What scrape_interval should be set
First of all, it is recommended to use global config to set interval.
In terms of resource usage and resolution of data, 15 – 60 sec is recommended in every page.
If the more detailed resolution of metrics is needed, that should be realized at application level. Application should observe the value and record the min / max of metrics of the specific duration and export it. All of the points metrics are not needed to be stored on the Prometheus storage.
What metrics should be observed for Prometheus
For memory
As they say, 3kB is used per time series data on Prometheus. So the following data should be monitored.
prometheus_tsdb_head_series: Total number of series in the head block
prometheus_tsdb_head_samples_appended_total: Total number of appended samples.
The following is the actual value of memory usage
process_resident_memory_bytes: RSS of prometheus process
For DiskUsage
The total of the following size will be the disk size Prometheus is using.
prometheus_tsdb_storage_blocks_bytes: The number of bytes that are currently used for local storage by all blocks
prometheus_tsdb_head_chunks_storage_size_bytes: Size of the chunks_head directory
prometheus_tsdb_wal_storage_size_bytes:Size of the write log directory.
