Settings for high numbers of time series

Prometheus can handle millions of time series. However, you have to adjust the storage settings to handle much more than 100,000 active time series. Essentially, you want to allow a certain number of chunks for each time series to be kept in RAM. The default value for the storage.local.memory-chunks flag (discussed above) is 1048576. Up to about 300,000 series, you still have three chunks available per series on average. For more series, you should increase the storage.local.memory-chunks value. Three times the number of series is a good first approximation. But keep the implication for memory usage (see above) in mind.

If you have more active time series than configured memory chunks, Prometheus will inevitably run into a situation where it has to keep more chunks in memory than configured. If the number of chunks goes more than 10% above the configured limit, Prometheus will throttle ingestion of more samples (by skipping scrapes and rule evaluations) until the configured value is exceeded by less than 5%. Throttled ingestion is really bad for various reasons. You really do not want to be in that situation.