Eliminate unnecessary instances from EC2 Service Discovery in Prometheus

EC2 Service Discovery is very useful in the Prometheus configuration especially when the autoscale is used. However, some temporary or test servers which do not need to be monitored will be cause of unnecessary alerts. So, I added some settings to avoid it.

For example, if I would like to eliminate instances which have names including "test" or "dev", I’ll add the following configurations.

1    relabel_configs:
2    - source_labels: [__meta_ec2_tag_Name]
3      regex: .*test.*
4      action: drop
5    - source_labels: [__meta_ec2_tag_Name]
6      regex: .*dev.*
7      action: drop

As a result of the above, the Prometheus will ignore those nodes. So, if I would like to use some instances which are temporary or no need to be monitored, the names should include one of them like "test-app" or "dev-web-server"