Prometheus: How to solve blackbox exporter icmp (ping) failure

Jul 28, 2021 · 3 min read · prometheus blackbox-exporter ·

I set up my blackbox exporter to monitor some network appliances via ICMP but its RTT was always zero. I'll share how to investigate this problem and solve it.

First I added the following module setting to blackbox.yml because I've never used ICMP monitoring before.

1modules:
2    icmp:
3        prober: icmp
4        timeout: 5s

Then, I added the following setting to the Prometheus configuration file.

1  - job_name: 'icmp_check'
2    metrics_path: /probe
3    params:
4      module: [icmp]
5    static_configs:
6      - targets:
7        - 'target01.hoge'
8        - 'target02.hoge'

I restarted the Prometheus and the blackbox exporter and Prometheus got started to monitor the above two network appliances. Then, I searched its result by the following query on Prometheus's web UI.

1probe_icmp_duration_seconds{job="icmp_check", phase="rtt"}

However, the second one's value had not been captured correctly even though the first one was correct. Actually the 'rtt' was always zero. So, I tried to execute the ping command in the blackbox-exporter's container but the result was no problem. I was not sure why it happened so I decided to add the debug flag to the blackbox-exporter in my docker-compose.yml file like this.

 1 blackbox-exporter:
 2  container_name: blackbox-exporter 
 3  build: ./blackbox_exporter
 4  restart: always
 5  ports:
 6   - "9115:9115"
 7  command:
 8   - "--log.level=debug"
 9  sysctls:
10    net.ipv4.ping_group_range: "0 2147483647"

After restarting the blackbox-exporter, I got the following logs.

1ts=2021-07-01T03:00:00.890Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Beginning probe" probe=icmp timeout_seconds=5
2ts=2021-07-01T03:00:00.890Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Resolving target address" ip_protocol=ip6
3ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Resolved target address" ip=xxxx:xxxx:xxxx:xxxx::1
4ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Creating socket"
5ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Creating ICMP packet" seq=57536 id=40265
6ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Writing out packet"
7ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Error writing to socket" err="write udp [::]:560->[xxxx:xxxx:xxxx:xxxx::1]:0: sendto: cannot assign requested address"
8ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Probe failed" duration_seconds=0.001720556
9ts=2021-07-01T03:07:19.099Z

And I realized that the second one's FQDN was solved as an IPv6 address. It caused the error because Docker uses the IPv4 only as default. So, it couldn't create a socket to send ICMP packets. That's why I added the following setting to the blackbox.yml file to use IPv4 preferably.

1modules:
2    icmp:
3        prober: icmp
4        timeout: 5s
5        icmp:
6            preferred_ip_protocol: "ip4"

Then, I restarted the blackbox-exporter again and checked the logs.

1ts=2021-07-01T03:16:27.891Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Beginning probe" probe=icmp timeout_seconds=5
2ts=2021-07-01T03:16:27.891Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Resolving target address" ip_protocol=ip4
3ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Resolved target address" ip=xxx.xxx.xxx.xxx
4ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Creating socket"
5ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Creating ICMP packet" seq=39633 id=48931
6ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Writing out packet"
7ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Waiting for reply packets"
8ts=2021-07-01T03:16:27.997Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Found matching reply packet"
9ts=2021-07-01T03:16:27.997Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Probe succeeded" duration_seconds=0.106222366

The exporter used the IPv4 address as I expected. So, I searched its result by the following query on the web UI again.

1probe_icmp_duration_seconds{job="icmp_check", phase="rtt"}

Finally, the value has been captured correctly. Last, don't forget to turn off the debug log setting for blackbox-exporter to avoid filling up the disk.

1 blackbox-exporter:
2  container_name: blackbox-exporter 
3  build: ./blackbox_exporter
4  restart: always
5  ports:
6   - "9115:9115"
7  # command:
8  #  - "--log.level=debug"