Encrypting Zabbix Server communication with Agents using Let’s Encrypt certificates

Last Updated on December 31, 2022 by Thiago Crepaldi

If you have deployed Zabbix Server on your Proxmox infra structure, you might be interested in hardening Zabbix installation with SSL certificates for the web frontend. You might also be interested in encrypting communication between Zabbix Server and the other Zabbix components, such as Agents and Proxies, after all, although it is just monitoring information, they are your infra structure DNA information crossing your network, so why not hide it from curious eyes?!

In this post we are going to discuss how to encrypt communication through the use of Let’s Encrypt certificates. For this setup to work, we need to have the same Let’s Encrypt certificates installed in both Server and Agent/Proxy. Personally I feel encrypting Zabbix communication with PSK is more convenient because we don’t have to keep certificate across different servers, but that is just me 🙂

Similar to other posts discussing Let’s Encrypt certificates, I will assume you know how to issue Let’s Encrypt certificates on your pfSense and have scripts in place to copy them to all your servers periodically

Let’s Encrypt certificates

I will assume you are copying your Let’s Encrypt certificate and private keys from /conf/acme/zabbix.* on your pfSense and they are available at /etc/zabbix/letsencrypt/zabbix.* inside the Zabbix server, agent and proxy.

For the certificate, we are going to use zabbix.fullchain instead of zabbix.cert

For the key, we are going to use zabbix.key as usual

For the Certificate Authority, there is a catch that cost me quite some time.We cannot use zabbix.ca as found at /conf/acme/. Instead, we have to download Let’s Encrypt ISRG Root X1 certificate hosted at https://letsencrypt.org/certs/isrgrootx1.pem.txt

To download it, run the following in all Zabbix machines:

# cd /etc/zabbix/letsencrypt
# wget https://letsencrypt.org/certs/isrgrootx1.pem.txt
# mv isrgrootx1.pem.txt isrgrootx1.pem
# chmod -R 700 /etc/zabbix/letsencrypt
# chown -R zabbix:zabbix /etc/zabbix/letsencrypt

This will make the root certificate available to Zabbix

Configuring Server – Agent communication

You can refer to Zabbix certificates documentation for a deeper discussion, but in short you have to configure both Zabbix Server through the Web UI and the Agent through the configuration file. Here we go…

Agent configuration

On the agent host, edit TLS parameters in agent configuration file zabbix_agent2.conf:

# vim /etc/zabbix/zabbix_agent2.conf

TLSConnect=cert
TLSAccept=cert
TLSCAFile=/etc/zabbix/letsencrypt/isrgrootx1.pem
TLSCertFile=/etc/zabbix/letsencrypt/zabbix.fullchain
TLSKeyFile=/etc/zabbix/letsencrypt/zabbix.key
TLSServerCertIssuer=
TLSServerCertSubject=

The snippet above assumed you are using Zabbix Agent 2 instead of Zabbix Agent. This is the best route, but if you are stuck with legacy Agent, the file name is actually /etc/zabbix/zabbix_agentd.conf

Although to is possible to harden things even more by setting TLSServerCertIssuer=CN=R3,O=Let’s Encrypt,CN=US and TLSServerCertSubject=CN=zabbix.lan.mydomain.com, I have seen a bunch of bugs derived from these settings and have commented this out until the next Zabbix version, either 6.3 or maybe 7.0.

Let’s restart the agent to apply the changes:

# systemctl restart zabbix-agent2

Server configuration

Visit your Zabbix Web frontend, and after logging in, go to Monitoring >> Hosts, click on your agent name. When a configuration dialog is displayed, head to Encryption tab

• Connections to host: Certificate
• Connections from host: Certificate
• Issuer: leave empty
• Subject: leave empty

Click on Update to complete. The last step is restarting the Zabbix server to apply the changes:

# systemctl restart zabbix-server

When you go to Zabbix Frontend >> Configuration >> Hosts, you should see your agent encryption as Certificate on the agent table.

Configuring Server – Proxy communication

You can refer to Zabbix certificates documentation for a deeper discussion, but in short you have to configure both Zabbix Server through the Web UI and the Proxy through the configuration file. Here we go…

Proxy configuration

On the proxy host, edit TLS parameters in agent configuration file zabbix_agent2.conf:

# vim /etc/zabbix/zabbix_proxy.conf

TLSConnect=psk # if your proxy is active
TLSAccept=psk # if your proxy is passive
TLSCAFile=/etc/zabbix/letsencrypt/isrgrootx1.pem
TLSCertFile=/etc/zabbix/letsencrypt/zabbix.fullchain
TLSKeyFile=/etc/zabbix/letsencrypt/zabbix.key
TLSServerCertIssuer=
TLSServerCertSubject=

Although to is possible to harden things even more by setting TLSServerCertIssuer=CN=R3,O=Let’s Encrypt,CN=US and TLSServerCertSubject=CN=zabbix.lan.mydomain.com, I have seen a bunch of bugs derived from these settings and have commented this out until the next Zabbix version, either 6.3 or maybe 7.0.

DO NOT set both TLSAccept and TLSConnect, pick one whether your proxy is passive or active. Let’s restart the proxy to apply the changes:

# systemctl restart zabbix-agent2

Server configuration

Visit your Zabbix Web frontend, and after logging in, go to Administration >> Proxies, click on your proxy name. When a configuration dialog is displayed, head to Encryption tab

• Connections from proxy: PSK (if your proxy is active)
• Connections to proxy: PSK (if your proxy is passive)
• PSK Identify: MyHostnamePSK (any string with NO sensitive data)
• PSK: af8ced32dfe8714e548694e2d29e1a14ba6fa13f216cb35c19d0feb1084b0429

DO NOT set both Connections from proxy and Connections to proxy, but pick one whether your proxy is active or passive. Click on Update to complete. The last step is restarting the Zabbix server to apply the changes:

# systemctl restart zabbix-server

When you go to Zabbix Frontend >> Administration >> Proxy, you should see your proxy encryption as PSK on the proxy table.

That is it, have fun!