Monitoring IT infrastructure with Zabbix server running on a Proxmox LXC container

Last Updated on December 30, 2022 by Thiago Crepaldi

After some time we accumulating VMs, containers, physical devices and keeping track of what is working as it should or not start to get challenging. Not rarely we start adding a bunch of homemade scripts to warns when some service dies, creating a management nightmare.

This is where Zabbix comes to the rescue! According to Wikipedia (by the way, how about donating a couple dollars to them after reading this post?!), “Zabbix is an open-source software tool to monitor IT infrastructure such as networks, servers, virtual machines, and cloud services.^[2] Zabbix collects and displays basic metrics.”

The way it works is using a central server with Zabbix server plus a database to collect and receive metris from zabbix agents installed in each device. Some metrics can be collected without an agent, such as connectivity through pings. Zabbix proxies can be used to offload zabbix server and intermediate metric collection on its behavior – which zabbix server will later sync with to grab the data.

The main selling point is that, according to zabbix.com, we can “get a single pane of glass view of your whole IT infrastructure stack”…

Installing Zabbix Server on LXC container

Had Zabbix published their software through LXC, this post would certainly be a lot shorter, but as can be seen at https://www.zabbix.com/container_images, they only provide Docker images… Because I have Proxmox VE running on my lab, and because I refuse to install Docker side by side with it, we are going to create a regular LXC container and install Zabbix on it.

The first step is creating a LXC container with the minimum requirements

• Debian 11 (Bullseye)
• 2 CPUs
• 8 GB of RAM
• 12 GB of Storage

If you think you will collect more than 1000 metrics/sec, visit the official Zabbix Requirements page and adjust the spec accordingly.

https://www.zabbix.com/download >> Zabbix Packages has detailed information on how to install it, but below you can find the summary for Zabbix 6.2. The steps below are valid based on the following assumptions:

• Zabbix version is 6.2
• Debian 11 Bullseye
• Zabbix Server + Frontend + Agent will be installed
• Postgres + TimescaleDB extension as database
• Apache2 as web server

Once the Debian is up and running (maybe with SMTP configured on it?), go to a terminal as root and make sure the OS locale is UTF-8. This is needed to make sure Zabbix Frontend will not cry about it later on.

# dpkg-reconfigure locales

Select en_US.UTF-8 UTF-8 in the dialog, click Ok and then select en_US.UTF-8 as the default locale for the system environment and finish the configuration clicking in Ok again.

The next step is installing the Zabbix repo on your Debian. The snippet below is from zabbix.com/download >> Zabbix Packages section

# wget https://repo.zabbix.com/zabbix/6.2/debian/pool/main/z/zabbix-release/zabbix-release_6.2-4%2Bdebian11_all.deb
# dpkg -i zabbix-release_6.2-4+debian11_all.deb
# apt update

Now it is time to install Zabbix itself. It includes PHP, Zabbix server, Zabbix frontend, Zabbix Agent2 and the web server and postgres connectors

# apt install -y zabbix-server-pgsql zabbix-frontend-php php7.4-pgsql zabbix-apache-conf zabbix-sql-scripts zabbix-agent2

We still need to install Postgres and an extension called TimescaleDB which does automatic partitioning across time and space. You can opt to just go with plain Postgres or MySQL. More details can be found at TimescaleDB page, but if you are lazy, just run:

# apt install gnupg postgresql-common apt-transport-https lsb-release wget

Run the PostgreSQL repository setup script:

# /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh

Add the TimescaleDB third party repository:

# echo "deb https://packagecloud.io/timescale/timescaledb/debian/ $(lsb_release -c -s) main" | tee /etc/apt/sources.list.d/timescaledb.list

Install Timescale GPG key

# wget --quiet -O - https://packagecloud.io/timescale/timescaledb/gpgkey | apt-key add -

Update your local repository list and install TimescaleDB:

# apt update
# apt install timescaledb-tools timescaledb-2-loader-postgresql-14=2.8.1~debian11 timescaledb-2-2.8.1-postgresql-14=2.8.1~debian11
systemctl restart postgresql

The snippet above pinned the Postgres version to 14 and TimescaleDB 2.8.1 because this is the latest Zabbix 6.2 supports – that can change over time, though.

At this point, Zabbix and the database are up and running, but not functional yet. Let’s populate the database with some template values:

# su - postgres -c "createuser --pwprompt zabbix"
# su - postgres -c "createdb -O zabbix zabbix --locale en_US.UTF-8 --encoding=UTF8 --template template0"

Note UTF-8 was explicitly specified in the command above. This is needed to prevent encoding issues with Zabbix Frontend later on. The same reason why we ensured the OS locale was UTF-8 in a previous step.

Let’s import the initial schema and data. You will be prompted to enter your newly created password:

# zcat /usr/share/zabbix-sql-scripts/postgresql/server.sql.gz | su - zabbix -c "psql zabbix"

Now we need to go to the postgres terminal and enable TimescaleDB for the zabbix database:

# psql -U zabbix -h localhost
zabbix=> \c zabbix
zabbix=> CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;
zabbix=> \q

TimescaleDB is enabled with default settings. Let’s automatically tune it based on your server settings:

# timescaledb-tune --yes
# systemctl restart postgresql

Additionally to the template data for Postgres, we have to do the same for TimescaleDB:

# cat /usr/share/zabbix-sql-scripts/postgresql/timescaledb.sql | su - zabbix -c "psql zabbix"

We are almost there, now we have to configure Zabbix with the database credentials. Edit file /etc/zabbix/zabbix_server.conf

DBPassword=<my_zabbix_password>

Start Zabbix server and agent processes and make them start at system boot

# systemctl restart zabbix-server zabbix-agent apache2
# systemctl enable zabbix-server zabbix-agent apache2

Visit http://ZABBIX_IP/zabbix/ or http://zabbix_hostname.mydomain.com/zabbix. Note that you must use HTTP and add a /zabbix at the of the URL. We will fix these “limitations” in a separate post.

When you visit Zabbix frontend, you will see a wizard. Go through it and make sure that:

• the default language is English (en_US) before clicking Next step
• all pre-requisites are marked as OK before clicking Next step
• zabbix’s user password is set and click Next step
• zabbix server name and timezone are set and click Next step
• pre-installation summary looks ok and click Next step
• click Finish to complete the configuration wizard

You will be redirected to the login page:

• User: Admin
• Password: zabbix

Let’s change the password. Follow to Administration >> Users and click Admin user. Next, click Change Password, type the new one and finish by clicking Update.

That is it, Zabbix server and agent are running on this server. Go to Monitoring >> Dashboard to see what you got so far. The dashboard should show some basic information about the zabbix server itself (this is why you installed the agent on the same machine as the server). You should see CPU utilization, etc

In future posts we are going to dive deeper on the monitoring capabilities itself, how to harden your Zabbix installation and go through Zabbix agents installation in your pfSense, web server, etc.

In the meantime, have fun going through the Monitoring tab of the Web frontend, in special at Dashboard, Problems, Hosts, Latest data sections. They have interesting information about your Zabbix server’s current status. More fun stuff can be found at Zabbix documentation.