Encrypting TCP traffic between rsyslog and Graylog

All my homelab servers and containers send their logs to a local Graylog instance. I wanted my VPS to do that as well — but sending unencrypted logs over the internet is not a good idea.

UDP (unsecure)

Lets first look at how we can send unencrypted logs over UDP from rsyslog to Graylog. That is fine on a local network where the clients are trusted, and the latency is low.

First, create an input in Graylog of type Syslog UDP:

Then configure our clients to send their log data to this input — create file /etc/rsyslog.d/60-graylog.conf:

*.* @graylog.lan.uctrl.net:5140;RSYSLOG_SyslogProtocol23Format

This means; send the logs, over UDP to port 5140 on host graylog.lan.uctrl.net and what format to use. Remember to change the hostname to your Graylog instance.

Easy peasy, but for secure TCP traffic there is a bit more involved.

TCP and TLS (secure)

For my VPS in Germany to send its logs to my local Graylog server, the traffic must be encrypted.

Graylog

Now we need a Syslog TCP input in Graylog. I’m not configuring TLS here because I want my haproxy load balancer to take care of that. More on that later…

Router

The home router now needs to be configured to allow and forward traffic on TCP port 5140 to the haproxy load balancer.

• Interface: WAN
• Protocol: TCP
• Remote port: 5140
• Destination: 10.0.0.10:5140

haproxy

Next, haproxy needs to listen on TCP port 5140 and to handle the TLS, so in /etc/haproxy/haproxy.cfg:

listen syslog
    bind :::5140 v4v6 ssl crt /etc/ssl/services/homelab.no.pem
    mode tcp
    option tcplog
    timeout client 30m
    timeout server 30m

    acl host_allowed src x.x.x.x
    tcp-request connection reject if !host_allowed

    server graylog-tcp graylog.lan.uctrl.net:5140 check

• I’m using Let’s Encrypt to generate a valid certificate.
• The connection will get rejected if the source IP address is not my VPS.

rsyslog

Create the file /etc/rsyslog.d/60-graylog.conf, but now with a bit more configuration:

$ActionResumeInterval 5
$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1     # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g       # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on     # save messages to disk on shutdown
$ActionQueueType LinkedList       # run asynchronously
$ActionResumeRetryCount -1        # infinite retries if host is down

$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer homelab.no

*.* @@homelab.no:5140;RSYSLOG_SyslogProtocol23Format

First, we are configuring some queuing, since the logs are getting sent over the internet. Next, we are setting up the certificate and TLS. We need to point to a certificate authority file, on Ubuntu it’s at /etc/ssl/certs/ca-certificates.crt but that may vary on other distros. We are also setting the permitted peer, in my case — homelab.no.

Notice the double @ (at) before the hostname on the last line? That instructs rsyslog to use TCP instead of UDP for sending data. Again remember to change the hostnames based on your setup.

Finito!

And that was it! With the router port open, both rsyslog and haproxy reloaded — I began seeing VPS system logs in my Graylog instance 🙂