Encrypting TCP traffic between rsyslog and Graylog
Table of contents
Lets first look at how we can send unencrypted logs over UDP from rsyslog to Graylog. That is fine on a local network where the clients are trusted, and the latency is low.
First, create an input in Graylog of type Syslog UDP:
Then configure our clients to send their log data to this input — create file
This means; send the logs, over UDP to port 5140 on host
graylog.lan.uctrl.net and what format to use. Remember to change the hostname to your Graylog instance.
Easy peasy, but for secure TCP traffic there is a bit more involved.
For my VPS in Germany to send its logs to my local Graylog server, the traffic must be encrypted.
Now we need a Syslog TCP input in Graylog. I’m not configuring TLS here because I want my haproxy load balancer to take care of that. More on that later…
The home router now needs to be configured to allow and forward traffic on TCP port 5140 to the haproxy load balancer.
- Remote port:
Next, haproxy needs to listen on TCP port 5140 and to handle the TLS, so in
listen syslog bind :::5140 v4v6 ssl crt /etc/ssl/services/homelab.no.pem mode tcp option tcplog timeout client 30m timeout server 30m acl host_allowed src x.x.x.x tcp-request connection reject if !host_allowed server graylog-tcp graylog.lan.uctrl.net:5140 check
- I’m using Let’s Encrypt to generate a valid certificate.
- The connection will get rejected if the source IP address is not my VPS.
Create the file
/etc/rsyslog.d/60-graylog.conf, but now with a bit more configuration:
$ActionResumeInterval 5 $WorkDirectory /var/spool/rsyslog # where to place spool files $ActionQueueFileName fwdRule1 # unique name prefix for spool files $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) $ActionQueueSaveOnShutdown on # save messages to disk on shutdown $ActionQueueType LinkedList # run asynchronously $ActionResumeRetryCount -1 # infinite retries if host is down $DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt $ActionSendStreamDriver gtls $ActionSendStreamDriverMode 1 $ActionSendStreamDriverAuthMode x509/name $ActionSendStreamDriverPermittedPeer homelab.no *.* @@homelab.no:5140;RSYSLOG_SyslogProtocol23Format
First, we are configuring some queuing, since the logs are getting sent over the internet. Next, we are setting up the certificate and TLS. We need to point to a certificate authority file, on Ubuntu it’s at
/etc/ssl/certs/ca-certificates.crt but that may vary on other distros. We are also setting the permitted peer, in my case —
Notice the double @ (at) before the hostname on the last line? That instructs rsyslog to use TCP instead of UDP for sending data. Again remember to change the hostnames based on your setup.
And that was it! With the router port open, both rsyslog and haproxy reloaded — I began seeing VPS system logs in my Graylog instance :)