Let’s take a look at how to monitor packets passing through the iptables firewall.

Initial configuration

Configure rsyslog to use a log file /var/log/firewall_trace.log for tracing the firewall.

$ cat << EOF | sudo tee /etc/rsyslog.d/01-firewall_trace.conf
# Log messages generated by iptables firewall to file
if $syslogfacility-text == 'kern' and $msg contains 'TRACE' then /var/log/firewall_trace.log
# stop processing it further
& stop
EOF

Apply rsyslog configuration.

$ sudo systemctl restart rsyslog

Rotate the log file to save disk space.

$ cat << EOF | sudo tee /etc/logrotate.d/firewall_trace.conf
/var/log/firewall_trace.log
{
  rotate 7
  daily
  missingok
  notifempty
  delaycompress
  compress
  postrotate
  invoke-rc.d rsyslog rotate > /dev/null
  endscript
}
EOF

You should be sure to rate these logs hourly by size or send them to an external logging service, which I highly recommend.

How to track incoming packages

Use raw and PREROUTING to monitor packets coming in on any network interface.

$ sudo iptables -t raw -A PREROUTING -p tcp --destination 1.2.3.4 --dport 443 -j TRACE

Let’s see the raw table

$ sudo iptables -t raw -L -v -n --line-numbers

Chain PREROUTING (policy ACCEPT 3501 packets, 946K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1      468 28159 TRACE      tcp  --  *      *       0.0.0.0/0            1.2.3.4       tcp dpt:443
Chain OUTPUT (policy ACCEPT 885 packets, 695K bytes)
num   pkts bytes target     prot opt in     out     source               destination

The trail to the internal network will look like this.

[...]
Jul 18 18:33:27 cerberus kernel: [68907.892027] TRACE: raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=00:15:17:c3:a1:aa:00:15:17:c3:fb:07:01:00 SRC=172.69.63.16 DST=1.2.3.4 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64783 DF PROTO=TCP SPT=62598 DPT=443 SEQ=234589096 ACK=404477568 WINDOW=82 RES=0x00 ACK URGP=0
Jul 18 18:33:27 cerberus kernel: [68907.892093] TRACE: mangle:INPUT:policy:1 IN=eth0 OUT= MAC=00:15:17:c3:a1:aa:00:15:17:c3:fb:07:01:00 SRC=172.69.63.16 DST=1.2.3.4 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64783 DF PROTO=TCP SPT=62598 DPT=443 SEQ=234589096 ACK=404477568 WINDOW=82 RES=0x00 ACK URGP=0
Jul 18 18:33:27 cerberus kernel: [68907.892113] TRACE: filter:INPUT:rule:6 IN=eth0 OUT= MAC=00:15:17:c3:a1:aa:00:15:17:c3:fb:07:01:00 SRC=172.69.63.16 DST=1.2.3.4 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64783 DF PROTO=TCP SPT=62598 DPT=443 SEQ=234589096 ACK=404477568 WINDOW=82 RES=0x00 ACK URGP=0
Jul 18 18:33:27 cerberus kernel: [68907.892150] TRACE: raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=00:15:17:c3:a1:aa:00:15:17:c3:fb:07:01:00 SRC=172.69.63.16 DST=1.2.3.4 LEN=40 TOS=0x00 PREC=0x00 TTL=56 ID=64784 DF PROTO=TCP SPT=62598 DPT=443 SEQ=234589096 ACK=404477569 WINDOW=82 RES=0x00 ACK RST URGP=0
[...]

Map the filyer table, INPUT chain, rule number 6, which will accept bound and established connections.

$ sudo iptables -t filter -L INPUT 6 -v -n --line-numbers

6     979K  851M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Remove the first rule in the raw table, the PREROUTING chain.

$ sudo iptables -t raw -D PREROUTING 1

How to track outgoing packets

Use raw table and OUTPUT to keep track of locally generated packets.

$ sudo iptables -t raw -A OUTPUT -p tcp --destination 8.8.8.8 --dport 53 -j TRACE

$ sudo iptables -t raw -A OUTPUT -p udp --destination 8.8.8.8 --dport 53 -j TRACE

Let’s see the raw table

$ sudo iptables -t raw -L -v -n --line-numbers

Chain PREROUTING (policy ACCEPT 1281 packets, 422K bytes)
num   pkts bytes target     prot opt in     out     source               destination
Chain OUTPUT (policy ACCEPT 379 packets, 324K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 TRACE      tcp  --  *      *       0.0.0.0/0            8.8.8.8              tcp dpt:53
2        0     0 TRACE      udp  --  *      *       0.0.0.0/0            8.8.8.8              udp d

The trail to the external DNS server will look like this.

[...]
Jul 18 18:48:38 cerberus kernel: [69819.286907] TRACE: raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=1.2.3.4 DST=8.8.8.8 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=27373 PROTO=UDP SPT=45407 DPT=53 LEN=58 UID=2018 GID=2018
Jul 18 18:48:38 cerberus kernel: [69819.286922] TRACE: nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=1.2.3.4 DST=8.8.8.8 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=27373 PROTO=UDP SPT=45407 DPT=53 LEN=58 UID=2018 GID=2018
Jul 18 18:48:38 cerberus kernel: [69819.286929] TRACE: filter:OUTPUT:rule:7 IN= OUT=eth0 SRC=1.2.3.4 DST=8.8.8.8 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=27373 PROTO=UDP SPT=45407 DPT=53 LEN=58 UID=2018 GID=2018
Jul 18 18:48:38 cerberus kernel: [69819.286939] TRACE: nat:POSTROUTING:policy:2 IN= OUT=eth0 SRC=1.2.3.4 DST=8.8.8.8 LEN=78 TOS=0x00 PREC=0x00 TTL=64 ID=27373 PROTO=UDP SPT=45407 DPT=53 LEN=58 UID=2018 GID=2018
[...]

Display the filter table, in the OUTPUT chain, rule number 7, which will everywhere accept outgoing udp connections on port 53.

$ sudo iptables -t filter -L OUTPUT 7 -v -n --line-numbers

7     2982  223K ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0

Remove the first and second rule in the raw table, the OUTPUT chain.

$ sudo iptables -t raw -D PREROUTING 1

$ sudo iptables -t raw -D PREROUTING 2