Sending DNS data to Spamhaus

Introduction

A passive DNS system is obviously worthless without the data accumulated in it. Collection of such data takes place using DNSTAP and shipping data to our systems, that will then elaborate it and put it into the system. This part of the documentation is intended to partners who want do ship us their DNS resolution data so that they can become part of the data corpus available through our Passive DNS.

What is DNSTAP ?

In order to collect DNS packets that a Passive DNS Sender will send to Spamhaus Technology, a DNS Resolver must be instructed to save DNS packets that it exchanges with an upstream DNS servers in files with a specific format.

The format of the files is called DNSTAP: a flexible, structured binary log format for DNS software. It uses Protocol Buffers to encode events that occur inside DNS software in an implementation-neutral format.

You can find more information by reading this website but the most important thing to remember is that the file will NOT contain the packets that the resolver sends to the client, thus preventing any personally indefinable information (PII) to be shared with anyone.

The DNS resolver, if configured appropriately, will only record the public queries that will be sent and received from its upstreams.

How to create the DNSTAP files

Your DNS resolver will need to be configured in order to enable the DNSTAP logging (This part will be explained later in this document). The tool that will connect to your resolver, read the DNSTAP flow and store it with the aforementioned DNSTAP format is called fstrm.

The sources of this tool can be found on github

How to send the DNSTAP files to Spamhaus

The files will need to be rotated every minute. fstrm will take care of this. To sync the files to our servers you will use rsync over SSH. Spamhaus will provide you with a username and you will send us a specific SSH public key in order to be able to connect and send the data in a private and secure manner.

You will also need to manage retries (in case of network problems) and manage the growth of your files queue when any network problems persist.

We have some automation ready for you to use or adapt, according to your needs.

Installation - Resolver Configuration

In order to enable the DNSTAP features on your resolver, you will need to add some configuration options and set it up appropriately.

The configuration changes depends on the resolver that you’ll be using.

Let’s dig into some implementation details.

AppArmor considerations

It may be possible that your system uses AppArmor. Since normally writing on a custom socket is not permitted, you’ll need to modify the AppArmor profile for your resolver (if present) to allow this.

You’ll need to edit the profile file associated with your resolver, in example:

  • /etc/apparmor.d/usr.sbin.named for Bind

  • /etc/apparmor.d/usr.sbin.unbound for Unbound

  • … etc

And add the following line to the file:

  /var/lib/spamhaus/dnstap.sock rw,

Finally reload the profile with:

# apparmor_parser -r /etc/apparmor.d/usr.sbin.named

or, if using Unbound:

# apparmor_parser -r /etc/apparmor.d/usr.sbin.unbound

Please change and adapt the profile name according to your resolver.

Fail to do that and the setup will not work.

Configuring Bind 9

The Bind9 online documentation to refer to is at this link.

Please note that you will need at least BIND 9.11.

You have to add the following configuration stanza:

        dnstap {
            auth response;
            forwarder response;
            resolver response;
        };
        dnstap-identity "IDENTIFIER";
        dnstap-output unix "/var/lib/spamhaus/dnstap.sock";

The file to edit depends on your Linux distribution. It is usually located in /etc/named.conf.options

Replace IDENTIFIER with your LOGIN_USER name as indicated in the email you will have received from Spamhaus.

And then create the directory:

# mkdir /var/lib/spamhaus
# chown bind.bind /var/lib/spamhaus

Restart your resolver when done.

Configuring Unbound

You have to add this to your configuration:

dnstap:
    dnstap-enable: yes
    dnstap-socket-path: "/var/lib/spamhaus/dnstap.sock"
    dnstap-send-identity: yes
    dnstap-send-version: yes
    dnstap-log-resolver-response-messages: yes
    dnstap-log-forwarder-response-messages: yes
    dnstap-log-client-query-messages: no

It’s usually convenient to add the previous items to a single file in /etc/unbound/unbound.conf.d/dnstap.conf but it depends on your distribution.

And then create the directory:

# mkdir /var/lib/spamhaus
# chown unbound.unbound /var/lib/spamhaus

Restart your resolver when done.

Configuring Knot

The relevant knot documentation can be found on this page.

Please be sure to run at least version 5.x. Use the instructions here to get the latest package.

In the “modules” section of your kresd.conf file, add the following:

    dnstap = {
        socket_path = "/var/lib/spamhaus/dnstap.sock",
        log_responses = true
    }

And then create the directory:

# mkdir /var/lib/spamhaus
# chown knot.knot /var/lib/spamhaus

Restart your resolver when done.

Configuring PowerDNS Recursor

To configure PowerDNS you can have a look at this document.

Be sure to use version 4.3.0+ compiled with “–enable-dnstap”

Create a file /etc/powerdns/dnstap.lua and paste inside it the following line:

dnstapFrameStreamServer("/var/lib/spamhaus/dnstap.sock", { logQueries=false, logResponses=true })

Now edit the recursor.conf and add

lua-config-file=/etc/powerdns/dnstap.lua

And then create the directory:

# mkdir /var/lib/spamhaus
# chown pdns.pdns /var/lib/spamhaus

Restart your resolver when done.

Configuring dnsdist

Be sure to use version 1.3.0+ compiled with libfstrm

Edit the file /etc/dnsdist/dnsdist.conf and insert the following lines:

dnstap=newFrameStreamUnixLogger("/var/lib/spamhaus/dnstap.sock")
addResponseAction(AllRule(), DnstapLogResponseAction("IDENTIFIER", dnstap))

Replace IDENTIFIER with your LOGIN_USER name as indicated in the email you will have received from Spamhaus.

And then create the directory:

# mkdir /var/lib/spamhaus
# chown _dnsdist._dnsdist /var/lib/spamhaus

Restart your resolver when done.

Stream collector setup (fstrm_capture)

The configuration of the DNS resolver will create a Unix socket listening on /var/lib/spamhaus/dnstap.sock (or any other path you may have configured) from which we will need to read the dnstap flow.

The tool that we will use is called fstrm_capture and it usually comes packaged for the bigger distributions. In Debian and Ubuntu the package is named fstrm-bin and in CentOS, RH or fedora (epel), the binary that we will need is in a package named fstrm-devel

NOTE: It’s highly suggested to run the latest version of fstrm. We’ve observed instability and crashes on older versions. Definitely, you shouldn’t use anything older than version 0.4.0

Create the spool directory

# mkdir -p /var/spool/spamhaus/dnstap/new
# mkdir /var/spool/spamhaus/dnstap/wait
# mkdir /var/spool/spamhaus/dnstap/lock
# mkdir /var/spool/spamhaus/dnstap/key

And remember to change its owner to the same user that your resolver is using. For example, if you use bind:

# chown bind.bind /var/spool/spamhaus/dnstap/new

The most appropriate command line to run this utility is the following:

/usr/bin/fstrm_capture \
    -t protobuf:dnstap.Dnstap \
    -u /var/lib/spamhaus/dnstap.sock \
    -w /var/spool/spamhaus/dnstap/new/spamhaus.cask.iad-cask-resolver-1-dnstap-%F-%T.dnstap \
    -s 60 \
    --gmtime

The options have the following reasons:

  • -t defines the frame streams content type

  • -u is the Unix socket path to read from. It must match the resolver option

  • -w is the file path to write Frame Streams data to. Notice that it contains a couple of strftime options because the file will be rotated and the filename must change

  • -s is the seconds before rotating the output file

  • --gmtime is used to filter -w path with strftime (UTC)

NOTE: The fstrm_capture needs to be run by the same user running your dns resolver (bind, named, unbound, nobody or whatever you configured on your system). If it won’t work, this is definitely the first thing to check, together with the permissions of the -u socket file.

If you manage to run fstrm_capture, every time you resolve a domain name through your dns server you should start seeing at least one dnstap file appear in your spool directory. If it doesn’t happen then you may have a problem. Please troubleshoot by:

  • checking if the resolver is working

  • if the resolver has opened the unix socket and check the permissions of such socket

  • check that the user running fstrm_capture has the permissions to read from such socket

  • check that the spool directory is writeable by the user running the tool

After a few minutes you should see more than one file appearing in such a directory.

systemd unit

What follows is an example systemd unit that you can use in order to run the capturing tool at boot and keep the daemon running.

[Unit]
Description=DNSTAP Sender
After=network.target nss-lookup.target

[Service]
Type=simple
ExecStart=/usr/bin/fstrm_capture -t protobuf:dnstap.Dnstap -u /var/lib/spamhaus/dnstap.sock -w /var/spool/spamhaus/dnstap/new/spamhaus.cask.iad-cask-resolver-1-dnstap-%%F-%%T.dnstap -s 60 --gmtime
User=-PLACE HERE THE USERNAME THAT YOUR RESOLVER IS USING-
TimeoutStopSec=5
KillMode=mixed
Restart=always
RestartSec=5s
LimitNOFILE=49152

[Install]
WantedBy=multi-user.target

Once you change the username to match the one your resolver is using, copy the file in

/etc/systemd/system/dnstap.service

and enable the service at boot time

# systemctl enable dnstap

Rotating DNSTAP files and preparing them for sending

To help you manage the rotation of files, you should create a bash script under ‘/usr/local/bin/dnstap-spool-maintenance’, with the following content:

#!/bin/bash 

array=(`ls -1 /var/spool/spamhaus/dnstap/new/* | sort -nr`)
unset 'array[0]'
for FILE in "${array[@]}"; do
  mv $FILE /var/spool/spamhaus/dnstap/wait/
done

Once its created, do the following:

# chmod +x /usr/local/bin/dnstap-spool-maintenance

and then add to ‘/etc/crontab’ the following line:

*  * * * * root test -x /usr/local/bin/dnstap-spool-maintenance && /usr/local/bin/dnstap-spool-maintenance || exit 0

The DNSTAP files will now be automatically moved to the “wait to be sent” directory every minute.

Sending the files

The DNSTAP files should be sent, as soon as they appear in the waiting directory, to Spamhaus Technology. An appropriate way to send the file is by a cron entry.

To send the files you will use rsync over SSH (both largely available for all linux flavours) using the remote host, remote port and login user that Spamhaus Technology will assign to you.

The first thing to do is generate a SSH key pair. A robust key can be generated with ssh-keygen -t ecdsa. Please email the public key to your point of contact in Spamhaus and you will receive the login user and the remote host in order to continue the configuration.

We suggest you move and rename the private key to /var/spool/spamhaus/dnstap/key/upload_key and then

# chmod 600 /var/spool/spamhaus/dnstap/key/upload_key

An example rsync command to send the files would be:

rsync \
    -q 
    -e "ssh -p $REMOTE_PORT -i /var/spool/spamhaus/dnstap/key/upload_key -o StrictHostKeyChecking=no -o HashKnownHosts=no" \
    --timeout=10 \
    --remove-source-files \
    /var/spool/spamhaus/dnstap/wait/*.dnstap \
    $LOGIN_USER@$REMOTE_SERVER:dnstap/

The REMOTE_SERVER , REMOTE_PORT and LOGIN_USER parameters will be provided by Spamhaus after receiving the public key.

A typical /etc/crontab line would be something like:

* * * * *       root test -x /usr/bin/rsync && rsync -q -e "ssh -p $REMOTE_PORT -i /var/spool/spamhaus/dnstap/key/upload_key -o StrictHostKeyChecking=no -o HashKnownHosts=no" --timeout=10 --remove-source-files /var/spool/spamhaus/dnstap/wait/*.dnstap $LOGIN_USER@$REMOTE_SERVER:dnstap/

Debian package

Spamhaus provides, upon request, the sources and the package of the whole process in a packet that is called spamhaus-dnstap-sender.

The package is a mainly a set of bash scripts and configuration files that automates (and adds more features) the process previously described.

Building from source

The package sources may be useful even if you don’t use Debian, as they may be easily portable to other distributions that don’t use the .deb packages, like RedHat, Fedora or CentOS.

If you’re on Debian/Ubuntu/Mint or any other Debian derivative, you may inspect the sources and compile the package yourself.

Compiling the package is not very difficult but you’ll need to set up the developer environment in Debian (this is out of scope so this item won’t be covered in this document, but very shortly the only thing that you should install is the dpkg-dev and the debhelper packages).

Once done, uncompress the sources and cd to the spamhaus-dnstap-sender directory that contains the required files.

In order to create the Debian package you will need to run a single command:

$ dpkg-buildpackage -b -uc

Once done you will find the Debian package in the upper directory.

...
dpkg-deb: building package 'spamhaus-dnstap-sender' in '../spamhaus-dnstap-sender_0.1.0_amd64.deb'.
...

Installing the package

Whether you have built the package on your own or you’re using a package provided by Spamhaus, you need to install it.

The package can be installed using dpkg with the following command (if you changed your working directory you may need to adjust the path):

$ dpkg -i ../spamhaus-dnstap-sender_0.1.0_amd64.deb

Should an error about the missing dependencies show up (and it’s very likely), you can run apt-get -f install in order to fix all the missing dependencies and finalize the package installation.

NOTE: The package will take care of installing fstrm-bin, rsync and sudo for you if they are not installed

...
...
Setting up spamhaus-dnstap-sender (0.1.0) ...

spamhaus-dnstap-collector
------------------------

This is the Spamhaus DNSTAP Collector package. It requires post-install configuration.

You should start looking at "/etc/default/spamhaus-dnstap-sender" and
setting up the proper variables.

Processing triggers for systemd (241-7~deb10u3) ...
...
...

Configuring the package

Once the package has been installed, you should generate a SSH key pair. The package has an handy script that does that for you. Just run:

$ spamhaus-dnstap-genkey

This script will generate the required keys. You should now mail the content of your public key (in /var/spool/spamhaus/keys/upload-key.pub) to your POC - point of contact - in Spamhaus in order to configure our side.

You will receive:

  • a login username

  • an endpoint ip address

  • a port number

You need to open the file /etc/default/spamhaus-dnstap-sender and modify the LOGIN_USER, the REMOTE_SERVER and the REMOTE_PORT in order to align the values to the data received by email.

The other very important parameter to edit in that file is DNSTAP_RUNAS. This is the user that will run the fstrm_capture binary that will connect to to the aforementioned unix socket. This value needs to be a system user that can access that socket. Usually it should match the DNS resolver user.

After setting up the configuration files, please run:

/usr/lib/spamhaus/dnstap/spamhaus-dnstap-setup

This script will take care to align the spool directory permissions to the proper selected user.

Before running the dnstap-sender scripts, you need to configure your DNS Resolver in order to activate DNSTAP and expose the unix socket. Please read the beginning of this document about how to do this.

Finally enable the dnstap-sender service:

systemctl enable spamhaus-dnstap-sender

and start it:

systemctl start spamhaus-dnstap-sender

RPM Package

Spamhaus provides, upon request, the sources and the package of the whole process in a packet that is called spamhaus-dnstap-sender.

The package is mainly a set of bash scripts and configuration files that automates and adds more features to the process previously described.

The package has been developed on CentOS 8.0 so the instructions for previous releases should be tested

Building from source

All commands should be executed as root.

In order to build it you should install the rpm-build package

dnf install -y rpm-build
# or yum install -y rpm-build

Then you should build the proper directory tree to build the sources:

cd; mkdir -p rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}

Then copy the tarball with the sources provided by spamhaus to the SOURCES dir:

cp spamhaus-dnstap-sender.tgz ~/rpmbuild/SOURCES

Finally, you need to extraxt a file in the tarball: the file is spamhaus-dnstap-sender/spamhaus-dnstap-sender.spec. The command to do this is:

tar -zxvf spamhaus-dnstap-sender.tgz spamhaus-dnstap-sender/spamhaus-dnstap-sender.spec

The .spec file is what you will need to build the package. The command is:

rpmbuild --target noarch -bb path/to/spamhaus-dnstap-sender.spec

And you’ll find the RPM package in your directory:

[root]# ls -1 *.rpm
spamhaus-dnstap-sender-0.1.0-1.noarch.rpm

Installing the package

In order to install the package, you only need to run

rpm -ivh spamhaus-dnstap-sender-0.1.0-1.noarch.rpm

This command may fail because of two missing dependencies. The first dependency is rsync and this is easily fixable with:

dnf install rsync
# or yum install rsync fstrm-devel

The other dependency is a bit more complicated. You may opt to compile the sources yourself or you may decide to install the fstrm-devel package from epel.

This is a short list of commands to have the package installed on your system:

dnf install epel-release
dnf update
dnf install rsync fstrm-devel

After this you can run the rpm -i mentioned earlier in this chapter to install the sender package.

Configuring the package

The setup process is very similar to the one shown for Debian so please refer to that chapter.

Notes for CentOS

The way the fstrm_camture program is started may interfere with selinux which must be disabled before running the sender.

If you see something like the following in your syslog or messages file:

spamhaus-dnstap-sender.service: Failed at step EXEC spawning /usr/bin/sudo: Permission denied

Then you should run setenforce 0 in order to fix this.

If your internal policy doesn’t allow you to do this, then feel free to modify the /usr/lib/systemd/system/spamhaus-dnstap-sender.service file. You will likely only need to remove the sudo invocation and change the User= parameter but this will prevent the systemd unit file to load the user from the configuration file located in /etc/default.

Checklist & Troubleshooting

Make sure that your resolver is properly configured and restarted. ps aux should show the process of your resolver. If it is not there, there is likely a configuration error. Inspect the syslog and act appropriately

Make sure that fstrm_capture is running. ps aux is your friend and it should show the running process. If it is not there then please check your syslog. Remember that fstrm_capture will run as DNSTAP_RUNAS user and such user will need read permissions on the Unix socket exposed by the dns resolver.

Make sure that the directory /var/spool/spamhaus/dnstap/new has files. It should contain a file that grows in size – the one that is being filled with the dnstap data coming from the resolver. There may be additional files, waiting to be picked up by cron, but no more than one or two.

If there are too many files either your cron daemon is off or there is something wrong with the set up. Please check your syslog.

The /var/spool/deteque/dnstap/wait directory should contain the files moved from /var/spool/spamhaus/dnstap/new and waiting to be delivered to Spamhaus through rsync. If the directory fills up with files, then there is a problem. Please check if rsync is working properly. It is run by cron and if an error pops up, cron should mail the system admin with an error. If rsync and cron are working, then please check that the remote settings in /etc/default/spamhaus-dnstap-sender are ok and check the network by telnetting to the REMOTE_SERVER on the TCP REMOTE_PORT