BackupPC is a very capable open-source enterprise-grade backup system that is readily available for most Linux distributions, e.g., Ubuntu, Debian and Fedora. However, since most computers today are behind a NAT router or firewall, this completely breaks the ping command that a remote BackupPC server uses to check if a host client is alive or reachable. The most common solutions which allow BackupPC to ping host clients that are behind a NAT router or firewall are discussed here.
BackupPC, being open source and free, is great in many aspects and thus has also become the backup system of choice in many enterprise environments as well as home networks. Although it usually works out-of-the box in most popular Linux distributions, such as Ubuntu, Debian and Fedora, some quirks do exist. For example, the broken GUI interface caused by Apache. Another example, which is discussed here, is the ping system that BackupPC uses to check if a host computer client is alive. In addition, it will be argued why some common solutions provided on the internet are better than others based on experience and logic.The scenario that is particularly considered is the case of a remote BackupPC server attempting to backup multiple computers that sit behind a NAT router with firewall, as illustrated in the picture above.
By default, BackupPC uses the ping
command ($PingCmd, /bin/ping) to check if a host client is alive and reachable, if it is,
BackupPC will backup the host according to a schedule. If no ping
response is received, BackupPC will eventually add the host to a
prioritized backup schedule, which means that BackupPC will ping it each
time it wakes up and issue the backup command as soon as the host is
alive again. This behavior certainly works great in a local area
network, unfortunately, it completely breaks down if BackupPC attempts
to ping hosts behind a NAT router or firewall. This is also the default
scenario for users wishing to have a remote BackupPC server to backup multiple of
their home computers that, today, almost always will be behind a NAT
router. Fortunately, there exist ways to fix the broken BackupPC ping
command to allow it to properly ping host clients to check if they are
alive. It is just too bad that the internet is flooded with either poor
alternatives, or highly questionable and strange answers, such as,
"disable ping", or "make your router always respond to pings". Here, we
suggest the
best way to ping hosts behind NAT routers or firewalls and also discuss the alternatives to "fix" the ping issue.
Using hping3 is arguably by far the best and most versatile way to "ping" hosts to
check if they are alive. While there are plenty of tools available to
send custom TCP/IP packets, hping3 has proven to work out-of-the box with BackupPC. The reason being that BackupPC uses the two regular expressions:
In the following example, a host client is reachable through SSH on port 17000 which is forwarded on the NAT. Running the regular PingCmd (/bin/ping) yields,
Use hping3 to send custom TCP/IP packets to ping hosts behind firewalls and NATs
Using hping3 is arguably by far the best and most versatile way to "ping" hosts to
check if they are alive. While there are plenty of tools available to
send custom TCP/IP packets, hping3 has proven to work out-of-the box with BackupPC. The reason being that BackupPC uses the two regular expressions:- rtt\s*min\/avg\/max\/mdev\s*=\s*[\d.]+\/([\d.]+)\/[\d.]+\/[\d.]+\s*(ms|usec)to look for a response from its $PingCmd. hping3 by coincidence (or not) actually returns data that matches the regular expressions.
- time=([\d.]+)\s*(ms|usec)
In the following example, a host client is reachable through SSH on port 17000 which is forwarded on the NAT. Running the regular PingCmd (/bin/ping) yields,
$ ping bhost1 -c 1No wonder, as ICMP ping replies are disabled for this NAT. However, running hping3 gives us,
PING bhost1 (1.1.1.1) 56(84) bytes of data.
--- bhost1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
$ sudo hping3 bhost1 -p 17000 -c 1 -SGreat! BackupPC actually understands the reply and a host that is offline is actually caught as well!
HPING bhost1 (eth1 1.1.1.1): S set, 40 headers + 0 data bytes
len=46 ip=1.1.1.1 ttl=55 DF id=0 sport=17000 flags=SA seq=0 win=29200 rtt=3.5 ms
Setting up hping3 with BackupPC in Ubuntu and Debian
hping3 is readily available in most Linux distributions and setting it up is easy through the following 3 steps.
While the above only shows an example for a host that is reachable through SSH, the same can be used for hosts that are reachable through rsyncd, smb, ftpd, or any other protocols, as long as the listening port itself is properly forwarded on the NAT router.
Using hping3 has so far solved the issues in my varieties of backup constellations while using BackupPC. However, there are of course alternatives. Two of the absolutely most commonly suggested ones are discussed below. The reason being that I do not understand why anyone would settle for these incomplete workarounds.
Pros: Simple and straight-forward to implement and works well with hosts that are almost always online and alive, i.e., hosts that would have responded to a ping request anyway.
Cons: This solution has obvious problems with hosts that are only occasionally online. By disabling ping checking, BackupPC thinks that the client is always alive and available and thus will strictly follow its schedule and blackout periods. In the worst case, clients that are only online within the blackout periods will never be backed up! Not optimal at all!
It is debatable if one should disable ICMP ping responses or not. Some claim that it increases security and others claim that this only adds pseudo security as it is easy to discover if the destination is online anyway. This debate is outside of the scope of this article, however, for a remote BackupPC server trying to access clients behind the firewall, this is a nonsense solution since it completely breaks down regardless of how ICMP ping responses are configured as seen in the pros-and-cons list below.
Pros: It is usually simple to enable ICMP ping responses and similar to the first solution, this one will work if all hosts behind the router are almost always online.
Cons: Again the problems are obvious here as this is almost the same as disabling ping checking on BackupPC altogether, since a NAT router is almost always online anyway. In addition, this does not address the issue of backing up multiple hosts behind a NAT router, since it is impossible to discover if each individual host is alive or not through ICMP ping packets. Again, in the worst case, one can end up with clients that are never backed up if they are online reachable during the configured blackout periods!
Pros: In contrast to the above two alternatives, this one actually does allow BackupPC to correctly determine if individual hosts behind a NAT router is reachable and alive.
Cons: The requirement that the clients need to establish a connection to BackupPC kind of defies the purpose the autonomous design. In addition, this may work great on Linux, *nix or Mac OS hosts, but what about Windows? One should not be forced to install cygwin just to be able to use BackupPC (encryption of the data stream can be achieved in other ways than SSH). This workaround is also more messy to maintain, since we would need to verify that the SSH connection is properly established on boot. A small thing perhaps, but the added complexity does not make sense when simpler ones exist. The beauty of BackupPC is after all that it does not explicitly require each client to install client software!
In conclusion, through my own experiences, the use of hping3 on a remote BackupPC server for host clients behind NAT routers and firewalls is the best way to ping-check. Although I respect that there certainly are other alternatives out there, they do not make much sense to me, but I may have missed some fundamental values in the BackupPC and *nix community...
In any case, I hope that this post will be of help to some of you out there who are wondering how to ping BackupPC hosts behind a NAT router or firewall!
- For Ubuntu and Debian we first need to install it. Using the terminal this is achieved by,
$ sudo apt-get install hping3
- Secondly one needs to execute hping3 as root. Thus, we must modify the sudoers list to allow BackupPC to run hping3 as root without being prompted for a password. This can be done by editing the file /etc/sudoers and append,
# Allow backuppc to run hping3 without a password
backuppc ALL=NOPASSWD: /usr/sbin/hping3 - Finally, we need to modify the PingCmd in BackupPC. This can be done through the web GUI under the Backup Settings heading. Cross the Override button and edit it to be,
/usr/bin/sudo /usr/sbin/hping3 $host -p <port_number> -c 1 -S
While the above only shows an example for a host that is reachable through SSH, the same can be used for hosts that are reachable through rsyncd, smb, ftpd, or any other protocols, as long as the listening port itself is properly forwarded on the NAT router.
Less optimal ways to fix BackupPC ping checks
Using hping3 has so far solved the issues in my varieties of backup constellations while using BackupPC. However, there are of course alternatives. Two of the absolutely most commonly suggested ones are discussed below. The reason being that I do not understand why anyone would settle for these incomplete workarounds.1. Disable BackupPC ping checking
Probably the most commonly suggested solution for users that cannot ping host clients behind a NAT router or firewall is to disable the ping keep-alive check for BackupPC altogether. This is not surprising as the BackupPC documentation even states:
If you want to disable ping checking, set this to some program that exits with 0 status, eg: $Conf{PingPath} = '/bin/echo';What the above does is to trick BackupPC into always thinking that the host is alive, regardless of if it is true or not, a poor man's solution.
Pros: Simple and straight-forward to implement and works well with hosts that are almost always online and alive, i.e., hosts that would have responded to a ping request anyway.
Cons: This solution has obvious problems with hosts that are only occasionally online. By disabling ping checking, BackupPC thinks that the client is always alive and available and thus will strictly follow its schedule and blackout periods. In the worst case, clients that are only online within the blackout periods will never be backed up! Not optimal at all!
2. Enable ICMP ping response on NAT router/firewall
Many routers and firewalls are configured by default to not reply to ICMP ping requests. In addition, some users may have manually turned this off. In either case, this will make BackupPC fail its ping check and thus mark the hosts as unreachable.
It is debatable if one should disable ICMP ping responses or not. Some claim that it increases security and others claim that this only adds pseudo security as it is easy to discover if the destination is online anyway. This debate is outside of the scope of this article, however, for a remote BackupPC server trying to access clients behind the firewall, this is a nonsense solution since it completely breaks down regardless of how ICMP ping responses are configured as seen in the pros-and-cons list below.
Pros: It is usually simple to enable ICMP ping responses and similar to the first solution, this one will work if all hosts behind the router are almost always online.
Cons: Again the problems are obvious here as this is almost the same as disabling ping checking on BackupPC altogether, since a NAT router is almost always online anyway. In addition, this does not address the issue of backing up multiple hosts behind a NAT router, since it is impossible to discover if each individual host is alive or not through ICMP ping packets. Again, in the worst case, one can end up with clients that are never backed up if they are online reachable during the configured blackout periods!
3. Use SSH instead of ping
In addition to the above alternatives, there are also more involved ones. A popular one being the use of SSH tunnels from the client side. The basic idea here is to make all clients establish an SSH connection to the BackupPC server, the server then checks the established connections to determine if the host clients are alive or not. It may sound like a somewhat elegant way, since it requires the clients to actively do something, there are again obvious downsides with this 'fix'.
Pros: In contrast to the above two alternatives, this one actually does allow BackupPC to correctly determine if individual hosts behind a NAT router is reachable and alive.
Cons: The requirement that the clients need to establish a connection to BackupPC kind of defies the purpose the autonomous design. In addition, this may work great on Linux, *nix or Mac OS hosts, but what about Windows? One should not be forced to install cygwin just to be able to use BackupPC (encryption of the data stream can be achieved in other ways than SSH). This workaround is also more messy to maintain, since we would need to verify that the SSH connection is properly established on boot. A small thing perhaps, but the added complexity does not make sense when simpler ones exist. The beauty of BackupPC is after all that it does not explicitly require each client to install client software!
In conclusion, through my own experiences, the use of hping3 on a remote BackupPC server for host clients behind NAT routers and firewalls is the best way to ping-check. Although I respect that there certainly are other alternatives out there, they do not make much sense to me, but I may have missed some fundamental values in the BackupPC and *nix community...
In any case, I hope that this post will be of help to some of you out there who are wondering how to ping BackupPC hosts behind a NAT router or firewall!
Thank's for the post. Despite old, it's still useful.
ReplyDelete