Rsnapreport.pl enhancements

I'm a big fan of rsnapshot.

rsnapshot is a filesystem snapshot utility for making backups of local and remote systems.

Using rsync and hard links, it is possible to keep multiple, full backups instantly available. The disk space required is just a little more than the space of one full backup, plus incrementals.

Having used a number of expensive commercial backup solutions over the years, rsnapshot is my favorite, the fact that it's Free and Open Source just makes it awesome.

This post is not about rsnapshot though, but about rsnapreport.pl, a utility rsnapshot ships with to produce a report that's easy on the eyes.

It's a fantastic Perl script, just a couple of small issues:

  1. A small bug that spits out "Use of uninitialized value in formline" errors when some values it expects are missing from the rsync stats output.
  2. The version that ships with Debian Lenny (1.3.0-2) doesn't sort the report by hostnames which makes it much less easy on the eye (latest version from the 1.3.1 source package does).
  3. The way it is typically called, looses the detailed rsync log, which I want to see in case rsnapreport points to a problem.

Solution to all three is easy enough.

First, let's get the latest version of rsnapreport.pl.

1. Grab the latest rsnapshot source packages from the site, as of this writing, it is 1.3.1.

2. Extract the archive, grab rsnapreport.pl from the utils directory and save it locally as rsnapreport-1.3.1

3. Create a patch named rsnapreport-1.3.1.patch (thanks to Dieter)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
--- rsnapreport-1.3.1	2010-10-29 14:55:41.324355024 -0700
+++ rsnapreport-1.3.1	2010-10-29 14:42:51.873835668 -0700
@@ -37,12 +37,12 @@ sub pretty_print(){
 
 	foreach my $source (sort keys %bkdata){
 		if($bkdata{$source} =~ /error/i) { print "ERROR $source $bkdata{$source}"; next; }
-		my $files = $bkdata{$source}{'files'};
-		my $filest = $bkdata{$source}{'files_tran'};
-		my $filelistgentime = $bkdata{$source}{'file_list_gen_time'};
-		my $filelistxfertime = $bkdata{$source}{'file_list_trans_time'};
-		my $bytes= $bkdata{$source}{'file_size'}/1000000; # convert to MB
-		my $bytest= $bkdata{$source}{'file_tran_size'}/1000000; # convert to MB
+		my $files = $bkdata{$source}{'files'} || "N/A";
+		my $filest = $bkdata{$source}{'files_tran'} || "N/A";
+		my $filelistgentime = $bkdata{$source}{'file_list_gen_time'} || "N/A";
+		my $filelistxfertime = $bkdata{$source}{'file_list_trans_time'} || "N/A";
+		my $bytes= $bkdata{$source}{'file_size'}/1000000 || "0"; # convert to MB
+		my $bytest= $bkdata{$source}{'file_tran_size'}/1000000 || "0"; # convert to MB
 		$source =~ s/^[^\@]+\@//; # remove username
 format BREPORTHEAD =
 SOURCE                          TOTAL FILES   FILES TRANS      TOTAL MB     MB TRANS   LIST GEN TIME  FILE XFER TIME

4. Apply patch

1
2
patch < rsnapreport-1.3.1.patch
patching file rsnapreport-1.3.1

5. Place script where you run them from and make executable

1
2
cp rsnapreport-1.3.1 /usr/local/bin/rsnapreport
chmod 755 /usr/local/bin/rsnapreport

The second (sorting) issue is solved by using the rsnapreport.pl from the 1.3.1 rsnapshot source package.

Now instead of this:

1
cat rsnapshot.log | ./rsnapreport-1.3.0-2
SOURCE                          TOTAL FILES   FILES TRANS      TOTAL MB     MB TRANS   LIST GEN TIME  FILE XFER TIME
--------------------------------------------------------------------------------------------------------------------
bfs4:/boot                    	         30            11         41.84        26.70   0.001 seconds   0.000 seconds
dlord:/var/lib/amavis/db      	         14             6          8.71         7.92   0.001 seconds   0.000 seconds
duck.mydomain:/var/www       	      12794             0        266.43         0.00   0.001 seconds   0.000 seconds
dlord:/var/backups            	        344            20        127.75         7.73   0.352 seconds   0.000 seconds
dtlx:/home                    	      21524           639       6229.47       422.95   0.001 seconds   0.000 seconds
duck.mydomain:/home          	        389             0          6.89         0.00   0.001 seconds   0.000 seconds
dlord:/var/www                	      52025             0        782.11         0.00   3.794 seconds   0.000 seconds
dtlx:/usr/local/bin           	         63             3          1.25         0.02   0.001 seconds   0.000 seconds
duck.mydomain:/var/backups   	        136             0         28.91         0.00   0.001 seconds   0.000 seconds
bfs4:/root                    	         74             0         13.93         0.00   0.001 seconds   0.000 seconds
duck.mydomain:/root          	        493             3          0.02         0.00   0.001 seconds   0.000 seconds
dlord:/var/lib/cyrus          	        131            13         52.23        32.85   0.298 seconds   0.000 seconds
Use of uninitialized value $filelistgentime in formline at ./rsnapreport-1.3.0-2 line 53.
Use of uninitialized value $filelistxfertime in formline at ./rsnapreport-1.3.0-2 line 53.
hawk.mydomain:/var/www       	          0             0          0.00         0.00
dlord:/var/lib/spamassassin   	         57             0          0.48         0.00   0.001 seconds   0.000 seconds
hawk.mydomain:/var/log       	         97            14          2.28         0.85   0.001 seconds   0.000 seconds
dlord:/var/log                	        502            41        102.24        15.28   0.063 seconds   0.000 seconds
dlord:/var/spool              	      75118           157       1432.01       393.27   2.348 seconds   0.000 seconds
hawk.mydomain:/var/backups   	         21             0          3.34         0.00   0.001 seconds   0.000 seconds
hawk.mydomain:/etc           	        997             0          1.23         0.00   0.001 seconds   0.000 seconds
dtlx:/boot                    	         27             0         26.17         0.00   0.001 seconds   0.000 seconds
duck.mydomain:/etc           	       1411             0          1.82         0.00   0.001 seconds   0.000 seconds
dlord:/usr/local/bin          	         12             0          0.03         0.00   0.001 seconds   0.000 seconds
bfs4:/usr/local/bin           	         33             1          0.18         0.00   0.001 seconds   0.000 seconds
duck.mydomain:/var/log       	        404           134        274.91       149.51   0.001 seconds   0.000 seconds
dtlx:/root                    	        153             0          0.56         0.00   0.001 seconds   0.000 seconds
dlord:/home                   	        506             0         10.85         0.00   0.162 seconds   0.000 seconds
dtlx:/var/backups             	         21             0          8.35         0.00   0.001 seconds   0.000 seconds
hawk.mydomain:/usr/local/bin 	          7             0          0.01         0.00   0.001 seconds   0.000 seconds
bfs4:/var/backups             	         26             1       4162.06         0.00   0.001 seconds   0.000 seconds
dlord:/boot                   	          5             0          7.31         0.00   0.012 seconds   0.000 seconds
dlord:/etc                    	       1498             0          6.25         0.00   0.190 seconds   0.000 seconds
bfs4:/home                    	        456             2        530.93         0.01   0.001 seconds   0.000 seconds
bfs4:/etc                     	       1430            26          2.09         0.17   0.001 seconds   0.000 seconds
dlord:/root                   	         18             0          0.02         0.00   0.026 seconds   0.000 seconds
dtlx:/etc                     	       2899             4          8.76         0.01   0.001 seconds   0.000 seconds
duck.mydomain:/usr/local/bin 	         10             0          0.02         0.00   0.001 seconds   0.000 seconds
hawk.mydomain:/home          	        293             0       3580.31         0.00   0.001 seconds   0.000 seconds
hawk.mydomain:/root          	         18             0          0.03         0.00   0.001 seconds   0.000 seconds

You should get this:

1
cat rsnapshot.log | ./rsnapreport-1.3.1.patched
SOURCE                          TOTAL FILES   FILES TRANS      TOTAL MB     MB TRANS   LIST GEN TIME  FILE XFER TIME
--------------------------------------------------------------------------------------------------------------------
bfs4:/boot                    	         30            11         41.84        26.70   0.001 seconds   0.000 seconds
bfs4:/etc                     	       1430            26          2.09         0.17   0.001 seconds   0.000 seconds
bfs4:/home                    	        456             2        530.93         0.01   0.001 seconds   0.000 seconds
bfs4:/root                    	         74           N/A         13.93         0.00   0.001 seconds   0.000 seconds
bfs4:/usr/local/bin           	         33             1          0.18         0.00   0.001 seconds   0.000 seconds
bfs4:/var/backups             	         26             1       4162.06         0.00   0.001 seconds   0.000 seconds
dlord:/boot                   	          5           N/A          7.31         0.00   0.012 seconds   0.000 seconds
dlord:/etc                    	       1498           N/A          6.25         0.00   0.190 seconds   0.000 seconds
dlord:/home                   	        506           N/A         10.85         0.00   0.162 seconds   0.000 seconds
dlord:/root                   	         18           N/A          0.02         0.00   0.026 seconds   0.000 seconds
dlord:/usr/local/bin          	         12           N/A          0.03         0.00   0.001 seconds   0.000 seconds
dlord:/var/backups            	        344            20        127.75         7.73   0.352 seconds   0.000 seconds
dlord:/var/lib/amavis/db      	         14             6          8.71         7.92   0.001 seconds   0.000 seconds
dlord:/var/lib/cyrus          	        131            13         52.23        32.85   0.298 seconds   0.000 seconds
dlord:/var/lib/spamassassin   	         57           N/A          0.48         0.00   0.001 seconds   0.000 seconds
dlord:/var/log                	        502            41        102.24        15.28   0.063 seconds   0.000 seconds
dlord:/var/spool              	      75118           157       1432.01       393.27   2.348 seconds   0.000 seconds
dlord:/var/www                	      52025           N/A        782.11         0.00   3.794 seconds   0.000 seconds
dtlx:/boot                    	         27           N/A         26.17         0.00   0.001 seconds   0.000 seconds
dtlx:/etc                     	       2899             4          8.76         0.01   0.001 seconds   0.000 seconds
dtlx:/home                    	      21524           639       6229.47       422.95   0.001 seconds   0.000 seconds
dtlx:/root                    	        153           N/A          0.56         0.00   0.001 seconds   0.000 seconds
dtlx:/usr/local/bin           	         63             3          1.25         0.02   0.001 seconds   0.000 seconds
dtlx:/var/backups             	         21           N/A          8.35         0.00   0.001 seconds   0.000 seconds
duck.mydomain:/etc           	       1411           N/A          1.82         0.00   0.001 seconds   0.000 seconds
duck.mydomain:/home          	        389           N/A          6.89         0.00   0.001 seconds   0.000 seconds
duck.mydomain:/root          	        493             3          0.02         0.00   0.001 seconds   0.000 seconds
duck.mydomain:/usr/local/bin 	         10           N/A          0.02         0.00   0.001 seconds   0.000 seconds
duck.mydomain:/var/backups   	        136           N/A         28.91         0.00   0.001 seconds   0.000 seconds
duck.mydomain:/var/log       	        404           134        274.91       149.51   0.001 seconds   0.000 seconds
duck.mydomain:/var/www       	      12794           N/A        266.43         0.00   0.001 seconds   0.000 seconds
hawk.mydomain:/etc           	        997           N/A          1.23         0.00   0.001 seconds   0.000 seconds
hawk.mydomain:/home          	        293           N/A       3580.31         0.00   0.001 seconds   0.000 seconds
hawk.mydomain:/root          	         18           N/A          0.03         0.00   0.001 seconds   0.000 seconds
hawk.mydomain:/usr/local/bin 	          7           N/A          0.01         0.00   0.001 seconds   0.000 seconds
hawk.mydomain:/var/backups   	         21           N/A          3.34         0.00   0.001 seconds   0.000 seconds
hawk.mydomain:/var/log       	         97            14          2.28         0.85   0.001 seconds   0.000 seconds
hawk.mydomain:/var/www       	        N/A           N/A          0.00         0.00             N/A             N/A

The solve the third issue, which is to get a concise backup report via email but keep a verbose log of every file backed up along with any errors, I set "verbose 5" in rsnapshot.conf and call it as follows:

1
grep rsnapshot /etc/crontab
# rsnapshot backups
00 2	* * *   root    rsnapshot sync 2>&1|tee -a /var/log/rsnapshot/log|rsnapreport|mail -s"rsnapreport" root
00 3    * * *   root    rsnapshot daily
45 2    * * 7   root    rsnapshot weekly
30 2    1 * *   root    rsnapshot monthly

Or if you don't sync first:

1
grep rsnapshot /etc/crontab
grep rsnapshot /etc/crontab
# rsnapshot backups
00 2    * * *   root    rsnapshot daily 2>&1|tee -a /var/log/rsnapshot/log|rsnapreport|mail -s"rsnapreport" root
30 1    * * 7   root    rsnapshot weekly
00 1    1 * *   root    rsnapshot monthly

5 Comments

  • 1. Alain Kelder is a Giant D&hellip replies at 30th October 2010, 3:35 pm :

    [...] a daily backup summary email (see my post on rsnapreport for [...]

  • 2. Steph3000 replies at 22nd March 2012, 3:29 am :

    This is helpful. Thank you !

  • 3. Mark replies at 25th May 2012, 8:46 am :

    Why is File Xfer Time zero for everything?
    Surely it takes some time to transfer files?

  • 4. Alain Kelder replies at 30th May 2012, 6:45 pm :

    Yes, surely. I’d say that’s another bug.

  • 5. dvg replies at 20th September 2013, 5:58 pm :

    This is not a bug. Rsync does not report the time it took to transfer files. It reports the time it take to transfer the file LIST. See below for sample output of rsync –stats:

    Number of files: 251424
    Number of files transferred: 213773
    Total file size: 6222284922 bytes
    Total transferred file size: 6027504827 bytes
    Literal data: 6027504827 bytes
    Matched data: 0 bytes
    File list size: 5882379
    File list generation time: 0.005 seconds
    File list transfer time: 0.000 seconds
    Total bytes sent: 4340587
    Total bytes received: 6043475004

    The bug (if any) is that this file list transfer time is being reported as “FILE XFER TIME” when it is not.

Leave a comment

NOTE: Enclose quotes in <blockquote></blockquote>. Enclose code in <pre lang="LANG"></pre> (where LANG is one of these).