Rsnapreport.pl enhancements
I'm a big fan of rsnapshot.
rsnapshot is a filesystem snapshot utility for making backups of local and remote systems.
Using rsync and hard links, it is possible to keep multiple, full backups instantly available. The disk space required is just a little more than the space of one full backup, plus incrementals.
Having used a number of expensive commercial backup solutions over the years, rsnapshot is my favorite, the fact that it's Free and Open Source just makes it awesome.
This post is not about rsnapshot though, but about rsnapreport.pl, a utility rsnapshot ships with to produce a report that's easy on the eyes.
It's a fantastic Perl script, just a couple of small issues:
- A small bug that spits out "Use of uninitialized value in formline" errors when some values it expects are missing from the rsync stats output.
- The version that ships with Debian Lenny (1.3.0-2) doesn't sort the report by hostnames which makes it much less easy on the eye (latest version from the 1.3.1 source package does).
- The way it is typically called, looses the detailed rsync log, which I want to see in case rsnapreport points to a problem.
Solution to all three is easy enough.
First, let's get the latest version of rsnapreport.pl.
1. Grab the latest rsnapshot source packages from the site, as of this writing, it is 1.3.1.
2. Extract the archive, grab rsnapreport.pl from the utils directory and save it locally as rsnapreport-1.3.1
3. Create a patch named rsnapreport-1.3.1.patch (thanks to Dieter)
--- rsnapreport-1.3.1 2010-10-29 14:55:41.324355024 -0700
+++ rsnapreport-1.3.1 2010-10-29 14:42:51.873835668 -0700
@@ -37,12 +37,12 @@ sub pretty_print(){
foreach my $source (sort keys %bkdata){
if($bkdata{$source} =~ /error/i) { print "ERROR $source $bkdata{$source}"; next; }
- my $files = $bkdata{$source}{'files'};
- my $filest = $bkdata{$source}{'files_tran'};
- my $filelistgentime = $bkdata{$source}{'file_list_gen_time'};
- my $filelistxfertime = $bkdata{$source}{'file_list_trans_time'};
- my $bytes= $bkdata{$source}{'file_size'}/1000000; # convert to MB
- my $bytest= $bkdata{$source}{'file_tran_size'}/1000000; # convert to MB
+ my $files = $bkdata{$source}{'files'} || "N/A";
+ my $filest = $bkdata{$source}{'files_tran'} || "N/A";
+ my $filelistgentime = $bkdata{$source}{'file_list_gen_time'} || "N/A";
+ my $filelistxfertime = $bkdata{$source}{'file_list_trans_time'} || "N/A";
+ my $bytes= $bkdata{$source}{'file_size'}/1000000 || "0"; # convert to MB
+ my $bytest= $bkdata{$source}{'file_tran_size'}/1000000 || "0"; # convert to MB
$source =~ s/^[^\@]+\@//; # remove username
format BREPORTHEAD =
SOURCE TOTAL FILES FILES TRANS TOTAL MB MB TRANS LIST GEN TIME FILE XFER TIME
4. Apply patch
patch < rsnapreport-1.3.1.patch
patching file rsnapreport-1.3.1
5. Place script where you run them from and make executable
cp rsnapreport-1.3.1 /usr/local/bin/rsnapreport
chmod 755 /usr/local/bin/rsnapreport
The second (sorting) issue is solved by using the rsnapreport.pl from the 1.3.1 rsnapshot source package.
Now instead of this:
cat rsnapshot.log | ./rsnapreport-1.3.0-2
SOURCE TOTAL FILES FILES TRANS TOTAL MB MB TRANS LIST GEN TIME FILE XFER TIME
--------------------------------------------------------------------------------------------------------------------
bfs4:/boot 30 11 41.84 26.70 0.001 seconds 0.000 seconds
dlord:/var/lib/amavis/db 14 6 8.71 7.92 0.001 seconds 0.000 seconds
duck.mydomain:/var/www 12794 0 266.43 0.00 0.001 seconds 0.000 seconds
dlord:/var/backups 344 20 127.75 7.73 0.352 seconds 0.000 seconds
dtlx:/home 21524 639 6229.47 422.95 0.001 seconds 0.000 seconds
duck.mydomain:/home 389 0 6.89 0.00 0.001 seconds 0.000 seconds
dlord:/var/www 52025 0 782.11 0.00 3.794 seconds 0.000 seconds
dtlx:/usr/local/bin 63 3 1.25 0.02 0.001 seconds 0.000 seconds
duck.mydomain:/var/backups 136 0 28.91 0.00 0.001 seconds 0.000 seconds
bfs4:/root 74 0 13.93 0.00 0.001 seconds 0.000 seconds
duck.mydomain:/root 493 3 0.02 0.00 0.001 seconds 0.000 seconds
dlord:/var/lib/cyrus 131 13 52.23 32.85 0.298 seconds 0.000 seconds
Use of uninitialized value $filelistgentime in formline at ./rsnapreport-1.3.0-2 line 53.
Use of uninitialized value $filelistxfertime in formline at ./rsnapreport-1.3.0-2 line 53.
hawk.mydomain:/var/www 0 0 0.00 0.00
dlord:/var/lib/spamassassin 57 0 0.48 0.00 0.001 seconds 0.000 seconds
hawk.mydomain:/var/log 97 14 2.28 0.85 0.001 seconds 0.000 seconds
dlord:/var/log 502 41 102.24 15.28 0.063 seconds 0.000 seconds
dlord:/var/spool 75118 157 1432.01 393.27 2.348 seconds 0.000 seconds
hawk.mydomain:/var/backups 21 0 3.34 0.00 0.001 seconds 0.000 seconds
hawk.mydomain:/etc 997 0 1.23 0.00 0.001 seconds 0.000 seconds
dtlx:/boot 27 0 26.17 0.00 0.001 seconds 0.000 seconds
duck.mydomain:/etc 1411 0 1.82 0.00 0.001 seconds 0.000 seconds
dlord:/usr/local/bin 12 0 0.03 0.00 0.001 seconds 0.000 seconds
bfs4:/usr/local/bin 33 1 0.18 0.00 0.001 seconds 0.000 seconds
duck.mydomain:/var/log 404 134 274.91 149.51 0.001 seconds 0.000 seconds
dtlx:/root 153 0 0.56 0.00 0.001 seconds 0.000 seconds
dlord:/home 506 0 10.85 0.00 0.162 seconds 0.000 seconds
dtlx:/var/backups 21 0 8.35 0.00 0.001 seconds 0.000 seconds
hawk.mydomain:/usr/local/bin 7 0 0.01 0.00 0.001 seconds 0.000 seconds
bfs4:/var/backups 26 1 4162.06 0.00 0.001 seconds 0.000 seconds
dlord:/boot 5 0 7.31 0.00 0.012 seconds 0.000 seconds
dlord:/etc 1498 0 6.25 0.00 0.190 seconds 0.000 seconds
bfs4:/home 456 2 530.93 0.01 0.001 seconds 0.000 seconds
bfs4:/etc 1430 26 2.09 0.17 0.001 seconds 0.000 seconds
dlord:/root 18 0 0.02 0.00 0.026 seconds 0.000 seconds
dtlx:/etc 2899 4 8.76 0.01 0.001 seconds 0.000 seconds
duck.mydomain:/usr/local/bin 10 0 0.02 0.00 0.001 seconds 0.000 seconds
hawk.mydomain:/home 293 0 3580.31 0.00 0.001 seconds 0.000 seconds
hawk.mydomain:/root 18 0 0.03 0.00 0.001 seconds 0.000 seconds
You should get this:
cat rsnapshot.log | ./rsnapreport-1.3.1.patched
SOURCE TOTAL FILES FILES TRANS TOTAL MB MB TRANS LIST GEN TIME FILE XFER TIME
--------------------------------------------------------------------------------------------------------------------
bfs4:/boot 30 11 41.84 26.70 0.001 seconds 0.000 seconds
bfs4:/etc 1430 26 2.09 0.17 0.001 seconds 0.000 seconds
bfs4:/home 456 2 530.93 0.01 0.001 seconds 0.000 seconds
bfs4:/root 74 N/A 13.93 0.00 0.001 seconds 0.000 seconds
bfs4:/usr/local/bin 33 1 0.18 0.00 0.001 seconds 0.000 seconds
bfs4:/var/backups 26 1 4162.06 0.00 0.001 seconds 0.000 seconds
dlord:/boot 5 N/A 7.31 0.00 0.012 seconds 0.000 seconds
dlord:/etc 1498 N/A 6.25 0.00 0.190 seconds 0.000 seconds
dlord:/home 506 N/A 10.85 0.00 0.162 seconds 0.000 seconds
dlord:/root 18 N/A 0.02 0.00 0.026 seconds 0.000 seconds
dlord:/usr/local/bin 12 N/A 0.03 0.00 0.001 seconds 0.000 seconds
dlord:/var/backups 344 20 127.75 7.73 0.352 seconds 0.000 seconds
dlord:/var/lib/amavis/db 14 6 8.71 7.92 0.001 seconds 0.000 seconds
dlord:/var/lib/cyrus 131 13 52.23 32.85 0.298 seconds 0.000 seconds
dlord:/var/lib/spamassassin 57 N/A 0.48 0.00 0.001 seconds 0.000 seconds
dlord:/var/log 502 41 102.24 15.28 0.063 seconds 0.000 seconds
dlord:/var/spool 75118 157 1432.01 393.27 2.348 seconds 0.000 seconds
dlord:/var/www 52025 N/A 782.11 0.00 3.794 seconds 0.000 seconds
dtlx:/boot 27 N/A 26.17 0.00 0.001 seconds 0.000 seconds
dtlx:/etc 2899 4 8.76 0.01 0.001 seconds 0.000 seconds
dtlx:/home 21524 639 6229.47 422.95 0.001 seconds 0.000 seconds
dtlx:/root 153 N/A 0.56 0.00 0.001 seconds 0.000 seconds
dtlx:/usr/local/bin 63 3 1.25 0.02 0.001 seconds 0.000 seconds
dtlx:/var/backups 21 N/A 8.35 0.00 0.001 seconds 0.000 seconds
duck.mydomain:/etc 1411 N/A 1.82 0.00 0.001 seconds 0.000 seconds
duck.mydomain:/home 389 N/A 6.89 0.00 0.001 seconds 0.000 seconds
duck.mydomain:/root 493 3 0.02 0.00 0.001 seconds 0.000 seconds
duck.mydomain:/usr/local/bin 10 N/A 0.02 0.00 0.001 seconds 0.000 seconds
duck.mydomain:/var/backups 136 N/A 28.91 0.00 0.001 seconds 0.000 seconds
duck.mydomain:/var/log 404 134 274.91 149.51 0.001 seconds 0.000 seconds
duck.mydomain:/var/www 12794 N/A 266.43 0.00 0.001 seconds 0.000 seconds
hawk.mydomain:/etc 997 N/A 1.23 0.00 0.001 seconds 0.000 seconds
hawk.mydomain:/home 293 N/A 3580.31 0.00 0.001 seconds 0.000 seconds
hawk.mydomain:/root 18 N/A 0.03 0.00 0.001 seconds 0.000 seconds
hawk.mydomain:/usr/local/bin 7 N/A 0.01 0.00 0.001 seconds 0.000 seconds
hawk.mydomain:/var/backups 21 N/A 3.34 0.00 0.001 seconds 0.000 seconds
hawk.mydomain:/var/log 97 14 2.28 0.85 0.001 seconds 0.000 seconds
hawk.mydomain:/var/www N/A N/A 0.00 0.00 N/A N/A
The solve the third issue, which is to get a concise backup report via email but keep a verbose log of every file backed up along with any errors, I set "verbose 5" in rsnapshot.conf and call it as follows:
grep rsnapshot /etc/crontab
# rsnapshot backups
00 2 * * * root rsnapshot sync 2>&1|tee -a /var/log/rsnapshot/log|rsnapreport|mail -s"rsnapreport" root
00 3 * * * root rsnapshot daily
45 2 * * 7 root rsnapshot weekly
30 2 1 * * root rsnapshot monthly
Or if you don't sync first:
grep rsnapshot /etc/crontab
grep rsnapshot /etc/crontab
# rsnapshot backups
00 2 * * * root rsnapshot daily 2>&1|tee -a /var/log/rsnapshot/log|rsnapreport|mail -s"rsnapreport" root
30 1 * * 7 root rsnapshot weekly
00 1 1 * * root rsnapshot monthly
5 Comments
1. Alain Kelder is a Giant D&hellip replies at 30th October 2010, 3:35 pm :
[…] a daily backup summary email (see my post on rsnapreport for […]
2. Steph3000 replies at 22nd March 2012, 3:29 am :
This is helpful. Thank you !
3. Mark replies at 25th May 2012, 8:46 am :
Why is File Xfer Time zero for everything?
Surely it takes some time to transfer files?
4. Alain Kelder replies at 30th May 2012, 6:45 pm :
Yes, surely. I’d say that’s another bug.
5. dvg replies at 20th September 2013, 5:58 pm :
This is not a bug. Rsync does not report the time it took to transfer files. It reports the time it take to transfer the file LIST. See below for sample output of rsync –stats:
Number of files: 251424
Number of files transferred: 213773
Total file size: 6222284922 bytes
Total transferred file size: 6027504827 bytes
Literal data: 6027504827 bytes
Matched data: 0 bytes
File list size: 5882379
File list generation time: 0.005 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 4340587
Total bytes received: 6043475004
The bug (if any) is that this file list transfer time is being reported as “FILE XFER TIME” when it is not.
Leave a comment