Got thousands of tar archives being transferred from various machines to one repository. Found that some of the archives are bad, either because the network transfer didn’t complete or the archive process was interrupted on the machine that created the archive (due to insufficient disk space, etc). Whatever the cause, needed a method to verify the archive integrity.
Tar provides options for verify the archive integrity by comparing it with the file system. In this case, I needed the ability to verify the archive without access to the original file system.
The BASH one liner below will find all archives below current directory, loop through each one attempting to list its contents, discard the archive content list, capture and log the tar command’s exit status along with archive name and redirect both to a log. Exit status of 0 means the archive is good, 2 means it’s bad (haven’t seen a 1). Redirecting output to /dev/null while capturing exit status turned out trickier than I thought, but this seems to work well:
1 | for f in $(find . -name "*tar.bz2"); do tar tfj $f &> /dev/null; err="$?"; echo $err $f >> tar-check.list; done |
I liked Jason’s solution of passing a file list to du via xargs to produce results sorted by size in human readable format, but wanted ability to limit file list by name and age. Combining the find command with du, worked out nicely:
find . -name "*tar.bz2" -mmin -120 -ls | sort -k7rn | awk '{print$NF}' | xargs du -sh
122M ./sls-monitor/830144.tar.bz2 98M ./sls-monitor/830156.tar.bz2 67M ./sls-off1/905895.tar.bz2 50M ./sls-off1/893748.tar.bz2 16M ./sls-off1/893759.tar.bz2 7.3M ./sls-off1/905897.tar.bz2 5.1M ./sls-off1/854850.tar.bz2 4.4M ./sls-monitor/804331.tar.bz2 3.8M ./sls-monitor/804333.tar.bz2 612K ./sls-off1/893755.tar.bz2 512K ./sls-off1/905898.tar.bz2
Cooked up a little shell script to produce monthly email statistics such as amount of email received, how much of it was spam, percentage of spam correctly identified, etc. Previously, I had manually ran the numbers and input into OpenOffice Calc to get the stats — boring!
Example output:
root@dpork:~# spam-stats-month Aug 2009 ------------------------------------ Stats for Aug 2009 ------------------------------------ Ham SpamC SpamR SpamM HamC 151 122 3444 7 0 -------------------------------------------------------------- 3724 Total messages 3573 Total Spam (Caught + Missed + Rejected) 95.94% Spam as % of all mail 96.38% % of Spam rejected by Postfix at SMTP time 0% False positive rate (Ham misclassified as Spam) .18% False negative rate (Spam misclassified as Ham) 99.80% Spam catch rate (Spam filter accuracy) --------------------------------------------------------------
A project I’m working required a comma delimited file containing date ranges, in one week increments, from July 1987 to the present. That’s 1166 week ranges. What sounds like a potentially time consuming ordeal, takes about 10 sec to produce with the following shell one liner:
for n in $(seq 0 1166); \ do echo -e \ $(date +%m/%d/%Y --date="1987-07-01 +$n week")","\ $(date +%m/%d/%Y --date="1987-07-01 +$n week +6 day") \ >> date.ranges; done
Most of the stuff I do is via the shell, but once in a while need to attach to an existing X session on my work desktop. My requirements are that the method:
- Uses SSH for authentication
- Tunnels VNC traffic over SSH for security and so no additional ports need to be opened
x11vnc is what I’ve been using. Here’s a little shell script I wrote that will take user@hostname as a variable, enumerate an existing GDM session on the remote computer, create an SSH tunnel and attach to it via VNC. It uses aggressive compression, which looks a bit ugly but still works over a low bandwidth connection. The auth file enumeration bit works with Gnome, but could probably be adapted for use with other window managers.
Pad filenames starting with a single digit year (e.g. 1, 2, etc) with a zero while leaving filesnames starting with two digit years (e.g. 11, 12) untouched.
The shell one line script below will:
1. Find all files below current dir
2. Rename the file with download Source and date range as part of the new file name
3. Renamed file will stay at current location (e.g. file “./path/to/dir/crappyname” becomes “./path/to/dir/DC.Circuit-2006.06.02-2006.07.01.txt”)
To trim a video clip, mencoder and ffmpeg expect a start time in hh:mm:ss, but instead of end time, want duration:
$ mencoder -ss 00:02:55 -endpos 00:07:15 -oac copy -ovc copy in.avi -o out.avi $ ffmpeg -ss 00:02:55 -t 00:07:15 -vcodec copy -acodec copy -i in.avi out.avi
Video players make it easy to identify start/stop times, but difficult to figure out duration, so I wrote a little shell script that will calculate duration from start/stop times.
Usage example:
$ calc-duration 00:02:55 00:10:10 Start: 00 hrs 02 min 55 sec Stop: 00 hrs 10 min 10 sec +----------------------+ | duration is 00:07:15 | +----------------------+
Display all lines from file except lines between two lines matching a pattern:
1 | sed '/10\/02\/1992/,/01\/02\/2001/d' input.txt |
Display lines between two lines matching a pattern:
1 | sed -n '/10\/02\/1992/,/01\/02\/2001/p' input.txt |