su: pam_env(su:session): Unable to open env file: /etc/default/locale: No such file or directory

I already had the “locales” package installed, to create the file:

dpkg-reconfigure locales

Follow the prompts to choose your locale(s). Result:

cat /etc/default/locale

#  File generated by update-locale
LANG=en_US.UTF-8

$ cat file.txt

1
2
3
4
5
6
7

blah
blah blah
append after me
blah bleh!
append after me
blah blah blah
bleeeh

This will add “new stuff” after every line that says “append after me”:

$ sed -i '/append after me/a new stuff' file.txt

Bingo:

$ cat file.txt

1
2
3
4
5
6
7
8
9

blah
blah blah
append after me
new stuff
blah bleh!
append after me
new stuff
blah blah blah
bleeeh

find . -name "*.zip" | while read f; do unzip -t "$f" &> /dev/null; err="$?"; echo checking "$f"; echo $err "$f" >> zip-check.list; done

Output:

2 ./1-02-cv-00319-RHB-JGS(170).zip
0 ./1-02-cv-00319-RHB-JGS(171).zip
2 ./1-02-cv-00319-RHB-JGS(172).zip
2 ./1-02-cv-00319-RHB-JGS(173).zip
0 ./1-02-cv-00319-RHB-JGS(174).zip
2 ./1-02-cv-00319-RHB-JGS(175).zip
0 ./1-02-cv-00319-RHB-JGS(176).zip
0 ./1-02-cv-00319-RHB-JGS(177).zip
2 ./1-02-cv-00319-RHB-JGS(178).zip
0 ./1-02-cv-00319-RHB-JGS(179).zip

But, wasted hours aside, I just got it working again. I had the nice 1366×768 resolution previously working with Ubuntu 9.04, but it wouldn’t let me turn on Compiz effects. A few days ago, swayed by promise of seemless iPhone 3GS syncing, I upgraded to Ubuntu 10.04, which broke video. Turns out, GMA500 support is not all there yet with Ubuntu 10.04 Lucid and I forgot to research that before upgrading. So then I do a fresh install of Ubuntu 9.10 Karmic and follow the install steps on the Ubuntu Poulsbo Wiki, only to find this did absolutely nothing. Turns out, it’s missing some critical steps. After scouring the web for a while and trying a bunch of settings, finally arrived at a configuration that works. Got the pretty resolution back and even got extra Compiz effects! This thing is a royal pain in the ass, but sure looks nice when it works. Anyway, here’s the config that’s working for me on Dell Mini 10 (aka Inspiron 1010).

First, we follow the Wiki and add a repository and install some packages:

1
2

ak@mini:~$ sudo add-apt-repository ppa:gma500/ppa && sudo apt-get update
ak@mini:~$ sudo apt-get install libdrm-poulsbo1 poulsbo-config poulsbo-driver-2d poulsbo-driver-3d psb-firmware psb-kernel-headers psb-modules xpsb-glx

Also install kernel sources, which the Wiki fails to mention:

1

ak@mini:~$ sudo apt-get psb-kernel-source

Then we tell the kernel to leave some RAM for Compiz. Edit “/etc/default/grub” and set:

GRUB_CMDLINE_LINUX="mem=786mb"

Be sure to update grub after:

1

ak@mini:~$ sudo update-grub

Now we edit “/usr/bin/compiz” to tell Compiz that psb driver is not a bad guy after all by adding it to the list of drivers to whitelist:

WHITELIST="nvidia intel ati radeon radeonhd i810 fglrx psb"

Then backup your /etc/X11/xorg.conf and replace it with:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Section "ServerFlags"
        Option "DontZap" "False"
EndSection
 
Section "Device"
        Identifier "Configured Video Device"
        Option "IgnoreACPI"
        Option "AccelMethod" "uxa"
        Option "MigrationHeuristic" "greedy"
        Option "NoDDC"
        Option "DRI" "on"
        Option "Tiling" "true"
        Driver "psb"
EndSection
 
Section "DRI"
        Mode 0666
EndSection
 
Section "Monitor"
        Identifier "Configured Monitor"
EndSection
 
Section "Screen"
        Identifier "Default Screen"
        Monitor "Configured Monitor"
        Device "Configured Video Device"
EndSection

Now reboot and enjoy your beautiful new screen (until next time it breaks).

/etc/postfix/main.cf:

# restrict based on message header content 
header_checks = pcre:/etc/postfix/header_checks

/etc/postfix/header_checks:

/^To:([^@]*@){10,}/	REJECT Sorry, your message has too many recepients.
/^Cc:([^@]*@){10,}/	REJECT Sorry, your message has too many recepients.

Solution based on a post at linuxquestions.org.

While script runs, it’ll produce a status report every 3 seconds of how many URLs it’s spidered so far. Example output:

ak@gd:~$ ./check.url http://giantdorks.org/
Removing existing log..
213 URLs checked thus far
624 URLs checked thus far
 
All done, calculating response codes..
 
Response counts, sorted by HTTP code
    545 200
     62 302
     17 301

Also useful to populate the Varnish cache as well as check effectiveness of the setup (run the script while running varnishhist on the web server). Then run it again to make sure you see mostly cache hits. In fact, I see all cache hits on the second run. Cache hits are pipes “|”, backend hits are pounds “#” — all pipes, no pounds make me happy.

Bad:

1:15, n = 635                                me.sweet.box
 
                                #
                                #
          |                     #
          |                     #
          |                     #
          |                     #
         ||                     #
         ||                     #
         |||                   ##
         |||                   ###
         |||               #   ###
+-----+-----+-----+-----+-----+-----+-----+-----+-----
|1e-6 |1e-5 |1e-4 |1e-3 |1e-2 |1e-1 |1e0  |1e1  |1e2

Good:

1:20, n = 634                                me.sweet.box
 
        |
        |
        |
        |
        |
        |
        |
       |||
       |||
       |||
       ||||
       |||| 
       |||||
+-----+-----+-----+-----+-----+-----+-----+-----+-----
|1e-6 |1e-5 |1e-4 |1e-3 |1e-2 |1e-1 |1e0  |1e1  |1e2

And here’s the script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

#!/bin/bash
 
# error handling
function err_exit { echo -e 1>&2; exit 1; }
 
# check if proper arguments are supplied
if [ $# -ne 1 ]; then
  echo -e "\n Usage error!\n Please provide URL to check.\n Example: $0 http://example.com\n"
  exit 1
fi
 
# normalize url for log name
url=$(echo $1 | 's_https?://__;s/www\.//;s_/_._g;s/\.+/\./g;s/\.$//')
 
# remove log if exists
if [ -f /tmp/$url.log ]; then
   echo "Removing existing log.."
   rm /tmp/$url.log || err_exit
fi
 
wget --spider -r $1 &> /tmp/$url.log || err_exit &
 
while [ $(pgrep -l -f $url | grep wget | wc -l) != 0 ]; do
  sleep 3
  total=$(grep "HTTP request sent" /tmp/$url.log | wc -l)
  echo "$total URLs checked thus far"
done
 
echo -e "\nAll done, calculating response codes.."
echo -e "\nResponse counts, sorted by HTTP code"
grep "^HTTP" /tmp/$url.log | awk '{print$6}' | sort | uniq -c | sort -nr || err_exit

First, I’ll say that wget is a powerful tool and can place a burden on the site we’re grabbing data from (and probably get us banned as we’ll be perceived as carrying out a DOS attack). To avoid this, the example below will use a 5 second wait between downloads so as to not trouble the web server too much and keep the site usable for others.

1

for dir2 in $(echo seq 100 499); do wget -A pdf -w 5 --random-wait --retr-symlinks -k -r -U "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.19) Gecko/2010040118 Debian Lenny Firefox/3.0.19" http://example.com/dir1/$dir2/; done

Wget created a dir structure I don’t want, e.g.:

./example.com/dir1/309/file123.pdf

Rename each file using parent directory name as part of the file name, e.g.:

309.file123.pdf

1

for f in $(find . -name "*.pdf"); do new=$(echo $f | sed 's_^\./__;s_/_\._g') && echo renaming $f to $new && mv $f $new; done

Then place all in a dir named example.com.dir1:

1
2

mkdir example.com.dir1
for f in $(find . -name "*.pdf"); do echo moving $f && mv $f example.com.dir1/; done

This presents a slew of quality of life reducing issues, unless of course your back end system is Apache, in which case you can use mod_rpaf to make your life pleasant again.

Without it, you’d have to do painful things like patch your applications, which grab the client IP from the REMOTE_ADDR server variable, to use the X_FORWARDED_FOR header, if present, instead. For example:

1
2
3
4
5
6

// If set, use IP provided by our front end rproxy
if ($_SERVER["HTTP_X_FORWARDED_FOR"]) {
     $realclientip = $_SERVER["HTTP_X_FORWARDED_FOR"];
} else {
     $realclientip = $_SERVER["REMOTE_ADDR"];
}

mod_rpaf solves this much more easily, it works as follows:

It changes the remote address of the client visible to other Apache modules when two conditions are satisfied. First condition is that the remote client is actually a proxy that is defined in httpd.conf. Secondly if there is an incoming X-Forwarded-For header and the proxy is in its list of known proxies it takes the last IP from the incoming X-Forwarded-For header and changes the remote address of the client in the request structure.

If you’re on Debian (or one of its derivatives — Ubuntu, etc), get it with:

ak@gd:~$ aptitude install libapache2-mod-rpaf

Modify “/etc/apache2/mods-available/rpaf.conf” to include IP(s) of your front end boxes and enable with:

ak@gd:~$ a2enmod rpaf
ak@gd:~$ apache2ctl -t
ak@gd:~$ apache2ctl restart

Now, mod_rpaf will grab IP from X_FORWARDED_FOR header and place it in REMOTE_ADDR server variable. You can verify with:

1
2
3
4
5
6
7
8

<?php
$clientip = $_SERVER["HTTP_CLIENT_IP"];
$remoteaddr = $_SERVER["REMOTE_ADDR"];
$xforwardedfor = $_SERVER["HTTP_X_FORWARDED_FOR"];
echo " <li>HTTP_CLIENT_IP:  $clientip</li>\n";
echo " <li>REMOTE_ADDR:  $remoteaddr</li>\n";
echo " <li>HTTP_X_FORWARDED_FOR:  $xforwardedfor</li>\n";
?>

Geesus, that was easy. Gotta love Apache with its plethora of delicious modules. Using a distro like Debian that makes them super easy to get is nice too. Now, I can stop patching apps and go ride a board.

Boring:

ak@gd:~$ cat file
blah
blah blah
bleh blah

Awesome:

ak@gd:~$ cat file | nl -n ln -w1 -s\|
1|blah
2|blah blah
3|bleh blah

To rip all titles and chapters, try this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

#!/bin/bash
# script to identify and rip the longest title of a dvd into xvid avi
# dvd-device can be a path to mounted iso (e.g. /mnt/iso/) or a real device (e.g. /media/cdrom0/)
# expects dvd as $1 and dvd name as $2
 
# error handling
function err_exit { echo -e 1>&2; exit 1; }
 
# Confirm two arguments were passed
if [ $# -ne 2 ]; then
  echo -e 1>&2 "\n Usage error! Need two arguments.\n Example: $0 dvd-device My.DVD\n"
  exit 1
fi
 
# Check if VIDEO_TS exists
ddir=$(echo "$1/VIDEO_TS" | sed 's_/\+_/_')
if ! [ -d "$ddir" ]; then
  echo -e 1>&2 "\n Error!\n Can't find $ddir \n"
  exit 1
fi
 
log="dvd2xvid-longest.log"
 
# remove log if exists
if [ -f $log ]; then
   echo -e 1>&2 "\n Removing existing $log.."
   rm $log || err_exit
fi
 
# test lsdvd
lsdvd -x $1 1> /dev/null || err_exit
 
# get longest track
title=$(lsdvd -x $1 2> /dev/null | grep "Longest track" | awk '{print$3}')
 
# get longest chapter of the longest title
chapter=$(lsdvd -c -t$title $1 2> /dev/null | sed 's/^\s\+//' | grep "Chapter:" | sort -k4r | head -1 | awk -F'Chapter: ' '{print$2}' | awk -F',' '{print$1}')
 
# log date
echo -e "$(date)" >> $log || err_exit
 
# real work starts here
rm -f divx2pass.log frameno.avi || err_exit
# define source
SRC="dvd://$title -chapter $chapter -dvd-device $1 -alang en"
# audio pass
echo -e "\n\n$(date +%H:%M:%S) started apass" >> $log || err_exit
nice -n 3 mencoder -oac mp3lame -lameopts mode=2:cbr:br=96:vol=0 \
-ovc frameno -o frameno.avi $SRC || err_exit && \
# video pass 1
echo -e "\n\n$(date +%H:%M:%S) started vpass1" >> $log || err_exit
nice -n 3 mencoder -sws 2 -oac copy -ovc lavc \
-lavcopts vcodec=mpeg4:vbitrate=2000:vhq:vpass=1 \
-ffourcc XVID $SRC -o /dev/null || err_exit && \
# video pass 2
echo -e "\n\n$(date +%H:%M:%S) started vpass2" >> $log || err_exit
nice -n 3 mencoder -sws 2 -oac copy -ovc lavc \
-lavcopts vcodec=mpeg4:vbitrate=2000:vhq:vpass=2 \
-ffourcc XVID $SRC -o "$2.longest.track.avi" || err_exit
 
# log more stuff
echo "$(date +%H:%M:%S) done" >> $log || err_exit
echo -e "\n\n\n" >> $log || err_exit
echo -e "\nLongest track is $title"  >> $log || err_exit
echo "Longest chapter is $chapter"  >> $log || err_exit
echo -e "\n########## lsdvd output ##########" >> $log || err_exit
echo "$(lsdvd -x $1)" >> $log || err_exit

Call the script with the following arguments:

$ ./smtp.netcat.test mx.example.com 25 from@example.com to@example.com

And here’s the script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

#!/bin/bash
# script to send test mail with netcat. 
# expects the following arguments:
# 1. recepient mail server
# 2. port (typically 25 or 465)
# 3. mail from (e.g. from@example.com)
# 4. mail to (e.g. to@example.com)
 
# for mail_input function
from=$3
to=$4
 
# error handling
function err_exit { echo -e 1>&2; exit 1; }
 
# check if proper arguments are supplied
if [ $# -ne 4 ]; then
  echo -e "\n Usage error!"
  echo " This script requires four arguments:"
  echo " 1. recepient mail server"
  echo " 2. port (typically 25 or 465)"
  echo " 3. mail from (e.g. from@example.com)"
  echo " 4. mail to (e.g. to@example.com)"
  exit 1
fi
 
# create message
function mail_input { 
  echo "ehlo $(hostname -f)"
  echo "MAIL FROM: <$from>"
  echo "RCPT TO: <$to>"
  echo "DATA"
  echo "From: <$from>"
  echo "To: <$to>"
  echo "Subject: Testing one two three"
  echo "This is only a test. Please do not panic. If this works, then all is well, else all is not well."
  echo "In closing, Lorem ipsum dolor sit amet, consectetur adipiscing elit."
  echo "."
  echo "quit"
}
 
# test
#mail_input
 
# send
mail_input | nc $1 $2 || err_exit

The name of the script is compress.sh, tar will run as a child of it, which I can locate using the parent PID:

me@my:~$ show.proc.sh compress.sh
USER     TT         PID  PPID %CPU %MEM    VSZ  STARTED     TIME COMMAND
me       pts/19    1810 24357  0.0  0.0   4496 10:15:37 00:00:00  |           \_ /bin/bash ./compress.sh
me       pts/19    1857  1810  0.5  0.0   3696 10:16:03 00:00:01  |           |   \_ tar cfj 833444.tar.bz2 833444

show.proc.sh:

1
2
3
4
5
6
7

#!/bin/bash
 
# get pid
pid=$(pgrep $1)
 
ps axfo user,tty,pid,ppid,pcpu,pmem,vsz,start,time,args | head -1
ps axfo user,tty,pid,ppid,pcpu,pmem,vsz,start,time,args | grep $pid | grep -v grep

compress.sh:

1
2
3

#!/bin/bash
 
for d in $(cat dir.list.2010-05-11); do tar cfj $d.tar.bz2 $d && rm -rf $d; done

Errors:

Warning: fsockopen() expects parameter 2 to be long, string given in /path/to/wordpress-varnish/wp-varnish.php on line 138
()

and

Warning: fsockopen() expects parameter 2 to be long, string given in /path/to/wordpress-varnish/wp-varnish.php on line 115
()

My hack to fix the errors and also to add MU blogs to list of common URLs to purge on changes:

diff -up wp-varnish.php.default wp-varnish.php

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

--- wp-varnish.php.default	2010-05-10 17:20:25.000000000 -0700
+++ wp-varnish.php	2010-05-10 21:10:05.000000000 -0700
@@ -54,6 +54,13 @@ class WPVarnish {
     $this->WPVarnishPurgeObject("/");
     $this->WPVarnishPurgeObject("/feed/");
     $this->WPVarnishPurgeObject("/feed/atom/");
+    $this->WPVarnishPurgeObject("/alain/");
+    $this->WPVarnishPurgeObject("/alain/feed/");
+    $this->WPVarnishPurgeObject("/alain/feed/atom/");
+    $this->WPVarnishPurgeObject("/jason/");
+    $this->WPVarnishPurgeObject("/jason/feed/");
+    $this->WPVarnishPurgeObject("/jason/feed/atom/");
+
   }
 
   function WPVarnishAdminMenu() {
@@ -112,7 +119,7 @@ class WPVarnish {
     $wpv_host = preg_replace($wpv_replace_wpurl, "", $wpv_wpurl);
 
     for ($i = 0; $i < count ($wpv_purgeaddr); $i++) {
-      $varnish_sock = fsockopen($wpv_purgeaddr[$i], $wpv_purgeport[$i], $errno, $errstr, 30);
+      $varnish_sock = fsockopen("127.0.0.1", "80", $errno, $errstr, 30);
       if (!$varnish_sock) {
         echo "$errstr ($errno)<br />\n";
       } else {
@@ -135,7 +142,7 @@ class WPVarnish {
     $wpv_host = preg_replace($wpv_replace_wpurl, "", $wpv_wpurl);
 
     for ($i = 0; $i < count ($wpv_purgeaddr); $i++) {
-      $varnish_sock = fsockopen($wpv_purgeaddr[$i], $wpv_purgeport[$i], $errno, $errstr, 30);
+      $varnish_sock = fsockopen("127.0.0.1", "80", $errno, $errstr, 30);
       if (!$varnish_sock) {
         echo "$errstr ($errno)<br />\n";
       } else {

1. Varnishadm

Purge a specific page (front page of this blog, for example):

varnishadm -T 127.0.0.1:6082 purge req.url == "/alain/"

Result:

varnishadm -T 127.0.0.1:6082 purge.list
    0 	req.url == /alain/

Purge any page matching a regular expression (front page of this blog, for example):

varnishadm -T 127.0.0.1:6082 purge.url "^/alain/$"

Result:

varnishadm -T 127.0.0.1:6082 purge.list
    0 	req.url ~ ^/alain/$

2. Telnet

Similar to running varnishadm, except we have to open a telnet connection first:

telnet localhost 6082
purge req.url == "/alain/"
200 0

My boxes don’t accept telnet connections from the Internets so I can’t think of a scenario where I’d be talking to varnishd via telnet, so varnishadm is what I’d use at the shell or in a script.

3. HTTP

This one is probably the one that will see the most use as it is most accessible for a web based application. The only caveat is that it doesn’t support regular expressions, so just one URL at a time can be passed this way.

Will test first via telnet:

Request:
telnet localhost 80
 
Response:
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
 
Request:
PURGE /alain/ HTTP/1.0
Host: giantdorks.org
 
Response:
HTTP/1.0 200 OK
Date: Tue, 11 May 2010 02:00:00 GMT
Server: Apache
X-Pingback: http://giantdorks.org/alain/xmlrpc.php
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=UTF-8
 
...
Followed by the HTML output of the page

Note that the HTTP request was “PURGE HTTP/1.0″ instead of the usual “GET HTTP/1.1″.

Now on to a more useful example. Here’s a PHP example using the built in fsockopen function, example based on the one offered at php.net:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<?php
$fp = fsockopen("127.0.0.1", "80", $errno, $errstr, 2);
if (!$fp) {
    echo "$errstr ($errno)<br />\n";
} else {
    $out = "PURGE /alain HTTP/1.0\r\n";
    $out .= "Host: giantdorks.org\r\n";
    $out .= "Connection: Close\r\n\r\n";
    fwrite($fp, $out);
    while (!feof($fp)) {
        echo fgets($fp, 128);
    }
    fclose($fp);
}
?>

Here’s another one. Presents user with a form to supply the URL and host (virtual host). Slightly modified example pulled from varnish-cache.org:

index.html:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
 
<html>
<head>
        <title>Purge Varnish cache</title>
</head>
 
<style type="text/css">
  body {
        font-size: 10px;
  }
  h1 {
        font-weight: bold;
        color: #000000;
        border-bottom: 1px solid #C6EC8C;
        margin-bottom: 2em;
  }
  label {
        font-size: 160%;
        float: left;
        text-align: right;
        margin-right: 0.5em;
        display: block
  }
  input[type="text"] {
        width: 500px;
  }
  .submit input {
        margin-left: 0em;
        margin-bottom: 1em;
  }
</style>
 
<body>
 
  <h1>Makes Varnish purge the supplied URL from its cache</h1>
 
  <form action="vpurge.php" method="post">
        <p><label>URL</label> <input type="text" name="url"></p>
        <p><label>HOST</label> <input type="text" name="host"></p>
        <p class="submit"><input value="Submit" type="submit"></p>
  </form>
 
</body>
</html>

vpurge.php:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

<?php
# get param
$url = $_POST["url"];
$host = $_POST["host"];
 
  $ip = "127.0.0.1";
  $port = "80";
 
  $timeout = 1;
  $verbose = 1;
 
  # inits
  $sock = fsockopen ($ip,$port,$errno, $errstr,$timeout);
  if (!$sock) { echo "connections failed $errno $errstr"; exit; }
 
  if ( !($url || $host) ) { echo "No params"; exit; }
 
  stream_set_timeout($sock,$timeout);
 
  $pcommand = "purge";
  # Send command
  $pcommand .= ".hash $url#$host#";
 
  put ($pcommand);
  put ("quit");
 
  fclose ($sock);
 
  function readit() {
    global $sock,$verbose;
    if (!$verbose) { return; }
    while ($sockstr = fgets($sock,1024)) {
      $str .= "rcv: " . $sockstr . "<br>";
    }
    if ($verbose) { echo "$str\n"; }
  }
 
  function put($str) {
    global $sock,$verbose;
    fwrite ($sock, $str . "\r\n");
    if ($verbose) { echo "send: $str <br>\n"; }
    readit();
  }
?>

To rip just the longest title, try this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

#!/bin/bash
# script to enumerate titles and chapters of a dvd and rip each into a separate xvid avi
# dvd-device can be a path to mounted iso (e.g. /mnt/iso/) or a real device (e.g. /media/cdrom0/)
# expects dvd as $1 and dvd name as $2
 
# error handling
function err_exit { echo -e 1>&2; exit 1; }
 
# Confirm two arguments were passed
if [ $# -ne 2 ]; then
  echo -e "\n Usage error! Need two arguments.\n Example: $0 dvd-device My.DVD\n"
  exit 1
fi
 
# Check if VIDEO_TS exists
ddir=$(echo "$1/VIDEO_TS" | sed 's_/\+_/_')
if ! [ -d "$ddir" ]; then
  echo -e "\n Error!\n Can't find $ddir \n"
  exit 1
fi
 
log="dvd2xvid-all.log"
tmp="/tmp/dvd.title.chapter.list"
 
# remove log if exists
if [ -f $log ]; then
   echo -e "\n Removing existing $log.."
   rm $log || err_exit
fi
 
# test lsdvd
lsdvd -x $1 1> /dev/null || err_exit
 
# create Title and Chapter list
lsdvd -x $1 | grep "^Title" > $tmp || err_exit
 
# log date
echo -e "$(date)" >> $log || err_exit
 
# outer loop goes through titles
for title in $(cat $tmp | grep -Eo "^Title: [0-9]{2}" | awk '{print$2}'); do 
  chapters=$(cat $tmp | grep "^Title: $title" | grep -Eo "Chapters: [0-9]{2}" | awk '{print$2}')
    # inner loop goes through chapters for each title
    for chapter in $(seq 01 $chapters); do 
      rm -f divx2pass.log frameno.avi || err_exit 
      echo -e "\n\n\n$(date +%H:%M:%S) Processing Title $title, Chapter $chapter" >> $log
      # real work happens here
      SRC="dvd://$title -chapter $chapter -dvd-device $1 -alang en"
      # audio pass
      echo -e "\n$(date +%H:%M:%S) started apass" >> $log
      nice -n 3 mencoder -oac mp3lame -lameopts mode=2:cbr:br=96:vol=0 \
      -ovc frameno -o frameno.avi $SRC || err_exit && \
      # video pass 1
      echo -e "\n\n$(date +%H:%M:%S) started vpass1" >> $log || err_exit
      nice -n 3 mencoder -sws 2 -oac copy -ovc lavc \
      -lavcopts vcodec=mpeg4:vbitrate=2000:vhq:vpass=1 \
      -ffourcc XVID $SRC -o /dev/null || err_exit && \
      # video pass 2
      echo -e "\n\n$(date +%H:%M:%S) started vpass2" >> $log || err_exit
      nice -n 3 mencoder -sws 2 -oac copy -ovc lavc \
      -lavcopts vcodec=mpeg4:vbitrate=2000:vhq:vpass=2 \
      -ffourcc XVID $SRC -o "$2.Title$title.Chapter$chapter.avi" || err_exit
    done
done
 
# log more stuff
echo "$(date +%H:%M:%S) done" >> $log || err_exit
echo -e "\n\n\n########## lsdvd output ##########" >> $log || err_exit
echo "$(lsdvd -x $1)" >> $log || err_exit
 
rm $tmp || err_exit

Our directory structure:

$ find . -type d
.
./test1
./test1/test1.1
./test2
./test2/test2.1
./test2/test2.1/test2.2
./test3
./test3/test3.1
./test3/test3.1/test3.2
./test3/test3.1/test3.2/test3.3

There are a number of ways to do this, but I found awk to be most elegant as I wanted to be able to substract an arbitrary number from the result and awk can count and do arithmetic all by itself:

$ find . -type d | while read f; do awk -F'/' '{print NF -1}' | sort | uniq -c | sort -nr; done
      3 2
      3 1
      2 3
      1 4

Failed to parse existing X config file '/etc/X11/xorg.conf'

Nothing further found in the logs. The solution was two-fold.

Firstly, running Nvidia config from the shell instead of the Administration menu, dumped a useful error to the screen, when attempting to save settings in the GUI:

$ sudo nvidia-settings 
 
VALIDATION ERROR:  Data incomplete in file /etc/X11/xorg.conf.
Undefined Device "(null)" referenced by Screen "Default Screen".

Secondly, now that I know that the problem is that nvidia-settings is upset with something in the existing config, one way to solve this is to just let nvidia-xconfig generate a new config.

$ sudo mv /etc/X11/xorg.conf /etc/X11/xorg.conf.bak
$ sudo nvidia-xconfig
 
WARNING: Unable to locate/open X configuration file.
New X configuration file written to '/etc/X11/xorg.conf'

A diff of the new xorg.conf against the backed up version, shows a bunch of new stuff was added along with TwinView options. Looks good to me. Now I reboot to confirm the new xorg.conf settings produce the result.

Yay, problem solved, on with my day.

(1) Specific model is:

$ sudo lspci | grep VGA
01:00.0 VGA compatible controller: nVidia Corporation G84 [GeForce 8600 GT] (rev a1)

$ string="abc.Cat.def.1.zip.hij.Cut.klm.1.zip.Cat.no.2.zippqrs"

Out of this string, you want to pick out Cat*zip (Cat.def.1.zip and Cat.no.2.zip), “.*” works but is greedy so this grabs too much:

$ echo $string | grep -Eo "Cat.*zip"
Cat.def.1.zip.hij.Cut.klm.1.zip.Cat.no.2.zip

This does what we want:

$ echo $string | grep -Eo "Ca[t][^t]*.zip"
Cat.def.1.zip
Cat.no.2.zip

The steps below probably don’t apply to you, UNLESS you made APT changes to pull in Boxee from the apt.boxee.tv Intrepid repository as per my previous post.

BTW, the version I got installed is 0.9.20.10263, yet I don’t see it available from apt.boxee.tv anymore. I’m guessing they’ve pulled beta releases (0.9.2x) off since I’ve grabbed it previously, as I now only see alpha releases (0.9.1x) listed:

sudo apt-cache madison boxee
     boxee | 0.9.14.6992 | http://apt.boxee.tv intrepid/main Packages
     boxee | 0.9.14.6992 | http://apt.boxee.tv jaunty/main Packages
     boxee | 0.9.14.6795 | http://apt.boxee.tv intrepid/main Packages
     boxee | 0.9.14.6795 | http://apt.boxee.tv jaunty/main Packages
     boxee | 0.9.13.6684 | http://apt.boxee.tv intrepid/main Packages
     boxee | 0.9.13.6684 | http://apt.boxee.tv jaunty/main Packages
     boxee | 0.9.12.6570 | http://apt.boxee.tv intrepid/main Packages
     boxee | 0.9.12.6570 | http://apt.boxee.tv jaunty/main Packages
     boxee | 0.9.11.5777 | http://apt.boxee.tv intrepid/main Packages
     boxee | 0.9.11.5777 | http://apt.boxee.tv hardy/main Packages
     boxee | 0.9.11.5777 | http://apt.boxee.tv jaunty/main Packages
     boxee | 0.9.7.4826 | http://apt.boxee.tv intrepid/main Packages
     boxee | 0.9.7.4826 | http://apt.boxee.tv hardy/main Packages
     boxee | 0.9.7.4826 | http://apt.boxee.tv jaunty/main Packages
     boxee | 0.9.6.4578 | http://apt.boxee.tv intrepid/main Packages
     boxee | 0.9.6.4578 | http://apt.boxee.tv hardy/main Packages
     boxee | 0.9.6.4578 | http://apt.boxee.tv jaunty/main Packages
     boxee |  0.9-r3970 | http://apt.boxee.tv hardy/main Packages
     boxee |  0.9-r3927 | http://apt.boxee.tv hardy/main Packages
     boxee |  0.9-r2492 | http://apt.boxee.tv hardy/main Packages
     boxee |  0.9-r2365 | http://apt.boxee.tv hardy/main Packages

First, I’m choosing to completely purge the current version along with any dependencies it pulled in.

sudo aptitude purge boxee
Reading package lists... Done
Building dependency tree
Reading state information... Done
Reading extended state information
Initializing package states... Done
The following packages will be REMOVED:
  boxee{p} comerr-dev{u} libcurl4-openssl-dev{u} libfaad-dev{u} libidn11-dev{u} libkadm5srv6{u} libkdb5-4{u} libkrb5-dev{u} libldap2-dev{u} libmysqlclient15off{u} libsdl-gfx1.2-4{u} libssl-dev{u} libtre4{u} libxmlrpc-c3{u} libxmlrpc-core-c3{u}
  python-sqlite{u} python2.4{u} xsel{u}
0 packages upgraded, 0 newly installed, 18 to remove and 0 not upgraded.
Need to get 0B of archives. After unpacking 28.5MB will be freed.

I don’t have any settings I need to keep, so I’m deleting them as well. This would give the new version a clean slate, which might be unnecessary, but can’t hurt either.

rm -r ~/.boxee/

Remove customizations we had added to install Boxee from Intrepid repository (WARNING: I don’t need these anymore because I had just added them for Boxee, YMMV!)

sudo rm /etc/apt/apt.conf /etc/apt/preferences /etc/apt/sources.list.d/intrepid.list /etc/apt/sources.list.d/boxee.list
sudo aptitude update

Now on to the manual install of the latest version:

wget http://www.boxee.tv/download/ubuntu -O boxee.latest.deb
sudo gdebi boxee.latest.deb

If all worked as it should, you should find shortcut to Boxee under “Applications” > “Sound & Video”.

------------------------------------
Stats for Feb 2010
------------------------------------
Ham	SpamC	SpamR	SpamM	HamC
192	126	1951	0	0
--------------------------------------------------------------
2269		Total messages
2077		Total Spam (Caught + Missed + Rejected)
91.53%		Spam as % of all mail
93.93%		% of Spam rejected by Postfix at SMTP time
0%		False positive rate (Ham misclassified as Spam)
0%		False negative rate (Spam misclassified as Ham)
100.00%		Spam catch rate (Spam filter accuracy)
--------------------------------------------------------------

As you can see, Postfix got most of it, then SpamAssasin got the rest. I heart Postfix and SpamAssassin.

Firstly, more on the problem. I was able to switch Bluetooth on/off with Fn+F2 keys, but the Wi-Fi icon remained dark. In the BIOS, both Bluetooth and Wireless were turned on and Fn+F2 was set to operate on both.

This laptop has the following adapter:

ak@lplx:~$ sudo lspci | grep 802.11
0b:00.0 Network controller: Broadcom Corporation BCM4311 802.11b/g WLAN (rev 01)

The syslog had errors like:

Feb 28 12:54:24 lplx kernel: [   24.670763] b43-phy0 ERROR: Firmware file "b43/ucode5.fw" not found
Feb 28 12:54:24 lplx kernel: [   24.670769] b43-phy0 ERROR: Firmware file "b43-open/ucode5.fw" not found
Feb 28 12:54:24 lplx kernel: [   24.670773] b43-phy0 ERROR: You must go to http://wireless.kernel.org/en/users/Drivers/b43#devicefirmware and download the correct firmware for this driver version. Please carefully read all instructions on this website.

Since the error was about missing drivers and lspci told me I’m using the BCM4311 adapter, first thing I tried was search APT descriptions for it:

ak@lplx:~$ sudo aptitude search ~dBCM4311
p   bcmwl-kernel-source                   - Broadcom 802.11 Linux STA wireless driver source
 
ak@lplx:~$ sudo aptitude show bcmwl-kernel-source
Package: bcmwl-kernel-source
New: yes
State: not installed
Automatically installed: no
Version: 5.10.91.9+bdcom-0ubuntu4
Priority: optional
Section: restricted/admin
Maintainer: Ubuntu Core Developers 
Uncompressed Size: 1,835k
Depends: dkms, linux-libc-dev, libc6-dev, linux-headers-generic
Description: Broadcom 802.11 Linux STA wireless driver source
 These package contains Broadcom 802.11 Linux STA wireless driver for use with Broadcom's BCM4311-, BCM4312-, BCM4321-, and BCM4322-based hardware.

Hmm, sounds promising, let’s try it:

ak@lplx:~$ sudo aptitude install bcmwl-kernel-source

After a reboot, the Wi-Fi icon on the keyboard lit up and Network Manager saw my wireless network. Sweet.

cp: cannot create regular file `./todo': File exists

And when creating a new file:

touch: cannot touch `alain.test': File exists

The error in the Apache log was:

[Thu Feb 25 22:52:42 2010] [error] [client xx.xx.xx.xx] Could not LOCK /path/to/alain.test due to a failed precondition (e.g. other locks).  [412, #0]

This is happening in Karmic with davfs2 1.4.1-1, but not in Intrepid with davfs2 1.3.3-1.

For now, running this on the client resolves the error:

echo "if_match_bug 1" >> /etc/davfs2/davfs2.conf

I used to accomplish this by setting up multiple user accounts and then using sux to startup multiple instances:

1
2

user1@madcow:/$ sux user2
user2@madcow:/$ firefox &

The above will fire up a firefox instance inside the user1’s desktop session running as user2. It works and maybe preferable in some situations, but there’s a better way for what I’m trying to do:

1

user1@madcow:/$ firefox -ProfileManager -no-remote &

The above will fire up the Profile Manager window allowing to start up separate instances of Firefox with multiple profiles:

1
2
3
4
5

user1@madcow:/$ ps auxf | head -1 && ps auxf | grep firefox
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
user1    21652  2.8  7.5 460416 274592 ?       Sl   Feb10  23:05 /usr/lib/firefox-3.0.17/firefox -p -no-remote
user1    22432  1.7  1.9 225680 69688 ?        Sl   Feb10   1:05 /usr/lib/firefox-3.0.17/firefox -p -no-remote
user1    23004  0.0  0.7  73964 28296 ?        Sl   Feb10   0:01 /usr/lib/firefox-3.0.17/firefox -p -no-remote

To make this the default way of starting up Firefox in Gnome, right click the Firefox application launcher in the panel, go to “Properties” and change “Command” to “firefox -ProfileManager -no-remote”.

Very ni-i-ice.

Tar provides options for verify the archive integrity by comparing it with the file system. In this case, I needed the ability to verify the archive without access to the original file system.

The BASH one liner below will find all archives below current directory, loop through each one attempting to list its contents, discard the archive content list, capture and log the tar command’s exit status along with archive name and redirect both to a log. Exit status of 0 means the archive is good, 2 means it’s bad (haven’t seen a 1). Redirecting output to /dev/null while capturing exit status turned out trickier than I thought, but this seems to work well:

1

for f in $(find . -name "*tar.bz2"); do tar tfj $f &> /dev/null; err="$?"; echo $err $f >> tar-check.list; done

Output:

1
2
3
4
5

cat tar-check.list
2 ./755806.tar.bz2
2 ./708955.tar.bz2
0 ./313854.tar.bz2
0 ./313857.tar.bz2

BTW, the specific error I’ve gotten, which the above method will identify, is:

bzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
        Input file = (stdin), output file = (stdout)
 
It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.
 
You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.
 
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

1

split -l $(echo $(( $(cat sourcefile | wc -l) / 3))) sourcefile sourcefile.

Assuming a file called sourcefile has 270 lines in it, the above will produce three smaller files of 90 lines each:

1
2
3
4
5

ak@pork:~$ wc -l sourcefile.*
  90 sourcefile.aa
  90 sourcefile.ab
  90 sourcefile.ac
 270 total

If the number of lines doesn’t devide up neatly (let’s say sourcefile contains 271 lines), split will produce an extra file with the difference:

1
2
3
4
5
6

ak@pork:~$ wc -l sourcefile.*
  90 sourcefile.aa
  90 sourcefile.ab
  90 sourcefile.ac
   1 sourcefile.ad
 271 total

I’m personally partial to numeric suffixes:

1
2
3
4
5
6
7

ak@pork:~$ split -l $(echo $(( $(cat sourcefile | wc -l) / 3))) -da3 sourcefile sourcefile.
ak@pork:~$ ls -1
sourcefile
sourcefile.000
sourcefile.001
sourcefile.002
sourcefile.003

Objectives:

• Be able to create multiple virtual users without having to create a shell account for every FTP user
• Individually set permissions for each FTP user
• FTP users should by default be locked inside a chroot
• However, need to be able to provide read/write access outside the chroot on exception basis

I used Proftpd on Debian Lenny to achieve the above.

Note: ip_conntrack_ftp kernel module is not loaded by default on Debian Lenny. Without it, everything will seem to work, except FTP LIST command will return “Connection timed out” and “Failed to retrieve directory listing” errors.

To resolve, load the module:

1

sudo modprobe ip_conntrack_ftp

To ensure it gets loaded on reboot, do:

1

sudo echo ip_conntrack_ftp >> /etc/modules

1. Install proftpd

1

sudo aptitude install proftpd

2. Create home directories for the virtual ftp users (we’re just creating two for this example):

1
2

sudo mkdir -p /var/ftp/user{1,2}/{read,write}
sudo chown -R proftpd:nogroup /var/ftp/

3. Get the uid and gid of the proftpd user (profptd install script should have created it):

1
2

sudo grep ftp /etc/passwd
proftpd:x:109:65534::/var/run/proftpd:/bin/false

4. Create virtual ftp users

1
2

sudo ftpasswd --passwd --name=user1 --uid=109 --gid=65534 --home=/var/ftp/user1 --shell=/bin/false --file=/etc/proftpd/passwd
sudo ftpasswd --passwd --name=user2 --uid=109 --gid=65534 --home=/var/ftp/user2 --shell=/bin/false --file=/etc/proftpd/passwd

Note: to change the password for the virtual FTP user, do:

1

sudo ftpasswd --change-password --passwd --name=user1 --file=/etc/proftpd/passwd

5. Add some directives to proftpd config file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

AuthUserFile            /etc/proftpd/passwd
DefaultRoot             ~
RequireValidShell       off
 
# VALID LOGINS
<Limit LOGIN>
   AllowUser user1
   AllowUser user2
   DenyALL
</Limit>
 
# USER 1
<Directory /var/ftp/user1/read>
    <Limit ALL>
        DenyAll
    </Limit>
    <Limit DIRS READ>
        AllowUser user1
    </Limit>
</Directory>
<Directory /var/ftp/user1/write>
    <Limit ALL>
        DenyAll
    </Limit>
    <Limit DIRS READ WRITE>
        AllowUser user1
    </Limit>
</Directory>
 
# USER 2
<Directory /var/ftp/user2/read>
    <Limit ALL>
        DenyAll
    </Limit>
    <Limit DIRS READ>
        AllowUser user2
    </Limit>
</Directory>
<Directory /var/ftp/user2/write>
    <Limit ALL>
        DenyAll
    </Limit>
    <Limit DIRS READ WRITE>
        AllowUser user2
    </Limit>
</Directory>

6. If you use a firewall, don’t forget to open up port 21. If you can get away with it, it’s of course best to open it up to a narrowly defined set of IPs.

7. Restart proftpd and you should be able to connect with an FTP client using the credentials for users we setup.

8. For read only file system access outside of the chroot:

1
2

mkdir /var/ftp/user1/read/blah
sudo mount --bind -r /path/to/somewhere/ /var/ftp/user1/read/blah/

Useful links:

1. ProFTPD-mini-HOWTO-Limit
2. ProFTPD-mini-HOWTO-VirtualUsers
3. ProFTPD-mini-HOWTO-AuthFiles
4. ProFTPD ftpasswd
5. ProFTPD-mini-HOWTO-Chroot

When I first took a StartCom cert for a spin a couple of years ago, the issue I had was spotty browser support. Today, I’m happy to report that web browser support for the free StartSSL certificate is excellent. Well, almost…

Firstly, I’d like to say that both pre-sales and post-sales support are unbelievably good. Terms “pre-sales” and “post-sales” here used loosely — the cert I got for giantdorks.org is free, after all. Anyway, I got almost instant email responses that actually made sense from people that run the thing, not some nonsense first level support crap you’d normally expect in the rare instances you do hear back.

Secondly, the setup is very fast. I was installing the certs about an hour after signing up. I haven’t been buying a lot of certs lately, but the last one I purchased was a $15 RapidSSL cert from Dynadot and while they were great, it took about a day to get the certificate (and that was actually a renewal).

Finally, StartSSL browser support seems great. All browsers I care about had no problems with the free StartSSL cert. I tested the following:

1
2
3
4
5
6
7
8
9
10

Linux ..... Galeon 2.0.6 
Linux ..... Epiphany 2.24.1
Linux ..... Firefox 3.0.15
Linux ..... Firefox 3.5.9
iPhone .... Safari
iPhone .... Mercury 2.1
Windows ... IE 7
Windows ... IE 8
Windows ... Safari 3.1.2
Windows ... Firefox 3.1

If you use other browsers on other platforms, please point them to https://www.startssl.com/ and feel free to comment below if your browser craps on itself — I’d love to hear about any browsers that don’t like the StartSSL certificate.

The only browser that complained about the free StartSSL cert was IE6 on Win2K. I personally don’t care — anyone still using IE6 should get off the Web immediately. If you know anyone still running Win2K, plead they switch to Firefox or Chrome — IE6 sucks ass — if they refuse, unplug their monitor, which should bewilder them (and the Meek Squad) for months. However, If you do need a SSL cert supported by Internet Explorer 6, but don’t need the green address bar (and don’t enjoy flushing money down the drain), try the $15 RapidSSL cert from Dynadot. I can confirm that IE6 is happy with it.

Other than that, I don’t see any reason to not go to StartCom for all my SSL certificate needs. If you do, I’d like to hear about it.

Update 5 March 2010: Since APT repository at apt.boxee.tv appears to be unmaintained, I’ve reverted back to manual installs until an updated repository becomes available. I don’t recommend you follow the steps below unless you want to get stuck with an outdated Boxee version. Here are the steps to revert back.

First we make sure the system is up to date:

sudo aptitude update
sudo aptitude safe-upgrade

Note: If you’re running Jaunty, substitute references to “karmic” with “jaunty”.

Create “/etc/apt/apt.conf” to tell APT that Karmic is our default release:

APT::Default-Release "karmic";

Create “/etc/apt/preferences” to give Karmic packages a higher priority than Intrepid packages:

Package: *
Pin: release a=karmic
Pin-Priority: 990
 
Package: *
Pin: release a=intrepid
Pin-Priority: 450

Create “/etc/apt/sources.list.d/intrepid.list” to tell APT where to get Intrepid packages:

deb http://us.archive.ubuntu.com/ubuntu/ intrepid main restricted universe multiverse

Create “/etc/apt/sources.list.d/boxee.list” to give APT a source for Boxee packages:

deb http://apt.boxee.tv intrepid main

Now we update APT to pick up our changes and install Boxee:

sudo aptitude update
sudo aptitude install boxee

If all worked as it should, you should find shortcut to Boxee under “Applications” > “Sound & Video”. Now, as new versions of Boxee are made available through their APT repo, we can get them through Synaptic APT GUI or via the console by running “sudo aptitude update && sudo aptitude safe-upgrade”.

Once Boxee adds Karmic support to their repo, we can remove the Intrepid Ubuntu sources and replace references to Intrepid with Karmic so APT will continue to get Boxee packages from the Boxee repo, just get Boxee packages for Karmic instead of Intrepid:

sudo rm /etc/apt/apt.conf /etc/apt/preferences /etc/apt/sources.list.d/intrepid.list
sudo sed -i 's/intrepid/karmic/' /etc/apt/sources.list.d/boxee.list
sudo aptitude update

Once Boxee packages become available from Ubuntu sources, we can remove the Boxee repo altogether:

sudo rm /etc/apt/sources.list.d/boxee.list
sudo aptitude update

Error: Unable create database backup "/var/lib/vnstat/.venet0".

By default, vnstat cron job runs as “nobody”, yet the database file “/var/lib/vnstat/venet0″ was owned by root, changing its ownership to “nobody”, fixed the problem:

chown nobody:nobody /var/lib/vnstat/*

In my tests, running rsync as a daemon produces file transfers almost twice as fast as when rsyncing over SSH.

Rsync daemon can be run either on the box that’s sending data or receiving it. In this example, the box receiving data will run the rsync daemon — so we’ll be using rsync in push mode (“client” pushing to “server”). I suppose, the other way around (pull mode), could be considered more secure since the rsync module can be setup in read only mode.

NOTE: the following setup should work on any Linux, but in my case the server was Debian 5.0.3 and client was Ubuntu 8.10.

On the server:

1. Create /etc/rsyncd.conf containing:

1
2
3
4
5
6
7
8
9
10
11

uid = root
gid = root
max connections = 5
hosts allow = 192.168.1.5 192.168.1.77
socket options = SO_KEEPALIVE
 
[modulename]
path = /path/to/dir
auth users = someuser
read only = no
secrets file = /etc/rsyncd.passwd

2. Create secrets file /etc/rsyncd.passwd containing:

1

someuser:somepass

3. Debian rsync daemon startup script will complain if secrets file is world readable so we set perms so it isn’t:

chmod 640 /etc/rsyncd.passwd

4. Start up the daemon:

/etc/init.d/rsync start

NOTE: any time /etc/rsyncd.conf is modified, daemon will need to be restarted.

On the client:

By default, rsync daemon on the server will prompt for password. You could supply a password file to suppress the prompt.

1. Create secrets file /etc/rsyncc.passwd containing:

1

somepass

2. Invoke rsync with:

rsync --password-file=/etc/rsyncc.passwd --progress -rh /path/to/local/dir/ someuser@server::modulename/path/to/remote/dir/

The above will read password on the client from /etc/rsyncc.passwd and sync contents of /path/to/local/dir/ on the client to /path/to/dir/path/to/remote/dir/ on the server (modulename expands to path in /etc/rsyncd.conf on the server).

------------------------------------
Stats for Dec 2009
------------------------------------
Ham	SpamC	SpamR	SpamM	HamC
160	392	1273	0	0
--------------------------------------------------------------
1825		Total messages
1665		Total Spam (Caught + Missed + Rejected)
91.23%		Spam as % of all mail
76.45%		% of Spam rejected by Postfix at SMTP time
0%		False positive rate (Ham misclassified as Spam)
0%		False negative rate (Spam misclassified as Ham)
100.00%		Spam catch rate (Spam filter accuracy)
--------------------------------------------------------------

Long version:

My company had switched to Google Apps for email and we saw an increase in the number of false negatives for spam sent to one of the email addresses published on our website. The amount was unbearable enough (a few dozen per day) that I asked our email guy to send mail from that alias to my personal email address to see if my anti spam setup could do better than Google’s. It did, but not well enough as the number of false negatives jumped from my usual 1-4 per month to 16 in Nov. The identification rate was still better than most commercial anti spam solutions out there (see test results by PC World, test results by Network Computing or test results by PC Magazine), but seeing more than a couple of spams per month is too much for me, regardless of how good a rate it might represent:

------------------------------------
Stats for Nov 2009
------------------------------------
Ham	SpamC	SpamR	SpamM	HamC
175	767	687	16	0
--------------------------------------------------------------
1645		Total messages
1470		Total Spam (Caught + Missed + Rejected)
89.36%		Spam as % of all mail
46.73%		% of Spam rejected by Postfix at SMTP time
0%		False positive rate (Ham misclassified as Spam)
.97%		False negative rate (Spam misclassified as Ham)
98.91%		Spam catch rate (Spam filter accuracy)
--------------------------------------------------------------

The reason for the jump in false negatives, was that my primary anti spam tool (rejecting spam at SMTP time with Postfix) was hampered by the fact that this spam was forwarded to me after having been accepted by Google, as such, it was coming from Google’s SMTP servers and because Google MX servers aren’t listed in RBL, my MX would accept the spam.

Previously, the amount of spam that got through was low enough that I didn’t even bother with statistical analysis tools. But in Nov, I enabled/configured the Bayes plugin for SpamAssassin. Looks like this one piece is what I needed to achieve 100% spam filtering accuracy. 0 spam is a number I can live with!

When I have a spare minute, I’ll see about complementing my existing toolkit with a solution based on Markovian discrimination, which, if you believe the Wikipedia article about it, might make for a more accurate statistical analysis based spam filter than the Bayesian method.

Update, Jan 13: I should note that after a few weeks, Google support finally came through with a solution which reduced false negatives from that particular email alias to zero. Aside from slow resolution time, I’m impressed with the quality of their filter.

Update, Feb 12: Spoke too soon — so far in Feb, Google let 11 spams through. During the last two weeks of Jan, it let through 10 spams. While Google’s spam filter may be better than those of some of the other free email providers, it still allows through an unacceptable amount of spam and falls well below what properly configured SpamAssassin setup can achieve.

cat file.txt 
leave me alone
leave me alone
comment me out
leave me alone
comment me out also
leave me alone

Comment out the lines that are begging for it:

sed -i '/comment me out/s/^/#/g' file.txt

Voila:

cat file.txt 
leave me alone
leave me alone
#comment me out
leave me alone
#comment me out also
leave me alone

Indeed, it looks like 1024 bit key length is the minimum currently recommended. In fact, depending on whom you ask, it may already be insufficient.

NIST guidelines are comparatively optimistic, but even according to their 2007 Recommendation for Key Management, time for 1024 bit keys is passing quick.

The trade off with longer key length is of course speed, decryption speed in particular, as that’s what the server will be doing. The penalty is significant. According to a study titled “Performance Analysis of Secure Web Server Based on SSL”, doubling of the key length resulted in nearly 8x performance penalty:

In my case, I’m working with low volume applications, 2048 bit key length should be fine:

openssl genrsa -rand /dev/urandom -out your.name.key 2048
openssl req -new -nodes -key your.name.key -out your.name.csr

But tonight is different. Tonight I’m gonna give the keyboard a break.

This will delete line 93 from known_hosts:

sed -i 93d ~/.ssh/known_hosts

• Hold down power and home buttons together until the Apple logo appears

And it came back to life. The battery was empty, plugged in, now it’s charging. It’s a relief, weird behavior, though, I think. I’ve let the battery on the other phones I’ve had over the years (Nokia, BlackBerry, Sony, Siemens, etc, etc) drain completely lots of times. Plugging in the charger always just let me start them up again with no issues.

Example output:

root@dpork:~# spam-stats-month Aug 2009
 
------------------------------------
Stats for Aug 2009
------------------------------------
Ham	SpamC	SpamR	SpamM	HamC
151	122	3444	7	0
--------------------------------------------------------------
3724		Total messages
3573		Total Spam (Caught + Missed + Rejected)
95.94%		Spam as % of all mail
96.38%		% of Spam rejected by Postfix at SMTP time
0%		False positive rate (Ham misclassified as Spam)
.18%		False negative rate (Spam misclassified as Ham)
99.80%		Spam catch rate (Spam filter accuracy)
--------------------------------------------------------------

My anti spam arsenal consists of Postfix anti-spam measures which reject obvious spam at the SMTP level and SpamAssassin via amavisd-new, which scans mail that was accepted for delivery. The latter consists of network tests (Razor, Pyzor), Bayes statistical analysis, FuzzyOcr for the image spam and some custom pattern matching rules by me.

After SA is done analyzing and tagging a message, it hands it back to Postfix, which then hands it to Cyrus for final delivery. Cyrus IMAP provides a powerful Sieve filtering interface, which I use to sort mail into my IMAP folders. My Sieve filter sorts spam over a certain threshold into the SpamCaught folder, while marking message with really high scores as read, so I don’t see them, the low scoring stuff I like to review and manually move any false positives to HamCaught folder and any false negatives to SpamMissed folder, which my nightly Bayes train script scans and learns from.

The spam stats script runs on the mail server (same Debian box runs Postfix and Cyrus), and very simply loops through my IMAP folders counting messages in the Inbox (ham), SpamCaught (correctly identified spam), SpamMissed (false negatives), HamCaught (false positives) and also parses mail logs to get a total for messages rejected by Postfix. It then does some basic bash arithmetic and calls bc to calculate percentages.

And here’s the script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

#!/bin/bash
# Get some spam numbers
# Usage: spam-stats-month Oct 2009
 
set -e
 
PATH=/usr/bin:/bin:/usr/local/bin; export PATH
 
  # Verify input
  if ([[ ! $1 =~ "\b[A-Z][a-z]{2}\b" ]]) || ([[ ! $2 =~ "\b[0-9]{4}\b" ]]); then
    echo -e 1>&2 "\n Usage error..\n Example: spam-stats-month Oct 2009\n"
    exit 127
  fi
 
# ----------------------------------------------------------------- #
# Convert text month input date format to numeric for use with "ls"
# (e.g. "Nov 2009" to "2009-11"")
case "$1" in
        Jan) nummon="01";;
        Feb) nummon="02";;
        Mar) nummon="03";;
        Apr) nummon="04";;
        May) nummon="05";;
        Jun) nummon="06";;
        Jul) nummon="07";;
        Aug) nummon="08";;
        Sep) nummon="09";;
        Oct) nummon="10";;
        Nov) nummon="11";;
        Dec) nummon="12";;
esac
# Construct numeric date
numdate=$2-$nummon
 
#1. Ham
cd /var/spool/cyrus/mail/domain/g/gd.org/a/user/ak/
Ham=$(ls -l *. | grep "$numdate" | wc -l)
 
#2. SpamRejected
file="/tmp/spam.stats.$1.rejected"
 
  if [ -f $file ]; then
    rm $file && touch $file
  else
    touch $file
  fi
 
  for f in $(ls -l /var/log/mail.log* | grep $numdate | awk '{print$NF}')
    do zgrep "postfix" $f | grep "reject:" | grep "^$1" | wc -l >> $file
  done
 
SpamR=$(awk '{s+=$0} END {print s}' $file)
 
#3. SpamCaught
cd /var/spool/cyrus/mail/domain/g/gd.org/a/user/ak/SpamCaught/
SpamC1=$(ls -l *. | grep "$numdate" | wc -l)
SpamC2=$(ls -l /var/lib/amavis/virusmails/ | grep "$numdate" | wc -l)
SpamC=$(( $SpamC1 + $SpamC2 ))
 
#4. SpamMissed
cd /var/spool/cyrus/mail/domain/g/gd.org/a/user/ak/SpamMissed/
SpamM=$(ls -l *. | grep "$numdate" | wc -l)
 
#5. HamCaught
cd /var/spool/cyrus/mail/domain/g/gd.org/a/user/ak/HamCaught/
HamC=$(ls -l *. | grep "$numdate" | wc -l)
 
echo ""
echo "------------------------------------"
echo -e "Stats for $1 $2"
echo "------------------------------------"
echo -e "Ham\tSpamC\tSpamR\tSpamM\tHamC"
echo -e "$Ham\t$SpamC\t$SpamR\t$SpamM\t$HamC"
 
# Calc percentages
TotalMsg=$(( $Ham + $SpamC + $SpamR + $SpamM + $HamC ))
TotalSpam=$(( $SpamC + $SpamR + $SpamM ))
TotalSpamI=$(( $SpamC + $SpamR ))
FPpercent=$( echo "scale=2; $HamC*100/$TotalMsg" | bc)
FNpercent=$( echo "scale=2; $SpamM*100/$TotalMsg" | bc)
SpamPercent=$( echo "scale=2; $TotalSpam*100/$TotalMsg" | bc)
SpamRPercent=$( echo "scale=2; $SpamR*100/$TotalSpam" | bc)
SpamCatchRate=$( echo "scale=2;  $TotalSpamI*100/$TotalSpam" | bc)
 
echo "--------------------------------------------------------------"
echo -e "$TotalMsg\t\tTotal messages"
echo -e "$TotalSpam\t\tTotal Spam (Caught + Missed + Rejected)"
echo -e "$SpamPercent%\t\tSpam as % of all mail"
echo -e "$SpamRPercent%\t\t% of Spam rejected by Postfix at SMTP time"
echo -e "$FPpercent%\t\tFalse positive rate (Ham misclassified as Spam)"
echo -e "$FNpercent%\t\tFalse negative rate (Spam misclassified as Ham)"
echo -e "$SpamCatchRate%\t\tSpam catch rate (Spam filter accuracy)"
echo "--------------------------------------------------------------"

It’s been working great for over a month until this morning, when instead of a login screen, I was greeted with an error: “Ubuntu running in low graphics mode”.

What’s up with that? The APT log contained the explanation:

1
2

 * Running DKMS auto installation service for kernel 2.6.28-16-generic        
 *  psb-kernel-source (4.41.1)...        psb-kernel-source (4.41.1): AUTOINSTALL not set in its dkms.conf.

I guess I could set AUTOINSTALL for the psb-kernel-source module to “Yes”. I’ll think about it, meanwhile, a manual fix is easy enough:

1
2

sudo dpkg-reconfigure psb-kernel-source
sudo shutdown -r now

POW. Back in business.

¹ By the way, all I had to do to get full resolution working in the first place, as I recall, was add the Ubuntu Mobile Team sources to my /etc/apt/sources.list:

1
2

deb http://ppa.launchpad.net/ubuntu-mobile/ppa/ubuntu jaunty main
deb-src http://ppa.launchpad.net/ubuntu-mobile/ppa/ubuntu jaunty main

Install the drivers and reboot:

1
2
3

sudo aptitude update
sudo aptitude install psb-kernel-headers psb-kernel-source xserver-xorg-video-psb
sudo shutdown -r now

for n in $(seq 0 1166); \
do echo -e \
$(date +%m/%d/%Y --date="1987-07-01 +$n week")","\
$(date +%m/%d/%Y --date="1987-07-01 +$n week +6 day") \
>> date.ranges; done

The result:

07/01/1987,07/07/1987
07/08/1987,07/14/1987
07/15/1987,07/21/1987
...
10/21/2009,10/27/2009
10/28/2009,11/03/2009
11/04/2009,11/10/2009

One way is to search package contents online at debian.org, or you could use apt-file.

First, we install it and update its database:

1
2

sudo aptitude install apt-file
sudo apt-file update

Now, let’s find what package provides command mimencode (since I’m looking for a program, I’ll prepend the search with “bin” since that’s where programs typically live):

1
2

apt-file search bin/mimencode
metamail: usr/bin/mimencode

So I’ll need to install package “metamail” to get program “mimencode”:

1
2
3

sudo aptitude install metamail
which mimencode
/usr/bin/mimencode

Sweet.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

SELECT COUNT(*) FROM wp_posts WHERE post_type = 'revision';
+----------+
| COUNT(*) |
+----------+
|      110 | 
+----------+
SELECT COUNT(*) FROM wp_posts WHERE post_type = 'revision' AND post_date < DATE_ADD(NOW(), INTERVAL -7 DAY);
+----------+
| COUNT(*) |
+----------+
|       84 | 
+----------+
DELETE FROM wp_posts WHERE post_type = 'revision' AND post_date < DATE_ADD(NOW(), INTERVAL -7 DAY);
SELECT COUNT(*) FROM wp_posts WHERE post_type = 'revision';
+----------+
| COUNT(*) |
+----------+
|       26 | 
+----------+

If you’re running WordPress MU, the process is the same, just need to specify the right table as each WPMU blog has its own. See this post for how to identify WPMU blog numbers and corresponding URLs.

• Uses SSH for authentication
• Tunnels VNC traffic over SSH for security and so no additional ports need to be opened

x11vnc is what I’ve been using. Here’s a little shell script I wrote that will take user@hostname as a variable, enumerate an existing GDM session on the remote computer, create an SSH tunnel and attach to it via VNC. It uses aggressive compression, which looks a bit ugly but still works over a low bandwidth connection. The auth file enumeration bit works with Gnome, but could probably be adapted for use with other window managers.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#!/bin/sh
 
die () {
    echo >&2 "$@"
    exit 1
}
 
echo $1 | grep -E -q '[a-zA-Z0-9]\@[a-zA-Z0-9]' || die "\n Missing username and host, try:\n ./vnc.sh user@host\n"
 
USERHOST=$1
USER=$(echo $USERHOST | awk -F@ '{print$1}')
AUTHFILE=/tmp/$(ssh $USERHOST 'ls -la /tmp | grep $USER | grep gdm' | awk '{print$NF}')
 
echo ""
echo " ------------------------------------------"
echo " Attempting to use the following variables:"
echo " ------------------------------------------"
echo " USERHOST... " $USERHOST
echo " USER....... " $USER
echo " AUTHFILE... " $AUTHFILE
echo ""
sleep 3
 
ssh -f -L 5900:localhost:5900 $USERHOST 'x11vnc -noxdamage -display :0 -auth' $AUTHFILE && sleep 5 && vncviewer -encodings "copyrect tight zrle hextile" localhost:0

Requirements: Any bluetooth enabled laptop and cellular phone should do, however, the steps below have been verified to work with Ubuntu 8.10 and 9.04 running on Dell Inspiron 1501 and Dell Mini 10. The phone used was Nokia E62.

I’ve already covered enabling bluetooth and connecting the two devices, so I’ll jump right into the bluetooth config. First, let’s install some software:

# aptitude install bluez-gnome bluez-utils wvdial

Identify the MAC address of your phone and channel used for Dial-Up Networking:

# hcitool scan
Scanning ...
00:18:C5:2A:xx:xx	E62
# sdptool browse 00:18:C5:2A:xx:xx | grep -A10 "Dial-Up Networking" | grep Channel
Channel: 1

Then enter the values. Below are mine, edit to fit your configuration.

My rfcomm config:

# cat /etc/bluetooth/rfcomm.conf
rfcomm0 {
bind yes;
device 00:18:C5:2A:xx:xx;
channel 1;
comment "Dial-up networking";
}

My wvdial config:

# cat /etc/wvdial.conf
[Dialer Defaults]
Phone =
Username =
Password =
New PPPD = yes
[Dialer E62]
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
Init3 = AT+CGDCONT=, ,"wap.cingular"
Modem = /dev/rfcomm0
Phone = *99***1#
Username = WAP@CINGULARGPRS.COM
Password = CINGULAR1
New PPPD = yes
BAUD = 460800
Stupid Mode = 1

Restart bluetooth:

# /etc/init.d/bluetooth restart

Now let’s attempt to connect:

# wvdial E62

If all is well, you should see something like:

--> WvDial: Internet dialer version 1.60
--> Cannot get information for serial port.
--> Initializing modem.
--> Sending: ATZ
ATZ
OK
--> Sending: ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
OK
--> Sending: AT+CGDCONT=, ,"wap.cingular"
AT+CGDCONT=, ,"wap.cingular"
OK
--> Modem initialized.
--> Sending: ATDT*99***1#
--> Waiting for carrier.
ATDT*99***1#
CONNECT
~[7f]}#@!}!} } }2}#}$@#}!}$}%\}"}&} }*} } g}%~
--> Carrier detected.  Starting PPP immediately.
--> Starting pppd at Sun Oct 11 21:22:26 2009
--> Pid of pppd: 20902
--> Using interface ppp0
--> pppd: ��?[08]
--> pppd: ��?[08]
--> pppd: ��?[08]
--> pppd: ��?[08]
--> pppd: ��?[08]
--> local  IP address 10.84.103.246
--> pppd: ��?[08]
--> remote IP address 10.6.6.6
--> pppd: ��?[08]
--> primary   DNS address 172.18.7.170
--> pppd: ��?[08]
--> secondary DNS address 172.18.7.170
--> pppd: ��?[08]

“Ifconfig” should show you a ppp0 device with an IP address and you should be able to access the Internet.

To disconnect, hit CTRL+C:

Caught signal 2:  Attempting to exit gracefully...
--> Terminating on signal 15
--> pppd: ��?[08]
--> Connect time 0.9 minutes.
--> pppd: ��?[08]
--> pppd: ��?[08]
--> pppd: ��?[08]
--> Disconnecting at Sun Oct 11 21:23:23 2009

The end.

fdisk  doesn’t  understand  GUID  Partition Table (GPT) and it is not designed for large 
partitions. In particular case use more advanced GNU parted(8).

Ok, parted it is then..

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# aptitude install parted
# parted /dev/xvde
 
(parted) mklabel gpt
(parted) mkpart primary 0 -1
(parted) print
 
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvde: 4548GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
 
Number  Start   End     Size    File system  Name     Flags
1       17.4kB  4548GB  4548GB               primary  
 
# mkfs.xfs /dev/xvde1 -f
# mount /dev/xvde1 /mnt/disk2
# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            247G   58G  177G  25% /
tmpfs                 2.0G     0  2.0G   0% /lib/init/rw
udev                   10M   28K   10M   1% /dev
tmpfs                 2.0G  4.0K  2.0G   1% /dev/shm
/dev/xvdc1            2.0T  1.1T  939G  55% /mnt/disk1
/dev/xvde1            4.2T  4.2M  4.2T   1% /mnt/disk2

This is because WordPress still considers itself at the old URL. You could use the following SQL find/replace old URL instances with new, directly in the database:

1

UPDATE wp_options SET option_value = REPLACE(option_value, 'http://old/', 'http://new/');

But wait, you’re not done, there are more places to update..

There are probably references to the old URL elsewhere as well. The ones I found were in the post_content and guid columns of the wp_posts table.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

UPDATE wp_posts SET post_content = REPLACE(post_content, 'http://old/', 'http://new/');
UPDATE wp_posts SET guid = REPLACE(guid, 'http://old/', 'http://new/');
 
SELECT COUNT(*) post_content FROM wp_posts WHERE post_content LIKE '%http://old%';
+--------------+
| post_content |
+--------------+
|            0 | 
+--------------+
 
SELECT COUNT(*) post_content FROM wp_posts WHERE post_content LIKE '%http://new%';
+--------------+
| post_content |
+--------------+
|           40 | 
+--------------+
 
SELECT COUNT(*) guid FROM wp_posts WHERE guid LIKE '%http://old%';
+------+
| guid |
+------+
|    0 | 
+------+
 
SELECT COUNT(*) guid FROM wp_posts WHERE guid LIKE '%http://new%';
+------+
| guid |
+------+
|   89 | 
+------+

If you’re running WordPress MU, the process is the same, just need to specify the right table as each WPMU blog has its own. See this post for how to enumerate WPMU blog numbers and URLs.

To modify a WPMU blog directly in the DB, one would usually want to identify which tables are used for which blog, to ensure the correct blog is being modified. Here’s a little shell one liner to do just that:

1
2
3
4

$ CMD="mysql giantdorks --skip-column-names -Be"; TBS=$($CMD "show tables like 'wp%options'"); for TB in $TBS; do URL=$($CMD "select option_value from $TB where option_name='siteurl'") && echo $TB is for $URL; done
wp_1_options is for http://giantdorks.org/
wp_2_options is for http://giantdorks.org/alain/
wp_3_options is for http://giantdorks.org/jason/

error: error running shared postrotate script for /var/log/mysql.log
/var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log
run-parts: /etc/cron.daily/logrotate exited with return code 1

The problem was that I imported all DBs from the prod server, which had a different password for the debian-sys-maint user so the following test would fail:

# mysqladmin --defaults-file=/etc/mysql/debian.cnf ping
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'debian-sys-maint'@'localhost' (using
password: YES)'

Easy fix is to copy the password from /etc/mysql/debian.cnf from the prod to the backup. Now this works:

# mysqladmin --defaults-file=/etc/mysql/debian.cnf ping
mysqld is alive

… and MySQL logs are getting rotated again.

1. Find all files below current dir
2. Rename the file with download Source and date range as part of the new file name
3. Renamed file will stay at current location (e.g. file “./path/to/dir/crappyname” becomes “./path/to/dir/DC.Circuit-2006.06.02-2006.07.01.txt”)

Note: For it to work as expected, it should find a line that starts with “Source:” and use first two words (e.g. “Source:DC Circuit – US Court of Appeals, District & Bankruptcy Cases, …” becomes “DC.Circuit”). It should also find a line that starts with “SearchTerms:” out of which it’ll extract the date range (e.g. “SearchTerms: date(geq (06/02/2006) and leq (07/01/2006))” becomes “2006.06.02-2006.07.01).

Do a dry run and examine rename.test for errors. This works to verify the quality of download as well, as it should reveal date ranges of downloaded files and their source.

1

$ for f in $(find . -type f); do dir=`dirname $f` && source=`head -20 $f | grep "^Source:" | awk -F: '{print$2}' | awk '{print$1"."$2}' | sed 's/ //g'` && dates=`head -20 $f | grep "^SearchTerms:" | grep -E -o "[0-9]{1,2}\/[0-9]{1,2}\/[0-9]{2,4}" | sed 's_/_._g' | tr "\n" "-" | sed 's/\([0-9]\{1,2\}.[0-9]\{1,2\}.\)\([0-9]\{4\}\)\(-\)\([0-9]\{1,2\}.[0-9]\{1,2\}.\)\([0-9]\{4\}\)\(-\)/\2.\1-\5.\4txt/' | sed 's/\.-/-/'` && echo would rename $f to $dir/$source-$dates && echo would rename $f to dir/$source-$dates >> rename.test; done

Examine output:

1

$ cat rename.test | sort -dk4 | less

Do the rename:

1

$ for f in $(find . -type f); do dir=`dirname $f` && source=`head -20 $f | grep "^Source:" | awk -F: '{print$2}' | awk '{print$1"."$2}' | sed 's/ //g'` && dates=`head -20 $f | grep "^SearchTerms:" | grep -E -o "[0-9]{1,2}\/[0-9]{1,2}\/[0-9]{2,4}" | sed 's_/_._g' | tr "\n" "-" | sed 's/\([0-9]\{1,2\}.[0-9]\{1,2\}.\)\([0-9]\{4\}\)\(-\)\([0-9]\{1,2\}.[0-9]\{1,2\}.\)\([0-9]\{4\}\)\(-\)/\2.\1-\5.\4txt/' | sed 's/\.-/-/'` && echo renaming $f to $dir/$source-$dates && mv $f $dir/$source-$dates; done

Test our SED by enclosing matches (back references) in parenthesis:

1
2
3
4
5

$ echo 01.01.1910-12.31.1914- | sed 's/\([0-9]\{2\}.[0-9]\{2\}.\)\([0-9]\{4\}\)\(-\)\([0-9]\{2\}.[0-9]\{2\}.\)\([0-9]\{4\}\)\(-\)/(\1)(\2)(\3)(\4)(\5)(\6)/'
(01.01.)(1910)(-)(12.31. (1914)(-)
 
$ echo 01.01.1910-12.31.1914- | sed 's/\([0-9]\{1,2\}.[0-9]\{1,2\}.\)\([0-9]\{4\}\)\(-\)\([0-9]\{1,2\}.[0-9]\{1,2\}.\)\([0-9]\{4\}\)\(-\)/\2.\1-\5.\4txt/' | sed 's/\.-/-/'
1910.01.01-1914.12.31.txt

Starting with Debian Lenny, libpam-umask is part of libpam-modules (which was auto installed on every Debian Lenny setup I’ve done so far), so just need to add the following to “/etc/pam.d/common-session”:

session    optional     pam_umask.so umask=002

Be sure to comment out conflicting umask directives elsewhere (e.g. /etc/profile)..

On Debian Etch, an addition step of installing libpam-modules is required:

# aptitude install libpam-umask

Lots of good info on UPG from a RedHat manual.

$ mencoder -ss 00:02:55 -endpos 00:07:15 -oac copy -ovc copy in.avi -o out.avi
$ ffmpeg -ss 00:02:55 -t 00:07:15 -vcodec copy -acodec copy -i in.avi out.avi

Video players make it easy to identify start/stop times, but difficult to figure out duration, so I wrote a little shell script that will calculate duration from start/stop times.

Usage example:

$ calc-duration 00:02:55 00:10:10
 
 Start: 00 hrs 02 min 55 sec
 Stop:  00 hrs 10 min 10 sec
 
 +----------------------+
 | duration is 00:07:15 |
 +----------------------+

Script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

#!/bin/bash
#============================================================#
# Calculates duration between start time and stop time
# USAGE: calc-duration 00:02:00 00:10:10
# Fix:
# 1. Currently expects start hour to be less than stop hour,
# so won't work for a clip that's longer than 24 hrs, not an
# issue for me as I don't have any of those... ;-)
#============================================================#
starthh=$(echo $1 | awk -F: '{print$1}')
startmm=$(echo $1 | awk -F: '{print$2}')
startss=$(echo $1 | awk -F: '{print$3}')
 
stophh=$(echo $2 | awk -F: '{print$1}')
stopmm=$(echo $2 | awk -F: '{print$2}')
stopss=$(echo $2 | awk -F: '{print$3}')
 
echo
echo " Start:" $starthh hrs $startmm min $startss sec
echo " Stop: " $stophh hrs $stopmm min $stopss sec
echo
 
if   [[ $((10#$startmm)) -gt $((10#$stopmm)) && $((10#$startss)) -gt $((10#$stopss)) ]]; then
        # echo start mm and ss are greater than stop
	# start mm is greater than stop mm, so we add 60 to stop mm and
	# substract 1 from stop hh
	newstopmm1=$((10#$stopmm+10#60))
	newstophh=$((10#$stophh-10#1))
	# start ss is greater than stop ss, so we add 60 to stop ss and
        # substract 1 from stop mm
        newstopss=$((10#$stopss+10#60))
        newstopmm=$((10#$newstopmm1-10#1))
        # calculate differences
	diffhh=$((10#$newstophh-10#$starthh))
	diffmm=$((10#$newstopmm-10#$startmm))
	diffss=$((10#$newstopss-10#$startss))
 
elif [[ $((10#$startmm)) -gt $((10#$stopmm)) ]]; then
        # echo startmm is greater than stopmm
	# start mm is greater than stop mm, so we add 60 to stop mm and
        # substract 1 from stop hh
        newstopmm=$((10#$stopmm+10#60))
        newstophh=$((10#$stophh-10#1))
	# calculate differences
        diffhh=$((10#$newstophh-10#$starthh))
        diffmm=$((10#$newstopmm-10#$startmm))
        diffss=$((10#$stopss-10#$startss))
 
elif [[ $((10#$startss)) -gt $((10#$stopss)) ]]; then
        # echo startss is greater than stopss
        # start ss is greater than stop ss, so we add 60 to stop ss and
        # substract 1 from stop mm
        newstopss=$((10#$stopss+10#60))
        newstopmm=$((10#$stopmm-10#1))
        # calculate differences
        diffhh=$((10#$stophh-10#$starthh))
        diffmm=$((10#$newstopmm-10#$startmm))
        diffss=$((10#$newstopss-10#$startss))
else
	# echo all stop values are greater than start values
        # calculate differences
        diffhh=$((10#$stophh-10#$starthh))
        diffmm=$((10#$stopmm-10#$startmm))
        diffss=$((10#$stopss-10#$startss))
fi
 
# pad single digit results with a zero (if + signs contain a single digit, add a zero, then remove + signs)
pdiffhh=$(echo +$diffhh+ | sed 's/+\([0-9]\{1\}\)+/+0\1+/g;s/+//g')
pdiffmm=$(echo +$diffmm+ | sed 's/+\([0-9]\{1\}\)+/+0\1+/g;s/+//g')
pdiffss=$(echo +$diffss+ | sed 's/+\([0-9]\{1\}\)+/+0\1+/g;s/+//g')
 
echo " +----------------------+"
echo " | duration is" $pdiffhh:$pdiffmm:$pdiffss "|"
echo " +----------------------+"
echo

1

sed '/10\/02\/1992/,/01\/02\/2001/d' input.txt

Display lines between two lines matching a pattern:

1

sed -n '/10\/02\/1992/,/01\/02\/2001/p' input.txt

Display all lines from file, excluding a line range between 2 and 90:

1

sed 2,90d input.txt

Display only lines between line numbers 2 and 90:

1

sed -n 2,90p input.txt

Display line two:

1

sed -n 2p input.txt

   ---------------8<---------------
       static-mapping HOST1 {
           ip-address 192.168.1.91
           mac-address aa:12:34:45:67:89
       }
       static-mapping HOST2 {
           ip-address 192.168.1.92
           mac-address bb:12:34:45:67:89
       }
       static-mapping HOST3 {
           ip-address 192.168.1.93
           mac-address cc:12:34:45:67:89
   ---------------8<---------------

The following shell one line script will convert the above into a format suitable for inclusion in a Bind DNS zone file:

$ grep -A1 static-mapping config.boot | awk '{print$2}' | tr "\n" " " | sed 's/\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\)/\1\n/g' | awk '{print$2 "\t" $1}'
192.168.1.91    HOST1
192.168.1.92    HOST2
192.168.1.93    HOST3

# wvdial E62
rfcomm connection refused

The solution was to remove the file containing the Link Keys used for the Bluetooth challenge-response scheme and letting the system recreate it after a successful bind. On my system this file was named “linkkeys” and lived in /var/lib/bluetooth/[device MAC address]

find /var/lib/bluetooth/ -name linkkeys | xargs rm

There are a number of packet sniffers out there, two of the most popular are Wireshark (formerly Ethereal) and tcpdump. Tshark is the console (command line) version of Wireshark.

Running a packet sniffer on even a moderately busy network will produce a ton of output so use of filters is a must. Since both Wireshark (tshark) and tcpdump use the pcap library (libpcap on UNIX and winpcap on Windows), they both use the same filter syntax.

There are lots of filter options. Filters can be combined with logical operators “and”, “not” and “or”.

Basic example below will capture DNS (port 53) traffic going to and from host 192.168.1.5:

1

tshark -i eth1 port 53 and host 192.168.1.5

When using multiple logical operators, filter syntax can get confusing. Parenthesis can be used to group multiple filter statements to make them easier on the eye.

A more complex filter example below will capture all traffic to/from hosts 192.168.1.4 and 192.168.1.5 except HTTP (port 80) and SMTP (port 25):

1

tshark -i eth1 '(host 192.168.1.4 or host 192.168.1.5) and not (port 80 or port 25)'

Please note that if I didn’t enclose the filter statement above in quotes, the shell would have tried to evaluate the parenthesis, returning an error similar to:

-sh: syntax error near unexpected token `('

Enclosing the filter statement in quotes tells the shell to just pass the arguments along without evaluating.

The solution is in the Apt How-To available from Debian. See section 3.8, title “How to keep a mixed system“.

First, a word of warning. APT is great but will dutifully do as you say, so you could get yourself into trouble. Mix packages from different releases with caution. When a package has lots of dependencies, often a newer version of a prerequisite package is listed, so to upgrade a single package to the testing version, APT might have to upgrade a number of other packages as well. This may become problematic. APT will warn you before doing anything, so consider those carefully. Personally, I don’t have a hard rule for mixing packages. Sometimes it makes sense to mix, sometimes it makes more sense to just setup a dedicated box or a VM to run Debian Testing/Unstable.

IMPORTANT: the “unstable” version of Debian is the version to which the newest versions of Debian packages are uploaded first. This distribution sees all of the changes that packages go through, both small ones and more drastic ones which affect many packages or the whole system. For this reason, this version of the distribution should not be used by inexperienced users or by those who need proven stability.
The ‘testing’ distribution is not necessarily better than ‘unstable’, because it does not receive security updates quickly. For servers and other production systems stable should always be used.

Here are the steps (this example is for getting the latest Bind9 package):

1. Edit /etc/apt/sources.list and add testing package source repository to your source list

# testing
deb http://ftp.us.debian.org/debian/ testing main contrib non-free
deb-src http://ftp.us.debian.org/debian/ testing main contrib non-free
 
# testing security
deb http://security.debian.org testing/updates main contrib non-free
deb-src http://security.debian.org testing/updates main contrib non-free

2. Create /etc/apt/apt.conf and add:

APT::Default-Release "stable";
APT::Cache-Limit "25165824";

This will tell apt not to automagically upgrade all your packages (essentially “upgrading” your entire system to the testing or unstable release) and also raise the cache limits as the default cache value is probably too small to accomodate this new larger set of packages.

3. Create /etc/apt/preferences and add:

Package: *
Pin: release a=stable
Pin-Priority: 990
 
Package: *
Pin: release a=testing
Pin-Priority: 450

This will tell apt to default to a stable version of a package, rather than to a higher release.

4. Update apt

aptitude update

5. Probably the easiest way to check what versions of Bind9 available from the repositories is to search packages.debian.org:

links2 http://packages.debian.org/bind9

6. Install Bind9 from the testing repository

aptitude -t testing install bind9

That’s it. Debian’s advanced package management system is one more reason I love it more every day.

As always with Linux, there are lots of quality projects worth serious consideration, so the hardest part was to narrow the plethora of choices down to a manageable number to test.

Specific requirements were:

• Runs on Linux
• Keeps historical data
• Not a resource hog (RAM, CPU or Disk)
• Keeps data in a format easily queried by other tools
• Released under an Open Source license

Out of the various tools I’ve tried, Netacct-mysql is currently my favorite. It collects bandwidth data through libpcap and stores in a MySQL database (reportedly, PostgreSQL and Oracle are also supported). It comes with a dedicated PHP front end, but to me the real power is that it stores the stats in the database, so through SQL SELECT statements, I’m able to get all the stats I need.

I was thinking about taking it a step further and getting some pretty graphs out of the data. Matt Dunlap, had suggested taking a look at PHP/SWF Charts.. Looks promising and it’s on my to do list.

Meanwhile, here are the installation steps for Netacct-mysql on Vyatta. Steps apply to Debian as well, since Vyatta is based on Debian (if you believe /etc/debian_version, which, on both Vyatta VC2 and VC3, reports “Debian 4.0″, aka “Etch”):

1
2
3
4
5
6
7

apt-get install mysql-server libpcap-dev gcc libmysqlclient15-dev make
wget http://netacct-mysql.gabrovo.com/download/netacct-mysql-0.78.tar.gz
tar xvfz netacct-mysql-0.78.tar.gz
cd netacct-mysql-0.78
mysql -uroot -p < netacct.sql
mysqladmin -u root password yourpasshere
mysql -u root -p yourpasshere

1
2
3

GRANT ALL PRIVILEGES ON netacct.* TO 'acct'@'localhost' IDENTIFIED BY 'somepass';
FLUSH PRIVILEGES;
quit

1
2
3
4

cd ~/netacct-mysql-0.78
./configure
make
make install

Edit nacct config files and define your options (e.g. enter the MySQL pass for user acct, define your network adapters and your networks)

1
2
3
4
5
6

vim /usr/local/etc/naccttab
vim /usr/local/etc/nacctpeering
cp contrib/nacctd.debian /etc/init.d/nacctd
update-rc.d nacctd defaults
/etc/init.d/nacctd start
ps aux | grep nacctd | grep -v grep

All done. Give the nacctd daemon about an hour (it keeps hourly stats) and you should start seeing numbers in your DB. Let’s see some stats!

1
2
3
4
5
6
7
8
9

mysql -u root -p
USE netacct;
SELECT * FROM traffic;
 
# Get total (in+out, in GBs) bandwidth used by 10.2.28.122 during Dec 07
SELECT SUM((input+output)/1073741824) FROM traffic WHERE IP='10.2.28.122' AND time LIKE '2007-12%';
 
# Get (in & out, in GBs) bandwidth used by all hosts during Dec 07
SELECT traffic.ip, sum(input)/1073741824 inp, sum(output)/1073741824 outp FROM traffic WHERE time LIKE '2007-12%' GROUP BY ip;

Hope you’re getting nice stats and that warm fuzzy feeling which comes with a false sense of security. For some additional info, check out the Netacct-mysql project page. Also, take a look at this post on Monitoring Bandwidth Usage for a Xen node, which is where I first learned about Netacct-mysql.

In addition to Netacct-mysql, I frequently use a couple of other tools also. Check them out!

1. Bmon
   I use this one for real time bandwidth monitoring at the console. Bmon tracks usage by interface. Looks great for a console tool. Description of the tool from the project page: “bmon is a portable bandwidth monitor and rate estimator running on various operating systems. It supports various input methods for different architectures. Various output modes exist including an interactive curses interface, lightweight HTML output but also formatable ASCII output.”
2. Vnstat
   Provides a historical view of bandwidth usage. Also tracks usage by interface. I use this one when I want a general overview of bandwidth usage, particularly on the edge router to see how close I am to overage charges from my colocation provider. The tool comes with a number of pre-configured display options (e.g. hours, days, weeks, months, top10). Can also measure and report back usage for a specified period time (e.g. 60 seconds). Description of the tool from the project page: “vnStat is a network traffic monitor for Linux that keeps a log of daily network traffic for the selected interface(s). vnStat isn’t a packet sniffer. The traffic information is analyzed from the /proc filesystem. That way vnStat can be used even without root permissions. However, at least a 2.2 series kernel is required.”

Well, that’s it. If you have some pointers, please leave a comment! Happy Festivus to all!

Below are a couple of personal notes I made on the subject to help myself out, as I was getting used to the Debian way of things…

List existing services:

ls /etc/rc2.d/S*

Manage services [start,stop,reload,etc]:

/etc/init.d/service command

Make service start on boot:

update-rc.d name defaults

The ‘defaults’ value makes the service start in runlevels 0123 and stop in 456. Alternatively, you can specify which runlevels the service will start and stop in with:

update-rc.d name start runlevel stop runlevel

To remove service from starting on boot:

update-rc.d name remove

BTW, for a really excellent breakdown of update-rc.d options, check out this well written post.

If you install sysvconfig (apt-get install sysvconfig) you can also use System V style commands to start/stop services:

service name command

Unfortunately, even having installed sysvconfig in Debian, the command to list all services with status (e.g. running, unused, etc) is still missing. In SUSE, you’d run it with:


service -s

And would get a nice little list like this:

In the absence of the above, you could just list running processes with ‘ps’.. Another option is to list files in /var/run whose filename ends with ‘pid’, as each of those corresponds to a service:


find . /var/run | grep pid

Yet another option, with sysvconfig installed on Debian, is to just run:


sysvconfig

…which produces a clean and simple ncurses GUI which allows to manage service startup and edit runlevels:

VIM

The example I’m using here is the Vyatta config file (BTW, not to digress, but Vyatta is a super awesome, flexible and capable Open Source router/firewall solution that runs on commodity x86 hardware). Anyway, by default, VIM doesn’t automatically syntax highlight the vyatta config file, config.boot. You can always enable it manually every time you open the file, by typing something like:


:set syntax=perl


…but that gets old quick and so I’m going to setup VIM to automatically highlight the syntax for this file.

First, make sure your VIM version supports syntax highlight by turning it on manually. If VIM complains, you probably just need to update to the latest version (using whatever software update facility is appropriate for your Linux distribution).

Secondly, create two files in your home directory:


cd ~
touch .vimrc
touch .vimcustomfiletypes


edit .vimrc and add:


"turn on syntax highlighting
syntax on
"load your custom syntax highlight filetypes file
so ~/.vimcustomfiletypes


edit .vimcustomfiletypes and add:


augroup filetype
au!
au! BufRead,BufNewFile *.boot* set filetype=cpp
augroup END


The above will use “cpp” (e.g. C++) syntax highlight rules for files which contain “.boot” in the filename (like Vyatta’s config.boot). There are other ways to identify the file besides the file name, just take a look at the VIM documentation. Also, if none of the provided syntax highlight rule sets work for you, you could define your own. I haven’t had the need to do that (yet?).

So, next time we open the Vyatta config.boot file, instead of boring ‘ol this:

We get this:

And what’s not to love about that?

As with most things in Linux, there are a million different ways of accomplishing the task. Here’s the ultra super easy method I like.

1
2

echo -e "\n# disable bell\nxset -b" >> ~/.bashrc
source ~/.bashrc

Sweet Jeesus, that was easy. This method should work for other Linux distributions as well. I haven’t tested it though since Ubuntu is the only desktop/laptop Linux I’m currently running.

Alright, here it goes.. Let’s say you got a great deal on some checks from an online vendor, however, each 8.5″ x 11″ sheet contains three checks and QuickBooks will only print the top one. What a waste! Fortunately, there’s a way..

Requirements:

• QuickBooks software (tested Quickbooks Premier Professional Services Edition 2006)
• Printer (tested HP LaserJet 4000)
• Checks (tested Business Size, 3-Per-Page Computer Checks Without Lines, Form # 59011)

1. Separate the individual checks along the perforated lines. Load a single check in your printer’s envelope tray and adjust plastic tray guides to the width of the check. Note: if the check has a Tear-Off Strip, remove it (typically the last check on the three page sheet would have the Tear-Off Strip)
2. In QuickBooks, go to print a check and
   • Under the “Settings” tab, choose Printer type: “Page-oriented (Single sheets)”
   • Under the “Settings” tab, set “Checks on 1st Page” to 1
   • Under the “Partial Page” tab, set “Partial Page Printing Style” to “Center”
   • Hit “Print” and answer “No” to the question about the Tear-Off Strip (which you removed in Step 1)

Now you should have a good looking check that should fit perfectly into your handsome double window self sealing envelope (which you probably purchased along with your checks). Seal, mail and impress your vendors with your professional looking setup.

Please note that you must have a data plan and an unlimited data plan is highly recommended to keep you from incurring additional connection charges.

Requirements: Any bluetooth enabled laptop and cellular phone should do, however, the steps below have been verified to work with Windows XP (Professional with SP2) running on the Dell Latitude D620 and Inspiron 1501 (Bluetooth Stack for Windows by Toshiba v.4.00.22(D)SP2) and the Nokia E62.

First thing is turn on (enable) Bluetooth on your laptop:

1. If not already enabled, turn on Bluetooth (in the BIOS, for Dell, hit F2 when you first switch on the computer)
2. Right click the Bluetooth icon in the system tray and choose “Enable” or “Install”. The Blutooth icon is red when Bluetooth is disabled: and white when enabled:

Next, enable Bluetooth visibility on your phone:

1. On the E62, go to Settings –> Bluetooth and change “My phone’s visibility” to “Shown to All

Now, we need to establish a Bluetooth connection between the laptop and the phone and create a new Dial Up Networking connection on the laptop utilizing the phone as the modem:

1. Right click the Bluetooth icon in the system tray and choose “Add New Connection”
2. Choose “Custom Mode”
3. Your laptop will scan for nearby Bluetooth devices and display a list. Choose your device (E62, in our case) and click “Next”

1. Laptop will ask for an authentication pin, make up a number and provide it (e.g. 1234) and click next, your phone should pop up an authentication box, provide the same code and choose to accept the connection.
2. In the box that follows you should have your phone as the connection name and *99# as the phone number. I turned off the “Set as Default Dial-up” option because I want to control when to establish this connection rather than let Windows control that.
3. Click “Dial” and you should get connected

To get the status of your connection (such as bandwidth used), place your mouse cursor over the network icon. To disconnect, right click and choose “Disconnect”.

That’s it, you’re all done! Now you can get an Internet connection wherever you have a cellular singal.

One issue that had been bugging me is that it wouldn’t search for Recipient (“To”) by default. Only for Sender (“From”) or Subject.

The fix is actually very easy (thanks to the well organized and thoroughly commented php files that make up the application). Just add the following highlighted items to /path/to/roundcube/program/steps/mail/search.inc


// search in subject and sender by default
else {
$search = $IMAP->search($mbox, "HEADER SUBJECT", trim($str), $imap_charset);
$search2 = $IMAP->search($mbox, "HEADER FROM", trim($str), $imap_charset);
$search3 = $IMAP->search($mbox, "HEADER TO", trim($str), $imap_charset);
finish_search($mbox, array_unique(array_merge($search, $search2, $search3)));
}


That’s it! Gotta love open source.

Mon May 10 13:56:47 PDT 2010
Mon May 10 13:57:33 PDT 2010
Mon May 10 13:58:15 PDT 2010
Mon May 10 14:15:46 PDT 2010
Mon May 10 14:49:55 PDT 2010
Mon May 10 15:16:08 PDT 2010