Exploring methods to purge Varnish cache

I'm considering implementing Varnish at my day job and have been playing with it for a while to understand implications and work out issues. One issue is purging pages from cache as they are modified or new pages are added. Varnish provides several methods to do this (varnishadm, telnet and http).

Purging via varnishadm

NOTE: In Varnish Cache 3, purge is now called ban. To use examples below in Varnish 3, replace 'purge' with 'ban' (e.g. 'purge.list' becomes 'ban.list'). See this.

0. Let's save the command as a variable:

cmd="sudo varnishadm -T 127.0.0.1:6082"

If you get an error "Authentication required", then you're using a newer version of Varnish Cache that uses contents of a secret file for auth, in which case the command should be:

cmd="sudo varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret"

1. Purge a specific page (front page of this blog, for example):

$cmd purge req.url == "/alain/"

Result:

$cmd purge.list | head -n 1

    0 	req.url == /alain/

2. Purge any page matching a regular expression (front page of this blog, for example):

$cmd purge.url "^/alain/$"

Result:

$cmd purge.list | head -n 1

    0 	req.url ~ ^/alain/$

3. If you're using Varnish Cache to cache multiple sites, you could limit your purges to specific hosts with:

$cmd "purge req.http.host == giantdorks.org && req.url ~ ^/alain/$"

Result:

$cmd purge.list | head -n 1

    0 	req.http.host == giantdorks.org && req.url ~ ^/alain/$

4. Here's a shell script that will both purge and refresh (e.g. populate cache with the changed version of the page):

#!/bin/bash

cmd="sudo varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret"
site=$(echo $1 | sed -r 's|https?://||;s|/(.*)||;')
page=$(echo $1 | awk -F'/' -v OFS='/' '{sub(/https?:\/\//, ""); $1=""; print $0}')

if [ -z "$1" ]; then
  echo "ERROR: please provide a URL to purge.."
  exit 1
fi

if [ -z "$site" ]; then
  site=giantdorks.org
fi

if [ -z "$page" ]; then
  page="/"
fi

echo "purging old version from cache.."
$cmd "purge req.http.host == $site && req.url ~ ^$page\$" | sed '/^$/d'

echo "populating cache with new page content.."
curl -sL -w "%{http_code} %{url_effective}\n" http://${site}${page} -o /dev/null

Script in action:

$ v-purge giantdorks.org/alain/

purging old version from cache..
populating cache with new page content..
200 http://giantdorks.org/alain/

Purging via telnet

NOTE: In Varnish Cache 3, purge is now called ban. To use examples below in Varnish 3, replace 'purge' with 'ban' (e.g. 'purge.list' becomes 'ban.list'). See this.

Similar to running varnishadm, except we have to open a telnet connection first:

telnet localhost 6082
purge req.url == "/alain/"

200 0

My boxes don't accept telnet connections from the Internets so I can't think of a scenario where I'd be talking to varnishd via telnet, so varnishadm is what I'd use at the shell or in a script.

Purging via HTTP

NOTE: Examples below apply to both Varnish 2 and Varnish 3.

This one is probably the one that will see the most use as it is most accessible for a web based application. The only caveat is that it doesn't support regular expressions, so just one URL at a time can be passed this way.

Will test first via telnet:

Request:
telnet localhost 80

Response:
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Request:
PURGE /alain/ HTTP/1.0
Host: giantdorks.org

Response:
HTTP/1.0 200 OK
Date: Tue, 11 May 2010 02:00:00 GMT
Server: Apache
X-Pingback: http://giantdorks.org/alain/xmlrpc.php
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=UTF-8

...
Followed by the HTML output of the page

Note that the HTTP request was "PURGE HTTP/1.0" instead of the usual "GET HTTP/1.1".

Now on to a more useful examples.

1. Here's a PHP example using the built in fsockopen function, example based on the one offered at php.net:

\n";
} else {
    $out = "PURGE /alain HTTP/1.0\r\n";
    $out .= "Host: giantdorks.org\r\n";
    $out .= "Connection: Close\r\n\r\n";
    fwrite($fp, $out);
    while (!feof($fp)) {
        echo fgets($fp, 128);
    }
    fclose($fp);
}
?>

2. Here's another one. Presents user with a form to supply the URL and host (virtual host). Slightly modified example pulled from varnish-cache.org:

index.html:

        






  
Makes Varnish purge the supplied URL from its cache

  

        
URL 
        
HOST 
        

  

vpurge.php:

";
    }
    if ($verbose) { echo "$str\n"; }
  }

  function put($str) {
    global $sock,$verbose;
    fwrite ($sock, $str . "\r\n");
    if ($verbose) { echo "send: $str 
\n"; }
    readit();
  }
?>

3. Finally, a shell script to purge one or more pages from several Varnish Cache boxes with one command. Because it purges via HTTP instead of varnishadm, this script can be used to purge from remote hosts (of course the remote hosts will need to be authorized in the varnish cache config):

#!/bin/bash
# shell script to purge objects from varnish cache servers
# If debugging:
# 1. remove "--silent" to get cURL errors
# 2. remove "| head -n1" to get full response from varnish

if [ "$#" -lt 1 ]; then
  echo "ERROR: expecting a list of URL to purge, e.g.:"
  echo "$(basename $0) http://example.com/foo http://example.net/bar"
  exit 1
fi

varnishes='
vc1.example.com
vc2.example.com
'

curl_opts="--silent"

Normalize()
{
  full=$(echo "$var" | sed -r 's|https?://||i')
}

Purge()
{
  if ( echo "$full" | grep -q "/" ); then
    host=$(echo "$full" | cut -d/ -f1)
    path="/"$(echo "$full" | cut -d/ -f2-)
  else
    host="$full"
    path=""
  fi
  
  for vc in $varnishes; do
    echo "----------------------------------------------------------------"
    echo " VC:  $vc"
    echo " URL: $path"
    echo "----------------------------------------------------------------"
    curl $curl_opts -I -X PURGE -H "Host: $host" http://$vc$path | head -n1
    echo
  done
}

for var in "$@"; do
  Normalize
  Purge
done

Script in action (in this example, both pages were cached on vc1, but not on vc2):

$ purge.sh http://example.com/about-us http://example.com/get-involved

----------------------------------------------------------------
 VC:  vc1.example.com
 URL: /about-us
----------------------------------------------------------------
HTTP/1.1 200 Purged

----------------------------------------------------------------
 VC:  vc2.example.com
 URL: /about-us
----------------------------------------------------------------
HTTP/1.1 404 Not in cache

----------------------------------------------------------------
 VC:  vc1.example.com
 URL: /get-involved
----------------------------------------------------------------
HTTP/1.1 200 Purged

----------------------------------------------------------------
 VC:  vc2.example.com
 URL: /get-involved
----------------------------------------------------------------
HTTP/1.1 404 Not in cache