Exploring methods to purge Varnish cache

I'm considering implementing Varnish at my day job and have been playing with it for a while to understand implications and work out issues. One issue is purging pages from cache as they are modified or new pages are added. Varnish provides several methods to do this (varnishadm, telnet and http).

Purging via varnishadm

NOTE: In Varnish Cache 3, purge is now called ban. To use examples below in Varnish 3, replace 'purge' with 'ban' (e.g. 'purge.list' becomes 'ban.list'). See this.

0. Let's save the command as a variable:

cmd="sudo varnishadm -T 127.0.0.1:6082"

If you get an error "Authentication required", then you're using a newer version of Varnish Cache that uses contents of a secret file for auth, in which case the command should be:

cmd="sudo varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret"

1. Purge a specific page (front page of this blog, for example):

$cmd purge req.url == "/alain/"

Result:

$cmd purge.list | head -n 1
    0 	req.url == /alain/

2. Purge any page matching a regular expression (front page of this blog, for example):

$cmd purge.url "^/alain/$"

Result:

$cmd purge.list | head -n 1
    0 	req.url ~ ^/alain/$

3. If you're using Varnish Cache to cache multiple sites, you could limit your purges to specific hosts with:

$cmd "purge req.http.host == giantdorks.org && req.url ~ ^/alain/$"

Result:

$cmd purge.list | head -n 1
    0 	req.http.host == giantdorks.org && req.url ~ ^/alain/$

4. Here's a shell script that will both purge and refresh (e.g. populate cache with the changed version of the page):

#!/bin/bash

cmd="sudo varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret"
site=$(echo $1 | sed -r 's|https?://||;s|/(.*)||;')
page=$(echo $1 | awk -F'/' -v OFS='/' '{sub(/https?:\/\//, ""); $1=""; print $0}')

if [ -z "$1" ]; then
  echo "ERROR: please provide a URL to purge.."
  exit 1
fi

if [ -z "$site" ]; then
  site=giantdorks.org
fi

if [ -z "$page" ]; then
  page="/"
fi

echo "purging old version from cache.."
$cmd "purge req.http.host == $site && req.url ~ ^$page\$" | sed '/^$/d'

echo "populating cache with new page content.."
curl -sL -w "%{http_code} %{url_effective}\n" http://${site}${page} -o /dev/null

Script in action:

$ v-purge giantdorks.org/alain/
purging old version from cache..
populating cache with new page content..
200 http://giantdorks.org/alain/

Purging via telnet

NOTE: In Varnish Cache 3, purge is now called ban. To use examples below in Varnish 3, replace 'purge' with 'ban' (e.g. 'purge.list' becomes 'ban.list'). See this.

Similar to running varnishadm, except we have to open a telnet connection first:

telnet localhost 6082
purge req.url == "/alain/"
200 0

My boxes don't accept telnet connections from the Internets so I can't think of a scenario where I'd be talking to varnishd via telnet, so varnishadm is what I'd use at the shell or in a script.

Purging via HTTP

NOTE: Examples below apply to both Varnish 2 and Varnish 3.

This one is probably the one that will see the most use as it is most accessible for a web based application. The only caveat is that it doesn't support regular expressions, so just one URL at a time can be passed this way.

Will test first via telnet:

Request:
telnet localhost 80

Response:
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Request:
PURGE /alain/ HTTP/1.0
Host: giantdorks.org

Response:
HTTP/1.0 200 OK
Date: Tue, 11 May 2010 02:00:00 GMT
Server: Apache
X-Pingback: http://giantdorks.org/alain/xmlrpc.php
Vary: Accept-Encoding
Connection: close
Content-Type: text/html; charset=UTF-8

...
Followed by the HTML output of the page

Note that the HTTP request was "PURGE HTTP/1.0" instead of the usual "GET HTTP/1.1".

Now on to a more useful examples.

1. Here's a PHP example using the built in fsockopen function, example based on the one offered at php.net:

\n";
} else {
    $out = "PURGE /alain HTTP/1.0\r\n";
    $out .= "Host: giantdorks.org\r\n";
    $out .= "Connection: Close\r\n\r\n";
    fwrite($fp, $out);
    while (!feof($fp)) {
        echo fgets($fp, 128);
    }
    fclose($fp);
}
?>

2. Here's another one. Presents user with a form to supply the URL and host (virtual host). Slightly modified example pulled from varnish-cache.org:

index.html:





        Purge Varnish cache






  

Makes Varnish purge the supplied URL from its cache

vpurge.php:

";
    }
    if ($verbose) { echo "$str\n"; }
  }

  function put($str) {
    global $sock,$verbose;
    fwrite ($sock, $str . "\r\n");
    if ($verbose) { echo "send: $str 
\n"; } readit(); } ?>

3. Finally, a shell script to purge one or more pages from several Varnish Cache boxes with one command. Because it purges via HTTP instead of varnishadm, this script can be used to purge from remote hosts (of course the remote hosts will need to be authorized in the varnish cache config):

#!/bin/bash
# shell script to purge objects from varnish cache servers
# If debugging:
# 1. remove "--silent" to get cURL errors
# 2. remove "| head -n1" to get full response from varnish

if [ "$#" -lt 1 ]; then
  echo "ERROR: expecting a list of URL to purge, e.g.:"
  echo "$(basename $0) http://example.com/foo http://example.net/bar"
  exit 1
fi

varnishes='
vc1.example.com
vc2.example.com
'

curl_opts="--silent"

Normalize()
{
  full=$(echo "$var" | sed -r 's|https?://||i')
}

Purge()
{
  if ( echo "$full" | grep -q "/" ); then
    host=$(echo "$full" | cut -d/ -f1)
    path="/"$(echo "$full" | cut -d/ -f2-)
  else
    host="$full"
    path=""
  fi
  
  for vc in $varnishes; do
    echo "----------------------------------------------------------------"
    echo " VC:  $vc"
    echo " URL: $path"
    echo "----------------------------------------------------------------"
    curl $curl_opts -I -X PURGE -H "Host: $host" http://$vc$path | head -n1
    echo
  done
}

for var in "$@"; do
  Normalize
  Purge
done

Script in action (in this example, both pages were cached on vc1, but not on vc2):

$ purge.sh http://example.com/about-us http://example.com/get-involved
----------------------------------------------------------------
 VC:  vc1.example.com
 URL: /about-us
----------------------------------------------------------------
HTTP/1.1 200 Purged

----------------------------------------------------------------
 VC:  vc2.example.com
 URL: /about-us
----------------------------------------------------------------
HTTP/1.1 404 Not in cache

----------------------------------------------------------------
 VC:  vc1.example.com
 URL: /get-involved
----------------------------------------------------------------
HTTP/1.1 200 Purged

----------------------------------------------------------------
 VC:  vc2.example.com
 URL: /get-involved
----------------------------------------------------------------
HTTP/1.1 404 Not in cache

21 Comments

  • 1. Matt Dunlap replies at 17th September 2010, 11:33 am :

    Does restarting varnish clear the cache also?

  • 2. Alain replies at 17th September 2010, 11:48 am :

    Yeah, I think so..

    http://www.mail-archive.com/varnish-misc@projects.linpro.no/msg03293.html

  • 3. battisti replies at 6th July 2011, 4:08 am :

    In the samples vcl config file i need to use: purge(“req.http.host == ” req.http.host); because the purge(“req.url == ” req.http.url ” && req.http.host == ” req.http.host);
    didn’t work!

  • 4. John McLear replies at 27th July 2011, 9:26 am :

    This article needs attention, note the > etc in the example. Also it’s worth mentioning this will only purge the cache on one server, not a cluster.

  • 5. Alain Kelder replies at 28th July 2011, 6:13 pm :

    Thanks, John! It seems lots of my older posts had this problem. Not sure when or why some characters got converted into HTML special characters, but it was easy enough to fix all of them at once using this method.

  • 6. Fask replies at 3rd September 2011, 2:44 pm :

    thanks for sharing, excellent tutorial

  • 7. Purging in varnish. | Net&hellip replies at 19th October 2011, 10:41 pm :

    […] purging, Alain Kelder wrote an article, Exploring methods to purge varnish cache, using varnishadm, telnet and http request through telnet. Its quite good article, which gives you […]

  • 8. Erik Shubeck replies at 25th March 2012, 8:19 am :

    I don’t have to update my blogroll every time I add a new blog to my Reader

  • 9. Micki Galler replies at 30th March 2012, 12:29 pm :

    It’s nice to see that we’re still reading each other. Thanks for listing me! Cheers

  • 10. raghu replies at 2nd April 2012, 12:34 am :

    Hi,
    how to know the sites IP and port. to execute the above code.

  • 11. Alain Kelder replies at 2nd April 2012, 7:37 am :

    @raghu
    Well, the IP is the IP of the host that’s running varnish cache. If you shell into the server, the IP can be localhost (127.0.0.1), otherwise, any IP the host is configured with that allows you to connect to the port. The port depends on your configuration. I suspect you have varnish cache listening on port 80. For telnet, try 6082 (that’s default, see DAEMON_OPTS in your varnish startup script or /etc/default/varnish on Debian).

  • 12. Sasha Mackowiak replies at 17th May 2012, 2:02 am :

    Truly beneficial, thanks for placing up it.

  • 13. Arianne Pignataro replies at 17th June 2012, 6:54 am :

    I merely found your website, I enjoy this (and I’ve decided upon your RSS feed)

  • 14. azfar replies at 18th July 2012, 9:37 am :

    Normally I clear the varnish 3.x cache with following command.

    varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret ban.url .

    Now when I am trying to clear it via vpurge.php code I am getting erros.

    send: ban.hash .#.#
    rcv: 200 203
    rcv: —————————–
    rcv: Varnish Cache CLI 1.0
    rcv: —————————–
    rcv: Linux,2.6.32-5-amd64,x86_64,-smalloc,-smalloc,-hcritbit
    rcv:
    rcv: Type ‘help’ for command list.
    rcv: Type ‘quit’ to close CLI session.
    rcv:
    rcv: 101 44
    rcv: Unknown request.
    rcv: Type ‘help’ for more info.
    rcv:
    send: quit
    rcv: 500 22
    rcv: Closing CLI connection

    I have few questions.

    1. In varnish 3.0 purge is replaced from ban so do I have to need to change it in vpurge.php?
    2.What are URL and HOST in index.html?
    3. If I am using varnish secret then how to authenticate in vpurge.php?

  • 15. azfar replies at 18th July 2012, 9:57 am :

    worked ban.

    $pcommand = “ban”;
    $pcommand .= ” req.url == .”;

  • 16. BestSTL replies at 21st August 2012, 1:26 am :

    Thanks, this really helped us. It’s been a pain logging in via SSH each time we wanted to clear something. Very clearly explained.

  • 17. freddy replies at 28th November 2012, 8:07 pm :

    hi azfar,

    may i ask what is the answer to your question 3?

    “3. If I am using varnish secret then how to authenticate in vpurge.php?”

  • 18. david replies at 13th November 2013, 4:01 am :

    hi, thanks for the nice writeup. you might want to add some warning in this article that this is varnish 2. for varnish 3, the command is called “ban”. either way, it does not free storage immediately but just mark content as obsolete.

  • 19. Alain Kelder replies at 13th November 2013, 10:25 am :

    Hi David,

    Purge is still available in varnish 3. Except it’s not the same as the purge in varnish 2. It’s a new method that removes the object from cache and frees up memory. The ban function in varnish 3 is, as you said, what purge was in varnish 2. Ban doesn’t free up memory, unless you use smart bans.

    I personally implement both, and use purge most of the time. When I need to purge multiple objects using a regex, I use ban. Pros and cons of each method are well described here: https://www.varnish-software.com/static/book/Cache_invalidation.html

    Since this change does affect my varnishadm and telnet examples, I’ve added a note as you suggested.

    Thanks,
    Alain

  • 20. Mike replies at 19th August 2015, 10:47 am :

    Thank you for your scripts, they helped inspire the new method of cache invalidation I use employing hash_always_miss which has a dual function: warms up your cache and refreshing current cached copies with new copies from the php backend so you are always serving cached pages, no more purging necessary!

    http://www.htpcguides.com/smart-warm-up-your-wordpress-varnish-cache-scripts/

  • 21. Alain Kelder replies at 20th August 2015, 9:48 am :

    @Mike, thanks for sharing your write up!

    Seems like the main benefit of using hash_always_miss is that while the new content is fetched from the backend, clients aren’t stuck waiting, which could be useful if backend response is slow.

    However it sounds like with hash_always_miss, the old object (e.g. old page, etc) will remain in the cache until it expires?

    Ref:

    https://www.varnish-cache.org/trac/wiki/VCLExampleEnableForceRefresh
    https://www.varnish-cache.org/trac/wiki/VCLExampleHashAlwaysMiss

Leave a comment

NOTE: Enclose quotes in <blockquote></blockquote>. Enclose code in <pre lang="LANG"></pre> (where LANG is one of these).