4 ms·

Varnish 4.0 Released

136 points by nwjsmith 13y ago

stefantalpalaru 13y ago
> Much of the VCL syntax has changed in Varnish 4. Bullet, meet foot.
- lkarsten 13y ago
  For most users it is a 5-10 minute job, no sweat.
  - stefantalpalaru 13y ago
    Take a look at the instructions: https://www.varnish-cache.org/docs/trunk/whats-new/upgrading.html https://www.varnish-cache.org/docs/trunk/whats-new/upgrading...
    - ruben_varnish 13y ago
      Ok, 5-10 min reading it. 5 min writing vcl3to4.pl and 5 min testing it. 5-10 min playing with the new varnishstat. In total a 30 min job. ;-)
      - ruben_varnish 12y ago
        See, it did not take that long: https://github.com/fgsch/varnish3to4 https://github.com/fgsch/varnish3to4
  - personZ 13y ago
    The biggest impact of these sorts of changes are future users who are trying to get their first configuration going, following various web tutorials. Many of those tutorials are now misleading, but via the nature of accumulated web content it won't be changed any time soon. Varnish is a nice project, but the lack of SSL significantly restricts its usefulness, especially in the "SSL everywhere" era. I've simply resorted to using nginx' proxy pass as the cache/load balancer.
    - kitsune_ 13y ago
      A lot of the technical tutorials (that also get posted here) are frankly, absolutely terrible. Tutorials that do not state the required levels of expertise? Tutorials without version numbers of the components that are described in it? Tutorials for absolute beginners that feature commands like echo 'something' | sudo ... without explaining them? Tutorials about using debian-installer without explaining whether locale=en_US will set the system locale during installation to en_US.UTF8 or simply en_US? In the same vein, tutorials about the installation of Postgres that do not mention that your system locale will impact the encoding / locale of your databases? Check, check, check, check and check. Something like RapGenius for technical tutorials might be useful here.
    - X-Istence 13y ago
      The Varnish authors have specifically stated they didn't want to introduce SSL into their process because of the added attack surface that adds. [0] If you want SSL for Varnish, use something like Pound [1], stunnel [2], or HAProxy [3] to do the SSL termination and pass it off to Varnish... I'm partial to Pound because it is lightweight and doesn't try to do any caching, it simply terminates the SSL and passes the request on. [0] https://www.varnish-cache.org/docs/trunk/phk/ssl.html https://www.varnish-cache.org/docs/trunk/phk/ssl.html [1] http://www.apsis.ch/pound http://www.apsis.ch/pound [2] https://www.stunnel.org/index.html https://www.stunnel.org/index.html [3] http://haproxy.1wt.eu http://haproxy.1wt.eu
      - jimktrains2 13y ago
        Re: HAProxy from http://haproxy.1wt.eu http://haproxy.1wt.eu : Update [2012/09/11] : native SSL support was implemented in 1.5-dev12. The points above about CPU usage are still valid though. Wow, I've been out of the loop for a while then. Last time I used HAProxy I was using stud (https://github.com/bumptech/stud https://github.com/bumptech/stud) to terminate.
        ruben_varnish 13y ago
        Well, SSL is there, but it is still under development and has yet to hit a stable HAProxy release.
- ihsw 13y ago
  It's not a point release, such compatibility-breaking changes should be expected.
- gulbrandr 13y ago
  Most of the changes seem to comes from a reorganization of the code, I think they were necessary.
- perbu 13y ago
  The semantics are completely different which has to be reflected in the VCL. I think keeping compatibility at the cost of sanity is pretty dangerous and disregarding compatibility has been one of the success factors for the Linux kernel.
  - stefantalpalaru 13y ago
    > disregarding compatibility has been one of the success factors for the Linux kernel It's actually the other way around. Linus Torvalds dixit[1]: "One of the core kernel rules has always been that we never ever break any external interfaces. That rule has been there since day one, although it's gotten much more explicit only in the last few years. The fact that we break internal interfaces that are not visible to userland is totally irrelevant, and a total red herring." [1]. https://plus.google.com/115250422803614415116/posts/hMT5kW8LKJk https://plus.google.com/115250422803614415116/posts/hMT5kW8L... - second comment
    - perbu 12y ago
      Valid point and I clearly was under the misconception that Linux didn't care that much about the external interfaces. However, VCL is more akin the the kernel modules, relying on the internal APIs which are much less stable. When the semantics on how Varnish operates (with 4.0, the backend threads are completely new) and so the VCL language _needs_ to change in order to reflect this. The external API, made available through HTTP hasn't changed much.
- dwb 13y ago
  Backwards compatibility is vastly overrated, especially when the changes are as modest as the ones that I can see for Varnish 4. Sweep away the cruft. Make things better.
  - stefantalpalaru 13y ago
    How many development hours have been wasted because of Python 3, Perl 6, Lua 5.2, any new Django version, Ruby 1.9, Rails 4, jQuery 1.9, Bootstrap 3, etc. ? Backwards compatibility is as simple as not wasting everybody's time. It should be high on the priority list, right under security and above performance. I like Linus' stance on the issue and I'm sure every developer dealing with syscalls appreciates it as well.
    - dwb 13y ago
      How much development time has been saved and codebase quality increased by making backwards-incompatible changes that improve things drastically? It's a balance, of course – I'm not suggesting old interfaces should be thrown under a bus at the earliest opportunity. But the super-conservative opposite does nothing for progress and ends up with excess technical debt. Varnish 4.0 is a major version release. 3.x still works and will be used by a lot of people for a long time. I really don't see the issue, certainly in this case.
ruben_varnish 13y ago
Don't forget the Varnish 4.0 Release Party on April 29th - http://v4party.varnish-cache.org http://v4party.varnish-cache.org #v4rp
smonte 13y ago
A bit off topic, but Varnish seems to have a valid point about using Varnish and SSL. https://www.varnish-cache.org/docs/trunk/phk/ssl.html https://www.varnish-cache.org/docs/trunk/phk/ssl.html "There is no other way we can guarantee that secret krypto-bits do not leak anywhere they should not, than by fencing in the code that deals with them in a child process, so the bulk of varnish never gets anywhere near the certificates, not even during a core-dump." I came across this when looking for https support few weeks ago.
- ruben_varnish 13y ago
  Indeed. Via one of PHK's latest tweets: https://twitter.com/bsdphk/status/453623583256760321 https://twitter.com/bsdphk/status/453623583256760321 You can see (video) another of his comments regarding OpenSSL from his KeyNote at FOSDEM: http://ftp.belnet.be/FOSDEM/2014/Janson/Sunday/NSA_operation_ORCHESTRA_Annual_Status_Report.webm http://ftp.belnet.be/FOSDEM/2014/Janson/Sunday/NSA_operation...
Dewie 13y ago
Watching that what-is-varnish video, I was surprised that this seems to be a Norwegian company. The narrator had an American (New England?) accent, not a naive/jovial Norwegian accent.
- ruben_varnish 13y ago
  That is correct. We wrote the script and had a native English speaker reading it for us ;-) Even if we have some colleagues who are from New England, we are a pretty multi-cultural Norwegian headquartered company as you can see in our site: https://www.varnish-software.com/about-us https://www.varnish-software.com/about-us Try the "What's coming in Varnish 4.0?" Hangout on Air video for a change: https://www.youtube.com/watch?feature=player_detailpage&v=Stnc_PoReUc#t=0 https://www.youtube.com/watch?feature=player_detailpage&v=St... You will then heard our amazing mix of accents :-)
galenko 13y ago
Can anyone who actually understands what the upgrade document states do a how to or a straight modification of a popular vcl, so people who don't have as much experience and pretty much just use these default vcl files? Here's one that I and a few of my friends use routinely: https://github.com/ewanleith/Wordpress-Server-Configuration-Files/blob/master/default.vcl https://github.com/ewanleith/Wordpress-Server-Configuration-... Like where do I start if I want to use this thing with 4.0? vcl_fetch needs to be renamed, will this work if I just rename it and add vcl version to the top?
- ruben_varnish 13y ago
  You need to read the documentation on this and understand it: https://www.varnish-cache.org/docs/4.0/whats-new/index.html https://www.varnish-cache.org/docs/4.0/whats-new/index.html Otherwise you should just be using a popular file from around the web. Take a look at the Varnish Utilities Directory: https://www.varnish-cache.org/utilities https://www.varnish-cache.org/utilities There you can find two of the best resources for ready to go VCL (for now Varnish 3 only): https://github.com/slashsBin/nuCache https://github.com/slashsBin/nuCache https://github.com/mattiasgeniar/varnish-3.0-configuration-templates https://github.com/mattiasgeniar/varnish-3.0-configuration-t... I am sure that if you ask nicely, the project members of these efforts might just give you a hand and upgrade to VCL 4.0 syntax a usable Wordpress template. Be willing to test and contribute back with whatever you can though.
  - galenko 12y ago
    Or if I currently don't have the time to do this, I have to either stop using varnish completely or not upgrade. I get the whole needing to drop backwards compatibility thing, but without an idiot-proof way of upgrading, you're sentencing users like me, who use Varnish as a sprinkle of magic on the server. Some examples of before/after of actual vcl files are a must in my opinion.
    - snowwrestler 12y ago
      I'm late to this party, but the answer for you is to just not upgrade to Varnish 4.0 yet. Keep using 3.0.5, which is still available and still stable. The "standard" VCL files for popular apps like Wordpress and Drupal will be updated eventually for Varnish 4.0 by the people who maintain them--which are typically not Varnish staff/volunteers, but people who work with those applications specifically. When services like Varnish go through major upgrades like this, it is normal for there to be a lag between the main release, and the general availability of updated "recipes" or plugins.
brunoqc 13y ago
I don't know much about caching. Does this work when serving dynamic content? I mean in a situation where your user absolutely need to see an up-to-date version of your data.
- theg2 13y ago
  Typically you serve up that content separately from the static content. We use it where it doesn't cache for logged in users but is on a rolling cache otherwise. Page updates invalidate the cache immediately as well. On a dynamic site with user content, you're going to be using some sort of object caching depending on your framework and language of choice, Varnish is great for news sites and things of that nature though.
  - byoung2 13y ago
    You can also cache some content on a page, even for logged-in users, and add in the dynamic parts using ESI. For example an ecommerce site could have a category page cached, and have one section at the top that says "Hello, John Doe" generated dynamically and added in using ESI. https://www.varnish-cache.org/trac/wiki/ESIfeatures https://www.varnish-cache.org/trac/wiki/ESIfeatures
    - troels 13y ago
      Or pull the dynamic parts in via ajax - I find that works quite well.
      - kiallmacinnes 12y ago
        I'm not disagreeing.. but lots of people either dislike relying on JS, or would prefer to have the initial page load be a complete page. If your AJAX is pulling down HTML fragments rather than JSON, then Varnish's ESI gives you the best of both worlds, allowing varnish the gather and assemble the initial page from the same endpoints your JS would hit.
- brokentone 13y ago
ryankshaw 13y ago
"Full support for streaming objects through from the backend on a cache miss. Bytes will be sent to 1..n requesting clients as they come in from the backend server." is an awesome feature in and of itself. thank you varnish team, this looks like a great release.

136 points by nwjsmith 13y ago

stefantalpalaru 13y ago

> Much of the VCL syntax has changed in Varnish 4. Bullet, meet foot.

lkarsten 13y ago
For most users it is a 5-10 minute job, no sweat.
- stefantalpalaru 13y ago
  Take a look at the instructions: https://www.varnish-cache.org/docs/trunk/whats-new/upgrading.html https://www.varnish-cache.org/docs/trunk/whats-new/upgrading...
  - ruben_varnish 13y ago
    Ok, 5-10 min reading it. 5 min writing vcl3to4.pl and 5 min testing it. 5-10 min playing with the new varnishstat. In total a 30 min job. ;-)
    - ruben_varnish 12y ago
      See, it did not take that long: https://github.com/fgsch/varnish3to4 https://github.com/fgsch/varnish3to4
- personZ 13y ago
  The biggest impact of these sorts of changes are future users who are trying to get their first configuration going, following various web tutorials. Many of those tutorials are now misleading, but via the nature of accumulated web content it won't be changed any time soon. Varnish is a nice project, but the lack of SSL significantly restricts its usefulness, especially in the "SSL everywhere" era. I've simply resorted to using nginx' proxy pass as the cache/load balancer.
  - kitsune_ 13y ago
    A lot of the technical tutorials (that also get posted here) are frankly, absolutely terrible. Tutorials that do not state the required levels of expertise? Tutorials without version numbers of the components that are described in it? Tutorials for absolute beginners that feature commands like echo 'something' | sudo ... without explaining them? Tutorials about using debian-installer without explaining whether locale=en_US will set the system locale during installation to en_US.UTF8 or simply en_US? In the same vein, tutorials about the installation of Postgres that do not mention that your system locale will impact the encoding / locale of your databases? Check, check, check, check and check. Something like RapGenius for technical tutorials might be useful here.
  - X-Istence 13y ago
    The Varnish authors have specifically stated they didn't want to introduce SSL into their process because of the added attack surface that adds. [0] If you want SSL for Varnish, use something like Pound [1], stunnel [2], or HAProxy [3] to do the SSL termination and pass it off to Varnish... I'm partial to Pound because it is lightweight and doesn't try to do any caching, it simply terminates the SSL and passes the request on. [0] https://www.varnish-cache.org/docs/trunk/phk/ssl.html https://www.varnish-cache.org/docs/trunk/phk/ssl.html [1] http://www.apsis.ch/pound http://www.apsis.ch/pound [2] https://www.stunnel.org/index.html https://www.stunnel.org/index.html [3] http://haproxy.1wt.eu http://haproxy.1wt.eu
    - jimktrains2 13y ago
      Re: HAProxy from http://haproxy.1wt.eu http://haproxy.1wt.eu : Update [2012/09/11] : native SSL support was implemented in 1.5-dev12. The points above about CPU usage are still valid though. Wow, I've been out of the loop for a while then. Last time I used HAProxy I was using stud (https://github.com/bumptech/stud https://github.com/bumptech/stud) to terminate.
      - ruben_varnish 13y ago
        Well, SSL is there, but it is still under development and has yet to hit a stable HAProxy release.
ihsw 13y ago
It's not a point release, such compatibility-breaking changes should be expected.
gulbrandr 13y ago
Most of the changes seem to comes from a reorganization of the code, I think they were necessary.
perbu 13y ago
The semantics are completely different which has to be reflected in the VCL. I think keeping compatibility at the cost of sanity is pretty dangerous and disregarding compatibility has been one of the success factors for the Linux kernel.
- stefantalpalaru 13y ago
  > disregarding compatibility has been one of the success factors for the Linux kernel It's actually the other way around. Linus Torvalds dixit[1]: "One of the core kernel rules has always been that we never ever break any external interfaces. That rule has been there since day one, although it's gotten much more explicit only in the last few years. The fact that we break internal interfaces that are not visible to userland is totally irrelevant, and a total red herring." [1]. https://plus.google.com/115250422803614415116/posts/hMT5kW8LKJk https://plus.google.com/115250422803614415116/posts/hMT5kW8L... - second comment
  - perbu 12y ago
    Valid point and I clearly was under the misconception that Linux didn't care that much about the external interfaces. However, VCL is more akin the the kernel modules, relying on the internal APIs which are much less stable. When the semantics on how Varnish operates (with 4.0, the backend threads are completely new) and so the VCL language _needs_ to change in order to reflect this. The external API, made available through HTTP hasn't changed much.
dwb 13y ago
Backwards compatibility is vastly overrated, especially when the changes are as modest as the ones that I can see for Varnish 4. Sweep away the cruft. Make things better.
- stefantalpalaru 13y ago
  How many development hours have been wasted because of Python 3, Perl 6, Lua 5.2, any new Django version, Ruby 1.9, Rails 4, jQuery 1.9, Bootstrap 3, etc. ? Backwards compatibility is as simple as not wasting everybody's time. It should be high on the priority list, right under security and above performance. I like Linus' stance on the issue and I'm sure every developer dealing with syscalls appreciates it as well.
  - dwb 13y ago
    How much development time has been saved and codebase quality increased by making backwards-incompatible changes that improve things drastically? It's a balance, of course – I'm not suggesting old interfaces should be thrown under a bus at the earliest opportunity. But the super-conservative opposite does nothing for progress and ends up with excess technical debt. Varnish 4.0 is a major version release. 3.x still works and will be used by a lot of people for a long time. I really don't see the issue, certainly in this case.

ruben_varnish 13y ago

Don't forget the Varnish 4.0 Release Party on April 29th - http://v4party.varnish-cache.org http://v4party.varnish-cache.org #v4rp

smonte 13y ago

A bit off topic, but Varnish seems to have a valid point about using Varnish and SSL. https://www.varnish-cache.org/docs/trunk/phk/ssl.html https://www.varnish-cache.org/docs/trunk/phk/ssl.html "There is no other way we can guarantee that secret krypto-bits do not leak anywhere they should not, than by fencing in the code that deals with them in a child process, so the bulk of varnish never gets anywhere near the certificates, not even during a core-dump." I came across this when looking for https support few weeks ago.

ruben_varnish 13y ago
Indeed. Via one of PHK's latest tweets: https://twitter.com/bsdphk/status/453623583256760321 https://twitter.com/bsdphk/status/453623583256760321 You can see (video) another of his comments regarding OpenSSL from his KeyNote at FOSDEM: http://ftp.belnet.be/FOSDEM/2014/Janson/Sunday/NSA_operation_ORCHESTRA_Annual_Status_Report.webm http://ftp.belnet.be/FOSDEM/2014/Janson/Sunday/NSA_operation...

Dewie 13y ago

Watching that what-is-varnish video, I was surprised that this seems to be a Norwegian company. The narrator had an American (New England?) accent, not a naive/jovial Norwegian accent.

ruben_varnish 13y ago
That is correct. We wrote the script and had a native English speaker reading it for us ;-) Even if we have some colleagues who are from New England, we are a pretty multi-cultural Norwegian headquartered company as you can see in our site: https://www.varnish-software.com/about-us https://www.varnish-software.com/about-us Try the "What's coming in Varnish 4.0?" Hangout on Air video for a change: https://www.youtube.com/watch?feature=player_detailpage&v=Stnc_PoReUc#t=0 https://www.youtube.com/watch?feature=player_detailpage&v=St... You will then heard our amazing mix of accents :-)

galenko 13y ago

Can anyone who actually understands what the upgrade document states do a how to or a straight modification of a popular vcl, so people who don't have as much experience and pretty much just use these default vcl files? Here's one that I and a few of my friends use routinely: https://github.com/ewanleith/Wordpress-Server-Configuration-Files/blob/master/default.vcl https://github.com/ewanleith/Wordpress-Server-Configuration-... Like where do I start if I want to use this thing with 4.0? vcl_fetch needs to be renamed, will this work if I just rename it and add vcl version to the top?

ruben_varnish 13y ago
You need to read the documentation on this and understand it: https://www.varnish-cache.org/docs/4.0/whats-new/index.html https://www.varnish-cache.org/docs/4.0/whats-new/index.html Otherwise you should just be using a popular file from around the web. Take a look at the Varnish Utilities Directory: https://www.varnish-cache.org/utilities https://www.varnish-cache.org/utilities There you can find two of the best resources for ready to go VCL (for now Varnish 3 only): https://github.com/slashsBin/nuCache https://github.com/slashsBin/nuCache https://github.com/mattiasgeniar/varnish-3.0-configuration-templates https://github.com/mattiasgeniar/varnish-3.0-configuration-t... I am sure that if you ask nicely, the project members of these efforts might just give you a hand and upgrade to VCL 4.0 syntax a usable Wordpress template. Be willing to test and contribute back with whatever you can though.
- galenko 12y ago
  Or if I currently don't have the time to do this, I have to either stop using varnish completely or not upgrade. I get the whole needing to drop backwards compatibility thing, but without an idiot-proof way of upgrading, you're sentencing users like me, who use Varnish as a sprinkle of magic on the server. Some examples of before/after of actual vcl files are a must in my opinion.
  - snowwrestler 12y ago
    I'm late to this party, but the answer for you is to just not upgrade to Varnish 4.0 yet. Keep using 3.0.5, which is still available and still stable. The "standard" VCL files for popular apps like Wordpress and Drupal will be updated eventually for Varnish 4.0 by the people who maintain them--which are typically not Varnish staff/volunteers, but people who work with those applications specifically. When services like Varnish go through major upgrades like this, it is normal for there to be a lag between the main release, and the general availability of updated "recipes" or plugins.

brunoqc 13y ago

I don't know much about caching. Does this work when serving dynamic content? I mean in a situation where your user absolutely need to see an up-to-date version of your data.

theg2 13y ago
Typically you serve up that content separately from the static content. We use it where it doesn't cache for logged in users but is on a rolling cache otherwise. Page updates invalidate the cache immediately as well. On a dynamic site with user content, you're going to be using some sort of object caching depending on your framework and language of choice, Varnish is great for news sites and things of that nature though.
- byoung2 13y ago
  You can also cache some content on a page, even for logged-in users, and add in the dynamic parts using ESI. For example an ecommerce site could have a category page cached, and have one section at the top that says "Hello, John Doe" generated dynamically and added in using ESI. https://www.varnish-cache.org/trac/wiki/ESIfeatures https://www.varnish-cache.org/trac/wiki/ESIfeatures
  - troels 13y ago
    Or pull the dynamic parts in via ajax - I find that works quite well.
    - kiallmacinnes 12y ago
      I'm not disagreeing.. but lots of people either dislike relying on JS, or would prefer to have the initial page load be a complete page. If your AJAX is pulling down HTML fragments rather than JSON, then Varnish's ESI gives you the best of both worlds, allowing varnish the gather and assemble the initial page from the same endpoints your JS would hit.
brokentone 13y ago

ryankshaw 13y ago

"Full support for streaming objects through from the backend on a cache miss. Bytes will be sent to 1..n requesting clients as they come in from the backend server." is an awesome feature in and of itself. thank you varnish team, this looks like a great release.