3 ms·

My uneducated hypothesis is that Fastly runs varnish. Presumably they have some process that collects data from their config system generates the VCL (varnishes

by Dobbs 5y ago

My uneducated hypothesis is that Fastly runs varnish. Presumably they have some process that collects data from their config system generates the VCL (varnishes psuedo-C configuration language which compiles directly to C). Somehow a customer configured something in such a way that it generated a bad VCL file which then caused it to lose all configuration, or caused one domain to incorrectly garble up traffic for all domains. I can poke plenty of holes in this hypothesis, like fastly likely not deploying configuration to all nodes but only subsets. Looking forward to the deeper post.

tyingq 5y ago
It's interesting to me that they still run Varnish, since it doesn't have https/tls built-in. I do get that VCL is more expressive than similar capabilities in Nginx, HAproxy, etc. But it would seem like less work to add expressiveness to one of those (via Lua maybe?) than to maintain both Varnish and the separate components needed for both tls ingress and egress.
- foobarbazetc 5y ago
  Varnish and VCL were the hotness at the time Fastly was coming up so it sort of makes sense, but Varnish also doesn’t support WebSockets, can’t proxy GRPC, etc so they’re very limited in functionality vs CloudFlare. I doubt they’d build it on Varnish today, but it’s a bit late now since they allow custom VCL (which has now proven to be a terrible idea) and will have to support that for eternity. They can run two or more serving stacks side by side though if it comes to that. And to add to your point, they also have a separate process that speaks QUIC. It’s an interesting tech stack with a lot of technical debt.
  - tyingq 5y ago
    >it’s a bit late now since they allow custom VCL (which has now proven to be a terrible idea) Ah, okay. I took a look, and it appears they at least didn't allow varnish modules or inline C. But, still, a fairly hefty anchor for the future.
  - anonydsfsfs 5y ago
    > Varnish also doesn’t support WebSockets Yes it does. I recently added WebSockets support to a Varnish instance. See https://varnish-cache.org/docs/trunk/users-guide/vcl-example-websockets.html https://varnish-cache.org/docs/trunk/users-guide/vcl-example...
- longwave 5y ago
  You can write custom VCL snippets directly in the Fastly control panel. Migrating all existing customer VCL to another language would be an enormous task.
LeifCarrotson 5y ago
They say as much here [1]. > Fastly is a shared infrastructure. By allowing the use of inline C code, we could potentially give a single user the power to read, write to, or write from everything. As a result, our varnish process (i.e., files on disk, memory of the varnish user's processes) would become unprotected because inline C code opens the potential for users to do things like crash servers, steal data, or run a botnet. Personally, my hypothesis is that somebody uploaded a configuration for their domain `https://IVCL_{raise(SIGSEGV)}.com https://IVCL_{raise(SIGSEGV)}.com` (edit: the preceding URL used to contain a heart emoji between I and VCL, apparently HN prefers ASCII, too) in a way that, rather than converting to Punycode, passed a few bytes that weren't in the 96 legal characters accepted by the VCC compiler and caused some kind of undefined behavior. [1] https://docs.fastly.com/en/guides/guide-to-vcl#embedding-inline-c-code-in-vcl https://docs.fastly.com/en/guides/guide-to-vcl#embedding-inl...
simlevesque 5y ago
Fastly directly allows you to run custom VCL: https://developer.fastly.com/reference/vcl/ https://developer.fastly.com/reference/vcl/

by Dobbs 5y ago

tyingq 5y ago

It's interesting to me that they still run Varnish, since it doesn't have https/tls built-in. I do get that VCL is more expressive than similar capabilities in Nginx, HAproxy, etc. But it would seem like less work to add expressiveness to one of those (via Lua maybe?) than to maintain both Varnish and the separate components needed for both tls ingress and egress.

foobarbazetc 5y ago
Varnish and VCL were the hotness at the time Fastly was coming up so it sort of makes sense, but Varnish also doesn’t support WebSockets, can’t proxy GRPC, etc so they’re very limited in functionality vs CloudFlare. I doubt they’d build it on Varnish today, but it’s a bit late now since they allow custom VCL (which has now proven to be a terrible idea) and will have to support that for eternity. They can run two or more serving stacks side by side though if it comes to that. And to add to your point, they also have a separate process that speaks QUIC. It’s an interesting tech stack with a lot of technical debt.
- tyingq 5y ago
  >it’s a bit late now since they allow custom VCL (which has now proven to be a terrible idea) Ah, okay. I took a look, and it appears they at least didn't allow varnish modules or inline C. But, still, a fairly hefty anchor for the future.
- anonydsfsfs 5y ago
  > Varnish also doesn’t support WebSockets Yes it does. I recently added WebSockets support to a Varnish instance. See https://varnish-cache.org/docs/trunk/users-guide/vcl-example-websockets.html https://varnish-cache.org/docs/trunk/users-guide/vcl-example...
longwave 5y ago
You can write custom VCL snippets directly in the Fastly control panel. Migrating all existing customer VCL to another language would be an enormous task.

LeifCarrotson 5y ago

They say as much here [1]. > Fastly is a shared infrastructure. By allowing the use of inline C code, we could potentially give a single user the power to read, write to, or write from everything. As a result, our varnish process (i.e., files on disk, memory of the varnish user's processes) would become unprotected because inline C code opens the potential for users to do things like crash servers, steal data, or run a botnet. Personally, my hypothesis is that somebody uploaded a configuration for their domain `https://IVCL_{raise(SIGSEGV)}.com https://IVCL_{raise(SIGSEGV)}.com` (edit: the preceding URL used to contain a heart emoji between I and VCL, apparently HN prefers ASCII, too) in a way that, rather than converting to Punycode, passed a few bytes that weren't in the 96 legal characters accepted by the VCC compiler and caused some kind of undefined behavior. [1] https://docs.fastly.com/en/guides/guide-to-vcl#embedding-inline-c-code-in-vcl https://docs.fastly.com/en/guides/guide-to-vcl#embedding-inl...

simlevesque 5y ago

Fastly directly allows you to run custom VCL: https://developer.fastly.com/reference/vcl/ https://developer.fastly.com/reference/vcl/