5 ms·

This is what happens: Within an hour of V8 pushing the fix for this, our build automation alerted me that it had picked up the patch and built a new release of

This is what happens: Within an hour of V8 pushing the fix for this, our build automation alerted me that it had picked up the patch and built a new release of the Workers Runtime for us. I clicked a button to start rolling it out. After quick one-click approvals from EM and SRE, the release went to canary. After running there for a short time to verify no problems, I clicked to roll it out world-wide, which is in progress now. It will be everywhere within a couple hours. Rolling out an update like this causes no visible impact to customers. In comparison, when a zero-day is dropped in a VM implementation used by a cloud service, it generally takes much longer to roll out a fix, and often requires rebooting all customer VMs which can be disruptive. > Their use is in some sense way worse than a web browser, where you would have to wait for someone to come to your likely-niche page rather than just push the attack to get run everywhere. I may be biased but I think this is debatable. If you want to target a specific victim, it's much easier to get that person to click a link than it is to randomly land on the same machine as them in a cloud service.

lazypenguin 6y ago
There is something so pleasing about this process being so fully automated that it can be managed in a few clicks. Kudos, I love reading things like this.
tatersolid 6y ago
What checks do you perform on that upstream code before building and running it? I imagine a supply-chain attack would be devastating, even if the code only made it to canaries. Your build infrastructure at least could easily be compromised by a nefarious makefile addition.
- kentonv 6y ago
  Someone would have to get a malicious patch merged into Google's V8 release branch first. I also personally do a sanity-check review of the patches before rolling out the update.
Thaxll 6y ago
Pretty interesting, so people don't have to follow updates on those major projects, your pipeline actually does it for you.
ec109685 6y ago
The other benefit you have is that anything running in the workers has to go through CI/CD, so you’d see anyone actively trying to exploit this. Hope you there is also some scanning for this put in place.
- kentonv 6y ago
  All code executed on Workers has to have been uploaded through our API -- we don't allow eval() or dynamically compiling Wasm at runtime. This ensures that we have a paper trail in case of an exploit.
saurik 6y ago
Thanks for the response! This didn't really answer what I was curious about, though: like, you answered what happens during the minutes after the fix being pushed, but I am curious about the minutes after the exploit being released, as the mention of "zero day" made me think that this bug could only have been fixed in the past few hours (and so there were likely hours of Cloudflare going "omg what now?" with engineers trying to help come up with a patch, etc.). However... for this comment I then wanted to see how long ago that patch did drop, and it turns out "a week ago" :/... and the real issue is that neither Chrome nor Edge have merged the patch?! https://therecord.media/security-researcher-drops-chrome-and-edge-zero-day-on-twitter/ https://therecord.media/security-researcher-drops-chrome-and... > Agarwal said he responsibly reported the V8 security issue to the Chromium team, which patched the bug in the V8 code last week; however, the patch has not yet been integrated into official releases of downstream Chromium-based browsers such as Chrome, Edge, and others. So uhh... damn ;P. > I may be biased but I think this is debatable. If you want to target a specific victim, ... FWIW, I had meant Cloudflare as the victim, not one of Cloudflare's users: I can push code to Cloudflare's servers and directly run it, but I can't do the same thing to a user (as they have to click a link). I appreciate your point, though (though I would also then want to look at "number of people I can quickly affect"). (I am curious about this because I want to better understand the mitigations in place by a service such as Cloudflare, as I am interested in the security ramifications of doing similar v8 work in distributed systems.)
- kentonv 6y ago
  > which patched the bug in the V8 code last week This does not appear to be true. AFAICT the first patch was merged today: https://chromium-review.googlesource.com/c/v8/v8/+/2820971 https://chromium-review.googlesource.com/c/v8/v8/+/2820971 (It was then rapidly cherry-picked into release branches, after which our automation picked it up.) > I am curious about this because I want to better understand the mitigations in place by a service such as Cloudflare, as I am interested in the security ramifications of doing similar v8 work in distributed systems. Here's a blog post with some more details about our security model and defenses-in-depth: https://blog.cloudflare.com/mitigating-spectre-and-other-security-threats-the-cloudflare-workers-security-model/ https://blog.cloudflare.com/mitigating-spectre-and-other-sec...
dboreham 6y ago
Forgot to begin post with "Horse's Mouth here".
bassdropvroom 6y ago
> Within an hour of V8 pushing the fix for this, our build automation alerted me that it had picked up the patch and built a new release of the Workers Runtime for us. I clicked a button to start rolling it out. After quick one-click approvals from EM and SRE, the release went to canary. After running there for a short time to verify no problems, I clicked to roll it out world-wide, which is in progress now. It will be everywhere within a couple hours. Rolling out an update like this causes no visible impact to customers. Great workflow! I long for the day when I can start for a company that actually has their automation as efficient as this. Few question, do you have a way of differentiating critical patches as this? If so, does that trigger an alert for the on-call person? Or do you still wait until working hours before such a change is pushed?
- Xunjin 6y ago
  Can you include me too?! I wish I could automate the hell like they do :P
  - lofties 6y ago
    You can!
    - throwaway823882 6y ago
      Specifically you can do two things: 1) planned incremental improvements, 2) simpler designs. For 1), write down the entire manual workflow. Start automating pieces that are easy to automate, even if someone has to run the automation manually. Continue to automate the in-between/manual pieces. For this you can use autonomation to fall back to manual work if complete automation is too difficult/risky. For 2), look at your system's design. See where the design/tools/implementation/etc limit the ability to easily automate. To replace a given workflow section, you can a) replace some part of your system with a functionally-equivalent but easier to automate solution, or b) embed some new functionality/logic into that section of the system that extends and slightly abstracts the functionality, so that you can later easily replace the old system with a simpler one. To get extra time/resources to spend on the automation, you can do a cost-benefit analysis. Record the manual processes' impact for a month, and compare this to an automated solution scaled out to 12-36 months (and the cost to automate it). Also include "costs" like time to market for deliverables and quality improvements. Business people really like charts, graphs, and cost saving estimates.
strstr 6y ago
Most VM providers now support live migration for common machine varieties, so forced reboots are uncommon.
DSingularity 6y ago
I think you are right — it is debatable [1]. I would argue that it is easier to find ways to exploit determinism of scheduling/allocation than finding ways to exploit humans. [1] https://hovav.net/ucsd/dist/cloudsec.pdf https://hovav.net/ucsd/dist/cloudsec.pdf
dtx1 6y ago
That's impressive, in fact I would argue that this is as close to best-practice as you can get. I would love to read a blog post with details how you set this up!

lazypenguin 6y ago

There is something so pleasing about this process being so fully automated that it can be managed in a few clicks. Kudos, I love reading things like this.

tatersolid 6y ago

What checks do you perform on that upstream code before building and running it? I imagine a supply-chain attack would be devastating, even if the code only made it to canaries. Your build infrastructure at least could easily be compromised by a nefarious makefile addition.

kentonv 6y ago
Someone would have to get a malicious patch merged into Google's V8 release branch first. I also personally do a sanity-check review of the patches before rolling out the update.

Thaxll 6y ago

Pretty interesting, so people don't have to follow updates on those major projects, your pipeline actually does it for you.

ec109685 6y ago

The other benefit you have is that anything running in the workers has to go through CI/CD, so you’d see anyone actively trying to exploit this. Hope you there is also some scanning for this put in place.

kentonv 6y ago
All code executed on Workers has to have been uploaded through our API -- we don't allow eval() or dynamically compiling Wasm at runtime. This ensures that we have a paper trail in case of an exploit.

saurik 6y ago

Thanks for the response! This didn't really answer what I was curious about, though: like, you answered what happens during the minutes after the fix being pushed, but I am curious about the minutes after the exploit being released, as the mention of "zero day" made me think that this bug could only have been fixed in the past few hours (and so there were likely hours of Cloudflare going "omg what now?" with engineers trying to help come up with a patch, etc.). However... for this comment I then wanted to see how long ago that patch did drop, and it turns out "a week ago" :/... and the real issue is that neither Chrome nor Edge have merged the patch?! https://therecord.media/security-researcher-drops-chrome-and-edge-zero-day-on-twitter/ https://therecord.media/security-researcher-drops-chrome-and... > Agarwal said he responsibly reported the V8 security issue to the Chromium team, which patched the bug in the V8 code last week; however, the patch has not yet been integrated into official releases of downstream Chromium-based browsers such as Chrome, Edge, and others. So uhh... damn ;P. > I may be biased but I think this is debatable. If you want to target a specific victim, ... FWIW, I had meant Cloudflare as the victim, not one of Cloudflare's users: I can push code to Cloudflare's servers and directly run it, but I can't do the same thing to a user (as they have to click a link). I appreciate your point, though (though I would also then want to look at "number of people I can quickly affect"). (I am curious about this because I want to better understand the mitigations in place by a service such as Cloudflare, as I am interested in the security ramifications of doing similar v8 work in distributed systems.)

kentonv 6y ago
> which patched the bug in the V8 code last week This does not appear to be true. AFAICT the first patch was merged today: https://chromium-review.googlesource.com/c/v8/v8/+/2820971 https://chromium-review.googlesource.com/c/v8/v8/+/2820971 (It was then rapidly cherry-picked into release branches, after which our automation picked it up.) > I am curious about this because I want to better understand the mitigations in place by a service such as Cloudflare, as I am interested in the security ramifications of doing similar v8 work in distributed systems. Here's a blog post with some more details about our security model and defenses-in-depth: https://blog.cloudflare.com/mitigating-spectre-and-other-security-threats-the-cloudflare-workers-security-model/ https://blog.cloudflare.com/mitigating-spectre-and-other-sec...

dboreham 6y ago

Forgot to begin post with "Horse's Mouth here".

bassdropvroom 6y ago

> Within an hour of V8 pushing the fix for this, our build automation alerted me that it had picked up the patch and built a new release of the Workers Runtime for us. I clicked a button to start rolling it out. After quick one-click approvals from EM and SRE, the release went to canary. After running there for a short time to verify no problems, I clicked to roll it out world-wide, which is in progress now. It will be everywhere within a couple hours. Rolling out an update like this causes no visible impact to customers. Great workflow! I long for the day when I can start for a company that actually has their automation as efficient as this. Few question, do you have a way of differentiating critical patches as this? If so, does that trigger an alert for the on-call person? Or do you still wait until working hours before such a change is pushed?

Xunjin 6y ago
Can you include me too?! I wish I could automate the hell like they do :P
- lofties 6y ago
  You can!
  - throwaway823882 6y ago
    Specifically you can do two things: 1) planned incremental improvements, 2) simpler designs. For 1), write down the entire manual workflow. Start automating pieces that are easy to automate, even if someone has to run the automation manually. Continue to automate the in-between/manual pieces. For this you can use autonomation to fall back to manual work if complete automation is too difficult/risky. For 2), look at your system's design. See where the design/tools/implementation/etc limit the ability to easily automate. To replace a given workflow section, you can a) replace some part of your system with a functionally-equivalent but easier to automate solution, or b) embed some new functionality/logic into that section of the system that extends and slightly abstracts the functionality, so that you can later easily replace the old system with a simpler one. To get extra time/resources to spend on the automation, you can do a cost-benefit analysis. Record the manual processes' impact for a month, and compare this to an automated solution scaled out to 12-36 months (and the cost to automate it). Also include "costs" like time to market for deliverables and quality improvements. Business people really like charts, graphs, and cost saving estimates.

strstr 6y ago

Most VM providers now support live migration for common machine varieties, so forced reboots are uncommon.

DSingularity 6y ago

I think you are right — it is debatable [1]. I would argue that it is easier to find ways to exploit determinism of scheduling/allocation than finding ways to exploit humans. [1] https://hovav.net/ucsd/dist/cloudsec.pdf https://hovav.net/ucsd/dist/cloudsec.pdf

dtx1 6y ago

That's impressive, in fact I would argue that this is as close to best-practice as you can get. I would love to read a blog post with details how you set this up!