13 ms·

OK... So, I'm the tech lead for Cloudflare Workers. In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until

OK... So, I'm the tech lead for Cloudflare Workers. In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don't know anything about either beyond what's publicly visible. But as long as we're already mud slinging, I'd like to take the opportunity to get a little something off my chest. Fastly has been trumpeting for years that Compute@Edge has 35 microsecond cold starts, or whatever, and repeatedly posting blog posts comparing that against 5 milliseconds for Workers, and implying that they are 150x faster. If you look at the details, it turns out that 35 microsecond time is actually how long they take to start a new request, given that the application is already loaded in memory. A hot start, not a cold start. Whereas Workers' 5ms includes time to load the application from disk (which is the biggest contributor to total time). Our hot start time is also a few microseconds, but that doesn't seem like an interesting number? We never called this out, it didn't seem worth arguing over. But excuse me if I'm not impressed by claims of false comparisons... On a serious note, I've been saying for decades that benchmarks are almost always meaningless, because different technologies will have different strengths and weaknesses, so you usually can't tell anything about how your use case will perform unless you actually test that use case. So, I would encourage everyone to run your own test and don't just go on other people's numbers. It's great that Fastly has opened up C@E for self-service testing so that people can actually try it out.

zimmerx 5y ago
I think this is all super fair commentary and very balanced. As a Compute@Edge user though how do they test CloudFlare when the ToS says otherwise? Test and break the ToS and hope for the best? It's a bit of a shame that it exists.
refulgentis 5y ago
I suggest deleting this. I'm a completely neutral third-party with little interest in the subject at hand, and their post doesn't come off like mud-slinging, at all. This then makes your response look accusatory and helps further a broader narrative post about Cloudflare's behavior here, which it seems your goal is to dispel, given your statement on whether engineering was involved.* A suggestion for something actionable that'll make things better in this difficult moment: email product counsel about starting the process to override your EULA and enable Compute@Edge to do benchmarking. It'll take forever and it's the only way to have an objective, evolving conversation about benchmarks, which your leads will (at least, should) want after this happened. * intentionally vague here, in order to give you space to delete and leave no record
- ethbr0 5y ago
  I found parent interesting and relevant. But mostly importantly, kentonv's last paragraph. At the end of the day, benchmarking for your use case is the only true north. Everything else, on all sides, is marketing fluff.
  - refulgentis 5y ago
    Please edit the username involved out of your post. Thank you for considering!! It's helpful to the community to leave people feeling comfortable contributing, and removing contributions is a core part of that. You only have a short amount of time, edits are disabled after...I'm not sure, perhaps an hour. If you regret removing it, you can always come back and reply again. :)
    - judge2020 5y ago
      Well, deletion is also prohibited as soon as anyone replies to the comment, so his username is set in stone even if he has the chance to edit it for a few hours.
    - dekhn 5y ago
      what are you even talking about. you mean the reference to kentonv? as stated, kenton is the tech lead for a major cloud project (as well as a respected open source developer). It's pretty clear from watching what he says that he is generally makes wise statements, while also providing some useful color (for which I am eternally grateful).
      - refulgentis 5y ago
        I did not, and do not, reference the username involved in any of my posts, until now. The first 3 replies to Kenton, including mine, don't recognize Kenton immediately or on a first name basis just from their account name. So, we all reacted the way people react to the generic situation at hand: be careful, you're attaching your name and position to something that further commits your employer to a narrative you're trying to dispel. I appreciate you providing this info, between that and Kenton's replies about themself (ex. about protos and cap'n proto) its clears that he has extensive experience in industry and at big companies, and certainly knows the tradeoff they're making. That's awesome, and different enough from the median situation to obviate early advice re: commenting on your employer.
hereforphone 5y ago
Agree with the other comment. Unless you have explicit permission I'd be careful what you post here about your employer
- NavinF 5y ago
  Did you notice his username? I’m pretty sure he has permission from himself lol.
  - hereforphone 5y ago
    Believe it or not, I don't have any clue who he is. And the post was edited btw.
    - kentonv 5y ago
      I made a very small edit to one sentence to change "don't believe either side's benchmarks" (or something like that) to "don't just go on other people's numbers", since the latter better reflects what I meant -- not that the numbers are false or intentionally misleading, but that they probably aren't reflective of your particular use case.
yellow_lead 5y ago
> So, I would encourage everyone to run your own test and don't believe what either side posts. I don't have the full TOS in front of me but wouldn't this violate it?
- howdydoo 5y ago
  I was about to disagree, but I just re-read Cloudflare's language, and you're right. Cloudflare forbids performing any tests or analyses, whether or not you publish the results. Even running something like `curl http://my-workers-site http://my-workers-site | time`, for your own personal curiosity, is in violation of their ToS. That's even more egregious than Oracle DB's infamous DeWitt clause. Then again, a Cloudflare employee just gave written consent in this post for us to perform benchmarks, so maybe we're all off the hook! :)
  - deleted 5y ago
    [deleted]
  - worik 5y ago
    I any civilised jurisdiction such ToS are unenforceable. Why do firms continue to make unenforceable claims? (My least favourite are restriction of trade clauses in employment contracts that are simply illegal, unless paid for, where I come from. But they appear all the time in contracts)
    - scoopertrooper 5y ago
      I just skimmed the ToS, the only remedy they offer to a violation of the terms is a denial of access to their service. Which is something they definitely can enforce.
    - Mezzie 5y ago
      Because it requires lawyers to prove that the claims are unenforceable, and most smaller companies aren't going to have the pockets needed to sustain the legal back-and-forth, which forces them to settle even if the big company is in the wrong. It's legal 'chicken'.
flumpcakes 5y ago
> I've been saying for decades that benchmarks are almost always meaningless I find this statement egregious: benchmarks are probably the most important thing in producing quality software and are often (outside of FAANG) the first thing to be ignored. I have the displeasure of working with extremely poor software for the majority of my career, where a simple benchmark and brain-power would illuminate the issues days-weeks-months before production issues occurred. I also see "benchmarks are useless" used a lot by vendors who perform poorly in said benchmarks. Or use useless benchmarks (like TTFB which can easily be fudged) which is just underhanded.
- kentonv 5y ago
  Oh absolutely, benchmarks are great for measuring your own performance and improving it. I meant they aren't great for comparing two very different technologies. I've looked at a lot of serialization benchmarks, having once maintained Protocol Buffers and later Cap'n Proto, and it's amazing how wildly different the results are depending on the shape of the data you're serializing.
  - _57jb 5y ago
    I absolutely agree with this sentiment. Many engineers, product advocates, and sales engineers/architects build careers, playbooks, and talk tracks around debunking or handwaving others (and sometimes their own) benchmarks. The pressure to make a sale and your livelihood against a customer that is looking at products on a top ten scorecard is a lot of pressure
  - seoaeu 5y ago
    It is extremely helpful to be able to estimate performance without building out a whole system. For instance 10ms vs 100ms vs 1s latency makes a world of difference as far as what is possible and what component might be the bottleneck. Though, I agree that the “horse race” between several highly optimized and broadly similar libraries may be far less interesting
deleted 5y ago
[deleted]
1cvmask 5y ago
Unfortunately not all companies follow your great obvious recommendation and listen to snake oil salesman dressed up as consultants and industry analysts: So, I would encourage everyone to run your own test and don't believe what either side posts. --- Otherwise there would be no business for the Gartners and Accentures of the world if customers were actually savvy themselves.
deleted 5y ago
[deleted]
nicoburns 5y ago
Sounds like you might be in a position to get some of these things changed and pave the way for fair benchmarks to take place.
ksec 5y ago
Personal Opinion only. I dont think Fastly has ever explicitly called out a 35us against Cloudflare 5ms. ( Please Correct me if I am wrong ) They may have some marketing material about 35us being faster than the rest of industry. But that doesn't imply it is Cloudflare because there are lots of player in the industry. First name comes to mind would be AWS. Compared to Cloudflare blog post directly naming Fastly in a benchmark is completely different. Second being Fastly was unhappy about this test and posted on twitter for weeks even quoting and mentioning @cloudflare. But I agree everyone should run their own benchmarks.
- dcow 5y ago
  >> So, I would encourage everyone to run your own test and don't just go on other people's numbers. > But I agree everyone should run their own benchmarks. But they literally can't because the ToS forbids it.
  - deleted 5y ago
    [deleted]
  - ewalk153 5y ago
    Anti benchmark clauses usually prohibit publication. This wouldn’t exclude running two systems with your workload and documenting the results for internal use.
    - hn_throwaway_99 5y ago
      Except the CloudFlare ToS (posted as an image from the tweet thread above) says "perform or publish benchmark tests or analyses..."
  - onion2k 5y ago
    They don't forbid it. They request you get permission first. No one seems to have bothered asking to find out if CF say no. Benchmarking accurately is hard, so CF trying to make sure they're not going to be subject to a "report" by someone whose tests are garbage, or by a competitor who's benchmarking poorly to make their own offering look better isn't totally unreasonable. Ideally, as far as CF are concerned, they'd be so far ahead of the competition it wouldn't matter, but presumably that wasn't the case when the ToS were written. The fact the CEO posted to say the clause is being removed means we can all freely test in future. This is how things are supposed to work. Someone does something, someone else reacts, and the first person updates their position. Expecting everyone to get things perfectly correct on the first try is nonsense.
dwwoelfel 5y ago
It has been two weeks since Fastly's VP of Eng called out your ToS and errors in your benchmark in her original tweet thread [1]. I would hope that Cloudflare would have a better response than this that directly addresses Fastly's claims. Are you going to remove the ToS clause or issue a correction on the blog post? [1] https://twitter.com/lxt/status/1462896850055352320 https://twitter.com/lxt/status/1462896850055352320
- kentonv 5y ago
  To be clear my comment here is not in any way, shape, or form, Cloudflare's response. I am here representing only myself.
  - dcow 5y ago
    Sure, but you self-identified as someone responsible for the project and then suggested people go violate the ToS by running their own benchmarks. I think the community here calling out the double standards and asking for an update/response is entirely fair. If you didn't want the flak then leave off the "I'm the tech lead on CF workers" intro. Seems to me like the ball is in your court to at least try and make your advice actionable.
    - kentonv 5y ago
      I do not believe I suggested doing anything against the ToS. I think you're misinterpreting the clause. But not being a lawyer, I don't really want to get into that discussion. If I didn't state upfront that I was the tech lead of Workers, someone would (rightly) call me out for astroturfing.
      - hn_throwaway_99 5y ago
        First off, just want to say thanks for your posts, I found they give useful context and I really appreciate them. I don't want the following to come off as unnecessarily argumentative, but regarding the ToS, I'm not a lawyer either, but my "ability to read English" interpretation of the section on "perform or publish benchmarks..." certainly sounds like it is prohibiting folks from doing their own side-by-side comparisons. Which is, of course, nonsense, because any engineer worth their salt would do their own analysis, even if they didn't publish it. Just sounds to me like the CloudFlare lawyers got a little too aggressive to the point of absurdity, but I still think it's fair to call out CloudFlare for this.
tiew9Vii 5y ago
> Fastly has been trumpeting for years that Compute@Edge This is the main problem with Compute@Edge, "for years". They have been advertising publicly C@E for years but it was a restricted private beta. In my opinion the state of C@E has historically been falsely advertised. You could not go to the website and sign up, put in your credit card details and use C@E. If you contacted support they told you it's not generally available, it's for exiting customers and no more seats are left, it was at capacity. After I bit of Twitter ranting one of the C@E advocates at Fastly did reach out and did get me test access and he did genuinely help as much as he could. I did like what I saw and in my personal testing found Fastly faster than Cloudflare Workers using Rust / Wasm / Wasi. Also one of my services where having issues in Cloudflare workers with ByteBuffers sending binary data, I don't know why but it just worked in Fastly Wasi runtime. Regardless today my preference would still be to use Cloudflare. You go to the Cloudflare website, sign up with minimal information and are then good to go. Need to move outside of the test tier? put in your credit card, click a button, done. The pricing for Cloudflare is transparent and hugely attractive for workers. Fastly, there is no clear pricing, they are applying there CDN pricing model with a $50 minimum fee which you need to go and read in the small print. That is ok for enterprise clients but obviously they are missing up on all the pay as you go small scale devs that Cloudflare attracts and all the other pay for what you use PaaS's attract that then take it in to their day jobs becoming advocates where the big contracts are signed. Fastly need to decide if they are a CDN or a internet company with a CDN product. They seem undecided what they are at the moment and have poor product structuring and it's genuinely very hard to go put in your credit card and give them money! Cloudflare are quite clear they are a internet technology company now not a CDN and make it extremely easy for people to give them money in a few clicks.
- numair 5y ago
  > They have been advertising publicly C@E for years but it was a restricted private beta. In my opinion the state of C@E has historically been falsely advertised. You could not go to the website and sign up, put in your credit card details and use C@E. If you contacted support they told you it's not generally available, it's for exiting customers and no more seats are left, it was at capacity. This is a MAJOR problem in our industry. Products that exist, but don't actually exist (this sitiuation). "Platforms" that are only platforms if you've got a secret deal nobody tells you that you need to have (Twitter, Facebook). "Open" systems that close themselves off once you hit a very, very low threshold of use, and then force you to do a lot of not-very-open things to regain access... I could go on and on. (I'll note that this is separate from the problem of AI-based fraud detection run amok.)
unityByFreedom 5y ago
Thank you for your hard work on workers and for your openness in these forums. As a dev, it sounds to me like Fastly's whole purpose is wrapped up in attacking Cloudflare whereas Cloudflare is focused on innovating.
resonious 5y ago
Sounds like there's a lack of communication if Cloudflare published a benchmark blog post without the tech lead even knowing (better yet, finding out about the competitor's post first!). I imagine it's pretty normal for the front-facing blog/PR department to be a bit disconnected from development, but we can't entirely blame Fastly for "complaining" seeing as Cloudflare already uploaded a bit of a diss post. Anyways I think I once read a phenomenon where corporate environments eventually reach a point where the outside world knows more about the company than the employees. Not sure if this is actually an example of that or not. Also I agree these blog posts are all kind of meaningless and people should do their own investigation if they're considering alternatives.
jiggawatts 5y ago
I suspect you're mixing up microseconds with milliseconds. 5 microseconds is a round trip distance of ~500 m in optical fibre, and is hence totally irrelevant for 100% of CloudFlare's end-users... unless they happen to live in a 2-block radius of a Cloudflare PoP and have their ISP providing them with Internet connectivity with direct fibre from the same data centre...
- kentonv 5y ago
  No, I didn't mix them up.
staticassertion 5y ago
Thanks for posting this. I think a lot of people are hyperfocusing on some of your wording, but this post was very informative.
co_bee31 5y ago
>> But as long as we're already mud slinging Strange statement. Didn't Cloudflare start the mud slinging? I have no shares in this. But Cloudflare publishing questionable benchmarks with competing services while prohibiting competitors publishing benchmarks with Cloudflare services is certainly not fair play. Reminds me of Oracle.
deanCommie 5y ago
> In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don't know anything about either beyond what's publicly visible. Take some responsibility and ownership for the authority and soft power you wield as a very senior individual contributor. You've been in this industry too long, and have risen too far to not feel a sense of personal investment and responsibility to 1) the benchmarks your company publishes about your product, and 2) the policies around public experimentation and transparency with that product. Talk to your manager, talk to his manager, figure out who is responsible for both and FIX IT. You are the only major provider that has a DeWitt Clause. It's embarrassing.
- robertlagrant 5y ago
  If you bother to read the thread the CEO has already commented to say they're changing that ToS. Why give advice that no-one asked for and no-one needed? Isn't that pretty much the worst?
- kentonv 5y ago
  My manager's manager is the CEO. Yes, we all discussed it and everyone agreed the clause should be removed, but at the time of my comment I wasn't sure if I was authorized to announce that publicly. He has since posted in this thread.
nix23 5y ago
Every benchmark should start with: For us/me XX works better because of...

zimmerx 5y ago

I think this is all super fair commentary and very balanced. As a Compute@Edge user though how do they test CloudFlare when the ToS says otherwise? Test and break the ToS and hope for the best? It's a bit of a shame that it exists.

refulgentis 5y ago

I suggest deleting this. I'm a completely neutral third-party with little interest in the subject at hand, and their post doesn't come off like mud-slinging, at all. This then makes your response look accusatory and helps further a broader narrative post about Cloudflare's behavior here, which it seems your goal is to dispel, given your statement on whether engineering was involved.* A suggestion for something actionable that'll make things better in this difficult moment: email product counsel about starting the process to override your EULA and enable Compute@Edge to do benchmarking. It'll take forever and it's the only way to have an objective, evolving conversation about benchmarks, which your leads will (at least, should) want after this happened. * intentionally vague here, in order to give you space to delete and leave no record

ethbr0 5y ago
I found parent interesting and relevant. But mostly importantly, kentonv's last paragraph. At the end of the day, benchmarking for your use case is the only true north. Everything else, on all sides, is marketing fluff.
- refulgentis 5y ago
  Please edit the username involved out of your post. Thank you for considering!! It's helpful to the community to leave people feeling comfortable contributing, and removing contributions is a core part of that. You only have a short amount of time, edits are disabled after...I'm not sure, perhaps an hour. If you regret removing it, you can always come back and reply again. :)
  - judge2020 5y ago
    Well, deletion is also prohibited as soon as anyone replies to the comment, so his username is set in stone even if he has the chance to edit it for a few hours.
  - dekhn 5y ago
    what are you even talking about. you mean the reference to kentonv? as stated, kenton is the tech lead for a major cloud project (as well as a respected open source developer). It's pretty clear from watching what he says that he is generally makes wise statements, while also providing some useful color (for which I am eternally grateful).
    - refulgentis 5y ago
      I did not, and do not, reference the username involved in any of my posts, until now. The first 3 replies to Kenton, including mine, don't recognize Kenton immediately or on a first name basis just from their account name. So, we all reacted the way people react to the generic situation at hand: be careful, you're attaching your name and position to something that further commits your employer to a narrative you're trying to dispel. I appreciate you providing this info, between that and Kenton's replies about themself (ex. about protos and cap'n proto) its clears that he has extensive experience in industry and at big companies, and certainly knows the tradeoff they're making. That's awesome, and different enough from the median situation to obviate early advice re: commenting on your employer.

hereforphone 5y ago

Agree with the other comment. Unless you have explicit permission I'd be careful what you post here about your employer

NavinF 5y ago
Did you notice his username? I’m pretty sure he has permission from himself lol.
- hereforphone 5y ago
  Believe it or not, I don't have any clue who he is. And the post was edited btw.
  - kentonv 5y ago
    I made a very small edit to one sentence to change "don't believe either side's benchmarks" (or something like that) to "don't just go on other people's numbers", since the latter better reflects what I meant -- not that the numbers are false or intentionally misleading, but that they probably aren't reflective of your particular use case.

yellow_lead 5y ago

> So, I would encourage everyone to run your own test and don't believe what either side posts. I don't have the full TOS in front of me but wouldn't this violate it?

howdydoo 5y ago
I was about to disagree, but I just re-read Cloudflare's language, and you're right. Cloudflare forbids performing any tests or analyses, whether or not you publish the results. Even running something like `curl http://my-workers-site http://my-workers-site | time`, for your own personal curiosity, is in violation of their ToS. That's even more egregious than Oracle DB's infamous DeWitt clause. Then again, a Cloudflare employee just gave written consent in this post for us to perform benchmarks, so maybe we're all off the hook! :)
- deleted 5y ago
  [deleted]
- worik 5y ago
  I any civilised jurisdiction such ToS are unenforceable. Why do firms continue to make unenforceable claims? (My least favourite are restriction of trade clauses in employment contracts that are simply illegal, unless paid for, where I come from. But they appear all the time in contracts)
  - scoopertrooper 5y ago
    I just skimmed the ToS, the only remedy they offer to a violation of the terms is a denial of access to their service. Which is something they definitely can enforce.
  - Mezzie 5y ago
    Because it requires lawyers to prove that the claims are unenforceable, and most smaller companies aren't going to have the pockets needed to sustain the legal back-and-forth, which forces them to settle even if the big company is in the wrong. It's legal 'chicken'.

flumpcakes 5y ago

> I've been saying for decades that benchmarks are almost always meaningless I find this statement egregious: benchmarks are probably the most important thing in producing quality software and are often (outside of FAANG) the first thing to be ignored. I have the displeasure of working with extremely poor software for the majority of my career, where a simple benchmark and brain-power would illuminate the issues days-weeks-months before production issues occurred. I also see "benchmarks are useless" used a lot by vendors who perform poorly in said benchmarks. Or use useless benchmarks (like TTFB which can easily be fudged) which is just underhanded.

kentonv 5y ago
Oh absolutely, benchmarks are great for measuring your own performance and improving it. I meant they aren't great for comparing two very different technologies. I've looked at a lot of serialization benchmarks, having once maintained Protocol Buffers and later Cap'n Proto, and it's amazing how wildly different the results are depending on the shape of the data you're serializing.
- _57jb 5y ago
  I absolutely agree with this sentiment. Many engineers, product advocates, and sales engineers/architects build careers, playbooks, and talk tracks around debunking or handwaving others (and sometimes their own) benchmarks. The pressure to make a sale and your livelihood against a customer that is looking at products on a top ten scorecard is a lot of pressure
- seoaeu 5y ago
  It is extremely helpful to be able to estimate performance without building out a whole system. For instance 10ms vs 100ms vs 1s latency makes a world of difference as far as what is possible and what component might be the bottleneck. Though, I agree that the “horse race” between several highly optimized and broadly similar libraries may be far less interesting

deleted 5y ago

[deleted]

1cvmask 5y ago

Unfortunately not all companies follow your great obvious recommendation and listen to snake oil salesman dressed up as consultants and industry analysts: So, I would encourage everyone to run your own test and don't believe what either side posts. --- Otherwise there would be no business for the Gartners and Accentures of the world if customers were actually savvy themselves.

deleted 5y ago

[deleted]

nicoburns 5y ago

Sounds like you might be in a position to get some of these things changed and pave the way for fair benchmarks to take place.

ksec 5y ago

Personal Opinion only. I dont think Fastly has ever explicitly called out a 35us against Cloudflare 5ms. ( Please Correct me if I am wrong ) They may have some marketing material about 35us being faster than the rest of industry. But that doesn't imply it is Cloudflare because there are lots of player in the industry. First name comes to mind would be AWS. Compared to Cloudflare blog post directly naming Fastly in a benchmark is completely different. Second being Fastly was unhappy about this test and posted on twitter for weeks even quoting and mentioning @cloudflare. But I agree everyone should run their own benchmarks.

dcow 5y ago
>> So, I would encourage everyone to run your own test and don't just go on other people's numbers. > But I agree everyone should run their own benchmarks. But they literally can't because the ToS forbids it.
- deleted 5y ago
  [deleted]
- ewalk153 5y ago
  Anti benchmark clauses usually prohibit publication. This wouldn’t exclude running two systems with your workload and documenting the results for internal use.
  - hn_throwaway_99 5y ago
    Except the CloudFlare ToS (posted as an image from the tweet thread above) says "perform or publish benchmark tests or analyses..."
- onion2k 5y ago
  They don't forbid it. They request you get permission first. No one seems to have bothered asking to find out if CF say no. Benchmarking accurately is hard, so CF trying to make sure they're not going to be subject to a "report" by someone whose tests are garbage, or by a competitor who's benchmarking poorly to make their own offering look better isn't totally unreasonable. Ideally, as far as CF are concerned, they'd be so far ahead of the competition it wouldn't matter, but presumably that wasn't the case when the ToS were written. The fact the CEO posted to say the clause is being removed means we can all freely test in future. This is how things are supposed to work. Someone does something, someone else reacts, and the first person updates their position. Expecting everyone to get things perfectly correct on the first try is nonsense.

dwwoelfel 5y ago

It has been two weeks since Fastly's VP of Eng called out your ToS and errors in your benchmark in her original tweet thread [1]. I would hope that Cloudflare would have a better response than this that directly addresses Fastly's claims. Are you going to remove the ToS clause or issue a correction on the blog post? [1] https://twitter.com/lxt/status/1462896850055352320 https://twitter.com/lxt/status/1462896850055352320

kentonv 5y ago
To be clear my comment here is not in any way, shape, or form, Cloudflare's response. I am here representing only myself.
- dcow 5y ago
  Sure, but you self-identified as someone responsible for the project and then suggested people go violate the ToS by running their own benchmarks. I think the community here calling out the double standards and asking for an update/response is entirely fair. If you didn't want the flak then leave off the "I'm the tech lead on CF workers" intro. Seems to me like the ball is in your court to at least try and make your advice actionable.
  - kentonv 5y ago
    I do not believe I suggested doing anything against the ToS. I think you're misinterpreting the clause. But not being a lawyer, I don't really want to get into that discussion. If I didn't state upfront that I was the tech lead of Workers, someone would (rightly) call me out for astroturfing.
    - hn_throwaway_99 5y ago
      First off, just want to say thanks for your posts, I found they give useful context and I really appreciate them. I don't want the following to come off as unnecessarily argumentative, but regarding the ToS, I'm not a lawyer either, but my "ability to read English" interpretation of the section on "perform or publish benchmarks..." certainly sounds like it is prohibiting folks from doing their own side-by-side comparisons. Which is, of course, nonsense, because any engineer worth their salt would do their own analysis, even if they didn't publish it. Just sounds to me like the CloudFlare lawyers got a little too aggressive to the point of absurdity, but I still think it's fair to call out CloudFlare for this.

tiew9Vii 5y ago

> Fastly has been trumpeting for years that Compute@Edge This is the main problem with Compute@Edge, "for years". They have been advertising publicly C@E for years but it was a restricted private beta. In my opinion the state of C@E has historically been falsely advertised. You could not go to the website and sign up, put in your credit card details and use C@E. If you contacted support they told you it's not generally available, it's for exiting customers and no more seats are left, it was at capacity. After I bit of Twitter ranting one of the C@E advocates at Fastly did reach out and did get me test access and he did genuinely help as much as he could. I did like what I saw and in my personal testing found Fastly faster than Cloudflare Workers using Rust / Wasm / Wasi. Also one of my services where having issues in Cloudflare workers with ByteBuffers sending binary data, I don't know why but it just worked in Fastly Wasi runtime. Regardless today my preference would still be to use Cloudflare. You go to the Cloudflare website, sign up with minimal information and are then good to go. Need to move outside of the test tier? put in your credit card, click a button, done. The pricing for Cloudflare is transparent and hugely attractive for workers. Fastly, there is no clear pricing, they are applying there CDN pricing model with a $50 minimum fee which you need to go and read in the small print. That is ok for enterprise clients but obviously they are missing up on all the pay as you go small scale devs that Cloudflare attracts and all the other pay for what you use PaaS's attract that then take it in to their day jobs becoming advocates where the big contracts are signed. Fastly need to decide if they are a CDN or a internet company with a CDN product. They seem undecided what they are at the moment and have poor product structuring and it's genuinely very hard to go put in your credit card and give them money! Cloudflare are quite clear they are a internet technology company now not a CDN and make it extremely easy for people to give them money in a few clicks.

numair 5y ago
> They have been advertising publicly C@E for years but it was a restricted private beta. In my opinion the state of C@E has historically been falsely advertised. You could not go to the website and sign up, put in your credit card details and use C@E. If you contacted support they told you it's not generally available, it's for exiting customers and no more seats are left, it was at capacity. This is a MAJOR problem in our industry. Products that exist, but don't actually exist (this sitiuation). "Platforms" that are only platforms if you've got a secret deal nobody tells you that you need to have (Twitter, Facebook). "Open" systems that close themselves off once you hit a very, very low threshold of use, and then force you to do a lot of not-very-open things to regain access... I could go on and on. (I'll note that this is separate from the problem of AI-based fraud detection run amok.)

unityByFreedom 5y ago

Thank you for your hard work on workers and for your openness in these forums. As a dev, it sounds to me like Fastly's whole purpose is wrapped up in attacking Cloudflare whereas Cloudflare is focused on innovating.

resonious 5y ago

Sounds like there's a lack of communication if Cloudflare published a benchmark blog post without the tech lead even knowing (better yet, finding out about the competitor's post first!). I imagine it's pretty normal for the front-facing blog/PR department to be a bit disconnected from development, but we can't entirely blame Fastly for "complaining" seeing as Cloudflare already uploaded a bit of a diss post. Anyways I think I once read a phenomenon where corporate environments eventually reach a point where the outside world knows more about the company than the employees. Not sure if this is actually an example of that or not. Also I agree these blog posts are all kind of meaningless and people should do their own investigation if they're considering alternatives.

jiggawatts 5y ago

I suspect you're mixing up microseconds with milliseconds. 5 microseconds is a round trip distance of ~500 m in optical fibre, and is hence totally irrelevant for 100% of CloudFlare's end-users... unless they happen to live in a 2-block radius of a Cloudflare PoP and have their ISP providing them with Internet connectivity with direct fibre from the same data centre...

kentonv 5y ago
No, I didn't mix them up.

staticassertion 5y ago

Thanks for posting this. I think a lot of people are hyperfocusing on some of your wording, but this post was very informative.

co_bee31 5y ago

>> But as long as we're already mud slinging Strange statement. Didn't Cloudflare start the mud slinging? I have no shares in this. But Cloudflare publishing questionable benchmarks with competing services while prohibiting competitors publishing benchmarks with Cloudflare services is certainly not fair play. Reminds me of Oracle.

deanCommie 5y ago

> In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don't know anything about either beyond what's publicly visible. Take some responsibility and ownership for the authority and soft power you wield as a very senior individual contributor. You've been in this industry too long, and have risen too far to not feel a sense of personal investment and responsibility to 1) the benchmarks your company publishes about your product, and 2) the policies around public experimentation and transparency with that product. Talk to your manager, talk to his manager, figure out who is responsible for both and FIX IT. You are the only major provider that has a DeWitt Clause. It's embarrassing.

robertlagrant 5y ago
If you bother to read the thread the CEO has already commented to say they're changing that ToS. Why give advice that no-one asked for and no-one needed? Isn't that pretty much the worst?
kentonv 5y ago
My manager's manager is the CEO. Yes, we all discussed it and everyone agreed the clause should be removed, but at the time of my comment I wasn't sure if I was authorized to announce that publicly. He has since posted in this thread.

nix23 5y ago

Every benchmark should start with: For us/me XX works better because of...