5 ms·

As mentioned in the article, Cloudflare expressly prohibits benchmarking their services in the tos. I think it's rather disingenuous for Cloudflare to publish

by jjeaff 5y ago

As mentioned in the article, Cloudflare expressly prohibits benchmarking their services in the tos. I think it's rather disingenuous for Cloudflare to publish their own benchmarks calling out competitors when they won't allow anyone to run their own tests and comparisons.

charcircuit 5y ago
They don't prohibit it outright. They just require you to ask first. The article made no mention of if they asked or not.
- deadmutex 5y ago
  Why tho?
  - hysan 5y ago
    Probably for the same reason that I've seen processes put in place for performance testing a service (internally or on production) - because it puts a significant load on the service and automatic rate limiting, flagging, etc can be triggered.
  - dekhn 5y ago
    if you benchmark your competitor, your competitor should have the ability to provide tuning so their numbers look as good as possible, before you run to the press.
    - deadmutex 5y ago
      Unless your competitor was offering a complimentary service to do that for all customers (tuning) then perhaps it shouldn't be done. I believe the idea of benchmarking is to get a sense of how the service would perform for a customer. Consider the following scenario: You calling your ISP that you're going to do a benchmark. Then the ISP gives you the best service, at the expense of other customers bandwidth, for the duration of the test. Then after the test, the ISP removes the QoS modification, and reverts to the old behavior. That could end up with a biased, unrealistic result.
      - dekhn 5y ago
        No, benchmarking aims to identify the peak performance of a service, not the average customer experience. Real Benchmarking is done for people who are serious about performance. marketing benchmarking is designed to attract naive customers. As for the latter, that benchmark wouldn't reproduce and would require extensive configuration (which in a sense is just a part of negotiation between the customer and the client). If a company got caught faking benchmarks run by competitors and it could be shown publicly they did, that's pretty much the end of that company's public reputation.
  - int_19h 5y ago
    https://en.wikipedia.org/wiki/David_DeWitt#DeWitt_Clause https://en.wikipedia.org/wiki/David_DeWitt#DeWitt_Clause
- girvo 5y ago
  > They don't prohibit it outright. They just require you to ask first Frankly, that's a distinction without a difference in this context, at least in my opinion.
- jchw 5y ago
  I feel like that's splitting hairs. if you say something is "prohibited without express written permission," to me that reads as "prohibited." Especially when it comes to benchmarking, because it effectively means that you can never do an impartial benchmark of the service.
  - deleted 5y ago
    [deleted]
  - onion2k 5y ago
    to me that reads as "prohibited." That's because you don't have a good enough reason to want to ask for permission though. If you were spending $xx thousands on edge workers every year you'd just ask, and you'd rule CF out if they said no. But, very likely, they'd say yes.
    - cryptonym 5y ago
      You have to ask for a written exemption that may or may not be granted. With enough money you can discuss any ToS. That's a fake opening with the same effect as prohibiting something by ToS.
- InGoodFaith 5y ago
  Asking before benchmarking seems to be a sure way to poison your results. Who's to say that the performance of the services won't coincidentally perform better after you've given your request to test? I am sure they might mention some legalese of "avoiding putting arduous strain on the services which may cause disruption to our other users" as a justification, but that PR speak won't fly with me since there is a separate section that already references that. I say this as a big fan of Cloudflare and I am sure they don't need to use such underhanded tactics to make their services look good.
  - deleted 5y ago
    [deleted]
- judge2020 5y ago
  In contrast to the other comments, this is a valid concern regarding whether or not they asked first. I assume legal cleared this blog post, which would likely infer that legal either (A) got permission to publish benchmarks, or (B) doesn't think that part of the TOS is enforceable/doesn't expect CF to actually do anything about it. The TOS clause, for reference: https://www.cloudflare.com/terms/#:~:text=(f)%20perform%20or%20publish%20any%20benchmark%20tests%20or%20analyses%20relating%20to%20the%20Services%20without%20Cloudflare%E2%80%99s%20written%20consent%3B https://www.cloudflare.com/terms/#:~:text=(f)%20perform%20or... (ctrl-f benchmark )
  - InGoodFaith 5y ago
    They stated in the blogpost that they only performed the benchmarks against their own services > So instead, we ran the same tests (echoing headers and measuring TTFB via Catchpoint) against our own platform
  - kailanb 5y ago
    To clarify, Fastly did not perform any benchmarks on Cloudflare's network. We ran our own benchmark, comparable to the one that Cloudflare ran on our platform, and compared those results to the ones that Cloudflare published as part of the original blog post. The Cloudflare terms prohibit benchmarking without their explicit permission.
    - dsizzle 5y ago
      > To clarify, Fastly did not perform any benchmarks on Cloudflare's network. We ran our own benchmark… Am I the only having cognitive dissonance here? I was confused how they complained about not being able to perform benchmarks but then… ran a benchmark? There must be some distinction here that I’m missing
      - kailanb 5y ago
        Sorry, I was trying to clarify in my comment but failed to put it through clearly. When I say "We ran our own benchmark" I mean Fastly ran a benchmark on a Fastly Compute@Edge service.
- barkingcat 5y ago
  In a totally different scenario - if someone requires you to ask before undergoing a drug test, for example, if an Olympic athlete made drug testing conditional upon having notice, it wouldn't be a real test.
  - robertlagrant 5y ago
    It's not quite the same. Cloudflare aren't going to rejig their global service every time someone wants to benchmark their code.
- Dylan16807 5y ago
  "If you get permission to bypass this, you can bypass it" is implicitly part of all agreements. So explicitly including that doesn't change it. It's the same as an outright prohibition.
- austhrow743 5y ago
  Every prohibited thing is not prohibited if you get an exception. Nothing is prohibited huzzah!
wraptile 5y ago
I'm not a lawyer but could someone explain to me could you be legally persecuted for this? It's "Terms of Service" - I broke the terms by performing the benchmarks and Cloudflare no longer provides service to me. Lets say I'm a scientist and I'm fine with never using cloudflare services but this benchmark is important for my field - would cloudflare have a case here? I'd imagine that would be quite absurd but corporate law very much already is.
- kingcharles 5y ago
  Yes. Here in Illinois violating terms of service carries up to 5 years in prison. https://www.ilga.gov/legislation/ilcs/fulltext.asp?DocName=072000050K17-51 https://www.ilga.gov/legislation/ilcs/fulltext.asp?DocName=0...
- gary_0 5y ago
  The feds ruined Aaron Swartz's life for violating "Terms of Service", so people are right to be fearful. https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act#Aaron_Swartz https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act#A...
- dathinab 5y ago
  Depends on the country you are in, in the US, thinks seem bad. See the other posts. In the EU there are various ways of ToS are limited in their power, if that applies here I'm not sure. But they are basically forbidding you to evaluate how well a product you (might be) paying for works, depending on what they advertised they might even prevent you from evaluation if the advertised product characteristics uphold. Especially in the later case I guess that the ToS clause would be void.
- lelandfe 5y ago
  Anti-benchmarking clauses are sometimes called DeWitt clauses after Oracle famously tried to get a professor fired for testing their offerings. https://danluu.com/anon-benchmark/ https://danluu.com/anon-benchmark/ I never knew how true that anecdote was but that’s typically the boogeyman story that gets passed around for things like this
hn_throwaway_99 5y ago
Note the CloudFlare CEO subsequently commented elsewhere on this post that they would remove the problematic language from their ToS: https://news.ycombinator.com/item?id=29468771 https://news.ycombinator.com/item?id=29468771

by jjeaff 5y ago

charcircuit 5y ago

They don't prohibit it outright. They just require you to ask first. The article made no mention of if they asked or not.

deadmutex 5y ago
Why tho?
- hysan 5y ago
  Probably for the same reason that I've seen processes put in place for performance testing a service (internally or on production) - because it puts a significant load on the service and automatic rate limiting, flagging, etc can be triggered.
- dekhn 5y ago
  if you benchmark your competitor, your competitor should have the ability to provide tuning so their numbers look as good as possible, before you run to the press.
  - deadmutex 5y ago
    Unless your competitor was offering a complimentary service to do that for all customers (tuning) then perhaps it shouldn't be done. I believe the idea of benchmarking is to get a sense of how the service would perform for a customer. Consider the following scenario: You calling your ISP that you're going to do a benchmark. Then the ISP gives you the best service, at the expense of other customers bandwidth, for the duration of the test. Then after the test, the ISP removes the QoS modification, and reverts to the old behavior. That could end up with a biased, unrealistic result.
    - dekhn 5y ago
      No, benchmarking aims to identify the peak performance of a service, not the average customer experience. Real Benchmarking is done for people who are serious about performance. marketing benchmarking is designed to attract naive customers. As for the latter, that benchmark wouldn't reproduce and would require extensive configuration (which in a sense is just a part of negotiation between the customer and the client). If a company got caught faking benchmarks run by competitors and it could be shown publicly they did, that's pretty much the end of that company's public reputation.
- int_19h 5y ago
  https://en.wikipedia.org/wiki/David_DeWitt#DeWitt_Clause https://en.wikipedia.org/wiki/David_DeWitt#DeWitt_Clause
girvo 5y ago
> They don't prohibit it outright. They just require you to ask first Frankly, that's a distinction without a difference in this context, at least in my opinion.
jchw 5y ago
I feel like that's splitting hairs. if you say something is "prohibited without express written permission," to me that reads as "prohibited." Especially when it comes to benchmarking, because it effectively means that you can never do an impartial benchmark of the service.
- deleted 5y ago
  [deleted]
- onion2k 5y ago
  to me that reads as "prohibited." That's because you don't have a good enough reason to want to ask for permission though. If you were spending $xx thousands on edge workers every year you'd just ask, and you'd rule CF out if they said no. But, very likely, they'd say yes.
  - cryptonym 5y ago
    You have to ask for a written exemption that may or may not be granted. With enough money you can discuss any ToS. That's a fake opening with the same effect as prohibiting something by ToS.
InGoodFaith 5y ago
Asking before benchmarking seems to be a sure way to poison your results. Who's to say that the performance of the services won't coincidentally perform better after you've given your request to test? I am sure they might mention some legalese of "avoiding putting arduous strain on the services which may cause disruption to our other users" as a justification, but that PR speak won't fly with me since there is a separate section that already references that. I say this as a big fan of Cloudflare and I am sure they don't need to use such underhanded tactics to make their services look good.
- deleted 5y ago
  [deleted]
judge2020 5y ago
In contrast to the other comments, this is a valid concern regarding whether or not they asked first. I assume legal cleared this blog post, which would likely infer that legal either (A) got permission to publish benchmarks, or (B) doesn't think that part of the TOS is enforceable/doesn't expect CF to actually do anything about it. The TOS clause, for reference: https://www.cloudflare.com/terms/#:~:text=(f)%20perform%20or%20publish%20any%20benchmark%20tests%20or%20analyses%20relating%20to%20the%20Services%20without%20Cloudflare%E2%80%99s%20written%20consent%3B https://www.cloudflare.com/terms/#:~:text=(f)%20perform%20or... (ctrl-f benchmark )
- InGoodFaith 5y ago
  They stated in the blogpost that they only performed the benchmarks against their own services > So instead, we ran the same tests (echoing headers and measuring TTFB via Catchpoint) against our own platform
- kailanb 5y ago
  To clarify, Fastly did not perform any benchmarks on Cloudflare's network. We ran our own benchmark, comparable to the one that Cloudflare ran on our platform, and compared those results to the ones that Cloudflare published as part of the original blog post. The Cloudflare terms prohibit benchmarking without their explicit permission.
  - dsizzle 5y ago
    > To clarify, Fastly did not perform any benchmarks on Cloudflare's network. We ran our own benchmark… Am I the only having cognitive dissonance here? I was confused how they complained about not being able to perform benchmarks but then… ran a benchmark? There must be some distinction here that I’m missing
    - kailanb 5y ago
      Sorry, I was trying to clarify in my comment but failed to put it through clearly. When I say "We ran our own benchmark" I mean Fastly ran a benchmark on a Fastly Compute@Edge service.
barkingcat 5y ago
In a totally different scenario - if someone requires you to ask before undergoing a drug test, for example, if an Olympic athlete made drug testing conditional upon having notice, it wouldn't be a real test.
- robertlagrant 5y ago
  It's not quite the same. Cloudflare aren't going to rejig their global service every time someone wants to benchmark their code.
Dylan16807 5y ago
"If you get permission to bypass this, you can bypass it" is implicitly part of all agreements. So explicitly including that doesn't change it. It's the same as an outright prohibition.
austhrow743 5y ago
Every prohibited thing is not prohibited if you get an exception. Nothing is prohibited huzzah!

wraptile 5y ago

I'm not a lawyer but could someone explain to me could you be legally persecuted for this? It's "Terms of Service" - I broke the terms by performing the benchmarks and Cloudflare no longer provides service to me. Lets say I'm a scientist and I'm fine with never using cloudflare services but this benchmark is important for my field - would cloudflare have a case here? I'd imagine that would be quite absurd but corporate law very much already is.

kingcharles 5y ago
Yes. Here in Illinois violating terms of service carries up to 5 years in prison. https://www.ilga.gov/legislation/ilcs/fulltext.asp?DocName=072000050K17-51 https://www.ilga.gov/legislation/ilcs/fulltext.asp?DocName=0...
gary_0 5y ago
The feds ruined Aaron Swartz's life for violating "Terms of Service", so people are right to be fearful. https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act#Aaron_Swartz https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act#A...
dathinab 5y ago
Depends on the country you are in, in the US, thinks seem bad. See the other posts. In the EU there are various ways of ToS are limited in their power, if that applies here I'm not sure. But they are basically forbidding you to evaluate how well a product you (might be) paying for works, depending on what they advertised they might even prevent you from evaluation if the advertised product characteristics uphold. Especially in the later case I guess that the ToS clause would be void.
lelandfe 5y ago
Anti-benchmarking clauses are sometimes called DeWitt clauses after Oracle famously tried to get a professor fired for testing their offerings. https://danluu.com/anon-benchmark/ https://danluu.com/anon-benchmark/ I never knew how true that anecdote was but that’s typically the boogeyman story that gets passed around for things like this

hn_throwaway_99 5y ago

Note the CloudFlare CEO subsequently commented elsewhere on this post that they would remove the problematic language from their ToS: https://news.ycombinator.com/item?id=29468771 https://news.ycombinator.com/item?id=29468771