3 ms·

The HN title is misleading: the terms do not forbid benchmarking, just public dissemination of the benchmarks. This is a standard software EULA clause. One rea

by mholt 8y ago

The HN title is misleading: the terms do not forbid benchmarking, just public dissemination of the benchmarks. This is a standard software EULA clause. One reason we decided to keep this clause in the Caddy EULA (which I should clarify here only applies to official binaries, not the open source, Apache-built binaries you can make yourself) is because we found out that very few people are expert enough to benchmark correctly. I've read a dozen Caddy benchmarks, for example, that turned out to be based on false assumptions or had hidden factors or were simply not reproducible (and not just by me). Benchmarking requires expertise that, it turns out, very few people have. I don't think I even have enough skills to do it correctly and meaningfully. Also, web servers are complex enough (in terms of both configuration and all the layers involved with networking stacks) that one correct benchmark is not generally useful to the next person. Spreading wrong performance information can hurt a business. It's not that there's anything to hide or any desire to take away your freedom -- and I would normally be one to assume the worst from any large company -- it's just business: they don't want the risk of bad PR based on a possibly false premise, especially when that information tends to only create negative hype rather than actually being useful. Anyway, this link doesn't seem like news. Just usual HN hype.

icelancer 8y ago
>> The HN title is misleading: the terms do not forbid benchmarking, just public dissemination of the benchmarks. If a tree falls in the forest, does it make a sound? The title is not misleading. Performing 69,000 benchmarks but being unable to publish them is de facto banning benchmarks, period. Hiding behind the idea that benchmarking is hard therefore it should be banned because it might be fake news is ludicrous. It is also not standard software EULA licensing, unless you think Oracle's practices are somehow industry standard and good for everyone.
- mesozoic 8y ago
  This way they can make you spend a ton of time bringing up a trial version on your internal systems to do the benchmarking. Effectively forcing lock in and sunk costs fallacy thinking then put you into tons of meetings that are really just attempts to upsell.
carapace 8y ago
> Benchmarking requires expertise that, it turns out, very few people have. I don't think I even have enough skills to do it correctly and meaningfully. Very important and often overlooked point. But I wonder, why not forbid public dissemination of inaccurate, non-reproducible benchmarks? > Spreading wrong performance information can hurt a business. Wouldn't that be libel? (IANAL)
- SpicyLemonZest 8y ago
  The problem is when the benchmark is accurate, reproducible, and based on a completely nonsensical scenario. It's not libel to say "Jira is 50% slower than Bugzilla" when your benchmark serially creates 25000 tickets, but it's not a fair claim either. That's not to say that anti-benchmarking licenses are the right solution, of course. I can just sympathize with why Atlassian wants one.
- mesozoic 8y ago
  Expertise so rare it seems Atlassian themselves don't have it.
- SamReidHughes 8y ago
  Suppose you state that a product performed poorly on measurement Y of benchmark X. The problem is not reproducibility or accuracy, it's that either Y or X are always really stupid in some way or another.
jkaplowitz 8y ago
Honestly, if I had widely used software with an enforceable EULA term which allowed me to benchmark but not publicly disclose bad results I found, it would make for even worse PR for the company: I'd be able to go to a tech industry reporter saying "I ran benchmarks on this software, and I think many people would be interested in the results, but the company forbade me to release them publicly. I will still share them non-publicly with any interested parties under NDA." Or if private dissemination were also forbidden, I'd change the wording accordingly. The better way for a company to handle this concern, if they feel it's important, is to proactively run and release benchmarks including commentary on the results, together with everything necessary for anyone to reproduce their results. Even better if they fund a trustworthy neutral third party to do this instead, with proper disclosure of the funding. They can then respond very effectively to bad PR about badly done benchmarks. Unless their performance is actually bad, or course.
- abecedarius 8y ago
  That's certainly how things should be. But this clause was one of Oracle's early innovations (https://danluu.com/anon-benchmark/ https://danluu.com/anon-benchmark/) and they did pretty well. Do we need to understand how they got away with it to have a good chance of changing the norm?
- RaleyField 8y ago
  Question for any lawyers - can they gag any benchmarks that are published against the terms? I'd imagine the license terminates - what kind of damages can companies realistically extract when you break your license agreement?
kodablah 8y ago
I think you should also include restrictions in your EULA to prevent people from publishing statements about your code quality too. Very few are qualified to measure it, and spreading potentially false information can hurt a business. I can think of several other items that have limited expertise in evaluating and could cause negative PR harm if done falsely, so you probably should just enumerate those in the EULA too. Also, I gather from your comment in support of these restrictions, they are more than boilerplate and you will pursue offenders.
- slavak 8y ago
  I think car companies should prevent people from publishing safety or performance analysis of their vehicles. And restaurants should disallow public reviews of their business. (Very few people have the culinary knowledge to properly and objectively assess a restaurant's quality.)
mesozoic 8y ago
Nope. You don't need to protect us from people who do a bad job benchmarking we can do that ourselves. If we read their methodology and disagree we can make up our own minds. Anyway my experience is terrible with Atlassian's services so without benchmarks I can just continue to assume that is true forever and always recommend against them when we're making infrastructure decisions. I guess I can thank you for a big bundle of products less to consider.
RaleyField 8y ago
> It's not that there's anything to hide Clauses like these make me think you think there is. Which is perhaps a bigger red flag than the attempted censorship in itself. > Spreading wrong performance information can hurt a business. Firstly, I'd suggest you should perhaps focus on trying to educate your users on how to make better performance tests - if they are bad at making benchmarks then they are likely bad at running your server as well. Secondly, boohoo. Not spreading unflattering but correct performance information might not hurt your business but will hurt your customers. Lastly, you are curtailing speech which isn't ethical and due to that I'm pretty ambivalent if any hurt visited your business. Imagine if everyone did that about everything. > complex > requires expertise I'm sure every despot anywhere used a variation of your argument, something about economics being complex offshoot of mathematics that requires expertise to handle so you best not share any overly rushed opinions. > Apache-built binaries Which people can benchmark to their heart's content and I'd hope the clause would irritate many people into publishing their own benchmarks..
jjoonathan 8y ago
> Spreading wrong performance information can hurt a business. Spreading correct performance information can hurt Atlassian's business. That's why they don't want us to.
djur 8y ago
Have you ever invoke that clause to demand that someone take down benchmarks of Caddy? Do you believe that clause in the Caddy EULA is enforceable in the United States under the Consumer Review Fairness Act?
9935c101ab17a66 8y ago
I'm surprised and disappointed by this stance. Other people in this thread have mentioned that such a clause isn't even legally enforceable, so I'm curious if you have any perspective on that. Ultimately, though I appreciate you sharing your perspective, I personally will be steering clear of caddy and your other work.
csydas 8y ago
I get your point, but I also think you're wrong about it only being harmful. Even if the review is done incorrectly, it's a data-point on a misconception your users have, and it gives you a chance to respond accordingly. The company I work for has to deal with performance complaints all the time, some very from very public and loud entities in our Market. A company deciding to move away from our product signals very clearly across the market and it shows in our renewals numbers -- yet, we've never considered saying "don't benchmark us" in our contract. Every single time these complaints kick up, we treat it as a chance to prove that we know our stuff, we reach out and offer to assist with re-evaluating, and explain our position, and most of the time, it works. It also shows a big public commitment to helping instead of just hiding behind our Legal Team, and it shows we know what we're doing. We're not a huge company by any means in terms of actual persons able to act, but we still make it work without the need to put out such terms. If your customers are frequently doing something to make your product less than ideal, then you have an education issue that needs to be resolved.
vbezhenar 8y ago
For me if benchmarks are forbidden, it means that software is significantly worse than its competitors. So this clause by itself is worse than any benchmarks. There might be exceptions like Oracle, but not everyone is Oracle.

by mholt 8y ago

icelancer 8y ago

>> The HN title is misleading: the terms do not forbid benchmarking, just public dissemination of the benchmarks. If a tree falls in the forest, does it make a sound? The title is not misleading. Performing 69,000 benchmarks but being unable to publish them is de facto banning benchmarks, period. Hiding behind the idea that benchmarking is hard therefore it should be banned because it might be fake news is ludicrous. It is also not standard software EULA licensing, unless you think Oracle's practices are somehow industry standard and good for everyone.

mesozoic 8y ago
This way they can make you spend a ton of time bringing up a trial version on your internal systems to do the benchmarking. Effectively forcing lock in and sunk costs fallacy thinking then put you into tons of meetings that are really just attempts to upsell.

carapace 8y ago

> Benchmarking requires expertise that, it turns out, very few people have. I don't think I even have enough skills to do it correctly and meaningfully. Very important and often overlooked point. But I wonder, why not forbid public dissemination of inaccurate, non-reproducible benchmarks? > Spreading wrong performance information can hurt a business. Wouldn't that be libel? (IANAL)

SpicyLemonZest 8y ago
The problem is when the benchmark is accurate, reproducible, and based on a completely nonsensical scenario. It's not libel to say "Jira is 50% slower than Bugzilla" when your benchmark serially creates 25000 tickets, but it's not a fair claim either. That's not to say that anti-benchmarking licenses are the right solution, of course. I can just sympathize with why Atlassian wants one.
mesozoic 8y ago
Expertise so rare it seems Atlassian themselves don't have it.
SamReidHughes 8y ago
Suppose you state that a product performed poorly on measurement Y of benchmark X. The problem is not reproducibility or accuracy, it's that either Y or X are always really stupid in some way or another.

jkaplowitz 8y ago

Honestly, if I had widely used software with an enforceable EULA term which allowed me to benchmark but not publicly disclose bad results I found, it would make for even worse PR for the company: I'd be able to go to a tech industry reporter saying "I ran benchmarks on this software, and I think many people would be interested in the results, but the company forbade me to release them publicly. I will still share them non-publicly with any interested parties under NDA." Or if private dissemination were also forbidden, I'd change the wording accordingly. The better way for a company to handle this concern, if they feel it's important, is to proactively run and release benchmarks including commentary on the results, together with everything necessary for anyone to reproduce their results. Even better if they fund a trustworthy neutral third party to do this instead, with proper disclosure of the funding. They can then respond very effectively to bad PR about badly done benchmarks. Unless their performance is actually bad, or course.

abecedarius 8y ago
That's certainly how things should be. But this clause was one of Oracle's early innovations (https://danluu.com/anon-benchmark/ https://danluu.com/anon-benchmark/) and they did pretty well. Do we need to understand how they got away with it to have a good chance of changing the norm?
RaleyField 8y ago
Question for any lawyers - can they gag any benchmarks that are published against the terms? I'd imagine the license terminates - what kind of damages can companies realistically extract when you break your license agreement?

kodablah 8y ago

I think you should also include restrictions in your EULA to prevent people from publishing statements about your code quality too. Very few are qualified to measure it, and spreading potentially false information can hurt a business. I can think of several other items that have limited expertise in evaluating and could cause negative PR harm if done falsely, so you probably should just enumerate those in the EULA too. Also, I gather from your comment in support of these restrictions, they are more than boilerplate and you will pursue offenders.

slavak 8y ago
I think car companies should prevent people from publishing safety or performance analysis of their vehicles. And restaurants should disallow public reviews of their business. (Very few people have the culinary knowledge to properly and objectively assess a restaurant's quality.)

mesozoic 8y ago

Nope. You don't need to protect us from people who do a bad job benchmarking we can do that ourselves. If we read their methodology and disagree we can make up our own minds. Anyway my experience is terrible with Atlassian's services so without benchmarks I can just continue to assume that is true forever and always recommend against them when we're making infrastructure decisions. I guess I can thank you for a big bundle of products less to consider.

RaleyField 8y ago

> It's not that there's anything to hide Clauses like these make me think you think there is. Which is perhaps a bigger red flag than the attempted censorship in itself. > Spreading wrong performance information can hurt a business. Firstly, I'd suggest you should perhaps focus on trying to educate your users on how to make better performance tests - if they are bad at making benchmarks then they are likely bad at running your server as well. Secondly, boohoo. Not spreading unflattering but correct performance information might not hurt your business but will hurt your customers. Lastly, you are curtailing speech which isn't ethical and due to that I'm pretty ambivalent if any hurt visited your business. Imagine if everyone did that about everything. > complex > requires expertise I'm sure every despot anywhere used a variation of your argument, something about economics being complex offshoot of mathematics that requires expertise to handle so you best not share any overly rushed opinions. > Apache-built binaries Which people can benchmark to their heart's content and I'd hope the clause would irritate many people into publishing their own benchmarks..

jjoonathan 8y ago

> Spreading wrong performance information can hurt a business. Spreading correct performance information can hurt Atlassian's business. That's why they don't want us to.

djur 8y ago

Have you ever invoke that clause to demand that someone take down benchmarks of Caddy? Do you believe that clause in the Caddy EULA is enforceable in the United States under the Consumer Review Fairness Act?

9935c101ab17a66 8y ago

I'm surprised and disappointed by this stance. Other people in this thread have mentioned that such a clause isn't even legally enforceable, so I'm curious if you have any perspective on that. Ultimately, though I appreciate you sharing your perspective, I personally will be steering clear of caddy and your other work.

csydas 8y ago

I get your point, but I also think you're wrong about it only being harmful. Even if the review is done incorrectly, it's a data-point on a misconception your users have, and it gives you a chance to respond accordingly. The company I work for has to deal with performance complaints all the time, some very from very public and loud entities in our Market. A company deciding to move away from our product signals very clearly across the market and it shows in our renewals numbers -- yet, we've never considered saying "don't benchmark us" in our contract. Every single time these complaints kick up, we treat it as a chance to prove that we know our stuff, we reach out and offer to assist with re-evaluating, and explain our position, and most of the time, it works. It also shows a big public commitment to helping instead of just hiding behind our Legal Team, and it shows we know what we're doing. We're not a huge company by any means in terms of actual persons able to act, but we still make it work without the need to put out such terms. If your customers are frequently doing something to make your product less than ideal, then you have an education issue that needs to be resolved.

vbezhenar 8y ago

For me if benchmarks are forbidden, it means that software is significantly worse than its competitors. So this clause by itself is worse than any benchmarks. There might be exceptions like Oracle, but not everyone is Oracle.