3 ms·

> After all, they are not apples-oranges comparison And more: Them are the MOST useful comparisons. Compare 2 oranges is not as useful than compare an apple an

by mamcx 6y ago

> After all, they are not apples-oranges comparison And more: Them are the MOST useful comparisons. Compare 2 oranges is not as useful than compare an apple and orange. You have MORE useful differences and more insights on what make them similar. In the case of speed: I compare a bike and a car. Is unfair? yes. But, if the GOAL is how fast is go to the next city, is totally worth to know that the car is better. However, if the goal is what is the most ENVIRONMENT FRIENDLY why to travel, the bike win. Comparing 2 very different vehicles is far more useful. --- What is important in benchmarks? To make clear WHAT is the goal to be measured. And not shy away to hammer if some choice become clearly superior.

rumanator 6y ago
> compare a bike and a car. Is unfair? yes. But, if the GOAL is how fast is go to the next city, is totally worth to know that the car is better. The problem with these benchmarks is they have a propensity to not make a fair comparison. For example, sometimes they try to compare underperforming naive implementations in one language with expertly tuned implementations in another. Sometimes a blatantly underperforming implementation is used. Sometimes results are turned upside down by flipping compiler flags. And sometimes photofinish results that fall within the standard deviation are sold as this absolute proof that language A outperforms language B. Let's look at these benchmarks to see how these things are spun to sound like objective assertions are derived from meaningless data. Does it really matter if someone could use language A to shave off about 1MB from build artifacts? Are we really expected to take a 1.5s/5% difference in a cherry picked synthetic benchmark as meaningful? Is anyone in their right mind expected to drive their decisions on their tech stack on two cherry-picked tricks posted on a blog? This is why no one takes these stunts seriously.

by mamcx 6y ago

rumanator 6y ago

> compare a bike and a car. Is unfair? yes. But, if the GOAL is how fast is go to the next city, is totally worth to know that the car is better. The problem with these benchmarks is they have a propensity to not make a fair comparison. For example, sometimes they try to compare underperforming naive implementations in one language with expertly tuned implementations in another. Sometimes a blatantly underperforming implementation is used. Sometimes results are turned upside down by flipping compiler flags. And sometimes photofinish results that fall within the standard deviation are sold as this absolute proof that language A outperforms language B. Let's look at these benchmarks to see how these things are spun to sound like objective assertions are derived from meaningless data. Does it really matter if someone could use language A to shave off about 1MB from build artifacts? Are we really expected to take a 1.5s/5% difference in a cherry picked synthetic benchmark as meaningful? Is anyone in their right mind expected to drive their decisions on their tech stack on two cherry-picked tricks posted on a blog? This is why no one takes these stunts seriously.