3 ms·

I think you took away the wrong lesson, personally. The point is that no single metric or even group of metrics is going to be able to cover the entire reality

I think you took away the wrong lesson, personally. The point is that no single metric or even group of metrics is going to be able to cover the entire reality of what's important about a database, even when what's important is performance. Just focusing on metrics is not enough. The post gives other examples about times when e.g. clean, maintainable code matters more (DuckDB) or how there are fundamental architectural limitations that make "worse performing" databases better for what you need (e.g. Clickhouse not being good at joins, some DB's friendlier SQL and CSV parsing all come up) but the message that unites all of its examples is "benchmark performance, even on very well-designed benchmarks, are often misleading to the actual reality of using a database".

taeric 3y ago
I'm with the other poster on this. I think there are good analogies to stuff like goodhart's law. At times you do need to reground why you are measuring something. That said, most of these lessons were about making the measurement to improve be more holistic. This is a part of an argument people will often use to talk about why end to end testing is superior to unit testing. It isn't that you don't want the units tested. Nor is it that end to end is easier. Rather, it is harder to focus on an end to end test for the sake of the end to end test. (No panacea, mind.)
thatfrenchguy 3y ago
> Just focusing on metrics is not enough This is not about focusing on the metrics itself, it's about focusing on what matters to your customers: the metric is a tool, not a end goal. It's obvious in the airplane example OP has, what matters is end to end time (and why HSR are more successful than you would think they would be at the travel game btw).
saghm 3y ago
It's interesting to me that you think that my comment is at odds with what you're saying, because even re-reading it I'm not sure where you got anything about "metrics" from what I said. Yes, I said that the issue did happen to be performance, but only because that's what happened to be where the customer's pain was. My point was that the idea of a feature being "enough" or "not enough" doesn't really make sense outside of the context of how people are actually experiencing the product. What's weird to me about the claim "performance is not enough" (and your longer-winded quote at the end) is the idea that the idea of trying to look at what customers are actually doing and finding what their pain points are is somehow best stated by decrying performance as a metric rather than decrying _any_ single factor as "enough". If I go to a restaurant, order spaghetti, and then get sauce spilled on my lap by the server, the takeaway isn't "having high quality sauce is not enough", it's "spilling things on customers makes for a bad restaurant experience".

taeric 3y ago

I'm with the other poster on this. I think there are good analogies to stuff like goodhart's law. At times you do need to reground why you are measuring something. That said, most of these lessons were about making the measurement to improve be more holistic. This is a part of an argument people will often use to talk about why end to end testing is superior to unit testing. It isn't that you don't want the units tested. Nor is it that end to end is easier. Rather, it is harder to focus on an end to end test for the sake of the end to end test. (No panacea, mind.)

thatfrenchguy 3y ago

> Just focusing on metrics is not enough This is not about focusing on the metrics itself, it's about focusing on what matters to your customers: the metric is a tool, not a end goal. It's obvious in the airplane example OP has, what matters is end to end time (and why HSR are more successful than you would think they would be at the travel game btw).

saghm 3y ago

It's interesting to me that you think that my comment is at odds with what you're saying, because even re-reading it I'm not sure where you got anything about "metrics" from what I said. Yes, I said that the issue did happen to be performance, but only because that's what happened to be where the customer's pain was. My point was that the idea of a feature being "enough" or "not enough" doesn't really make sense outside of the context of how people are actually experiencing the product. What's weird to me about the claim "performance is not enough" (and your longer-winded quote at the end) is the idea that the idea of trying to look at what customers are actually doing and finding what their pain points are is somehow best stated by decrying performance as a metric rather than decrying _any_ single factor as "enough". If I go to a restaurant, order spaghetti, and then get sauce spilled on my lap by the server, the takeaway isn't "having high quality sauce is not enough", it's "spilling things on customers makes for a bad restaurant experience".