5 ms·

They don't even bother trying to make this sound believable. You claim it's 1.5x-2x faster? Sure I might believe that despite your prohibition against benchmar

They don't even bother trying to make this sound believable. You claim it's 1.5x-2x faster? Sure I might believe that despite your prohibition against benchmarks. 10x faster? You better show how your engineers pulled out that miracle. 1000x? You've either picked out a very weird edge case, used something in a way it's not meant to be used, or just made that up. Anyone who's done any serious benchmarking in a competitive space will instantly recognize that as a 'that cant be right' result.

wredue 3y ago
Python vs C/C++/Rust/Zig has even more startling performance differences. If you’re looking for benchmarks of the same classes of software, among techempower benchmarks, the fastest on the charts are often well over 1000x faster. Being 1000x slower than shit needs to be is, very literally, the default state of the vast majority of software today. I don’t even blink at 1000x claims cause that’s just the standard gains you get over popular software.
- RandomBK 3y ago
  Perhaps, but Python vs C is hardly an apples-for-apples comparison. I could get 1000x differences benchmarking Aurora vs Redis for example. That's what I meant by "used in a way not meant to be used". Broadly speaking, there's plenty of public benchmarks indicating that a properly-configured Aurora is within the same order magnitude as a properly-configured RDS Postgres which is in the same ballpark as MySQL. That's an indication to me that Aurora isn't fundamentally broken in some way, making a 1000x claim quite extraordinary.
  - re-thc 3y ago
    > That's an indication to me that Aurora isn't fundamentally broken in some way, making a 1000x claim quite extraordinary. This isn't the open source MySQL being compared. MySQL Heatwave is an in-memory analytics cluster that works as a MySQL plugin. It's cloud only. In a way it's similar to AlloyDB from GCP, which has similar style claims (maybe not 1000x)
  - wredue 3y ago
    And that’s why I also discussed the several web api benchmarks where you’ll see 10,000x differences from bottom ranking to top. But you didn’t like that, so felt compelled to ignore it. 1,000x is literally nothing when discussing modern development practices. I’m not saying that oracle is definitely 1000x faster, just that writing off claims of 1000x cause you dislike the number is not proper in software when you can immediately find 10,000x benchmarks of two directly competing software packages.
    - RandomBK 3y ago
      Show me a serious (i.e. used in production) web framework that is 10,000x faster or slower than React or Vue or Spring when measured on a real-life workload (i.e. not testing a single function in a hot loop). All I'm saying is that if you're approaching a benchmark between serious competitors providing similar functionality used in the intended way, you almost never see 1000x+ performance differences.
      - wredue 3y ago
        [flagged]
  - wredue 3y ago
    The claim that “1000x is extraordinary” is demonstrably wrong. You can browse virtually any modern website today and see time to full load of 15+ seconds, followed by time to link load of 7+ seconds. Back in the days of single core pentiums, 1 second loads were considered severely broken. Modern software on modern hardware takes tens of seconds to start, whereas software with more features in the pentium days loaded instantly. 1000x is not only not “extraordinary”, it’s actually the demonstrable norm today. You think it’s extraordinary as a result of simply not knowing what computers are capable of.
    - RandomBK 3y ago
      15s vs 1s is only a factor of 15x, not 1000x. The load time of the oft-considered-bloated reddit.com for me is ~5s. Let's use that as a benchmark. If you told me you got reddit.com to load 1000x faster (5ms), then yes that is extraordinary. You're also missing the point of my 'apples-to-apples' caveat. A modern webpage might load slowly, but it also does a lot more than a static html one-pager. Sure you might not consider that to be useful, but those features (tracking, A/B testing, ads, redundancy, CDN, etc.) are useful to someone.
- xboxnolifes 3y ago
  Maybe if you compare the fastest to the slowest. But if actually compare the fastest options in each language, you rarely see things past a 10-20x slowdown.
- deaddodo 3y ago
  > Python vs C/C++/Rust/Zig has even more startling performance differences. There's no way you're seeing a >1000x speed difference in Python/Ruby/JS and C/C++/Rust/Zig/D without building some heavily contrived comparisons. That is not a normal differential. The "high-level" languages are slower, certainly, but usually only on a single order of magnitude.
re-thc 3y ago
It's MySQL Heatwave. Pretty sure this is referring to analytics usage. Clickhouse has a similar claim. If you look at https://benchmark.clickhouse.com/ https://benchmark.clickhouse.com/ and pick Clickhouse and Aurora for MySQL you get something in that ballpark. Heatwave claims to be faster than both Snowflake and Clickhouse.
- RandomBK 3y ago
  In that case the fairer comparison should have been to Redshift, though I will concede that Amazon walked into this one themselves by overhyping Aurora's OLAP capabilities. OLTP != OLAP, and the sooner database vendors learn to stay in their lane and focus on one use case, the better.
  - re-thc 3y ago
    Oracle claims it works for both OLTP and OLAP. I mean on the fundamentals Heatwave adds a giant memory cache / query engine in front. That alone makes it faster than anything disk based. As to Redshift, yes, they claim it's faster than Redshift too by quite a bit. (I use claim as it's against the ToS to benchmark it independently)
    - zX41ZdbW 3y ago
      There should not be such a distinction between in-memory and disk based databases. Being in-memory does not make OLAP databases faster. And typically, "in-memory" is a disadvantage and a limiting factor in OLAP database architecture. I have a presentation on this topic: https://presentations.clickhouse.com/meetup53/optimizations/ https://presentations.clickhouse.com/meetup53/optimizations/ It is similar to how frequently someone says "lock-free" to highlight imaginary performance advantages. If someone says a database engine is fast because it is in-memory and lock-free, it smells wrong.
tfehring 3y ago
It seems like he was comparing an in-memory columnar datastore to a disk-based row-oriented database for OLAP workloads. So it certainly falls in the "used something in a way it's not meant to be used" bucket, and maybe also in the "very weird edge case" bucket if the implication is that a big company is going to keep a material chunk of its analytics data in RAM.
ttfkam 3y ago
And of course Oracle famously forbids publishing benchmarks on their databases, so independent verification is indeed impossible.
- xkcd-sucks 3y ago
  Just wondering, is there a site hosted in a US unfriendly jurisdiction / on Tor which does publish Oracle benchmarks?

wredue 3y ago

Python vs C/C++/Rust/Zig has even more startling performance differences. If you’re looking for benchmarks of the same classes of software, among techempower benchmarks, the fastest on the charts are often well over 1000x faster. Being 1000x slower than shit needs to be is, very literally, the default state of the vast majority of software today. I don’t even blink at 1000x claims cause that’s just the standard gains you get over popular software.

RandomBK 3y ago
Perhaps, but Python vs C is hardly an apples-for-apples comparison. I could get 1000x differences benchmarking Aurora vs Redis for example. That's what I meant by "used in a way not meant to be used". Broadly speaking, there's plenty of public benchmarks indicating that a properly-configured Aurora is within the same order magnitude as a properly-configured RDS Postgres which is in the same ballpark as MySQL. That's an indication to me that Aurora isn't fundamentally broken in some way, making a 1000x claim quite extraordinary.
- re-thc 3y ago
  > That's an indication to me that Aurora isn't fundamentally broken in some way, making a 1000x claim quite extraordinary. This isn't the open source MySQL being compared. MySQL Heatwave is an in-memory analytics cluster that works as a MySQL plugin. It's cloud only. In a way it's similar to AlloyDB from GCP, which has similar style claims (maybe not 1000x)
- wredue 3y ago
  And that’s why I also discussed the several web api benchmarks where you’ll see 10,000x differences from bottom ranking to top. But you didn’t like that, so felt compelled to ignore it. 1,000x is literally nothing when discussing modern development practices. I’m not saying that oracle is definitely 1000x faster, just that writing off claims of 1000x cause you dislike the number is not proper in software when you can immediately find 10,000x benchmarks of two directly competing software packages.
  - RandomBK 3y ago
    Show me a serious (i.e. used in production) web framework that is 10,000x faster or slower than React or Vue or Spring when measured on a real-life workload (i.e. not testing a single function in a hot loop). All I'm saying is that if you're approaching a benchmark between serious competitors providing similar functionality used in the intended way, you almost never see 1000x+ performance differences.
    - wredue 3y ago
      [flagged]
- wredue 3y ago
  The claim that “1000x is extraordinary” is demonstrably wrong. You can browse virtually any modern website today and see time to full load of 15+ seconds, followed by time to link load of 7+ seconds. Back in the days of single core pentiums, 1 second loads were considered severely broken. Modern software on modern hardware takes tens of seconds to start, whereas software with more features in the pentium days loaded instantly. 1000x is not only not “extraordinary”, it’s actually the demonstrable norm today. You think it’s extraordinary as a result of simply not knowing what computers are capable of.
  - RandomBK 3y ago
    15s vs 1s is only a factor of 15x, not 1000x. The load time of the oft-considered-bloated reddit.com for me is ~5s. Let's use that as a benchmark. If you told me you got reddit.com to load 1000x faster (5ms), then yes that is extraordinary. You're also missing the point of my 'apples-to-apples' caveat. A modern webpage might load slowly, but it also does a lot more than a static html one-pager. Sure you might not consider that to be useful, but those features (tracking, A/B testing, ads, redundancy, CDN, etc.) are useful to someone.
xboxnolifes 3y ago
Maybe if you compare the fastest to the slowest. But if actually compare the fastest options in each language, you rarely see things past a 10-20x slowdown.
deaddodo 3y ago
> Python vs C/C++/Rust/Zig has even more startling performance differences. There's no way you're seeing a >1000x speed difference in Python/Ruby/JS and C/C++/Rust/Zig/D without building some heavily contrived comparisons. That is not a normal differential. The "high-level" languages are slower, certainly, but usually only on a single order of magnitude.

re-thc 3y ago

It's MySQL Heatwave. Pretty sure this is referring to analytics usage. Clickhouse has a similar claim. If you look at https://benchmark.clickhouse.com/ https://benchmark.clickhouse.com/ and pick Clickhouse and Aurora for MySQL you get something in that ballpark. Heatwave claims to be faster than both Snowflake and Clickhouse.

RandomBK 3y ago
In that case the fairer comparison should have been to Redshift, though I will concede that Amazon walked into this one themselves by overhyping Aurora's OLAP capabilities. OLTP != OLAP, and the sooner database vendors learn to stay in their lane and focus on one use case, the better.
- re-thc 3y ago
  Oracle claims it works for both OLTP and OLAP. I mean on the fundamentals Heatwave adds a giant memory cache / query engine in front. That alone makes it faster than anything disk based. As to Redshift, yes, they claim it's faster than Redshift too by quite a bit. (I use claim as it's against the ToS to benchmark it independently)
  - zX41ZdbW 3y ago
    There should not be such a distinction between in-memory and disk based databases. Being in-memory does not make OLAP databases faster. And typically, "in-memory" is a disadvantage and a limiting factor in OLAP database architecture. I have a presentation on this topic: https://presentations.clickhouse.com/meetup53/optimizations/ https://presentations.clickhouse.com/meetup53/optimizations/ It is similar to how frequently someone says "lock-free" to highlight imaginary performance advantages. If someone says a database engine is fast because it is in-memory and lock-free, it smells wrong.

tfehring 3y ago

It seems like he was comparing an in-memory columnar datastore to a disk-based row-oriented database for OLAP workloads. So it certainly falls in the "used something in a way it's not meant to be used" bucket, and maybe also in the "very weird edge case" bucket if the implication is that a big company is going to keep a material chunk of its analytics data in RAM.

ttfkam 3y ago

And of course Oracle famously forbids publishing benchmarks on their databases, so independent verification is indeed impossible.

xkcd-sucks 3y ago
Just wondering, is there a site hosted in a US unfriendly jurisdiction / on Tor which does publish Oracle benchmarks?