6 ms·

*50-100x faster for analytical queries. Columnar storage. Vertica was the real innovator. Redshift, BigQuery, Snowflake, Citus, Timescale, etc just took the id

*50-100x faster for analytical queries. Columnar storage. Vertica was the real innovator. Redshift, BigQuery, Snowflake, Citus, Timescale, etc just took the idea and ran with it.

metadat 4y ago
Ah, so it's actually more marketing, since it's comparing completely different approaches. I wonder if there are performance tradeoffs sometimes leading to slower than vanilla postgres?
- zaphar 4y ago
  The sweet spot for columnar storage is calculating aggregations across a subset of columns. It can be really fast at that. But if you what you need is select * it might actually be slower in that case.
- tehlike 4y ago
  If you are reading all columns from same row in your queries that'd break columnar storage use case, and would be slower.
- sa46 4y ago
  They claim 4x faster against vanilla postgres for "commercial workloads" so it sounds really impressive. > I wonder if there are performance tradeoffs sometimes leading to slower than vanilla postgres? Speculation: alloydb probably has both a row store and a column store and some fancy logic to decide which one to use. That's usually what HTAP means. In very broad strokes, columnar data stores struggle if you select many columns.
  - cavisne 4y ago
    Not really speculation, its mentioned in the article. Theres not really any free lunch here, vanilla postgres is optimized to perform well within the constraints of a single scale-up host, minimizing storage usage as much as possible. These Aurora style systems instead take the assumption that storage is relatively cheap in a cloud environment, and likewise with any compute task that can be scaled out instead of up. So move as much as possible out of the scale-up instance to scale-out storage and compute. Additionally Google is claiming their system is better than Aurora because google has a distributed filesystem, where AWS only has block storage (tied to specific compute) and object storage (much worse performance).
    - rektide 4y ago
      > and likewise with any compute task that can be scaled out instead of up. So move as much as possible out of the scale-up instance to scale-out storage and compute. This was very much the main lesson I got reading up on DataDog's third-gen event storage system Husky[1]. Great read. [1] https://www.datadoghq.com/blog/engineering/introducing-husky/ https://www.datadoghq.com/blog/engineering/introducing-husky... https://news.ycombinator.com/item?id=31416843 https://news.ycombinator.com/item?id=31416843 (185 points, 9d ago, 15 comments)
- trhway 4y ago
  Doing OLTP writes into a columnar storage is pretty complex. Also columnar storage needs to be in memory or on SSD at least, not HDD where read performance would be abysmal without serious gymnastics (thus in particular, given the RAM and SSD prices back then, you just couldn't really have a columnar db widespread adoption 15+ years ago).
avinassh 4y ago
What were the innovations by Vertica? Is there any link you have
- AlisdairO 4y ago
  This is the c-store paper, which was evolved into Vertica: http://www.cs.umd.edu/~abadi/vldb.pdf http://www.cs.umd.edu/~abadi/vldb.pdf . It's very readable, worth a look. edit: column stores existed before c-store, but c-store did some very nifty stuff around integrating compression awareness into the query executor
  - avinassh 4y ago
    Thank you, I will read the paper!
jerryjerryjerry 4y ago
If I remember correctly, IBM published the idea even before Vertica.
dehrmann 4y ago
How did the timeline work with Vertica and Paraccel? I know Paraccel was around the same time, but I'm not sure who was really first, or if it's even an important distinction.
- mdasen 4y ago
  Vertica came out of the C-Store paper (https://web.archive.org/web/20100619191833/http://db.lcs.mit.edu/projects/cstore/vldb.pdf https://web.archive.org/web/20100619191833/http://db.lcs.mit...). The C-Store paper pre-dates ParAccel being incorporated. Many of the C-Store authors were involved in Vertica's creation. If you want to be technical, C-Store was the real innovator...and those people refined and commercialized it as Vertica.

*50-100x faster for analytical queries. Columnar storage. Vertica was the real innovator. Redshift, BigQuery, Snowflake, Citus, Timescale, etc just took the idea and ran with it.

metadat 4y ago

Ah, so it's actually more marketing, since it's comparing completely different approaches. I wonder if there are performance tradeoffs sometimes leading to slower than vanilla postgres?

zaphar 4y ago
The sweet spot for columnar storage is calculating aggregations across a subset of columns. It can be really fast at that. But if you what you need is select * it might actually be slower in that case.
tehlike 4y ago
If you are reading all columns from same row in your queries that'd break columnar storage use case, and would be slower.
sa46 4y ago
They claim 4x faster against vanilla postgres for "commercial workloads" so it sounds really impressive. > I wonder if there are performance tradeoffs sometimes leading to slower than vanilla postgres? Speculation: alloydb probably has both a row store and a column store and some fancy logic to decide which one to use. That's usually what HTAP means. In very broad strokes, columnar data stores struggle if you select many columns.
- cavisne 4y ago
  Not really speculation, its mentioned in the article. Theres not really any free lunch here, vanilla postgres is optimized to perform well within the constraints of a single scale-up host, minimizing storage usage as much as possible. These Aurora style systems instead take the assumption that storage is relatively cheap in a cloud environment, and likewise with any compute task that can be scaled out instead of up. So move as much as possible out of the scale-up instance to scale-out storage and compute. Additionally Google is claiming their system is better than Aurora because google has a distributed filesystem, where AWS only has block storage (tied to specific compute) and object storage (much worse performance).
  - rektide 4y ago
    > and likewise with any compute task that can be scaled out instead of up. So move as much as possible out of the scale-up instance to scale-out storage and compute. This was very much the main lesson I got reading up on DataDog's third-gen event storage system Husky[1]. Great read. [1] https://www.datadoghq.com/blog/engineering/introducing-husky/ https://www.datadoghq.com/blog/engineering/introducing-husky... https://news.ycombinator.com/item?id=31416843 https://news.ycombinator.com/item?id=31416843 (185 points, 9d ago, 15 comments)
trhway 4y ago
Doing OLTP writes into a columnar storage is pretty complex. Also columnar storage needs to be in memory or on SSD at least, not HDD where read performance would be abysmal without serious gymnastics (thus in particular, given the RAM and SSD prices back then, you just couldn't really have a columnar db widespread adoption 15+ years ago).

avinassh 4y ago

What were the innovations by Vertica? Is there any link you have

AlisdairO 4y ago
This is the c-store paper, which was evolved into Vertica: http://www.cs.umd.edu/~abadi/vldb.pdf http://www.cs.umd.edu/~abadi/vldb.pdf . It's very readable, worth a look. edit: column stores existed before c-store, but c-store did some very nifty stuff around integrating compression awareness into the query executor
- avinassh 4y ago
  Thank you, I will read the paper!

jerryjerryjerry 4y ago

If I remember correctly, IBM published the idea even before Vertica.

dehrmann 4y ago

How did the timeline work with Vertica and Paraccel? I know Paraccel was around the same time, but I'm not sure who was really first, or if it's even an important distinction.

mdasen 4y ago
Vertica came out of the C-Store paper (https://web.archive.org/web/20100619191833/http://db.lcs.mit.edu/projects/cstore/vldb.pdf https://web.archive.org/web/20100619191833/http://db.lcs.mit...). The C-Store paper pre-dates ParAccel being incorporated. Many of the C-Store authors were involved in Vertica's creation. If you want to be technical, C-Store was the real innovator...and those people refined and commercialized it as Vertica.