3 ms·

What about iceberg tables and a lake approach on GCS and then picking a querying engine?

• whalesalad 3y ago
  These are terms I’m sorta familiar with but not sure. Data lake = bunch of noise (everything), iceberg = generated tables or views to read relevant/hot data from the lake?
  • gigatexal 3y ago
    Basically that’s it. Yeah. If you can afford BigQuery just use that but otherwise building off of blob storage and bolting on query engines and catalogs makes for a flexible approach but I find BigQuery solves most problems rather well just throwing money at the problem lol
• benjaminwootton 3y ago
  This is the way the industry is going. Table formats such as Delta, Hudi, Iceberg stored on cloud object stores. Though it works amazingly well, it is certainly slower than ingesting the data to be stored and manipulated in native formats.