5 ms·
Crazy that the project took almost 4 years end-to-end, and it's still ongoing. I had no idea anything at AWS had that long of an attention span. It's funny an
by mannyv 2y ago
Crazy that the project took almost 4 years end-to-end, and it's still ongoing.
I had no idea anything at AWS had that long of an attention span.
It's funny and telling that in the end, it's all backed by CSVs in s3. Long live CSV!
- jerrygenser 2y agoThey reference parquet files, not sure if it's only CSV or CSV even figures in that heavily other than the first iteration before migrating to spark
- thedood 2y agoHi mannyv - one of the devs that worked on the migration here. It has been a pretty long project - approached with caution due to the criticality of keeping our BI datasets healthy - but the preliminary results produced year-over-year kept looking promising enough to keep after it. =) Also, we mostly have Parquet data cataloged in S3 today, but delimited text is indeed ubiquitous and surprisingly sticky, so we continue to maintain some very large datasets natively in this format. However, while the table's data producer may prefer to write delimited text, they are almost always converted to Parquet during the compaction process to produce a read-optimized table variant downstream.
- gregw2 2y agoAre you all shifting over to storing as iceberg-enriched parquet yet and letting it (within, say Athena) manage compaction or thinking about it, or is it not worth it since this new Ray+Parquet thing is working for you?
- thedood 2y agoAs alluded to in the blog post, Ray+Parquet+Iceberg is the next frontier we'd like to make our compactor and similar procedures available on in open source so that the community can start bringing similar benefits for their Iceberg workloads. Stay tuned. =)
- sdfsdfadf 2y ago[flagged]
- wenc 2y agoMost people are moving away from CSV for big datasets, except in exceptional cases involving linear reads (append only ETL). CSV has one big upside which is human readability. But so many downsides: poor random access, no typing, no compression, complex parser needing to handle exceptions.
- 100pctremote 2y agoMost people don't directly query or otherwise operate on raw CSV, though. Large source datasets in CSV format still reign in many enterprises, but these are typically read into a dataframe, manipulated and stored as Parquet and the like, then operated upon by DuckDB, Polars, etc., or modeled (E.g. DBT) and pushed to an OLAP target.
- wenc 2y agoThere are folks who still directly query CSV formats in a data lake using a query engine like Athena or Spark or Redshift Spectrum — which ends up being much slower and consuming more resources than is necessary due to full table scans. CSV is only good for append only. But so is Parquet and if you can write Parquet from the get go, you save on storage as well has have a directly queryable column store from the start. CSV still exists because of legacy data generating processes and dearth of Parquet familiarity among many software engineers. CSV is simple to generate and easy to troubleshoot without specialized tools (compared to Parquet which requires tools like Visidata). But you pay for it elsewhere.
- whoevercares 2y agoIs it really AWS? I don’t recall any service called BDT
- p0rkbelly 2y agoThe second paragraph discusses that BDT is an internal team at Amazon Retail. They used AWS and Ray to do this.
- qwerp 2y agoour plans are measured in centuries