2 ms·
I have. I think it's a pretty easy situation for certain kinds of startups to find themselves in: - Someone decides on CSV because it's easy to produce and yo
by moregrist 1y ago
I have. I think it's a pretty easy situation for certain kinds of startups to find themselves in:
- Someone decides on CSV because it's easy to produce and you don't have that much data. Plus it's easier for the <non-software people> to read so they quit asking you to give them Excel sheets. Here <non-software people> is anyone who has a legit need to see your data and knows Excel really well. It can range from business types to lab scientists.
- Your internal processes start to consume CSV because it's what you produce. You build out key pipelines where one or more steps consume CSV.
- Suddenly your data increases by 10x or 100x or more because something started working: you got some customers, your sensor throughput improved, the science part started working, etc.
Then it starts to make sense to optimize ingesting millions or billions of lines of CSV. It buys you time so you can start moving your internal processes (and maybe some other teams' stuff) to a format more suited for this kind of data.
- nly 1y agoWe use parquet extensively at work, and it's really slow to ingest. Slower than a hand rolled binary column oriented format. Sometimes using something standardized is just worth it though.