4 ms·
Mostly because dependencies are hard and extra so when you need another team using a different language to also neeeds support the same format. I’d love to pas
by elteto 1y ago
Mostly because dependencies are hard and extra so when you need another team using a different language to also neeeds support the same format.
I’d love to pass parquet data around, or SQLite dbs, or something else, but that requires dedicated support from other teams upstream/downstream.
Everyone and everything supports CSV, and when they don’t they can hack a simple parser quickly. I know that getting a CSV parser right for all the edge cases is very hard, but they don’t need to. They just need to support the features we use. That’s simple and quick and everyone quickly moves on to the actual work of processing the data.
- Neywiny 1y agoYeah there's no format to support that way. Maybe I'm more biased towards numeric data (sensor readings, etc), but I never have to worry about libraries and dependencies to say data = (uint32_t *)read(f); Or data = struct.unpack... Sounds like you're dealing with more heavily formatted or variably formatted data that benefits from more structure to it
- elteto 1y agoLucky you! We produce/consume several files from other teams, and those teams use Python, Java, Go, and C++ internally. I can try to bend those teams to my will by pushing my own custom serialization library (and would that even be a fair thing to do?) or I can just pass them a CSV.