3 ms·
Hmmm if they're just internal tools, why not just an array of structs? No parsing needed. Can have optionals. Can't go faster than nothing.
by Neywiny 1y ago
Hmmm if they're just internal tools, why not just an array of structs? No parsing needed. Can have optionals. Can't go faster than nothing.
- trollbridge 1y agoThanks to everyone above for some great responses. Cap'n Proto seems to do exactly what you're describing (the in-memory representation is identical to what's on the wire, and then getter/setter methods are generated which look at that).
- elteto 1y agoMostly because dependencies are hard and extra so when you need another team using a different language to also neeeds support the same format. I’d love to pass parquet data around, or SQLite dbs, or something else, but that requires dedicated support from other teams upstream/downstream. Everyone and everything supports CSV, and when they don’t they can hack a simple parser quickly. I know that getting a CSV parser right for all the edge cases is very hard, but they don’t need to. They just need to support the features we use. That’s simple and quick and everyone quickly moves on to the actual work of processing the data.
- Neywiny 1y agoYeah there's no format to support that way. Maybe I'm more biased towards numeric data (sensor readings, etc), but I never have to worry about libraries and dependencies to say data = (uint32_t *)read(f); Or data = struct.unpack... Sounds like you're dealing with more heavily formatted or variably formatted data that benefits from more structure to it
- elteto 1y agoLucky you! We produce/consume several files from other teams, and those teams use Python, Java, Go, and C++ internally. I can try to bend those teams to my will by pushing my own custom serialization library (and would that even be a fair thing to do?) or I can just pass them a CSV.