3 ms·
I think this is somewhat incomplete information, and I say this as a SQLite lover. It depends on your type of data, but both HDF5 and SQLite are more than adequ
by batbomb 11y ago
I think this is somewhat incomplete information, and I say this as a SQLite lover. It depends on your type of data, but both HDF5 and SQLite are more than adequate formats for data, and you should be using one of them.
I think HDF5 is actually a better format for long term data management, but I think SQLite is a easier to use in everyday life. There's also Parquet and Avro for these as well.
High-Energy physics also has the "code is the documentation" clusterfuck known as ROOT files.
- rch 11y agoDo you have experience with parallel HDF5? It looks like it has been improving a lot as of late, but I can't really test it myself at present.
- chacham15 11y agoThis again misses the point. The point is that you should use the right tool for the job. If your application needs a particular format for whatever reason, use that format. For example, could you imagine trying to encode a video file in a sqlite database? Similarly, would you use a sqlite file for a plaintext document (like c++) or a binary executable? To give a more custom example: I was working on a project which tries to build a database of solutions to games (like Connect 4). Connect 4 has such a large search state that using a database is terrible because it doesnt meet the requirements of really fast lookups (with great caching), small size, and fast parallel writing that the solver needs.