5 ms·
SQLite should be the default way someone manages information on a binary data file, as opposed to just inventing your format from scratch. I remember on uni wh
by harperlee 11y ago
SQLite should be the default way someone manages information on a binary data file, as opposed to just inventing your format from scratch.
I remember on uni when we started defining our own formats, to later on learn about what goes inside a DBMS, and thinking, "Why on earth would I ever do the first thing again?" (not that there's no space for it, but it's not so big).
Also, when I've found a couple of times that a file that I thought had a proprietary binary format was a SQLite file, it was bliss. So much easier to navigate, inspect and work with that a huge XML file or a plain text one.
- batou 11y agoPretty much everything I've built that is standalone uses exactly this model. It's a wonderful bit of tech!
- batbomb 11y agoI think this is somewhat incomplete information, and I say this as a SQLite lover. It depends on your type of data, but both HDF5 and SQLite are more than adequate formats for data, and you should be using one of them. I think HDF5 is actually a better format for long term data management, but I think SQLite is a easier to use in everyday life. There's also Parquet and Avro for these as well. High-Energy physics also has the "code is the documentation" clusterfuck known as ROOT files.
- rch 11y agoDo you have experience with parallel HDF5? It looks like it has been improving a lot as of late, but I can't really test it myself at present.
- chacham15 11y agoThis again misses the point. The point is that you should use the right tool for the job. If your application needs a particular format for whatever reason, use that format. For example, could you imagine trying to encode a video file in a sqlite database? Similarly, would you use a sqlite file for a plaintext document (like c++) or a binary executable? To give a more custom example: I was working on a project which tries to build a database of solutions to games (like Connect 4). Connect 4 has such a large search state that using a database is terrible because it doesnt meet the requirements of really fast lookups (with great caching), small size, and fast parallel writing that the solver needs.
- Ygor 11y agoGreat part from the docs: "SQLite does not compete with client/server databases. SQLite competes with fopen()."
- protomyth 11y agoI agree with that for a lot of things, but I would trade SQLite in a heartbeat for a Newton Soup[1] implementation. 1) https://en.wikipedia.org/wiki/Soup_(Apple) https://en.wikipedia.org/wiki/Soup_(Apple)
- scrollaway 11y agoCould you elaborate why? I imagine not many people are familiar with Soup.
- protomyth 11y agoChapter 11 of the Newton Programmers Guide[1] gives a fairly detailed description with pictures. Relational Databases are nice, but I would rather deal in objects (well frames). 1) https://manuals.info.apple.com/MANUALS/1000/MA1510/en_US/NewtonProgrammerGuide20.PDF https://manuals.info.apple.com/MANUALS/1000/MA1510/en_US/New...
- jbaiter 11y agoThis sounds really intriguing, do you know of any open source implementations for other operating systems/programming languages?
- protomyth 11y agoI haven't found just a soup implementation (would be interesting for javascript given the prototype objects). NEWT/0[1] is a full NewtoScript but it looks unused. 1) http://gnue.github.io/NEWT0/ http://gnue.github.io/NEWT0/
- dr_zoidberg 11y agoIt is indeed one of their use cases: https://www.sqlite.org/whentouse.html https://www.sqlite.org/whentouse.html