3 ms·

A long time ago, I wrote an ORM to serialize/deserialize object data seamlessly into SQLite, with massive arrays of floats being stored as blobs. In code, I co

by alterom 1mo ago

A long time ago, I wrote an ORM to serialize/deserialize object data seamlessly into SQLite, with massive arrays of floats being stored as blobs. In code, I could effectively mark which class members need to be stored/restored, and optionally provide a custom serialization function for them if needed. The latter was effectively never necessary, because all the bases types and multi-dimensional arrays were handled by templates. Really wish I open-sourced that thing then, but the corporate bureaucracy around that was tricky. I remember sufficiently little about implementation details now that I think I can get to writing it again, without producing a copypasta of that code - and maybe I should :)

inigyou 1mo ago
Did it end up becoming more painful than just writing the SQL?
- Hendrikto 28d ago
  It almost always does. People put so much effort into building layers of abstractions to avoid using SQL, while just writing it directly would be much faster, easier, more readable, easier to maintain, modify, and debug. I do not get it.
  - alterom 24d ago
    >It almost always does. People put so much effort into building layers of abstractions to avoid using SQL, while just writing it directly would be much faster, easier, more readable, easier to maintain, modify, and debug. I do not get it. When your data is literally hierarchical and all you need to do is read/write one Big Object, "just using SQL" makes no sense. I used SQLite because it makes a lot of sense to use as an application file format[1], i.e. as an alternative to writing directly to the filesystem. The hierarchical data was a mix of parameters and huge blobs, so text-based JSON was a no-go, size-limited BSON was a no-go, at which point I asked myself if there was a reason to not just use SQLite, and found none. [1] https://sqlite.org/appfileformat.html https://sqlite.org/appfileformat.html
- alterom 24d ago
  No it did not. The data wasn't relational. The ORM was the smallest part of it, I could have written/read from JSON too, but SQLite was just a far better choice for a mix of structured data and big blobs. The thing it replaced was writing to a file at a predetermined offsets. Which became very fun as the file format changed between versions and backwards compatibility needed to be maintained.

by alterom 1mo ago

inigyou 1mo ago

Did it end up becoming more painful than just writing the SQL?

Hendrikto 28d ago
It almost always does. People put so much effort into building layers of abstractions to avoid using SQL, while just writing it directly would be much faster, easier, more readable, easier to maintain, modify, and debug. I do not get it.
- alterom 24d ago
  >It almost always does. People put so much effort into building layers of abstractions to avoid using SQL, while just writing it directly would be much faster, easier, more readable, easier to maintain, modify, and debug. I do not get it. When your data is literally hierarchical and all you need to do is read/write one Big Object, "just using SQL" makes no sense. I used SQLite because it makes a lot of sense to use as an application file format[1], i.e. as an alternative to writing directly to the filesystem. The hierarchical data was a mix of parameters and huge blobs, so text-based JSON was a no-go, size-limited BSON was a no-go, at which point I asked myself if there was a reason to not just use SQLite, and found none. [1] https://sqlite.org/appfileformat.html https://sqlite.org/appfileformat.html
alterom 24d ago
No it did not. The data wasn't relational. The ORM was the smallest part of it, I could have written/read from JSON too, but SQLite was just a far better choice for a mix of structured data and big blobs. The thing it replaced was writing to a file at a predetermined offsets. Which became very fun as the file format changed between versions and backwards compatibility needed to be maintained.