4 ms·

> database is an append-only file of JSON objects separated by newlines. When the app restarts, it reads the file and rebuilds its memory image. All data is in

by dmw_ng 5y ago

> database is an append-only file of JSON objects separated by newlines. When the app restarts, it reads the file and rebuilds its memory image. All data is in RAM Apps like this tend to perform like an absolute whippet too (or if they dont, getting them to perform well is often a 5 line change). It's really freeing to be able to write scans and filters with simple loops that still return results faster than a network roundtrip to a database. The problem is always growth, either GC jank from a massive heap, running out of RAM, or those loops eventually catching up with you. Fixing any one of these eventually involves either serialization or IO, at which point the balance is destroyed and a real database wins again.

bob1029 5y ago
> The problem is always growth, either GC jank from a massive heap, running out of RAM, or those loops eventually catching up with you Absolutely. The challenge is having enough faith that it will take long enough to catch up to you. Statistically speaking, it won't catch up to you and if it does, it will take so long you should have seen it coming from miles away and had time to prepare. In my systems that use an in-memory/append-only technique, I try to keep only the pointers and basic indexes in memory. With modern PCIe flash storage, there is no good justification for keeping big fat blobs around in memory anymore.
- aprdm 5y ago
  Could you expand what you mean by keeping pointers and basic index in memory?
  - bob1029 5y ago
    Pointers are tuples of (Id, LogOffset) and are used to map logical identities to positions of those objects in the append-only log. Indexes are usually a tuple of (Some64BitKey, Id) and are used to map physical business keys to logical object identities. These entries are only candidates in the case where the key material needs to be hashed and inspected for actual equivalence. One big advantage with this approach is that you can stream big blobs directly out of the log to a caller-supplied buffer or stream. No intermediate allocations required aside from some small buffers.
    - aprdm 5y ago
      Awesome, thank you for sharing!
Beltalowda 5y ago
Another issue with "just a JSON file" as a database is that you need to be a bit careful to avoid race conditions and the like, e.g. if two web pages try to write the same database at the same time. It's not an issue for all applications, and not that hard to get right, but does require some effort. This is a huge reason I prefer SQLite for simple file storage needs.
- brundolf 5y ago
  A normal Express app (assuming it's one process per JSON file) shouldn't have that problem, because JavaScript is single-threaded
  - djbusby 5y ago
    Is it possible a write is interrupted on it's turn in the event-loop, and crossed with another?
    - brundolf 5y ago
      Hmm. I wouldn't think so, but I don't actually know Still, given the strategy at hand, the in-memory JS object (exclusively single-threaded) is the source of truth, and just gets mirrored in the file system (and doesn't get read again until the next startup). So you should have an eventual-consistency situation in the worst case (any racing issue between file-writes would just put the file in a stale state, and the next file-write would bring it back up to consistency)
  - spiffytech 5y ago
    It can definitely be a problem in Node.js. Assuming the workflow is read from disk -> modify -> write to disk, and that you're using the async fs functions, two async code paths running at the same time will have last-write-wins semantics and will lose data. That's the naive scenario. If all code paths write out a global data structure, then it'd be fine. Or if the file is written append-only instead of as a single, atomic data structure, then it could be fine.
  - midrus 5y ago
pavlov 5y ago
Yes, you need to be sure that you understand the growth pattern if you want to YOLO in RAM. If your product aims to be the next Instagram, this is clearly not the architecture. But a lot of small businesses are genuinely small. They may not sign up new customers that often. When they do, the impact to the service is often very predictable ("Amy at customer X uses this every other day, she's very happy, it generates 100 requests / week"). If growth picks up, there would be signs well in advance of the toy service becoming an actual problem.
- jkaptur 5y ago
  > If your product aims to be the next Instagram, this is clearly not the architecture. But maybe! https://instagram-engineering.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172 https://instagram-engineering.com/dismissing-python-garbage-...

by dmw_ng 5y ago

bob1029 5y ago

> The problem is always growth, either GC jank from a massive heap, running out of RAM, or those loops eventually catching up with you Absolutely. The challenge is having enough faith that it will take long enough to catch up to you. Statistically speaking, it won't catch up to you and if it does, it will take so long you should have seen it coming from miles away and had time to prepare. In my systems that use an in-memory/append-only technique, I try to keep only the pointers and basic indexes in memory. With modern PCIe flash storage, there is no good justification for keeping big fat blobs around in memory anymore.

aprdm 5y ago
Could you expand what you mean by keeping pointers and basic index in memory?
- bob1029 5y ago
  Pointers are tuples of (Id, LogOffset) and are used to map logical identities to positions of those objects in the append-only log. Indexes are usually a tuple of (Some64BitKey, Id) and are used to map physical business keys to logical object identities. These entries are only candidates in the case where the key material needs to be hashed and inspected for actual equivalence. One big advantage with this approach is that you can stream big blobs directly out of the log to a caller-supplied buffer or stream. No intermediate allocations required aside from some small buffers.
  - aprdm 5y ago
    Awesome, thank you for sharing!

Beltalowda 5y ago

Another issue with "just a JSON file" as a database is that you need to be a bit careful to avoid race conditions and the like, e.g. if two web pages try to write the same database at the same time. It's not an issue for all applications, and not that hard to get right, but does require some effort. This is a huge reason I prefer SQLite for simple file storage needs.

brundolf 5y ago
A normal Express app (assuming it's one process per JSON file) shouldn't have that problem, because JavaScript is single-threaded
- djbusby 5y ago
  Is it possible a write is interrupted on it's turn in the event-loop, and crossed with another?
  - brundolf 5y ago
    Hmm. I wouldn't think so, but I don't actually know Still, given the strategy at hand, the in-memory JS object (exclusively single-threaded) is the source of truth, and just gets mirrored in the file system (and doesn't get read again until the next startup). So you should have an eventual-consistency situation in the worst case (any racing issue between file-writes would just put the file in a stale state, and the next file-write would bring it back up to consistency)
- spiffytech 5y ago
  It can definitely be a problem in Node.js. Assuming the workflow is read from disk -> modify -> write to disk, and that you're using the async fs functions, two async code paths running at the same time will have last-write-wins semantics and will lose data. That's the naive scenario. If all code paths write out a global data structure, then it'd be fine. Or if the file is written append-only instead of as a single, atomic data structure, then it could be fine.
- midrus 5y ago

pavlov 5y ago

Yes, you need to be sure that you understand the growth pattern if you want to YOLO in RAM. If your product aims to be the next Instagram, this is clearly not the architecture. But a lot of small businesses are genuinely small. They may not sign up new customers that often. When they do, the impact to the service is often very predictable ("Amy at customer X uses this every other day, she's very happy, it generates 100 requests / week"). If growth picks up, there would be signs well in advance of the toy service becoming an actual problem.

jkaptur 5y ago
> If your product aims to be the next Instagram, this is clearly not the architecture. But maybe! https://instagram-engineering.com/dismissing-python-garbage-collection-at-instagram-4dca40b29172 https://instagram-engineering.com/dismissing-python-garbage-...