4 ms·

I once saw a project in the wild where the "database" was implemented using filesystem directories as "tables" with JSON files inside as "rows". When I asked p

by lpapez 2y ago

I once saw a project in the wild where the "database" was implemented using filesystem directories as "tables" with JSON files inside as "rows". When I asked people working on it if they considered Redis or Mongo or Postgres with jsonb columns, they just said they considered all of those things but decided to roll out their own db anyway because "they understood it better". This article gives off the same energy. I really hope it works out for you, but IMO spending innovation tokens to build a database is nuts.

ActorNightly 2y ago
This isn't innovation though. You literally just write your server like you would for a single machine, then wrap it any of the available Raft libraries. AWS and other cloud providers are money printers because a lot of engineers are insanely tied into established patterns of doing things and can't think through things at a fundamental level. Ive seen company backends where their entire AWS stacks could be replaced by a 2 EC2 instances behind a load balancer with a domain name, without affecting business flow. We did something similar to the work in the OP post at my work, we had a bunch of ECS tasks for a service, where the service did another call to an upstream service to fetch some intermediate results. We wanted to cache results for lower response latency. People were working to set up a Redis cluster. Except the TPS of the service was like 0.1. Took me one day to code a /sync api endpoint, which was just a replica of the main endpoint. The only difference is that the main endpoint would spin of a thread to call the /sync endpoint, whereas the /sync endpoint didn't. Both endpoints ended with caching the results in memory before returning. Easy as day, no additional infra costs necessary. But overall, personally, I don't hate the "spending innovation tokens to build a database is nuts" sentiment too much, because it keeps me employed at high salary while doing minimal work, where things that really should be basic CS are considered innovation.
- gtirloni 2y ago
  Software Engineering is different than CS though.
- bastawhiz 2y ago
  > then wrap it any of the available Raft libraries. Raft does consensus. Raft does not do persistence to disk, WAL, crash recovery, indexing, vacuuming (you're using tombstones for your deletes, right?), or any of the other necessary pieces of a database. That's not mentioning how such a system has no query engine, so every piece of data you're looking up in every place you need data is traversing your bespoke data structures. What you described isn't a database. Keeping some disposable values cached isn't a database.
  - tdrhq 2y ago
    Raft does do persistence and crash recovery, at least of the transaction logs. What you need from your side (and there are libraries that already do this): a) A mechanism to snapshot all the data b) An easy in-memory mechanism to create indexes on fields--not strictly needed, but definitely makes things a lot more easier to work with. Bespoke data structures are just simple classes, so if you're familiar with traversing simple objects in the language of your choice, you're all set. You might be over-estimating the benefits of a query engine (and I have worked at multiple places that used MySQL extensively, and used MySQL to build heavily scaled software in the past).
    - bastawhiz 2y ago
      > Raft does do persistence and crash recovery, at least of the transaction logs. It simply does not. The paper that definitionally is Raft doesn't tell you how to interact with durable storage. The raft protocol handles crash recovery in so far as it allows one or more nodes to rebuild state after a crash, but Raft doesn't talk about serialization or WAL or any of the other things you inevitably have to do for reliability and performance. It gives you a way to go from some existing state to the state of the leader (even if that means downloading the full state from scratch), but it doesn't give you a way to go from a pile of bits on a disk to that existing state. If you have a library that implements Raft and gives you those things, that's not Raft giving you things. And that library could just be SQLite. > You might be over-estimating the benefits of a query engine No, I'm not. It's great to describe the data I want and get, say, an array of strings back without having to crawl some Btrees by hand.
jmull 2y ago
I get your point and I don’t doubt the project you’re talking about was a mess, but the file system is a database, and can be a very good choice, depending on exactly what you’re doing.
- rrrix1 2y ago
  The file system is a database and an API. Magic!
philipwhiuk 2y ago
> I once saw a project in the wild where the "database" was implemented using filesystem directories as "tables" with JSON files inside as "rows". I did this sort of thing recently. I felt bad doing it, I still objectively hate it, because I do know enough to know that basically I'm re-implementing what years of hardworking O/S developers have done, piecemeal. But at least I'm going in with my eyes open which feels better. The only real mitigating factor I have is that the application is largely 'never-read' and then when reading is done, it's sequential batches. Which is not normally something databases optimise for and works okay for file-storage. (If someone does know a lightweight database architecture that performs like this, let me know).
- sahil-kang 2y ago
  fwiw, I normally think of LMDB when the idea of using the fs instead of a db crosses my mind
- djhn 2y ago
  Why not sqlite? put the json in a single column, maybe copy some parts of it or metadata to another two or three. Should be faster than the filesystem for reading multiple rows.
- lpapez 2y ago
  Why not start with SQLite? There are benchmarks out there proving that for some use cases (i.e. many small updates) where using SQLite is faster than using the filesystem. [1] So not only do you get all of the benefits of a relational database, and literally centuries of engineering hours and bugfixes invested into SQLite, you might also get better performance (which is why I presume you even considered rolling your own). [1] https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html

by lpapez 2y ago

ActorNightly 2y ago

This isn't innovation though. You literally just write your server like you would for a single machine, then wrap it any of the available Raft libraries. AWS and other cloud providers are money printers because a lot of engineers are insanely tied into established patterns of doing things and can't think through things at a fundamental level. Ive seen company backends where their entire AWS stacks could be replaced by a 2 EC2 instances behind a load balancer with a domain name, without affecting business flow. We did something similar to the work in the OP post at my work, we had a bunch of ECS tasks for a service, where the service did another call to an upstream service to fetch some intermediate results. We wanted to cache results for lower response latency. People were working to set up a Redis cluster. Except the TPS of the service was like 0.1. Took me one day to code a /sync api endpoint, which was just a replica of the main endpoint. The only difference is that the main endpoint would spin of a thread to call the /sync endpoint, whereas the /sync endpoint didn't. Both endpoints ended with caching the results in memory before returning. Easy as day, no additional infra costs necessary. But overall, personally, I don't hate the "spending innovation tokens to build a database is nuts" sentiment too much, because it keeps me employed at high salary while doing minimal work, where things that really should be basic CS are considered innovation.

gtirloni 2y ago
Software Engineering is different than CS though.
bastawhiz 2y ago
> then wrap it any of the available Raft libraries. Raft does consensus. Raft does not do persistence to disk, WAL, crash recovery, indexing, vacuuming (you're using tombstones for your deletes, right?), or any of the other necessary pieces of a database. That's not mentioning how such a system has no query engine, so every piece of data you're looking up in every place you need data is traversing your bespoke data structures. What you described isn't a database. Keeping some disposable values cached isn't a database.
- tdrhq 2y ago
  Raft does do persistence and crash recovery, at least of the transaction logs. What you need from your side (and there are libraries that already do this): a) A mechanism to snapshot all the data b) An easy in-memory mechanism to create indexes on fields--not strictly needed, but definitely makes things a lot more easier to work with. Bespoke data structures are just simple classes, so if you're familiar with traversing simple objects in the language of your choice, you're all set. You might be over-estimating the benefits of a query engine (and I have worked at multiple places that used MySQL extensively, and used MySQL to build heavily scaled software in the past).
  - bastawhiz 2y ago
    > Raft does do persistence and crash recovery, at least of the transaction logs. It simply does not. The paper that definitionally is Raft doesn't tell you how to interact with durable storage. The raft protocol handles crash recovery in so far as it allows one or more nodes to rebuild state after a crash, but Raft doesn't talk about serialization or WAL or any of the other things you inevitably have to do for reliability and performance. It gives you a way to go from some existing state to the state of the leader (even if that means downloading the full state from scratch), but it doesn't give you a way to go from a pile of bits on a disk to that existing state. If you have a library that implements Raft and gives you those things, that's not Raft giving you things. And that library could just be SQLite. > You might be over-estimating the benefits of a query engine No, I'm not. It's great to describe the data I want and get, say, an array of strings back without having to crawl some Btrees by hand.

jmull 2y ago

I get your point and I don’t doubt the project you’re talking about was a mess, but the file system is a database, and can be a very good choice, depending on exactly what you’re doing.

rrrix1 2y ago
The file system is a database and an API. Magic!

philipwhiuk 2y ago

> I once saw a project in the wild where the "database" was implemented using filesystem directories as "tables" with JSON files inside as "rows". I did this sort of thing recently. I felt bad doing it, I still objectively hate it, because I do know enough to know that basically I'm re-implementing what years of hardworking O/S developers have done, piecemeal. But at least I'm going in with my eyes open which feels better. The only real mitigating factor I have is that the application is largely 'never-read' and then when reading is done, it's sequential batches. Which is not normally something databases optimise for and works okay for file-storage. (If someone does know a lightweight database architecture that performs like this, let me know).

sahil-kang 2y ago
fwiw, I normally think of LMDB when the idea of using the fs instead of a db crosses my mind
djhn 2y ago
Why not sqlite? put the json in a single column, maybe copy some parts of it or metadata to another two or three. Should be faster than the filesystem for reading multiple rows.
lpapez 2y ago
Why not start with SQLite? There are benchmarks out there proving that for some use cases (i.e. many small updates) where using SQLite is faster than using the filesystem. [1] So not only do you get all of the benefits of a relational database, and literally centuries of engineering hours and bugfixes invested into SQLite, you might also get better performance (which is why I presume you even considered rolling your own). [1] https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html