4 ms·

"Always start with in-memory data structure with straightforward persistence (i.e. load and save it all at once, using some popular format)." That sounds like

"Always start with in-memory data structure with straightforward persistence (i.e. load and save it all at once, using some popular format)." That sounds like a bad idea in many cases. Prone to many kinds of problems, like accidentally corrupting the stored state beyond recognition from a tiny application bug.

int_19h 8y ago
How exactly would you corrupt it? For all the stuff that's in-memory, it's as safe as your language is. If you're writing it to disk all at once, it's easy to implement this atomically with rename - this guarantees that you have either the old file or the new file, but not some weird in-between state. And if you mean corrupting by the app going amok and asking to corrupt data because of logic errors in the app itself, that's just as doable with a database, and especially doable with local-file DBs (which usually don't try to limit what you can do).
- tim333 8y ago
  I had a python script that was roughly: read file, unpickle, do stuff in memory, pickle, write. A bug in "do stuff in memory" could easily muck things up and I didn't get round to renaming the file for every edit. Yeah you could do some auto rename thing but it might be simpler to just use sqlite.
  - int_19h 8y ago
    > I didn't get round to renaming the file for every edit. That was your problem then, no? If you don't do that from the get go, you're essentially saying that you don't care about atomicity, consistency and durability. > Yeah you could do some auto rename thing but it might be simpler to just use sqlite. You mean atomic rename? The simplest way is to just use a library that does it for you. One does exist for Python. It takes 2 lines of code to safely dump your JSON data with it.
- jeffdavis 8y ago
  Create some ID that's supposed to be unique, and due to a race condition, it's not. Then, before you know it, the data is a mess. Unique keys, foreign keys, check constraints, and DDL changes were all meant to solve real problems. I guess some apps might not benefit as much, but it seems like asking for trouble. Given how easy it is to just use postgres, why invite all the risk?
  - int_19h 8y ago
    If your data is in-memory, why would you have IDs and foreign keys? You'd just have objects referencing other objects (or maps referencing maps, or whatever is idiomatic). I wasn't talking about re-implementing a relational database in-memory, but simply working with data in the way that is natural in your language. If you do actually need advanced schema features, then I agree, just use PostgreSQL. But none of the things that you've listed are advanced features, and all of them have some trivial equivalent for in-memory data. Side note: do you maybe have an image of a very particular kind of app in mind? Like, a moderately sized web app? Keep in mind that I'm talking in general here, about web/desktop/mobile, and about all sizes. And in practice, even for web apps, people often underestimate the actual usage and perf requirements.
    - jeffdavis 8y ago
      If you make code changes, that usually means you need to adjust your data structures, too. That means you need to manage multiple on disk versions, or maybe migrate the old data to the new version. I mean, you can do all of this stuff another way. But PostgreSQL represents a lot of lessons learned over decades with all kinds of applications. Sure, there may be some level at which it's just easier to serialize the in-memory data. But that level is lower than most people realize and postgres is pretty darn easy to use given all the kinds of problems it helps you avoid. Even an MVP of a trivial web app is probably fastest to deliver with postgres, and less risky. And there are tons of things that are really convenient even for the tiniest app. For instance, being able to go in and browse your data to find out what's causing a minor bug.

int_19h 8y ago

How exactly would you corrupt it? For all the stuff that's in-memory, it's as safe as your language is. If you're writing it to disk all at once, it's easy to implement this atomically with rename - this guarantees that you have either the old file or the new file, but not some weird in-between state. And if you mean corrupting by the app going amok and asking to corrupt data because of logic errors in the app itself, that's just as doable with a database, and especially doable with local-file DBs (which usually don't try to limit what you can do).

tim333 8y ago
I had a python script that was roughly: read file, unpickle, do stuff in memory, pickle, write. A bug in "do stuff in memory" could easily muck things up and I didn't get round to renaming the file for every edit. Yeah you could do some auto rename thing but it might be simpler to just use sqlite.
- int_19h 8y ago
  > I didn't get round to renaming the file for every edit. That was your problem then, no? If you don't do that from the get go, you're essentially saying that you don't care about atomicity, consistency and durability. > Yeah you could do some auto rename thing but it might be simpler to just use sqlite. You mean atomic rename? The simplest way is to just use a library that does it for you. One does exist for Python. It takes 2 lines of code to safely dump your JSON data with it.
jeffdavis 8y ago
Create some ID that's supposed to be unique, and due to a race condition, it's not. Then, before you know it, the data is a mess. Unique keys, foreign keys, check constraints, and DDL changes were all meant to solve real problems. I guess some apps might not benefit as much, but it seems like asking for trouble. Given how easy it is to just use postgres, why invite all the risk?
- int_19h 8y ago
  If your data is in-memory, why would you have IDs and foreign keys? You'd just have objects referencing other objects (or maps referencing maps, or whatever is idiomatic). I wasn't talking about re-implementing a relational database in-memory, but simply working with data in the way that is natural in your language. If you do actually need advanced schema features, then I agree, just use PostgreSQL. But none of the things that you've listed are advanced features, and all of them have some trivial equivalent for in-memory data. Side note: do you maybe have an image of a very particular kind of app in mind? Like, a moderately sized web app? Keep in mind that I'm talking in general here, about web/desktop/mobile, and about all sizes. And in practice, even for web apps, people often underestimate the actual usage and perf requirements.
  - jeffdavis 8y ago
    If you make code changes, that usually means you need to adjust your data structures, too. That means you need to manage multiple on disk versions, or maybe migrate the old data to the new version. I mean, you can do all of this stuff another way. But PostgreSQL represents a lot of lessons learned over decades with all kinds of applications. Sure, there may be some level at which it's just easier to serialize the in-memory data. But that level is lower than most people realize and postgres is pretty darn easy to use given all the kinds of problems it helps you avoid. Even an MVP of a trivial web app is probably fastest to deliver with postgres, and less risky. And there are tons of things that are really convenient even for the tiniest app. For instance, being able to go in and browse your data to find out what's causing a minor bug.