4 ms·

i struggle to see why in-memory data structure is the way to start can you please explain. do you create json for all your users?

by ousta 8y ago

chrisseaton 8y ago
> i struggle to see why in-memory data structure is the way to start can you please explain. It’s much simpler. The idea is to start simple and avoid complexity. Do you need the extra complexity of a PostgreSQL instance or can you get away with doing it in-memory? If so you can save yourself a lot of complexity.
- lowercased 8y ago
  > If so you can save yourself a lot of complexity. The complexity exists, you're just shifting it around some. The "complexity" of setting up a populate db server is non-zero, but generally a known-entity. The complexity of managing your own stuff 'in-memory', with all the attendant issues about concurrency/corruption/performance/etc - would almost always end up being far more custom to your particular project.
  - int_19h 8y ago
    If that were true, we wouldn't store any data in memory at all - everything would be in the database to avoid "the attendant issues about concurrency/corruption/performance/etc". But we don't. Perhaps because the attendant issues often aren't issues in practice? There's no concurrency problem if it's just your app working with data, and it's logically single threaded (i.e. possibly async, but serialized) - which is by far the most common model for non-web apps. There's also no concurrency problem if you use immutable data structures. There's no corruption problem if you are using a memory-safe language, and your data structures enforce their invariants, as they should in any good design regardless of anything else. Performance is going to be better, often significantly so, until you have enough data that dumb queries over it are slower than the overhead of communicating with the database. And so on. So, no, the complexity doesn't necessarily exist. The complexity of your data is constant, and of the kind of processing that you need to do with it, true. But we're talking about complexity of the pipeline you use to access that data - and there, you can absolutely make it more complicated that it needs to be, and I'm arguing that it is what usually happens in practice.
- ousta 8y ago
  surely you are using angular, nodejs or react right? why not just raw html and css? its much simpler
int_19h 8y ago
You create a JSON (for example) for all your users when you need to store it. But in memory, it's just a collection of User objects - or whatever is idiomatic for your PL. And you query it with the same tools your language offers - e.g. sequence comprehensions. So there's no impedance mismatch, and no need for the vastly more complicated code that is needed to bridge it. Even if it's not your code - i.e. if you're using an ORM or something similar - it's still unneeded complexity, if you don't have too many users. Think of it from the opposite perspective - if you can work directly with in-memory data, why would you prefer to run SQL queries instead? The latter is obviously more complex, but what's the advantage? There are many valid cases that have a good answer to that question, but there are many more that don't. I can't help but think that we're conditioned to use DBMS (SQL or not) largely by inertia. When we learn about them, the examples are necessarily toy ones - tables with a dozen of records, that sort of thing. But one side effect is that it subtly normalizes the notion that this much data warrants a dedicated DBMS - so people don't balk at even patently ridiculous setups, like a separate SQL server used to hold a grand total of a couple thousand records in all its tables (this is a real thing, something that I personally did in a LOB app many years ago; I didn't have a good justification, it was pure cargo cult, and the app could have done everything it did in-memory, faster, and easier to code). And sometimes, when you ask, people say, "yeah, it's only 1000 records now, but we're going to grow later, and then we'll need a DBMS". But you'd need to grow by many orders of magnitude to get there - and if you get that opportunity, you'll also have the resources to adapt. But everybody dreams of being Google. And then you have apps like, say, a todo list manager app. Is it ever going to deal with millions of todo records? I don't think so. Then why does it need an SQLite DB?
- dchest 8y ago
  Unfortunately, I've rarely seen such a simple thing as file saving implemented correctly by many developers. Missing fsync(), missing checking of errors on close(), in-place non-atomic data overwrites. One of the good things about using SQLite/Postgres/MySQL is that at least they save your data correctly.
- fauigerzigerk 8y ago
  I don't think the number of records is the right benchmark. The reason why data is better kept in a database system is because it often has a very different life cycle than applications. It often lives longer and gets used by more than one application. That's why modelling and storing data somewhat separately from applications often makes sense regardless of the amount of data you have. Also, procedural code is often far more complicated than a SQL query regardless of the number of records being processed. But that obviously depends very much on the specific problem, on the programming language and on the developer's skill set.
seanwilson 8y ago
To clarify, I was talking about client-side code where there isn't a server component e.g. so your choices are usually to store in files, SQLite or IndexedDB.

i struggle to see why in-memory data structure is the way to start can you please explain. do you create json for all your users?

by ousta 8y ago

i struggle to see why in-memory data structure is the way to start can you please explain. do you create json for all your users?

chrisseaton 8y ago

> i struggle to see why in-memory data structure is the way to start can you please explain. It’s much simpler. The idea is to start simple and avoid complexity. Do you need the extra complexity of a PostgreSQL instance or can you get away with doing it in-memory? If so you can save yourself a lot of complexity.

lowercased 8y ago
> If so you can save yourself a lot of complexity. The complexity exists, you're just shifting it around some. The "complexity" of setting up a populate db server is non-zero, but generally a known-entity. The complexity of managing your own stuff 'in-memory', with all the attendant issues about concurrency/corruption/performance/etc - would almost always end up being far more custom to your particular project.
- int_19h 8y ago
  If that were true, we wouldn't store any data in memory at all - everything would be in the database to avoid "the attendant issues about concurrency/corruption/performance/etc". But we don't. Perhaps because the attendant issues often aren't issues in practice? There's no concurrency problem if it's just your app working with data, and it's logically single threaded (i.e. possibly async, but serialized) - which is by far the most common model for non-web apps. There's also no concurrency problem if you use immutable data structures. There's no corruption problem if you are using a memory-safe language, and your data structures enforce their invariants, as they should in any good design regardless of anything else. Performance is going to be better, often significantly so, until you have enough data that dumb queries over it are slower than the overhead of communicating with the database. And so on. So, no, the complexity doesn't necessarily exist. The complexity of your data is constant, and of the kind of processing that you need to do with it, true. But we're talking about complexity of the pipeline you use to access that data - and there, you can absolutely make it more complicated that it needs to be, and I'm arguing that it is what usually happens in practice.
ousta 8y ago
surely you are using angular, nodejs or react right? why not just raw html and css? its much simpler

int_19h 8y ago

You create a JSON (for example) for all your users when you need to store it. But in memory, it's just a collection of User objects - or whatever is idiomatic for your PL. And you query it with the same tools your language offers - e.g. sequence comprehensions. So there's no impedance mismatch, and no need for the vastly more complicated code that is needed to bridge it. Even if it's not your code - i.e. if you're using an ORM or something similar - it's still unneeded complexity, if you don't have too many users. Think of it from the opposite perspective - if you can work directly with in-memory data, why would you prefer to run SQL queries instead? The latter is obviously more complex, but what's the advantage? There are many valid cases that have a good answer to that question, but there are many more that don't. I can't help but think that we're conditioned to use DBMS (SQL or not) largely by inertia. When we learn about them, the examples are necessarily toy ones - tables with a dozen of records, that sort of thing. But one side effect is that it subtly normalizes the notion that this much data warrants a dedicated DBMS - so people don't balk at even patently ridiculous setups, like a separate SQL server used to hold a grand total of a couple thousand records in all its tables (this is a real thing, something that I personally did in a LOB app many years ago; I didn't have a good justification, it was pure cargo cult, and the app could have done everything it did in-memory, faster, and easier to code). And sometimes, when you ask, people say, "yeah, it's only 1000 records now, but we're going to grow later, and then we'll need a DBMS". But you'd need to grow by many orders of magnitude to get there - and if you get that opportunity, you'll also have the resources to adapt. But everybody dreams of being Google. And then you have apps like, say, a todo list manager app. Is it ever going to deal with millions of todo records? I don't think so. Then why does it need an SQLite DB?

dchest 8y ago
Unfortunately, I've rarely seen such a simple thing as file saving implemented correctly by many developers. Missing fsync(), missing checking of errors on close(), in-place non-atomic data overwrites. One of the good things about using SQLite/Postgres/MySQL is that at least they save your data correctly.
fauigerzigerk 8y ago
I don't think the number of records is the right benchmark. The reason why data is better kept in a database system is because it often has a very different life cycle than applications. It often lives longer and gets used by more than one application. That's why modelling and storing data somewhat separately from applications often makes sense regardless of the amount of data you have. Also, procedural code is often far more complicated than a SQL query regardless of the number of records being processed. But that obviously depends very much on the specific problem, on the programming language and on the developer's skill set.

seanwilson 8y ago

To clarify, I was talking about client-side code where there isn't a server component e.g. so your choices are usually to store in files, SQLite or IndexedDB.