3 ms·

Every time I read explanations of any database anything, it looks childishly simplistic to what I've done with C pointers and structs. Basically it all exists

Every time I read explanations of any database anything, it looks childishly simplistic to what I've done with C pointers and structs. Basically it all exists because of storage devices: - large transfer factors (spinning platter hard drive track access, flash erase blocks) - latency (spinning platter track to track seek, and rotational latency) Once you have fast random access, it's data structures and algorithms: forget the database cruft.

mkleczek 2y ago
That’s a very simplistic view. For once: you _always_ have to deal with memory hierarchy as there is no way to fit all data to the fastest memory. So you need to design with this in mind. Secondly: the point of databases is not to implement a single specific task efficiently but: a) provide means to maintain data so that it is easy to implement many different tasks (both current and future) b) to safely _share_ data between various tasks/programs/users
- kazinator 2y ago
  There are all kinds of situations in which, if all your data fits into dynamic RAM, you don't have to give any other damn. (This is basically the bulk of the premise of technology like Python, for instance). Nobody ever rebooted their machine because it became unusable due to their L1 or L2 cache thrashing. You really have to care more about that particular hierarchy level transition, RAM to external storage, more than the others, because traditionally the transition is very steep, like suddenly hitting a brick wall of slowness. The higher levels hierarchies can be left to generic management mechanisms, like the hardware itself or compiler register allocations. The ram to storage transition is managed at run-time by software. Databases have to bypass the operating system version of that management, and provide their own, because it is so critical.
petalmind 2y ago
> Once you have fast random access, it's data structures and algorithms: forget the database cruft. I mean, yeah, but in practice you have to provide ACID or other invariants, you have to provide distributed processing and storage, you have to provide replication etc., etc. Also, "once you have fast random access" sounds like "once you have gajillion dollars". I too want a pony!
- kazinator 2y ago
  Systems which handle everything in memory can also be distributed and be required to provide consistency guarantees. "Once you have fast random access" happens just from moving up a step in the memory hierarchy; that's what I mean.
RaftPeople 2y ago
> Every time I read explanations of any database anything, it looks childishly simplistic to what I've done with C pointers and structs. Basically it all exists because of storage devices: One of the key motivations for relational was to have a standardized system for data storage so data consumption was simpler. Prior to relational every system had it's own unique method of storing data (e.g. your pointers and structs, and other things etc.) which required unique custom programming for every new data consumption use case. With relational, there is a simpler (not simple, but much simpler) method of access that no longer requires a developer write a custom program to get the data for each new request.

mkleczek 2y ago

That’s a very simplistic view. For once: you _always_ have to deal with memory hierarchy as there is no way to fit all data to the fastest memory. So you need to design with this in mind. Secondly: the point of databases is not to implement a single specific task efficiently but: a) provide means to maintain data so that it is easy to implement many different tasks (both current and future) b) to safely _share_ data between various tasks/programs/users

kazinator 2y ago
There are all kinds of situations in which, if all your data fits into dynamic RAM, you don't have to give any other damn. (This is basically the bulk of the premise of technology like Python, for instance). Nobody ever rebooted their machine because it became unusable due to their L1 or L2 cache thrashing. You really have to care more about that particular hierarchy level transition, RAM to external storage, more than the others, because traditionally the transition is very steep, like suddenly hitting a brick wall of slowness. The higher levels hierarchies can be left to generic management mechanisms, like the hardware itself or compiler register allocations. The ram to storage transition is managed at run-time by software. Databases have to bypass the operating system version of that management, and provide their own, because it is so critical.

petalmind 2y ago

> Once you have fast random access, it's data structures and algorithms: forget the database cruft. I mean, yeah, but in practice you have to provide ACID or other invariants, you have to provide distributed processing and storage, you have to provide replication etc., etc. Also, "once you have fast random access" sounds like "once you have gajillion dollars". I too want a pony!

kazinator 2y ago
Systems which handle everything in memory can also be distributed and be required to provide consistency guarantees. "Once you have fast random access" happens just from moving up a step in the memory hierarchy; that's what I mean.

RaftPeople 2y ago

> Every time I read explanations of any database anything, it looks childishly simplistic to what I've done with C pointers and structs. Basically it all exists because of storage devices: One of the key motivations for relational was to have a standardized system for data storage so data consumption was simpler. Prior to relational every system had it's own unique method of storing data (e.g. your pointers and structs, and other things etc.) which required unique custom programming for every new data consumption use case. With relational, there is a simpler (not simple, but much simpler) method of access that no longer requires a developer write a custom program to get the data for each new request.