3 ms·

This shows that we lack good abstractions over storage.

by Kinrany 3y ago

osigurdson 3y ago
You cannot abstract away a 3 order of magnitude difference in bandwidth and latency.
- pclmulqdq 3y ago
  Where did you get a 3 order of magnitude difference? Are you still using hard drives for your storage medium?
  - osigurdson 3y ago
    Adding two numbers together takes on the order of a nanosecond. Doing the same thing using a rest / http service (like an idiot) in the same datacenter takes on the order of a millisecond. Six orders of magnitude actually.
    - pclmulqdq 3y ago
      I'm pretty sure the parent comment was about storage media, not about network hops and service boundaries. Also, extremely basic REST/HTTP services definitely do not take 1 millisecond even if you are bad at software - the overhead of that stack is in the tens of microseconds if you are doing nothing. For the comparison being referenced here, if you want to compare RAM, the storage medium that backs compute, to modern persistent storage, here it is: * 40 GB/s per DIMM vs 5-10 GB/s per NVMe SSD. At most one order of magnitude off, but you can pack enough disks into a computer that the throughput ratio is almost 1:1. AWS EBS is about 1 order of magnitude different here, and that is with network-attached storage. * 100-200 ns latency (RAM) vs 10-50 us (fast SSD) - about 2 orders of magnitude, but also possible to hide with batching.
ben_jones 3y ago
It’s more like data storage needs are not one size fits all so it’s better left to the user who best knows their storage needs.
- miningape 3y ago
  This is interesting, makes me wonder if a "dockerised" database is something people could use. I mean a database frontend with its own language/protocols/whatever that allows you to define the data structure but leaves the specific storage engine or format as a backend detail that can change from platform to platform.
  - mjr00 3y ago
    > I mean a database frontend with its own language/protocols/whatever that allows you to define the data structure but leaves the specific storage engine or format as a backend detail that can change from platform to platform. That's more or less a description of SQL.
  - rekwah 3y ago
    Postgres wire format is indirectly getting there. Plenty of tools use that with wildly different storage engines on the other end. A clean room implementation would likely yield different results but there appears to be some appetite for a solution.
- crabbone 3y ago
  Nah. It's not that. We lack a concept that can organize storage. Let me illustrate this. So, until some years ago there was complete nonsense and anarchy in Linux networking management. That is until we got the "ip" program. There's still nonsense and anarchy, because the "ip" program doesn't cover everything, but it's on the right track to organize everything Linux knows about networking under one roof. So, vendors today, like, say, Melanox (i.e. NVidia) choose to interface with "ip" and work with that stack rather than invent their own interfaces. When it's extendable in predictable and convenient ways, user will extend and enrich functionality. Now, compare this to Linux storage... I want to scream and kill somebody every time I have to deal with any aspect of it because of how poorly mismanaged it is. There's no uniformity, plenty of standards where at most one is necessary, duplication upon duplication, layers... well, forget layers. Like, say, you wanted a RAID0, well, you have MD RAIDs, you have LVM RAIDs, you have ZFS RAIDs, you have multipassing with DM (is that a RAID, well sorta' depends on what you expected...) also, well, Ceph RDB are also kind of like RAIDs, DRBD can also sort of be like a RAID... Do you maybe also want snapshots? How about encryption? -- Every solution will end up so particularly tailored to the needs of your organization that even an experienced admin in this very area your org is specializing will have no clue what's going on with your storage. Needs can be studied, understood, catalogued, rolled into some sort of a hierarchy or some other structure amenable to management. We haven't solved this problem. But we have an even bigger one: no coordination and no desire to coordinate even within Linux core components, forget third-party vendors.
yjftsjthsd-h 3y ago
The space is complex enough that I wonder if it's possible to make abstractions that aren't horribly leaky.
- pphysch 3y ago
  I blame the SQL "standard". It's a massive, unnecessary abstraction layer that only complicates attempts to build bridges between code and relational databases (which I believe is the most general-purpose paradigm). Personally, I am working on a modern Python ORM for PostgreSQL and PostgreSQL alone.
jandrese 3y ago
Isn't this what SQL is supposed to be? You bring the DBI for your database and plug it into the app. Shame that it doesn't work out so well in practice.
Too 3y ago
Cloud providers are (counter to intuition about their lock-in incentives) improving the space of this. Lots of tools now allow configuring storage by just pointing to various cloud stores, most often a S3 compatible api, not exclusively though. K8s PersistentVolume is another decent shot at storage abstraction, only a bit raw. Finally, more and more tools expect you to have a Postgres they can plug into as backend. All above assumes you want to treat the library data as a big unknown blob. Once data start being corrupted and need bespoke repair, things are less fun. Access and retention is another fun rabbit hole. Data is complicated.
chii 3y ago
But that and abstraction exist, because SQL exists. If the library is designed to send sql to another storage library...

This shows that we lack good abstractions over storage.

osigurdson 3y ago

You cannot abstract away a 3 order of magnitude difference in bandwidth and latency.

pclmulqdq 3y ago
Where did you get a 3 order of magnitude difference? Are you still using hard drives for your storage medium?
- osigurdson 3y ago
  Adding two numbers together takes on the order of a nanosecond. Doing the same thing using a rest / http service (like an idiot) in the same datacenter takes on the order of a millisecond. Six orders of magnitude actually.
  - pclmulqdq 3y ago
    I'm pretty sure the parent comment was about storage media, not about network hops and service boundaries. Also, extremely basic REST/HTTP services definitely do not take 1 millisecond even if you are bad at software - the overhead of that stack is in the tens of microseconds if you are doing nothing. For the comparison being referenced here, if you want to compare RAM, the storage medium that backs compute, to modern persistent storage, here it is: * 40 GB/s per DIMM vs 5-10 GB/s per NVMe SSD. At most one order of magnitude off, but you can pack enough disks into a computer that the throughput ratio is almost 1:1. AWS EBS is about 1 order of magnitude different here, and that is with network-attached storage. * 100-200 ns latency (RAM) vs 10-50 us (fast SSD) - about 2 orders of magnitude, but also possible to hide with batching.

ben_jones 3y ago

It’s more like data storage needs are not one size fits all so it’s better left to the user who best knows their storage needs.

miningape 3y ago
This is interesting, makes me wonder if a "dockerised" database is something people could use. I mean a database frontend with its own language/protocols/whatever that allows you to define the data structure but leaves the specific storage engine or format as a backend detail that can change from platform to platform.
- mjr00 3y ago
  > I mean a database frontend with its own language/protocols/whatever that allows you to define the data structure but leaves the specific storage engine or format as a backend detail that can change from platform to platform. That's more or less a description of SQL.
- rekwah 3y ago
  Postgres wire format is indirectly getting there. Plenty of tools use that with wildly different storage engines on the other end. A clean room implementation would likely yield different results but there appears to be some appetite for a solution.
crabbone 3y ago
Nah. It's not that. We lack a concept that can organize storage. Let me illustrate this. So, until some years ago there was complete nonsense and anarchy in Linux networking management. That is until we got the "ip" program. There's still nonsense and anarchy, because the "ip" program doesn't cover everything, but it's on the right track to organize everything Linux knows about networking under one roof. So, vendors today, like, say, Melanox (i.e. NVidia) choose to interface with "ip" and work with that stack rather than invent their own interfaces. When it's extendable in predictable and convenient ways, user will extend and enrich functionality. Now, compare this to Linux storage... I want to scream and kill somebody every time I have to deal with any aspect of it because of how poorly mismanaged it is. There's no uniformity, plenty of standards where at most one is necessary, duplication upon duplication, layers... well, forget layers. Like, say, you wanted a RAID0, well, you have MD RAIDs, you have LVM RAIDs, you have ZFS RAIDs, you have multipassing with DM (is that a RAID, well sorta' depends on what you expected...) also, well, Ceph RDB are also kind of like RAIDs, DRBD can also sort of be like a RAID... Do you maybe also want snapshots? How about encryption? -- Every solution will end up so particularly tailored to the needs of your organization that even an experienced admin in this very area your org is specializing will have no clue what's going on with your storage. Needs can be studied, understood, catalogued, rolled into some sort of a hierarchy or some other structure amenable to management. We haven't solved this problem. But we have an even bigger one: no coordination and no desire to coordinate even within Linux core components, forget third-party vendors.

yjftsjthsd-h 3y ago

The space is complex enough that I wonder if it's possible to make abstractions that aren't horribly leaky.

pphysch 3y ago
I blame the SQL "standard". It's a massive, unnecessary abstraction layer that only complicates attempts to build bridges between code and relational databases (which I believe is the most general-purpose paradigm). Personally, I am working on a modern Python ORM for PostgreSQL and PostgreSQL alone.

jandrese 3y ago

Isn't this what SQL is supposed to be? You bring the DBI for your database and plug it into the app. Shame that it doesn't work out so well in practice.

Too 3y ago

Cloud providers are (counter to intuition about their lock-in incentives) improving the space of this. Lots of tools now allow configuring storage by just pointing to various cloud stores, most often a S3 compatible api, not exclusively though. K8s PersistentVolume is another decent shot at storage abstraction, only a bit raw. Finally, more and more tools expect you to have a Postgres they can plug into as backend. All above assumes you want to treat the library data as a big unknown blob. Once data start being corrupted and need bespoke repair, things are less fun. Access and retention is another fun rabbit hole. Data is complicated.

chii 3y ago

But that and abstraction exist, because SQL exists. If the library is designed to send sql to another storage library...