5 ms·

We have a production service running for years that just mmaps an entire SSD and casts the pointer to the desired C++ data structure. That SSD doesn't even hav

We have a production service running for years that just mmaps an entire SSD and casts the pointer to the desired C++ data structure. That SSD doesn't even have a file system on it, instead it directly stores one monstrous struct array filled with data. There's also no recovery, if the SSD breaks you need to recover all data from a backup. But it works and it's mind-boggingly fast and cheap.

brysonreece 3y ago
Wow. How do design decisions get made that result in these types of situations in the first place?
- username135 3y ago
  If I had to guess: Doing it this way = $ Doing it that way = $$$
- MBCook 3y ago
  Honestly it’s not too far off from what many databases do if they can. They manage one giant file as if it’s their own personal drive of memory and ignore the concept of a filesystem completely. Obviously that breaks down when you need to span multiple disks, but conceptually it really is quite simple. A lot of the other stuff file systems do are to help keep things consistent. But if there’s only one “file“ and you don’t ever need metadata then you don’t really need that. Very smart solution really.
  - jasonwatkinspdx 3y ago
    Yeah, a lot of database storage engines use O_DIRECT because the OS's general purpose cache heuristics are inferior vs them doing their own buffer pool management. That said if you try this naively you're likely to end up doing something a lot worse than the Linux kernel.
- _zoltan_ 3y ago
  it's a very reasonable thing to do if you need performance.
- taneq 3y ago
  Someone says "hey if we had 900GB of RAM we could make a lot of money" and then someone else says "that's ridiculous and impossib- hang on a minute" and scurries off to hack together some tech heresy over their lunch break.
  - speedgoose 3y ago
    By the way, you can find single servers with 32TB of ram nowadays.
whartung 3y ago
I've always wanted a Smalltalk VM that did this. Eternally persistent VM, without having to "save". It just "lives". Go ahead, map a 10GB or 100GB file to the VM and go at it. Imagine your entire email history (everyone seems to have large email histories) in the "email array", all as ST objects. Just as an example. Is that "good"? I dunno. But, simply, there is no impedance mismatch. There's no persistence layer, your entire heap is simply mmap'd into a blob of storage with some lightweight flushing mechanic. Obviously it's not that simple, there's all sorts of caveats. It just feels like it should be that simple, and we've had the tech to do this since forever. It doesn't even have to be blistering fast, simply "usable".
- MBCook 3y ago
  Isn’t that sort of the original idea for how Forth would work? Everything is just one big memory space and you do whatever you need? I’m going from very hazy memory here.
  - _0ffh 3y ago
    I think it is, although you have to manually save the current image if you want to keep the changes you made. Which I find entirely reasonable. I also think that what gp is looking for is Scratch. IIRC it's a complete graphical Smalltalk environment where everything is saved in one big image file. You change some function, it stays changed.
- jdougan 3y ago
  Arguably you could use GemStone/S like that, though it's probably not the kind of capabilities you want.
- tonyarkles 3y ago
  That is so wonderfully fascinating to me. You could just download a file into a variable and when that variable goes out of scope/has no more references it’d just be automatically “deleted”. Since there’s no longer a concrete “thing” called a file, you can organize them however you want and with whatever “metadata” you want by having a dict with the metadata you want and some convention like :file as the key that points to the body. Arbitrary indexes too; any number of data structures could all share a reference to the same variable. Simple databases are just made up of collections of objects. Foreign key constraints? Just make the instance variable type a non-nullable type. Indexes? Lists of tuples that point to the objects. More complex databases and queries can provide a set of functions as an API. You can write queries in SQL or you can just provide a map/filter/reduce function with the predicate written in normal code. Graph databases too: you can just run Dijkstra’s algorithm or TSP or whatever directly on a rich persistent data structure. Thanks for the neat idea to riff on. I like it! Thinking about it in practice makes me a little anxious, but the theory is beautiful.
lepisma 3y ago
Very interesting. Can you give a sense of the speed up factor?
nurettin 3y ago
I do similar things with mmap and dumping raw structs to get insane speeds one wouldn't expect to get from traditional databases. Perhaps you could even pause the operations, snapshot with dd and resume everything back in order to get a backup.
jasonwatkinspdx 3y ago
LMDB has a mode designed to do something similar, if anyone wants something like this with just a bit more structure to it like transactional updates via CoW and garbage collection of old versions. It's single writer via a lock but readers are lock/coordination free. A long running read transaction can delay garbage collection however.
anthk 3y ago
Linux had methods to avoid fysnc on filesystems and if you used an SSD and a SAI you would usually have no problems. Pixar used that to write GB's of renders and media for instance.
Shorel 3y ago
I love this one. If anyone from AWS reads your comment, they could have an idea for a new "product" xD
MaxBarraclough 3y ago
Sounds fragile, C++ compilers are permitted to do struct padding essentially as they please. A change in compiler could break the SSD<-->struct mapping (i.e. the member offsets). C++ arrays, on the other hand, are guaranteed not to have padding. That's essentially what memory-mapped IO gives you out of the box. https://stackoverflow.com/a/5398498 https://stackoverflow.com/a/5398498 http://www.catb.org/esr/structure-packing/#_structure_alignment_and_padding http://www.catb.org/esr/structure-packing/#_structure_alignm...
- lights0123 3y ago
  I'm not familiar with C++ rules (the linked answer seems very suspect to me, their argument being "if you change the struct, the struct changes!"), but they could absolutely just be declaring them as extern "C" to use C's layout rules.

brysonreece 3y ago

Wow. How do design decisions get made that result in these types of situations in the first place?

username135 3y ago
If I had to guess: Doing it this way = $ Doing it that way = $$$
MBCook 3y ago
Honestly it’s not too far off from what many databases do if they can. They manage one giant file as if it’s their own personal drive of memory and ignore the concept of a filesystem completely. Obviously that breaks down when you need to span multiple disks, but conceptually it really is quite simple. A lot of the other stuff file systems do are to help keep things consistent. But if there’s only one “file“ and you don’t ever need metadata then you don’t really need that. Very smart solution really.
- jasonwatkinspdx 3y ago
  Yeah, a lot of database storage engines use O_DIRECT because the OS's general purpose cache heuristics are inferior vs them doing their own buffer pool management. That said if you try this naively you're likely to end up doing something a lot worse than the Linux kernel.
_zoltan_ 3y ago
it's a very reasonable thing to do if you need performance.
taneq 3y ago
Someone says "hey if we had 900GB of RAM we could make a lot of money" and then someone else says "that's ridiculous and impossib- hang on a minute" and scurries off to hack together some tech heresy over their lunch break.
- speedgoose 3y ago
  By the way, you can find single servers with 32TB of ram nowadays.

whartung 3y ago

I've always wanted a Smalltalk VM that did this. Eternally persistent VM, without having to "save". It just "lives". Go ahead, map a 10GB or 100GB file to the VM and go at it. Imagine your entire email history (everyone seems to have large email histories) in the "email array", all as ST objects. Just as an example. Is that "good"? I dunno. But, simply, there is no impedance mismatch. There's no persistence layer, your entire heap is simply mmap'd into a blob of storage with some lightweight flushing mechanic. Obviously it's not that simple, there's all sorts of caveats. It just feels like it should be that simple, and we've had the tech to do this since forever. It doesn't even have to be blistering fast, simply "usable".

MBCook 3y ago
Isn’t that sort of the original idea for how Forth would work? Everything is just one big memory space and you do whatever you need? I’m going from very hazy memory here.
- _0ffh 3y ago
  I think it is, although you have to manually save the current image if you want to keep the changes you made. Which I find entirely reasonable. I also think that what gp is looking for is Scratch. IIRC it's a complete graphical Smalltalk environment where everything is saved in one big image file. You change some function, it stays changed.
jdougan 3y ago
Arguably you could use GemStone/S like that, though it's probably not the kind of capabilities you want.
tonyarkles 3y ago
That is so wonderfully fascinating to me. You could just download a file into a variable and when that variable goes out of scope/has no more references it’d just be automatically “deleted”. Since there’s no longer a concrete “thing” called a file, you can organize them however you want and with whatever “metadata” you want by having a dict with the metadata you want and some convention like :file as the key that points to the body. Arbitrary indexes too; any number of data structures could all share a reference to the same variable. Simple databases are just made up of collections of objects. Foreign key constraints? Just make the instance variable type a non-nullable type. Indexes? Lists of tuples that point to the objects. More complex databases and queries can provide a set of functions as an API. You can write queries in SQL or you can just provide a map/filter/reduce function with the predicate written in normal code. Graph databases too: you can just run Dijkstra’s algorithm or TSP or whatever directly on a rich persistent data structure. Thanks for the neat idea to riff on. I like it! Thinking about it in practice makes me a little anxious, but the theory is beautiful.

lepisma 3y ago

Very interesting. Can you give a sense of the speed up factor?

nurettin 3y ago

I do similar things with mmap and dumping raw structs to get insane speeds one wouldn't expect to get from traditional databases. Perhaps you could even pause the operations, snapshot with dd and resume everything back in order to get a backup.

jasonwatkinspdx 3y ago

LMDB has a mode designed to do something similar, if anyone wants something like this with just a bit more structure to it like transactional updates via CoW and garbage collection of old versions. It's single writer via a lock but readers are lock/coordination free. A long running read transaction can delay garbage collection however.

anthk 3y ago

Linux had methods to avoid fysnc on filesystems and if you used an SSD and a SAI you would usually have no problems. Pixar used that to write GB's of renders and media for instance.

Shorel 3y ago

I love this one. If anyone from AWS reads your comment, they could have an idea for a new "product" xD

MaxBarraclough 3y ago

Sounds fragile, C++ compilers are permitted to do struct padding essentially as they please. A change in compiler could break the SSD<-->struct mapping (i.e. the member offsets). C++ arrays, on the other hand, are guaranteed not to have padding. That's essentially what memory-mapped IO gives you out of the box. https://stackoverflow.com/a/5398498 https://stackoverflow.com/a/5398498 http://www.catb.org/esr/structure-packing/#_structure_alignment_and_padding http://www.catb.org/esr/structure-packing/#_structure_alignm...

lights0123 3y ago
I'm not familiar with C++ rules (the linked answer seems very suspect to me, their argument being "if you change the struct, the struct changes!"), but they could absolutely just be declaring them as extern "C" to use C's layout rules.