4 ms·

> Well, today computers really only have one kind of storage, and it is usually some sort of disk, the operating system and the virtual memory management hardwa

> Well, today computers really only have one kind of storage, and it is usually some sort of disk, the operating system and the virtual memory management hardware has converted the RAM to a cache for the disk storage. > So what happens with squids elaborate memory management is that it gets into fights with the kernels elaborate memory management, and like any civil war, that never gets anything done. ctrl+f mmap on this article doesn't return anything so.. is it talking it's better to mmap files and having files store in-memory data structures like cap'n'proto does? (if it's entirely in-memory with no file in disk, then it's surely comparing apples to oranges? if a program needs the persistent memory, it needs the disk) Also: is mmaping a file and using it directly from memory what databases should do today to avoid fighting the OS disk cache?

genericlemon24 5y ago
Yes, I understood that to mean mmap(). > is mmaping a file and using it directly from memory what databases should do today Some do; for example, SQLite: https://www.sqlite.org/mmap.html https://www.sqlite.org/mmap.html (the page has pros and cons, and explains how SQLite does memory mapped I/O).
atonse 5y ago
I thought the fact that mongodb data files were essentially mmapped was used as a critique against its durability as a primary database. Is that not true? Or was that not a relevant critique?
- nextaccountic 5y ago
  Welp, I don't know! From the link of the other comment here, Sqlite disables mmap by default because Linux provides a poor API regarding I/O errors (they are sent as a signal instead)
- jfindley 5y ago
  A database like mongodb is a very different animal from a http cache. The mongodb authors likely read this article (it's from 2006) when they were designing the first version of their database, and due to their almost complete unfamiliarity with what they were doing didn't understand that these two things are not alike. Databases have different classes of objects (think index vs data, though there are more than just these two), different types of access (e.g. full table scan vs single record select) and a whole lot more complexity than I can cover here. You don't really ever want to page out your indicies, but paging out data is probably fine. You likely want to avoid replacing your entire cache for full table scan of a big table. You probably want to have some concept of ensuring a single user doesn't starve out all your other 1000+ users. Etc etc. In other words, you can't get away with just opting out of memory management for a database - you need to treat different things differently. I understand that a few years in, the mongo devs eventually caught up to 1990s era mysql and realised this, leading to a new DB engine that was a bit less bad.
- jd_mongodb 5y ago
  MongoDB released the Wired Tiger storage engine in 2015 which replaced the MMAP storage engine. So whatever your read about MMAP is obsolete. https://www.mongodb.com/presentations/a-technical-introduction-to-wiredtiger https://www.mongodb.com/presentations/a-technical-introducti...
  - atonse 5y ago
    Yep I know that. I’m saying mmap used to be a critique. I left the place I introduced mongo to, a couple months before they bought WiredTiger
- aidenn0 5y ago
  It's a valid critique, but A cache has no durability requirements.
jasonwatkinspdx 5y ago
Mmaping directly readable data is certainly a way of avoiding the problems PHK is talking about, and can be reasonably efficient. Sendfile can be even better if you can structure things to have a metadata index that's in memory or mmaped, and never need to read the actual sent data. But mmap is no panacea. It's big flaw is that the kernel scheduler often doesn't know as much as your app about what you're doing. So when you touch an out of core page and it page faults, your thread just gets locked up. Worse, the kernel's read ahead logic may make the wrong guess about how many pages you're going to read, or when you want data written back out. You can try to mitigate this using madvise and msync, but in practice it's kinda brittle. Some systems use blocking io with a dedicated thread pool to wait on the block. This way the main threads can move on to other useful work rather than sitting stalled on a page fault. Golang's runtime does something like this for you automatically. Databases for the most part do their own scheduling, and may even use async io. This is because they know far better what they're actually doing than the kernel can guess. Another reason is the database's buffer cache is usually tightly coupled with the consistency control algorithm, so again, you don't want the kernel guessing, you want total control. Databases often use direct io and take on the burden of scheduling everything themselves precisely for this reason. LMDB uses mmap, but should not really be considered a database. It's just an embedded CoW btree, though a fine one. Andy Pavlo's openly published course materials are a great resource for learning about database internels. He used to be emphatically against mmap for databases, but in the last couple years has moderated that position somewhat. io_uring is looking like it will be the best overall solution moving forward, at least for linux only.
- nextaccountic 5y ago
  Thanks for the explanation. > io_uring is looking like it will be the best overall solution moving forward, at least for linux only. But io_uring means duplicating memory, right? There will be both my in-memory representation and the kernel cache for the file and they will be fighting each other (or at least, consuming more memory than necessary.
  - jasonwatkinspdx 5y ago
    You can use io_uring with O_DIRECT, with the usual alignment restrictions. io_uring itself is carefully designed to eliminate copying even request/completion event structures.

genericlemon24 5y ago

Yes, I understood that to mean mmap(). > is mmaping a file and using it directly from memory what databases should do today Some do; for example, SQLite: https://www.sqlite.org/mmap.html https://www.sqlite.org/mmap.html (the page has pros and cons, and explains how SQLite does memory mapped I/O).

atonse 5y ago

I thought the fact that mongodb data files were essentially mmapped was used as a critique against its durability as a primary database. Is that not true? Or was that not a relevant critique?

nextaccountic 5y ago
Welp, I don't know! From the link of the other comment here, Sqlite disables mmap by default because Linux provides a poor API regarding I/O errors (they are sent as a signal instead)
jfindley 5y ago
A database like mongodb is a very different animal from a http cache. The mongodb authors likely read this article (it's from 2006) when they were designing the first version of their database, and due to their almost complete unfamiliarity with what they were doing didn't understand that these two things are not alike. Databases have different classes of objects (think index vs data, though there are more than just these two), different types of access (e.g. full table scan vs single record select) and a whole lot more complexity than I can cover here. You don't really ever want to page out your indicies, but paging out data is probably fine. You likely want to avoid replacing your entire cache for full table scan of a big table. You probably want to have some concept of ensuring a single user doesn't starve out all your other 1000+ users. Etc etc. In other words, you can't get away with just opting out of memory management for a database - you need to treat different things differently. I understand that a few years in, the mongo devs eventually caught up to 1990s era mysql and realised this, leading to a new DB engine that was a bit less bad.
jd_mongodb 5y ago
MongoDB released the Wired Tiger storage engine in 2015 which replaced the MMAP storage engine. So whatever your read about MMAP is obsolete. https://www.mongodb.com/presentations/a-technical-introduction-to-wiredtiger https://www.mongodb.com/presentations/a-technical-introducti...
- atonse 5y ago
  Yep I know that. I’m saying mmap used to be a critique. I left the place I introduced mongo to, a couple months before they bought WiredTiger
aidenn0 5y ago
It's a valid critique, but A cache has no durability requirements.

jasonwatkinspdx 5y ago

Mmaping directly readable data is certainly a way of avoiding the problems PHK is talking about, and can be reasonably efficient. Sendfile can be even better if you can structure things to have a metadata index that's in memory or mmaped, and never need to read the actual sent data. But mmap is no panacea. It's big flaw is that the kernel scheduler often doesn't know as much as your app about what you're doing. So when you touch an out of core page and it page faults, your thread just gets locked up. Worse, the kernel's read ahead logic may make the wrong guess about how many pages you're going to read, or when you want data written back out. You can try to mitigate this using madvise and msync, but in practice it's kinda brittle. Some systems use blocking io with a dedicated thread pool to wait on the block. This way the main threads can move on to other useful work rather than sitting stalled on a page fault. Golang's runtime does something like this for you automatically. Databases for the most part do their own scheduling, and may even use async io. This is because they know far better what they're actually doing than the kernel can guess. Another reason is the database's buffer cache is usually tightly coupled with the consistency control algorithm, so again, you don't want the kernel guessing, you want total control. Databases often use direct io and take on the burden of scheduling everything themselves precisely for this reason. LMDB uses mmap, but should not really be considered a database. It's just an embedded CoW btree, though a fine one. Andy Pavlo's openly published course materials are a great resource for learning about database internels. He used to be emphatically against mmap for databases, but in the last couple years has moderated that position somewhat. io_uring is looking like it will be the best overall solution moving forward, at least for linux only.

nextaccountic 5y ago
Thanks for the explanation. > io_uring is looking like it will be the best overall solution moving forward, at least for linux only. But io_uring means duplicating memory, right? There will be both my in-memory representation and the kernel cache for the file and they will be fighting each other (or at least, consuming more memory than necessary.
- jasonwatkinspdx 5y ago
  You can use io_uring with O_DIRECT, with the usual alignment restrictions. io_uring itself is carefully designed to eliminate copying even request/completion event structures.