7 ms·

Investigating Linux phantom disk reads

302 points by kamaraju 3y ago

rajnathani 3y ago
I know sharing ChatGPT/GPT/AI generated text in comments here can be unappealing, but I would like to share this one as I feel that I managed to get ChatGPT-4 to summarize this article using a non-computer analogy pretty well: "Imagine you work in a library where you store books on shelves. Your primary task is to take new books and put them on the shelves (write-only load). You don't expect to read the books often, so the number of times you need to open and read the books should be minimal. One day, you notice that several books are being opened and read more often than expected, even though your main task is to put away new books. This is confusing and unexpected, so you start investigating why this is happening. After some investigation, you find out that the library assistant (the operating system) is trying to be helpful by anticipating which books might be needed next and opening them ahead of time (readahead). This anticipation works well when there is plenty of shelf space (memory) available. However, when the library gets crowded (memory pressure), the assistant starts anticipating the wrong books, causing unnecessary book openings (phantom reads). To resolve this issue, you tell the library assistant to stop anticipating which books to open (disabling readahead) when you're just putting away new books. This solves the problem and reduces the number of unnecessary book openings. The experience teaches you the importance of understanding how the library assistant works and shows that addressing unexpected issues can lead to improvements in the overall library system."
sytse 3y ago
TLDR; "Ingestion of a high number of column files under memory pressure led to the kernel starting readahead disk read operations, which you wouldn't expect from a write-only load. The rest was as simple as using madvise in our code to disable the readahead in table writers."
- EE84M3i 3y ago
  The article kind of dances around it, but AIUI the reason that their "weite-only load" caused reads (and thus readahead) was because they were writing to a mapped page that had already been evicted - so the kernel was reading/faulting those pages because it can only write in block/page sized chunks. In some sense maybe this could be thought of as readahead in preparation for writing to those pages, which is undesirable in this case. However, what confused be about this article was if the data files are append only, how is there a "next" block to read ahead to? I guess maybe the files are pre-allocated or the kernel is reading previous pages.
  - deleted 3y ago
    [deleted]
  - bremac 3y ago
    Reading between the lines, it sounds as if they're using mmap. There is no "append" operation on a memory mapping, so the file would need to be preallocated before mapping it. If the preallocation is done using fallocate or just writing zeros, then by default it's backed by blocks on disk, and readahead must hit the disk since there is data there. On the other hand, preallocating with fallocate using FALLOC_FL_ZERO_RANGE or (often) with ftruncate() will just update the logical file length, and even if readahead is triggered it won't actually hit the disk.
    - EE84M3i 3y ago
      For the file being entirely pre-allocated case I understand, but for the file hole case I'm not sure I understand why you'd get such high disk activity. If the index block also got evicted from the page cache, then could reading into a file hole still trigger a fault? Or is the "holiness" of a page for a mapping stored in the page table?
      - loeg 3y ago
        I suspect page size/aligned file holes could be backed by a read-only zero page via PTE as an optimization, but they might not be (I'm not as familiar with Linux mmap/filesystems as with FreeBSD). It is quite possible the filesystem caches, e.g., the file extent tree (including holiness) separately from the backing inode/on-disk sectors for the tree.
    - ayende 3y ago
      Using _ftruncate_ or FALLOC_FL_ZERO_RANGE is a bad idea for a database. The problem is that you may get an out of disk space error mid operation. If you are using mmap, that will express itself as a segmentation fault, which you really don't want. You _need_ to allocate the file ahead of time, so you can properly behave there.
  - pengaru 3y ago
pengaru 3y ago
Going through mmap for bulk-ingest sucks because the kernel has to fault in the contents to make what's in-core reflect what's on-disk before your write access to the mapped memory occurs. It's basically a read-modify-write pattern even when all you intended to do was write the entire page. When you just use a write call you provide a unit of arbitrary size, and if you've done your homework that size is a multiple of page size and the offset page-aligned. Then there's no need for the kernel to load anything in for the written pages; you're providing everything in the single call. Then you go down the O_DIRECT rabbithole every fast linux database has historically gone down.
- bluestreak 3y ago
  It is not always read-write-modify. There is no evidence of this pattern in Ubuntu when there is no memory pressure. Merge occurs when block is partially updated after kenel had lost state of the block, which can happen under memory pressure.
  - amluto 3y ago
    Sorry, try again. On x86, and I think every architecture, when you write to a memory mapping that is not already backed by a writable page, the kernel is notified that user code is trying to write. And the kernel needs to fill in the contents of the page, which requires a read if the page isn’t already loaded. It has to be this way! The write could be a read-modify-write instruction. Or it could be a plain store, but I’ve never heard of hardware with write-only memory with fine enough granularity to make this work. The sole exception is if the page in question is all zeros and the kernel can know this without needing to read the file. This might sometimes be the case for an append-only database. I don’t know exactly what QuestDB does. Also: > As soon as you mmap a file, the kernel allocates page table entries (PTEs) for the virtual memory to reserve an address range for your file, Nope. It just makes a record of the existence of the mapping. This is called a VMA in Linux. No PTEs are created unless you set MAP_POPULATE. > but it doesn't read the file contents at this point. The actual data is read into the page when you access the allocated memory, i.e. start reading (LOAD instruction in x86) or writing (STORE instruction in x86) the memory. What are these LOAD and STORE instructions in x86? There are architectures reasonably described as load-store architectures, and x86 isn’t one of them.
davidhyde 3y ago
Seems like using memory mapped files for a write-only load is the sub optimal choice. Maybe I’m mistaken but surely using an append-only file handle would be simpler than changing the behaviour of how memory mapped files are cached like they did for their solution?
addisonj 3y ago
I am going to write this comment with a large preface: I don't think it is ever helpful to be an absolutist. For every best-practice/"right way" to do things, there are circumstances when doing it another way makes sense. That can be a ton of reasons for that, be it technical, money/time, etc. The best engineering teams aren't those that just blindly follow what others say is a best practice but understand the options and make an informed choice. None of the following comment is at all commentary on questDB, as they mention in the article, many databases use similar tools. With that said, after reading the first paragraph I immediately searched the article for "mmap" and had a good sense of where the rest of this was going. Put simply, it is just really hard to consider what the OS is going to do in all situations when using mmap. Based on my experience, I would guess that a ton of people reading this comment have hit issues that, I would argue, is due to using mmap. (Particularly looking at you prometheus). All things told, this is a pretty innocuous incident of mmap causing problems, but I would encourage any aspiring DB engineers to read https://db.cs.cmu.edu/mmap-cidr2022 https://db.cs.cmu.edu/mmap-cidr2022 as it gives a great overview of the range of problems that can occur when using mmap I think some would argue that mmap is "fine" for append only workloads (and is certainly more reasonable compared to a DB with arbitrary updates) but even here, lots of factors like metadata, scaling number of tables, etc will eventually bring you to hit some fundamental problems when using mmap. The interesting opportunity in my mind, especially with improvements in async IO (both at FS level and in tools like rust), is to build higher level abstractions that bring the "simplicity" of mmap, but with more purpose-built semantics ideal for databases.
- Sytten 3y ago
  When I read they were using mmap I immediately thought of Andy Pavlo since he warns against it every time he can in the CMU videos. Enough that I thought it was somewhat of a concensus now that mmap should be avoided specially for databases, guess I was wrong.
  - eatonphil 3y ago
    > a concensus now that mmap should be avoided specially for databases Maybe, but table 1 in Andy Pavlo's paper shows 7 of 10 databases surveyed do still use mmap. Furthermore, that paper clearly demonstrating the issues with mmap came out in 2022 and most databases have been around longer than that. That mmap isn't the future is maybe more certain than that mmap isn't common practice today (because it does sorta seem to be).
0xbadcafebee 3y ago
There are other methods you can use to increase performance under memory pressure, but you'd end up handling i/o directly and maintaining your own index of memory and disk accesses, page-aligned reads/writes, etc. It would be easier to just require your users buy more memory, but when there's a hack like this available, that seems preferable to implementing your own VMM and disk i/o subsystem.
dmazin 3y ago
Am I the only one surprised to read that this database relies on periodic flushing (every 30s by default) with no manual syncs at all? I guess it’s metrics so 30s of data loss is fine? I dunno about that. Data loss is usually due to a power failure, and the metrics collected right before a power failure are important.
- eatonphil 3y ago
  I was surprised by this too. I asked the author about it on Twitter [0]. At the very least it seems like fsync is something you can opt into in their configuration, even if it's not the default. https://twitter.com/eatonphil/status/1653373246929027075 https://twitter.com/eatonphil/status/1653373246929027075
  - dmazin 3y ago
    Nice. Glad you asked. From a developer: “As long as the OS & the HW doesn't crash, the data is safe thanks to the page cache” This is so strange to me. A database that is non-durable by default. OK…
    - josephg 3y ago
      A few years ago I was doing some consulting for a medical tech startup run by an ex-doctor. They were thinking of using mongodb until I explained how it had a reputation for losing data. I'll never forget the look of horror, disgust and confusion on his face. He turned to me and said "A database that forgets things!? Why would anyone want that??". I still don't have an answer for him. It sounds just as strange to me too.
      - matthews2 3y ago
        It's like Snapchat, but for databases!
      - nhourcard 3y ago
        MongoDB is one of the most successful open-source databases of all time. The parent company is a listed company and worth $15BN, 3x more than Elastic to put some perspective. This reflection [1] came from the founders of RethinkDB, a competitor of MongoDB at the time: "It turned out that correctness, simplicity of the interface, and consistency are the wrong metrics of goodness for most users. The majority of users wanted these three trade-offs instead: - A use case. We set out to build a good database system, but users wanted a good way to do X (e.g. a good way to store JSON documents from hapi, a good way to store and analyze logs, a good way to create reports, etc.). - Timely arrival. They wanted the product to actually exist when they needed it, not three years later. - Palpable speed [...]. MongoDB mastered these workloads brilliantly, while we fought the losing battle of educating the market." MongoDB narrowed things down for a specific use case, and became the best for that use case. This comes with trade-offs. MongoDB was probably not the best database for healthcare back in the days, but that is OK. It did the job very well for other use cases and industries. And over time, they fixed the issue around losing data and became more stable. Essentially, they made developers feel like superheroes, and over time improved their product, and eventually grabbed a massive market share. [1] https://www.defmacro.org/2017/01/18/why-rethinkdb-failed.html https://www.defmacro.org/2017/01/18/why-rethinkdb-failed.htm...
deleted 3y ago
[deleted]
speedgoose 3y ago
> It's also important to note that the above percentages… Has this article being written using ChatGPT by any chance ?

302 points by kamaraju 3y ago

I know sharing ChatGPT/GPT/AI generated text in comments here can be unappealing, but I would like to share this one as I feel that I managed to get ChatGPT-4 to summarize this article using a non-computer analogy pretty well: "Imagine you work in a library where you store books on shelves. Your primary task is to take new books and put them on the shelves (write-only load). You don't expect to read the books often, so the number of times you need to open and read the books should be minimal. One day, you notice that several books are being opened and read more often than expected, even though your main task is to put away new books. This is confusing and unexpected, so you start investigating why this is happening. After some investigation, you find out that the library assistant (the operating system) is trying to be helpful by anticipating which books might be needed next and opening them ahead of time (readahead). This anticipation works well when there is plenty of shelf space (memory) available. However, when the library gets crowded (memory pressure), the assistant starts anticipating the wrong books, causing unnecessary book openings (phantom reads). To resolve this issue, you tell the library assistant to stop anticipating which books to open (disabling readahead) when you're just putting away new books. This solves the problem and reduces the number of unnecessary book openings. The experience teaches you the importance of understanding how the library assistant works and shows that addressing unexpected issues can lead to improvements in the overall library system."

sytse 3y ago

TLDR; "Ingestion of a high number of column files under memory pressure led to the kernel starting readahead disk read operations, which you wouldn't expect from a write-only load. The rest was as simple as using madvise in our code to disable the readahead in table writers."

EE84M3i 3y ago
The article kind of dances around it, but AIUI the reason that their "weite-only load" caused reads (and thus readahead) was because they were writing to a mapped page that had already been evicted - so the kernel was reading/faulting those pages because it can only write in block/page sized chunks. In some sense maybe this could be thought of as readahead in preparation for writing to those pages, which is undesirable in this case. However, what confused be about this article was if the data files are append only, how is there a "next" block to read ahead to? I guess maybe the files are pre-allocated or the kernel is reading previous pages.
- deleted 3y ago
  [deleted]
- bremac 3y ago
  Reading between the lines, it sounds as if they're using mmap. There is no "append" operation on a memory mapping, so the file would need to be preallocated before mapping it. If the preallocation is done using fallocate or just writing zeros, then by default it's backed by blocks on disk, and readahead must hit the disk since there is data there. On the other hand, preallocating with fallocate using FALLOC_FL_ZERO_RANGE or (often) with ftruncate() will just update the logical file length, and even if readahead is triggered it won't actually hit the disk.
  - EE84M3i 3y ago
    For the file being entirely pre-allocated case I understand, but for the file hole case I'm not sure I understand why you'd get such high disk activity. If the index block also got evicted from the page cache, then could reading into a file hole still trigger a fault? Or is the "holiness" of a page for a mapping stored in the page table?
    - loeg 3y ago
      I suspect page size/aligned file holes could be backed by a read-only zero page via PTE as an optimization, but they might not be (I'm not as familiar with Linux mmap/filesystems as with FreeBSD). It is quite possible the filesystem caches, e.g., the file extent tree (including holiness) separately from the backing inode/on-disk sectors for the tree.
  - ayende 3y ago
    Using _ftruncate_ or FALLOC_FL_ZERO_RANGE is a bad idea for a database. The problem is that you may get an out of disk space error mid operation. If you are using mmap, that will express itself as a segmentation fault, which you really don't want. You _need_ to allocate the file ahead of time, so you can properly behave there.
- pengaru 3y ago

pengaru 3y ago

Going through mmap for bulk-ingest sucks because the kernel has to fault in the contents to make what's in-core reflect what's on-disk before your write access to the mapped memory occurs. It's basically a read-modify-write pattern even when all you intended to do was write the entire page. When you just use a write call you provide a unit of arbitrary size, and if you've done your homework that size is a multiple of page size and the offset page-aligned. Then there's no need for the kernel to load anything in for the written pages; you're providing everything in the single call. Then you go down the O_DIRECT rabbithole every fast linux database has historically gone down.

bluestreak 3y ago
It is not always read-write-modify. There is no evidence of this pattern in Ubuntu when there is no memory pressure. Merge occurs when block is partially updated after kenel had lost state of the block, which can happen under memory pressure.
- amluto 3y ago
  Sorry, try again. On x86, and I think every architecture, when you write to a memory mapping that is not already backed by a writable page, the kernel is notified that user code is trying to write. And the kernel needs to fill in the contents of the page, which requires a read if the page isn’t already loaded. It has to be this way! The write could be a read-modify-write instruction. Or it could be a plain store, but I’ve never heard of hardware with write-only memory with fine enough granularity to make this work. The sole exception is if the page in question is all zeros and the kernel can know this without needing to read the file. This might sometimes be the case for an append-only database. I don’t know exactly what QuestDB does. Also: > As soon as you mmap a file, the kernel allocates page table entries (PTEs) for the virtual memory to reserve an address range for your file, Nope. It just makes a record of the existence of the mapping. This is called a VMA in Linux. No PTEs are created unless you set MAP_POPULATE. > but it doesn't read the file contents at this point. The actual data is read into the page when you access the allocated memory, i.e. start reading (LOAD instruction in x86) or writing (STORE instruction in x86) the memory. What are these LOAD and STORE instructions in x86? There are architectures reasonably described as load-store architectures, and x86 isn’t one of them.

davidhyde 3y ago

Seems like using memory mapped files for a write-only load is the sub optimal choice. Maybe I’m mistaken but surely using an append-only file handle would be simpler than changing the behaviour of how memory mapped files are cached like they did for their solution?

addisonj 3y ago

I am going to write this comment with a large preface: I don't think it is ever helpful to be an absolutist. For every best-practice/"right way" to do things, there are circumstances when doing it another way makes sense. That can be a ton of reasons for that, be it technical, money/time, etc. The best engineering teams aren't those that just blindly follow what others say is a best practice but understand the options and make an informed choice. None of the following comment is at all commentary on questDB, as they mention in the article, many databases use similar tools. With that said, after reading the first paragraph I immediately searched the article for "mmap" and had a good sense of where the rest of this was going. Put simply, it is just really hard to consider what the OS is going to do in all situations when using mmap. Based on my experience, I would guess that a ton of people reading this comment have hit issues that, I would argue, is due to using mmap. (Particularly looking at you prometheus). All things told, this is a pretty innocuous incident of mmap causing problems, but I would encourage any aspiring DB engineers to read https://db.cs.cmu.edu/mmap-cidr2022 https://db.cs.cmu.edu/mmap-cidr2022 as it gives a great overview of the range of problems that can occur when using mmap I think some would argue that mmap is "fine" for append only workloads (and is certainly more reasonable compared to a DB with arbitrary updates) but even here, lots of factors like metadata, scaling number of tables, etc will eventually bring you to hit some fundamental problems when using mmap. The interesting opportunity in my mind, especially with improvements in async IO (both at FS level and in tools like rust), is to build higher level abstractions that bring the "simplicity" of mmap, but with more purpose-built semantics ideal for databases.

Sytten 3y ago
When I read they were using mmap I immediately thought of Andy Pavlo since he warns against it every time he can in the CMU videos. Enough that I thought it was somewhat of a concensus now that mmap should be avoided specially for databases, guess I was wrong.
- eatonphil 3y ago
  > a concensus now that mmap should be avoided specially for databases Maybe, but table 1 in Andy Pavlo's paper shows 7 of 10 databases surveyed do still use mmap. Furthermore, that paper clearly demonstrating the issues with mmap came out in 2022 and most databases have been around longer than that. That mmap isn't the future is maybe more certain than that mmap isn't common practice today (because it does sorta seem to be).

0xbadcafebee 3y ago

There are other methods you can use to increase performance under memory pressure, but you'd end up handling i/o directly and maintaining your own index of memory and disk accesses, page-aligned reads/writes, etc. It would be easier to just require your users buy more memory, but when there's a hack like this available, that seems preferable to implementing your own VMM and disk i/o subsystem.

dmazin 3y ago

Am I the only one surprised to read that this database relies on periodic flushing (every 30s by default) with no manual syncs at all? I guess it’s metrics so 30s of data loss is fine? I dunno about that. Data loss is usually due to a power failure, and the metrics collected right before a power failure are important.

eatonphil 3y ago
I was surprised by this too. I asked the author about it on Twitter [0]. At the very least it seems like fsync is something you can opt into in their configuration, even if it's not the default. https://twitter.com/eatonphil/status/1653373246929027075 https://twitter.com/eatonphil/status/1653373246929027075
- dmazin 3y ago
  Nice. Glad you asked. From a developer: “As long as the OS & the HW doesn't crash, the data is safe thanks to the page cache” This is so strange to me. A database that is non-durable by default. OK…
  - josephg 3y ago
    A few years ago I was doing some consulting for a medical tech startup run by an ex-doctor. They were thinking of using mongodb until I explained how it had a reputation for losing data. I'll never forget the look of horror, disgust and confusion on his face. He turned to me and said "A database that forgets things!? Why would anyone want that??". I still don't have an answer for him. It sounds just as strange to me too.
    - matthews2 3y ago
      It's like Snapchat, but for databases!
    - nhourcard 3y ago
      MongoDB is one of the most successful open-source databases of all time. The parent company is a listed company and worth $15BN, 3x more than Elastic to put some perspective. This reflection [1] came from the founders of RethinkDB, a competitor of MongoDB at the time: "It turned out that correctness, simplicity of the interface, and consistency are the wrong metrics of goodness for most users. The majority of users wanted these three trade-offs instead: - A use case. We set out to build a good database system, but users wanted a good way to do X (e.g. a good way to store JSON documents from hapi, a good way to store and analyze logs, a good way to create reports, etc.). - Timely arrival. They wanted the product to actually exist when they needed it, not three years later. - Palpable speed [...]. MongoDB mastered these workloads brilliantly, while we fought the losing battle of educating the market." MongoDB narrowed things down for a specific use case, and became the best for that use case. This comes with trade-offs. MongoDB was probably not the best database for healthcare back in the days, but that is OK. It did the job very well for other use cases and industries. And over time, they fixed the issue around losing data and became more stable. Essentially, they made developers feel like superheroes, and over time improved their product, and eventually grabbed a massive market share. [1] https://www.defmacro.org/2017/01/18/why-rethinkdb-failed.html https://www.defmacro.org/2017/01/18/why-rethinkdb-failed.htm...

deleted 3y ago

[deleted]

speedgoose 3y ago

> It's also important to note that the above percentages… Has this article being written using ChatGPT by any chance ?