3 ms·
TLDR; "Ingestion of a high number of column files under memory pressure led to the kernel starting readahead disk read operations, which you wouldn't expect fro
by sytse 3y ago
TLDR; "Ingestion of a high number of column files under memory pressure led to the kernel starting readahead disk read operations, which you wouldn't expect from a write-only load. The rest was as simple as using madvise in our code to disable the readahead in table writers."
- EE84M3i 3y agoThe article kind of dances around it, but AIUI the reason that their "weite-only load" caused reads (and thus readahead) was because they were writing to a mapped page that had already been evicted - so the kernel was reading/faulting those pages because it can only write in block/page sized chunks. In some sense maybe this could be thought of as readahead in preparation for writing to those pages, which is undesirable in this case. However, what confused be about this article was if the data files are append only, how is there a "next" block to read ahead to? I guess maybe the files are pre-allocated or the kernel is reading previous pages.
- deleted 3y ago[deleted]
- bremac 3y agoReading between the lines, it sounds as if they're using mmap. There is no "append" operation on a memory mapping, so the file would need to be preallocated before mapping it. If the preallocation is done using fallocate or just writing zeros, then by default it's backed by blocks on disk, and readahead must hit the disk since there is data there. On the other hand, preallocating with fallocate using FALLOC_FL_ZERO_RANGE or (often) with ftruncate() will just update the logical file length, and even if readahead is triggered it won't actually hit the disk.
- EE84M3i 3y agoFor the file being entirely pre-allocated case I understand, but for the file hole case I'm not sure I understand why you'd get such high disk activity. If the index block also got evicted from the page cache, then could reading into a file hole still trigger a fault? Or is the "holiness" of a page for a mapping stored in the page table?
- loeg 3y agoI suspect page size/aligned file holes could be backed by a read-only zero page via PTE as an optimization, but they might not be (I'm not as familiar with Linux mmap/filesystems as with FreeBSD). It is quite possible the filesystem caches, e.g., the file extent tree (including holiness) separately from the backing inode/on-disk sectors for the tree.
- ayende 3y agoUsing _ftruncate_ or FALLOC_FL_ZERO_RANGE is a bad idea for a database. The problem is that you may get an out of disk space error mid operation. If you are using mmap, that will express itself as a segmentation fault, which you really don't want. You _need_ to allocate the file ahead of time, so you can properly behave there.
- pengaru 3y agoThe readahead is a bit of a readaround when I last checked, as in it'll pull in some stuff before the fault as well. There used to be a sys-wide tunable in /sys to control how large an area readahead would extend to, but I'm not seeing it anymore on this 6.1 laptop. I think there's been some work changing stuff to be more clever in this area in recent years. It used to be interesting to make that value small vs. large and see how things like uncached journalctl (heavy mmap user) were affected in terms of performance vs. IO generated.
- EE84M3i 3y agoThe article distinguishes "readaround" from a linear predicted "readahead", but then says the output of blktrace indicates a "potential readahead", which is where I got confused. Does MADV_RANDOM disable both "readahead" and "readaround"?