9 ms·

It's certainly good to understand that there is caching, journaling, etc. though I think that most people realize it by now. It's very easy to notice at least p

by jchw 3y ago

It's certainly good to understand that there is caching, journaling, etc. though I think that most people realize it by now. It's very easy to notice at least part of this by removing a USB stick without unmounting it on Linux :) However, if you're one of today's lucky 10,000, welcome to the curse of knowledge. On that note, I can only imagine the war stories that PostgreSQL developers could tell about disk syncing. > How slow? An SSD is 1,000 times slower than memory. A spinning hard drive is one MILLION times slower. Disks are multiple orders of magnitude slower than memory! Latency? Throughput? I don't think SSDs are 1,000 times slower than RAM in either category, assuming fairly typical consumer machines, but I could be wrong.

dmazin 3y ago
Thank you for engaging. Also, thank you for finding that imprecise language. An original draft had stated that I'm talking about I/O duration, but I lost that language. I will re-add it. My numbers are for "main memory access", "SSD I/O" and "rotational disk I/O" from Gregg System Performance (chapter 2). In retrospect, with all respect to Gregg, that wording is itself pretty imprecise. I think what really matters here is I/O duration rather than throughput. That is, the reason we use the cache so much is because a single I/O is really slow against disks...
- jchw 3y ago
  Yeah, that's true. Raw speeds don't really mean a whole lot without more context, and in practice writing to disk is going to be a lot slower even if the peak throughputs are not quite that far off. The performance characteristics are quite different in practice. I assume at one point in history though, these numbers were closer to true even for raw throughput... PCIe and NVMe have probably helped close the gap a bit, I would guess.
marginalia_nu 3y ago
> Latency? Throughput? I, don't think SSDs are 1,000 times slower than RAM in either category, assuming fairly typical consumer machines, but I could be wrong. Everything in this business is like a great big onion of caching and buffering layers. Disks have them too. This is why you often see consumer SSDs have very fast writes only as long as you don't write continuously. Continuous writes causes their internal buffers to fill up and the real write performance to be revealed. There's also factors like write amplification and overprovisioning that enter into this set of equations. But if there's any takeaway from this, it's that I/O benchmarking is hell.
- dmazin 3y ago
  I actually want to explore this further sometime. It's not like it's a secret that buffers are everywhere in computers, but I am still surprised at how many buffers one comes across. Like, let's say you are doing some writing using the C standard library... * the library itself will buffer your writes, to reduce the number of syscalls * I think your bio objects might get buffered, via plugging, to reduce the number of block requests * your block requests will get buffered, so that they can be canceled or merged (I'm actually not sure if this happens in the "software staging queue" part of blk-mq, or prior...) * as you mentioned, the disk controller may very well buffer the actual write operations
  - hoten 3y ago
    John Carmack's lecture on systems engineering / reducing latency for VR mentions buffers repeatedly: https://youtu.be/lHLpKzUxjGk https://youtu.be/lHLpKzUxjGk
  - formerly_proven 3y ago
    > I actually want to explore this further sometime. It's not like it's a secret that buffers are everywhere in computers, but I am still surprised at how many buffers one comes across. Well there's a reason it takes a click or key press to register about as long as it takes sending a packet halfway across the world (or a few times around the world, depending on what software you're running).
    - ryandrake 3y ago
      It must have been a real blast back in the 8-bit days, when you'd have some peripheral's port mapped to a specific address in the memory map, and setting a byte to that address actually slammed the electrons into that peripheral's register. No wasteful 15 layers of OS abstractions waiting for buffers to fill and copying bytes from buffer to buffer all the way down.
- jchw 3y ago
  Oh yeah, that I can't deny. The peak throughput numbers used in marketing are definitely questionable at times. Kinda makes sense considering SSDs and HDDs are basically computers of their own that give you some abstraction over the actual raw storage media. You could compare memcpy to sequential write speed or something, but it'd definitely not tell the full story.
Palomides 3y ago
SSDs, even nvme, are indeed about 1000x more latency tham RAM normal RAM only has like 10x the bandwidth of a high end SSD, though
cratermoon 3y ago
> the war stories that PostgreSQL developers could tell about disk syncing. The man page for glibc open(2) used to say, "Most Linux filesystems don't actually implement the POSIX O_SYNC semantics, which require all metadata updates of a write to be on disk on returning to userspace", but now it notes that "before Linux 2.6.33" it wasn't implemented correctly I remember a rather eye-opening discussion on either the postgresql-hackers or lkml about the semantics of O_SYNC not actually being synchronous. In addition to the implementation details, which depended on the version of the kernel and the underlying filesystem type, it was also noted that underlying disk hardware also had caches which did not necessarily behave synchronously. Some places I can find that cover these details include https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg272253.html https://www.mail-archive.com/linux-kernel@vger.kernel.org/ms... http://milek.blogspot.com/2010/12/linux-osync-and-write-barriers.html http://milek.blogspot.com/2010/12/linux-osync-and-write-barr... https://lwn.net/Articles/352229/ https://lwn.net/Articles/352229/
- sroussey 3y ago
  RAID cards back in the day, and some enterprise SSDs today either had batteries or capacitors in order to finish the write, even if the power turned off. They tell the OS that the write has synced even when it has not as it has confidence that it will.
foobiekr 3y ago
Not only are they 1000x slower than memory, if you bump into thermal management, much slower than that. Worse, if you run into wear leveling write amplification, orders of magnitude slower. Yes, you can use caching to hide this, in main memory and on the device (safely and unsafely, depending on the design and whether it is an enterprise drive with sufficient capacitors), but you can do the same with SRAM vs. DRAM and it doesn't make DRAM as fast as SRAM, or registers-vs-SRAM, likewise.

by jchw 3y ago

dmazin 3y ago

Thank you for engaging. Also, thank you for finding that imprecise language. An original draft had stated that I'm talking about I/O duration, but I lost that language. I will re-add it. My numbers are for "main memory access", "SSD I/O" and "rotational disk I/O" from Gregg System Performance (chapter 2). In retrospect, with all respect to Gregg, that wording is itself pretty imprecise. I think what really matters here is I/O duration rather than throughput. That is, the reason we use the cache so much is because a single I/O is really slow against disks...

jchw 3y ago
Yeah, that's true. Raw speeds don't really mean a whole lot without more context, and in practice writing to disk is going to be a lot slower even if the peak throughputs are not quite that far off. The performance characteristics are quite different in practice. I assume at one point in history though, these numbers were closer to true even for raw throughput... PCIe and NVMe have probably helped close the gap a bit, I would guess.

marginalia_nu 3y ago

> Latency? Throughput? I, don't think SSDs are 1,000 times slower than RAM in either category, assuming fairly typical consumer machines, but I could be wrong. Everything in this business is like a great big onion of caching and buffering layers. Disks have them too. This is why you often see consumer SSDs have very fast writes only as long as you don't write continuously. Continuous writes causes their internal buffers to fill up and the real write performance to be revealed. There's also factors like write amplification and overprovisioning that enter into this set of equations. But if there's any takeaway from this, it's that I/O benchmarking is hell.

dmazin 3y ago
I actually want to explore this further sometime. It's not like it's a secret that buffers are everywhere in computers, but I am still surprised at how many buffers one comes across. Like, let's say you are doing some writing using the C standard library... * the library itself will buffer your writes, to reduce the number of syscalls * I think your bio objects might get buffered, via plugging, to reduce the number of block requests * your block requests will get buffered, so that they can be canceled or merged (I'm actually not sure if this happens in the "software staging queue" part of blk-mq, or prior...) * as you mentioned, the disk controller may very well buffer the actual write operations
- hoten 3y ago
  John Carmack's lecture on systems engineering / reducing latency for VR mentions buffers repeatedly: https://youtu.be/lHLpKzUxjGk https://youtu.be/lHLpKzUxjGk
- formerly_proven 3y ago
  > I actually want to explore this further sometime. It's not like it's a secret that buffers are everywhere in computers, but I am still surprised at how many buffers one comes across. Well there's a reason it takes a click or key press to register about as long as it takes sending a packet halfway across the world (or a few times around the world, depending on what software you're running).
  - ryandrake 3y ago
    It must have been a real blast back in the 8-bit days, when you'd have some peripheral's port mapped to a specific address in the memory map, and setting a byte to that address actually slammed the electrons into that peripheral's register. No wasteful 15 layers of OS abstractions waiting for buffers to fill and copying bytes from buffer to buffer all the way down.
jchw 3y ago
Oh yeah, that I can't deny. The peak throughput numbers used in marketing are definitely questionable at times. Kinda makes sense considering SSDs and HDDs are basically computers of their own that give you some abstraction over the actual raw storage media. You could compare memcpy to sequential write speed or something, but it'd definitely not tell the full story.

Palomides 3y ago

SSDs, even nvme, are indeed about 1000x more latency tham RAM normal RAM only has like 10x the bandwidth of a high end SSD, though

cratermoon 3y ago

> the war stories that PostgreSQL developers could tell about disk syncing. The man page for glibc open(2) used to say, "Most Linux filesystems don't actually implement the POSIX O_SYNC semantics, which require all metadata updates of a write to be on disk on returning to userspace", but now it notes that "before Linux 2.6.33" it wasn't implemented correctly I remember a rather eye-opening discussion on either the postgresql-hackers or lkml about the semantics of O_SYNC not actually being synchronous. In addition to the implementation details, which depended on the version of the kernel and the underlying filesystem type, it was also noted that underlying disk hardware also had caches which did not necessarily behave synchronously. Some places I can find that cover these details include https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg272253.html https://www.mail-archive.com/linux-kernel@vger.kernel.org/ms... http://milek.blogspot.com/2010/12/linux-osync-and-write-barriers.html http://milek.blogspot.com/2010/12/linux-osync-and-write-barr... https://lwn.net/Articles/352229/ https://lwn.net/Articles/352229/

sroussey 3y ago
RAID cards back in the day, and some enterprise SSDs today either had batteries or capacitors in order to finish the write, even if the power turned off. They tell the OS that the write has synced even when it has not as it has confidence that it will.

foobiekr 3y ago

Not only are they 1000x slower than memory, if you bump into thermal management, much slower than that. Worse, if you run into wear leveling write amplification, orders of magnitude slower. Yes, you can use caching to hide this, in main memory and on the device (safely and unsafely, depending on the design and whether it is an enterprise drive with sufficient capacitors), but you can do the same with SRAM vs. DRAM and it doesn't make DRAM as fast as SRAM, or registers-vs-SRAM, likewise.