4 ms·

> there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O I guess most people

by _flux 3y ago

> there are zero benefits in using NVMe/TCP, as he just ends up doing a serial block copy using dd(1) so he's not leveraging concurrent I/O I guess most people don't have faster local network than an SSD can transfer. I wonder though, for those people who do, does a concurrent I/O block device replicator tool exist? Btw, you might want also use pv in the pipeline to see an ETA, although it might have a small impact on performance.

Palomides 3y ago
dd has status=progress to show bytes read/written now, I just use that
crote 3y ago
I doubt it makes a difference. SSDs are an awful lot better at sequential writes than random writes, and concurrent IO would mainly speed up random access. Besides, I don't think anyone really has a local network which is faster than their SSD. Even a 4-year-old consumer Samsung 970 Pro can sustain full-disk writes at 2.000M Byte/s, easily saturating a 10Gbit connection. If we're looking at state-of-the-art consumer tech, the fastest you're getting is a USB4 40Gbit machine-to-machine transfer - but at that point you probably have something like the Crucial T700, which has a sequential write speed of 11.800 MByte/s. The enterprise world probably doesn't look too different. You'd need a 100Gbit NIC to saturate even a single modern SSD, but any machine with such a NIC is more likely to have closer to half a dozen SSDs. At that point you're starting to be more worried about things like memory bandwidth instead. [0] [0]: http://nabstreamingsummit.com/wp-content/uploads/2022/05/2022-Streaming-Summit-Netflix.pdf http://nabstreamingsummit.com/wp-content/uploads/2022/05/202...
- JoeAltmaier 3y ago
  Can't find corroboration for the assertion 'SSDs are an awful lot better at sequential writes than random writes'. Doesn't make sense at first glance. There's no head to move, as in an old-style hard drive. What else could make random write take longer on an SSD?
  - crote 3y ago
    Long story short: changing a bit from 1 to 0 is really easy, but changing it from 0 to 1 is quite difficult and requires an expensive erase operation. The erase works on a full 4k block, so writing a random byte means reading 4k to a buffer, changing one byte, erasing the page, and writing back 4k. Sequential writing means erasing the page once, and writing the bytes in known-pre-erased sections. Any modern (last few decades) SSD has a lot of sauce on top of it to reduce this penalty, but random writes are still a fair bit slower - especially once the buffer of pre-prepared replacement pages runs out. Sequential access is also just a lot easier to predict, so the SSD can do a lot of speculative work to speed it up even more.
  - wtallis 3y ago
    The main problem is that random writes tend to be smaller than the NAND flash erase block size, which is in the range of several MB. You can check literally any SSD benchmark that tests both random and sequential IO. They're both vastly better than a mechanical hard drive, but sequential IO is still faster than random IO.
    - JoeAltmaier 3y ago
      Seems to be the case, to some degree. My SSD is 8% slower doing random writes. I guess your mileage may vary.
  - PeterisP 3y ago
    The key aspect is that such memory generally works on a "block" level so making any smaller-than-block write on a SSD requires reading a whole block (which can be quite large), erasing that whole block and then writing back the whole modified block; as you physically can't toggle a bit without erasing the whole block first. So if large sequential writes mean that you only write full whole blocks, that can be done much faster than writing the same data in random order.
    - wtallis 3y ago
      In practice, flash based SSDs basically never do a full read-modify-write cycle to do an in-place update of an entire erase block. They just write the new data elsewhere and keep track of the fragmentation (consequently, sequential reads of data that wasn't written sequentially may not be as fast as sequential reads of data that was written sequentially). RMW cycles (though not in-place) are common for writes smaller than a NAND page (eg. 16kB) and basically unavoidable for writes smaller than the FTL's 4kB granularity
- wolrah 3y ago
  > Besides, I don't think anyone really has a local network which is faster than their SSD. Even a 4-year-old consumer Samsung 970 Pro can sustain full-disk writes at 2.000M Byte/s, easily saturating a 10Gbit connection. You might be surprised if you take a look at how cheap high speed NICs are on the used market. 25G and 40G can be had for around $50, and 100G around $100. If you need switches things start to get expensive but for the "home lab" crowd since most of these cards are dual port a three-node mesh can be had for just a few hundred bucks. I've had a 40G link to my home server for a few years now mostly just because I could do it for less than the cost of a single hard drive.
  - bpye 3y ago
    Depending on your sensitivity to power/noise a 40Gb switch can be had somewhat inexpensively too - something like the Brocade ICX6610 costs <200$ on eBay.
  - asgrdz 3y ago
    This is exactly what I'm looking for. Would you mind sharing what specific 40G card(s) you're using?
    - wolrah 3y ago
      I'm using Mellanox ConnectX-3 cards, IIRC they're HP branded. They shipped in Infiniband mode and required a small amount of command line fiddling to put them in ethernet mode but it was pretty close to trivial. They're PCIe 3.0 x8 cards so they can't max out both ports, but realistically no one who's considering cheap high speed NICs cares about maxing out more than one port.
- BeeOnRope 3y ago
  In EC2, most of the "storage optimized" instances (which have the largest/fastest SSDs) generally have more advertised network throughput than SSD throughput, by a factor usually in the range of 1 to 2 (though it depends on exactly how you count it, e.g., how you normalize for the full-duplex nature of network speeds and same for SSD).

by _flux 3y ago

Palomides 3y ago

dd has status=progress to show bytes read/written now, I just use that

crote 3y ago

I doubt it makes a difference. SSDs are an awful lot better at sequential writes than random writes, and concurrent IO would mainly speed up random access. Besides, I don't think anyone really has a local network which is faster than their SSD. Even a 4-year-old consumer Samsung 970 Pro can sustain full-disk writes at 2.000M Byte/s, easily saturating a 10Gbit connection. If we're looking at state-of-the-art consumer tech, the fastest you're getting is a USB4 40Gbit machine-to-machine transfer - but at that point you probably have something like the Crucial T700, which has a sequential write speed of 11.800 MByte/s. The enterprise world probably doesn't look too different. You'd need a 100Gbit NIC to saturate even a single modern SSD, but any machine with such a NIC is more likely to have closer to half a dozen SSDs. At that point you're starting to be more worried about things like memory bandwidth instead. [0] [0]: http://nabstreamingsummit.com/wp-content/uploads/2022/05/2022-Streaming-Summit-Netflix.pdf http://nabstreamingsummit.com/wp-content/uploads/2022/05/202...

JoeAltmaier 3y ago
Can't find corroboration for the assertion 'SSDs are an awful lot better at sequential writes than random writes'. Doesn't make sense at first glance. There's no head to move, as in an old-style hard drive. What else could make random write take longer on an SSD?
- crote 3y ago
  Long story short: changing a bit from 1 to 0 is really easy, but changing it from 0 to 1 is quite difficult and requires an expensive erase operation. The erase works on a full 4k block, so writing a random byte means reading 4k to a buffer, changing one byte, erasing the page, and writing back 4k. Sequential writing means erasing the page once, and writing the bytes in known-pre-erased sections. Any modern (last few decades) SSD has a lot of sauce on top of it to reduce this penalty, but random writes are still a fair bit slower - especially once the buffer of pre-prepared replacement pages runs out. Sequential access is also just a lot easier to predict, so the SSD can do a lot of speculative work to speed it up even more.
- wtallis 3y ago
  The main problem is that random writes tend to be smaller than the NAND flash erase block size, which is in the range of several MB. You can check literally any SSD benchmark that tests both random and sequential IO. They're both vastly better than a mechanical hard drive, but sequential IO is still faster than random IO.
  - JoeAltmaier 3y ago
    Seems to be the case, to some degree. My SSD is 8% slower doing random writes. I guess your mileage may vary.
- PeterisP 3y ago
  The key aspect is that such memory generally works on a "block" level so making any smaller-than-block write on a SSD requires reading a whole block (which can be quite large), erasing that whole block and then writing back the whole modified block; as you physically can't toggle a bit without erasing the whole block first. So if large sequential writes mean that you only write full whole blocks, that can be done much faster than writing the same data in random order.
  - wtallis 3y ago
    In practice, flash based SSDs basically never do a full read-modify-write cycle to do an in-place update of an entire erase block. They just write the new data elsewhere and keep track of the fragmentation (consequently, sequential reads of data that wasn't written sequentially may not be as fast as sequential reads of data that was written sequentially). RMW cycles (though not in-place) are common for writes smaller than a NAND page (eg. 16kB) and basically unavoidable for writes smaller than the FTL's 4kB granularity
wolrah 3y ago
> Besides, I don't think anyone really has a local network which is faster than their SSD. Even a 4-year-old consumer Samsung 970 Pro can sustain full-disk writes at 2.000M Byte/s, easily saturating a 10Gbit connection. You might be surprised if you take a look at how cheap high speed NICs are on the used market. 25G and 40G can be had for around $50, and 100G around $100. If you need switches things start to get expensive but for the "home lab" crowd since most of these cards are dual port a three-node mesh can be had for just a few hundred bucks. I've had a 40G link to my home server for a few years now mostly just because I could do it for less than the cost of a single hard drive.
- bpye 3y ago
  Depending on your sensitivity to power/noise a 40Gb switch can be had somewhat inexpensively too - something like the Brocade ICX6610 costs <200$ on eBay.
- asgrdz 3y ago
  This is exactly what I'm looking for. Would you mind sharing what specific 40G card(s) you're using?
  - wolrah 3y ago
    I'm using Mellanox ConnectX-3 cards, IIRC they're HP branded. They shipped in Infiniband mode and required a small amount of command line fiddling to put them in ethernet mode but it was pretty close to trivial. They're PCIe 3.0 x8 cards so they can't max out both ports, but realistically no one who's considering cheap high speed NICs cares about maxing out more than one port.
BeeOnRope 3y ago
In EC2, most of the "storage optimized" instances (which have the largest/fastest SSDs) generally have more advertised network throughput than SSD throughput, by a factor usually in the range of 1 to 2 (though it depends on exactly how you count it, e.g., how you normalize for the full-duplex nature of network speeds and same for SSD).