4 ms·

Optimizing on the filesystem is interesting, but the underlying hardware should not be neglected -- and benchmarks should be done before anything else! I recen

Optimizing on the filesystem is interesting, but the underlying hardware should not be neglected -- and benchmarks should be done before anything else! I recently needed a large and fast SQL database. I decided to start by doing some simple tests with dd and some more advanced ones with fio. On a very decent AMD EPYC 7401P 24-Core with 128G of ram and 6 NVMe drives, I found that performance started declining once I added the 6th drive. Based on my benchmarks, the most I can get out of this machine is about 4GB/s in read/write performance. This makes me suspect it is the speed limit from the CPU to the PCH. So instead of this 1 server, I will deploy a cluster of smaller servers each equipped with 2 NVME and hope the network drivers do a decent job (just checked the dmesg: eth2: (PCIe:5.0Gb/s:Width x4)

hamandcheese 9y ago
But how do you distribute your SQL db? Read only followers?
gsich 9y ago
>So instead of this 1 server, I will deploy a cluster of smaller servers each equipped with 2 NVME and hope the network drivers do a decent job (just checked the dmesg: eth2: (PCIe:5.0Gb/s:Width x4) What does that output suggest? Unless you have a 10Gbe card, this won't matter.
- devereaux 9y ago
  The output suggest to me that I should do more benchmarks to avoid another bottleneck, as I seem to be hitting PCIe lanes issues :-) I also want to get my hands on a few intel Optanes to test how well VROC performs in practice. FYI, here are some previous results where I configured partitions of similar sizes, then assembled them like: (...) mdadm --create --verbose /dev/md6 --level=stripe --raid-devices=6 /dev/nvme[0-5]n1p6 mdadm --create --verbose /dev/md5 --level=stripe --raid-devices=5 /dev/nvme[0-4]n1p5 (...) On repeated benchmarks, the best results were consistently obtained with 5 drives: Testing md5 randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16 ... fio-2.1.11 Starting 8 processes Jobs: 8 (f=8): [w(8)] [100.0% done] [0KB/4397MB/0KB /s] [0/1126K/0 iops] [eta 00m:00s] randwrite: (groupid=0, jobs=8): err= 0: pid=76554: Mon Feb 12 01:47:56 2018 write: io=966497MB, bw=4027.4MB/s, iops=1030.1K, runt=240002msec slat (usec): min=1, max=3852, avg= 3.45, stdev= 1.49 clat (usec): min=0, max=17533, avg=119.88, stdev=258.75 lat (usec): min=10, max=17535, avg=123.40, stdev=258.76 clat percentiles (usec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 20], | 30.00th=[ 27], 40.00th=[ 36], 50.00th=[ 50], 60.00th=[ 71], | 70.00th=[ 106], 80.00th=[ 167], 90.00th=[ 270], 95.00th=[ 366], | 99.00th=[ 1020], 99.50th=[ 1992], 99.90th=[ 3536], 99.95th=[ 3952], | 99.99th=[ 4960] bw (KB /s): min=402136, max=748520, per=12.50%, avg=515423.89, stdev=44403.70 lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=19.43%, 50=30.18% lat (usec) : 100=18.95%, 250=19.84%, 500=9.14%, 750=1.12%, 1000=0.32% lat (msec) : 2=0.52%, 4=0.45%, 10=0.05%, 20=0.01% cpu : usr=17.59%, sys=50.20%, ctx=50843447, majf=0, minf=71 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=247423104/d=0, short=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: io=966497MB, aggrb=4027.4MB/s, minb=4027.4MB/s, maxb=4027.4MB/s, mint=240002msec, ma xt=240002msec Disk stats (read/write): md5: ios=0/247193645, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/49484620, a ggrmerge=0/0, aggrticks=0/5476932, aggrin_queue=5571618, aggrutil=90.41% nvme0n1: ios=0/49483811, merge=0/0, ticks=0/5518696, in_queue=5613760, util=90.41% nvme4n1: ios=0/49480722, merge=0/0, ticks=0/5573820, in_queue=5663004, util=90.29% nvme1n1: ios=0/49485862, merge=0/0, ticks=0/5390388, in_queue=5483560, util=90.07% nvme2n1: ios=0/49488566, merge=0/0, ticks=0/5395484, in_queue=5496916, util=90.12% nvme3n1: ios=0/49484143, merge=0/0, ticks=0/5506276, in_queue=5600852, util=90.30%
  - praseodym 9y ago
    4027.4MB/s (~32Gbps) with 100GB of random writes isn’t bad at all. Maybe you’re just maxing out the SSDs themselves, not the platform around it?
    - devereaux 9y ago
      Unfortunately, with md6 (using 6 drives) the performance start degrading. It is perfectly reproductible: (...) Run status group 0 (all jobs): WRITE: io=963949MB, aggrb=4016.5MB/s, minb=4016.5MB/s, maxb=4016.5MB/s, mint=240001msec, maxt=240001msec Disk stats (read/write): md6: ios=0/246593949, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/41128473, aggrmerge=0/0, aggrticks=0/4307426, aggrin_queue=4365219, aggrutil=89.21% nvme2n1: ios=0/41137801, merge=0/0, ticks=0/2860992, in_queue=2909124, util=81.31% nvme3n1: ios=0/41125375, merge=0/0, ticks=0/2861640, in_queue=2896800, util=80.94% nvme0n1: ios=0/41136785, merge=0/0, ticks=0/7713176, in_queue=7809532, util=89.21% nvme4n1: ios=0/41142237, merge=0/0, ticks=0/3141888, in_queue=3185880, util=81.73% nvme1n1: ios=0/41116666, merge=0/0, ticks=0/6495928, in_queue=6578204, util=87.12% nvme5n1: ios=0/41111977, merge=0/0, ticks=0/2770936, in_queue=2811776, util=81.20% I would welcome any idea. Tell me your email if you want to play with the machine (there's nothing there yet, I was just setting up md5 for the next step: pgsql benchmarks)
      - user5994461 9y ago
        It's not degrading, it's the same. You've reached a ceiling on something. Try increasing the queue from 16 to 64.
        devereaux 9y ago
        Just tried changing a few things with a different benchmark after reading http://middoraid.blogspot.com/2013/01/tweaking.html http://middoraid.blogspot.com/2013/01/tweaking.html, no change.
        user5994461 9y ago
Thaxll 9y ago
If you go from one big server to a many smaller it's a different kind of problem ( sharding, ha, replication ect ... )
praseodym 9y ago
The AMD Epyc 7401P you’re using has 128 PCIe 3.0 lanes, which can provide way more bandwidth than the 4Gbps figure you’re getting (8 Gbps per lane, so 1024 Gbps on paper). The AMD Epyc platform has been benchmarked at 57 GBps (bytes, not bits) of random storage bandwidth: http://www.legitreviews.com/one-amd-epyc-processor-reaches-57-gbs-of-random-storage-bandwidth_195653 http://www.legitreviews.com/one-amd-epyc-processor-reaches-5... There’s simply no way a PCIe x4 NIC is going to be able to compete with the bandwidth available within that server.
- devereaux 9y ago
  I just can't get such results at the moment, so I am trying some creative solutions to get over the limit I'm hitting. Would you have some suggestions to diagnose the issue better?
zlynx 9y ago
That speed is surprising to me considering that the EPYC is supposed to have 128 PCIe lanes. I think that you should check the motherboard manual, if you have one, or contact the manufacturer, and see how the board's PCIe is laid out. An EPYC has no need to share PCIe lanes through a PCH and shouldn't have one for the NVMe drives. It's possible that by placing cards in different slots you could gain more speed. There may also be some sort of setting to configure PCIe between 8 16-lane slots (accelerator cards) or 32 4-lane slots (NVMe drives). If you're running a single-threaded application you may be hitting the limits of the DMA and memory controllers.
- devereaux 9y ago
  I am not familiar with the EPYC. I will check that. I posted some benchmarks above using fio -- I think it it properly multithreaded.
saltcured 9y ago
You say 4 Gb/s here but in the thread below show 4GB/s, i.e. 8-10x faster depending on how you think about link speeds. What happens when you test each drive individually? Do they all perform identically or is the 6th device slower? An array will be limited by its slowest member even if there is no bus contention. Also, there are many tunable parameters and I am not sure the defaults are equivalent for RAID5 or RAID6 in the Linux-MD layer. What happens if you use all 6 drives in a RAID5 array? What happens if you use 5 drives in a RAID6 array? Did you try changing the chunk size to match the SSD page size? Are you sure the partitions are all aligned properly to avoid any fragmentation during SSD writes? There are countless variables here...
- devereaux 9y ago
  Sorry for the typo. I will edit the post to show an uppercase B. This is still at least an order of magnitude below the 57 GB/s reported by praseodym This is for a raid0, as I am only concerned with performance here. The chunk size is the default 512K. I can try 4k, but I haven't found the Samsung NVMe SM961 page size. The partitions are of course 4k aligned: Number Start (sector) End (sector) Size Code Name 1 4096 209719295 100.0 GiB FD00 Linux RAID 2 209719296 419434495 100.0 GiB FD00 Linux RAID 3 419434496 629149695 100.0 GiB FD00 Linux RAID 4 629149696 838864895 100.0 GiB FD00 Linux RAID 5 838864896 1048580095 100.0 GiB FD00 Linux RAID 6 1048580096 1258295295 100.0 GiB FD00 Linux RAID 7 1258295296 1468010495 100.0 GiB FD00 Linux RAID 8 1468010496 1677725695 100.0 GiB FD00 Linux RAID 9 1677725696 1875384974 94.3 GiB 8300 Linux filesystem 10 34 4095 2.0 MiB EF02 BIOS boot partition No drive is slower on independant benchmarks. I created the various partitions to also test that scenario by testing various combination of the 6 drives. Performance increases linearly until the 6th drive. If you want to try, give me your email and I'll send you the root password to play with SSH. I was setting md5 for psql tests but there's nothing on the server yet, you can also play as much as you want with /dev/md6 You can also get a AX160-NVMe at Hetzner.de. If you cancel before 14 days, there is no charge.

hamandcheese 9y ago

But how do you distribute your SQL db? Read only followers?

gsich 9y ago

>So instead of this 1 server, I will deploy a cluster of smaller servers each equipped with 2 NVME and hope the network drivers do a decent job (just checked the dmesg: eth2: (PCIe:5.0Gb/s:Width x4) What does that output suggest? Unless you have a 10Gbe card, this won't matter.

devereaux 9y ago
The output suggest to me that I should do more benchmarks to avoid another bottleneck, as I seem to be hitting PCIe lanes issues :-) I also want to get my hands on a few intel Optanes to test how well VROC performs in practice. FYI, here are some previous results where I configured partitions of similar sizes, then assembled them like: (...) mdadm --create --verbose /dev/md6 --level=stripe --raid-devices=6 /dev/nvme[0-5]n1p6 mdadm --create --verbose /dev/md5 --level=stripe --raid-devices=5 /dev/nvme[0-4]n1p5 (...) On repeated benchmarks, the best results were consistently obtained with 5 drives: Testing md5 randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=16 ... fio-2.1.11 Starting 8 processes Jobs: 8 (f=8): [w(8)] [100.0% done] [0KB/4397MB/0KB /s] [0/1126K/0 iops] [eta 00m:00s] randwrite: (groupid=0, jobs=8): err= 0: pid=76554: Mon Feb 12 01:47:56 2018 write: io=966497MB, bw=4027.4MB/s, iops=1030.1K, runt=240002msec slat (usec): min=1, max=3852, avg= 3.45, stdev= 1.49 clat (usec): min=0, max=17533, avg=119.88, stdev=258.75 lat (usec): min=10, max=17535, avg=123.40, stdev=258.76 clat percentiles (usec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 20], | 30.00th=[ 27], 40.00th=[ 36], 50.00th=[ 50], 60.00th=[ 71], | 70.00th=[ 106], 80.00th=[ 167], 90.00th=[ 270], 95.00th=[ 366], | 99.00th=[ 1020], 99.50th=[ 1992], 99.90th=[ 3536], 99.95th=[ 3952], | 99.99th=[ 4960] bw (KB /s): min=402136, max=748520, per=12.50%, avg=515423.89, stdev=44403.70 lat (usec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=19.43%, 50=30.18% lat (usec) : 100=18.95%, 250=19.84%, 500=9.14%, 750=1.12%, 1000=0.32% lat (msec) : 2=0.52%, 4=0.45%, 10=0.05%, 20=0.01% cpu : usr=17.59%, sys=50.20%, ctx=50843447, majf=0, minf=71 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=247423104/d=0, short=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: io=966497MB, aggrb=4027.4MB/s, minb=4027.4MB/s, maxb=4027.4MB/s, mint=240002msec, ma xt=240002msec Disk stats (read/write): md5: ios=0/247193645, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/49484620, a ggrmerge=0/0, aggrticks=0/5476932, aggrin_queue=5571618, aggrutil=90.41% nvme0n1: ios=0/49483811, merge=0/0, ticks=0/5518696, in_queue=5613760, util=90.41% nvme4n1: ios=0/49480722, merge=0/0, ticks=0/5573820, in_queue=5663004, util=90.29% nvme1n1: ios=0/49485862, merge=0/0, ticks=0/5390388, in_queue=5483560, util=90.07% nvme2n1: ios=0/49488566, merge=0/0, ticks=0/5395484, in_queue=5496916, util=90.12% nvme3n1: ios=0/49484143, merge=0/0, ticks=0/5506276, in_queue=5600852, util=90.30%
- praseodym 9y ago
  4027.4MB/s (~32Gbps) with 100GB of random writes isn’t bad at all. Maybe you’re just maxing out the SSDs themselves, not the platform around it?
  - devereaux 9y ago
    Unfortunately, with md6 (using 6 drives) the performance start degrading. It is perfectly reproductible: (...) Run status group 0 (all jobs): WRITE: io=963949MB, aggrb=4016.5MB/s, minb=4016.5MB/s, maxb=4016.5MB/s, mint=240001msec, maxt=240001msec Disk stats (read/write): md6: ios=0/246593949, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/41128473, aggrmerge=0/0, aggrticks=0/4307426, aggrin_queue=4365219, aggrutil=89.21% nvme2n1: ios=0/41137801, merge=0/0, ticks=0/2860992, in_queue=2909124, util=81.31% nvme3n1: ios=0/41125375, merge=0/0, ticks=0/2861640, in_queue=2896800, util=80.94% nvme0n1: ios=0/41136785, merge=0/0, ticks=0/7713176, in_queue=7809532, util=89.21% nvme4n1: ios=0/41142237, merge=0/0, ticks=0/3141888, in_queue=3185880, util=81.73% nvme1n1: ios=0/41116666, merge=0/0, ticks=0/6495928, in_queue=6578204, util=87.12% nvme5n1: ios=0/41111977, merge=0/0, ticks=0/2770936, in_queue=2811776, util=81.20% I would welcome any idea. Tell me your email if you want to play with the machine (there's nothing there yet, I was just setting up md5 for the next step: pgsql benchmarks)
    - user5994461 9y ago
      It's not degrading, it's the same. You've reached a ceiling on something. Try increasing the queue from 16 to 64.
      - devereaux 9y ago
        Just tried changing a few things with a different benchmark after reading http://middoraid.blogspot.com/2013/01/tweaking.html http://middoraid.blogspot.com/2013/01/tweaking.html, no change.
        user5994461 9y ago

Thaxll 9y ago

If you go from one big server to a many smaller it's a different kind of problem ( sharding, ha, replication ect ... )

praseodym 9y ago

The AMD Epyc 7401P you’re using has 128 PCIe 3.0 lanes, which can provide way more bandwidth than the 4Gbps figure you’re getting (8 Gbps per lane, so 1024 Gbps on paper). The AMD Epyc platform has been benchmarked at 57 GBps (bytes, not bits) of random storage bandwidth: http://www.legitreviews.com/one-amd-epyc-processor-reaches-57-gbs-of-random-storage-bandwidth_195653 http://www.legitreviews.com/one-amd-epyc-processor-reaches-5... There’s simply no way a PCIe x4 NIC is going to be able to compete with the bandwidth available within that server.

devereaux 9y ago
I just can't get such results at the moment, so I am trying some creative solutions to get over the limit I'm hitting. Would you have some suggestions to diagnose the issue better?

zlynx 9y ago

That speed is surprising to me considering that the EPYC is supposed to have 128 PCIe lanes. I think that you should check the motherboard manual, if you have one, or contact the manufacturer, and see how the board's PCIe is laid out. An EPYC has no need to share PCIe lanes through a PCH and shouldn't have one for the NVMe drives. It's possible that by placing cards in different slots you could gain more speed. There may also be some sort of setting to configure PCIe between 8 16-lane slots (accelerator cards) or 32 4-lane slots (NVMe drives). If you're running a single-threaded application you may be hitting the limits of the DMA and memory controllers.

devereaux 9y ago
I am not familiar with the EPYC. I will check that. I posted some benchmarks above using fio -- I think it it properly multithreaded.

saltcured 9y ago

You say 4 Gb/s here but in the thread below show 4GB/s, i.e. 8-10x faster depending on how you think about link speeds. What happens when you test each drive individually? Do they all perform identically or is the 6th device slower? An array will be limited by its slowest member even if there is no bus contention. Also, there are many tunable parameters and I am not sure the defaults are equivalent for RAID5 or RAID6 in the Linux-MD layer. What happens if you use all 6 drives in a RAID5 array? What happens if you use 5 drives in a RAID6 array? Did you try changing the chunk size to match the SSD page size? Are you sure the partitions are all aligned properly to avoid any fragmentation during SSD writes? There are countless variables here...

devereaux 9y ago
Sorry for the typo. I will edit the post to show an uppercase B. This is still at least an order of magnitude below the 57 GB/s reported by praseodym This is for a raid0, as I am only concerned with performance here. The chunk size is the default 512K. I can try 4k, but I haven't found the Samsung NVMe SM961 page size. The partitions are of course 4k aligned: Number Start (sector) End (sector) Size Code Name 1 4096 209719295 100.0 GiB FD00 Linux RAID 2 209719296 419434495 100.0 GiB FD00 Linux RAID 3 419434496 629149695 100.0 GiB FD00 Linux RAID 4 629149696 838864895 100.0 GiB FD00 Linux RAID 5 838864896 1048580095 100.0 GiB FD00 Linux RAID 6 1048580096 1258295295 100.0 GiB FD00 Linux RAID 7 1258295296 1468010495 100.0 GiB FD00 Linux RAID 8 1468010496 1677725695 100.0 GiB FD00 Linux RAID 9 1677725696 1875384974 94.3 GiB 8300 Linux filesystem 10 34 4095 2.0 MiB EF02 BIOS boot partition No drive is slower on independant benchmarks. I created the various partitions to also test that scenario by testing various combination of the 6 drives. Performance increases linearly until the 6th drive. If you want to try, give me your email and I'll send you the root password to play with SSH. I was setting md5 for psql tests but there's nothing on the server yet, you can also play as much as you want with /dev/md6 You can also get a AX160-NVMe at Hetzner.de. If you cancel before 14 days, there is no charge.