28 ms·

Author here: This article was intended to explain some modern hardware bottlenecks (and non-bottlenecks), but unexpectedly ended up covering a bunch of Linux ke

Author here: This article was intended to explain some modern hardware bottlenecks (and non-bottlenecks), but unexpectedly ended up covering a bunch of Linux kernel I/O stack issues as well :-) AMA

PragmaticPulp 6y ago
This is a great article. Thanks for writing it up and sharing.
jeffbee 6y ago
Great article, I learned! Can you tell me if you looked into aspects of the NVMe device itself, such as whether it supports 4K logical blocks instead of 512B? Use `nvme id-ns` to read out the supported logical block formats.
- tanelpoder 6y ago
  Doesn't seem to support 4k out of the box? Some drives - like Intel Optane SSDs allow changing this in firmware (and reformatting) with a manufacturer's utility... $ lsblk -t /dev/nvme0n1 NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE RA WSAME nvme0n1 0 512 0 512 512 0 none 1023 128 0B $ sudo nvme id-ns -H /dev/nvme0n1 | grep Size LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)
  - guerby 6y ago
    Here is an article about nvme-cli tool : https://nvmexpress.org/open-source-nvme-management-utility-nvme-command-line-interface-nvme-cli/ https://nvmexpress.org/open-source-nvme-management-utility-n... On Samsung SSD 970 EVO 1TB it seems only 512 bytes LBA are supported: # nvme id-ns /dev/nvme0n1 -n 1 -H|grep "^LBA Format" LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)
  - jeffbee 6y ago
    Thanks for checking. SSD review sites never mention this important detail. For some reason the Samsung datacenter SSDs support 4K LBA format, and they are very similar to the retail SSDs which don't seem to. I have the a retail 970 Evo that only provides 512.
    - wtallis 6y ago
      I just checked my logs, and none of Samsung's consumer NVMe drives have ever supported sector sizes other than 512B. They seem to view this feature as part of their product segmentation strategy. Some consumer SSD vendors do enable 4kB LBA support. I've seen it supported on consumer drives from WD, SK hynix and a variety of brands using Phison or SMI SSD controllers (including Kingston, Seagate, Corsair, Sabrent). But I haven't systematically checked to see which brands consistently support it.
      - 1996 6y ago
        Is it genuine 512? As in, what ashift value do you use with zfs?
        wtallis 6y ago
        Regardless of what sector size you configure the SSD to expose, the drive's flash translation layer still manages logical to physical mappings at a 4kB granularity, the underlying media page size is usually on the order of 16kB, and the erase block size is several MB. So what ashift value you want to use depends very much on what kind of tradeoffs you're okay with in terms of different aspects of performance and write endurance/write amplification. But for most flash-based SSDs, there's no reason to set ashift to anything less than 12 (corresponding to 4kB blocks).
        1996 6y ago
        > for most flash-based SSDs, there's no reason to set ashift to anything less than 12 (corresponding to 4kB blocks). matching the page size? > the underlying media page size is usually on the order of 16kB I'd say that's a good reason to set ashift=14 as 2^14=16kb
guerby 6y ago
71 GB/s is 568 Gbit/s so you'll need about 3 dual 100 Gbit/s cards to pump data out at the rate you can read it from the NVMe drives. And ethernet (unless LAN jumbo frames) is about 1.5kByte per frame (not 4kB). One such PC should be able to do 100k simultaneous 5 Mbps HD streams. Testing this would be fun :)
- tanelpoder 6y ago
  I should (finally) receive my RTX 3090 card today (PCIe 4.0 too!), I guess here goes my weekend (and the following weekends over a couple of years)!
- zamadatix 6y ago
  Mellanox has a 2x200 Gbps NIC these days. Haven't gotten to play with it yet though.
  - tanelpoder 6y ago
    Which NICs would you recommend for me to buy for testing at least 1x100 Gbps (ideally 200 Gbps?) networking between this machine (PCIe 4.0) and an Intel Xeon one that I have with PCIe 3.0. Don't want to spend much money, so the cards don't need to be too enterprisey, just fast. And - do such cards even allow direct "cross" connection without a switch in between?
    - drewg123 6y ago
      All 100G is enterprisy. For a cheap solution, I'd get a pair of used Mellanox ConnectX4 or Chelsio T6, and a QSFP28 direct attach copper cable.
    - zamadatix 6y ago
      +1 on what the sibling comment said. As for directly connecting them absolutely, works great. Id recommend a cheap DAC off fs.com to connect them in that case.
    - namibj 6y ago
KaiserPro 6y ago
Excellent write up. I used to work for a VFX company in 2008. At that point we used lustre to get high throughput file storage. From memory we had something like 20 racks of server/disks to get a 3-6 gigabyte/s (sustained) throughput on a 300tb filesystem. It is hilarious to think that a 2u box can now theoretically saturate 2x100gig nics.
vinay_ys 6y ago
Very cool rig and benchmark. Kudos. Request: add network io load to your benchmarking load while nvme io load is running.
- tanelpoder 6y ago
  Thanks, will do in a future article! I could share the disks out via NFS or iSCSI or something and hammer them from a remote machine...
sitkack 6y ago
Could you explain some of your thought processes and methodologies when approaching problems like this? What is your mental model like? How much experimentation do you do verses reading kernel code? How do you know what questions to start asking? *edit, btw I understand that a response to these questions could be an entire book, you get the question-space.
- tanelpoder 6y ago
  Good question. I don't ever read kernel code as a starting point, only if some profiling or tracing tool points me towards an interesting function or codepath. And interesting usually is something that takes most CPU in perf output or some function call with an unusually high latency in ftrace, bcc/bpftrace script output. Or just a stack trace in a core- or crashdump. As far as mindset goes - I try to apply the developer mindset to system performance. In other words, I don't use much of what I call the "old school sysadmin mindset", from a time where better tooling was not available. I don't use systemwide utilization or various get/hit ratios for doing "metric voodoo" of Unix wizards. The developer mindset dictates that everything you run is an application. JVM is an application. Kernel is an application. Postgres, Oracle are applications. All applications execute one or more threads that run on CPU or do not run on CPU. There are only two categories of reasons why a thread does not run on CPU (is sleeping): The OS put the thread to sleep (involuntary blocking) or the thread voluntarily wanted to go to go to sleep (for example, it realized it can't get some application level lock). And you drill down from there. Your OS/system is just a bunch of threads running on CPU, sleeping and sometimes communicating with each other. You can directly measure all of these things easily nowadays with profilers, no need for metric voodoo. I have written my own tools to complement things like perf, ftrace and BPF stuff - as a consultant I regularly see 10+ year old Linux versions, etc - and I find sampling thread states from /proc file system is a really good (and flexible) starting point for system performance analysis and even some drilldown - all this without having to install new software or upgrading to latest kernels. Some of the tools I showed in my article too: https://tanelpoder.com/psnapper https://tanelpoder.com/psnapper & https://0x.tools https://0x.tools In the end of my post I mentioned that I'll do a webinar "hacking session" next Thursday, I'll show more how I work there :-)
rafaelturk 6y ago
Thanks for well written article, makes me think about inefficiencies in our over-hyped cloud environment.
- tanelpoder 6y ago
  Oh yes - and incorrectly configured on-premises systems too!
tarasglek 6y ago
You should look at cpu usage. There is a good chance all your interrupts are hitting cpu-0. you can run hwloc to see what chiplet the pci cards are on and handle interrupts on those cores.
- jeffbee 6y ago
  Why would that happen with the linux nvme stack that puts a completion queue on each CPU?
  - wtallis 6y ago
    I think that in addition to allocating a queue per CPU, you need to be able to allocate a MSI(-X) vector per CPU. That shouldn't be a problem for the Samsung 980 PRO, since it supports 128 queues and 130 interrupt vectors.
- tanelpoder 6y ago
  Thanks for the "hwloc" tip. I hadn't thought about that. I was thinking of doing something like that. Weirdly I got sustained throughput differences when I killed & restarted fio. So, if I got 11M IOPS, it stayed at that level until I killed fio & restarted. If I got 10.8M next, it stayed like it until I killed & restarted it. This makes me think that I'm hitting some PCIe/memory bottleneck, dependent on process placement (which process happens to need to move data across infinity fabric due to accessing data through a "remote" PCIe root complex or something like that). But then I realized that Zen 2 has a central IO hub again, so there shouldn't be a "far edge of I/O" like on current gen Intel CPUs (?) But there's definitely some workload placement and I/O-memory-interrupt affinity that I've wanted to look into. I could even enable the NUMA-like-mode from BIOS, but again with Zen 2, the memory access goes through the central infinity-fabric chip too, I understand, so not sure if there's any value in trying to achieve memory locality for individual chiplets on this platform (?)
  - wtallis 6y ago
    The PCIe is all on a single IO die, but internally it is organized into quadrants that can produce some NUMA effects. So it is probably worth trying out the motherboard firmware settings to expose your CPU as multiple NUMA nodes, and using the FIO options to allocate memory only on the local node, and restricting execution to the right cores.
ksec 6y ago
I just love this article. Especially when the norm is always about scaling out instead of scaling up. We can have 128 Core CPU, 2TB Memory, PCI-E 4.0 SSD, ( and soon PCI-E 5.0 ). We could even fit a Petabyte in 1U for SSD Storage. I remember WhatsApp used to operate its 500M user with only a dozen of large FreeBSD boxes. ( Only to be taken apart by Facebook ) So Thank you for raising awareness. Hopefully the pendulum is swinging back to conceptually simple design. >I also have a 380 GB Intel Optane 905P SSD for low latency writes I would love to see that. Although I am waiting for someone to do a review on the Optane SSD P5800X [1]. Random 4K IOPS up to 1.5M with lower than 6us Latency. [1] https://www.servethehome.com/new-intel-optane-p5800x-100-dwpd-ssd-dominates-pcie-gen4/ https://www.servethehome.com/new-intel-optane-p5800x-100-dwp...
- texasbigdata 6y ago
  Second on Optane.
- phkahler 6y ago
  >> I remember WhatsApp used to operate its 500M user with only a dozen of large FreeBSD boxes. With 1TB of RAM you can have 256 bytes for every person on earth live in memory. With SSD either as virtual memory or keeping an index in RAM, you can do meaningful work in real time, probably as fast as the network will allow.
  - swader999 6y ago
    Faster then they can type!
  - zie 6y ago
    My math doesn't compute with yours: depending on how you define a TB(memory tends to favour the latter definition, but YMMV): 1,000,000,000,000 / 7.8billion = 128.21 bytes per human. 1,099,511,627,776 / 7.8billion = 140.96 bytes per human. population source via Wikipedia.
- rektide 6y ago
  Intel killing off prosumer optane 2 weeks ago[1] made me so so so sad. The new P5800X should be sick. [1] https://news.ycombinator.com/item?id=25805779 https://news.ycombinator.com/item?id=25805779
maerF0x0 6y ago
When I first moved to the bay area, the company that hired me asked me what kind of computer I wanted and gave me a budget (like $3000 or something)... I spent a few days crafting a parts list so I could build an awesome workstation. Once I sent it over they were like "Uh, we just meant which macbook do you want?" and kind of gave me some shade about it. They joked, so how are you going to do meetings or on call? I rolled with it, but really wondered if they knew I could get 2x the hardware and have a computer at home and at work for less money than the MBP ... Most of the people didnt seem to understand that laptop CPUs are not the same as desktop/workstation ones, especially when they hit thermal down throttling.
- walrus01 6y ago
  I mean it would have been a totally valid answer to say that you intended to use a $600 laptop as effectively a thin client, and spend $2400 on a powerful workstation PC to drive remotely.
- noir_lord 6y ago
  Last but one job boss offered me an iMac Pro, I asked if I could just have the equivalent money for hardware and he said sure. Which is how I ended up with an absolute monster of a work machine, these days I WFH and while work issued me a Macbook Pro it sits on the shelf behind me. Fedora on a (still fast) Ryzen/2080 and 2x4K 27" screens vs a Macbook Pro is a hilarious no brainer for me. Upgrading soon but can't decide whether I need the 5950X or merely want it - realistically except for gaming I'm nowhere near tapping out this machine (and it's still awesome for that an VR which is why the step-son is about to get a in his words "sick" PC).
  - technofiend 6y ago
    Were you forced to make it a hackintosh to run company-required software or did you end up with a linux build?
    - noir_lord 6y ago
      Linux (Fedora) I was only on-site dev and everything was linux anyway.
antongribok 6y ago
Great article! Any chance you could post somewhere the output of: lstopo --of ascii Or similar?
- tanelpoder 6y ago
  I can do it tomorrow, please drop me an email (email listed in my blog)
perryizgr8 6y ago
It would be interesting to know what you intend to use this rig for, if that is not some secret :)
- tanelpoder 6y ago
  Valid question! 1) Learning & researching capabilities of modern HW 2) Running RDBMS stress tests (until breaking point), Oracle, Postgres+TimescaleDB, MySQL, probably ScyllaDB soon too 3) Why? As a performance troubleshooter consultant+trainer, I regularly have to reproduce complex problems that show up only under high concurrency & load - stuff that you can't just reproduce in a VM in a laptop. 4) Fun - seeing if the "next gen" hardware's promised performance is actually possible! FYI I have some videos from my past complex problem troubleshooting adventures, mostly Oracle stuff so far and some Linux performance troubleshooting: https://tanelpoder.com/videos/ https://tanelpoder.com/videos/
nicioan 6y ago
Excellent article, thank you! I really like the analysis and profiling part of the evaluation. I also have some experience in I/O performance in linux -- we measured 30GiB/s in a pcie Gen3 box (shameless plug[0]). I have one question / comment: did you use multiple jobs for the BW (large IO) experiments? If yes, then did you set randrepeat to 0? I'm asking this because fio by default uses the same sequence of offsets for each job, in which case there might be data re-used across jobs. I had verified that with blktrace a few years back, but it might have changed recently. [0]https://www.usenix.org/conference/fast19/presentation/kourtis https://www.usenix.org/conference/fast19/presentation/kourti... edit: fixed typo
- tanelpoder 6y ago
  Looks interesting! I wonder whether there'd be interesting new database applications on NVMe when doing as small as 512 byte I/Os (with more efficient "IO engine" than Linux bio, that has too high CPU overhead with such small requests). I mean, currently OLTP RDBMS engines tend to use 4k, 8k (and some) 16k block size and when doing completely random I/O (or, say traversing an index on customer_id that now needs to read random occasional customer orders across years of history). So you may end up reading 1000 x 8 kB blocks just to read 1000 x 100B order records "randomly" scattered across the table from inserts done over the years. Optane persistent memory can do small, cache line sized I/O I understand, but that's a different topic. When being able to do random 512B I/O on "commodity" NVMe SSDs efficiently, this would open some interesting opportunities for retrieving records that are scattered "randomly" across the disks. edit: to answer your question, I used 10 separate fio commands with numjobs=3 or 4 for each and randrepeat was set to default.

Author here: This article was intended to explain some modern hardware bottlenecks (and non-bottlenecks), but unexpectedly ended up covering a bunch of Linux kernel I/O stack issues as well :-) AMA

PragmaticPulp 6y ago

This is a great article. Thanks for writing it up and sharing.

jeffbee 6y ago

Great article, I learned! Can you tell me if you looked into aspects of the NVMe device itself, such as whether it supports 4K logical blocks instead of 512B? Use `nvme id-ns` to read out the supported logical block formats.

tanelpoder 6y ago
Doesn't seem to support 4k out of the box? Some drives - like Intel Optane SSDs allow changing this in firmware (and reformatting) with a manufacturer's utility... $ lsblk -t /dev/nvme0n1 NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE RA WSAME nvme0n1 0 512 0 512 512 0 none 1023 128 0B $ sudo nvme id-ns -H /dev/nvme0n1 | grep Size LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)
- guerby 6y ago
  Here is an article about nvme-cli tool : https://nvmexpress.org/open-source-nvme-management-utility-nvme-command-line-interface-nvme-cli/ https://nvmexpress.org/open-source-nvme-management-utility-n... On Samsung SSD 970 EVO 1TB it seems only 512 bytes LBA are supported: # nvme id-ns /dev/nvme0n1 -n 1 -H|grep "^LBA Format" LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)
- jeffbee 6y ago
  Thanks for checking. SSD review sites never mention this important detail. For some reason the Samsung datacenter SSDs support 4K LBA format, and they are very similar to the retail SSDs which don't seem to. I have the a retail 970 Evo that only provides 512.
  - wtallis 6y ago
    I just checked my logs, and none of Samsung's consumer NVMe drives have ever supported sector sizes other than 512B. They seem to view this feature as part of their product segmentation strategy. Some consumer SSD vendors do enable 4kB LBA support. I've seen it supported on consumer drives from WD, SK hynix and a variety of brands using Phison or SMI SSD controllers (including Kingston, Seagate, Corsair, Sabrent). But I haven't systematically checked to see which brands consistently support it.
    - 1996 6y ago
      Is it genuine 512? As in, what ashift value do you use with zfs?
      - wtallis 6y ago
        Regardless of what sector size you configure the SSD to expose, the drive's flash translation layer still manages logical to physical mappings at a 4kB granularity, the underlying media page size is usually on the order of 16kB, and the erase block size is several MB. So what ashift value you want to use depends very much on what kind of tradeoffs you're okay with in terms of different aspects of performance and write endurance/write amplification. But for most flash-based SSDs, there's no reason to set ashift to anything less than 12 (corresponding to 4kB blocks).
        1996 6y ago
        > for most flash-based SSDs, there's no reason to set ashift to anything less than 12 (corresponding to 4kB blocks). matching the page size? > the underlying media page size is usually on the order of 16kB I'd say that's a good reason to set ashift=14 as 2^14=16kb

guerby 6y ago

71 GB/s is 568 Gbit/s so you'll need about 3 dual 100 Gbit/s cards to pump data out at the rate you can read it from the NVMe drives. And ethernet (unless LAN jumbo frames) is about 1.5kByte per frame (not 4kB). One such PC should be able to do 100k simultaneous 5 Mbps HD streams. Testing this would be fun :)

tanelpoder 6y ago
I should (finally) receive my RTX 3090 card today (PCIe 4.0 too!), I guess here goes my weekend (and the following weekends over a couple of years)!
zamadatix 6y ago
Mellanox has a 2x200 Gbps NIC these days. Haven't gotten to play with it yet though.
- tanelpoder 6y ago
  Which NICs would you recommend for me to buy for testing at least 1x100 Gbps (ideally 200 Gbps?) networking between this machine (PCIe 4.0) and an Intel Xeon one that I have with PCIe 3.0. Don't want to spend much money, so the cards don't need to be too enterprisey, just fast. And - do such cards even allow direct "cross" connection without a switch in between?
  - drewg123 6y ago
    All 100G is enterprisy. For a cheap solution, I'd get a pair of used Mellanox ConnectX4 or Chelsio T6, and a QSFP28 direct attach copper cable.
  - zamadatix 6y ago
    +1 on what the sibling comment said. As for directly connecting them absolutely, works great. Id recommend a cheap DAC off fs.com to connect them in that case.
  - namibj 6y ago

KaiserPro 6y ago

Excellent write up. I used to work for a VFX company in 2008. At that point we used lustre to get high throughput file storage. From memory we had something like 20 racks of server/disks to get a 3-6 gigabyte/s (sustained) throughput on a 300tb filesystem. It is hilarious to think that a 2u box can now theoretically saturate 2x100gig nics.

vinay_ys 6y ago

Very cool rig and benchmark. Kudos. Request: add network io load to your benchmarking load while nvme io load is running.

tanelpoder 6y ago
Thanks, will do in a future article! I could share the disks out via NFS or iSCSI or something and hammer them from a remote machine...

sitkack 6y ago

Could you explain some of your thought processes and methodologies when approaching problems like this? What is your mental model like? How much experimentation do you do verses reading kernel code? How do you know what questions to start asking? *edit, btw I understand that a response to these questions could be an entire book, you get the question-space.

tanelpoder 6y ago
Good question. I don't ever read kernel code as a starting point, only if some profiling or tracing tool points me towards an interesting function or codepath. And interesting usually is something that takes most CPU in perf output or some function call with an unusually high latency in ftrace, bcc/bpftrace script output. Or just a stack trace in a core- or crashdump. As far as mindset goes - I try to apply the developer mindset to system performance. In other words, I don't use much of what I call the "old school sysadmin mindset", from a time where better tooling was not available. I don't use systemwide utilization or various get/hit ratios for doing "metric voodoo" of Unix wizards. The developer mindset dictates that everything you run is an application. JVM is an application. Kernel is an application. Postgres, Oracle are applications. All applications execute one or more threads that run on CPU or do not run on CPU. There are only two categories of reasons why a thread does not run on CPU (is sleeping): The OS put the thread to sleep (involuntary blocking) or the thread voluntarily wanted to go to go to sleep (for example, it realized it can't get some application level lock). And you drill down from there. Your OS/system is just a bunch of threads running on CPU, sleeping and sometimes communicating with each other. You can directly measure all of these things easily nowadays with profilers, no need for metric voodoo. I have written my own tools to complement things like perf, ftrace and BPF stuff - as a consultant I regularly see 10+ year old Linux versions, etc - and I find sampling thread states from /proc file system is a really good (and flexible) starting point for system performance analysis and even some drilldown - all this without having to install new software or upgrading to latest kernels. Some of the tools I showed in my article too: https://tanelpoder.com/psnapper https://tanelpoder.com/psnapper & https://0x.tools https://0x.tools In the end of my post I mentioned that I'll do a webinar "hacking session" next Thursday, I'll show more how I work there :-)

rafaelturk 6y ago

Thanks for well written article, makes me think about inefficiencies in our over-hyped cloud environment.

tanelpoder 6y ago
Oh yes - and incorrectly configured on-premises systems too!

tarasglek 6y ago

You should look at cpu usage. There is a good chance all your interrupts are hitting cpu-0. you can run hwloc to see what chiplet the pci cards are on and handle interrupts on those cores.

jeffbee 6y ago
Why would that happen with the linux nvme stack that puts a completion queue on each CPU?
- wtallis 6y ago
  I think that in addition to allocating a queue per CPU, you need to be able to allocate a MSI(-X) vector per CPU. That shouldn't be a problem for the Samsung 980 PRO, since it supports 128 queues and 130 interrupt vectors.
tanelpoder 6y ago
Thanks for the "hwloc" tip. I hadn't thought about that. I was thinking of doing something like that. Weirdly I got sustained throughput differences when I killed & restarted fio. So, if I got 11M IOPS, it stayed at that level until I killed fio & restarted. If I got 10.8M next, it stayed like it until I killed & restarted it. This makes me think that I'm hitting some PCIe/memory bottleneck, dependent on process placement (which process happens to need to move data across infinity fabric due to accessing data through a "remote" PCIe root complex or something like that). But then I realized that Zen 2 has a central IO hub again, so there shouldn't be a "far edge of I/O" like on current gen Intel CPUs (?) But there's definitely some workload placement and I/O-memory-interrupt affinity that I've wanted to look into. I could even enable the NUMA-like-mode from BIOS, but again with Zen 2, the memory access goes through the central infinity-fabric chip too, I understand, so not sure if there's any value in trying to achieve memory locality for individual chiplets on this platform (?)
- wtallis 6y ago
  The PCIe is all on a single IO die, but internally it is organized into quadrants that can produce some NUMA effects. So it is probably worth trying out the motherboard firmware settings to expose your CPU as multiple NUMA nodes, and using the FIO options to allocate memory only on the local node, and restricting execution to the right cores.

ksec 6y ago

I just love this article. Especially when the norm is always about scaling out instead of scaling up. We can have 128 Core CPU, 2TB Memory, PCI-E 4.0 SSD, ( and soon PCI-E 5.0 ). We could even fit a Petabyte in 1U for SSD Storage. I remember WhatsApp used to operate its 500M user with only a dozen of large FreeBSD boxes. ( Only to be taken apart by Facebook ) So Thank you for raising awareness. Hopefully the pendulum is swinging back to conceptually simple design. >I also have a 380 GB Intel Optane 905P SSD for low latency writes I would love to see that. Although I am waiting for someone to do a review on the Optane SSD P5800X [1]. Random 4K IOPS up to 1.5M with lower than 6us Latency. [1] https://www.servethehome.com/new-intel-optane-p5800x-100-dwpd-ssd-dominates-pcie-gen4/ https://www.servethehome.com/new-intel-optane-p5800x-100-dwp...

texasbigdata 6y ago
Second on Optane.
phkahler 6y ago
>> I remember WhatsApp used to operate its 500M user with only a dozen of large FreeBSD boxes. With 1TB of RAM you can have 256 bytes for every person on earth live in memory. With SSD either as virtual memory or keeping an index in RAM, you can do meaningful work in real time, probably as fast as the network will allow.
- swader999 6y ago
  Faster then they can type!
- zie 6y ago
  My math doesn't compute with yours: depending on how you define a TB(memory tends to favour the latter definition, but YMMV): 1,000,000,000,000 / 7.8billion = 128.21 bytes per human. 1,099,511,627,776 / 7.8billion = 140.96 bytes per human. population source via Wikipedia.
rektide 6y ago
Intel killing off prosumer optane 2 weeks ago[1] made me so so so sad. The new P5800X should be sick. [1] https://news.ycombinator.com/item?id=25805779 https://news.ycombinator.com/item?id=25805779

maerF0x0 6y ago

When I first moved to the bay area, the company that hired me asked me what kind of computer I wanted and gave me a budget (like $3000 or something)... I spent a few days crafting a parts list so I could build an awesome workstation. Once I sent it over they were like "Uh, we just meant which macbook do you want?" and kind of gave me some shade about it. They joked, so how are you going to do meetings or on call? I rolled with it, but really wondered if they knew I could get 2x the hardware and have a computer at home and at work for less money than the MBP ... Most of the people didnt seem to understand that laptop CPUs are not the same as desktop/workstation ones, especially when they hit thermal down throttling.

walrus01 6y ago
I mean it would have been a totally valid answer to say that you intended to use a $600 laptop as effectively a thin client, and spend $2400 on a powerful workstation PC to drive remotely.
noir_lord 6y ago
Last but one job boss offered me an iMac Pro, I asked if I could just have the equivalent money for hardware and he said sure. Which is how I ended up with an absolute monster of a work machine, these days I WFH and while work issued me a Macbook Pro it sits on the shelf behind me. Fedora on a (still fast) Ryzen/2080 and 2x4K 27" screens vs a Macbook Pro is a hilarious no brainer for me. Upgrading soon but can't decide whether I need the 5950X or merely want it - realistically except for gaming I'm nowhere near tapping out this machine (and it's still awesome for that an VR which is why the step-son is about to get a in his words "sick" PC).
- technofiend 6y ago
  Were you forced to make it a hackintosh to run company-required software or did you end up with a linux build?
  - noir_lord 6y ago
    Linux (Fedora) I was only on-site dev and everything was linux anyway.

antongribok 6y ago

Great article! Any chance you could post somewhere the output of: lstopo --of ascii Or similar?

tanelpoder 6y ago
I can do it tomorrow, please drop me an email (email listed in my blog)

perryizgr8 6y ago

It would be interesting to know what you intend to use this rig for, if that is not some secret :)

tanelpoder 6y ago
Valid question! 1) Learning & researching capabilities of modern HW 2) Running RDBMS stress tests (until breaking point), Oracle, Postgres+TimescaleDB, MySQL, probably ScyllaDB soon too 3) Why? As a performance troubleshooter consultant+trainer, I regularly have to reproduce complex problems that show up only under high concurrency & load - stuff that you can't just reproduce in a VM in a laptop. 4) Fun - seeing if the "next gen" hardware's promised performance is actually possible! FYI I have some videos from my past complex problem troubleshooting adventures, mostly Oracle stuff so far and some Linux performance troubleshooting: https://tanelpoder.com/videos/ https://tanelpoder.com/videos/

nicioan 6y ago

Excellent article, thank you! I really like the analysis and profiling part of the evaluation. I also have some experience in I/O performance in linux -- we measured 30GiB/s in a pcie Gen3 box (shameless plug[0]). I have one question / comment: did you use multiple jobs for the BW (large IO) experiments? If yes, then did you set randrepeat to 0? I'm asking this because fio by default uses the same sequence of offsets for each job, in which case there might be data re-used across jobs. I had verified that with blktrace a few years back, but it might have changed recently. [0]https://www.usenix.org/conference/fast19/presentation/kourtis https://www.usenix.org/conference/fast19/presentation/kourti... edit: fixed typo

tanelpoder 6y ago
Looks interesting! I wonder whether there'd be interesting new database applications on NVMe when doing as small as 512 byte I/Os (with more efficient "IO engine" than Linux bio, that has too high CPU overhead with such small requests). I mean, currently OLTP RDBMS engines tend to use 4k, 8k (and some) 16k block size and when doing completely random I/O (or, say traversing an index on customer_id that now needs to read random occasional customer orders across years of history). So you may end up reading 1000 x 8 kB blocks just to read 1000 x 100B order records "randomly" scattered across the table from inserts done over the years. Optane persistent memory can do small, cache line sized I/O I understand, but that's a different topic. When being able to do random 512B I/O on "commodity" NVMe SSDs efficiently, this would open some interesting opportunities for retrieving records that are scattered "randomly" across the disks. edit: to answer your question, I used 10 separate fio commands with numjobs=3 or 4 for each and randrepeat was set to default.