21 ms·
Achieving 11M IOPS and 66 GB/S IO on a Single ThreadRipper Workstation
- tyingq 6y agoI'm somewhat curious what happens to the long standing 4P/4U servers from companies like Dell and HP. The Ryzen/EPYC has really made going past 2P/2U a more rare need.
- wtallis 6y agoI think at this point the only reasons to go beyond 2U are to make room for either 3.5" hard drives, or GPUs.
- rektide 6y agoWould love to see some very dense blade style ryzen offerings. The 4 2P nodes in 2U is great. Good way to share some power supies, fan, chassis, ideally multi-home nic too. Turn those sleds into blades though, put em on their side, & go even denser. It should be a way to save costs, but density alas is a huge upsell, even though it should be a way to scale costs down.
- tanelpoder 6y agoIndeed, 128 EPYC cores in 2 sockets (with total 16 memory channels) will give a lot of power. I guess it's worth mentioning that the 64-core chips have much lower clock rate than 16/32 core ones though. And with some expensive software that's licensed by CPU core (Oracle), you'd want faster cores, but possibly pay a higher NUMA price when going with a single 4 or 8 sockets machine for your "sacred monolith".
- thinkingkong 6y agoYou might be able to buy a smaller server but the rack density doesnt necessarily change. You still have to worry about cooling and power so lots of DCs would have 1/4 or 1/2 racks.
- tyingq 6y agoSure. I wasn't really thinking of density, just the interesting start of the "death" of 4 socket servers. Being an old-timer, it's interesting to me because "typical database server" has been synonymous with 4P/4U for a long, long time.
- vinay_ys 6y agoI haven't seen a 4 socket machine in a long time.
- StillBored 6y agoThere always seems to be buyers for more exotic high end hardware. That market has been shrinking and expanding, well since the first computer, as mainstream machines become more capable and people discover more uses for large coherent machines. But users of 16 socket machines, will just step down to 4 socket epyc machines with 512 cores (or whatever). And someone else will realize that moving their "web scale" cluster from 5k machines, down to a single machine with 16 sockets results in lower latency and less cost. (or whatever).
- toast0 6y agoAt least when I was actively looking at hardware (2011-2018), 4 socket Xeon was available off the shelf, but at quite the premium over 2 socket Xeon. If your load scaled horizontally, it still made sense to get a 2P Xeon over 2x 1P Xeon, but 2x 2P Xeon was way more cost efficient than a 4P Xeon. 8P or 16P seemed to exist, but maybe only in catalogs. I'm not really in the market anymore, but Epyc looks like 1P is going to solve a lot of needs, and 2P will be available at a reasonable premium, but 4P will probably be out of reach.
- tanelpoder 6y agoAuthor here: This article was intended to explain some modern hardware bottlenecks (and non-bottlenecks), but unexpectedly ended up covering a bunch of Linux kernel I/O stack issues as well :-) AMA
- PragmaticPulp 6y agoThis is a great article. Thanks for writing it up and sharing.
- jeffbee 6y agoGreat article, I learned! Can you tell me if you looked into aspects of the NVMe device itself, such as whether it supports 4K logical blocks instead of 512B? Use `nvme id-ns` to read out the supported logical block formats.
- tanelpoder 6y agoDoesn't seem to support 4k out of the box? Some drives - like Intel Optane SSDs allow changing this in firmware (and reformatting) with a manufacturer's utility... $ lsblk -t /dev/nvme0n1 NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE RA WSAME nvme0n1 0 512 0 512 512 0 none 1023 128 0B $ sudo nvme id-ns -H /dev/nvme0n1 | grep Size LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)
- guerby 6y agoHere is an article about nvme-cli tool : https://nvmexpress.org/open-source-nvme-management-utility-nvme-command-line-interface-nvme-cli/ https://nvmexpress.org/open-source-nvme-management-utility-n... On Samsung SSD 970 EVO 1TB it seems only 512 bytes LBA are supported: # nvme id-ns /dev/nvme0n1 -n 1 -H|grep "^LBA Format" LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)
- jeffbee 6y agoThanks for checking. SSD review sites never mention this important detail. For some reason the Samsung datacenter SSDs support 4K LBA format, and they are very similar to the retail SSDs which don't seem to. I have the a retail 970 Evo that only provides 512.
- tyingq 6y agoI wonder if "huge pages" would make a difference, since some of the bottlenecks seemed to be lock contention on memory pages.
- tanelpoder 6y agoLinux pagecache doesn't use hugepages, but definitely when doing direct I/O into application buffers, would make sense to use hugepages for that. I plan to run tests on various database engines next - and many of them support using hugepages (for shared memory areas at least).
- tyingq 6y agoThanks! Apparently, they did add it for tmpfs, and discussed it for ext4. https://lwn.net/Articles/718102/ https://lwn.net/Articles/718102/
- tanelpoder 6y agoGood point - something to test, once I get to the filesystem benchmarks!
- guerby 6y agoIn the networking world (DPDK) huge pages and static pinning everything is a huge deal as you have very few cpu cycles per network packet.
- tanelpoder 6y agoYep - and there's SPDK for direct NVMe storage access without going through the Linux block layer: https://spdk.io https://spdk.io (it's in my TODO list too)
- pbalcer 6y agoWhat I find interesting about the performance of this type of hardware is how it affects the software we are using for storage. The article talked about how the Linux kernel just can't keep up, but what about databases or kv stores. Are the trade-offs those types of solutions make still valid for this type of hardware? RocksDB, and LSM algorithms in general, seem to be designed with the assumption that random block I/O is slow. It appears that, for modern hardware, that assumption no longer holds, and the software only slows things down [0]. [0] - https://github.com/BLepers/KVell/blob/master/sosp19-final40.pdf https://github.com/BLepers/KVell/blob/master/sosp19-final40....
- ddorian43 6y agoDisappointed there was no lmdb comparison in there.
- tyingq 6y agoA paper on making LSM more SSD friendly: https://users.cs.duke.edu/~rvt/ICDE_2017_CameraReady_427.pdf https://users.cs.duke.edu/~rvt/ICDE_2017_CameraReady_427.pdf
- pbalcer 6y agoThanks for sharing this article - I found it very insightful. I've seen similar ideas being floated around before, and they often seem to focus on what software can be added on top of an already fairly complex solution (while LSM can appear to be conceptually simple, its implementations are anything but). To me, what the original article shows is an opportunity to remove - not add.
- jeffbee 6y agoIf you think about it from the perspective of the authors of large-scale databases, linear access is still a lot cheaper than random access in a datacenter filesystem.
- 1MachineElf 6y agoReminds me of the Solid-State Drive checkbox that VirtualBox has for any VM disks. Checking it will make sure that the VM hardware emulation doesn't wait for the filesystem journal to be written, which would normally be advisable with spinning disks.
- 37ef_ced3 6y agoSomebody please tell me how many ResNet50 inferences you can do per second on one of these chips Here is the standalone AVX-512 ResNet50 code (C99 .h and .c files): https://nn-512.com/browse/ResNet50 https://nn-512.com/browse/ResNet50 Oops, AMD doesn't support AVX-512 yet. Even Zen 3? Incredible
- tanelpoder 6y agoYou mean on the CPU, right? This CPU doesn't support AVX-512: $ grep ^flags /proc/cpuinfo | egrep "avx|sse|popcnt" | sed 's/ /\n/g' | egrep "avx|sse|popcnt" | sort | uniq avx avx2 misalignsse popcnt sse sse2 sse4_1 sse4_2 sse4a ssse3 What compile/build options should I use?
- 37ef_ced3 6y agoNo AVX-512, forget it then
- xxpor 6y agoThey don't have avx512 instructions.
- wyldfire 6y agoWhoa, this code looks interesting. Must've been emitted by something higher-level? Something like PyTorch/TF/MLIR/TVM/Glow maybe? If that is the case, then maybe it could be emitted again while masking the instruction sets Ryzen doesn't support yet.
- wiradikusuma 6y agoI've been thinking about this. Would traditional co-location (e.g. 2x 2U from DELL) in a local data center be cheaper if e.g. you're serving local (country-wise) market?
- derefr 6y agoDepends on how long you need the server, and the ownership model you've chosen to pursue for it. If you purchase a server and stick it in a co-lo somewhere, and your business plans to exist for 10+ years — well, is that server still going to be powering your business 10 years from now? Or will you have moved its workloads to something newer? If so, you'll probably want to decommission and sell the server at some point. The time required to deal with that might not be worth the labor costs of your highly-paid engineers. Which means you might not actually end up re-capturing the depreciated value of the server, but instead will just let it rot on the shelf, or dispose of it as e-waste. Hardware leasing is a lot simpler. Whether you lease servers from an OEM like Dell, there's a quick, well-known path to getting the EOLed hardware shipped back to Dell and the depreciated value paid back out to you. And, of course, hardware renting is simpler still. Renting the hardware of the co-lo (i.e. "bare-metal unmanaged server" hosting plans) means never having to worry about the CapEx of the hardware in the first place. You just walk away at the end of your term. But, of course, that's when you start paying premiums on top of the hardware. Renting VMs, then, is like renting hardware on a micro-scale; you never have to think about what you're running on, as — presuming your workload isn't welded to particular machine features like GPUs or local SSDs — you'll tend to automatically get migrated to newer hypervisor hardware generations as they become available. When you work it out in terms of "ten years of ops-staff labor costs of dealing with generational migrations and sell-offs" vs. "ten years of premiums charged by hosting rentiers", the pricing is surprisingly comparable. (In fact, this is basically the math hosting providers use to figure out what they can charge without scaring away their large enterprise customers, who are fully capable of taking a better deal if there is one.)
- rodgerd 6y ago> If you purchase a server and stick it in a co-lo somewhere, and your business plans to exist for 10+ years — well, is that server still going to be powering your business 10 years from now? Or will you have moved its workloads to something newer? Which, if you have even the remotest fiscal competence, you'll have funded by using the depreciation of the book value of the asset after 3 years.
- qaq 6y agoNow price this in terms of AWS and marvel at the markup
- speedgoose 6y agoI'm afraid Jeff Bezos himself couldn't afford such IOs on AWS.
- whalesalad 6y agoThis post is fantastic. I wish there was more workstation porn like this for those of us who are not into the RGB light show ripjaw hacksaw aorus elite novelty stuff that gamers are so into. Benchmarks in the community are almost universally focused on gaming performance and FPS. I want to build an epic rig that will last a long time with professional grade hardware (with ECC memory for instance) and would love to get a lot of the bleeding-edge stuff without compromising on durability. Where do these people hang out online?
- piinbinary 6y agoThe level1techs forums seems to have a lot of people with similar interests
- zhdc1 6y agoLook at purchasing used enterprise hardware. You can buy a reliable x9 or X10 generation supermicro server (rack or tower) for around a couple of hundred.
- ashkankiani 6y agoI've been planning to do this, but enterprise hardware seems like it requires a completely different set of knowledge on how to purchase it and maintain it, and especially as a consumer. It's not quite as trivial of a barrier to entry as consumer desktops, but I suppose that's the point. Still, it would be nice if there was a guide that could help me make good decisions to start.
- jayonsoftware1 6y agohttps://www.asus.com/us/Motherboard-Accessories/HYPER-M-2-X16-GEN-4-CARD/ https://www.asus.com/us/Motherboard-Accessories/HYPER-M-2-X1... vs https://highpoint-tech.com/USA_new/nvme_raid_controllers.htm https://highpoint-tech.com/USA_new/nvme_raid_controllers.htm . One card is about x10 expensive, but looks like performance is same. Am I missing some thing.
- tanelpoder 6y agoThe ASUS one doesn't have its own RAID controller nor PCIe switch onboard. It relies on the motherboard-provided PCIe bifurcation and if using hardware RAID, it'd use AMD's built-in RAID solution (but I'll use software RAID via Linux dm/md). The HighPoint SSD7500 seems to have a proprietary RAID controller built in to it and some management/monitoring features too (it's the "somewhat enterprisey" version)
- wtallis 6y agoThe HighPoint card doesn't have a hardware RAID controller, just a PCIe switch and an option ROM providing boot support for their software RAID. PCIe switch chips were affordable in the PCIe 2.0 era when multi-GPU gaming setups were popular, but Broadcom decided to price them out of the consumer market for PCIe 3 and later.
- tanelpoder 6y agoOk, thanks, good to know. I misunderstood from their website.
- rektide 6y agopcie switches getting expensive is so the suck.
- deleted 6y ago[deleted]
- RobLach 6y agoExcellent article. Worth a read even if you're not maxing IO.
- anarazel 6y agoHave you checked if using the fio options (--iodepth_batch_*) to batch submissions helps? Fio doesn't do that by default, and I found that that can be a significant benefit. Particularly submitting multiple up requests can amortize the cost of setting the nvme doorbell (the expensive part as far as I understand it) across multiple requests.
- tanelpoder 6y agoI tested various fio options, but didn't notice this one - I'll check it out! It might explain why I still kept seeing lots of interrupts raised even though I had enabled the I/O completion polling instead, with io_uring's --hipri option. edit: I ran a quick test with various IO batch sizes and it didn't make a difference - I guess because thanks to using io_uring, my bottleneck is not in IO submission, but deeper in the block IO stack...
- wtallis 6y agoI think on recent kernels, using the hipri option doesn't get you interrupt-free polled IO unless you've configured the nvme driver to allocate some queues specifically for polled IO. Since these Samsung drives support 128 queues and you're only using a 16C/32T processor, you have more than enough for each drive to have one poll queue and one regular IO queue allocated to each (virtual) CPU core.
- tanelpoder 6y agoThat would explain it. Do you recommend any docs/links I should read about allocating queues for polled IO?
- anarazel 6y agoIt's terribly documented :(. You need to set the nvme.poll_queues to the number of queues you want, before the disks are attached. I.e. either at boot, or you need to set the parameter and then cause the NVMe to be rescanned (you can do that in sysfs, but I can't immediately recall the steps with high confidence).
- ogrisel 6y agoAs a nitpicking person, I really like to read a post that does not confuse GB/s for GiB/s :) https://en.wikipedia.org/wiki/Byte#Multiple-byte_units https://en.wikipedia.org/wiki/Byte#Multiple-byte_units
- ogrisel 6y agoActually now I realize that the title and the intro paragraph are contradicting each other...
- tanelpoder 6y agoYeah, I used the formally incorrect GB in the title when I tried to make it look as simple as possible... GiB just didn't look as nice in the "marketing copy" :-) I may have missed using the right unit in some other sections too. At least I hope that I've conveyed that there's a difference!
- secondcoming 6y ago> For final tests, I even disabled the frequent gettimeofday system calls that are used for I/O latency measurement I was knocking up some profiling code and measured the performance of gettimeofday as a proof-of-concept test. The performance difference between running the test on my personal desktop Linux VM versus running it on a cloud instance Linux VM was quite interesting (cloud was worse) I think I read somewhere that cloud instances cannot use the VDSO code path because your app may be moved to a different machine. My recollection of the reason is somewhat cloudy.
- ashkankiani 6y agoWhen I bought a bunch of NVME drives, I was disappointed with how slow the maximum speed I could achieve with them was given my knowledge and available time at the time. Thanks for making this post to give me more points of insight into the problem. I'm on the same page with your thesis that "hardware is fast and clusters are usually overkill," and disk I/O was a piece that I hadn't really figured out yet despite making great strides in the software engineering side of things. I'm trying to make a startup this year and disk I/O will actually be a huge factor in how far I can scale without bursting costs for my application. Good stuff!
- namero999 6y agoYou should be farming Chia on that thing [0] Amazing, congrats! [0] https://github.com/Chia-Network/chia-blockchain/wiki/FAQ https://github.com/Chia-Network/chia-blockchain/wiki/FAQ
- benlwalker 6y agoPlug for a post I wrote a few years ago demonstrating nearly the same result but using only a single CPU core: https://spdk.io/news/2019/05/06/nvme/ https://spdk.io/news/2019/05/06/nvme/ This is using SPDK to eliminate all of the overhead the author identified. The hardware is far more capable than most people expect, if the software would just get out of the way.
- tanelpoder 6y agoYes I had seen that one (even more impressive!) When I have more time again, I'll run fio with the SPDK plugin on my kit too. And would be interested in seeing what happens when doing 512B random I/Os?
- benlwalker 6y agoThe system that was tested there was PCIe bandwidth constrained because this was a few years ago. With your system, it'll get a bigger number - probably 14 or 15 million 4KiB IO per second per core. But while SPDK does have an fio plug-in, unfortunately you won't see numbers like that with fio. There's way too much overhead in the tool itself. We can't get beyond 3 to 4 million with that. We rolled our own benchmarking tool in SPDK so we can actually measure the software we produce. Since the core is CPU bound, 512B IO are going to net the same IO per second as 4k. The software overhead in SPDK is fixed per IO, regardless of size. You can also run more threads with SPDK than just one - it has no locks or cross thread communication so it scales linearly with additional threads. You can push systems to 80-100M IO per second if you have disks and bandwidth that can handle it.
- tanelpoder 6y agoYeah, that’s what I wondered - I’m ok with using multiple cores, would I get even more IOPS when doing smaller I/Os. Is the benchmark suite you used part of the SPDK toolkit (and easy enough to run?)
- benlwalker 6y ago
- rektide 6y agoNice follow up @ttanelpoder to "RAM is the new disk" (2015)[1] which we talked about not even two weeks ago! I was quite surprised to hear in that thread that AMD's infiniband was so oversubscribed. There's 256GBps of pcie on a 1P butit seems like this 66GBps is all the fabric can do. A little under a 4:1 oversubscription! [1] https://news.ycombinator.com/item?id=25863093 https://news.ycombinator.com/item?id=25863093
- electricshampo1 6y ago66GBps is from each of 10 drives doing ~6.6 GBps; don't think the infinity fabric is the limiter here
- rektide 6y agoI'd been going off this link[1] from the previous "Ram is the new diskx thread, but I think last time I read it I'd only counted one Infinity Fabric Inter-Socket on the 1P diagram (which provides the PCIe). On review, willing to bet, yeah, the pcie lanes aren't all sharing the one IFIS. The diagram is to give an idea, not the actual configuration. [1] https://en.wikichip.org/wiki/amd/infinity_fabric#Scalable_Data_Fabric_.28SDF.29 https://en.wikichip.org/wiki/amd/infinity_fabric#Scalable_Da...
- qaq 6y agoNow honestly say for how long two boxes like this behind a load balancer would be more than enough for your startup.
- qaq 6y agoWould be cool to see pgbench score for this setup
- muro 6y agoThis article was great, thanks for sharing! Anyone has advice on optimizing a windows 10 system? I have a haswell workstation (E5-1680 v3) that I find reasonably fast and works very well under Linux. In windows, I get lost. I tried to run the userbenchark suite which told me I'm below median for most of my components. Is there any good advice how to improve that? Which tools give good insight into what the machine is doing under windows? I'd like first to try to optimize what I have, before upgrading to the new shiny :).
- nwmcsween 6y agoSo Linus was wrong on his rant to Dave about the page cache being detremental on fast devices
- tutfbhuf 6y agoThis article focuses on IOPS and throughput, but what is also important for many applications is I/O latency, which can be measured with ioping (apt-get install ioping). Unfortunately, even 10x PCIe 4.0 NVMe do not provide any better latency than a single NVMe drive. If you are constrained by disk latency then 11M IOPS won't gain you much.
- cheeze 6y agoDoes this come up in practice? What kind of use cases suffer from disk latency? This stuff is all fascinating to me. I have a zfs NAS but I feel like I've barely scratched the surface of SSDs
- tutfbhuf 6y ago> Does this come up in practice? What kind of use cases suffer from disk latency? One popular example is HFT. And from my experience on a desktop PC it is better to disable swap and have the OOM killer do his work, instead of swapping to disk, which makes my system noticeable laggy, even with a fast NVMe.
- sitkack 6y agoAnything with transaction SLOs in the microsecond or millisecond range. Adtech, fintech, fraud detection, call records, shopping carts. Two big players in this space are Aerospike and ScyllaDB.
- tutfbhuf 6y agoThis is a very synthetical fio benchmark, I would like to see how actual applications like a postgres databases would perform on such a tuned machine.
- tanelpoder 6y agoYep, some “real” workload tests are coming next (using filesystems). I wanted to start from low level basics and later build on top of that.
- jacquesm 6y agoLovely article, zero fluff, tons of good content and modest to boot. Thank you for this write-up, I'll pass it around to some people who feel that the need for competent system administration skills has passed.
- maerF0x0 6y ago> Shouldn’t I be building a 50-node cluster in the cloud “for scalability”? This is exactly the point of my experiment - do you really want to have all the complexity of clusters or performance implications of remote storage if you can run your I/O heavy workload on just one server with local NVMe storage? Anyone have a story to share about their company doing just this? "Scale out" has basically been the only acceptable answer across most of my career. Not to mention High Availability.
- tanelpoder 6y agoYou can get high availability without a "distibuted system", just an active/passive failover cluster may be enough for some requirements. Even failover (sometimes seamless) on a VMWare cluster can help with planned maintenance scenarios without downtime, etc. Another way of achieving HA together with satisfying disaster recovery requirements is replication (either app level or database log replication, etc). So, no distributed system is necessary unless, you have legit scaling requirements. If you work on ERP-like databases for traditional Fortune 500-like companies, few people run such "sacred monolith" applications on modern distributed NoSQL databases, it's all Oracle, MSSQL or some Postgres nowadays. Data warehouses used to be all Oracle, Teradata too - although these DBs support some cluster scale-out, they're still "sacred monoliths" from a different era (they are still doing - what they were designed for - very well). Now of course Snowflake, BigQuery, etc are taking over the DW/analytics world for new greenfield projects, existing systems usually stay as they are due to lock-in & extremely high cost of rewriting decades of existing reports and apps.
- maerF0x0 6y ago> "distibuted system", just an active/passive failover cluster I would call this a distributed system. To me HA means 0 downtime deploys, are there SQL/RDBMS that offer that even for schema changes?
- MrFoof 6y agoU.2 form factor drives (also NVMe protocol) can achieve higher IOPS (particularly writes) still over M.2 form factor (especially M.2 2280), with higher durability, but you'll need your own controllers which are sparse on the market for the moment. Throughput (MB/sec, not IOPS) will be about the same, but the U.2 drives can do it for longer. U.2 means more NAND to parallelize over, more spare area (and higher overall durability), potentially larger DRAM caches, and a far larger area to dissipate heat. Plus it has all the fancy bleeding-edge features you aren't going to see on consumer-grade drives. -- ----- The big issue with U.2 for "end user" applications like workstations is you can't get drivers from Samsung for things like the PM1733 or PM9A3 (which blow the doors off the 980 Pro, especially for writes and $/GB, plus other neat features like Fail-In-Place) unless you're an SI, in which you also co-developed the firmware. The same goes for SanDisk, KIOXIA and other makers of enterprise SSDs. The kicker is enterprise U.2 drives are about the same $/GB as SATA drives, but being NVMe PCIe 4.0 x4. blow the doors off about everything. There's also the EDSFF, NF1 and now E.1L form factors, but U.2 is very prevalent. Enterprise SSDs are attractive as that's where the huge volume is (hence the low $/GB), but end-user support is really limited. You can use "generic drivers", but you won't see anywhere near the peak performance of the drives. The good news is both Micron and Intel have great support for end-users, where you can get optimized drivers and updated firmware. Intel has the D7-P5510 probably hitting VARs and some retail sellers (maybe NewEgg) within about 60 days. Similar throughput to the Samsung drives, far more write IOPS (especially sustained), lower latencies, FAR more durability (with a big warranty), far more capacity, and not too bad a price (looking like ~$800USD for 3.84TB with ~7.2PB of warrantied writes over 5 years). -- ----- My plan once Genesis Peak (Threadripper 5XXX) hits is four 3.84TB Intel D7-P5510s in RAID10, connected to a HighPoint SSD7580 PCIe 4.0 x16 controller. Figure ~$4,000 for a storage setup of ~7.3TB usable space after formatting, 26GB/sec peak writes, ~8GB/sec peak reads, with 2.8M 4K read iops, 700K 4K write iops, and ~14.3PB of warrantied write durability.
- floatboth 6y agoHow would a model-specific driver for something that speaks NVMe even work? Is it for Linux? Is it open? Is it just modifications to the stock Linux NVMe driver that take some drive specifics into account? Or is it some stupid proprietary NVMe stack?
- chx 6y agoA terabyte of RAM on your desktop. It's been less than a quarter century ago, 1997, when Microsoft and Compaq launched the TerraServer which was a wordplay on terabyte -- it stored a terabyte of data and it was a Big Deal. Today's that not storage, that's main RAM, unencumbered by NUMA.
- drmadera 6y agoGreat article. Did you consider doing Optane tests? I built a 3990x WS with all-optanes and I get blazing fast access times, but 3gb/s top speeds. It might be interesting to look at them for these tests, specially in time-sensitive scenarios.
- tanelpoder 6y agoI have 2 Optane 905P M.2 cards and I intend to run some database engine tests, putting their transaction logs (and possibly temporary spill areas for sorts, hashes) on Optane. When I think about Optane, I think about optimizing for low latency where it's needed and not that much about bandwidth of large ops.
- svacko 6y agoI wonder, is increasing temperature of the M.2 NVMe disks affecting the measured performance? Or is P620 cooling system efficient enough to keep temp of the number of disks low? Anyway, thanks for the inspirative post!
- tanelpoder 6y agoBoth quad SSDs adapters had a fan on it and the built in M.2 ones had a heatsink, right in front of one large chassis fan & air intake. I didn't measure the SSD temperatures, but the I/O rate didn't drop over time. I was bottlenecked by CPU when doing small I/O tests, I monitored the current MHz from /proc/cpuinfo to make sure that the CPU speeds didn't drop lower than their nominal 3.9 GHz (and they didn't). Btw, even the DIMMs have dedicated fans and enclosure (one per 4 DIMMs) on the P620.
- occamschainsaw 6y agoI love articles like these, taking a deep dive into achieving the absolute best on neglected metrics like IO. I am trying to get very high resolution (~ 2 * 10^6 samples/s) voltage and current measurements of a sensor for a control system. Has anyone tried that? Should it be done through PCIe?
- sitkack 6y agoFor a lab environment I would look at https://www.picotech.com/oscilloscope/2000/picoscope-2000-specifications https://www.picotech.com/oscilloscope/2000/picoscope-2000-sp... or https://redpitaya.readthedocs.io/en/latest/developerGuide/125-10/vs.html https://redpitaya.readthedocs.io/en/latest/developerGuide/12...