6 ms·
I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that
by tyingq 6y ago
I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me.
64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU.
If you price it out on Dell's site, you get a retail price north of $200k. That is really what made it clear for me. That you could fit $200k+ worth of DIMMS, Drives, CPUS into a 2U spot :)
- vmception 6y agoI have a motherboard from 2012 and I just put 2x 8TB NVMe SSDs on it, on a PCIe 2.0 x16 slot Works great. The PCIe card itself has 2 more slots for SSDs The GPU is on the 2.0 x8 slot because they don't really transfer that much data over the lanes. I honestly didn't realize PCIe was up to 4.0 now, and I am pushing up against the limits of PCIe 2.0 but it still works! And I’m “only” at the limits, and its only a limit when I want faster than 3,000 megabytes per second, which is amazing. Granted, this would have been considered a good enthusiast motherboard in 2012. Buying new but cheap is the mistake.
- foota 6y agoWhat drives did you get? I think you need PCI 4 to stress most SSDs these days?
- deleted 6y ago[deleted]
- vmception 6y agoI have a hunch that the pcie card itself is most important as it is doing bifurcation. So each drive acts like it has its own slower (but fast enough) pcie slot, and then the raid0 combines the bits back to double the performance. Could be wrong but I get 2,900 megabytes per second transfers from RAM to disk and back. And this is PCIe 2.0 x16 so maybe if you want , 3,000, 4,000 or 6,500 megabytes per second then I have nothing to brag about. I’m pretty amazed though and will be content for all my use cases.
- jaas 6y agoJust so everyone here is aware re: pricing... If you buy through a VAR and/or Dell reps you don't pay the price on the website. What you actually pay is typically significantly lower. I don't think anyone actually buys servers like these by just ordering from the website. We (Let's Encrypt) certainly don't. These are expensive servers, crossing into six digits, but not $200k.
- tyingq 6y agoYes, that's worth noting. The $200k is the list/retail, which nobody would pay for a purchase of this size.
- walrus01 6y agoAt this size/scale I wonder what the actual price would be for a comparable Supermicro system. If I had to make a wild guess, well under half the 200k previously quoted.
- tyingq 6y agoThe NVME drives are what's making it so expensive. Dell retail is $6044 for each of the Intel P4610 6.4TB drives. That's $145k.
- walrus01 6y agoafter 15 seconds of googling, the P4610 U.2 format 6.4TB seems something more like a single unit street price of $2400 from non-Dell vendors. I'm mildly surprised it's that low considering that the U.2 format stuff, for serious servers, will always command a premium price. Probably in the range of $2100 to $2200 per unit from a x86-64 component distributor in moderate quantities.
- tyingq 6y agoJust noticed this in the Dell cart... "42% off list price: use code SERVER42" Doesn't make the price reasonable exactly, but it's kind of funny.
- walrus01 6y agoOne of the interesting things about threadripper and epyc systems with a lot of PCI-E 4.0 lanes on one motherboard, if your goal is not NVME disk I/O, but rather network throughput, it also provides the opportunity for a DIY approach to very high capacity multi-100GbE ethernet interface routers. Such as a fairly low cost 4U threadripper box, running VyOS (ultimately based on a very normal debian Linux) with multiple Intel 100GbE NICs in PCI-E slots. The Intel linux kernel (and FreeBSD) driver support for those is generally excellent and robust. Intel has had full time developers doing the Linux drivers for that series of cards since the earliest days of their 64-bit/66MHz PCI-X 10Gbps NICs about 17 years ago. Also worth mentioning that FRR is now an official Linux Foundation supported project. https://frrouting.org/ https://frrouting.org/ https://www.intel.com/content/www/us/en/products/docs/network-io/ethernet/network-adapters/ethernet-800-series-network-adapters/e810-cqda1-cqda2-100gbe-brief.html https://www.intel.com/content/www/us/en/products/docs/networ...
- jart 6y agoAltavista used to run their entire search engine on a single computer twenty years ago. Imagine what Mike Burrows could accomplish today with one of these 2U EPYC NVME 200gbps PCs. It'd probably trigger the singularity.
- theandrewbailey 6y agoDell-controlled VMWare is listed under "major sponsors and funders". I wonder if Let's Encrypt got a discount on that server. Good for them!
- wpietri 6y agoTotally. 2 TB of RAM! In one box! I think the first servers I had in production had 8 MB RAM. No more, certainly. Soon we'll be at 1000x that. My dad's first "server" was 3 orders of magnitude smaller, with 8 KB of RAM (hand-wound wire core memory). In that time, the US population hasn't even doubled.
- jagger27 6y agoMore like 1,000,000x
- deleted 6y ago[deleted]
- wpietri 6y agoOops! Yes, that's correct.
- buzer 6y agohttps://yourdatafitsinram.net/ https://yourdatafitsinram.net/ The listed servers are from few years ago so I guess there might be some bigger ones available these days.
- midasuni 6y agoMy phone in my pocket has 1,000 times 8MB of ram. A million of your first serves in one box. In about 30 years?
- amluto 6y agoDell makes it easier than it should be by having absurd markups on storage.
- infogulch 6y agoSTH recently did a video about this dense mass of storage & compute they call a server. iirc they specced it with their discounts to end up at a much more reasonable ~$90k. https://youtu.be/9PLvt3WATMU https://youtu.be/9PLvt3WATMU
- ggregoire 6y agoWe made so much progress in that field over the last 40 years... I sometimes get lost trying to imagine what will be the computing performance available in 100, 1000, 10000 years from now...
- gnarbarian 6y agoSticks and rocks most likely.
- adolph 6y agoYep, quantum entanglement materials thru and thru
- eppp 6y agoSingle threaded performance isn't keeping pace unfortunately.
- SulfurHexaFluri 6y agoThat just means multithreaded will become more and more important. Languages like rust may gain some usage as we start to require safe multithreading.
- eppp 6y agoThat may be so, but that doesnt make it any less depressing. Some things cant be run parallel. I cant mail a letter until it is written.
- JohnJamesRambo 6y agoAnd I'm afraid it will still obey this trend. https://danluu.com/input-lag/ https://danluu.com/input-lag/
- kornbattery 6y agoThat doesen't seem like some fundamental restriction, but rather a compromise we're always willing to make. 30ms is not noticeable, so we don't try to lower it and sacrifice something else. Even the super slow ones like on kindle, are _choices_ that have been made in favor of something else. a second to turn a page on a book isn't unbearable
- dan-robertson 6y agoFor a comparison, you could get a 64 processor, 256Gb ram Sun Starfire around 20 years ago[1]. Wikipedia claims these cost well over a million dollars ($1.5 million in 2021 dollars). This machine was enormous (bigger than a rack), would have had more non-uniform memory access to deal with, and the processors were clocked at something like 250-650MHz. [1] https://en.wikipedia.org/wiki/Sun_Enterprise https://en.wikipedia.org/wiki/Sun_Enterprise
- tyingq 6y agoThat's a pretty good comparison. It weighed 2000lbs, and was 38 inches wide, or basically 2 full racks, which I guess you could call an 84U server. It was also 49 inches deep, versus a standard rack which is 36 inches deep.
- angled 6y agoiirc, anton got a kernel compile down to about a minute with -j32 back in the day while at ozlabs?
- segmondy 6y agoI'm surprised that's what did it for me. I admined some sun enterprise servers and was blown away when I bought a pi 3 yrs and realized the pi was probably faster than it. That enterprise server would have 100-200 engineering students logged in and working at once. :-/. We're living in the future.
- spijdar 6y ago> the pi was probably faster than it. If we're talking about 90s or early 2000s Sun, probably. Even then, though, these systems probably had substantially better I/O performance. I have a SPARC laptop lying around from 1995 that gets a whopping 20 MB/s in disk read/write speeds across its dual half height SCSI drives. That still beats all but the best SD cards. The full size systems with SAS and other options would get even better disk performance.
- bloopernova 6y agoI like to play a game called "A current Raspberry Pi (or mobile phone) is equivalent to what year's global computing power?" Also "fun" is "The solid state storage I can buy for $100 is equivalent to all the world's computing storage in what year?" I'm super fun at parties!
- alberth 6y agoAnd if you don't want to buy Dell, you can get a SuperMicro server that has about 2x the specs for half the price ($102k). https://www.siliconmechanics.com/system/a+-ultra-superserver-2123us-tn24r25m https://www.siliconmechanics.com/system/a+-ultra-superserver... 2U 128-Core 307TB NVME 4 TB RAM ------- = $102k
- haltingproblem 6y agoWe profiled Supermicro vs. Dell vs. HP vs. IBM servers a decade ago for trading applications. Supermicro was cheaper by 20-30% had a quantitatively better power profile which basically meant it paid for itself and just rocked on every dimension. And their sales reps were able to answer every question we had including sending us a excel model of their power usage. The dell folks never got back to us on power usage. I dunno why folks insist on Dell crapola.
- m463 6y agoYou can build a pretty nice home 32-core/256gb threadripper system for ~ $5k. (24 nvme drives not included)