27 ms·
SSDs have become fast, except in the cloud
- teaearlgraycold 3y agoWhat’s a good small cloud competitor to AWS? For teams that just need two AZs to get HA and your standard stuff like VMs, k8s, etc.
- leroman 3y agoDigitalOcean has been great, have managed k8s and generally good bang for your buck
- ThrowawayTestr 3y agoBuy a server
- nullindividual 3y agoI don't like this answer. When I look at cloud, I get to think "finally! No more hardware to manage. No OS to manage". It's the best thing about the cloud, provided your workload is amenable to PaaS. It's great because I don't have to manage Windows or IIS. Microsoft does that part for me and significantly cheaper than it would be to employ me to do that work.
- adastra22 3y agoThere is tooling to provide the same PaaS interface though, so your cost doing those things amounts to running OS updates.
- cynicalsecurity 3y agoA cloud is just someone else's computer. When you rent a bare metal server, you don't manage your hardware either. The failed parts are replaced for you. Unless you can't figure out what hardware configuration you need - which would be a really big red flag for your level of expertise.
- kikimora 3y agoA cloud is network and tools, less hardware. For example see this [1] Reddit thread discussing network in Hetzner. Any other bare-metal would have same challenges. Once you solve network security you have to deal with server access. People hired and fired, hardcoded SSH keys is a bad idea. Once you solve access you likely have AD, LDAP and SSO of some sort. Then backups, and automated test suite + periodical test recoveries. Then database and backups. Then secrets, does all members of your team know production db password? And so on and on. Maybe TCO still favors bare-metal but you have to spend a lot of time on configuration. [1] https://www.reddit.com/r/hetzner/comments/rjuzcs/securing_network_in_hetzner/ https://www.reddit.com/r/hetzner/comments/rjuzcs/securing_ne...
- tjoff 3y agoAnd now you have to manage the cloud instead. Which turns out to be more hassle and with no overlap with the actual problem you are trying to solve. So not only do you spend time on the wrong thing you don't even know how it works. And the providers goals are not aligned either as all they care about is locking you in. How is that better?
- nullindividual 3y agoThat may be your experience, but not mine. Yes, the cloud is _different_ to manage and has some of the same fundamentals to overcome such as security and networking, but lacks some of the very large pain points of managing an OS, like updates, ancillary local services, local accounts, and so on. I'm not sure why you would state that it doesn't solve the problem I'm invested in -- namely operating websites. It is the perfect cloud workload.
- deathanatos 3y ago> No more hardware to manage. No OS to manage We must be using different clouds. For some of the much higher-level services … maybe some semblance of that statement holds. But for VMs? Definitely not "no OS to manage" … the OS is usually on the customer. There might be OS-level agents from your cloud of choice that make certain operations easier … but I'm still on the hook for updates. Even "No machine" is a stretch, though I've found this is much more dependent on cloud. AWS typically notices failures before I do, and by the time I notice something is up, the VM has been migrated to a new host and I'm none the wiser sans the reboot that cost. But other clouds I've been less lucky with: we've caught host failures well before the cloud provider, to an extent where I've wished there was a "vote of no confidence" API call I could make to say "give me new HW, and I personally think this HW is suss". Even on higher level services like RDS, or S3, I've noticed failures prior to AWS … or even to the extent that I don't know that AWS would have noticed those failures unless we had opened the support ticket. (E.g., in the S3 case, even though we clearly reported the problem, and the problem was occurring on basically every request, we still had to provide example request IDs before they'd believe us. The service was basically in an outage as far as we could tell … though I think AWS ended up claiming it was "just us".) That said, S3 in particular is still an excellent service, and I'd happily use it again. But cloud == 0 time on my part. It depends heavily on the cloud, and less heavily on the service how much time, and sometimes, it is still worthwhile.
- nullindividual 3y agoI'm baffled after you quoted 'no OS to manage' why you'd start discussing virtual machines.
- pritambarhate 3y agoDigital Ocean. But they don’t have concept of multi az as far as I know. But they have multiple data centers in same region. But I am not aware if there is any private networking between DCs in the same region.
- emseetech 3y agoReally needs more information for a good answer. How many req/s are you handling? Do you have dedicated devops? Write heavy, read heavy or pipeline/etl heavy? Is autoscaling a must or would you rather failure over a bank breaking cloud bill?
- dimgl 3y agoI am _super_ impressed with Supabase. I can see how a lot of people may not like it given how reliant it is on Postgres, but I find it to be absolutely genius. It basically allows me to forego having to make a server for the CRUD operations so I can focus on the actual business implications. My REST API is automatically managed for me (mostly with lightweight views and functions) and all of my other core logic is either spread out through edge functions or in a separate data store (Redis) where I perform the more CPU intensive operations related to my business. There's some rough edges around their documentation and DX but I'm really loving it so far.
- catherinecodes 3y agoHetzner and Entrywan are pure-play cloud companies with good prices and support. Hetzner is based in Germany and Entrywan in the US.
- infamia 3y agoThanks for mentioning Entrywan, they look great from what I can tell on their site. Have you used their services? If so, I'm curious about your experiences with them.
- catherinecodes 3y agoMy irc bouncer and two kubernetes clusters are running there. So far the service has been good.
- madars 3y agoHetzner has a reputation for locking accounts for "identity verification" (Google "hetzner kyc" or "hetzner identity verification"). Might be worthwhile to go through a reseller just to avoid downtime like that.
- infecto 3y agoAWS is pretty great and I think reasonably cheap, I would include any of the other large cloud players. The amount saved is just not reasonable enough for me, I would rather work with something that I have used over and over with. Along with that, ChatGPT has knocked down most of the remaining barriers I have had when permissions get confusing in one of the cloud services.
- mciancia 3y agoI like using scaleway for personal projects, but they are available only in Europe
- folmar 3y agoHetzner, OVH, Scaleway
- michellezzz 3y agoKubeBlocks. It's open source and can manage data infra on K8s.
- bombcar 3y agoI think the obvious answer is there's not much demand, and keeping it "low" allows trickery and funny business with the virtualization layer (think: SAN, etc) that you can't do with "raw hardware speed".
- _Rabs_ 3y agoSure, but it does make me wonder what kind of speeds we are paying for if we can't even get raw hardware speeds... Sounds like one more excuse for AWS to obfuscate any meaning in their billing structure and take control of the narrative. How much are they getting away with by virtualization. (Think how banks use your money for loans and stuff) You actually don't get to really see the internals other than IOPS which doesn't help when it's gatekept already.
- bombcar 3y agoThe biggest "scam" if you can call it that is reducing all factors of CPU performance to "cores".
- Nextgrid 3y agoI'd argue the even bigger scam is charging for egress data transfer rather than just for the size of the pipe.
- zokier 3y agoAWS is pretty transparent about what sort of cores you are exactly getting, and has different types available for different use-cases; typical example would something like r7iz that is aimed for peak single-threaded perf https://aws.amazon.com/ec2/instance-types/r7iz/ https://aws.amazon.com/ec2/instance-types/r7iz/
- s1gnp0st 3y agoTrue but the funny business buys a lot of fault tolerance, and predictable performance if not maximum performance.
- 3y ago
- pclmulqdq 3y agoThis was a huge technical problem I worked on at Google, and is sort of fundamental to a cloud. I believe this is actually a big deal that drives peoples' technology directions. SSDs in the cloud are attached over a network, and fundamentally have to be. The problem is that this network is so large and slow that it can't give you anywhere near the performance of a local SSD. This wasn't a problem for hard drives, which was the backing technology when a lot of these network attached storage systems were invented, because they are fundamentally slow compared to networks, but it is a problem for SSD.
- brucethemoose2 3y agoYeah this was my impression. I am but an end user, but I noticed that disk IO for a certain app was glacial compared to a local test deployment, and I chalked it up to networking/VM overhead
- vlovich123 3y agoWhy do they fundamentally need to be network attached storage instead of local to the VM?
- Filligree 3y agoThey don't. Some cloud providers (i.e. Hetzner) let you rent VMs with locally attached NVMe, which is dramatically faster than network-attached even factoring in the VM tax. Of course then you have a single point of failure, in the PCIe fabric of the machine you're running on if not the NVMe itself. But if you have good backups, which you should, then the juice really isn't worth the squeeze for NAS storage.
- ssl-3 3y agoA network adds more points of failure. It does not reduce them.
- supriyo-biswas 3y ago
- siliconc0w 3y agoCore count plus modern nvme actually make a great case for moving away from the cloud- before it was, "your data probably fits into memory". These are so fast that they're close enough to memory so it's "your data surely fits on disk". This reduces the complexity of a lot of workloads so you can just buy a beefy server and do pretty insane caching/calculation/serving with just a single box or two for redundancy.
- echelon 3y agoThe reasons to switch away from cloud keep piling up. We're doing some amount of on-prem, and I'm eager to do more.
- aarmenaa 3y agoI've previously worked for a place that ran most of their production network "on-prem". They had a few thousand physical machines spread across 6 or so colocation sites on three continents. I enjoyed that job immensely; I'd jump at the chance to build something like it from the ground up. I'm not sure if that actually makes sense for very many businesses though.
- kuchenbecker 3y agoI'm getting that opportunity, however I expect it will be the last as most have migrated to the cloud and smaller companies are appealing to me; smallest company (of 5) I've worked for had 4.4k employees and large companies have the resources to roll their own. Unless there is an onprem movement I expect cloud to be the future as maintaining the tech stack onprem is difficult and we need to nake decisions down to the hardware we order.
- malfist 3y agoI keep hearing that, but that's simply not true. SSDs are fast, but they're several orders of magnitude slower than RAM, which is orders of magnitude slower than CPU Cache. Samsung 990 Pro 2TB has a latency of 40 μs DDR4-2133 with a CAS 15 has a latency of 14 nano seconds. DDR4 latency is 0.035% of one of the fastest SSDs, or to put it another way, DDR4 is 2,857x faster than an SSD. L1 cache is typically accessible in 4 clock cycles, in 4.8 ghz cpu like the i7-10700, L1 cache latency is sub 1ns.
- zokier 3y ago> Since then, several NVMe instance types, including i4i and im4gn, have been launched. Surprisingly, however, the performance has not increased; seven years after the i3 launch, we are still stuck with 2 GB/s per SSD. AWS marketing claims otherwise: Up to 800K random write IOPS Up to 1 million random read IOPS Up to 5600 MB/second of sequential writes Up to 8000 MB/second of sequential reads https://aws.amazon.com/blogs/aws/new-storage-optimized-amazon-ec2-i4g-instances-graviton-processors-and-aws-nitro-ssds/ https://aws.amazon.com/blogs/aws/new-storage-optimized-amazo...
- sprachspiel 3y agoThis is for 8 SSDs and a single modern PCIe 5.0 has better specs than this.
- jeffbee 3y agoThose claims are per device. There isn't even an instance in that family with 8 devices.
- nik_0_0 3y agoIs it? The line preceding the bullet list on that page seems to state otherwise: “” Each storage volume can deliver the following performance (all measured using 4 KiB blocks): * Up to 8000 MB/second of sequential reads “”
- sprachspiel 3y agoJust tested a i4i.32xlarge: $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 24.9M 1 loop /snap/amazon-ssm-agent/7628 loop1 7:1 0 55.7M 1 loop /snap/core18/2812 loop2 7:2 0 63.5M 1 loop /snap/core20/2015 loop3 7:3 0 111.9M 1 loop /snap/lxd/24322 loop4 7:4 0 40.9M 1 loop /snap/snapd/20290 nvme0n1 259:0 0 8G 0 disk ├─nvme0n1p1 259:1 0 7.9G 0 part / ├─nvme0n1p14 259:2 0 4M 0 part └─nvme0n1p15 259:3 0 106M 0 part /boot/efi nvme2n1 259:4 0 3.4T 0 disk nvme4n1 259:5 0 3.4T 0 disk nvme1n1 259:6 0 3.4T 0 disk nvme5n1 259:7 0 3.4T 0 disk nvme7n1 259:8 0 3.4T 0 disk nvme6n1 259:9 0 3.4T 0 disk nvme3n1 259:10 0 3.4T 0 disk nvme8n1 259:11 0 3.4T 0 disk Since nvme0n1 is the EBS boot volume, we have 8 SSDs. And here's the read bandwidth for one of them: $ sudo fio --name=bla --filename=/dev/nvme2n1 --rw=read --iodepth=128 --ioengine=libaio --direct=1 --blocksize=16m bla: (g=0): rw=read, bs=(R) 16.0MiB-16.0MiB, (W) 16.0MiB-16.0MiB, (T) 16.0MiB-16.0MiB, ioengine=libaio, iodepth=128 fio-3.28 Starting 1 process ^Cbs: 1 (f=1): [R(1)][0.5%][r=2704MiB/s][r=169 IOPS][eta 20m:17s] So we should have a total bandwidth of 2.7*8=21 GB/s. Not that great for 2024.
- cogman10 3y agoThere's a 4th option. Cost. The fastest SSDs tend to also be MLC which tend to have much lower write life vs other technologies. This isn't unusual, increasing data density generally also makes it easier to increase performance. However, it's at the cost that the writes are typically done for a block/cell in memory rather than for single bits. So if one cell goes bad, they all fail. But even if that's not the problem, there is a problem of upgrading the fleet in a cost effective mechanism. When you start introducing new tech into the stack, replacing that tech now requires your datacenters to have 2 different types of hardware on hand AND for the techs swapping drives to have a way to identify and replace that stuff when it goes bad.
- kilmanio 3y agoSure, MLC write life is worse than SLC, but significantly better than TLC and QLC -edit: this comment was purely focused at your first sentence: >The fastest SSDs tend to also be MLC which tend to have much lower write life vs other technologies. I'm not sure what you mean with "other technologies" in this case, SLC is indeed truly expensive for a significantly higher write life, and HDDs are debatable for their lifespan.
- c0l0 3y agoSeeing the really just puny "provisioned IOPS" numbers on hugely expensive cloud instances made me chuckle (first in disbelief, then in horror) when I joined a "cloud-first" enterprise shop in 2020 (having come from a company that hosted their own hardware at a colo). It's no wonder that many people nowadays, esp. those who are so young that they've never experienced anything but cloud instances, seem to have little idea of how much performance you can actually pack in just one or two RUs today. Ultra-fast (I'm not parroting some marketing speak here - I just take a look at IOPS numbers, and compare them to those from highest-end storage some 10-12 years ago) NVMe storage is a big part of that astonishing magic.
- Aurornis 3y ago> It's no wonder that many people nowadays, esp. those who are so young that they've never experienced anything but cloud instances, seem to have little idea of how much performance you can actually pack in just one or two RUs today. On the contrary, young people often show up having learned on their super fast Apple SSD or a top of the line gaming machine with NVMe SSD. Many know what hardware can do. There’s no need to dunk on young people. Anyway, the cloud performance realities are well know to anyone who works in cloud performance. It’s part of the game and it’s learned by anyone scaling a system. It doesn’t really matter what you could do if you build a couple RUs yourself and hauled them down to the data center, because beyond simple single-purpose applications with flexible uptime requirements, that’s not a realistic option.
- zten 3y ago> On the contrary, young people often show up having learned on their super fast Apple SSD or a top of the line gaming machine with NVMe SSD. Yes, this is often a big surprise. You can test out some disk-heavy app locally on your laptop and observe decent performance, and then have your day completely ruined when you provision a slice of an NVMe SSD instance type (like, i4i.2xlarge) and discover you're only paying for SATA SSD performance.
- seabrookmx 3y ago
- zokier 3y agoHas anyone disk benchmarks for M7gd (or C/R equivalents) instance stores? While probably not at the level of I4g, would still be interesting comparison.
- Twirrim 3y agoDisclaimer: I work for OCI, opinion my own etc. We offer faster NVMe drives in instances. Our E4 Dense shapes ship with SAMSUNG MZWLJ7T6HALA-00AU3, which supports Sequential Reads of 7000 MB/s, and Sequential Write 3800 MB/s. From a general perspective, I would say the _likely_ answer to why AWS doesn't have faster NVMes at the moment is likely to be lack of specific demand. That's a guess, but that's generally how things go. If there's not enough specific demand being fed in through TAMs and the like for faster disks, upgrades are likely to be more of an after-thought, or reflecting supply chain. I know there's a tendency when you engineer things, to just work around, or work with the constraints, and grumble amongst your team, but it's incredibly invaluable if you can make sure your account manager knows what shortcomings you've had to work around.
- nine_k 3y agoI very much expect AWS SSD / NVMe upgrades to be well thought-out ahead of time, and optimized for both upfront cost and for longevity / durability. Speed may be a third consideration.
- flaminHotSpeedo 3y agoYeah, hardware and forecasting for cloud providers is basically the definition of deliberate. If anything I'd guess it's a procurement issue, parity between regions is a big thing and it's hard to supply dozens of regions around the world with the latest hardware hotness
- e12e 3y ago> I work for OCI Ah: Oracle cloud infra https://blogs.oracle.com/cloud-infrastructure/post/announcing-e4-denseio-instances-with-2x-performance-for-database-and-analytics-workloads https://blogs.oracle.com/cloud-infrastructure/post/announcin... I keep forgetting Oracle is in the cloud business too. "Make it rain", I guess :)
- Twirrim 3y agoDoh, sorry. I'm usually good about not using that acronym!
- fabioyy 3y agoit is not worth to use cloud if you need a lot of iops/bandwidth heck, its not worth for anything besides scalability dedicated servers are wayyyy cheaper
- kstrauser 3y agoI'm not certain that's true if you look at TCO. Yes, you can probably buy a server for less than the yearly rent on the equivalent EC2 instance. But then you've got to put that server somewhere, with reliable power and probably redundant Internet connections. You have to pay someone's salary to set it up and load it to the point that a user can SSH in and configure it. You have to maintain an inventory of spares, and pay someone to swap it out if it breaks. You have to pay to put its backups somewhere. Yeah, you can skip a lot of that if your goal is to get a server online as cheaply as possible, reliability be damned. As soon as you start caring about keeping it in a business-ready state, costs start to skyrocket. I've worn the sysadmin hat. If AWS burned down, I'd be ready and willing to recreate the important parts locally so that my company could stay in business. But wow, would they ever be in for some sticker shock.
- Nextgrid 3y ago> But then you've got to put that server somewhere, with reliable power and probably redundant Internet connections. You have to pay someone's salary to set it up and load it to the point that a user can SSH in and configure it. You have to maintain an inventory of spares, and pay someone to swap it out if it breaks. There's a middle-ground between cloud and colocation. There are plenty of providers such as OVH, Hetzner, Equinix, etc which will do all of the above for you.
- the8472 3y agoAt least in the workstation segment cloud doesn't compete. We use Threadrippers + A6000 GPUs at work. Getting the equivalent datacenter-type GPUs and EPYC processors is more expensive, even after accounting for IT and utilization.
- layer8 3y agoWhere I live, a number of SMEs are doing this. It’s really not that costly, unless you are a tiny startup I guess.
- pkstn 3y agoUpCloud has super fast MaxIOPS: https://upcloud.com/products/block-storage https://upcloud.com/products/block-storage Here's referral link with free credits: https://upcloud.com/signup/?promo=J3JYWZ https://upcloud.com/signup/?promo=J3JYWZ
- rbranson 3y agoIt's awfully similar to a lane of PCIe 4.0, if we're talking about instance storage. It reads like they've chosen to map each physical device to a single PCIe lane. Surely the AWS Nitro hardware platform has longer cycle times than PCIe. Note that once the instance type has multiple block devices exposed (i.e. im4gn.8xlarge or higher), striping across these will reach higher throughput (2 devices yields 4G/s, 4 yields 8G/s).
- 0cf8612b2e1e 3y agoSerious question, for a consumer does it make any sense to compare SSD benchmarks? I assume the best and worst models give a user an identical experience in 99% of cases, and it is only prosumer activities (video? sustained writes?) which would differentiate them.
- wmf 3y agoYeah, that's pretty much the case. Cheap SSDs provide good enough performance for desktop use.
- eisa01 3y agoWould this be a consequence of the cloud providers not being on the latest technology CPU-wise? At least I have the impression they are lagging, eg., still offering things like: z1d: Skylake (2017) https://aws.amazon.com/ec2/instance-types/z1d/ https://aws.amazon.com/ec2/instance-types/z1d/ x2i: Cascade Lake (2019) and Ice lake (2021) https://aws.amazon.com/ec2/instance-types/x2i/ https://aws.amazon.com/ec2/instance-types/x2i/ I have not been able to find instances powered by the 4th (Q1 2023) or 5th generation (Q4 2023) Xeons? We solve large capacity expansion power market models that need as fast single-threaded performance as possible coupled with lots of RAM (32:1 ratio or higher ideal). One model may take 256-512 GB RAM, but not being able to use more than 4 threads effectively (interior point algorithms have very diminishing returns past this point) Our dispatch models do not have the same RAM requirement, but you still wish to have the fastest single-threaded processors available (and then parallelize)
- deadmutex 3y agoYou can find Intel Sapphire Rapids powered VM instances on GCE
- zokier 3y agoAWS was offering Sapphire Rapids instances before those CPUs became even publicly available https://aws.amazon.com/about-aws/whats-new/2022/11/introducing-amazon-ec2-r7iz-instances/ https://aws.amazon.com/about-aws/whats-new/2022/11/introduci...
- deleted 3y ago[deleted]
- kwillets 3y agoAWS docs and blogs describe the Nitro SSD architecture, which is locally attached with custom firmware. > The Nitro Cards are physically connected to the system main board and its processors via PCIe, but are otherwise logically isolated from the system main board that runs customer workloads. https://docs.aws.amazon.com/whitepapers/latest/security-design-of-aws-nitro-system/the-components-of-the-nitro-system.html https://docs.aws.amazon.com/whitepapers/latest/security-desi... > In order to make the [SSD] devices last as long as possible, the firmware is responsible for a process known as wear leveling.... There’s some housekeeping (a form of garbage collection) involved in this process, and garden-variety SSDs can slow down (creating latency spikes) at unpredictable times when dealing with a barrage of writes. We also took advantage of our database expertise and built a very sophisticated, power-fail-safe journal-based database into the SSD firmware. https://aws.amazon.com/blogs/aws/aws-nitro-ssd-high-performance-storage-for-your-i-o-intensive-applications/ https://aws.amazon.com/blogs/aws/aws-nitro-ssd-high-performa... This firmware layer seems like a good candidate for the slowdown.
- someguydave 3y agoYeah I wonder how Nitro balances the latency & bandwidth demands of multiple VMs while also minimizing memory cache misses on the CPU (I am assuming it uses DMA to talk to the main CPU cores)
- dan-robertson 3y agoYeah, I’m curious how they would respond to the claims in the article. In [1], they talk about aiming for low latency, for consistent performance (apparently other SSDs could stall at inopportune times), and support on-disk encryption. Latency is often in direct conflict with throughput (eg batching usually trades one for the other), and also matters a lot for plenty of filesystem or database tasks (indeed the OP links to a paper showing that popular databases, even column stores, struggle to use the full disk throughput, though I didn’t read why). Encryption is probably not the reason – dedicated hardware on modern chips can do AES at 50GB/s, though maybe it is if it increases latency? So maybe there’s something else to it like sharing between many vms on one host [1] https://m.youtube.com/watch?v=Cxie0FgLogg https://m.youtube.com/watch?v=Cxie0FgLogg
- dan-robertson 3y agoIs read/write throughput the only difference? Eg I don’t know how latency compares, or indeed failure rates or whether fsync lies or not and writes won’t always survive power failures.
- jiggawatts 3y agoThere's a lot of talk about cloud network and disk performance in this thread. I recently benchmarked both Azure and AWS and found that: - Azure network latency is about 85 microseconds. - AWS network latency is about 55 microseconds. - Both can do better, but only in special circumstances such as RDMA NICs in HPC clusters. - Cross-VPC or cross-VNET is basically identical. Some people were saying it's terribly slow, but I didn't see that in my tests. - Cross-zone is 300-1200 microseconds due to the inescapable speed of light delay. - VM-to-VM bandwidth is over 10 Gbps (>1 GB/s) for both clouds, even for the smallest two vCPU VMs! - Azure Premium SSD v1 latency varies between about 800 to 3,000 microseconds, which is many times worse than the network latency. - Azure Premium SSD v2 latency is about 400 to 2,000 microseconds, which isn't that much better, because: - Local SSD caches in Azure are so much faster than remote disk that we found that Premium SSD v1 is almost always faster than Premium SSD v2 because the latter doesn't support caching. - Again in Azure, the local SSD "cache" and also the local "temp disks" both have latency as low as 40 microseconds, on par with a modern laptop NVMe drive. We found that switching to the latest-gen VM SKU and turning on the "read caching" for the data disks was the magic "go-fast" button for databases... without the risk of losing out data. We investigated the various local-SSD VM SKUs in both clouds such as the Lasv3 series, and as the article mentioned, the performance delta didn't blow my skirt up, but the data loss risk made these not worth the hassle.
- computerdork 3y agoInteresting. And would you happen to have the numbers on the performance of the local SSD? Is it's read and write throughput up to the level of modern SSD's?
- jiggawatts 3y agoIt's pretty much like how the article said. The cloud local SSDs are notably slower than what you'd get in an ordinary laptop, let alone a high-end server. I'm not an insider and don't have any exclusive knowledge, but from reading a lot about the topic my impression is that the issue in both clouds is the virtualization overheads. That is, having the networking or storage go through any hypervisor software layer is what kills the performance. I've seen similar numbers with on-prem VMware, Xen, and Nutanix setups as well. Both clouds appear to be working on next-generation VM SKUs where the hypervisor network and storage functions are offloaded into 100% hardware, either into FPGAs or custom ASICs. "Azure Boost" is Microsoft's marketing name for this, and it basically amounts to both local and remote disks going through an NVMe controller directly mapped into the memory space of the VM. That is, the VM OS kernel talks directly to the hardware, bypassing the hypervisor completely. This is shown in their documentation diagrams: https://learn.microsoft.com/en-us/azure/azure-boost/overview https://learn.microsoft.com/en-us/azure/azure-boost/overview They're claiming up to 3.8M IOPS for a single VM, which is 3-10x what you'd get out of a single NVMe SSD stick, so... not too shabby at all! Similarly, Microsoft Azure Network Adapter (MANA) is the equivalent for the NIC, which will similarly connect the VM OS directly into the network, bypassing the hypervisor software. I'm not an AWS expert, but from what I've seen they've been working on similar tech (Nitro) for years.
- StillBored 3y agoIts worse than the article mentions. Because bandwidth isn't the problem its IOPS that are the problem. Last time (about a year ago) I ran a couple random IO benchmarks against a storage optimized instances and the random IOPs behavior is closer to a large spinning RAID array than SSDs if the disk size is over some threshold. IIRC, What it looks like is that there is a fast local SSD cache with a couple hundred GB of storage and then the rest is backed by remote spinning media. Its one of the many reasons I have a hard time taking cloud optimization seriously, the lack of direct tiering controls means that database/etc style workloads are not going to optimize well and that will end up costing a lot of $$$$$. So, maybe it was the instance types/configuration I was using, but <shrug> it was just something I was testing in passing.
- zokier 3y ago# fio --name=read_iops_test --filename=/dev/nvme1n1 --filesize=1500G --time_based --ramp_time=1s --runtime=15s --ioengine=io_uring --fixedbufs --direct=1 --verify=0 --randrep eat=0 --bs=4K --iodepth=256 --rw=randread --iodepth_batch_submit=256 --iodepth_batch_complete_max=256 read_iops_test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=256 fio-3.32 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=2082MiB/s][r=533k IOPS][eta 00m:00s] read_iops_test: (groupid=0, jobs=1): err= 0: pid=34235: Tue Feb 20 22:57:00 2024 read: IOPS=534k, BW=2086MiB/s (2187MB/s)(30.6GiB/15001msec) slat (nsec): min=713, max=255840, avg=31174.74, stdev=16248.45 clat (nsec): min=1419, max=1175.6k, avg=443782.26, stdev=277389.66 lat (usec): min=133, max=1240, avg=474.96, stdev=274.50 clat percentiles (usec): | 1.00th=[ 169], 5.00th=[ 198], 10.00th=[ 217], 20.00th=[ 243], | 30.00th=[ 265], 40.00th=[ 285], 50.00th=[ 306], 60.00th=[ 334], | 70.00th=[ 396], 80.00th=[ 865], 90.00th=[ 922], 95.00th=[ 947], | 99.00th=[ 996], 99.50th=[ 1012], 99.90th=[ 1045], 99.95th=[ 1057], | 99.99th=[ 1074] bw ( MiB/s): min= 2080, max= 2092, per=100.00%, avg=2086.72, stdev= 2.35, samples=30 iops : min=532548, max=535738, avg=534199.13, stdev=601.82, samples=30 lat (usec) : 2=0.01%, 100=0.01%, 250=23.06%, 500=50.90%, 750=0.28% lat (usec) : 1000=24.90% lat (msec) : 2=0.87% cpu : usr=14.17%, sys=67.83%, ctx=156851, majf=0, minf=37 IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0% submit : 0=0.0%, 4=7.8%, 8=11.3%, 16=39.7%, 32=30.6%, 64=10.5%, >=64=0.1% complete : 0=0.0%, 4=5.3%, 8=9.5%, 16=40.3%, 32=32.4%, 64=12.4%, >=64=0.1% issued rwts: total=8010661,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=256 Run status group 0 (all jobs): READ: bw=2086MiB/s (2187MB/s), 2086MiB/s-2086MiB/s (2187MB/s-2187MB/s), io=30.6GiB (32.8GB), run=15001-15001msec Disk stats (read/write): nvme1n1: ios=8542481/0, merge=0/0, ticks=3822266/0, in_queue=3822266, util=99.37% tldr: random 4k reads pretty much saturate the available 2GB/s bandwidth (this is on m6id)
- Ericson2314 3y agoThe cloud really is a scam for those afraid of hardware
- spintin 3y agoI'm going with hybrid: - 2011 X-25E 64GB (2W write and almost nothing read/idle) at 100.000 writes per bit for OS - 2021 PM897 3.7TB (2.3 Watt (read) ¦ 3 Watt (write) ¦ 1.4 Watt (idle) down from the PM983 (8.7 Watt (read) ¦ 10.6 Watt (write) ¦ 4 Watt (idle)) for DB. This way I can get the most robust solution, with largest DB at lowest power. They are both in a 8-core Atom Mini-ITX board at 25W TDP.
- ksec 3y agoThis got me thinking about the recent Rails improvement on moving cache from RAM to SSD. The test they did from what I remembered was RAM at 0.8ms and SSD was ~1.5ms. Moving to SSD you could afford to cache 10x more data and still be cheaper. Now I wonder if the test results would be the same on cloud. Assuming they tested it locally.
- samtheprogram 3y agoThey (Basecamp) recently moved off of cloud, so I’m wondering if they even care for their particular use case. I’m sure this is configurable in general though?
- aditya 3y agogot a link to the rails improvement? nvm, found it: https://dev.37signals.com/solid-cache/ https://dev.37signals.com/solid-cache/
- HippoBaro 3y agoAs a database engineer who worked extensively with both i3 and i4 instances, I want to add that although i4 has lower IOPS, the latencies distribution of IO ops is an order of magnitude better. IOPS indeed matters a lot, but so does latency! For our use case, it was much easier to saturate those disks than the old i3s, and we attribute it to the better latencies, making IO scheduling a lot more accurate.
- jnsaff2 3y agoThe article ends with the question about why the bandwidth is capped. I think the answer could be mclock scheduler. https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Gulati.pdf https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...
- vinay_ys 3y agoWell, as hardware becomes more and more powerful, what's possible in a small footprint becomes bigger and bigger. And distributed software for disaggregated storage is becoming more accessible. You put these two together, running on-prem footprints at the scale of 50-100M$ capex makes a lot of sense. In my personal experience, at this scale, (if your cloud bill for compute+storage+local-network is $50M+/year), you can get 2-4x more on-prem private-cloud capacity for the same money. Of course, this only makes sense if you have an in-house software engineering team already and the marginal cost of adding another 50-100 engineers to build and operate this is strategically valuable to your business. In big data AI space, this is exactly what's happening with the top 20th to 100th companies in the world right now.
- Temporary_31337 3y agoThe main reason is that cloud providers charge for IOPS
- wistlo 3y agoAt my job at a telco, I had a 13 billion record file to scan and index for duplicates and bad addresses. Consultants brought in to move our apps (some of which were Excel macros, others SAS scripts running on old desktop) to Azure. The Azure architects identified Postgres as the best tool. Consultants attempted to create a Postgres index in a small Azure instance but their tests would fail without completion (they were string concatenation rather than the native indexing function). Consultants' conclusion: file too big for Postgres. I disputed this. Plenty of literature out there on Pg handling bigger files. The Postgres (for Windows!) instance on my Core I7 laptop with an nVME drive could index the file about an hour. As an experiment I spun up a bare metal nVME instance on a Ryzen 7600 (lowest power, 6 core) Zen 4 CPU pc with a 1TB Samsung PCIe 4 nVME drive. Got my index in 10 minutes. I then tried to replicate this in Azure, upping the CPUs, memory, and to the nVME Azure CPU family (Ebsv5). Even at a $2000/mo level, I could not get the Azure instance any faster than one fifth (about an hour) of the speed of my bare metal experiment. I probably could have matched it eventually with more cores, but did not want to get called on the carpet for a ten grand Azure bill. All this happened while I was working from home (one can't spin up an experimental bare metal system at a drop-in spot in the communal workroom). What happened next I don't know, because I left in the midst of RTO fever. I was given the option of moving 1000 miles to commute to a hub office, or retire "voluntarily with severance." I chose the latter.
- bcaxis 3y agoThis matches my experience. Something about cloud systems makes them incredibly slow compared to real hardware. Not just the disk, the CPU is more limited too. I fire up vCPU or dedicated or bare metal in the cloud, doesn't matter, I simply cannot match the equivalent compute of real hardware and it's not even close.
- silent_cal 3y agoIsn't that expected? I would assume cloud stuff is slower because it's essentially an emulation of the real thing.
- 3y ago