7 ms·

For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an

For databases, local storage beats SAN storage massively. I can get better performance and IOPS from an Intel NUC with a decent PCIe SSD than I can get from an AWS RDS instance that costs as much per month as the NUC did to buy. If I optimize a proper rack mount server for database I can get an insane amount and storage and performance compared to even a few months of RDS. Even a pair of 40 Gbps SAN links would not compare.

YZF 8y ago
I'm not sure this is necessarily true. 2 x 40Gbps is bandwidth which is typically not the limiting factor. If it was you can go with higher bandwidth like FC. RDS is a database service, not SAN. Look at SAN boxes from storage companies, you can get things like 60 SSDs in a single box. To match the IOPs of that you'd need a lot of servers with local storage. I think in general dis-aggregating compute and storage is the optimal approach, whether some particular solution is better/price effective is a different question. Having all your drives in a box means they're easier to share, easier to service, easier to replace the servers as well ... lots of wins there.
- Aeolun 8y ago
  I’m sure you can have some 12 drives in the same box as your database.
- latch 8y ago
  If you're doing it locally, you'd ideally use RAID with a battery backed unit and your fsync essentially end up going to memory.
  - zbentley 8y ago
    Until a battery relearn takes your app down in the middle of the night. I don't miss those.
  - sharms 8y ago
    The servers I buy now have large capacitors which don't fail like batteries, and you can then have the server on a UPS
justinclift 8y ago
> local storage beats SAN storage massively Only if your (storage) network is slow (eg 1/10/25 GbE). With a decent network (eg modern infiniband, 40+GbE, etc) for the storage, the latency and throughput to the storage shouldn't make a difference. For example (years ago), I used to set up SSD arrays - SATA at the time, as M.2 wasn't a thing - and have them served over a 20Gbs Infiniband network to hosts in the same data centre. The access times from an OS perspective to hit that storage over the network were the same as for hitting local disk. But the networked storage was higher bandwidth (multiple SSD's, instead of a single per host). Worked really well. :)
- justinclift 8y ago
  For reference, the SCST project was the software used (back-in-the-day) for high performance block storage over infiniband: http://scst.sourceforge.net http://scst.sourceforge.net Looks like it's still an active project too, as there's a new release listed from November 2018.
a11595 8y ago
Don't know where you are getting this from, but it's literally not how every enterprise-level company runs. You have something like and EMC Symmetrix all flash full of optane or nvme flash drives - hundreds of them. your data is striped across all of them. they're connected to something like a cisco mds. usually you have 4 paths to each LUN, and they're usually 32Gb FC. you can certainly have as many paths as you want if you need more bandwidth. The array also has 10-20TB of cache to front-end the flash. your response time is about 100-200 microseconds. I don't know how much ram or disk you're going to shove into your server, but good luck running a bunch of half-petabyte databases in it, and expanding it for growth. this is literally how 90% the world runs. your little theories are cute though.
- manigandham 8y ago
  Leave out the last sentence next time, that attitude overshadows your entire comment and isn't a good fit for HN.
  - a11595 8y ago
    Not a chance buddy. I've been here for a while, and this place has become reddit, with mods from /r/incels. I have two accounts. When I see someone with zero knowledge of subject matter, like OP, authoritatively and dismissively spew complete bs, I reply like I do. People with a clear agenda spewing fake crap gets upvoted here now. Disagree with a mod - your comments are shadow removed, and possibly your account is shadow banned, with no warning, so the insecure mod can feel better. I give actual valuable information from 20 years of experience at almost all the fortune100 firms. Snarkness? Yes. Much better and funnier than the guy I replied to. I know you disagree. That's specifically why I include sentences like the last - to draw attention to the issue. comment karma has become useless here. Stuff is greyed out, stuff is shadow deleted - good things, filtered by idiots. I don't want my content filtered by insecure idiots.
    - jordanbeiber 8y ago
      Hey budz, many around here have 20+ years of experience. I've fought all my career to get "enterprise" folks to understand that spending $100s of mill on hardware vendors and 3rd party software is madness, if what you are doing is actually mission critical. Money should be spend on acquiring talent and adapting to actual business needs. The business you support could not care less about your shiny toys, they want to transform, and probably yesteryear! A digression here, from one of your other posts: if your business applications rely on live migration of vm's, you're doing it wrong. You should take this up with your CIO or even better the CTO, but he's probably busy looking at Gartner magic quadrants and contemplating the dire need of a blockchain. Keep an open mind, and keep it simple. Best advice ever in this crazy business!
KaiserPro 8y ago
There are a few issues here. EBS is a compromise, its allows your data to run anywhere in a region, on any machine. That means lots of hops and lots of interconnects. the latest SAS stuff runs at 12 gigs(most likely 4 lanes per cable, and dual linked too.), _but_ thats dedicated for local traffic. The performance difference between having a SAS disk inside a box, or in the next rack is negligible. A decent SAN that exports over a 56gig connect (inifiniband et al) will be >> than a local pcie in terms of iops and sustained bandwith (at the expense of latency.) Crucially, in somewhere that runs it's own datacenter, the storage is modelled for a specific workload. In VFX land we had 32 60 disk file servers, each capable of saturating two 10 gig ethernet links. But it'd be appalling for mixed VM hosting. EBS is a hedged bet, it had a _boatload_ of caching to make any sort of performance. It had a huge amount of QoS to stop selfish loads stealing all the IOPs. To get the best performance from EBS you have to have large volumes (5Tb+) and large latest generation instances (c5, r5, m5) 4 or 8/9 xlarge ec2 instances. This costs money to run.
- KaiserPro 8y ago
  > its allows your data to run anywhere in a region that should read AZ
- pas 8y ago
  This sort of portability is usually handled by the DB routing layer in cloud native ("webscale") database deployments. eg in Cassandra. of course there are a lot of systems where one MySQL DB instance is needed and sufficient for the forseeable future. so that's a very static resource allocation one VM on a hypervisor with local disk, (and with backups every day), makes a lot of smaller sites very happy, and if they get the local provisioned performance instead of the EBS variant they will be happier for longer. of course, EBS is a lot more fault tolerant.
manigandham 8y ago
There's a wide range of performance with storage systems but why are you comparing that to RDS? That's a managed database service that just runs on EC2 instances with EBS volumes, and EBS is designed for affordable and scalable persistence instead of extreme performance. EBS is not your typical SAN, which usually offer much more throughput, IOPS, and reliability in exchange for more complexity and latency, however you probably won't even notice that latency if your SAN system is close enough and using highend links.
tw04 8y ago
That has almost nothing to do with SAN and everything to do with Amazon's implementation. A nuc will never give you the resiliency or performance of a properly sized SAN. Furthermore, you'll never get close to saturating a 40gbps link with a database workload, your limitation is iops, not throughput, and you don't have anywhere near enough cpu to max out those links in a loaded server much less a nuc.

YZF 8y ago

I'm not sure this is necessarily true. 2 x 40Gbps is bandwidth which is typically not the limiting factor. If it was you can go with higher bandwidth like FC. RDS is a database service, not SAN. Look at SAN boxes from storage companies, you can get things like 60 SSDs in a single box. To match the IOPs of that you'd need a lot of servers with local storage. I think in general dis-aggregating compute and storage is the optimal approach, whether some particular solution is better/price effective is a different question. Having all your drives in a box means they're easier to share, easier to service, easier to replace the servers as well ... lots of wins there.

Aeolun 8y ago
I’m sure you can have some 12 drives in the same box as your database.
latch 8y ago
If you're doing it locally, you'd ideally use RAID with a battery backed unit and your fsync essentially end up going to memory.
- zbentley 8y ago
  Until a battery relearn takes your app down in the middle of the night. I don't miss those.
- sharms 8y ago
  The servers I buy now have large capacitors which don't fail like batteries, and you can then have the server on a UPS

justinclift 8y ago

> local storage beats SAN storage massively Only if your (storage) network is slow (eg 1/10/25 GbE). With a decent network (eg modern infiniband, 40+GbE, etc) for the storage, the latency and throughput to the storage shouldn't make a difference. For example (years ago), I used to set up SSD arrays - SATA at the time, as M.2 wasn't a thing - and have them served over a 20Gbs Infiniband network to hosts in the same data centre. The access times from an OS perspective to hit that storage over the network were the same as for hitting local disk. But the networked storage was higher bandwidth (multiple SSD's, instead of a single per host). Worked really well. :)

justinclift 8y ago
For reference, the SCST project was the software used (back-in-the-day) for high performance block storage over infiniband: http://scst.sourceforge.net http://scst.sourceforge.net Looks like it's still an active project too, as there's a new release listed from November 2018.

a11595 8y ago

Don't know where you are getting this from, but it's literally not how every enterprise-level company runs. You have something like and EMC Symmetrix all flash full of optane or nvme flash drives - hundreds of them. your data is striped across all of them. they're connected to something like a cisco mds. usually you have 4 paths to each LUN, and they're usually 32Gb FC. you can certainly have as many paths as you want if you need more bandwidth. The array also has 10-20TB of cache to front-end the flash. your response time is about 100-200 microseconds. I don't know how much ram or disk you're going to shove into your server, but good luck running a bunch of half-petabyte databases in it, and expanding it for growth. this is literally how 90% the world runs. your little theories are cute though.

manigandham 8y ago
Leave out the last sentence next time, that attitude overshadows your entire comment and isn't a good fit for HN.
- a11595 8y ago
  Not a chance buddy. I've been here for a while, and this place has become reddit, with mods from /r/incels. I have two accounts. When I see someone with zero knowledge of subject matter, like OP, authoritatively and dismissively spew complete bs, I reply like I do. People with a clear agenda spewing fake crap gets upvoted here now. Disagree with a mod - your comments are shadow removed, and possibly your account is shadow banned, with no warning, so the insecure mod can feel better. I give actual valuable information from 20 years of experience at almost all the fortune100 firms. Snarkness? Yes. Much better and funnier than the guy I replied to. I know you disagree. That's specifically why I include sentences like the last - to draw attention to the issue. comment karma has become useless here. Stuff is greyed out, stuff is shadow deleted - good things, filtered by idiots. I don't want my content filtered by insecure idiots.
  - jordanbeiber 8y ago
    Hey budz, many around here have 20+ years of experience. I've fought all my career to get "enterprise" folks to understand that spending $100s of mill on hardware vendors and 3rd party software is madness, if what you are doing is actually mission critical. Money should be spend on acquiring talent and adapting to actual business needs. The business you support could not care less about your shiny toys, they want to transform, and probably yesteryear! A digression here, from one of your other posts: if your business applications rely on live migration of vm's, you're doing it wrong. You should take this up with your CIO or even better the CTO, but he's probably busy looking at Gartner magic quadrants and contemplating the dire need of a blockchain. Keep an open mind, and keep it simple. Best advice ever in this crazy business!

KaiserPro 8y ago

There are a few issues here. EBS is a compromise, its allows your data to run anywhere in a region, on any machine. That means lots of hops and lots of interconnects. the latest SAS stuff runs at 12 gigs(most likely 4 lanes per cable, and dual linked too.), _but_ thats dedicated for local traffic. The performance difference between having a SAS disk inside a box, or in the next rack is negligible. A decent SAN that exports over a 56gig connect (inifiniband et al) will be >> than a local pcie in terms of iops and sustained bandwith (at the expense of latency.) Crucially, in somewhere that runs it's own datacenter, the storage is modelled for a specific workload. In VFX land we had 32 60 disk file servers, each capable of saturating two 10 gig ethernet links. But it'd be appalling for mixed VM hosting. EBS is a hedged bet, it had a _boatload_ of caching to make any sort of performance. It had a huge amount of QoS to stop selfish loads stealing all the IOPs. To get the best performance from EBS you have to have large volumes (5Tb+) and large latest generation instances (c5, r5, m5) 4 or 8/9 xlarge ec2 instances. This costs money to run.

KaiserPro 8y ago
> its allows your data to run anywhere in a region that should read AZ
pas 8y ago
This sort of portability is usually handled by the DB routing layer in cloud native ("webscale") database deployments. eg in Cassandra. of course there are a lot of systems where one MySQL DB instance is needed and sufficient for the forseeable future. so that's a very static resource allocation one VM on a hypervisor with local disk, (and with backups every day), makes a lot of smaller sites very happy, and if they get the local provisioned performance instead of the EBS variant they will be happier for longer. of course, EBS is a lot more fault tolerant.

manigandham 8y ago

There's a wide range of performance with storage systems but why are you comparing that to RDS? That's a managed database service that just runs on EC2 instances with EBS volumes, and EBS is designed for affordable and scalable persistence instead of extreme performance. EBS is not your typical SAN, which usually offer much more throughput, IOPS, and reliability in exchange for more complexity and latency, however you probably won't even notice that latency if your SAN system is close enough and using highend links.

tw04 8y ago

That has almost nothing to do with SAN and everything to do with Amazon's implementation. A nuc will never give you the resiliency or performance of a properly sized SAN. Furthermore, you'll never get close to saturating a 40gbps link with a database workload, your limitation is iops, not throughput, and you don't have anywhere near enough cpu to max out those links in a loaded server much less a nuc.