4 ms·

Am I the only one seeing a lot of advantages with local storage? I don't think it's idiosyncratic at all – that's how DigitalOcean became the company it is toda

by nik736 2y ago

Am I the only one seeing a lot of advantages with local storage? I don't think it's idiosyncratic at all – that's how DigitalOcean became the company it is today, with simple local storage VMs. The performance of local NVMes is way better, it is more predictable and you don't have to factor in latency, bandwidth, network issues, bottlenecks and more. Redundancy can be achieved with a multi host setup, so even if the host fails the underlying application or database is not impacted. The one disadvantage I can see is that you can't "scale"/change the underlying disks. The disks you bought when setting up the server are probably there to stay.

jandrewrogers 2y ago
No, you are correct. The database performance difference between using local NVMe and network-attached storage is integer factor on what is otherwise the same hardware. It would be surprising if this was not the case since storage bandwidth of local NVMe is typically integer factor higher and the performance of modern database engines is strongly correlated with available storage bandwidth for many workloads. The network-attached storage does some things for you that local NVMe cannot, like easily moving storage between hosts, but it is also relatively expensive. Nonetheless, it can makes sense if performance and minimizing hardware footprint is not a major concern. If your main business is data-intensive on the other hand, the economics of network-attached storage start to make a lot less sense.
adrian_b 2y ago
Most good servers use removable SSDs or HDDs, which can be replaced or moved from one server to another during the normal operation of the servers, without shutting them down. Even the consumer SATA connectors are designed for hot-plugging, which is also true for the enterprise NVMe or SAS connectors. Only the M.2 connectors are an exception among the modern SSD connectors, because they do not support hot-plugging, which is why they are not used in serious servers. The only advantage of shared storage that is concentrated in some special place is that when such operations like replacement or moving are required, which require the presence of a human, the human may need to walk a much shorter distance to the rack with shared storage than if passing by each server had been required.
- nik736 2y ago
  I know. It's called hot swapping. But what I meant is the size. Having 6x 3.84 TB NVMe in an existing raid array and wanting to upgrade to 6x 7.68 TB is possible but not as straight forward.
  - adrian_b 2y ago
    It would be less straight forward only if you have a single server and that server has only a few SSD slots, which are all occupied. Otherwise, you can just move the old SSDs to another server, plug the new SSDs, copy the old content in the new SSDs and that is it. No reboot needed. Normally any server should have a separate small boot SSD, typically an M.2 SSD on the MB or sometimes a small USB memory plugged internally in the MB, with the operating system, which is not affected in any way by the mounting or unmounting of the other SSDs or HDDs that contain user data. When the second server does not have free SSD slots, it must be taken temporarily offline and its own SSDs must be removed until the old SSDs of the first server are copied. In the worst case one would have to do three pairs of SSD unplugging and plugging multiplied with the number of SSDs in one RAID array, six in your example, to replace the SSDs in the first server with the new SSDs, then replace the SSDs in the second server with the old SSDs, then restore the SSDs of the second server. Even so, I would consider such a procedure as straight forward, as all the operations would take a few minutes at most, which would be a much shorter time than required for the copying of the old content to the new SSDs. When you have a single server, then the alternative of using shared storage would also not exist. Except for home servers, the use of single servers should be rare, because even when a single server has enough capacity there should be a backup for it.
ngrilly 2y ago
I fully agree with you, for what it's worth. I really like what Fly.io is doing here. Most of the advantages of being closer to the metal, combined to most advantages of being in the cloud.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]

by nik736 2y ago

jandrewrogers 2y ago

No, you are correct. The database performance difference between using local NVMe and network-attached storage is integer factor on what is otherwise the same hardware. It would be surprising if this was not the case since storage bandwidth of local NVMe is typically integer factor higher and the performance of modern database engines is strongly correlated with available storage bandwidth for many workloads. The network-attached storage does some things for you that local NVMe cannot, like easily moving storage between hosts, but it is also relatively expensive. Nonetheless, it can makes sense if performance and minimizing hardware footprint is not a major concern. If your main business is data-intensive on the other hand, the economics of network-attached storage start to make a lot less sense.

adrian_b 2y ago

Most good servers use removable SSDs or HDDs, which can be replaced or moved from one server to another during the normal operation of the servers, without shutting them down. Even the consumer SATA connectors are designed for hot-plugging, which is also true for the enterprise NVMe or SAS connectors. Only the M.2 connectors are an exception among the modern SSD connectors, because they do not support hot-plugging, which is why they are not used in serious servers. The only advantage of shared storage that is concentrated in some special place is that when such operations like replacement or moving are required, which require the presence of a human, the human may need to walk a much shorter distance to the rack with shared storage than if passing by each server had been required.

nik736 2y ago
I know. It's called hot swapping. But what I meant is the size. Having 6x 3.84 TB NVMe in an existing raid array and wanting to upgrade to 6x 7.68 TB is possible but not as straight forward.
- adrian_b 2y ago
  It would be less straight forward only if you have a single server and that server has only a few SSD slots, which are all occupied. Otherwise, you can just move the old SSDs to another server, plug the new SSDs, copy the old content in the new SSDs and that is it. No reboot needed. Normally any server should have a separate small boot SSD, typically an M.2 SSD on the MB or sometimes a small USB memory plugged internally in the MB, with the operating system, which is not affected in any way by the mounting or unmounting of the other SSDs or HDDs that contain user data. When the second server does not have free SSD slots, it must be taken temporarily offline and its own SSDs must be removed until the old SSDs of the first server are copied. In the worst case one would have to do three pairs of SSD unplugging and plugging multiplied with the number of SSDs in one RAID array, six in your example, to replace the SSDs in the first server with the new SSDs, then replace the SSDs in the second server with the old SSDs, then restore the SSDs of the second server. Even so, I would consider such a procedure as straight forward, as all the operations would take a few minutes at most, which would be a much shorter time than required for the copying of the old content to the new SSDs. When you have a single server, then the alternative of using shared storage would also not exist. Except for home servers, the use of single servers should be rare, because even when a single server has enough capacity there should be a backup for it.

ngrilly 2y ago

I fully agree with you, for what it's worth. I really like what Fly.io is doing here. Most of the advantages of being closer to the metal, combined to most advantages of being in the cloud.

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]