4 ms·

I don't understand, why is spinning metal not good enough? For the large amount of storage required it seems like it would be a good idea.

by tomc1985 5y ago

vvpan 5y ago
Sometime in 2017/18 it has been found that one of the biggest bottlenecks for the network were propagation speeds and that those were largely weighed down by disk writes. Thus pretty much the whole network switched to SSDs and things improved tremendously. Some other networks have implemented stateless clients and Ethereum has been working on going in that direction for a while with a lot of effort being dedicated to it this year specifically.
- eminence32 5y ago
  Does the node need bulk write speeds? or does it need fast IO ops per second?
  - trianglesphere 5y ago
    It needs low latency. There is a huge amount of inherently sequential lookups in the state (~9 leveldb lookups per 1 state acess in a smart contract) and you can't store the state in RAM on most consumer hardware.
  - jlokier 5y ago
    It's read as well as write IOPS and seek time. The Ethereum state database uses hashes as keys and is larger than anyone's reasonable amount of RAM. So most accesses are all over the database in terms of physical location, and it can't be cached in RAM effectively. There is various work to improve this, but the basic architecture does depend on random access storage to a great extent.
- nly 5y ago
  Doesn't make sense. Most people have faster spinning rust write speeds than they do internet bandwidth. Unless the client is generating a lot of data client side, how can writes ever be a problem? Just buffer appropriately
  - deleted 5y ago
    [deleted]
  - delaaxe 5y ago
    It's not like clients just download blobs and dump them on disk. They download programs that they execute and then store the updated state on disk
3np 5y ago
If you use spinning rust for the initial sync you’ll most likely never catch up. If you have craploads of RAM to allocate to ARC, you may actually be able to do it on ZFS I guess.
- jlokier 5y ago
  I think this may no longer be true with the Snap sync protocol (introduced in Geth this year). Even though transactions are very random access, Snap iterates sequentially over storage to transfer it all from one node to another, which is HDD friendly. However I'm not confident it will be able to stay in sync afterwards using HDD.
  - yokem55 5y ago
    The new fork of geth, called erigon (formerly turbogeth) supposedly can do a full archive sync on a spinning disk drive and is much more space efficient.
jlokier 5y ago
It's a great idea. It just doesn't work. HDDs don't have the right performance characteristics, unfortunately. You can do about 300 random access per second on a good HDD, which isn't enough to keep up with Ethereum's transaction rate, and each transaction needs quite a few of those each. You can do about 300,000 per second on NVMe. 1000 times faster. That's plenty.
- lupire 5y ago
  what happens when eth get popular and nvme can't keep up?
  - deleted 5y ago
    [deleted]
  - Freestyler_3 5y ago
    Multiply, or live in ram?
xur17 5y ago
You can actually put the older, less frequently accessed data on a spinning disk using this parameter: > --datadir.ancient /media/disk1/goethereum/ancient I've been using it, and I have 225gb on my ssd, 229gb on my regular hard drive.

I don't understand, why is spinning metal not good enough? For the large amount of storage required it seems like it would be a good idea.

vvpan 5y ago

Sometime in 2017/18 it has been found that one of the biggest bottlenecks for the network were propagation speeds and that those were largely weighed down by disk writes. Thus pretty much the whole network switched to SSDs and things improved tremendously. Some other networks have implemented stateless clients and Ethereum has been working on going in that direction for a while with a lot of effort being dedicated to it this year specifically.

eminence32 5y ago
Does the node need bulk write speeds? or does it need fast IO ops per second?
- trianglesphere 5y ago
  It needs low latency. There is a huge amount of inherently sequential lookups in the state (~9 leveldb lookups per 1 state acess in a smart contract) and you can't store the state in RAM on most consumer hardware.
- jlokier 5y ago
  It's read as well as write IOPS and seek time. The Ethereum state database uses hashes as keys and is larger than anyone's reasonable amount of RAM. So most accesses are all over the database in terms of physical location, and it can't be cached in RAM effectively. There is various work to improve this, but the basic architecture does depend on random access storage to a great extent.
nly 5y ago
Doesn't make sense. Most people have faster spinning rust write speeds than they do internet bandwidth. Unless the client is generating a lot of data client side, how can writes ever be a problem? Just buffer appropriately
- deleted 5y ago
  [deleted]
- delaaxe 5y ago
  It's not like clients just download blobs and dump them on disk. They download programs that they execute and then store the updated state on disk

3np 5y ago

If you use spinning rust for the initial sync you’ll most likely never catch up. If you have craploads of RAM to allocate to ARC, you may actually be able to do it on ZFS I guess.

jlokier 5y ago
I think this may no longer be true with the Snap sync protocol (introduced in Geth this year). Even though transactions are very random access, Snap iterates sequentially over storage to transfer it all from one node to another, which is HDD friendly. However I'm not confident it will be able to stay in sync afterwards using HDD.
- yokem55 5y ago
  The new fork of geth, called erigon (formerly turbogeth) supposedly can do a full archive sync on a spinning disk drive and is much more space efficient.

jlokier 5y ago

It's a great idea. It just doesn't work. HDDs don't have the right performance characteristics, unfortunately. You can do about 300 random access per second on a good HDD, which isn't enough to keep up with Ethereum's transaction rate, and each transaction needs quite a few of those each. You can do about 300,000 per second on NVMe. 1000 times faster. That's plenty.

lupire 5y ago
what happens when eth get popular and nvme can't keep up?
- deleted 5y ago
  [deleted]
- Freestyler_3 5y ago
  Multiply, or live in ram?

xur17 5y ago

You can actually put the older, less frequently accessed data on a spinning disk using this parameter: > --datadir.ancient /media/disk1/goethereum/ancient I've been using it, and I have 225gb on my ssd, 229gb on my regular hard drive.