4 ms·

It takes me about a day to sync parity on a consumer desktop with an ssd and a 7 year old i5 Intel cpu

by snissn 8y ago

just_one_time_ 8y ago
I worry that disk space will stop increasing and manufacturers will stop producing hard drives one day, what will Ethereum do then?
erulabs 8y ago
For me, just about 18 hours with parity on a fairly new i7 and fairly fast (200mbps) net and NVMe disks. Haven't repeated enough to be sure, but I can't figure out how to replicate the "weeks" argument I keep hearing. I suppose on average American broadband (~15mbps) and spinning HDDs the week-long numbers start to seem reasonable?
- erulabs 8y ago
  Edit: It seems (based on the conversations in this thread) the argument is versus "full", "fast" and "light" sync methods. Full implies personally reverifying the entire chain and every transaction, which takes a week or more. Fast pulls down the entire chain (and all of its data), and takes ~18 hours (what I mentioned above), but still results in a complete Ethereum node. It becomes a philosophical argument if the "fast" sync method implies "enough trust", or if everyone should be expected to reprocess every transaction ever. Personally (and i'm not a mathematician) from what I understand of merkle trees, as additional transactions are verified, the "trust in the past" grows, as it would be increasingly difficult to have lied about (or rather, to change) a previous transaction. Philosophy aside, it's really painless and takes less than a day to get a Ethereum node up and running and chugging away, hard stop.
  - bouncycastle 8y ago
    My understanding that a full sync in Bitcoin means something different than a full sync on Ethereum. Bitcoin's database is modeled on UTXO (https://en.m.wikipedia.org/wiki/Unspent_transaction_output https://en.m.wikipedia.org/wiki/Unspent_transaction_output) therefore all clients need to validate all transactions from genesis to calculate final balances. Meanwhile, Ethereum is account based and state is global, and to determine the balances you only need to check the merkle proof, no need to store or re-validate each and every transaction, just the final result. Source: https://github.com/ethereum/wiki/wiki/Design-Rationale#accounts-and-not-utxos https://github.com/ethereum/wiki/wiki/Design-Rationale#accou... quote "The benefits of accounts are:Large space savings" In Ethereum, the closet to a Bitcoin full node would be an 'full archive' node, those are keeping all the transactions, down to the stack traces and events, without pruning. These are usually run by block explorers, but you certainly don't need to store that much detail if you just want to interact with a dapps.
    - CryptoPunk 8y ago
      Bitcoin nodes could download the most recent UTXO index too. Validating the entire history is necessary in both if you want to trust nothing except the client code you're given. An archival node in Ethereum goes beyond a full node, and stores data not needed to do full validation. A full node in Ethereum needs to download about 80 GB I believe, which is a tenth of what an archival node needs to download and store.
      - bouncycastle 8y ago
        archival node doesn't need to download anything extra. It just reconstructs everything just from the blocks.
    - keymone 8y ago
      You say “check merkle proof”, how do you know what it’s supposed to be? Trust somebody gives you the right hash? There is no difference between BTC’s UTXO model and Ethereum’s account model in this sense - both are aggregate structures and to have full certainty the aggregate you’re looking at is correct you have to run through all the transactions. To speedup sync in BTC it is assumed that blocks older than some amount of YEARS are valid but you still have to go through their transactions to build up the aggregate UTXO set. In ETH instead it is common to just download the result and trust it’s valid. And since it’s now impossible to sync from origin you can never tell if some account doesn’t have ether created from nothing.
      - bouncycastle 8y ago
        > how do you know what it’s supposed to be The information is in the block headers. > In ETH instead it is common to just download the result and trust it’s valid Nope. You still have to download all the block headers from block 0 and check if they are valid, including the PoW and other hashes.. The blocks are basically tamper proof as in Bitcoin (but can also be vulnerable to a 51% attack) Then that's where it all diverges from Bitcoin: in Ethereum, there are three merkle trees: one for the state, one for transactions and one for the transaction results (receipts). The roots of these trees are in the blocks, once you have these, you can verify a particular value of, say, a state value, without the need to verify each and every other value, by walking through the tree and doing a merkle proof. There's plenty of info on the web explaining how it works if you look it up, eg. Search for "ethereum state trie" As for syncing, recently it improved a lot. I did it a few times last month with Geth on an old machine just to test it out. Synced within 2 days, a big improvement over the previous versions of Geth. And yes, it got all the blocks from 0 to current. Blocks by themselves are very light.
nostrademons 8y ago
Similar experience with geth on an m5.large EC2 instance and 500G SSD. The SSD vs. hard disk distinction is key. I ran into the "it will literally never complete" issue trying to sync to an external HD on my home MBP - I ran it for about a week and it didn't work. Tried parity as well, same result. Once I decided I'd just rent a big SSD from Amazon I had no problem.
- xiphias2 8y ago
  Were you passing --gcmode=archive --syncmode=full flags? If not, then it's just a light node. Here's an example: https://github.com/ethereum/go-ethereum/issues/16237 https://github.com/ethereum/go-ethereum/issues/16237 ,,Seems like it does indeed proceed but very slowly. After 12 days on the 96gig ram machine I am finally at block 4732597... I will report back how long it takes to finish.''
  - nostrademons 8y ago
    No, I was doing a fast sync (but not a light sync, which is different). I wanted and got the transaction history & full code for each smart contract on the chain, but didn't care about verification of each transaction in the chain. You can run a full node that verifies everything on an ongoing basis without doing a full sync from the genesis block - just start it in fast sync, let the chain catch up, and then run it again in full mode. Unless you believe the Ethereum blockchain has already been compromised, there's no reason why your personal node has to verify everything from the genesis block.
  - thinkmassive 8y ago
    gcmode=archive is not required for a full node. It indexes every transaction in a database, which is unnecessary for most applications. It’s similar to txindex in bitcoind, but the Ethereum db is already much larger, so it takes weeks to sync in this mode.
    - xiphias2 8y ago
      I see, the problem was that it's hard to find documentation about what's needed for an Ethereum full node. As I'm just running Bitcoin full node, I don't have any personal experience with it.
xiphias2 8y ago
I guess you're not doing a full sync. Here's an example of a person trying to do it: https://ethereum.stackexchange.com/questions/37348/parity-full-history-node https://ethereum.stackexchange.com/questions/37348/parity-fu...
- CryptoPunk 8y ago
  An archival node goes beyond a full node and downloads data not needed for full validation.
  - nootropicat 8y ago
    It doesn't download them, the state history is generated from the blockchain locally. The blockchain itself is 68GB (uncompressed, decimal G)
- nootropicat 8y ago
  Ssd is needed (or enough ram) due to random access of state. See my reddit post from a year ago, full sync took 25 hours: https://www.reddit.com/r/ethereum/comments/7n4v64/correcting_rbitcoins_misinformation_i_just_fully/ https://www.reddit.com/r/ethereum/comments/7n4v64/correcting... There's a lot of misinformation, some originating due to genuine problems (trying to use a hdd or geth with barely working garbage collector) and misunderstandings, some from intentional lies (especially two r/bitcoin moderators - thieflar and stopanddecrypt).

It takes me about a day to sync parity on a consumer desktop with an ssd and a 7 year old i5 Intel cpu

by snissn 8y ago

It takes me about a day to sync parity on a consumer desktop with an ssd and a 7 year old i5 Intel cpu

I worry that disk space will stop increasing and manufacturers will stop producing hard drives one day, what will Ethereum do then?

erulabs 8y ago

For me, just about 18 hours with parity on a fairly new i7 and fairly fast (200mbps) net and NVMe disks. Haven't repeated enough to be sure, but I can't figure out how to replicate the "weeks" argument I keep hearing. I suppose on average American broadband (~15mbps) and spinning HDDs the week-long numbers start to seem reasonable?

erulabs 8y ago
Edit: It seems (based on the conversations in this thread) the argument is versus "full", "fast" and "light" sync methods. Full implies personally reverifying the entire chain and every transaction, which takes a week or more. Fast pulls down the entire chain (and all of its data), and takes ~18 hours (what I mentioned above), but still results in a complete Ethereum node. It becomes a philosophical argument if the "fast" sync method implies "enough trust", or if everyone should be expected to reprocess every transaction ever. Personally (and i'm not a mathematician) from what I understand of merkle trees, as additional transactions are verified, the "trust in the past" grows, as it would be increasingly difficult to have lied about (or rather, to change) a previous transaction. Philosophy aside, it's really painless and takes less than a day to get a Ethereum node up and running and chugging away, hard stop.
- bouncycastle 8y ago
  My understanding that a full sync in Bitcoin means something different than a full sync on Ethereum. Bitcoin's database is modeled on UTXO (https://en.m.wikipedia.org/wiki/Unspent_transaction_output https://en.m.wikipedia.org/wiki/Unspent_transaction_output) therefore all clients need to validate all transactions from genesis to calculate final balances. Meanwhile, Ethereum is account based and state is global, and to determine the balances you only need to check the merkle proof, no need to store or re-validate each and every transaction, just the final result. Source: https://github.com/ethereum/wiki/wiki/Design-Rationale#accounts-and-not-utxos https://github.com/ethereum/wiki/wiki/Design-Rationale#accou... quote "The benefits of accounts are:Large space savings" In Ethereum, the closet to a Bitcoin full node would be an 'full archive' node, those are keeping all the transactions, down to the stack traces and events, without pruning. These are usually run by block explorers, but you certainly don't need to store that much detail if you just want to interact with a dapps.
  - CryptoPunk 8y ago
    Bitcoin nodes could download the most recent UTXO index too. Validating the entire history is necessary in both if you want to trust nothing except the client code you're given. An archival node in Ethereum goes beyond a full node, and stores data not needed to do full validation. A full node in Ethereum needs to download about 80 GB I believe, which is a tenth of what an archival node needs to download and store.
    - bouncycastle 8y ago
      archival node doesn't need to download anything extra. It just reconstructs everything just from the blocks.
  - keymone 8y ago
    You say “check merkle proof”, how do you know what it’s supposed to be? Trust somebody gives you the right hash? There is no difference between BTC’s UTXO model and Ethereum’s account model in this sense - both are aggregate structures and to have full certainty the aggregate you’re looking at is correct you have to run through all the transactions. To speedup sync in BTC it is assumed that blocks older than some amount of YEARS are valid but you still have to go through their transactions to build up the aggregate UTXO set. In ETH instead it is common to just download the result and trust it’s valid. And since it’s now impossible to sync from origin you can never tell if some account doesn’t have ether created from nothing.
    - bouncycastle 8y ago
      > how do you know what it’s supposed to be The information is in the block headers. > In ETH instead it is common to just download the result and trust it’s valid Nope. You still have to download all the block headers from block 0 and check if they are valid, including the PoW and other hashes.. The blocks are basically tamper proof as in Bitcoin (but can also be vulnerable to a 51% attack) Then that's where it all diverges from Bitcoin: in Ethereum, there are three merkle trees: one for the state, one for transactions and one for the transaction results (receipts). The roots of these trees are in the blocks, once you have these, you can verify a particular value of, say, a state value, without the need to verify each and every other value, by walking through the tree and doing a merkle proof. There's plenty of info on the web explaining how it works if you look it up, eg. Search for "ethereum state trie" As for syncing, recently it improved a lot. I did it a few times last month with Geth on an old machine just to test it out. Synced within 2 days, a big improvement over the previous versions of Geth. And yes, it got all the blocks from 0 to current. Blocks by themselves are very light.

nostrademons 8y ago

Similar experience with geth on an m5.large EC2 instance and 500G SSD. The SSD vs. hard disk distinction is key. I ran into the "it will literally never complete" issue trying to sync to an external HD on my home MBP - I ran it for about a week and it didn't work. Tried parity as well, same result. Once I decided I'd just rent a big SSD from Amazon I had no problem.

xiphias2 8y ago
Were you passing --gcmode=archive --syncmode=full flags? If not, then it's just a light node. Here's an example: https://github.com/ethereum/go-ethereum/issues/16237 https://github.com/ethereum/go-ethereum/issues/16237 ,,Seems like it does indeed proceed but very slowly. After 12 days on the 96gig ram machine I am finally at block 4732597... I will report back how long it takes to finish.''
- nostrademons 8y ago
  No, I was doing a fast sync (but not a light sync, which is different). I wanted and got the transaction history & full code for each smart contract on the chain, but didn't care about verification of each transaction in the chain. You can run a full node that verifies everything on an ongoing basis without doing a full sync from the genesis block - just start it in fast sync, let the chain catch up, and then run it again in full mode. Unless you believe the Ethereum blockchain has already been compromised, there's no reason why your personal node has to verify everything from the genesis block.
- thinkmassive 8y ago
  gcmode=archive is not required for a full node. It indexes every transaction in a database, which is unnecessary for most applications. It’s similar to txindex in bitcoind, but the Ethereum db is already much larger, so it takes weeks to sync in this mode.
  - xiphias2 8y ago
    I see, the problem was that it's hard to find documentation about what's needed for an Ethereum full node. As I'm just running Bitcoin full node, I don't have any personal experience with it.

xiphias2 8y ago

I guess you're not doing a full sync. Here's an example of a person trying to do it: https://ethereum.stackexchange.com/questions/37348/parity-full-history-node https://ethereum.stackexchange.com/questions/37348/parity-fu...

CryptoPunk 8y ago
An archival node goes beyond a full node and downloads data not needed for full validation.
- nootropicat 8y ago
  It doesn't download them, the state history is generated from the blockchain locally. The blockchain itself is 68GB (uncompressed, decimal G)
nootropicat 8y ago
Ssd is needed (or enough ram) due to random access of state. See my reddit post from a year ago, full sync took 25 hours: https://www.reddit.com/r/ethereum/comments/7n4v64/correcting_rbitcoins_misinformation_i_just_fully/ https://www.reddit.com/r/ethereum/comments/7n4v64/correcting... There's a lot of misinformation, some originating due to genuine problems (trying to use a hdd or geth with barely working garbage collector) and misunderstandings, some from intentional lies (especially two r/bitcoin moderators - thieflar and stopanddecrypt).