7 ms·

FYI do not run blockchain related software on Hetzner nodes, even their dedicated bare metal. They are likely to shut the host down. I was running a Solana Tes

by SEJeff 3y ago

FYI do not run blockchain related software on Hetzner nodes, even their dedicated bare metal. They are likely to shut the host down. I was running a Solana Testnet Proof of Stake validator for work and they decided to shut the host down at the network layer and refused to give us access to the host. After my employer's legal threatened them, they offered to give us access back, but to be a pain in the butt, they only gave us console login access. Luckily, it was just our Testnet validator, and we keep the keys in cold storage which we were able to recover from. Their nodes are cheap, but beware of Hetzner. Something that they didn't mind hosting for almost 4 years they suddenly changed their minds on and shut the host down. See: https://www.theblock.co/post/182283/1000-solana-validators-go-offline-as-hetzner-blocks-server-access https://www.theblock.co/post/182283/1000-solana-validators-g...

krig 3y ago
Thanks for writing about this. Makes me like Hetzner even more.
- lutoma 3y ago
  I was just about to write that. No crypto scam nonsense in the same datacenter sounds like a plus to me. (On a more serious note, I imagine the reason Hetzner does not want crypto-related stuff is for the same reasons most hosting companies are not very fond of porn-related sites: High rates of fraud, charge-backs and other potential legal liability).
vmfunction 3y ago
They did officially stated on reddit [1] that they are not supporting crypto related hosting. Just wonder why? Is the GPU/CPU/Network related over use? Or legality related? [1]: https://www.reddit.com/r/hetzner/comments/wucxs4/comment/ilfoj8u/?context=3 https://www.reddit.com/r/hetzner/comments/wucxs4/comment/ilf...
- vegardx 3y ago
  Wild guess: They use significantly more electricity than the average customer that does just about anything else with their hardware. The GPU can easily run at hundreds of watts, and when you add in the required cooling you're probably at 1.5-2x that in total power usage.
  - deleted 3y ago
    [deleted]
  - SEJeff 3y ago
    This is not a proof of work (bitcoin or pre merge ethereum) node and it does not use more electricity. They singled out solana seemingly and I’m unsure why. We moved about 300 services off of dedicated higher end Hetzner nodes within a week.
  - mnahkies 3y ago
    What implication does this have for AI workloads - the computation intensity would be similar? (Assuming a popular service saturating the machines) I see some other comments mentioning ssd wear which I don't think would apply to AI stuff, but the utilisation/power consumption certainly would.
- SEJeff 3y ago
  They stated that AFTER they just cut the network access to the hosts. It isn’t as though we were doing anything ethically or legally wrong. We were running rpc nodes for legit applications and a validator to secure the network. For me, who’s worked in 50,000+ bare metal Linux environments before, it was just so surprising how sudden and violent it was. Their customer support was amazing until it wasn’t. Just let it be a lesson, they will arbitrarily change their mind and put your nodes running your business at risk. Even if you’ve broken no laws and are operating in good faith. We were not running and proof of work nonsense that breaks the hardware or uses excess electricity.
  - sgift 3y ago
    > It isn’t as though we were doing anything ethically or legally wrong. Legally? Well, they probably have a "if we think you do something we don't like we can kick you off" clause. Ethically? Eh .. Crypto. Ethical. Opinions differ.
    - shaism 3y ago
      I am not a big fan of crypto either, but don’t understand why crypto (the technology itself) would be unethical. Could you explain?
      - Cryptoanti 3y ago
        I don't know any working thing using crypto in a way that it acutally make sense and couldn't be already be done easier and similiar safe. Even in big companies you always see examples which break the technology like creating something on the blockchain which has a connection to the real world.
        ajimix 3y ago
        Try to send some big amount of money internationally and you will see why crypto is the future. With regular banks it takes days, weeks and many times months due to intermediate banks, AML questions, reviews, etc and have very high costs associated for simple transfers of money hardly earned. With cryptocurrencies it takes seconds or minutes and no questions asked with very low fees
- gosub100 3y ago
  My guess is it's a huge liability if keys are stolen, and it puts a huge target on their back from people around the world who want to steal crypto keys or DDoS crypto systems.
  - SEJeff 3y ago
    Most of the Hetzner DdoS come from gaming fyi. They’re pretty good at handling them but we’d get paged every few months due to this. Our uptime is better since we left hetzner. We went with a host that’s more commonly used for electronic trading firms, Teraswitch. They’re great.
    - neoromantique 3y ago
      Thanks for recommendation, we are looking to move our servers off Hetzner for precisely the same reasons.
- Croftengea 3y ago
  Many hosters banned all kind of crypto after advent of file coins, which wear out ssds too quicky.
  - SEJeff 3y ago
    Indeed. Some bad actors (which is most of the industry) ruin it for those of us who just want to build cool tech. Very annoying.
    - stouset 3y ago
      You know what they say, a bunch of bad apples spoils the few.
- grw_ 3y ago
  AFAIK it's the high disk IO causing wear on the SSDs, since they specifically mention Solana nodes
  - SEJeff 3y ago
    Likely. We put the accounts db in memory (on a tmpfs) to mitigate this and were paying them for a beefcake node with 1T of memory.
DonHopkins 3y ago
[flagged]
- SEJeff 3y ago
  Yeah, that’s definitely not what we were doing buddy. Keep the politics out of this please. We don’t accept any customer money either for this usage. Zero. Those nodes secured the network and were also used for analysis of hacks and forensic incident response. We work a lot with the US FBI btw. We’re trying to prevent literally all the things you just mentioned, so no need to be jerk here buddy.
  - DonHopkins 3y ago
    [flagged]
  - lutoma 3y ago
    > Keep the politics out of this please Apart from how nonsensical it is to want to "keep politics" out of inherently political discussions: Your last comments on hacker news were about US trade policy, Navalny's treatment by Russia and the development of fringe political groups in the EU.
    - SEJeff 3y ago
      The conversation was not about views on blockchain. The conversion was hetzner decided a client running several hundred nodes was suddenly running afoul of the ToS after running happily almost four years and they cut off all access without any notification. They then updated their ToS to clarify this was not allowed, but they happily took our money for four years and we built out a fair bit of tooling on this. We do DFIR and security. Do you even know anything about digital forensics? It’s not a “bad” thing to investigate hacks and work with law enforcement to recover funds. It’s necessary as not everyone is a crook or thief. So yes, I think it’s entirely reasonable to keep conversations on hacker news on topic and not inject politics where the conversation is not about politics. Obviously the cases you mentioned are inherently about politics so that’s a nice straw man you beat up there. Congratulations sir.
tichiian 3y ago
Your fault for not reading their ToS: https://www.hetzner.com/de/legal/terms-and-conditions/ https://www.hetzner.com/de/legal/terms-and-conditions/ number 8.1 in the German version, number 4 in the English one: "Furthermore, the operation of applications for mining cryptocurrencies is prohibited. This includes, but is not limited to, mining, farming and plotting of cryptocurrencies. In the case of non-compliance, we are entitled to lock your access to the Service and/or to your account." Lots of hosters and cloud providers have this and similar exclusions, to keep out the typically shady cryptocurrency crowd, their excessive resource usage, high payment risk and reputation damage. E.g. Google Cloud also doesn't allow cryptocurrency stuff without prior approval, 3.3 iv: https://cloud.google.com/terms https://cloud.google.com/terms , same for AWS, 1.25: https://aws.amazon.com/service-terms/ https://aws.amazon.com/service-terms/ So I hope you got your written approval first, otherwise AWS might be equally nasty to you.
- SEJeff 3y ago
  Their ToS did not reference blockchain at the time. This is why our lawyers got involved. This was updated after they caused an outage unannounced for a service running almost four years. Google actually offers blockchain rpc. Fwiw, Solana nodes run poorly in the cloud and generally only make sense on bare metal. As far as slow distributed databases go (aka blockchains), it’s 1000x+ faster than the ethereum style blockchains.
  - tichiian 3y ago
    Ah, if that is new, then you are right, that's probably a breach of contract on their side. Cloud doesn't necessarily mean virtual machines with scheduling and sharing problems. You can get non-shared machines with almost all popular cloud platforms, e.g. https://cloud.google.com/compute/docs/compute-optimized-machines https://cloud.google.com/compute/docs/compute-optimized-mach... Of course those aren't cheap.

by SEJeff 3y ago

krig 3y ago

Thanks for writing about this. Makes me like Hetzner even more.

lutoma 3y ago
I was just about to write that. No crypto scam nonsense in the same datacenter sounds like a plus to me. (On a more serious note, I imagine the reason Hetzner does not want crypto-related stuff is for the same reasons most hosting companies are not very fond of porn-related sites: High rates of fraud, charge-backs and other potential legal liability).

vmfunction 3y ago

They did officially stated on reddit [1] that they are not supporting crypto related hosting. Just wonder why? Is the GPU/CPU/Network related over use? Or legality related? [1]: https://www.reddit.com/r/hetzner/comments/wucxs4/comment/ilfoj8u/?context=3 https://www.reddit.com/r/hetzner/comments/wucxs4/comment/ilf...

vegardx 3y ago
Wild guess: They use significantly more electricity than the average customer that does just about anything else with their hardware. The GPU can easily run at hundreds of watts, and when you add in the required cooling you're probably at 1.5-2x that in total power usage.
- deleted 3y ago
  [deleted]
- SEJeff 3y ago
  This is not a proof of work (bitcoin or pre merge ethereum) node and it does not use more electricity. They singled out solana seemingly and I’m unsure why. We moved about 300 services off of dedicated higher end Hetzner nodes within a week.
- mnahkies 3y ago
  What implication does this have for AI workloads - the computation intensity would be similar? (Assuming a popular service saturating the machines) I see some other comments mentioning ssd wear which I don't think would apply to AI stuff, but the utilisation/power consumption certainly would.
SEJeff 3y ago
They stated that AFTER they just cut the network access to the hosts. It isn’t as though we were doing anything ethically or legally wrong. We were running rpc nodes for legit applications and a validator to secure the network. For me, who’s worked in 50,000+ bare metal Linux environments before, it was just so surprising how sudden and violent it was. Their customer support was amazing until it wasn’t. Just let it be a lesson, they will arbitrarily change their mind and put your nodes running your business at risk. Even if you’ve broken no laws and are operating in good faith. We were not running and proof of work nonsense that breaks the hardware or uses excess electricity.
- sgift 3y ago
  > It isn’t as though we were doing anything ethically or legally wrong. Legally? Well, they probably have a "if we think you do something we don't like we can kick you off" clause. Ethically? Eh .. Crypto. Ethical. Opinions differ.
  - shaism 3y ago
    I am not a big fan of crypto either, but don’t understand why crypto (the technology itself) would be unethical. Could you explain?
    - Cryptoanti 3y ago
      I don't know any working thing using crypto in a way that it acutally make sense and couldn't be already be done easier and similiar safe. Even in big companies you always see examples which break the technology like creating something on the blockchain which has a connection to the real world.
      - ajimix 3y ago
        Try to send some big amount of money internationally and you will see why crypto is the future. With regular banks it takes days, weeks and many times months due to intermediate banks, AML questions, reviews, etc and have very high costs associated for simple transfers of money hardly earned. With cryptocurrencies it takes seconds or minutes and no questions asked with very low fees
gosub100 3y ago
My guess is it's a huge liability if keys are stolen, and it puts a huge target on their back from people around the world who want to steal crypto keys or DDoS crypto systems.
- SEJeff 3y ago
  Most of the Hetzner DdoS come from gaming fyi. They’re pretty good at handling them but we’d get paged every few months due to this. Our uptime is better since we left hetzner. We went with a host that’s more commonly used for electronic trading firms, Teraswitch. They’re great.
  - neoromantique 3y ago
    Thanks for recommendation, we are looking to move our servers off Hetzner for precisely the same reasons.
Croftengea 3y ago
Many hosters banned all kind of crypto after advent of file coins, which wear out ssds too quicky.
- SEJeff 3y ago
  Indeed. Some bad actors (which is most of the industry) ruin it for those of us who just want to build cool tech. Very annoying.
  - stouset 3y ago
    You know what they say, a bunch of bad apples spoils the few.
grw_ 3y ago
AFAIK it's the high disk IO causing wear on the SSDs, since they specifically mention Solana nodes
- SEJeff 3y ago
  Likely. We put the accounts db in memory (on a tmpfs) to mitigate this and were paying them for a beefcake node with 1T of memory.

DonHopkins 3y ago

[flagged]

SEJeff 3y ago
Yeah, that’s definitely not what we were doing buddy. Keep the politics out of this please. We don’t accept any customer money either for this usage. Zero. Those nodes secured the network and were also used for analysis of hacks and forensic incident response. We work a lot with the US FBI btw. We’re trying to prevent literally all the things you just mentioned, so no need to be jerk here buddy.
- DonHopkins 3y ago
  [flagged]
- lutoma 3y ago
  > Keep the politics out of this please Apart from how nonsensical it is to want to "keep politics" out of inherently political discussions: Your last comments on hacker news were about US trade policy, Navalny's treatment by Russia and the development of fringe political groups in the EU.
  - SEJeff 3y ago
    The conversation was not about views on blockchain. The conversion was hetzner decided a client running several hundred nodes was suddenly running afoul of the ToS after running happily almost four years and they cut off all access without any notification. They then updated their ToS to clarify this was not allowed, but they happily took our money for four years and we built out a fair bit of tooling on this. We do DFIR and security. Do you even know anything about digital forensics? It’s not a “bad” thing to investigate hacks and work with law enforcement to recover funds. It’s necessary as not everyone is a crook or thief. So yes, I think it’s entirely reasonable to keep conversations on hacker news on topic and not inject politics where the conversation is not about politics. Obviously the cases you mentioned are inherently about politics so that’s a nice straw man you beat up there. Congratulations sir.

tichiian 3y ago

Your fault for not reading their ToS: https://www.hetzner.com/de/legal/terms-and-conditions/ https://www.hetzner.com/de/legal/terms-and-conditions/ number 8.1 in the German version, number 4 in the English one: "Furthermore, the operation of applications for mining cryptocurrencies is prohibited. This includes, but is not limited to, mining, farming and plotting of cryptocurrencies. In the case of non-compliance, we are entitled to lock your access to the Service and/or to your account." Lots of hosters and cloud providers have this and similar exclusions, to keep out the typically shady cryptocurrency crowd, their excessive resource usage, high payment risk and reputation damage. E.g. Google Cloud also doesn't allow cryptocurrency stuff without prior approval, 3.3 iv: https://cloud.google.com/terms https://cloud.google.com/terms , same for AWS, 1.25: https://aws.amazon.com/service-terms/ https://aws.amazon.com/service-terms/ So I hope you got your written approval first, otherwise AWS might be equally nasty to you.

SEJeff 3y ago
Their ToS did not reference blockchain at the time. This is why our lawyers got involved. This was updated after they caused an outage unannounced for a service running almost four years. Google actually offers blockchain rpc. Fwiw, Solana nodes run poorly in the cloud and generally only make sense on bare metal. As far as slow distributed databases go (aka blockchains), it’s 1000x+ faster than the ethereum style blockchains.
- tichiian 3y ago
  Ah, if that is new, then you are right, that's probably a breach of contract on their side. Cloud doesn't necessarily mean virtual machines with scheduling and sharing problems. You can get non-shared machines with almost all popular cloud platforms, e.g. https://cloud.google.com/compute/docs/compute-optimized-machines https://cloud.google.com/compute/docs/compute-optimized-mach... Of course those aren't cheap.