6 ms·

> Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to

> Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to make things more balanced. I may know one of the culprits -- whom I will leave unnamed here. But the company, who is fairly popular, built out their own CDN via putting a bunch of nginx caching proxies on various Hetzner servers around the world. It apparently was really cheap and very effective. Given that they were bootstrapped and this was prior to Cloudflare really being that popular, it was a great strategy. This was true like 8 years ago, so maybe it has changed in the meantime.

justinclift 2y ago
Sounds like a completely legitimate use though. Hetzner were widely telling people about that 20TB limit, so why would they be surprised when people use them as CDN boxes?
- yurishimo 2y ago
  Until you remember that marketing is a separate department from finance which is a separate department from ops/engineering. The engineers said 20TB in aggregate was fine but likely didn’t consider the “bad apples”. Marketing obviously wants to use the biggest numbers and then finance comes in with the hammer and dev points to egress as an simple way to upset rhe fewest number of real customers.
  - hyperpape 2y ago
    As an engineer, if you don't qualify whether your answer is average or max, you've messed up.
    - drpossum 2y ago
      I'm confident the marketing folks would just say something akin to "shut up nerd", cash their bonuses, and leave you with the problem.
    - cluckindan 2y ago
      As an engineer, NEVER give out averages without checking that they are consistent with the distribution of the actual data.
      - phil21 2y ago
        Averages tend to skew once exposed to adverse selection.
      - Moru 2y ago
        Don't think you find much engineers in the marketing department though.
  - bmicraft 2y ago
    It does make sense: The average was low enough that a 20TB cap worked. Then marketing started boasting about the 20TB limit and attracted a bunch of high-bandwith customers, thereby driving up the average making 20TB decidedly "not fine".
- sdwr 2y ago
  Where's the surprise? It's the classic business 2-step - drum up interest with "too good to be true" features, then cut them back. The marginal customers who need those features leave (and are too expensive to keep), everyone else is used to your product and stays.
  - baq 2y ago
    MBA value extraction 101.
  - gchamonlive 2y ago
    That or they just didn't dimension well their prices and now it's biting them back. I prefer to assume naivety over malice/negligence.
  - asyx 2y ago
    Hetzner has been in this business for a very long time and I’m sure half the German small to medium sized companies use Hetzner. They have costs as well and with virtual servers they just assumed the usage pattern was similar to their eu data center and that didn’t work out so they increase the prices a bit. So that on average their calculation works out again. Like, this is not Netflix tripling the prices over a few years.
learnedbytes 2y ago
Want to give a clue on who the culprit is?
- bhouston 2y ago
  I was curious and checked if they are still using Hetzner. It appears not, so I can share who it was. It was https://artstation.com https://artstation.com. Basically heavily oriented towards serving static images, so the CDN could have been really expensive. Doing a reverse IP lookup on cdn.artstation.com servers now resolves to Cloudflare and it has cloudflare headers on the response.
tssva 2y ago
I think culprits is a poor choice of words since it means someone suspected of a misdeed. I could perhaps understand using it for example for someone that tried to store a petrabyte of storage on a consumer unlimited storage plan. But in this case Hetzner set a specific data usage amount you are paying for so using that amount is not a misdeed.
- KomoD 2y ago
  > I could perhaps understand using it for example for someone that tried to store a petrabyte of storage on a consumer unlimited storage plan. I couldn't, don't call it unlimited if it's not unlimited, using what you paid for is not a "misdeed".
paxys 2y ago
“Culprits” because they used the service they paid for within its advertised limits? It’s the same with cloud storage providers. First give out a massive amount of storage and rapidly gain users, then cut it down after blaming people for “abusing” it. How about you advertise your correct capacity to begin with? They are simply deflecting blame for their own enshittification.
- macspoofing 2y ago
  >“Culprits” because they used the service they paid for within its advertised limits? "Culprits" because it was their (legal) use of the service that made Hetzner rethink and change their service plan.
  - paxys 2y ago
    So then the culprits are the company’s own engineering and marketing departments for not correctly anticipating user demand.
    - ghaff 2y ago
      Or you take your best shot and then adjust as needed.
    - macspoofing 2y ago
      Sure .. also them. Having said that, I am usually empathetic to these kinds of 'unlimited' deals because even though they aren't really 'unlimited', they do tend to be generous to the average use-case and average user .. Inevitably, and unfortunately, someone decides to test the limits and the entire thing collapses. It reminds me of the Blockbuster "No more late fees" policy, which was a really good customer-friendly policy (speaking as someone who regularly returned rentals late) .. but then they were sued because an aspect of the policy had Blockbuster charging the cost of the rental to the customer if it wasn't returned in some period of time .. and because that charge looked like a 'late fee' they got sued. Urgh.
      - 2y ago
Aeolun 2y ago
Then why does this only apply in the US? Are they saying EU customers are well behaved?
- 1oooqooq 2y ago
  probably streaming platforms have better content on the US and everyone wants to exit there? so they are mostly serving US traffic for several vpns all over the world connecting to CDNs in the US.
- Hilift 2y ago
  Isn't Hetzner impacted in Europe at the moment from a cable cut? Wouldn't surprise me if there is a wave of people moving stuff over to the US because of that. I don't know much about cloud though. I believe "The Ship has arrived and repairs are underway, which will still take some time." and "the repair may take up to two weeks.".
  - mrweasel 2y ago
    Hetzner did pay somewhere around 20% of the cost of C-Lion1, so it wouldn't be surprising if it has hurt hem.
  - Hamuko 2y ago
    Last I heard from Cinia was that the cable should be fixed by the end of November, so sometime within the next 55 hours.
    - master-lincoln 2y ago
      Seems done https://www.cinia.fi/en/news/cinias-c-lion1-submarine-cable-has-fully-restored https://www.cinia.fi/en/news/cinias-c-lion1-submarine-cable-...
- citrin_ru 2y ago
  It also may depend on peering arrangements Hetzner has. If EU ISP more inclined to peer with Hetzner than US one bandwidth in EU will be cheaper for them.
BonoboIO 2y ago
8 years ago, hetzner had no cloud offerings The dedicated servers still have 20 tb traffic included
- lostmsu 2y ago
  Do they have dedicated servers in US?
  - no_wizard 2y ago
    In Hillsboro, OR they been building up their data centers but this hasn’t converted to dedicated servers in the US yet
    - nik736 2y ago
      Do you have a source for this? My last info was they rented space in an existing data center.
- dumbledoren 2y ago
  What? Afaik they are unmetered. And real unmetered - there are people who are saturating their connections 24/7 while using them.
Dylan16807 2y ago
Well, I doubt an extra $1/TB is going to stop Hetzner from being very price effective.

justinclift 2y ago

Sounds like a completely legitimate use though. Hetzner were widely telling people about that 20TB limit, so why would they be surprised when people use them as CDN boxes?

yurishimo 2y ago
Until you remember that marketing is a separate department from finance which is a separate department from ops/engineering. The engineers said 20TB in aggregate was fine but likely didn’t consider the “bad apples”. Marketing obviously wants to use the biggest numbers and then finance comes in with the hammer and dev points to egress as an simple way to upset rhe fewest number of real customers.
- hyperpape 2y ago
  As an engineer, if you don't qualify whether your answer is average or max, you've messed up.
  - drpossum 2y ago
    I'm confident the marketing folks would just say something akin to "shut up nerd", cash their bonuses, and leave you with the problem.
  - cluckindan 2y ago
    As an engineer, NEVER give out averages without checking that they are consistent with the distribution of the actual data.
    - phil21 2y ago
      Averages tend to skew once exposed to adverse selection.
    - Moru 2y ago
      Don't think you find much engineers in the marketing department though.
- bmicraft 2y ago
  It does make sense: The average was low enough that a 20TB cap worked. Then marketing started boasting about the 20TB limit and attracted a bunch of high-bandwith customers, thereby driving up the average making 20TB decidedly "not fine".
sdwr 2y ago
Where's the surprise? It's the classic business 2-step - drum up interest with "too good to be true" features, then cut them back. The marginal customers who need those features leave (and are too expensive to keep), everyone else is used to your product and stays.
- baq 2y ago
  MBA value extraction 101.
- gchamonlive 2y ago
  That or they just didn't dimension well their prices and now it's biting them back. I prefer to assume naivety over malice/negligence.
- asyx 2y ago
  Hetzner has been in this business for a very long time and I’m sure half the German small to medium sized companies use Hetzner. They have costs as well and with virtual servers they just assumed the usage pattern was similar to their eu data center and that didn’t work out so they increase the prices a bit. So that on average their calculation works out again. Like, this is not Netflix tripling the prices over a few years.

learnedbytes 2y ago

Want to give a clue on who the culprit is?

bhouston 2y ago
I was curious and checked if they are still using Hetzner. It appears not, so I can share who it was. It was https://artstation.com https://artstation.com. Basically heavily oriented towards serving static images, so the CDN could have been really expensive. Doing a reverse IP lookup on cdn.artstation.com servers now resolves to Cloudflare and it has cloudflare headers on the response.

tssva 2y ago

I think culprits is a poor choice of words since it means someone suspected of a misdeed. I could perhaps understand using it for example for someone that tried to store a petrabyte of storage on a consumer unlimited storage plan. But in this case Hetzner set a specific data usage amount you are paying for so using that amount is not a misdeed.

KomoD 2y ago
> I could perhaps understand using it for example for someone that tried to store a petrabyte of storage on a consumer unlimited storage plan. I couldn't, don't call it unlimited if it's not unlimited, using what you paid for is not a "misdeed".

paxys 2y ago

“Culprits” because they used the service they paid for within its advertised limits? It’s the same with cloud storage providers. First give out a massive amount of storage and rapidly gain users, then cut it down after blaming people for “abusing” it. How about you advertise your correct capacity to begin with? They are simply deflecting blame for their own enshittification.

macspoofing 2y ago
>“Culprits” because they used the service they paid for within its advertised limits? "Culprits" because it was their (legal) use of the service that made Hetzner rethink and change their service plan.
- paxys 2y ago
  So then the culprits are the company’s own engineering and marketing departments for not correctly anticipating user demand.
  - ghaff 2y ago
    Or you take your best shot and then adjust as needed.
  - macspoofing 2y ago
    Sure .. also them. Having said that, I am usually empathetic to these kinds of 'unlimited' deals because even though they aren't really 'unlimited', they do tend to be generous to the average use-case and average user .. Inevitably, and unfortunately, someone decides to test the limits and the entire thing collapses. It reminds me of the Blockbuster "No more late fees" policy, which was a really good customer-friendly policy (speaking as someone who regularly returned rentals late) .. but then they were sued because an aspect of the policy had Blockbuster charging the cost of the rental to the customer if it wasn't returned in some period of time .. and because that charge looked like a 'late fee' they got sued. Urgh.
    - 2y ago

Aeolun 2y ago

Then why does this only apply in the US? Are they saying EU customers are well behaved?

1oooqooq 2y ago
probably streaming platforms have better content on the US and everyone wants to exit there? so they are mostly serving US traffic for several vpns all over the world connecting to CDNs in the US.
Hilift 2y ago
Isn't Hetzner impacted in Europe at the moment from a cable cut? Wouldn't surprise me if there is a wave of people moving stuff over to the US because of that. I don't know much about cloud though. I believe "The Ship has arrived and repairs are underway, which will still take some time." and "the repair may take up to two weeks.".
- mrweasel 2y ago
  Hetzner did pay somewhere around 20% of the cost of C-Lion1, so it wouldn't be surprising if it has hurt hem.
- Hamuko 2y ago
  Last I heard from Cinia was that the cable should be fixed by the end of November, so sometime within the next 55 hours.
  - master-lincoln 2y ago
    Seems done https://www.cinia.fi/en/news/cinias-c-lion1-submarine-cable-has-fully-restored https://www.cinia.fi/en/news/cinias-c-lion1-submarine-cable-...
citrin_ru 2y ago
It also may depend on peering arrangements Hetzner has. If EU ISP more inclined to peer with Hetzner than US one bandwidth in EU will be cheaper for them.

BonoboIO 2y ago

8 years ago, hetzner had no cloud offerings The dedicated servers still have 20 tb traffic included

lostmsu 2y ago
Do they have dedicated servers in US?
- no_wizard 2y ago
  In Hillsboro, OR they been building up their data centers but this hasn’t converted to dedicated servers in the US yet
  - nik736 2y ago
    Do you have a source for this? My last info was they rented space in an existing data center.
dumbledoren 2y ago
What? Afaik they are unmetered. And real unmetered - there are people who are saturating their connections 24/7 while using them.

Dylan16807 2y ago

Well, I doubt an extra $1/TB is going to stop Hetzner from being very price effective.