7 ms·

How Ahrefs Gets a Billion Dollar-Worth Infrastructure with a 90% Discount

65 points by botsbreeder 2y ago

CoolCold 2y ago
Nice writing! My gut feeling says part of the problem is C-levels, like CTOs, in recent time, coming solely from programming experience and its just mentally hard for them to even consider having physical hardware (yet on colocation!) and managing such systems. It's just out of equation. On the article itself, billions scale payments are astonishing. And how much paying for traffic would add here too.
- RicoElectrico 2y ago
  Arguing with programmers is oftentimes pointless these days. Pragmatism is out of the equation. I can see why, though. First, there was so much money flowing into IT since 20-ish years that cost was not a consideration both on infra and salary side. Second, connecting all the piping gives an illusion of doing work and feeling "important". You know, we need to be scalable because Google does, and they're successful, right?
  - CoolCold 2y ago
    > Arguing with programmers is oftentimes pointless these days. Pragmatism is out of the equation. I partially agree with you, but my point is - things are even worse at times - arguing happens when there is more than single person involved - CTO may just not even considering/thinking on using hardware, it's a very strong habit to relay on cloudy stuff, atrophy of some sort. > Second, connecting all the piping gives an illusion of doing work and feeling "important". This happens as well, but I think it's more about "keeping things under control" - humans feel more comfortable when they control or tend to believe they control things, all humans, not just developers. From this perspective "gee, this Cloud has API and Console, I can manage it" gives this assurance -> leads to have feeling of control => preferable solution.
    - electric_mayhem 2y ago
      I’ve been doing infrastructure for 27 years. Around a dozen years ago, my business was designing building and supporting physical infrastructure for startups. One company was humming along nicely on $4000 of used hardware and a $2000 a month cage in a Colo facility. Their business had ramped up and got some funding, so we got them another 60k worth of hardware in nother facility. They onboarded still more customers to where they needed a few thousand dollars worth of SSDs to keep up with their random io demands. But their new VC-installed CTO was like… No! We spent all this money on hardware and it’s not doing what we need it to! That’s crap! We’re gonna move to the cloud and save money. So they moved to Amazon. Their first month’s bill was $50,000. And it only went up from there. The cloud is dandy for small workloads. But where you’ve got a consistently large workload the break even point on owning your hardware is a lot lower than most people think, even factoring in infra management expenses.
      - CoolCold 2y ago
        > But their new VC-installed CTO was like… No! Largely the point I'm trying to put here > Their first month’s bill was $50,000. And it only went up from there. Well, I've read it's not uncommon to have salaries of 200-500k USD/year per programmer, on that scale may be it doesn't matter is it 50k spending on infra or 100k . For others such extra spending is _something_ though.
        Nullabillity 2y ago
        You're mixing yearly salary figures and monthly infra bills.
        throwaway11460 2y ago
        50k*12=600k, equivalent to 1-3 SWE employees; practically not even a single small team. It's nothing. Individual contributors see these numbers and act like it's some huge amount of money but in the grander scheme of things when you're approving budgets for multiple teams of senior SWEs and other roles, it doesn't even register.
- RamblingCTO 2y ago
  Running your own data center is just a PITA no one wants to do. Non-C-levels just love to ignore the costs you have when doing such a thing by just comparing hardware costs to the cloud costs. There's always the liability aspect, managing less people, externalizing some complexity to another company etc. etc. All worth a lot of money, at least for me personally. I wanna focus on software, not also having to manage hardware lifecycles, fire stuff, more engineers etc. etc. You deal with enough complexity nowadays already. I'd say that if you really want to have hardware on your own, you get the right people to do the job for you. CTOs are not managing that on the floor. So I'm not sure it's the real reason.
  - CoolCold 2y ago
    > Running your own data center is just a PITA I believe in the article they clearly say they just use colo, not running own DCs > I wanna focus on software, not also having to manage hardware lifecycles, fire stuff, more engineers etc. etc. You deal with enough complexity nowadays already. This looks strange to me, to focus on software but not on company goals, one of them quite often is to increase profits/decrease spending. Focus on software may happen on lower level of responsibility from my PoV. No offense, but for me it sounds like someone having talent for bakery and cooking, decided to open franchise of McDonald's, becoming General Manager of that shop and then suddenly says - I wanna focus on burgers only, not hire/fire workers.
    - RamblingCTO 2y ago
      Maybe my point didn't come across properly. Me personally, I prefer to keep the org complexity down and externalize some headcaches if I can. Having to also manage hardware, people that manage this hardware and not being able to keep other people liable for a given SLA. It's just yet another thing that other companies are really good at and for me it's just not worth it to do on our own. If I don't really need it, why would I? And again, there are more costs associated with this approach than just infra vs hardware + people costs. Granted, I'm never using edge/runner stuff and choosing the stack from the get go as something performant. I'm also talking about headcount < 100, which I also prefer to stay that way. In short: it's not the background (I'm not the one directly managing hardware anyway!), but the upside is not enough for the risk, costs and headaches in general. You need to have a real good use case to do CoLo/DC. If you build web software, you probably don't need it.
- SkyPuncher 2y ago
  In my eyes, physical hardware is something non-growth companies do to improve margins. When you’re a growth stage company, the flexibility and leanness that comes from the cloud is important. Once your core product and computing demands stabilize, it’s a good time to figure out what makes sense to move to cheaper compute.
  - nojvek 2y ago
    Exactly. Dropbox was cloud and then later moved to on-prem to save on storage costs. Anyone running >1000 servers with >80% utilization should consider going partially colocated. Comma.ai and Tesla come to mind who run their own training clusters. If you are serious about compute and cost of compute and storage is ~100s of engineers, then moving off cloud is a wise strategy. Cloud providers have huge margins. One can run a very profitable business if they get a pie of that margin.
- barryrandall 2y ago
  It's investors and the one-size-fits-all metrics many rely on. They want a company's financial statements to look like the financial statements from other financial instruments (companies) they're familiar with. If your industry has 30% EBITDA, your company needs to have ~30% EBITDA. If your industry is 100% cloud, your company is expected to be ~100% cloud. If your numbers deviate from expectations, they'll demand an explanation. If that explanation isn't your company's secret to success, then it's probably not going to be accepted.
EcommerceFlow 2y ago
Fun random fact, Ahrefs is STILL denied a Wikipedia article, even though their competitors like SEMRUSH have one. Wouldn't be surprised if bad actors were involved. I mean hell, I've heard their crawling spider was the 2nd largest on the web after Google. Super interesting article though, and a nice addition to their prior one.
- EE84M3i 2y ago
  https://en.m.wikipedia.org/wiki/Wikipedia:Articles_for_deletion/Ahrefs https://en.m.wikipedia.org/wiki/Wikipedia:Articles_for_delet... seems to be the most recent deletion discussion, from 2020
  - mirochnik 2y ago
    First they ignore you...
  - RamblingCTO 2y ago
    This conversation is cringy. Ahrefs is a great tool.
    - ianhawes 2y ago
      This reads like 99% of the other Wikipedia debates I've seen.
mannyv 2y ago
I think these articles are great. It's about understanding what you do and what infra is appropriate. I have to wonder how that would look with a negotiated AWS discount. But they might get beaten to death by incoming bandwidth costs anyway.
- CoolCold 2y ago
  from AWS/GCP/Azure/you_name_it's perspective, it may be worth to give that 90% discount extra, just to ensure 1. keep shrinking workforce being able to keep doing such kind of setups 2. avoid such writeups with _numbers_ at all
- mirochnik 2y ago
  Netflix is a flagship example of AWS usage. https://aws.amazon.com/solutions/case-studies/innovators/netflix/ https://aws.amazon.com/solutions/case-studies/innovators/net... We don't know what are Netflix discounts. Maybe Netflix uses AWS just for free to make others pay. Maybe AWS even pay to Netflix to continue attracting the other guys. On the other hand, just read this disclosure in the Netflix 2023 annual financial report: "We have architected our software and computer systems so as to utilize data processing, storage capabilities and other services provided by AWS... Given this, along with the fact that we cannot easily switch our AWS operations to another cloud provider, any disruption of or interference with our use of AWS would impact our operations and our business would be adversely impacted. While the retail side of Amazon competes with us, we do not believe that Amazon will use the AWS operation in such a manner as to gain competitive advantage against our service, although if it were to do so it could harm our business."
  - fragmede 2y ago
    > We don't know what are Netflix discounts What you do, if you're a VC backed SaaS company with a six or seven figure monthly spend on AWS, is hire away someone from Netflix who does know what their discount is and use that when your next contract negotiation is up with AWS.
    - ttymck 2y ago
krade 2y ago
I don't doubt that physical hardware would be cheaper than AWS at their scale, but their graph seems off: They are showing no difference between on demand and reserved pricing, with reserved pricing often even being more expensive. Are they actually taking the full 3-year up-front cost and comparing it to monthly on-demand pricing?
- mirochnik 2y ago
  > They are showing no difference between on demand and reserved pricing, with reserved pricing often even being more expensive. That's exactly what is happening when you look at things without depreciation. You pay 3y all upfront. And it's a lot of money: a high step on the graph. On demand will grow steadily and in three years, it will surely be higher than 3y reserved instance. But we are not comparing one instance. We add more ever more powerful servers as they are added in any infrastructure. And we pay upfront. On demand price can't catch up to those all upfront costs on our period of time. To see it clearly, we fix the infra on the 4th graph. And then on-demand becomes much more expensive. Also, you may look at the last paragraph in the Appendix to see why 3y all upfront is not horizontal between the steps. > Are they actually taking the full 3-year up-front cost and comparing it to monthly on-demand pricing? Yep.

65 points by botsbreeder 2y ago

Nice writing! My gut feeling says part of the problem is C-levels, like CTOs, in recent time, coming solely from programming experience and its just mentally hard for them to even consider having physical hardware (yet on colocation!) and managing such systems. It's just out of equation. On the article itself, billions scale payments are astonishing. And how much paying for traffic would add here too.

RicoElectrico 2y ago
Arguing with programmers is oftentimes pointless these days. Pragmatism is out of the equation. I can see why, though. First, there was so much money flowing into IT since 20-ish years that cost was not a consideration both on infra and salary side. Second, connecting all the piping gives an illusion of doing work and feeling "important". You know, we need to be scalable because Google does, and they're successful, right?
- CoolCold 2y ago
  > Arguing with programmers is oftentimes pointless these days. Pragmatism is out of the equation. I partially agree with you, but my point is - things are even worse at times - arguing happens when there is more than single person involved - CTO may just not even considering/thinking on using hardware, it's a very strong habit to relay on cloudy stuff, atrophy of some sort. > Second, connecting all the piping gives an illusion of doing work and feeling "important". This happens as well, but I think it's more about "keeping things under control" - humans feel more comfortable when they control or tend to believe they control things, all humans, not just developers. From this perspective "gee, this Cloud has API and Console, I can manage it" gives this assurance -> leads to have feeling of control => preferable solution.
  - electric_mayhem 2y ago
    I’ve been doing infrastructure for 27 years. Around a dozen years ago, my business was designing building and supporting physical infrastructure for startups. One company was humming along nicely on $4000 of used hardware and a $2000 a month cage in a Colo facility. Their business had ramped up and got some funding, so we got them another 60k worth of hardware in nother facility. They onboarded still more customers to where they needed a few thousand dollars worth of SSDs to keep up with their random io demands. But their new VC-installed CTO was like… No! We spent all this money on hardware and it’s not doing what we need it to! That’s crap! We’re gonna move to the cloud and save money. So they moved to Amazon. Their first month’s bill was $50,000. And it only went up from there. The cloud is dandy for small workloads. But where you’ve got a consistently large workload the break even point on owning your hardware is a lot lower than most people think, even factoring in infra management expenses.
    - CoolCold 2y ago
      > But their new VC-installed CTO was like… No! Largely the point I'm trying to put here > Their first month’s bill was $50,000. And it only went up from there. Well, I've read it's not uncommon to have salaries of 200-500k USD/year per programmer, on that scale may be it doesn't matter is it 50k spending on infra or 100k . For others such extra spending is _something_ though.
      - Nullabillity 2y ago
        You're mixing yearly salary figures and monthly infra bills.
        throwaway11460 2y ago
        50k*12=600k, equivalent to 1-3 SWE employees; practically not even a single small team. It's nothing. Individual contributors see these numbers and act like it's some huge amount of money but in the grander scheme of things when you're approving budgets for multiple teams of senior SWEs and other roles, it doesn't even register.
RamblingCTO 2y ago
Running your own data center is just a PITA no one wants to do. Non-C-levels just love to ignore the costs you have when doing such a thing by just comparing hardware costs to the cloud costs. There's always the liability aspect, managing less people, externalizing some complexity to another company etc. etc. All worth a lot of money, at least for me personally. I wanna focus on software, not also having to manage hardware lifecycles, fire stuff, more engineers etc. etc. You deal with enough complexity nowadays already. I'd say that if you really want to have hardware on your own, you get the right people to do the job for you. CTOs are not managing that on the floor. So I'm not sure it's the real reason.
- CoolCold 2y ago
  > Running your own data center is just a PITA I believe in the article they clearly say they just use colo, not running own DCs > I wanna focus on software, not also having to manage hardware lifecycles, fire stuff, more engineers etc. etc. You deal with enough complexity nowadays already. This looks strange to me, to focus on software but not on company goals, one of them quite often is to increase profits/decrease spending. Focus on software may happen on lower level of responsibility from my PoV. No offense, but for me it sounds like someone having talent for bakery and cooking, decided to open franchise of McDonald's, becoming General Manager of that shop and then suddenly says - I wanna focus on burgers only, not hire/fire workers.
  - RamblingCTO 2y ago
    Maybe my point didn't come across properly. Me personally, I prefer to keep the org complexity down and externalize some headcaches if I can. Having to also manage hardware, people that manage this hardware and not being able to keep other people liable for a given SLA. It's just yet another thing that other companies are really good at and for me it's just not worth it to do on our own. If I don't really need it, why would I? And again, there are more costs associated with this approach than just infra vs hardware + people costs. Granted, I'm never using edge/runner stuff and choosing the stack from the get go as something performant. I'm also talking about headcount < 100, which I also prefer to stay that way. In short: it's not the background (I'm not the one directly managing hardware anyway!), but the upside is not enough for the risk, costs and headaches in general. You need to have a real good use case to do CoLo/DC. If you build web software, you probably don't need it.
SkyPuncher 2y ago
In my eyes, physical hardware is something non-growth companies do to improve margins. When you’re a growth stage company, the flexibility and leanness that comes from the cloud is important. Once your core product and computing demands stabilize, it’s a good time to figure out what makes sense to move to cheaper compute.
- nojvek 2y ago
  Exactly. Dropbox was cloud and then later moved to on-prem to save on storage costs. Anyone running >1000 servers with >80% utilization should consider going partially colocated. Comma.ai and Tesla come to mind who run their own training clusters. If you are serious about compute and cost of compute and storage is ~100s of engineers, then moving off cloud is a wise strategy. Cloud providers have huge margins. One can run a very profitable business if they get a pie of that margin.
barryrandall 2y ago
It's investors and the one-size-fits-all metrics many rely on. They want a company's financial statements to look like the financial statements from other financial instruments (companies) they're familiar with. If your industry has 30% EBITDA, your company needs to have ~30% EBITDA. If your industry is 100% cloud, your company is expected to be ~100% cloud. If your numbers deviate from expectations, they'll demand an explanation. If that explanation isn't your company's secret to success, then it's probably not going to be accepted.

EcommerceFlow 2y ago

Fun random fact, Ahrefs is STILL denied a Wikipedia article, even though their competitors like SEMRUSH have one. Wouldn't be surprised if bad actors were involved. I mean hell, I've heard their crawling spider was the 2nd largest on the web after Google. Super interesting article though, and a nice addition to their prior one.

EE84M3i 2y ago
https://en.m.wikipedia.org/wiki/Wikipedia:Articles_for_deletion/Ahrefs https://en.m.wikipedia.org/wiki/Wikipedia:Articles_for_delet... seems to be the most recent deletion discussion, from 2020
- mirochnik 2y ago
  First they ignore you...
- RamblingCTO 2y ago
  This conversation is cringy. Ahrefs is a great tool.
  - ianhawes 2y ago
    This reads like 99% of the other Wikipedia debates I've seen.

mannyv 2y ago

I think these articles are great. It's about understanding what you do and what infra is appropriate. I have to wonder how that would look with a negotiated AWS discount. But they might get beaten to death by incoming bandwidth costs anyway.

CoolCold 2y ago
from AWS/GCP/Azure/you_name_it's perspective, it may be worth to give that 90% discount extra, just to ensure 1. keep shrinking workforce being able to keep doing such kind of setups 2. avoid such writeups with _numbers_ at all
mirochnik 2y ago
Netflix is a flagship example of AWS usage. https://aws.amazon.com/solutions/case-studies/innovators/netflix/ https://aws.amazon.com/solutions/case-studies/innovators/net... We don't know what are Netflix discounts. Maybe Netflix uses AWS just for free to make others pay. Maybe AWS even pay to Netflix to continue attracting the other guys. On the other hand, just read this disclosure in the Netflix 2023 annual financial report: "We have architected our software and computer systems so as to utilize data processing, storage capabilities and other services provided by AWS... Given this, along with the fact that we cannot easily switch our AWS operations to another cloud provider, any disruption of or interference with our use of AWS would impact our operations and our business would be adversely impacted. While the retail side of Amazon competes with us, we do not believe that Amazon will use the AWS operation in such a manner as to gain competitive advantage against our service, although if it were to do so it could harm our business."
- fragmede 2y ago
  > We don't know what are Netflix discounts What you do, if you're a VC backed SaaS company with a six or seven figure monthly spend on AWS, is hire away someone from Netflix who does know what their discount is and use that when your next contract negotiation is up with AWS.
  - ttymck 2y ago

krade 2y ago

I don't doubt that physical hardware would be cheaper than AWS at their scale, but their graph seems off: They are showing no difference between on demand and reserved pricing, with reserved pricing often even being more expensive. Are they actually taking the full 3-year up-front cost and comparing it to monthly on-demand pricing?

mirochnik 2y ago
> They are showing no difference between on demand and reserved pricing, with reserved pricing often even being more expensive. That's exactly what is happening when you look at things without depreciation. You pay 3y all upfront. And it's a lot of money: a high step on the graph. On demand will grow steadily and in three years, it will surely be higher than 3y reserved instance. But we are not comparing one instance. We add more ever more powerful servers as they are added in any infrastructure. And we pay upfront. On demand price can't catch up to those all upfront costs on our period of time. To see it clearly, we fix the infra on the 4th graph. And then on-demand becomes much more expensive. Also, you may look at the last paragraph in the Appendix to see why 3y all upfront is not horizontal between the steps. > Are they actually taking the full 3-year up-front cost and comparing it to monthly on-demand pricing? Yep.