11 ms·

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is

This is an industry we're[0] in. Owning is at one end of the spectrum, with cloud at the other, and a broadly couple of options in-between: 1 - Cloud – This is minimising cap-ex, hiring, and risk, while largely maximising operational costs (its expensive) and cost variability (usage based). 2 - Managed Private Cloud - What we do. Still minimal-to-no cap-ex, hiring, risk, and medium-sized operational cost (around 50% cheaper than AWS et al). We rent or colocate bare metal, manage it for you, handle software deployments, deploy only open-source, etc. Only really makes sense above €$5k/month spend. 3 - Rented Bare Metal – Let someone else handle the hardware financing for you. Still minimal cap-ex, but with greater hiring/skilling and risk. Around 90% cheaper than AWS et al (plus time). 4 - Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills, scale, cap-ex, and if you plan to run the servers for at least 3-5 years. A good provider for option 3 is someone like Hetzner. Their internal ROI on server hardware seems to be around the 3 year mark. After which I assume it is either still running with a client, or goes into their server auction system. Options 3 & 4 generally become more appealing either at scale, or when infrastructure is part of the core business. Option 1 is great for startups who want to spend very little initially, but then grow very quickly. Option 2 is pretty good for SMEs with baseline load, regular-sized business growth, and maybe an overworked DevOps team! [0] https://lithus.eu https://lithus.eu, adam@

mgaunard 8mo ago
you're missing 5, what they are doing. There is a world of difference between renting some cabinets in an Equinix datacenter and operating your own.
- adamcharnock 8mo ago
  Fair point! 5 - Datacenter (DC) - Like 4, except also take control of the space/power/HVAC/transit/security side of the equation. Makes sense either at scale, or if you have specific needs. Specific needs could be: specific location, reliability (higher or lower than a DC), resilience (conflict planning). There are actually some really interesting use cases here. For example, reliability: If your company is in a physical office, how strong is the need to run your internal systems in a data centre? If you run your servers in your office, then there's no connectivity reliability concerns. If the power goes out, then the power is out to your staff's computers anyway (still get a UPS though). Or perhaps you don't need as high reliability if you're doing only batch workloads? Do you need to pay the premium for redundant network connections and power supplies? If you want your company to still function in the event of some kind of military conflict, do you really want to rely on fibre optic lines between your office and the data center? Do you want to keep all your infrastructure in such a high-value target? I think this is one of the more interesting areas to think about, at least for me!
  - mgaunard 8mo ago
    Personally I haven't seen a scenario where it makes sense beyond a small experimental lab where you value the ability to tinker physically with the hardware regularly. Offices are usually very expensive real estate in city centers and with very limited cooling capabilities. Then again the US is a different place, they don't have cities like in Europe (bar NYC).
    - kryptiskt 8mo ago
      If you are a bank or a bookmaker or similar you may well want to have total control of physical access to the machines. I know one bookmaker I worked with had their own mini-datacenter, mainly because of physical security.
      - tomcam 8mo ago
        I am pretty forward-thinking but even when I started writing my first web server 30+ years ago I didn’t foresee the day when the phrase “my bookie’s datacenter” might cross my lips.
      - mgaunard 8mo ago
        Most trading venues are in Equinix data centers.
  - noosphr 8mo ago
    5 was a great option for ml work last year since colo rented didn't come with a 10kW cable. With ram, sd and GPU prices the way they are now I have no idea what you'd need to do. Thank goodness we did all the capex before the OpenAI ram deal and expensive nvidia gpus were the worst we had to deal with.
  - direwolf20 8mo ago
    If you have less than a rack of hardware, if you have physical security requirements, and/or your hardware is used in the office more than from the internet, it can make sense.
  - 8mo ago
bpavuk 8mo ago
if someone on the DevOps team knows Nix, option 3 becomes a lot cheaper time-wise! yeah, Nix flakes still need maintenance, especially on the `nixos-unstable` branch, but you get the quickest disaster recovery route possible! plus, infra flexibility removes random constraints that e.g. Cloudflare Workers have
- adamcharnock 8mo ago
  Indeed! We've yet to go down this route, but it's something we're thinking on. A friend and I have been talking about how to bring Nix-like constructs to Kubernetes as well, which has been interesting. (https://github.com/clotodex/kix https://github.com/clotodex/kix, very much in the "this is fun to think about" phase)
- muvlon 8mo ago
  Option 4 as well, that's how we do it at work and it's been great. However, it can't really be "someone on the team knows Nix", anyone working on Ops will need Nix skills in order to be effective.
  - lstodd 8mo ago
    Why this fixation on Nix? You don't need Nix to run bare metal.
    - bpavuk 8mo ago
      Nix makes sure that everything is exactly as you declared, and that in case of [INSERT APOCALYPTIC EVENT], you'll be able to recover much faster
- preisschild 8mo ago
  I'm a NixOS fan, but been using Talos Linux on Hetzner nodes (using Cluster-API) to form a Kubernetes Cluster. Works great too!
- aequitas 8mo ago
  This is what we do, I gave a talk about our setup earlier this week at CfgMgmtCamp: https://www.youtube.com/watch?v=DBxkVVrN0mA&t=8457s https://www.youtube.com/watch?v=DBxkVVrN0mA&t=8457s
DyslexicAtheist 8mo ago
this is what we did in the 90ies into mid 2000: > Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills back then this type of "skill" was abundant. You could easily get sysadmin contractors who would take a drive down to the data-center (probably rented facilities in a real-estate that belonged to a bank or insurance) to exchange some disks that died for some reason. such a person was full stack in a sense that they covered backups, networking, firewalls, and knew how to source hardware. the argument was that this was too expensive and the cloud was better. so hundreds of thousands of SME's embraced the cloud - most of them never needed Google-type of scale, but got sucked into the "recurring revenue" grift that is SaaS. If you opposed this mentality you were basically saying "we as a company will never scale this much" which was at best "toxic" and at worst "career-ending". The thing is these ancient skills still exist. And most orgs simply do not need AWS type of scale. European orgs would do well to revisit these basic ideas. And Hetzner or Lithus would be a much more natural (and honest) fit for these companies.
- theodric 8mo ago
  > ancient skills https://youtu.be/ZtYU87QNjPw?&t=10 https://youtu.be/ZtYU87QNjPw?&t=10 It baffles me that my career trajectory somehow managed to insulate me from ever having to deal with the cloud, while such esoteric skills as swapping a hot swap disk or racking and cabling a new blade chassis are apparently on the order of finding a COBOL developer now. Really? I can promise you that large financial institutions still have datacenters. Many, many, many datacenters!
  - direwolf20 8mo ago
    we had two racks in our office of mostly developers. If you have an office you already have a rack for switches and patch panels. Adding a few servers is obvious. Software development isn't a typical SME however. Mike's Fish and Chips will not buy a server and that's fine.
- belorn 8mo ago
  I wonder how much companies pay yearly in order to avoid having an employee pick up a drive from a local store, drive to the data center, pull the disk drive, screw out the failing hard drive and put in the new one, add it in the raid, verify the repair process has started, and then return to the office.
Lucasoato 8mo ago
Hetzner is definitely an interesting option. I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) but the price difference makes it so appealing. From our financial models, Hetzner can win over AWS when you spend over 10~15K per month on infrastructure and you’re hiring really well. It’s still a risk, but a risk that definitely can be worthy.
- adamcharnock 8mo ago
  You sum it up very neatly. We've heard this from quite a few companies, and that's kind of why we started our ours. We figured, "Okay, if we can do this well, reliably, and de-risk it; then we can offer that as a service and just split the difference on the cost savings" (plus we include engineering time proportional to cluster size, and also do the migration on our own dime as part of the de-risking)
- iso1631 8mo ago
  > I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) Out of interest, how old are you? This was quite normal expectation of a technical department even 15 years ago.
  - christophilus 8mo ago
    I’m curious to know the answer, too. I used to deploy my software on-prem back in the day, and that always included an installation of Microsoft SQL Server. So, all of my clients had at least one database server they had to keep operational. Most of those clients didn’t have an IT staff at all, so if something went wrong (which was exceedingly rare), they’d call me and I’d walk them through diagnosing and fixing things, or I’d Remote Desktop into the server if their firewalls permitted and fix it myself. Backups were automated and would produce an alert if they failed to verify. It’s not rocket science, especially when you’re talking about small amounts of data (small credit union systems in my example).
  - infecto 8mo ago
    No it was not. 15 years ago Heroku was the rage. Even the places that had bare metal usually had someone running something similar to devops and at least core infrar was not being touched. I am sure places existed but 15 years while far away was already pretty far along from what you describe. At least in SV.
preisschild 8mo ago
Been using Hetzner Cloud for Kubernetes and generally like it, but it has its limitations. The network is highly unpredictable. You at best get 2Gbit/s, but at worst a few hundreds of Mbit/s. https://docs.hetzner.com/cloud/technical-details/faq/#what-kind-of-connection-do-the-instances-have https://docs.hetzner.com/cloud/technical-details/faq/#what-k...
- victorbjorklund 8mo ago
  Is that for the virtual private network? I heard some people say that you actually get higher bandwidth if you're using the public network instead of the private network within Hetzner, which is a little bit crazy.
  - direwolf20 8mo ago
    Hetzner dedicated is pretty bad at private networks, so bad you should use a VPN instead. Don't know about the cloud side of things.
weavie 8mo ago
What is the upper limit of Hertzner? Say you have an AWS bill in the $100s of millions, could Hertzner realistically take on that scale?
- adamcharnock 8mo ago
  An interesting question, so time for some 100% speculation. It sounds like they probably have revenue in the €500mm range today. And given that the bare metal cost of AWS-equivalent bills tends to be a 90% reduction, we'll say a €10mm+ bare metal cost. So I would say a cautious and qualified "yes". But I know even for smaller deployments of tens or hundreds of servers, they'll ask you what the purpose is. If you say something like "blockchain," they're going to say, "Actually, we prefer not to have your business." I get the strong impression that while they naturally do want business, they also aren't going to take a huge amount of risk on board themselves. Their specialism is optimising on cost, which naturally has to involve avoiding or mitigating risk. I'm sure there'd be business terms to discuss, put it that way.
  - StilesCrisis 8mo ago
    Why would a client who wants to run a Blockchain be risky for Herzner? I'm not a fan, I just don't see the issue. If the client pays their monthly bill, who cares if they're using the machine to mine for Bitcoin?
    - Symbiote 8mo ago
      They are certain to run the machines at 100% continually, which will cost more than a typical customer who doesn't do this, and leave the old machines with less second-hand value for their auction thing afterwards.
      - mbreese 8mo ago
        I’d bet that main reason would be power. Running machines at 100% doesn’t subtract much extra , but a server running hard for 24 hours would use more power than a bursty workload. (While we’re all speculating)
Schlagbohrer 8mo ago
Can someone explain 2 to me. How is a managed private cloud different from full cloud? Like you are still using AWS or Azure but you are keeping all your operation in a bundled, portable way, so you can leave that provider easily at any time, rather than becoming very dependent on them? Is it like staying provider-agnostic but still cloud based?
- adamcharnock 8mo ago
  To put it plainly: We deploy a Kubernetes cluster on Hetzner dedicated servers and become your DevOps team (or a part thereof). It works because bare metal is about 10% the cost of cloud, and our value-add is in 1) creating a resilient platform on top of that, 2) supporting it, 3) being on-call, and 4) being or supporting your DevOps team. This starts with us providing a Kubernetes cluster which we manage, but we also take responsibility for the services run on it. If you want Postgres, Redis, Clickhouse, NATS, etc, we'll deploy it and be SLA-on-call for any issues. If you don't want to deal with Kubernetes then you don't have to. Just have your software engineers hand us the software and we'll handle deployment. Everything is deployed on open source tooling, you have access to all the configuration for the services we deploy. You have server root access. If you want to leave you can do. Our customers have full root access, and our engineers (myself included) are in a Slack channel with you engineers. And, FWIW, it doesn't have to be Hetzner. We can colocate or use other providers, but Hetzner offer excellent bang-per-buck. Edit: And all this is included in the cluster price, which comes out cheaper than the same hardware on the major cloud providers
  - Annatar 8mo ago
    [dead]
  - mancerayder 8mo ago
    You give customers root but you're on call when something goes tits up? You're a brave DevOps team. That would cause a lot of friction in my experience, since people with root or other administrative privileges do naughty things, but others are getting called in on Saturday afternoon.
CrzyLngPwd 8mo ago
#2.5ish We rent hardware and also some VPS, as well as use AWS for cheap things such as S3 fronted with Cloudflare, and SES for priority emails. We have other services we pay for, such as AI content detection, disposable email detection, a small postal email server, and more. We're only a small business, so having predictable monthly costs is vital. Our servers are far from maxed out, and we process ~4 million dynamic page and API requests per day.
torginus 8mo ago
I think the issue with this formulation is what drives the cost at cloud providers isn't necessarily that their hardware is too expensive (which it is), but that they push you towards overcomplicated and inefficient architectures that cost too much to run. A core at this are all the 'managed' services - if you have a server box, its in your financial interest to squeeze as much per out of it as possible. If you're using something like ECS or serverless, AWS gains nothing by optimizing the servers to make your code run faster - their hard work results in less billed infrastructure hours. This 'microservices' push usually means that instead of having an on-server session where you can serve stuff from a temporary cache, all the data that persists between requests needs to be stored in a db somewhere, all the auth logic needs to re-check your credentials, and something needs to direct the traffic and load balance these endpoint, and all this stuff costs money. I think if you have 4 Java boxes as servers with a redundant DB with read replicas on EC2, your infra is so efficient and cheap that even paying 4x for it rather than going for colocation is well worth it because of the QoL and QoS. These crazy AWS bills usually come from using every service under the sun.
- mrweasel 8mo ago
  Just this week a friend of mine was spinning up some AWS managed service, complaining about the complexity, and how any reconfiguration took 45 minutes to reload. It's a service you can just install with apt, the default configuration is fine. Not only is many service no longer cheaper in the cloud, the management overhead also exceed that of on-prem.
  - mystifyingpoi 8mo ago
    I'd gladly use (and maybe even pay for!) an open-source reimplementation of AWS RDS Aurora. All the bells and whistles with failover, clustering, volume-based snaps, cross-region replication, metrics etc. As far as I know, nothing comes close to Aurora functionality. Even in vibecoding world. No, 'apt-get install postgres' is not enough.
    - SOLAR_FIELDS 8mo ago
eru 8mo ago
> 4 - Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills, scale, cap-ex, and if you plan to run the servers for at least 3-5 years. Is it still the cheapest after you take into account that skills, scale, cap-ex and long term lock-in also have opportunity costs?
- graemep 8mo ago
  That is why the the second "if" is there. You can get locked into cloud too. The lock in is not really long term as it is an easy option to migrate off.
- chrisandchris 8mo ago
  Personal experience: Did some cloud stuff for SME, and later on started colocation. I think my learning curve for all the cloud-stuff was the same as for all the colocation stuff, except the cloud will not get you rid of firewalls, NAT, DHCP and all that stuff. Cloud isn't that much easier, it's just a little bit different. IMHO, the largest disadvantage of colocation is that it requires (sometimes) physical presence at a datacenter.
bojangleslover 8mo ago
Great comment. I agree it's a spectrum and those of us who are comfortable on (4) like yourself and probably us at Carolina Cloud [0] as well, (4) seems like a no brainer. But there's a long tail of semi-technical users who are more comfortable in 2-3 or even 1, which is what ultimately traps them into the ransomware-adjacent situation that is a lot of the modern public cloud. I would push back on "usage-based". Yes it is technically usage-based but the base fee also goes way up and there are also sometimes retainers on these services (ie minimum spend). So of course "usage-based" is not wrong but what it usually means is "more expensive and potentially far more expensive". [0] https://carolinacloud.io https://carolinacloud.io, derek@
- spwa4 8mo ago
  The problem is that clouds have easily become 3 or 5 times the price of managed services, 10x the price of option 3, and 20x the price of option 4. To say nothing of the fact that almost all businesses can run fine on "pc under desk" type situations. So in practice cloud has become the more expensive option the second your spend goes over the price of 1 engineer.
Archelaos 8mo ago
I am using something inbetween 2 and 3, a hosted Web-site and database service with excellent customer support. On shared hardware it is 22 €/month. A managed server on dedicated hardware starts at about 50 €/month.
boplicity 8mo ago
I don't know. I rent a bare metal server for $500 a month, which is way overkill. It takes almost no time to manage -- maybe a few hours a year -- and can handle almost anything I throw at it. Maybe my needs are too simple though?
- edge17 8mo ago
  Just curious, what is the spec you pay $6000/year for? Where/what is the line between rent vs buy?
  - boplicity 8mo ago
    It's a server with: - 2x Intel Xeon 5218 - 128gb Ram - 2x960GB SSD - 30TB monthly bandwidth I pay around an extra $200/month for "premium" support and Acronis backups, both of which have come in handy, but are probably not necessary. (Automated backups to AWS are actually pretty cheap.) It definitely helps with peace of mind, though.
    - cheema33 8mo ago
      I have a similar system from Hetzner. I pay around $100 for it. No bandwidth cap. I have setup encrypted backups to go to my backup server in the office. We have a gigabit service at the office. Critical data changes are backed up every hour and full backup once a day.
      - boplicity 8mo ago
        Yeah -- I know I could probably get a better deal. I pay more for premium support ($200), as well as a North American location. Plus, probably an addition premium for not wanting to go through the effort of switching servers.
jgalt212 8mo ago
We looked at option 4. And colocation is not cheap. It was cheaper for us to lease VMs from Hetzner than to buy boxes and colocate at Equinix.
whiplash451 8mo ago
> Option 1 is great for startups Unfortunately, (successful) startups can quickly get trapped into this option. If they're growing fast, everyone on the board will ask why you'd move to another option at the first place. The cloud becomes a very deep local minimum that's hard to get out off.
adarsh2321 8mo ago
[dead]
ibejoeb 8mo ago
Dead on. Recently, 3 and 4 have been compelling. Cloud costs have rocketed up. I started my casual transition to co-lo 2 years ago and just in december finished everything. I have more capacity at about 30% of the cost. If you go option 3, you even get the benefit of 6+ month retro pricing for RAM/storage. I'm running all DDR4, but I have so much of it I don't know what to do with it. The flip side is that compliance is a little more involved. Rather than, say, carve out a whole swathe of SOC-2 ops, I have to coordinate some controls. It's not a lot, and it's still a lot lighter than I used to do 10+ years ago. Just something to consider.
sanderjd 8mo ago
This space of #2 like Lithus is not something I'm very familiar with, so thank you for the comment that piqued my interest! If you're willing to share, I'm curious who else you would describe as being in this space. My last decade and a half or so of experience has all been in cloud services, and prior to that it was #3 or #4. What was striking to me when I went to the Lithus website was that I couldn't figure out any details without hitting a "Schedule a Call" button. This makes it difficult for me to map my experiences in using cloud services onto what Lithus offers. Can I use terraform? How does the kubernetes offering work? How does the ML/AI data pipelines work? To me, it would be nice if I could try it out in a very limited way as self-service, or at least read some technical documentation. Without that, I'm left wondering how it works. I'm sure this is a conscious decision to not do this, and for good reasons, but I thought I'd share my impressions!
- adamcharnock 8mo ago
  Hello! I think this is a fair question, and improving the communication on the website is something that is steadily climbing up our priority list. We're not really that kind of product company; we're more of a services company. What we do is deploy Kubernetes clusters onto bare metal servers. That's the core technical offering. However, everything beyond that is somewhat per-client. Some clients need a lot of compute. Some clients need a custom object storage cluster. Some clients need a lot of high-speed internal networking. Which is why we prefer to have a call to figure out specifically what your needs are. But I can also see how this isn't necessarily satisfying if you're used to just grabbing the API docs and having a look around. What we will do is take your company's software stack and migrate it off AWS/Azure/Google and deploy it onto our new infrastructure. We will then become (or work with) your DevOps team to supporting you. This can be anything from containerising workloads to diagnosing performance issues to deploying a new multi-region Postgres cluster. Whatever you need done on your hardware that we feel we can reasonably support. We are the ones on-call should NATS fall over at 4am. Your team also has full access to the Kubernetes cluster to deploy to as you wish. I think the pricing page is the most concrete thing on our website, and it is entirely accurate. If you were to phone us and say, "I want that exact hardware," we would do it for you. But the real value we also offer is in the DevOps support we provide, actually doing the migration up-front (at our own cost), and being there working with your team every week.
rcpt 8mo ago
5. On-premise and engineers touch the wires every few days.
megggan 8mo ago
Getting rid of bureaucratic internal IT department is a game changer for productivity. That alone is worth 10x infra costs, especially for big companies where work can grind to a halt dealing with obstructionists through service now. Good leaders understand this.
- bell-cot 8mo ago
  Sadly true. Or, the so-called internal IT Dept. can be a shambolic mess of PHB's, Brunchlords, Catberts, metric maximizers, and micromanagers, presiding over the hollowed-out and burned out remains of the actual workforce that you'd need to reliably do the job.
doctorpangloss 8mo ago
Where do AWS reserved instances come into your hierarchy? What if there existed a “perpetual” reserved instance? Is cap-ex vs. op-ex really the key distinction?
themafia 8mo ago
> while largely maximising operational costs The core services are cheap. S3 is cheap. Dynamo is cheap. Lambda is exceedingly cheap. Not understanding these services on their own terms and failing to read the documentation can lead one to use them in highly inefficient ways. The "cloud" isn't just "another type of server." It's another type of /service/. Every costly stack I've seen fails to accept this truth.
LeFantome 8mo ago
This is a good summary and my take as well. The cloud is great when you just need to start and when you do not know what scale you will need. Minimal initial cost and no wasted time over planning things you do not know enough about. The cloud is horrible for steady-state demand. You are over-paying for your base load. If your demand does not scale that much, you do not benefit from the flexibility. Distance from the edge can cause performance problems. In an effort to “save money” you will chase complexity and bake in total reliance on your cloud provider. AAA l Starting with the cloud makes sense. Just make sure not to engineer a solution you cannot take somewhere else. As you scale and demand becomes known, you can start to migrate some stuff on premises or to other managed providers. The great thing about “cloud architecture” is that you can use a hybrid model. You can selectively move parts of the stack. You can host your baseline demand and still rely on the cloud for scalability. Where you need to spend the money and gain the expertise is in design. Not a giant features waterfall but rather knowing how to build an application and infrastructure that is adaptable and portable as you scale. Keep it simple but also keep it modular. At least, that had been my experience.

mgaunard 8mo ago

you're missing 5, what they are doing. There is a world of difference between renting some cabinets in an Equinix datacenter and operating your own.

adamcharnock 8mo ago
Fair point! 5 - Datacenter (DC) - Like 4, except also take control of the space/power/HVAC/transit/security side of the equation. Makes sense either at scale, or if you have specific needs. Specific needs could be: specific location, reliability (higher or lower than a DC), resilience (conflict planning). There are actually some really interesting use cases here. For example, reliability: If your company is in a physical office, how strong is the need to run your internal systems in a data centre? If you run your servers in your office, then there's no connectivity reliability concerns. If the power goes out, then the power is out to your staff's computers anyway (still get a UPS though). Or perhaps you don't need as high reliability if you're doing only batch workloads? Do you need to pay the premium for redundant network connections and power supplies? If you want your company to still function in the event of some kind of military conflict, do you really want to rely on fibre optic lines between your office and the data center? Do you want to keep all your infrastructure in such a high-value target? I think this is one of the more interesting areas to think about, at least for me!
- mgaunard 8mo ago
  Personally I haven't seen a scenario where it makes sense beyond a small experimental lab where you value the ability to tinker physically with the hardware regularly. Offices are usually very expensive real estate in city centers and with very limited cooling capabilities. Then again the US is a different place, they don't have cities like in Europe (bar NYC).
  - kryptiskt 8mo ago
    If you are a bank or a bookmaker or similar you may well want to have total control of physical access to the machines. I know one bookmaker I worked with had their own mini-datacenter, mainly because of physical security.
    - tomcam 8mo ago
      I am pretty forward-thinking but even when I started writing my first web server 30+ years ago I didn’t foresee the day when the phrase “my bookie’s datacenter” might cross my lips.
    - mgaunard 8mo ago
      Most trading venues are in Equinix data centers.
- noosphr 8mo ago
  5 was a great option for ml work last year since colo rented didn't come with a 10kW cable. With ram, sd and GPU prices the way they are now I have no idea what you'd need to do. Thank goodness we did all the capex before the OpenAI ram deal and expensive nvidia gpus were the worst we had to deal with.
- direwolf20 8mo ago
  If you have less than a rack of hardware, if you have physical security requirements, and/or your hardware is used in the office more than from the internet, it can make sense.
- 8mo ago

bpavuk 8mo ago

if someone on the DevOps team knows Nix, option 3 becomes a lot cheaper time-wise! yeah, Nix flakes still need maintenance, especially on the `nixos-unstable` branch, but you get the quickest disaster recovery route possible! plus, infra flexibility removes random constraints that e.g. Cloudflare Workers have

adamcharnock 8mo ago
Indeed! We've yet to go down this route, but it's something we're thinking on. A friend and I have been talking about how to bring Nix-like constructs to Kubernetes as well, which has been interesting. (https://github.com/clotodex/kix https://github.com/clotodex/kix, very much in the "this is fun to think about" phase)
muvlon 8mo ago
Option 4 as well, that's how we do it at work and it's been great. However, it can't really be "someone on the team knows Nix", anyone working on Ops will need Nix skills in order to be effective.
- lstodd 8mo ago
  Why this fixation on Nix? You don't need Nix to run bare metal.
  - bpavuk 8mo ago
    Nix makes sure that everything is exactly as you declared, and that in case of [INSERT APOCALYPTIC EVENT], you'll be able to recover much faster
preisschild 8mo ago
I'm a NixOS fan, but been using Talos Linux on Hetzner nodes (using Cluster-API) to form a Kubernetes Cluster. Works great too!
aequitas 8mo ago
This is what we do, I gave a talk about our setup earlier this week at CfgMgmtCamp: https://www.youtube.com/watch?v=DBxkVVrN0mA&t=8457s https://www.youtube.com/watch?v=DBxkVVrN0mA&t=8457s

DyslexicAtheist 8mo ago

this is what we did in the 90ies into mid 2000: > Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills back then this type of "skill" was abundant. You could easily get sysadmin contractors who would take a drive down to the data-center (probably rented facilities in a real-estate that belonged to a bank or insurance) to exchange some disks that died for some reason. such a person was full stack in a sense that they covered backups, networking, firewalls, and knew how to source hardware. the argument was that this was too expensive and the cloud was better. so hundreds of thousands of SME's embraced the cloud - most of them never needed Google-type of scale, but got sucked into the "recurring revenue" grift that is SaaS. If you opposed this mentality you were basically saying "we as a company will never scale this much" which was at best "toxic" and at worst "career-ending". The thing is these ancient skills still exist. And most orgs simply do not need AWS type of scale. European orgs would do well to revisit these basic ideas. And Hetzner or Lithus would be a much more natural (and honest) fit for these companies.

theodric 8mo ago
> ancient skills https://youtu.be/ZtYU87QNjPw?&t=10 https://youtu.be/ZtYU87QNjPw?&t=10 It baffles me that my career trajectory somehow managed to insulate me from ever having to deal with the cloud, while such esoteric skills as swapping a hot swap disk or racking and cabling a new blade chassis are apparently on the order of finding a COBOL developer now. Really? I can promise you that large financial institutions still have datacenters. Many, many, many datacenters!
- direwolf20 8mo ago
  we had two racks in our office of mostly developers. If you have an office you already have a rack for switches and patch panels. Adding a few servers is obvious. Software development isn't a typical SME however. Mike's Fish and Chips will not buy a server and that's fine.
belorn 8mo ago
I wonder how much companies pay yearly in order to avoid having an employee pick up a drive from a local store, drive to the data center, pull the disk drive, screw out the failing hard drive and put in the new one, add it in the raid, verify the repair process has started, and then return to the office.

Lucasoato 8mo ago

Hetzner is definitely an interesting option. I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) but the price difference makes it so appealing. From our financial models, Hetzner can win over AWS when you spend over 10~15K per month on infrastructure and you’re hiring really well. It’s still a risk, but a risk that definitely can be worthy.

adamcharnock 8mo ago
You sum it up very neatly. We've heard this from quite a few companies, and that's kind of why we started our ours. We figured, "Okay, if we can do this well, reliably, and de-risk it; then we can offer that as a service and just split the difference on the cost savings" (plus we include engineering time proportional to cluster size, and also do the migration on our own dime as part of the de-risking)
iso1631 8mo ago
> I’m a bit scared of managing the services on my own (like Postgres, Site2Site VPN, …) Out of interest, how old are you? This was quite normal expectation of a technical department even 15 years ago.
- christophilus 8mo ago
  I’m curious to know the answer, too. I used to deploy my software on-prem back in the day, and that always included an installation of Microsoft SQL Server. So, all of my clients had at least one database server they had to keep operational. Most of those clients didn’t have an IT staff at all, so if something went wrong (which was exceedingly rare), they’d call me and I’d walk them through diagnosing and fixing things, or I’d Remote Desktop into the server if their firewalls permitted and fix it myself. Backups were automated and would produce an alert if they failed to verify. It’s not rocket science, especially when you’re talking about small amounts of data (small credit union systems in my example).
- infecto 8mo ago
  No it was not. 15 years ago Heroku was the rage. Even the places that had bare metal usually had someone running something similar to devops and at least core infrar was not being touched. I am sure places existed but 15 years while far away was already pretty far along from what you describe. At least in SV.

preisschild 8mo ago

Been using Hetzner Cloud for Kubernetes and generally like it, but it has its limitations. The network is highly unpredictable. You at best get 2Gbit/s, but at worst a few hundreds of Mbit/s. https://docs.hetzner.com/cloud/technical-details/faq/#what-kind-of-connection-do-the-instances-have https://docs.hetzner.com/cloud/technical-details/faq/#what-k...

victorbjorklund 8mo ago
Is that for the virtual private network? I heard some people say that you actually get higher bandwidth if you're using the public network instead of the private network within Hetzner, which is a little bit crazy.
- direwolf20 8mo ago
  Hetzner dedicated is pretty bad at private networks, so bad you should use a VPN instead. Don't know about the cloud side of things.

weavie 8mo ago

What is the upper limit of Hertzner? Say you have an AWS bill in the $100s of millions, could Hertzner realistically take on that scale?

adamcharnock 8mo ago
An interesting question, so time for some 100% speculation. It sounds like they probably have revenue in the €500mm range today. And given that the bare metal cost of AWS-equivalent bills tends to be a 90% reduction, we'll say a €10mm+ bare metal cost. So I would say a cautious and qualified "yes". But I know even for smaller deployments of tens or hundreds of servers, they'll ask you what the purpose is. If you say something like "blockchain," they're going to say, "Actually, we prefer not to have your business." I get the strong impression that while they naturally do want business, they also aren't going to take a huge amount of risk on board themselves. Their specialism is optimising on cost, which naturally has to involve avoiding or mitigating risk. I'm sure there'd be business terms to discuss, put it that way.
- StilesCrisis 8mo ago
  Why would a client who wants to run a Blockchain be risky for Herzner? I'm not a fan, I just don't see the issue. If the client pays their monthly bill, who cares if they're using the machine to mine for Bitcoin?
  - Symbiote 8mo ago
    They are certain to run the machines at 100% continually, which will cost more than a typical customer who doesn't do this, and leave the old machines with less second-hand value for their auction thing afterwards.
    - mbreese 8mo ago
      I’d bet that main reason would be power. Running machines at 100% doesn’t subtract much extra , but a server running hard for 24 hours would use more power than a bursty workload. (While we’re all speculating)

Schlagbohrer 8mo ago

Can someone explain 2 to me. How is a managed private cloud different from full cloud? Like you are still using AWS or Azure but you are keeping all your operation in a bundled, portable way, so you can leave that provider easily at any time, rather than becoming very dependent on them? Is it like staying provider-agnostic but still cloud based?

adamcharnock 8mo ago
To put it plainly: We deploy a Kubernetes cluster on Hetzner dedicated servers and become your DevOps team (or a part thereof). It works because bare metal is about 10% the cost of cloud, and our value-add is in 1) creating a resilient platform on top of that, 2) supporting it, 3) being on-call, and 4) being or supporting your DevOps team. This starts with us providing a Kubernetes cluster which we manage, but we also take responsibility for the services run on it. If you want Postgres, Redis, Clickhouse, NATS, etc, we'll deploy it and be SLA-on-call for any issues. If you don't want to deal with Kubernetes then you don't have to. Just have your software engineers hand us the software and we'll handle deployment. Everything is deployed on open source tooling, you have access to all the configuration for the services we deploy. You have server root access. If you want to leave you can do. Our customers have full root access, and our engineers (myself included) are in a Slack channel with you engineers. And, FWIW, it doesn't have to be Hetzner. We can colocate or use other providers, but Hetzner offer excellent bang-per-buck. Edit: And all this is included in the cluster price, which comes out cheaper than the same hardware on the major cloud providers
- Annatar 8mo ago
  [dead]
- mancerayder 8mo ago
  You give customers root but you're on call when something goes tits up? You're a brave DevOps team. That would cause a lot of friction in my experience, since people with root or other administrative privileges do naughty things, but others are getting called in on Saturday afternoon.

CrzyLngPwd 8mo ago

#2.5ish We rent hardware and also some VPS, as well as use AWS for cheap things such as S3 fronted with Cloudflare, and SES for priority emails. We have other services we pay for, such as AI content detection, disposable email detection, a small postal email server, and more. We're only a small business, so having predictable monthly costs is vital. Our servers are far from maxed out, and we process ~4 million dynamic page and API requests per day.

torginus 8mo ago

I think the issue with this formulation is what drives the cost at cloud providers isn't necessarily that their hardware is too expensive (which it is), but that they push you towards overcomplicated and inefficient architectures that cost too much to run. A core at this are all the 'managed' services - if you have a server box, its in your financial interest to squeeze as much per out of it as possible. If you're using something like ECS or serverless, AWS gains nothing by optimizing the servers to make your code run faster - their hard work results in less billed infrastructure hours. This 'microservices' push usually means that instead of having an on-server session where you can serve stuff from a temporary cache, all the data that persists between requests needs to be stored in a db somewhere, all the auth logic needs to re-check your credentials, and something needs to direct the traffic and load balance these endpoint, and all this stuff costs money. I think if you have 4 Java boxes as servers with a redundant DB with read replicas on EC2, your infra is so efficient and cheap that even paying 4x for it rather than going for colocation is well worth it because of the QoL and QoS. These crazy AWS bills usually come from using every service under the sun.

mrweasel 8mo ago
Just this week a friend of mine was spinning up some AWS managed service, complaining about the complexity, and how any reconfiguration took 45 minutes to reload. It's a service you can just install with apt, the default configuration is fine. Not only is many service no longer cheaper in the cloud, the management overhead also exceed that of on-prem.
- mystifyingpoi 8mo ago
  I'd gladly use (and maybe even pay for!) an open-source reimplementation of AWS RDS Aurora. All the bells and whistles with failover, clustering, volume-based snaps, cross-region replication, metrics etc. As far as I know, nothing comes close to Aurora functionality. Even in vibecoding world. No, 'apt-get install postgres' is not enough.
  - SOLAR_FIELDS 8mo ago

eru 8mo ago

> 4 - Buy and colocate the hardware yourself – Certainly the cheapest option if you have the skills, scale, cap-ex, and if you plan to run the servers for at least 3-5 years. Is it still the cheapest after you take into account that skills, scale, cap-ex and long term lock-in also have opportunity costs?

graemep 8mo ago
That is why the the second "if" is there. You can get locked into cloud too. The lock in is not really long term as it is an easy option to migrate off.
chrisandchris 8mo ago
Personal experience: Did some cloud stuff for SME, and later on started colocation. I think my learning curve for all the cloud-stuff was the same as for all the colocation stuff, except the cloud will not get you rid of firewalls, NAT, DHCP and all that stuff. Cloud isn't that much easier, it's just a little bit different. IMHO, the largest disadvantage of colocation is that it requires (sometimes) physical presence at a datacenter.

bojangleslover 8mo ago

Great comment. I agree it's a spectrum and those of us who are comfortable on (4) like yourself and probably us at Carolina Cloud [0] as well, (4) seems like a no brainer. But there's a long tail of semi-technical users who are more comfortable in 2-3 or even 1, which is what ultimately traps them into the ransomware-adjacent situation that is a lot of the modern public cloud. I would push back on "usage-based". Yes it is technically usage-based but the base fee also goes way up and there are also sometimes retainers on these services (ie minimum spend). So of course "usage-based" is not wrong but what it usually means is "more expensive and potentially far more expensive". [0] https://carolinacloud.io https://carolinacloud.io, derek@

spwa4 8mo ago
The problem is that clouds have easily become 3 or 5 times the price of managed services, 10x the price of option 3, and 20x the price of option 4. To say nothing of the fact that almost all businesses can run fine on "pc under desk" type situations. So in practice cloud has become the more expensive option the second your spend goes over the price of 1 engineer.

Archelaos 8mo ago

I am using something inbetween 2 and 3, a hosted Web-site and database service with excellent customer support. On shared hardware it is 22 €/month. A managed server on dedicated hardware starts at about 50 €/month.

boplicity 8mo ago

I don't know. I rent a bare metal server for $500 a month, which is way overkill. It takes almost no time to manage -- maybe a few hours a year -- and can handle almost anything I throw at it. Maybe my needs are too simple though?

edge17 8mo ago
Just curious, what is the spec you pay $6000/year for? Where/what is the line between rent vs buy?
- boplicity 8mo ago
  It's a server with: - 2x Intel Xeon 5218 - 128gb Ram - 2x960GB SSD - 30TB monthly bandwidth I pay around an extra $200/month for "premium" support and Acronis backups, both of which have come in handy, but are probably not necessary. (Automated backups to AWS are actually pretty cheap.) It definitely helps with peace of mind, though.
  - cheema33 8mo ago
    I have a similar system from Hetzner. I pay around $100 for it. No bandwidth cap. I have setup encrypted backups to go to my backup server in the office. We have a gigabit service at the office. Critical data changes are backed up every hour and full backup once a day.
    - boplicity 8mo ago
      Yeah -- I know I could probably get a better deal. I pay more for premium support ($200), as well as a North American location. Plus, probably an addition premium for not wanting to go through the effort of switching servers.

jgalt212 8mo ago

We looked at option 4. And colocation is not cheap. It was cheaper for us to lease VMs from Hetzner than to buy boxes and colocate at Equinix.

whiplash451 8mo ago

> Option 1 is great for startups Unfortunately, (successful) startups can quickly get trapped into this option. If they're growing fast, everyone on the board will ask why you'd move to another option at the first place. The cloud becomes a very deep local minimum that's hard to get out off.

adarsh2321 8mo ago

[dead]

ibejoeb 8mo ago

Dead on. Recently, 3 and 4 have been compelling. Cloud costs have rocketed up. I started my casual transition to co-lo 2 years ago and just in december finished everything. I have more capacity at about 30% of the cost. If you go option 3, you even get the benefit of 6+ month retro pricing for RAM/storage. I'm running all DDR4, but I have so much of it I don't know what to do with it. The flip side is that compliance is a little more involved. Rather than, say, carve out a whole swathe of SOC-2 ops, I have to coordinate some controls. It's not a lot, and it's still a lot lighter than I used to do 10+ years ago. Just something to consider.

sanderjd 8mo ago

This space of #2 like Lithus is not something I'm very familiar with, so thank you for the comment that piqued my interest! If you're willing to share, I'm curious who else you would describe as being in this space. My last decade and a half or so of experience has all been in cloud services, and prior to that it was #3 or #4. What was striking to me when I went to the Lithus website was that I couldn't figure out any details without hitting a "Schedule a Call" button. This makes it difficult for me to map my experiences in using cloud services onto what Lithus offers. Can I use terraform? How does the kubernetes offering work? How does the ML/AI data pipelines work? To me, it would be nice if I could try it out in a very limited way as self-service, or at least read some technical documentation. Without that, I'm left wondering how it works. I'm sure this is a conscious decision to not do this, and for good reasons, but I thought I'd share my impressions!

adamcharnock 8mo ago
Hello! I think this is a fair question, and improving the communication on the website is something that is steadily climbing up our priority list. We're not really that kind of product company; we're more of a services company. What we do is deploy Kubernetes clusters onto bare metal servers. That's the core technical offering. However, everything beyond that is somewhat per-client. Some clients need a lot of compute. Some clients need a custom object storage cluster. Some clients need a lot of high-speed internal networking. Which is why we prefer to have a call to figure out specifically what your needs are. But I can also see how this isn't necessarily satisfying if you're used to just grabbing the API docs and having a look around. What we will do is take your company's software stack and migrate it off AWS/Azure/Google and deploy it onto our new infrastructure. We will then become (or work with) your DevOps team to supporting you. This can be anything from containerising workloads to diagnosing performance issues to deploying a new multi-region Postgres cluster. Whatever you need done on your hardware that we feel we can reasonably support. We are the ones on-call should NATS fall over at 4am. Your team also has full access to the Kubernetes cluster to deploy to as you wish. I think the pricing page is the most concrete thing on our website, and it is entirely accurate. If you were to phone us and say, "I want that exact hardware," we would do it for you. But the real value we also offer is in the DevOps support we provide, actually doing the migration up-front (at our own cost), and being there working with your team every week.

rcpt 8mo ago

5. On-premise and engineers touch the wires every few days.

megggan 8mo ago

Getting rid of bureaucratic internal IT department is a game changer for productivity. That alone is worth 10x infra costs, especially for big companies where work can grind to a halt dealing with obstructionists through service now. Good leaders understand this.

bell-cot 8mo ago
Sadly true. Or, the so-called internal IT Dept. can be a shambolic mess of PHB's, Brunchlords, Catberts, metric maximizers, and micromanagers, presiding over the hollowed-out and burned out remains of the actual workforce that you'd need to reliably do the job.

doctorpangloss 8mo ago

Where do AWS reserved instances come into your hierarchy? What if there existed a “perpetual” reserved instance? Is cap-ex vs. op-ex really the key distinction?

themafia 8mo ago

> while largely maximising operational costs The core services are cheap. S3 is cheap. Dynamo is cheap. Lambda is exceedingly cheap. Not understanding these services on their own terms and failing to read the documentation can lead one to use them in highly inefficient ways. The "cloud" isn't just "another type of server." It's another type of /service/. Every costly stack I've seen fails to accept this truth.

LeFantome 8mo ago

This is a good summary and my take as well. The cloud is great when you just need to start and when you do not know what scale you will need. Minimal initial cost and no wasted time over planning things you do not know enough about. The cloud is horrible for steady-state demand. You are over-paying for your base load. If your demand does not scale that much, you do not benefit from the flexibility. Distance from the edge can cause performance problems. In an effort to “save money” you will chase complexity and bake in total reliance on your cloud provider. AAA l Starting with the cloud makes sense. Just make sure not to engineer a solution you cannot take somewhere else. As you scale and demand becomes known, you can start to migrate some stuff on premises or to other managed providers. The great thing about “cloud architecture” is that you can use a hybrid model. You can selectively move parts of the stack. You can host your baseline demand and still rely on the cloud for scalability. Where you need to spend the money and gain the expertise is in design. Not a giant features waterfall but rather knowing how to build an application and infrastructure that is adaptable and portable as you scale. Keep it simple but also keep it modular. At least, that had been my experience.