4 ms·

A bit less terrible way in my opinion: Find a dedicated server provider and rent the hardware. These companies rent some part of the datacenter (or sometimes b

by krab 2y ago

A bit less terrible way in my opinion: Find a dedicated server provider and rent the hardware. These companies rent some part of the datacenter (or sometimes build their own). Bonus points if they offer KVM - as in remote console, not the Linux hypervisor. Also ask if they do hardware monitoring and proactively replace the failed parts. All of this is still way cheaper than cloud. Usually with unmetered networking. Way less hassle. They'll even take your existing stuff and put it into the same rack with the rented hardware. The difference from cloud, apart from the price, is mainly that they have a sales rep instead of an API. And getting a server may take a from few hours to a few days. But in the end you get the same SSH login details you would get from a cloud provider. Or, if you really want to just collocate your boxes, the providers offer "remote hands" service, so you can have geo-redundancy or just choose a better deal instead of one that's physically close to your place.

rsanheim 2y ago
One hurdle that many companies who have only known cloud hosting will face here is significant: how do you find a reliable, trustworthy datacenter? One who actually monitors the hardware and also has a real human if your networking access gets screwed or if you need a critical component swapped at 2 am on a Saturday. I used to have a short list of trustworthy companies like this I'd recommend to clients ~20 years ago when doing consulting. I think 3/4 of them have been gobbled up by private equity chop shops or are just gone. Nowadays noone gets fired for going with AWS, or resold AWS with a 100% markup from a 'private enterprise cloud' provider.
- krab 2y ago
  You're right you need to find a company you can trust. And for a lot of startups it really makes sense to use AWS. But if you do something resource or bandwidth intensive (and I'm not even talking about Llama now), the costs add up quickly. In our case, switching to AWS would increase our costs by an equivalent of 4 - 8 devs salaries. After AWS discounts. That's a hard sell in a 15-person team even though half of our infra costs already are with AWS (S3).
  - vidarh 2y ago
    I often recommend a "embrace, extend, extinguish" approach to AWS: Starting there for simplicity is fine, then "wrap" anything bandwidth intensive with caches elsewhere (every 1TB in egress from AWS will pay for a fleet of Hetzner instances with 5TB included, or one or more dedicated servers). Gradually shift workloads, leaving anything requiring super-high durability last (optionally keeping S3, or competitors, as a backup storage option) as getting durability right is one of the more difficult things to get confidence in and most dangerous ones to get wrong. Wrapping S3 with a write-through cache setup can often be the biggest cost win if your egress costs are high. Sometimes caching the entire dataset is worth it, sometimes just a small portion.
    - krab 2y ago
      Well, S3 is hard to beat for our use case. We make a heavy use of their various tires, we store a somewhat large amount of data but only a minor part ever goes out. The compute and network heavy stuff we do is still out of AWS.
      - vidarh 2y ago
        That's pretty much the one situation where they're competitive, so sounds very reasonable. Some of their competitors (Cloudflare, Backblaze) might compete, but the biggest cost issue with S3 by far is the egress so if not much goes out it might still be best for you to stay there. Sounds like (unlike most people who use AWS) you've done your homework. It's great to see. I've used AWS a lot, and will again, because it's often convenient, but so often I see people doing it uncritically without modeling their costs even as it skyrockets with scale.
      - throwaway384638 2y ago
        S3 is a decent product with zero competition. You should keep s3, it’s a fair price.
        vidarh 2y ago
        S3 has plenty of competition. It can be a fair price if you rarely read and need it's extreme durability, but that leaves plenty of uses it's totally wrong for.
- chpatrick 2y ago
  I think if you want to host in Europe then Hetzner is the clear choice. They won't monitor your hardware for you though, you need to let them know if something breaks and they'll replace it very quickly.
- jareklupinski 2y ago
  > how do you find a reliable, trustworthy datacenter? drive to a few, and shake some hands. in my exp, the difference between colos is usually "actual SOC2/ISO compliance" on one side, and "there are no locked doors between the parking lot and my rack" on the other, with not much in-between that's not for some specialty (radio), and these things can only really be seen for yourself
  - hedora 2y ago
    That’s unfortunate. I consider SoC 2 compliance as a negative indicator of security (I’ve been on the vendor side of it, and have seen it have significant negative impacts on security and reliability in multiple organizations). Ideally, there’d be locked doors, and the data center wouldn’t be subsidizing performative checkboxing.
    - wadadadad 2y ago
      I'm curious, can you tell us what the negative impacts you've seen are? Are there any audits that you can say are a positive indicator for security?
      - agartner 2y ago
        Spending all their security time/budget on box checking rather than actual security. I'd rather see open ended red team pentest reports.
        ensignavenger 2y ago
        This is my complaint with "cyber insurance". Companies spending money on insurance premiums and checklists for the insurance company rather than spending money on security.
        phil21 2y ago
        Yep. My experience as well. Once a place starts doing useless box checking stuff like SOC2 it’s time to find a new job or switch vendors. Positive indicators would be talking to employees and getting an idea of organizational clue level. There are no shortcuts here I’ve ever found beyond doing this sort of old fashioned “know your vendor” style work.
- jve 2y ago
  As a technical person working at datacenter and at still handle technical support requests at some capacity, interesting to read this stuff from the customer perspective. Good to know what is considered and important by customers. Maybe sales staff knows all that too well, but for me just invokes a smile and some pride in the job I do :)
  - krab 2y ago
    The absolute trust killer is when your customer finds out about some issue on their own - even though you were supposed to monitor that. :-) More important than raw uptime.
vidarh 2y ago
This. I used to colo lots of stuff, but now mostly use Hetzner. But there are many in this space, and some of them even offer an API. And some of them (like Hetzner) also offer at least basic cloud services, so you can mix and match (which allows for even steeper cost cuts - instead of loading your dedicated hardware to 60% or whatever you're comfortable with to have headroom, you can load it higher and scale into their cloud offering to handle spikes). The boundary where colo and dedicated server offerings intersect in price tend to be down to land and power costs - Hetzner finally became the cheaper option for me as London land values skyrocketed relative to their locations in Germany, and colo prices with them. (We could have looked at coko somewhere remote, but the savings would've been too low to be worth it)
bcrl 2y ago
After years of dealing with colocation, I would never deploy a server without KVM in a data center. The cost of truck rolls is just too damned high! Even with KVM, I have had to make an emergency trip to a data center that is a 3.5 hour drive away due to a hardware errata on circa 2010 Intel Xeon boards where the network port shared between the KVM and the host would lock up under certain rare circumstances. The second time that happened I pulled the system from production. If you do happen to have a system without on-board KVM, check out NanoKVM, which is a cheap (~$40) option for an add on KVM. It's rather more affordable than PiKVM. https://github.com/sipeed/NanoKVM https://github.com/sipeed/NanoKVM

by krab 2y ago

rsanheim 2y ago

One hurdle that many companies who have only known cloud hosting will face here is significant: how do you find a reliable, trustworthy datacenter? One who actually monitors the hardware and also has a real human if your networking access gets screwed or if you need a critical component swapped at 2 am on a Saturday. I used to have a short list of trustworthy companies like this I'd recommend to clients ~20 years ago when doing consulting. I think 3/4 of them have been gobbled up by private equity chop shops or are just gone. Nowadays noone gets fired for going with AWS, or resold AWS with a 100% markup from a 'private enterprise cloud' provider.

krab 2y ago
You're right you need to find a company you can trust. And for a lot of startups it really makes sense to use AWS. But if you do something resource or bandwidth intensive (and I'm not even talking about Llama now), the costs add up quickly. In our case, switching to AWS would increase our costs by an equivalent of 4 - 8 devs salaries. After AWS discounts. That's a hard sell in a 15-person team even though half of our infra costs already are with AWS (S3).
- vidarh 2y ago
  I often recommend a "embrace, extend, extinguish" approach to AWS: Starting there for simplicity is fine, then "wrap" anything bandwidth intensive with caches elsewhere (every 1TB in egress from AWS will pay for a fleet of Hetzner instances with 5TB included, or one or more dedicated servers). Gradually shift workloads, leaving anything requiring super-high durability last (optionally keeping S3, or competitors, as a backup storage option) as getting durability right is one of the more difficult things to get confidence in and most dangerous ones to get wrong. Wrapping S3 with a write-through cache setup can often be the biggest cost win if your egress costs are high. Sometimes caching the entire dataset is worth it, sometimes just a small portion.
  - krab 2y ago
    Well, S3 is hard to beat for our use case. We make a heavy use of their various tires, we store a somewhat large amount of data but only a minor part ever goes out. The compute and network heavy stuff we do is still out of AWS.
    - vidarh 2y ago
      That's pretty much the one situation where they're competitive, so sounds very reasonable. Some of their competitors (Cloudflare, Backblaze) might compete, but the biggest cost issue with S3 by far is the egress so if not much goes out it might still be best for you to stay there. Sounds like (unlike most people who use AWS) you've done your homework. It's great to see. I've used AWS a lot, and will again, because it's often convenient, but so often I see people doing it uncritically without modeling their costs even as it skyrockets with scale.
    - throwaway384638 2y ago
      S3 is a decent product with zero competition. You should keep s3, it’s a fair price.
      - vidarh 2y ago
        S3 has plenty of competition. It can be a fair price if you rarely read and need it's extreme durability, but that leaves plenty of uses it's totally wrong for.
chpatrick 2y ago
I think if you want to host in Europe then Hetzner is the clear choice. They won't monitor your hardware for you though, you need to let them know if something breaks and they'll replace it very quickly.
jareklupinski 2y ago
> how do you find a reliable, trustworthy datacenter? drive to a few, and shake some hands. in my exp, the difference between colos is usually "actual SOC2/ISO compliance" on one side, and "there are no locked doors between the parking lot and my rack" on the other, with not much in-between that's not for some specialty (radio), and these things can only really be seen for yourself
- hedora 2y ago
  That’s unfortunate. I consider SoC 2 compliance as a negative indicator of security (I’ve been on the vendor side of it, and have seen it have significant negative impacts on security and reliability in multiple organizations). Ideally, there’d be locked doors, and the data center wouldn’t be subsidizing performative checkboxing.
  - wadadadad 2y ago
    I'm curious, can you tell us what the negative impacts you've seen are? Are there any audits that you can say are a positive indicator for security?
    - agartner 2y ago
      Spending all their security time/budget on box checking rather than actual security. I'd rather see open ended red team pentest reports.
      - ensignavenger 2y ago
        This is my complaint with "cyber insurance". Companies spending money on insurance premiums and checklists for the insurance company rather than spending money on security.
      - phil21 2y ago
        Yep. My experience as well. Once a place starts doing useless box checking stuff like SOC2 it’s time to find a new job or switch vendors. Positive indicators would be talking to employees and getting an idea of organizational clue level. There are no shortcuts here I’ve ever found beyond doing this sort of old fashioned “know your vendor” style work.
jve 2y ago
As a technical person working at datacenter and at still handle technical support requests at some capacity, interesting to read this stuff from the customer perspective. Good to know what is considered and important by customers. Maybe sales staff knows all that too well, but for me just invokes a smile and some pride in the job I do :)
- krab 2y ago
  The absolute trust killer is when your customer finds out about some issue on their own - even though you were supposed to monitor that. :-) More important than raw uptime.

vidarh 2y ago

This. I used to colo lots of stuff, but now mostly use Hetzner. But there are many in this space, and some of them even offer an API. And some of them (like Hetzner) also offer at least basic cloud services, so you can mix and match (which allows for even steeper cost cuts - instead of loading your dedicated hardware to 60% or whatever you're comfortable with to have headroom, you can load it higher and scale into their cloud offering to handle spikes). The boundary where colo and dedicated server offerings intersect in price tend to be down to land and power costs - Hetzner finally became the cheaper option for me as London land values skyrocketed relative to their locations in Germany, and colo prices with them. (We could have looked at coko somewhere remote, but the savings would've been too low to be worth it)

bcrl 2y ago

After years of dealing with colocation, I would never deploy a server without KVM in a data center. The cost of truck rolls is just too damned high! Even with KVM, I have had to make an emergency trip to a data center that is a 3.5 hour drive away due to a hardware errata on circa 2010 Intel Xeon boards where the network port shared between the KVM and the host would lock up under certain rare circumstances. The second time that happened I pulled the system from production. If you do happen to have a system without on-board KVM, check out NanoKVM, which is a cheap (~$40) option for an add on KVM. It's rather more affordable than PiKVM. https://github.com/sipeed/NanoKVM https://github.com/sipeed/NanoKVM