6 ms·

I am really starting to feel that co-location will make a big comeback. It seems cloud costs are just becoming too high for the convenience they once offered. F

by onebot 5y ago

I am really starting to feel that co-location will make a big comeback. It seems cloud costs are just becoming too high for the convenience they once offered. For small projects and scale probably makes a ton of sense, but at some point the costs to scale aren't worth the up front developer cost savings.

fulmicoton 5y ago
It depends on the use case does not it. Shared nothing is the best architecture for e-commerce search for instance. But if you have one query every minutes or so for a 1TB dataset, it feels a bit silly to have a couple of servers dedicated to it doesn't it? Imagine this is the case for all big data search you can think of... Logs, emails, etc. This is a waste of CPU and RAM.
curryst 5y ago
Where they get you is that it very rarely makes financial sense to do both cloud and colo/on-prem (unless you're a massive company). It ends up being way more expensive to use the cloud, but also hire engineers to work on making an on-prem cloud. Most companies have a mixed bag of projects that are either better served by the cloud, or are okay with colo and the savings it can bring. Assuming you don't want to do a hybrid approach, then you either push everyone onto the cloud and accept paying more, or you push everyone into colo and force the small and scaling out projects to deal with stuff like having to order hardware 3 months in advance. Then, depending on how nice you want it to be to interact with your infrastructure, you can end up paying a lot to have people build abstractions over it. Do you want developers to be able to create their own database from a merge request or API call? If so, now you're going to have to hire someone with a 6 figure salary to figure out how to do that. It's easy to forget how many things are involved in that. You're going to have a lot of databases, so you need a system to track them. A lot of these databases are presumably not big enough to warrant a full physical server, so you have to sort out multi-tenancy. If you have multi-tenancy, you need a way to handle RBAC so one user can't bork all the databases on the host. You will also need some way to handle what happens when one user is throwing so much load at the RDBMS it's impacting other apps on that database. To accomplish that, you're going to need a way to gather metrics that are sharded per-database and a way to monitor those (which is admittedly one of the easier bits). You also generally just straight up lose a lot of the scaling features. I don't have a way to just give you more IOPS to your database on-prem. The best I can do is add more disks, but your database will be down for a long time if I have to put a disk in, expand the RAID, let it redistribute data and then power it back up. That's several hours of downtime for you, along with anyone who's on the same database. Of course, we can do replicas, and swap the master, but everyone will have to reconfigure their apps or we need something like Consul to handle that (which means more engineers to manage that stuff). You're also probably going to need more than one of those expensive infra people, because they presumably need an on-call rotation, and no one is going to agree to be on-call the time. And every time someone quits, you have to train the new person, which is several months of salary basically wasted. That's not to say that you don't need infra people on AWS, but you a) need a lot less of them, because they only need to manage the systems AWS has, not build them, and b) you can hire cheaper ops people, again because you don't need people that are capable of building those kinds of systems. Once you factor in all of that stuff, AWS' prices start looking more reasonable. They're still a little higher, but they're not double the price. If anything more than a tiny, tiny subset of the AWS features are appealing, it's going to cost you almost as much to build your own as it does to just pay Amazon/Google/Microsoft/whoever. Also, a massive thing people overlook is that AWS is fairly well documented. I can Google exactly how to set up permissions on an S3 bucket, or how to use an S3 bucket as a website. It only takes seconds, the cognitive burden is low, and the low-friction doesn't cause anyone stress. In-house systems tend to be poorly documented, and doing anything slightly outside the norm becomes a "set up a meeting with the infra team" kind of thing. It takes forever, but more importantly, it takes a lot of thought and it's frustrating.
- rossmohax 5y ago
  You save on specialized engineers (Database, RabbitMQ, Ceph administrators), but you lose elsewhere. What used to be an apache serving static files, now is S3 bucket, but it wont be easy, because you wanted your own domain, so now you need a Cloudfront because of SSL support. Their tutorial conveniently mentions it only at the step 7 ("Test your website endpoint"). You buy into Cognito, great, saved money on Keycloak administrator, but in the worst moment deep in the project you learn that there is absolutely no way to support multiple regions, even if you are willing to do some leg work for AWS. Or find that Cognito email reset flow can't go through your existing customer contact system and must go through SES only, suddenly you find developing elaborate log/event processing tool just that your customer service agent can see password reset event on their interface. GCP CloudSQL, managed RDBMS, great! No upgrade for you other than SQL dump/restore your 10TB instance, have fun. Cloud might be a net win still, but it is very much not as rosy as cloud evangelists want us think.
  - Fgffhjj 5y ago
    I realized how great Aws and co was when I took over the tech lead on a small startup. That well defined setup with load balancer, multiple VMS, snapshotting as backup, von/ipsec and vlan was something I couldn't have build and maintained by myself and doing other things 15 years ago without cloud. The worst is that a lot of "specialized" people are actually not experts at all. They are just there as support person's. From 3 DB admins one was really good and the other 2 sucked (other company again). The one you never saw comes to you, starts thing on an oracle shell basically blind and does complex things and the other two you tell them what they forgot. As an admin your job is seldomly to optimize queries etc. But more adding and removing users, configuring backups, playing back backups, upgrading etc. At least for me it opened up what I can do as a single expert and faster and in better quality. I would highly recommend normal companies who think they are not an it company (everyone is one today) to run in the cloud only.
- ElFitz 5y ago
  > Also, a massive thing people overlook is that AWS is fairly well documented. I can Google exactly how to set up permissions on an S3 bucket, or how to use an S3 bucket as a website. > In-house systems tend to be poorly documented, and doing anything slightly outside the norm becomes a "set up a meeting with the infra team" kind of thing. I usually wasn’t really happy with AWS’ documentation. But now, considering the alternative, it find it quite lovely. Thank you for making me realize that.
toast0 5y ago
Bare metal hosting is a happy medium between co-lo and cloud. You don't have much control over the network, so it might not be enough if you need faster NICs than they offer, but if you fit in their offerings, it can work well. Otoh, the bare metal hoster I worked with is now owned by IBM, and a big competitor is owned by private equity; bare metal from cloud providers still has a lot of cloudiness associated too. Maybe colo is the way to go.
- mwcampbell 5y ago
  How about OVH? They now have data centers in Canada and the US as well as Europe.
  - toast0 5y ago
    I hadn't looked at their offerings before, but they seem at least superficially good. A decent collection of datacenters, network with link aggregation is important to me. I'd be curious to see any reports about their fire response in their recent incident as well as any changes they made or didn't make to other sites.
Fgffhjj 5y ago
Always calculate what it would cost you as a company to actual my hire a good sys admin, keeping that person happy and spending the money for operating a average quality setup vs. fire and forget. A few grant infrastructure cost for any modern company is nothing and an outage costs you much more. You can also bet that proper backup is either missing or was paid for expensive enough. On Aws you activate snapshots and that's it.

by onebot 5y ago

fulmicoton 5y ago

It depends on the use case does not it. Shared nothing is the best architecture for e-commerce search for instance. But if you have one query every minutes or so for a 1TB dataset, it feels a bit silly to have a couple of servers dedicated to it doesn't it? Imagine this is the case for all big data search you can think of... Logs, emails, etc. This is a waste of CPU and RAM.

curryst 5y ago

Where they get you is that it very rarely makes financial sense to do both cloud and colo/on-prem (unless you're a massive company). It ends up being way more expensive to use the cloud, but also hire engineers to work on making an on-prem cloud. Most companies have a mixed bag of projects that are either better served by the cloud, or are okay with colo and the savings it can bring. Assuming you don't want to do a hybrid approach, then you either push everyone onto the cloud and accept paying more, or you push everyone into colo and force the small and scaling out projects to deal with stuff like having to order hardware 3 months in advance. Then, depending on how nice you want it to be to interact with your infrastructure, you can end up paying a lot to have people build abstractions over it. Do you want developers to be able to create their own database from a merge request or API call? If so, now you're going to have to hire someone with a 6 figure salary to figure out how to do that. It's easy to forget how many things are involved in that. You're going to have a lot of databases, so you need a system to track them. A lot of these databases are presumably not big enough to warrant a full physical server, so you have to sort out multi-tenancy. If you have multi-tenancy, you need a way to handle RBAC so one user can't bork all the databases on the host. You will also need some way to handle what happens when one user is throwing so much load at the RDBMS it's impacting other apps on that database. To accomplish that, you're going to need a way to gather metrics that are sharded per-database and a way to monitor those (which is admittedly one of the easier bits). You also generally just straight up lose a lot of the scaling features. I don't have a way to just give you more IOPS to your database on-prem. The best I can do is add more disks, but your database will be down for a long time if I have to put a disk in, expand the RAID, let it redistribute data and then power it back up. That's several hours of downtime for you, along with anyone who's on the same database. Of course, we can do replicas, and swap the master, but everyone will have to reconfigure their apps or we need something like Consul to handle that (which means more engineers to manage that stuff). You're also probably going to need more than one of those expensive infra people, because they presumably need an on-call rotation, and no one is going to agree to be on-call the time. And every time someone quits, you have to train the new person, which is several months of salary basically wasted. That's not to say that you don't need infra people on AWS, but you a) need a lot less of them, because they only need to manage the systems AWS has, not build them, and b) you can hire cheaper ops people, again because you don't need people that are capable of building those kinds of systems. Once you factor in all of that stuff, AWS' prices start looking more reasonable. They're still a little higher, but they're not double the price. If anything more than a tiny, tiny subset of the AWS features are appealing, it's going to cost you almost as much to build your own as it does to just pay Amazon/Google/Microsoft/whoever. Also, a massive thing people overlook is that AWS is fairly well documented. I can Google exactly how to set up permissions on an S3 bucket, or how to use an S3 bucket as a website. It only takes seconds, the cognitive burden is low, and the low-friction doesn't cause anyone stress. In-house systems tend to be poorly documented, and doing anything slightly outside the norm becomes a "set up a meeting with the infra team" kind of thing. It takes forever, but more importantly, it takes a lot of thought and it's frustrating.

rossmohax 5y ago
You save on specialized engineers (Database, RabbitMQ, Ceph administrators), but you lose elsewhere. What used to be an apache serving static files, now is S3 bucket, but it wont be easy, because you wanted your own domain, so now you need a Cloudfront because of SSL support. Their tutorial conveniently mentions it only at the step 7 ("Test your website endpoint"). You buy into Cognito, great, saved money on Keycloak administrator, but in the worst moment deep in the project you learn that there is absolutely no way to support multiple regions, even if you are willing to do some leg work for AWS. Or find that Cognito email reset flow can't go through your existing customer contact system and must go through SES only, suddenly you find developing elaborate log/event processing tool just that your customer service agent can see password reset event on their interface. GCP CloudSQL, managed RDBMS, great! No upgrade for you other than SQL dump/restore your 10TB instance, have fun. Cloud might be a net win still, but it is very much not as rosy as cloud evangelists want us think.
- Fgffhjj 5y ago
  I realized how great Aws and co was when I took over the tech lead on a small startup. That well defined setup with load balancer, multiple VMS, snapshotting as backup, von/ipsec and vlan was something I couldn't have build and maintained by myself and doing other things 15 years ago without cloud. The worst is that a lot of "specialized" people are actually not experts at all. They are just there as support person's. From 3 DB admins one was really good and the other 2 sucked (other company again). The one you never saw comes to you, starts thing on an oracle shell basically blind and does complex things and the other two you tell them what they forgot. As an admin your job is seldomly to optimize queries etc. But more adding and removing users, configuring backups, playing back backups, upgrading etc. At least for me it opened up what I can do as a single expert and faster and in better quality. I would highly recommend normal companies who think they are not an it company (everyone is one today) to run in the cloud only.
ElFitz 5y ago
> Also, a massive thing people overlook is that AWS is fairly well documented. I can Google exactly how to set up permissions on an S3 bucket, or how to use an S3 bucket as a website. > In-house systems tend to be poorly documented, and doing anything slightly outside the norm becomes a "set up a meeting with the infra team" kind of thing. I usually wasn’t really happy with AWS’ documentation. But now, considering the alternative, it find it quite lovely. Thank you for making me realize that.

toast0 5y ago

Bare metal hosting is a happy medium between co-lo and cloud. You don't have much control over the network, so it might not be enough if you need faster NICs than they offer, but if you fit in their offerings, it can work well. Otoh, the bare metal hoster I worked with is now owned by IBM, and a big competitor is owned by private equity; bare metal from cloud providers still has a lot of cloudiness associated too. Maybe colo is the way to go.

mwcampbell 5y ago
How about OVH? They now have data centers in Canada and the US as well as Europe.
- toast0 5y ago
  I hadn't looked at their offerings before, but they seem at least superficially good. A decent collection of datacenters, network with link aggregation is important to me. I'd be curious to see any reports about their fire response in their recent incident as well as any changes they made or didn't make to other sites.

Fgffhjj 5y ago

Always calculate what it would cost you as a company to actual my hire a good sys admin, keeping that person happy and spending the money for operating a average quality setup vs. fire and forget. A few grant infrastructure cost for any modern company is nothing and an outage costs you much more. You can also bet that proper backup is either missing or was paid for expensive enough. On Aws you activate snapshots and that's it.