4 ms·

>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different

>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list of self-hosting companies shrunk by a huge degree. Doesn't that seem interesting?

lixtra 7y ago
Ok, I will try to counter that. The stack grows higher. More specialized employees become more effective. In 200X you just needed a Linux admin. Now on top of being good a good Linux admin you also need to know docker, kubernetes (the admin part, not the user part) etc. If you don’t need that daily, because you also need to maintain some apps, you will be less professional than a cloud provider. Of course it’s a trade-off. If you’re large, with constant high resource needs and benefit from custom optimization, you may be better off with your own team/infrastructure. If you’re small (2 it guys), fast changing, normal speed requirements / easy to scale out then the cloud should be interesting to you.
- apple4ever 7y ago
  Really good points. Maintaining your own infrastructure may be "boring" with "old" technologies. But its a known process. Its really easy. But with the cloud its a whole different domain. It may be easy too, but I don't think its as easy as hosting your own. And your last two lines are great. It really comes down to the company about which tradeoff is appropriate.
jasode 7y ago
>In 200X lots of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. But my point is that companies' IT departments did not maintain infrastructure just like Amazon did. In ~2005 when companies were first experimenting with AWS cloud, they might start with dev & test servers. They click a few buttons and are amazed when new servers get spun up in minutes and their programmers are productive immediately. The natural question that company execs ask is, "why can't our own internal IT department spin up servers for us in 10 minutes?!? Why does it take them so damn long?!?" They wouldn't have asked those hard questions if their internal IT capability was equivalent to AWS. Eventually, their improved experiences with AWS on Dev&Test&QA convinced them to migrate mission-critical Production workloads to AWS as well. >We have cheaper hardware. We have (supposedly) better software. You're still focusing on on hardware+software and not considering the IT employees' speed of execution in how company executives compare the situation. Even Netflix as a tech company maintained their own datacenters for over 10 years but ended up migrating to AWS. Their "Guardian" moment was a big database corruption in 2008. They also had ambitious plans to expand into countries outside of USA. Those were some of the factors that convinced them they didn't want to invest anymore in their own datacenters and preferred AWS take care of it. Amazon employees iterated on datacenter capabilities faster than Netflix engineers could do it.
- nopzor 7y ago
  netflix does not depend on aws to deliver their video bits. they do it themselves based on a big network of core and edge pni and caches. this infra, that netflix has built, in datacenters, is a big competitive advantage. it would be worse performance and much higher cost to do this over something like aws cloudfront.
  - jasode 7y ago
    >this infra, that netflix has built, in datacenters, Unless things have changed since 2015, reports say Netflix eliminated their last datacenter already.[1] The Netflix "edge appliances" for CDN streaming are located in others' datacenters owned by Verizon,Comcast,ISPs,etc. [1] https://arstechnica.com/information-technology/2015/08/netflix-shuts-down-its-last-data-center-but-still-runs-a-big-it-operation/ https://arstechnica.com/information-technology/2015/08/netfl...
    - pathseeker 7y ago
      That's still in datacenters that aren't Amazon's, Google's, etc. It's the most critical component of netflix (the actual video delivery) and it's not "in the cloud".
      - jasode 7y ago
        >That's still in datacenters The point isn't that they are still _in_ datacenters. Yes, of course, they are. Even the "cloud" ultimately resolves down to somebody's datacenter somewhere. The point is that Reed Hastings & Netflix wanted to get out of managing their own datacenters. Putting their Netflix appliances inside of ISP owned datacenters still lets them avoid managing their own datacenters. Their critical user accounts signup, monthly billing, and analytics, etc workloads are at AWS. And as the article mentions, even updating the cache on the Netflix appliances is coordinated through AWS. The combination of those strategies keeps them out of the "datacenter business" and let's them stay focused on their core competency of "video content".
dwild 7y ago
His whole comment was about how it's not about hardware cost, but staff cost. > It's in-house IT employees' speed of tech innovation vs Amazon's engineers' The cloud shrunk the cost of IT team. Sure hardware is cheaper, but that cost is marginal versus employee cost.
acdha 7y ago
> In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. I would question that equivalence. Speaking as someone who was trying to get what we now call devops going at the time, there were a _few_ companies starting big wins from automation and a ton of places which were content pouring huge amounts of human time into doing it the hard way and slow metrics for time to patch, provision new services, recover after failures, etc. When they had automation, it was known-bad practices like cloning VMs and had accordingly greater cost and lower benefits. The companies in the latter groups were the ones who were faced with either spending a huge amount of time and money catching up or cutting a cloud contract and not having do a large percentage of that remediation work at all, while being able to deliver results immediately.
hn_throwaway_99 7y ago
I worked at a couple of those self-hosted companies in the 200Xs. For me, as a developer, it was painful and way, way, way worse than AWS or GCP. Planning for big expected spikes of traffic was extremely painful, and once the cloud vendors started adding new services we felt continually behind the times. I can only think it was "interesting" if you've forgotten how difficult and expensive managing your own infrastructure can be.
- pas 7y ago
  It's a false dichotomy (or at least a false equivalence). So, there were a lot of, and still there a lot of corporations running their own stuff. Some of these are still running their own stuff well. Some got big, some stayed small in the past decade (some downsized). It's not necessarily hard to run your own stuff either. Of course, of course, managed platforms are easy (or easier) from a lot of aspects. (That's their value offering after all.) And the IT expertise market was never exceptionally great in a lot of places. Meaning it was hard to find good sysadmins, good ops people, good devops folks, good programmers, leads, tech PMs, POs, etc. who were able to work together and effectively run their shit, build their product/service, run with the flow (of open source or whatever vendor they used), and so on. AWS/GCP/Azure is a big paradigm change. No longer do you need to argue about what and how, at best you pick one of the three, and that's it. No longer you need to think about hardware, colo, uplink, switches, peering, multi-homing, picking the right dedicated server provider, and so on. (Sure, you need network guys, who at least have some basic understanding of VPC - overlay networking, but the basics are easy, and the rest is YAGNI anyway.) And this is due to their size. They are big enough, that their price premium - compared even to whatever random small hosting provider you might find on the 'net - is irrelevant, because they are usually more efficient, more secure, more geo-available than the small ones. But, ... of course, while generalizing all self-hosted companies (meaning that all self-hosted is a pain) leads to a falsehood, on the margin, especially during the IT consolidation of the past years, most companies moved away from self-hosting. And developers usually rejoiced. And similarly, generalizing the problem of self-hosting from your experiences again leads to a flawed conclusion (that it's necessarily hard and expensive to manage your infra), on average, outsourcing a cost-center in an ever increasing complexity world (that is potentially unbounded costs) and focusing on profit-centers is a sane decision.
marcinzm 7y ago
>In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. I worked at a decently large tech-ish company at that time. It took 3 months to provision a handful of dev servers in the data center. After filling out forms in triplicate. We asked for extras since we had almost zero ability to manage them (reboot, etc.) or get them re-built if something broke. No backups of any kind as far as I knew. We once had one break at night and apparently someone had to psychically drive there at 3am to reboot the thing after we filed a ticket. Used some sort of in-house distribution of linux that had it's own quirks. I'll take the cloud any day of the week.
- apple4ever 7y ago
  That sounds like a problem with process, not technology. I can provision a handful of production servers in 5 minutes.

lixtra 7y ago

Ok, I will try to counter that. The stack grows higher. More specialized employees become more effective. In 200X you just needed a Linux admin. Now on top of being good a good Linux admin you also need to know docker, kubernetes (the admin part, not the user part) etc. If you don’t need that daily, because you also need to maintain some apps, you will be less professional than a cloud provider. Of course it’s a trade-off. If you’re large, with constant high resource needs and benefit from custom optimization, you may be better off with your own team/infrastructure. If you’re small (2 it guys), fast changing, normal speed requirements / easy to scale out then the cloud should be interesting to you.

apple4ever 7y ago
Really good points. Maintaining your own infrastructure may be "boring" with "old" technologies. But its a known process. Its really easy. But with the cloud its a whole different domain. It may be easy too, but I don't think its as easy as hosting your own. And your last two lines are great. It really comes down to the company about which tradeoff is appropriate.

jasode 7y ago

>In 200X lots of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. But my point is that companies' IT departments did not maintain infrastructure just like Amazon did. In ~2005 when companies were first experimenting with AWS cloud, they might start with dev & test servers. They click a few buttons and are amazed when new servers get spun up in minutes and their programmers are productive immediately. The natural question that company execs ask is, "why can't our own internal IT department spin up servers for us in 10 minutes?!? Why does it take them so damn long?!?" They wouldn't have asked those hard questions if their internal IT capability was equivalent to AWS. Eventually, their improved experiences with AWS on Dev&Test&QA convinced them to migrate mission-critical Production workloads to AWS as well. >We have cheaper hardware. We have (supposedly) better software. You're still focusing on on hardware+software and not considering the IT employees' speed of execution in how company executives compare the situation. Even Netflix as a tech company maintained their own datacenters for over 10 years but ended up migrating to AWS. Their "Guardian" moment was a big database corruption in 2008. They also had ambitious plans to expand into countries outside of USA. Those were some of the factors that convinced them they didn't want to invest anymore in their own datacenters and preferred AWS take care of it. Amazon employees iterated on datacenter capabilities faster than Netflix engineers could do it.

nopzor 7y ago
netflix does not depend on aws to deliver their video bits. they do it themselves based on a big network of core and edge pni and caches. this infra, that netflix has built, in datacenters, is a big competitive advantage. it would be worse performance and much higher cost to do this over something like aws cloudfront.
- jasode 7y ago
  >this infra, that netflix has built, in datacenters, Unless things have changed since 2015, reports say Netflix eliminated their last datacenter already.[1] The Netflix "edge appliances" for CDN streaming are located in others' datacenters owned by Verizon,Comcast,ISPs,etc. [1] https://arstechnica.com/information-technology/2015/08/netflix-shuts-down-its-last-data-center-but-still-runs-a-big-it-operation/ https://arstechnica.com/information-technology/2015/08/netfl...
  - pathseeker 7y ago
    That's still in datacenters that aren't Amazon's, Google's, etc. It's the most critical component of netflix (the actual video delivery) and it's not "in the cloud".
    - jasode 7y ago
      >That's still in datacenters The point isn't that they are still _in_ datacenters. Yes, of course, they are. Even the "cloud" ultimately resolves down to somebody's datacenter somewhere. The point is that Reed Hastings & Netflix wanted to get out of managing their own datacenters. Putting their Netflix appliances inside of ISP owned datacenters still lets them avoid managing their own datacenters. Their critical user accounts signup, monthly billing, and analytics, etc workloads are at AWS. And as the article mentions, even updating the cache on the Netflix appliances is coordinated through AWS. The combination of those strategies keeps them out of the "datacenter business" and let's them stay focused on their core competency of "video content".

dwild 7y ago

His whole comment was about how it's not about hardware cost, but staff cost. > It's in-house IT employees' speed of tech innovation vs Amazon's engineers' The cloud shrunk the cost of IT team. Sure hardware is cheaper, but that cost is marginal versus employee cost.

acdha 7y ago

> In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. I would question that equivalence. Speaking as someone who was trying to get what we now call devops going at the time, there were a _few_ companies starting big wins from automation and a ton of places which were content pouring huge amounts of human time into doing it the hard way and slow metrics for time to patch, provision new services, recover after failures, etc. When they had automation, it was known-bad practices like cloning VMs and had accordingly greater cost and lower benefits. The companies in the latter groups were the ones who were faced with either spending a huge amount of time and money catching up or cutting a cloud contract and not having do a large percentage of that remediation work at all, while being able to deliver results immediately.

hn_throwaway_99 7y ago

I worked at a couple of those self-hosted companies in the 200Xs. For me, as a developer, it was painful and way, way, way worse than AWS or GCP. Planning for big expected spikes of traffic was extremely painful, and once the cloud vendors started adding new services we felt continually behind the times. I can only think it was "interesting" if you've forgotten how difficult and expensive managing your own infrastructure can be.

pas 7y ago
It's a false dichotomy (or at least a false equivalence). So, there were a lot of, and still there a lot of corporations running their own stuff. Some of these are still running their own stuff well. Some got big, some stayed small in the past decade (some downsized). It's not necessarily hard to run your own stuff either. Of course, of course, managed platforms are easy (or easier) from a lot of aspects. (That's their value offering after all.) And the IT expertise market was never exceptionally great in a lot of places. Meaning it was hard to find good sysadmins, good ops people, good devops folks, good programmers, leads, tech PMs, POs, etc. who were able to work together and effectively run their shit, build their product/service, run with the flow (of open source or whatever vendor they used), and so on. AWS/GCP/Azure is a big paradigm change. No longer do you need to argue about what and how, at best you pick one of the three, and that's it. No longer you need to think about hardware, colo, uplink, switches, peering, multi-homing, picking the right dedicated server provider, and so on. (Sure, you need network guys, who at least have some basic understanding of VPC - overlay networking, but the basics are easy, and the rest is YAGNI anyway.) And this is due to their size. They are big enough, that their price premium - compared even to whatever random small hosting provider you might find on the 'net - is irrelevant, because they are usually more efficient, more secure, more geo-available than the small ones. But, ... of course, while generalizing all self-hosted companies (meaning that all self-hosted is a pain) leads to a falsehood, on the margin, especially during the IT consolidation of the past years, most companies moved away from self-hosting. And developers usually rejoiced. And similarly, generalizing the problem of self-hosting from your experiences again leads to a flawed conclusion (that it's necessarily hard and expensive to manage your infra), on average, outsourcing a cost-center in an ever increasing complexity world (that is potentially unbounded costs) and focusing on profit-centers is a sane decision.

marcinzm 7y ago

>In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. I worked at a decently large tech-ish company at that time. It took 3 months to provision a handful of dev servers in the data center. After filling out forms in triplicate. We asked for extras since we had almost zero ability to manage them (reboot, etc.) or get them re-built if something broke. No backups of any kind as far as I knew. We once had one break at night and apparently someone had to psychically drive there at 3am to reboot the thing after we filed a ticket. Used some sort of in-house distribution of linux that had it's own quirks. I'll take the cloud any day of the week.

apple4ever 7y ago
That sounds like a problem with process, not technology. I can provision a handful of production servers in 5 minutes.