6 ms·

Last physical datacenter my team at the time ran our app in we spent weeks troubleshooting some hardware network driver issues that caused the network to drop.

Last physical datacenter my team at the time ran our app in we spent weeks troubleshooting some hardware network driver issues that caused the network to drop. Was an enormous distraction and Dell and VMware support were useless at resolving it for us. I’m glad to have others deal with those low level “oh it must just be your setup” issues.

closeparen 7y ago
At the scale where owning infrastructure makes sense, that's not a distraction but some team's full-time mission.
- argd678 7y ago
  We did actually have a team who handles hardware and a networking department, and they couldn’t figure it out. So lots of people had to get involved to do a binary search for the problem. I should mention this is a company with considerable resources.
- falcolas 7y ago
  Even before that tipping point, you're likely to have a team dedicated to working with infrastructure hosted by a cloud provider. Based off past (hardware) and current (AWS) experience, the team sizes aren't that different.
  - deleted 7y ago
    [deleted]
qaq 7y ago
Until you hit this can not happen we will not spend time looking into it issue in a cloud :)
lima 7y ago
Anecdotal evidence. I've also seen colleagues spend weeks arguing with AWS support while debugging a weird performance degradation issue, that would have been straight-forward to investigate in a bare metal deployment with full control over everything. It's not like the cloud is a magical place where no unexpected issues ever happen. Cloud providers can be surprisingly buggy, especially AWS, particularly at scale when you start hitting the implicit scaling limits their docs conveniently forgot to mention. Each technology has their unique challenges.
- argd678 7y ago
  I’ve had performance issues with GCP, however I can switch cloud providers in that case without too much effort. With physical hardware that’s much harder.
  - lima 7y ago
    If you have a well-built and automated bare-metal deployment, it's no harder than switching cloud providers. At the scale where it makes sense to go bare metal, either would be a lot of effort. An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds for many companies.
    - windexh8er 7y ago
      > An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds for many companies. OpenShift is built on a prescriptive mentality. They choose everything for you from OS through pipeline and many deviations are unsupported. As you mentioned there are others, but depending on your environment OpenShift is either a very good fit or square-peg-round-hole.
      - lima 7y ago
        OpenShift is a superset of k8s, so you're free to customize it (you just won't get support if it breaks).
      - humbleMouse 7y ago
        Openshift doesn't choose the pipeline. You can deploy to it however you want. Openshift is literally the same exact thing as kubernetes, with a simpler api and a nice ui. Openshift is extremely flexible and you can use it however you want. Don't know why you think it's so restrictive.
nemonemo 7y ago
In my last team, our team suffered from a CPU bug and a SSD bug that caused painful debugging nights. We used multithreading heavily, and it could have been library/kernel issues or some bugs in our code, which cover massive amount of potential target code that needed to sort through initially. I don't think small companies can afford such gigantic debugging effort, so either they should rely on their luck (or flaky tests) or on someone else who can afford it (a cloud/infra service provider or a startup that depends on you.)
- lima 7y ago
  How is that related to bare metal vs. clouds? That could have happened on either.
  - nemonemo 7y ago
    Big cloud providers or some big VC backed companies have the resource to figure out the issues by wrestling with the gigantic and unyielding vendors, or by working around it with their engineering resource. Not many teams have the time for even wanting to understand the issues. This is not an inherent problem of bare metal, but one aspect to consider if you were to choose the path. But you are right, cloud providers have their own quirks.
    - nocman 7y ago
      > Big cloud providers or some big VC backed companies have the resource to figure out the issues by wrestling with the gigantic and unyielding vendors, or by working around it with their engineering resource. This is all operating on the assumption that those entities will care enough about your problem enough to do something meaningful to fix it. As others have stated here, at least in the context of "Big cloud providers", that frequently is not the case. Often you just get the runaround (continuous delays, requests for "more information", other attempts to stall in first level support). Add a third party "managing" some piece of your company's infrastructure via a cloud provider into the mix and it often gets even worse. It is true that there are real costs to having your own hardware/software onsite and people who know how to manage it. However, the promised reductions in cost/hassle of moving things offsite are frequently offset or even exceeded by the costs/hassles you get by not having control of things yourself.
dilyevsky 7y ago
You wait until you have to debug something like that with a VPC. We had to spend couple days to convince our cloud support the issue was on their end during full site outage.
ori_b 7y ago
Counterpoint: These issues still exist, you just lose visibility into them by putting someone else in between. And usually, the someone else is big enough that you're not a priority.
- nemonemo 7y ago
  with limited resource, you have to choose to balance between the stability or more control over your infra. IMO, small companies can't afford both, but also likely they would not need both at that size.
  - rhizome 7y ago
    B.S. I was solo IT for a 50+ppl company, on a three-person team with a CIO/DBA and a data manager. I supported a full Windows/Exchange LAN with heavy printing needs, tape backups, internal/external infrastructural services and webservers with LOB java webapps, remote users, etc. Through all this I learned Perl from scratch, rearchitected the business systems (hardware) and reduced their production time by like 90% (two weeks to two days) with zero downtime (primordial blue-green, as it turns out), and wrote a completely new ETL flow with my new Perl skills. From four servers under a table to four racks, SaaS architecture is exceedingly simple next to all this, and the company spent probably around $10K/mo all told, maybe up to $20k while I was buying hardware. Maybe you simply haven't seen this kind of operation in action before?

closeparen 7y ago

At the scale where owning infrastructure makes sense, that's not a distraction but some team's full-time mission.

argd678 7y ago
We did actually have a team who handles hardware and a networking department, and they couldn’t figure it out. So lots of people had to get involved to do a binary search for the problem. I should mention this is a company with considerable resources.
falcolas 7y ago
Even before that tipping point, you're likely to have a team dedicated to working with infrastructure hosted by a cloud provider. Based off past (hardware) and current (AWS) experience, the team sizes aren't that different.
- deleted 7y ago
  [deleted]

qaq 7y ago

Until you hit this can not happen we will not spend time looking into it issue in a cloud :)

lima 7y ago

Anecdotal evidence. I've also seen colleagues spend weeks arguing with AWS support while debugging a weird performance degradation issue, that would have been straight-forward to investigate in a bare metal deployment with full control over everything. It's not like the cloud is a magical place where no unexpected issues ever happen. Cloud providers can be surprisingly buggy, especially AWS, particularly at scale when you start hitting the implicit scaling limits their docs conveniently forgot to mention. Each technology has their unique challenges.

argd678 7y ago
I’ve had performance issues with GCP, however I can switch cloud providers in that case without too much effort. With physical hardware that’s much harder.
- lima 7y ago
  If you have a well-built and automated bare-metal deployment, it's no harder than switching cloud providers. At the scale where it makes sense to go bare metal, either would be a lot of effort. An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds for many companies.
  - windexh8er 7y ago
    > An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds for many companies. OpenShift is built on a prescriptive mentality. They choose everything for you from OS through pipeline and many deviations are unsupported. As you mentioned there are others, but depending on your environment OpenShift is either a very good fit or square-peg-round-hole.
    - lima 7y ago
      OpenShift is a superset of k8s, so you're free to customize it (you just won't get support if it breaks).
    - humbleMouse 7y ago
      Openshift doesn't choose the pipeline. You can deploy to it however you want. Openshift is literally the same exact thing as kubernetes, with a simpler api and a nice ui. Openshift is extremely flexible and you can use it however you want. Don't know why you think it's so restrictive.

nemonemo 7y ago

In my last team, our team suffered from a CPU bug and a SSD bug that caused painful debugging nights. We used multithreading heavily, and it could have been library/kernel issues or some bugs in our code, which cover massive amount of potential target code that needed to sort through initially. I don't think small companies can afford such gigantic debugging effort, so either they should rely on their luck (or flaky tests) or on someone else who can afford it (a cloud/infra service provider or a startup that depends on you.)

lima 7y ago
How is that related to bare metal vs. clouds? That could have happened on either.
- nemonemo 7y ago
  Big cloud providers or some big VC backed companies have the resource to figure out the issues by wrestling with the gigantic and unyielding vendors, or by working around it with their engineering resource. Not many teams have the time for even wanting to understand the issues. This is not an inherent problem of bare metal, but one aspect to consider if you were to choose the path. But you are right, cloud providers have their own quirks.
  - nocman 7y ago
    > Big cloud providers or some big VC backed companies have the resource to figure out the issues by wrestling with the gigantic and unyielding vendors, or by working around it with their engineering resource. This is all operating on the assumption that those entities will care enough about your problem enough to do something meaningful to fix it. As others have stated here, at least in the context of "Big cloud providers", that frequently is not the case. Often you just get the runaround (continuous delays, requests for "more information", other attempts to stall in first level support). Add a third party "managing" some piece of your company's infrastructure via a cloud provider into the mix and it often gets even worse. It is true that there are real costs to having your own hardware/software onsite and people who know how to manage it. However, the promised reductions in cost/hassle of moving things offsite are frequently offset or even exceeded by the costs/hassles you get by not having control of things yourself.

dilyevsky 7y ago

You wait until you have to debug something like that with a VPC. We had to spend couple days to convince our cloud support the issue was on their end during full site outage.

ori_b 7y ago

Counterpoint: These issues still exist, you just lose visibility into them by putting someone else in between. And usually, the someone else is big enough that you're not a priority.

nemonemo 7y ago
with limited resource, you have to choose to balance between the stability or more control over your infra. IMO, small companies can't afford both, but also likely they would not need both at that size.
- rhizome 7y ago
  B.S. I was solo IT for a 50+ppl company, on a three-person team with a CIO/DBA and a data manager. I supported a full Windows/Exchange LAN with heavy printing needs, tape backups, internal/external infrastructural services and webservers with LOB java webapps, remote users, etc. Through all this I learned Perl from scratch, rearchitected the business systems (hardware) and reduced their production time by like 90% (two weeks to two days) with zero downtime (primordial blue-green, as it turns out), and wrote a completely new ETL flow with my new Perl skills. From four servers under a table to four racks, SaaS architecture is exceedingly simple next to all this, and the company spent probably around $10K/mo all told, maybe up to $20k while I was buying hardware. Maybe you simply haven't seen this kind of operation in action before?