18 ms·

Rolling your own servers with Kubernetes

409 points by old-gregg 7y ago

rjzzleep 7y ago
isn't this what they all sell nowadays? A prettier UI on top of kubernetes? - Kubermatic by loodse - rancheros - even mesosphere is kinda that now, except the first two are opensource, but definitely not the only solutions I personally deploy kubespray here and there, but I would still recommend smartos for normal humans, unless they want to setup an elk + kafka cluster or whatever else you might want to do.
- jskaggz 7y ago
  upvoted for triton+smartos - hear hear
deleted 7y ago
[deleted]
cagenut 7y ago
I used to specialize in 2 - 10 rack build outs, but its been nearly a decade since those gigs evaporated. I wonder if k8s creates a new niche for it.
- nineteen999 7y ago
  These kind of gigs are definitely nowhere near as common as they once were, but they haven't evaporated completely - I work in emergency services infrastructure in the process of a 10 rack build and that is something I don't see changing in the next 10 years, where we will do a refresh possibly once, or twice more. The complexity in our case is in the backend transmission network and unless we control everything (hardware, network) end-to-end as much as possible, we can't guarantee to getting close meet our SLA's for our customer. They wouldn't allow us to run our system in the cloud even if we wanted to.
huangbong 7y ago
So what is Gravity? kops/kubespray without an MIT license?
- alexk 7y ago
  Without MIT.... but with Apache2 ! :) https://github.com/gravitational/gravity https://github.com/gravitational/gravity In all seriousness though, gravity is an open core toolkit to package and deliver complicated sets of micro services in air gapped and restricted environments as a virtual installable appliance. It takes care of both application and Kubernetes lifecycle, software distribution and licensing workflows.
a012 7y ago
You can colo your server and roll your own k8s clusters. But you can't afford the luxury of high throughput networking, EBS, etc., and don't forget HA options (Multi-AZ) for your clusters.
- linuxdude314 7y ago
  Untrue. I purchased a 48 port 10GbE Arista 7148 the other month on eBay for $320.
  - ericd 7y ago
    Nice! OT, but have you seen any reasonably priced RJ45 10GBe out there? Preferably without screaming fans :-)
    - linuxdude314 7y ago
      I haven't. I've also not heard that great of things about it, but we run it in our DCs so I assume it works well enough.
      - ericd 7y ago
        Ah, yeah, looking for something for my home office (hence the preference for no screaming 1U fans), so I wouldn't have very long runs. But I have heard that 10 GbE RJ45 can be finicky.
    - olavgg 7y ago
      You really do not want RJ45 / 10GBase-T for higher network speeds. They consume A LOT more power and has significantly higher latency than SFP+/SFP28 and fiber cables. I have a Brocade ICX 6450 with 4 10G ports in my homelab, replaced the fan with a whisper silent one. Cannot hear and is right next to me. Brocade ICX 6610 has 16 10G ports and 2 40G ports, not super quiet, but not that noisy either.
      - ericd 7y ago
rmbryan 7y ago
The next-to-last line cracked me up: "The costs can be much lower… or much higher!"
- duxup 7y ago
  The cost of shiny new tech. "Hey look at all these cool things you could / can do that would lower costs and do other cool things." Weeks or months later you're still learning about how you do the thing that will maybe lower costs and do cool things ....
  - SteveNuts 7y ago
    Meanwhile the market and mindshare has moved on to the next shiny new tech.
    - geezerjay 7y ago
      Kubernetes has been here for a while. I'm sure we can agree that it ceased to be a candidate for a meteoric fad for a while now.
      - duxup 7y ago
        I am not a Kubernetes expert but I feel like Kubernetes still has such a high rate of change going on that ... it still is in that phase where you just don't know what you might get out of it... maybe not "just a fad" but still variable enough that who knows where we'll be or if in two years it's all different.
      - imtringued 7y ago
        I've probably wasted more than three weeks installing kubernetes on 3 nodes. I've tried Racher, RKE, kube-admin, kubespray and doing everything myself. I always failed and the way it failed was completely opaque and I didn't feel like I could even understand what was going wrong.
        7y ago
NickBusey 7y ago
I am curious to see how they recommend handling storage on bare metal k8s.
- lima 7y ago
  In many environments, you don't actually need distributed storage: - Application servers are stateless. - All state lives in stateful databases. - Those database handle high availability at the application layer by replication data, so you can use local storage and deploy them using StatefulSets. Has the extra benefit of avoiding the network storage latency penalty.
  - turk73 7y ago
    But StatefulSets require storage...someplace. We're dealing with that exact issue right now. Our K8s cluster is not playing nice with a NetApp NAS. We are exploring other options such as local iscsi storage volumes ($$$) or an external state store db (slow).
    - lima 7y ago
      The point is that with native HA, it doesn't have to be distributed storage. You can use locally attached SSDs or NVMe drives, which are straight forward to manage and fast.
      - acdha 7y ago
        Importantly, they also have independent failure modes. I’ve never known a SAN to be within an order of magnitude of the advertised uptime — and some interesting failures have been caused by HA features. Local storage avoids whole classes of problem and has much greater aggregate performance if you can make it work for your app.
- q3k 7y ago
  I highly recommend Rook [1], which is based on Ceph, provides PersistentVolumes to k8s workloads and is also running on k8s itself. [1] - https://rook.io/ https://rook.io/
argd678 7y ago
Last physical datacenter my team at the time ran our app in we spent weeks troubleshooting some hardware network driver issues that caused the network to drop. Was an enormous distraction and Dell and VMware support were useless at resolving it for us. I’m glad to have others deal with those low level “oh it must just be your setup” issues.
- closeparen 7y ago
  At the scale where owning infrastructure makes sense, that's not a distraction but some team's full-time mission.
  - argd678 7y ago
    We did actually have a team who handles hardware and a networking department, and they couldn’t figure it out. So lots of people had to get involved to do a binary search for the problem. I should mention this is a company with considerable resources.
  - falcolas 7y ago
    Even before that tipping point, you're likely to have a team dedicated to working with infrastructure hosted by a cloud provider. Based off past (hardware) and current (AWS) experience, the team sizes aren't that different.
    - deleted 7y ago
      [deleted]
- qaq 7y ago
  Until you hit this can not happen we will not spend time looking into it issue in a cloud :)
- lima 7y ago
  Anecdotal evidence. I've also seen colleagues spend weeks arguing with AWS support while debugging a weird performance degradation issue, that would have been straight-forward to investigate in a bare metal deployment with full control over everything. It's not like the cloud is a magical place where no unexpected issues ever happen. Cloud providers can be surprisingly buggy, especially AWS, particularly at scale when you start hitting the implicit scaling limits their docs conveniently forgot to mention. Each technology has their unique challenges.
xrisk 7y ago
The article seems to focus on K8s with reference to micro-services. How well does K8s do if you're running a monolith?
- mfatica 7y ago
  Using kubernetes without microservices is like using hadoop on a 1000 row csv file
  - xrisk 7y ago
    Ah, as someone not well versed with K8s, thanks for that analogy.
  - humbleMouse 7y ago
    I don't know if that's a fair comparison. If you deploy a monolith you have to set up a custom multi data center deploy and manage all the load balancing/disaster recovery/volume management(databases) yourself. Kubernetes isn't that hard to setup and gives you out of box solutions to all that. It also gives more flexibility on adding additional services around your monolith. Elk stack, kafka, that kind of stuff. Also gives you a standard api to deploy against. I don't think your metaphor holds up.
  - yowlingcat 7y ago
    This is a poor analogy. To quote a sibling comment: "If you consider what it takes to manage the end-to-end lifecycle of a single application, monolith or micro-service, you need a solution for the following items: deployments, application configuration, high availability, log and metrics aggregation, autoscaling, and load balancing across multiple application instances. Kubernetes provides an opinionated way of doing all of those things. For example, Kubernetes leverages container images and declarative configs for packaging and deploying applications. For many people this approach is much simpler than what Puppet, Chef, and Ansible bring to the table in terms managing applications." Using Kubernetes even with a monolith can be pretty nice, if (big if here) Kubernetes behaves.
- kelseyhightower 7y ago
t0m4 7y ago
addblock pleaz
webwanderings 7y ago
This is basically the "on-prem/hybrid cloud" business which everyone's after (IBM, Google, AWS, MS). The market seems to be pushing towards this model. The old businesses were naturally not moving to "cloud" so the cloud wants to come to their house (which makes sense).
acd 7y ago
There is also kubespray Ansible provisioning of Kubernetes, kops and kubeadm. Rolling Kubernetes on your own is quite hard though especially the networking part so there should be a market for a company helping out with provisioning. Using public cloud can be very expensive for some work loads which were not written for the cloud due to higher resource usage. It is still quite hard to forecast cost of the cloud due to a lot of moving parts and micro billing for each item. With traditional service providers it can be easier to budget for the service expenses.
- djsumdog 7y ago
  I did a post on installing DC/OS a few years back. I was going to follow up with a K8s post, but between the official Ansible scripts, the (at the time) very alpha kubeadm, broken pod logging and WeaveNet that would constantly fail with no logs, I ended up giving up on that post: https://penguindreams.org/blog/installing-mesosphere-dcos-on-small-digital-ocean-droplets/ https://penguindreams.org/blog/installing-mesosphere-dcos-on... I remember the manual-ish server provisioning days of Autoyast and RedHat's Kickstart. It's not that difficult to bootstrap physical servers to get them to the point of being nodes in a Docker scheduling system. Rancher is another great tool for setting up K8s (which you can run against physical nodes once you get them past the initial install/bootstrap). I worked at one place where they took a hybrid approach. They had DC/OS running in the local VMWare data center (which we were migrating to from AWS to save costs), but there were nodes that could scale to AWS as well and you could label your deployments for which data center you needed them to run in. A lot of high volume loads we were able to move back to our data center, but we also had a kick-ass platform team and our AWS bill was well over $150k/month. The trouble is it's almost impossible to do a real cost comparison. There are just too many factors, and when you start going AWS, GCE or Azure, you start using all their other managed services that may not have self-hosted replacements, not to mention off-setting the costs of setting up master/slaves, backups, snapshots, redundancy and other operations tasks that come by default or with an additional price from a hosted solution.
- ljm 7y ago
  From my limited experience, you'd save a lot on the cloud if you replaced the default IngressController. Google Cloud, for instance, will spin up a fresh load balancer for every service by default and those cost money. Of course, it's not an easy task as a beginner to use nginx or traefik in its place, and to handle the complexity of that deploy.
gambler 7y ago
>Should you roll your own servers? If you are not certain, the answer is most likely “no”. The staggering growth of AWS happened for a reason. Funny how for many decades companies and people were running their own servers. The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure.
- icebraining 7y ago
  It's kinda annoying how the valid criticism of cloud solutions is getting drowned out by criticism on strawmen. Who exactly is saying that no one can afford their own infra anymore? What some people say is that cloud is usually cheaper than running your own infra. They may be wrong (I'm doubtful myself), but it's a wholly different argument.
  - rhizome 7y ago
    It usually takes the form of "ugh then you have to hire dedicated people to manage it."
    - RussianCow 7y ago
      Which is a valid argument, isn't it? Engineers are expensive! Whether they're more expensive than AWS depends entirely on what you're doing, but the answer is almost always "yes" at a small enough scale.
      - rhizome 7y ago
        See my other comment.
        acdha 7y ago
        At least link it
segmondy 7y ago
I use to enjoy having a cheap desktop under my table or in a closet serving traffic to people across the internet. Computers have gotten faster, software has gotten better, network has gotten faster, things have gotten cheaper. Sadly instead of seeing more of these, most of these are now being outsourced to cloud providers, we have bought and drank the kool-aid that they can do it better and cheap. Which is not true, perhaps this is why it's so hard to find good unix and network admins these days.
- deleted 7y ago
  [deleted]
- johnwyles 7y ago
  Looking at Knative to mature in a couple of years and with it on top of k8s we may see a slow migration back to self-hosting where it makes sense
- chocolatebunny 7y ago
  I don't think cost is the benefit of cloud. For tech companies the benefit is scale (you won't know how popular a product is until you deploy). For non-tech companies the benefit is not having to deal with things that aren't related to your core business. I think you can argue that tech is always related to your core business and you only need a few servers to handle a million simultaneous users but those kinds of arguments aren't usually good enough to businessmen or entrepreneurs.
  - ericd 7y ago
    Yeah, you don't know how big you'll need to scale, but I think that's a bit of a boogeyman for a lot of companies. Servers have a relatively massive amount of cores and memory these days, along with screaming fast IO, and you can handle quite a lot of web traffic with only a few dual socket servers if you don't bog them down with incredibly inefficient code.
- lalos 7y ago
  How does it work with the consumer ISP? Can you elaborate on any special considerations? Obviously not for customers, but I'm interested for it as a hobby thing.
shereadsthenews 7y ago
If you get to build your own servers why torment yourself with a 2-socket solution? It's just harder to program with no benefits. A single-socket UMA solution will be more than large enough single nodes and a lot easier to program and manage.
peterwwillis 7y ago
Some thoughts: 1) Do you really need to invest a million dollars in bin-packing containerized stateless microservices? 2) You don't have to use K8s to get the benefits of rolling your own servers. In fact, I'd argue you should do the latter well before you do the former. 3) Definitely hire someone who has done it before. You will save so much time and money your head will spin. 4) Do not just build a rack full of random commodity gear. Make sure it is suited specifically for your purposes, and then weigh the cost of service contracts and managed colo against a $100k+ cage monkey on call 24/7. 5) Do not fall for the "We've got <insert tech hype>, we don't need redundant hardware!" lie. The more parts of your system rely on lots of hardware, the more fragile your system becomes. Distributed decentralized services become a PITA when the underlying gear is flaky, and centralized services require it. Do not underestimate the shittiness of your colo; always design for the most redundancy you can get for what you have. If you can run dual power, do it. Dual network stacks, do it. Redundant disks, do it. Remote management, do it. Outside modem to a management port on the router, do it. Always be postponing entropy. Later, when you become a FAANG (yeah right) you'll have the time and money to automate away some of the redundancy issues. 6) It's sometimes harder to upgrade disk or bandwidth on an existing machine than it is to just buy another machine, but the more machines you have, the more problems (and overhead costs), and there is an upward limit on the scaling of most services without rebuilding them. So buy big on local disk and network, and buy new machines to expand cpu and memory. The redundancy and performance issues inherent to SAN/NAS often make local disks a better choice, as they are unlikely to impact your whole network at once, you can fix/upgrade them piecemeal, and they are less difficult to manage/operate. 7) Don't DIY critical components such as data replication; it's surprisingly hard. In fact, don't even believe vendors if they claim they can solve X for you with some magic software or hardware. Get them to show you a live demo on your own network. 8) Don't forget that in 3 years you'll be replacing it all.
diehunde 7y ago
So before we had an IT team that would maintain the bare metal servers. Now we need "Cloud Engineers" to maintain the cloud infrastructure working properly. I don't know if the argument of externalize the maintenance of the servers is valid since the complexity of the cloud services is just increasing everyday.
- jononor 7y ago
  True for those that go for Kubernetes etc straight away. If you just have a few apps to run and put it on a PaaS like Heroku, things are pretty manageable as a side-thing for the regular software developers. No dedicated team, or even person, needed.
  - jrumbut 7y ago
    When your infrastructure is too simple, you have simple regrets. When your infrastructure is too exotic, you get exciting, bleeding edge regrets. This is my view based on the work I've done. I like the ability to dispense with capacity planning or dealing with power supplies and fire suppression you get from cloud hosting, but when in any doubt I set up what I need using VMs (be those droplets or EC2 instances or whatever). I like to imagine that I've avoided wasting weeks by wasting a few hours here and there.
MuffinFlavored 7y ago
> A “starter pack” 15amp cabinet with a gigabit connection can be rented for as little as $400 per month. Link?...
- ShakataGaNai 7y ago
  He.net
- snuxoll 7y ago
  https://he.net/colocation.html https://he.net/colocation.html Can't speak about the power given on their $400/mo cabinet deal.
- mlrtime 7y ago
  Yeah, this isn't possible for a entire rack, maybe 1U if it includes network and power if it is in any decent datacenter.
  - MuffinFlavored 7y ago
    How much would a 48U rack like described in the article cost a month? I would guess in the thousands if not ten thousands....
    - mlrtime 7y ago
      It mostly goes by power not rack units, if you have 48U and only 15amps, you're not going to fill it up. 15A is not a lot.
      - MuffinFlavored 7y ago
        So, $400/mo is overkill power wise. What would the bandwidth bill look like? How much of an up front investment is it to buy 48U worth of hardware? Seems expensive...
        dk- 7y ago
fcantournet 7y ago
The problem is not running kubernetes on "bare-metal". That's the somewhat easy and cheap part. The problem is the supply chain to spec, purchase, rack, validate, provision and keep up to date your servers, their drivers and firmware. It's your capacity planning. It's keeping up with the rate of hardware evolution. What happens when you have to go from 10G to 25G networking ? Oh and you also need a solid scalable L4 loadbalancing solution (facebook has a nice opensource dataplane based on ebpf) You can build / assemble from open-source bits most of the classic IaaS bits (compute/storage/networkLB). It's doable, the open-source solutions are pretty good. But unless you have 10k compute nodes how can you bear the cost of having the expert knowledge necessary to debug this ? Hardware sucks and is always on fire. Once you take this into account, the economic equation changes dramatically.
vuln 7y ago
https://github.com/aquasecurity/kube-hunter https://github.com/aquasecurity/kube-hunter https://labs.mwrinfosecurity.com/blog/attacking-kubernetes-through-kubelet/ https://labs.mwrinfosecurity.com/blog/attacking-kubernetes-t...
ranrotx 7y ago
AWS employee here--thoughts and opinions are my own. Prior to AWS, I was in IT Operations at a large financial services company. I saw the writing on the wall that over time, companies would not want to manage this part of their IT infrastructure themselves. Keep in mind, I was someone who was responsible for keeping the lights on for a decent number of Linux severs. For an individual company, there really isn't much value in having to maintain firmware levels on all your hardware, patch hypervisors (and try to coordinate all of the maintenance around a fixed pool of hardware), perform months-long evaluation of new hardware before purchasing, test and validate configurations on new hardware, etc. I used to do all of this. I don't miss it either. Yes, the items above are important, but doing them right is really table-stakes for any reliable IT Operations department. You can choose to spend time getting these right, or delegate that responsibility to a service provider whose main job is to get that stuff right (and recoups that cost across a much larger customer base).
- latch 7y ago
  What seems to often go unsaid in these discussions is that the choice isn't between cloud and colo. There's a third, hugely popular and mature option: dedicated providers - which address all of your issues. It's convenient for cloud vendors to have people believe the choice is between them or having to deal with hardware.
  - dhruvkar 7y ago
    what's an example of a dedicated provider? I also worked in ITOps at a medium-ish company and we were moving our colo to Azure, when I left.
    - arianvanp 7y ago
      Hetzner, scaleway and packet.net come to mind
    - acjacobson 7y ago
      Rackspace
haolez 7y ago
I’ve done this for small projects using AWS and Scaleway. The setup is simple and it just works, but the maintenance and security aspects were more worrying. One time I found out that one of my nodes failed to automatically apply security patches from my distro. I couldn’t detect it sooner because my monitoring was lacking. The overhead escalates fast. If I start an effort like this again today, I would make sure to get monitoring right from day 1. Possibly a combination of Prometheus with Grafana, or something along the lines of ElasticSearch APM.
momofarm 7y ago
I once heard a story that the reason some people would build their own server is they don't trust those service provider on the internet. They feared that the data store there are quite easily breached. Thanks to these guys, they just lower the unemployment rate a little more!
prisar 7y ago
link is broken
- numlock86 7y ago
  Yup, 404. Even manually going to their blog I can't find the entry anymore. Bookmarked because it was supposed to be an ongoing series with multiple parts. Weird provider. Probably best to avoid.
  - compuguy 7y ago
    I have a feeling they deleted the blog post. There is an archived copy here: https://web.archive.org/web/20190508123542/https://gravitational.com/blog/aws_vs_colocation/ https://web.archive.org/web/20190508123542/https://gravitati...

409 points by old-gregg 7y ago

isn't this what they all sell nowadays? A prettier UI on top of kubernetes? - Kubermatic by loodse - rancheros - even mesosphere is kinda that now, except the first two are opensource, but definitely not the only solutions I personally deploy kubespray here and there, but I would still recommend smartos for normal humans, unless they want to setup an elk + kafka cluster or whatever else you might want to do.

jskaggz 7y ago
upvoted for triton+smartos - hear hear

deleted 7y ago

[deleted]

cagenut 7y ago

I used to specialize in 2 - 10 rack build outs, but its been nearly a decade since those gigs evaporated. I wonder if k8s creates a new niche for it.

nineteen999 7y ago
These kind of gigs are definitely nowhere near as common as they once were, but they haven't evaporated completely - I work in emergency services infrastructure in the process of a 10 rack build and that is something I don't see changing in the next 10 years, where we will do a refresh possibly once, or twice more. The complexity in our case is in the backend transmission network and unless we control everything (hardware, network) end-to-end as much as possible, we can't guarantee to getting close meet our SLA's for our customer. They wouldn't allow us to run our system in the cloud even if we wanted to.

huangbong 7y ago

So what is Gravity? kops/kubespray without an MIT license?

alexk 7y ago
Without MIT.... but with Apache2 ! :) https://github.com/gravitational/gravity https://github.com/gravitational/gravity In all seriousness though, gravity is an open core toolkit to package and deliver complicated sets of micro services in air gapped and restricted environments as a virtual installable appliance. It takes care of both application and Kubernetes lifecycle, software distribution and licensing workflows.

a012 7y ago

You can colo your server and roll your own k8s clusters. But you can't afford the luxury of high throughput networking, EBS, etc., and don't forget HA options (Multi-AZ) for your clusters.

linuxdude314 7y ago
Untrue. I purchased a 48 port 10GbE Arista 7148 the other month on eBay for $320.
- ericd 7y ago
  Nice! OT, but have you seen any reasonably priced RJ45 10GBe out there? Preferably without screaming fans :-)
  - linuxdude314 7y ago
    I haven't. I've also not heard that great of things about it, but we run it in our DCs so I assume it works well enough.
    - ericd 7y ago
      Ah, yeah, looking for something for my home office (hence the preference for no screaming 1U fans), so I wouldn't have very long runs. But I have heard that 10 GbE RJ45 can be finicky.
  - olavgg 7y ago
    You really do not want RJ45 / 10GBase-T for higher network speeds. They consume A LOT more power and has significantly higher latency than SFP+/SFP28 and fiber cables. I have a Brocade ICX 6450 with 4 10G ports in my homelab, replaced the fan with a whisper silent one. Cannot hear and is right next to me. Brocade ICX 6610 has 16 10G ports and 2 40G ports, not super quiet, but not that noisy either.
    - ericd 7y ago

rmbryan 7y ago

The next-to-last line cracked me up: "The costs can be much lower… or much higher!"

duxup 7y ago
The cost of shiny new tech. "Hey look at all these cool things you could / can do that would lower costs and do other cool things." Weeks or months later you're still learning about how you do the thing that will maybe lower costs and do cool things ....
- SteveNuts 7y ago
  Meanwhile the market and mindshare has moved on to the next shiny new tech.
  - geezerjay 7y ago
    Kubernetes has been here for a while. I'm sure we can agree that it ceased to be a candidate for a meteoric fad for a while now.
    - duxup 7y ago
      I am not a Kubernetes expert but I feel like Kubernetes still has such a high rate of change going on that ... it still is in that phase where you just don't know what you might get out of it... maybe not "just a fad" but still variable enough that who knows where we'll be or if in two years it's all different.
    - imtringued 7y ago
      I've probably wasted more than three weeks installing kubernetes on 3 nodes. I've tried Racher, RKE, kube-admin, kubespray and doing everything myself. I always failed and the way it failed was completely opaque and I didn't feel like I could even understand what was going wrong.
      - 7y ago

NickBusey 7y ago

I am curious to see how they recommend handling storage on bare metal k8s.

lima 7y ago
In many environments, you don't actually need distributed storage: - Application servers are stateless. - All state lives in stateful databases. - Those database handle high availability at the application layer by replication data, so you can use local storage and deploy them using StatefulSets. Has the extra benefit of avoiding the network storage latency penalty.
- turk73 7y ago
  But StatefulSets require storage...someplace. We're dealing with that exact issue right now. Our K8s cluster is not playing nice with a NetApp NAS. We are exploring other options such as local iscsi storage volumes ($$$) or an external state store db (slow).
  - lima 7y ago
    The point is that with native HA, it doesn't have to be distributed storage. You can use locally attached SSDs or NVMe drives, which are straight forward to manage and fast.
    - acdha 7y ago
      Importantly, they also have independent failure modes. I’ve never known a SAN to be within an order of magnitude of the advertised uptime — and some interesting failures have been caused by HA features. Local storage avoids whole classes of problem and has much greater aggregate performance if you can make it work for your app.
q3k 7y ago
I highly recommend Rook [1], which is based on Ceph, provides PersistentVolumes to k8s workloads and is also running on k8s itself. [1] - https://rook.io/ https://rook.io/

argd678 7y ago

Last physical datacenter my team at the time ran our app in we spent weeks troubleshooting some hardware network driver issues that caused the network to drop. Was an enormous distraction and Dell and VMware support were useless at resolving it for us. I’m glad to have others deal with those low level “oh it must just be your setup” issues.

closeparen 7y ago
At the scale where owning infrastructure makes sense, that's not a distraction but some team's full-time mission.
- argd678 7y ago
  We did actually have a team who handles hardware and a networking department, and they couldn’t figure it out. So lots of people had to get involved to do a binary search for the problem. I should mention this is a company with considerable resources.
- falcolas 7y ago
  Even before that tipping point, you're likely to have a team dedicated to working with infrastructure hosted by a cloud provider. Based off past (hardware) and current (AWS) experience, the team sizes aren't that different.
  - deleted 7y ago
    [deleted]
qaq 7y ago
Until you hit this can not happen we will not spend time looking into it issue in a cloud :)
lima 7y ago
Anecdotal evidence. I've also seen colleagues spend weeks arguing with AWS support while debugging a weird performance degradation issue, that would have been straight-forward to investigate in a bare metal deployment with full control over everything. It's not like the cloud is a magical place where no unexpected issues ever happen. Cloud providers can be surprisingly buggy, especially AWS, particularly at scale when you start hitting the implicit scaling limits their docs conveniently forgot to mention. Each technology has their unique challenges.

xrisk 7y ago

The article seems to focus on K8s with reference to micro-services. How well does K8s do if you're running a monolith?

mfatica 7y ago
Using kubernetes without microservices is like using hadoop on a 1000 row csv file
- xrisk 7y ago
  Ah, as someone not well versed with K8s, thanks for that analogy.
- humbleMouse 7y ago
  I don't know if that's a fair comparison. If you deploy a monolith you have to set up a custom multi data center deploy and manage all the load balancing/disaster recovery/volume management(databases) yourself. Kubernetes isn't that hard to setup and gives you out of box solutions to all that. It also gives more flexibility on adding additional services around your monolith. Elk stack, kafka, that kind of stuff. Also gives you a standard api to deploy against. I don't think your metaphor holds up.
- yowlingcat 7y ago
  This is a poor analogy. To quote a sibling comment: "If you consider what it takes to manage the end-to-end lifecycle of a single application, monolith or micro-service, you need a solution for the following items: deployments, application configuration, high availability, log and metrics aggregation, autoscaling, and load balancing across multiple application instances. Kubernetes provides an opinionated way of doing all of those things. For example, Kubernetes leverages container images and declarative configs for packaging and deploying applications. For many people this approach is much simpler than what Puppet, Chef, and Ansible bring to the table in terms managing applications." Using Kubernetes even with a monolith can be pretty nice, if (big if here) Kubernetes behaves.
kelseyhightower 7y ago

t0m4 7y ago

addblock pleaz

webwanderings 7y ago

This is basically the "on-prem/hybrid cloud" business which everyone's after (IBM, Google, AWS, MS). The market seems to be pushing towards this model. The old businesses were naturally not moving to "cloud" so the cloud wants to come to their house (which makes sense).

acd 7y ago

There is also kubespray Ansible provisioning of Kubernetes, kops and kubeadm. Rolling Kubernetes on your own is quite hard though especially the networking part so there should be a market for a company helping out with provisioning. Using public cloud can be very expensive for some work loads which were not written for the cloud due to higher resource usage. It is still quite hard to forecast cost of the cloud due to a lot of moving parts and micro billing for each item. With traditional service providers it can be easier to budget for the service expenses.

djsumdog 7y ago
I did a post on installing DC/OS a few years back. I was going to follow up with a K8s post, but between the official Ansible scripts, the (at the time) very alpha kubeadm, broken pod logging and WeaveNet that would constantly fail with no logs, I ended up giving up on that post: https://penguindreams.org/blog/installing-mesosphere-dcos-on-small-digital-ocean-droplets/ https://penguindreams.org/blog/installing-mesosphere-dcos-on... I remember the manual-ish server provisioning days of Autoyast and RedHat's Kickstart. It's not that difficult to bootstrap physical servers to get them to the point of being nodes in a Docker scheduling system. Rancher is another great tool for setting up K8s (which you can run against physical nodes once you get them past the initial install/bootstrap). I worked at one place where they took a hybrid approach. They had DC/OS running in the local VMWare data center (which we were migrating to from AWS to save costs), but there were nodes that could scale to AWS as well and you could label your deployments for which data center you needed them to run in. A lot of high volume loads we were able to move back to our data center, but we also had a kick-ass platform team and our AWS bill was well over $150k/month. The trouble is it's almost impossible to do a real cost comparison. There are just too many factors, and when you start going AWS, GCE or Azure, you start using all their other managed services that may not have self-hosted replacements, not to mention off-setting the costs of setting up master/slaves, backups, snapshots, redundancy and other operations tasks that come by default or with an additional price from a hosted solution.
ljm 7y ago
From my limited experience, you'd save a lot on the cloud if you replaced the default IngressController. Google Cloud, for instance, will spin up a fresh load balancer for every service by default and those cost money. Of course, it's not an easy task as a beginner to use nginx or traefik in its place, and to handle the complexity of that deploy.

gambler 7y ago

>Should you roll your own servers? If you are not certain, the answer is most likely “no”. The staggering growth of AWS happened for a reason. Funny how for many decades companies and people were running their own servers. The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure.

icebraining 7y ago
It's kinda annoying how the valid criticism of cloud solutions is getting drowned out by criticism on strawmen. Who exactly is saying that no one can afford their own infra anymore? What some people say is that cloud is usually cheaper than running your own infra. They may be wrong (I'm doubtful myself), but it's a wholly different argument.
- rhizome 7y ago
  It usually takes the form of "ugh then you have to hire dedicated people to manage it."
  - RussianCow 7y ago
    Which is a valid argument, isn't it? Engineers are expensive! Whether they're more expensive than AWS depends entirely on what you're doing, but the answer is almost always "yes" at a small enough scale.
    - rhizome 7y ago
      See my other comment.
      - acdha 7y ago
        At least link it

segmondy 7y ago

I use to enjoy having a cheap desktop under my table or in a closet serving traffic to people across the internet. Computers have gotten faster, software has gotten better, network has gotten faster, things have gotten cheaper. Sadly instead of seeing more of these, most of these are now being outsourced to cloud providers, we have bought and drank the kool-aid that they can do it better and cheap. Which is not true, perhaps this is why it's so hard to find good unix and network admins these days.

deleted 7y ago
[deleted]
johnwyles 7y ago
Looking at Knative to mature in a couple of years and with it on top of k8s we may see a slow migration back to self-hosting where it makes sense
chocolatebunny 7y ago
I don't think cost is the benefit of cloud. For tech companies the benefit is scale (you won't know how popular a product is until you deploy). For non-tech companies the benefit is not having to deal with things that aren't related to your core business. I think you can argue that tech is always related to your core business and you only need a few servers to handle a million simultaneous users but those kinds of arguments aren't usually good enough to businessmen or entrepreneurs.
- ericd 7y ago
  Yeah, you don't know how big you'll need to scale, but I think that's a bit of a boogeyman for a lot of companies. Servers have a relatively massive amount of cores and memory these days, along with screaming fast IO, and you can handle quite a lot of web traffic with only a few dual socket servers if you don't bog them down with incredibly inefficient code.
lalos 7y ago
How does it work with the consumer ISP? Can you elaborate on any special considerations? Obviously not for customers, but I'm interested for it as a hobby thing.

shereadsthenews 7y ago

If you get to build your own servers why torment yourself with a 2-socket solution? It's just harder to program with no benefits. A single-socket UMA solution will be more than large enough single nodes and a lot easier to program and manage.

peterwwillis 7y ago

Some thoughts: 1) Do you really need to invest a million dollars in bin-packing containerized stateless microservices? 2) You don't have to use K8s to get the benefits of rolling your own servers. In fact, I'd argue you should do the latter well before you do the former. 3) Definitely hire someone who has done it before. You will save so much time and money your head will spin. 4) Do not just build a rack full of random commodity gear. Make sure it is suited specifically for your purposes, and then weigh the cost of service contracts and managed colo against a $100k+ cage monkey on call 24/7. 5) Do not fall for the "We've got <insert tech hype>, we don't need redundant hardware!" lie. The more parts of your system rely on lots of hardware, the more fragile your system becomes. Distributed decentralized services become a PITA when the underlying gear is flaky, and centralized services require it. Do not underestimate the shittiness of your colo; always design for the most redundancy you can get for what you have. If you can run dual power, do it. Dual network stacks, do it. Redundant disks, do it. Remote management, do it. Outside modem to a management port on the router, do it. Always be postponing entropy. Later, when you become a FAANG (yeah right) you'll have the time and money to automate away some of the redundancy issues. 6) It's sometimes harder to upgrade disk or bandwidth on an existing machine than it is to just buy another machine, but the more machines you have, the more problems (and overhead costs), and there is an upward limit on the scaling of most services without rebuilding them. So buy big on local disk and network, and buy new machines to expand cpu and memory. The redundancy and performance issues inherent to SAN/NAS often make local disks a better choice, as they are unlikely to impact your whole network at once, you can fix/upgrade them piecemeal, and they are less difficult to manage/operate. 7) Don't DIY critical components such as data replication; it's surprisingly hard. In fact, don't even believe vendors if they claim they can solve X for you with some magic software or hardware. Get them to show you a live demo on your own network. 8) Don't forget that in 3 years you'll be replacing it all.

diehunde 7y ago

So before we had an IT team that would maintain the bare metal servers. Now we need "Cloud Engineers" to maintain the cloud infrastructure working properly. I don't know if the argument of externalize the maintenance of the servers is valid since the complexity of the cloud services is just increasing everyday.

jononor 7y ago
True for those that go for Kubernetes etc straight away. If you just have a few apps to run and put it on a PaaS like Heroku, things are pretty manageable as a side-thing for the regular software developers. No dedicated team, or even person, needed.
- jrumbut 7y ago
  When your infrastructure is too simple, you have simple regrets. When your infrastructure is too exotic, you get exciting, bleeding edge regrets. This is my view based on the work I've done. I like the ability to dispense with capacity planning or dealing with power supplies and fire suppression you get from cloud hosting, but when in any doubt I set up what I need using VMs (be those droplets or EC2 instances or whatever). I like to imagine that I've avoided wasting weeks by wasting a few hours here and there.

MuffinFlavored 7y ago

> A “starter pack” 15amp cabinet with a gigabit connection can be rented for as little as $400 per month. Link?...

ShakataGaNai 7y ago
He.net
snuxoll 7y ago
https://he.net/colocation.html https://he.net/colocation.html Can't speak about the power given on their $400/mo cabinet deal.
mlrtime 7y ago
Yeah, this isn't possible for a entire rack, maybe 1U if it includes network and power if it is in any decent datacenter.
- MuffinFlavored 7y ago
  How much would a 48U rack like described in the article cost a month? I would guess in the thousands if not ten thousands....
  - mlrtime 7y ago
    It mostly goes by power not rack units, if you have 48U and only 15amps, you're not going to fill it up. 15A is not a lot.
    - MuffinFlavored 7y ago
      So, $400/mo is overkill power wise. What would the bandwidth bill look like? How much of an up front investment is it to buy 48U worth of hardware? Seems expensive...
      - dk- 7y ago

fcantournet 7y ago

The problem is not running kubernetes on "bare-metal". That's the somewhat easy and cheap part. The problem is the supply chain to spec, purchase, rack, validate, provision and keep up to date your servers, their drivers and firmware. It's your capacity planning. It's keeping up with the rate of hardware evolution. What happens when you have to go from 10G to 25G networking ? Oh and you also need a solid scalable L4 loadbalancing solution (facebook has a nice opensource dataplane based on ebpf) You can build / assemble from open-source bits most of the classic IaaS bits (compute/storage/networkLB). It's doable, the open-source solutions are pretty good. But unless you have 10k compute nodes how can you bear the cost of having the expert knowledge necessary to debug this ? Hardware sucks and is always on fire. Once you take this into account, the economic equation changes dramatically.

vuln 7y ago

https://github.com/aquasecurity/kube-hunter https://github.com/aquasecurity/kube-hunter https://labs.mwrinfosecurity.com/blog/attacking-kubernetes-through-kubelet/ https://labs.mwrinfosecurity.com/blog/attacking-kubernetes-t...

ranrotx 7y ago

AWS employee here--thoughts and opinions are my own. Prior to AWS, I was in IT Operations at a large financial services company. I saw the writing on the wall that over time, companies would not want to manage this part of their IT infrastructure themselves. Keep in mind, I was someone who was responsible for keeping the lights on for a decent number of Linux severs. For an individual company, there really isn't much value in having to maintain firmware levels on all your hardware, patch hypervisors (and try to coordinate all of the maintenance around a fixed pool of hardware), perform months-long evaluation of new hardware before purchasing, test and validate configurations on new hardware, etc. I used to do all of this. I don't miss it either. Yes, the items above are important, but doing them right is really table-stakes for any reliable IT Operations department. You can choose to spend time getting these right, or delegate that responsibility to a service provider whose main job is to get that stuff right (and recoups that cost across a much larger customer base).

latch 7y ago
What seems to often go unsaid in these discussions is that the choice isn't between cloud and colo. There's a third, hugely popular and mature option: dedicated providers - which address all of your issues. It's convenient for cloud vendors to have people believe the choice is between them or having to deal with hardware.
- dhruvkar 7y ago
  what's an example of a dedicated provider? I also worked in ITOps at a medium-ish company and we were moving our colo to Azure, when I left.
  - arianvanp 7y ago
    Hetzner, scaleway and packet.net come to mind
  - acjacobson 7y ago
    Rackspace

haolez 7y ago

I’ve done this for small projects using AWS and Scaleway. The setup is simple and it just works, but the maintenance and security aspects were more worrying. One time I found out that one of my nodes failed to automatically apply security patches from my distro. I couldn’t detect it sooner because my monitoring was lacking. The overhead escalates fast. If I start an effort like this again today, I would make sure to get monitoring right from day 1. Possibly a combination of Prometheus with Grafana, or something along the lines of ElasticSearch APM.

momofarm 7y ago

I once heard a story that the reason some people would build their own server is they don't trust those service provider on the internet. They feared that the data store there are quite easily breached. Thanks to these guys, they just lower the unemployment rate a little more!

prisar 7y ago

link is broken

numlock86 7y ago
Yup, 404. Even manually going to their blog I can't find the entry anymore. Bookmarked because it was supposed to be an ongoing series with multiple parts. Weird provider. Probably best to avoid.
- compuguy 7y ago
  I have a feeling they deleted the blog post. There is an archived copy here: https://web.archive.org/web/20190508123542/https://gravitational.com/blog/aws_vs_colocation/ https://web.archive.org/web/20190508123542/https://gravitati...