10 ms·

Running servers and services well is not trivial (2018)

117 points by rognjen 5y ago

a-dub 5y ago
they forgot "document everything you did so that your replacement can recreate everything exactly as it was from instructions and backups"
- sokoloff 5y ago
  That’s one of the appeals of infrastructure as code. For my own hobby work (where I might put down a project for months between serious work sessions, I often am that replacement). Some projects I have well enough structured or documented to pick right back inside of 30 minutes; others are more archeological in nature.
  - cersa8 5y ago
    I wish there was a statically typed version of Ansible. There is Pulumi but it's mostly tied to cloud API's and not Linux system administration (setting up HAProxy, NGINX, PostgreSQL, users)
    - nijave 5y ago
      I wish Ansible just had better data handling. Adding variable assignment plays and jamming things in Jinja is pretty clunky (if you want to, say, pull some data from a REST API and loop over a subset of the info) You could always make an Ansible module but then there's the overhead to managing/installing that Data wrangling can also make idempotent playbooks a bit clunky, too. You get into this 2-4 play "run check, reshape results, conditionally run play" pattern
    - edude03 5y ago
      Me too, though nix (https://nixos.org https://nixos.org) and eventually nickel (https://nickel-lang.org https://nickel-lang.org) get you very close today.
    - baq 5y ago
      Not exactly that, but definitely take a look: https://dhall-lang.org/ https://dhall-lang.org/
  - pid-1 5y ago
    Most software is hostile to IaC, unfortunately. Even cloud providers, who have APIs for everything, will fuck up at making services that can be deployed as code in a sane way.
    - sokoloff 5y ago
      If I’m being honest with myself, more of my side projects have a bin/publish script than are actually IaC.
      - maccard 5y ago
        bin/publish.sh is better than remembering whether the service is on kubernetes, containers or a raw ec2 box, and "do I need to restart the service after update or just wait"
  - bezospen15 5y ago
    Lol you're overthinking it. We will just take a snapshot and call it a day
- throwaway984393 5y ago
  "If it's not written down it doesn't exist"
- voiper1 5y ago
  Not just your replacement! If I have to check or change something or recreate it even 6 months later I often have no clue what the heck I did.
goodpoint 5y ago
Sooner or later the pendulum will swing back, hopefully, and we'll have a new generation of secure and easy to maintain local services.
nijave 5y ago
Article leaves out cost of migration. If you're trying to get better uptime, you're likely going to incur a downtime migrating everything over so you've already taken a few step back uptime wise unless the platform you're moving off of is already horrible. The article touches on it, but there's also compliance concerns (access control is mentioned, also retention policies, DR, ability to redact/remove improperly added data) They briefly mentioned monitoring but, like auth, can be non trivial for a well built system. Something like email is easy until the entire system is down and the alerts don't go out anymore (so you need an external monitoring system) I also don't see labor cost mentioned very often. How many hours will it take to support?
kqr 5y ago
This is one thing feel intuitively but have trouble arguing. When a solution requires running a server, I mentally count it as much more expensive than if it's done locally and synchronously. I've noticed not everyone shares this bias, and I'm wondering if I'm unnecessarily conservative or other people are underestimating maintenance costs.
- pid-1 5y ago
  "just run your own git server" comments always make me scratch my head. IMO folks managing services for personal use vastly underestimate how much harder everything can get in an actual business environment.
  - nijave 5y ago
    I don't think it's necessarily just "personal use". The complexity quickly scales with the company size. Replacing Github with Gitlab might be a week-long project at a small company but could easily turn into a multi-year project at a large one. It's not just a technical problem, either. Bigger companies tend to have more expertise-oriented teams (security, compliance, developer tooling, operations, internal infrastructure) which tends to make decisions more difficult than when a single person or team can do it themselves.
    - vlunkr 5y ago
      > Replacing Github with Gitlab might be a week-long project at a small company A week-long project initially. Now you have to install updates, set up and maintain secure access, reboot or troubleshoot when it dies, etc. Installing things is the easy part.
      - netizen-936824 5y ago
        What kind of a company doesn't have at least one sysadmin, I feel like that's kind of a critical position for maintaining systems?
krnlpnc 5y ago
Writing good code is not trivial either, so while at it just buy a SAAS.
- speedgoose 5y ago
  Buying SaaS is not trivial either.
  - maccard 5y ago
    I do most of the service work for a team of 25 - half developers and half non technical product people (art/design). Saas is substantially easier than writing it yourself or even running it yourself. Github costs $4/user/month, sentry costs $25/month, and 2 digitalocean k8s clusters costs $20/month. For $175/month I can have a decelopment environment with basically 0 maintenance for a team of 25 people, including monitoring and alerting for my app. Compared to running a local gitlab instance, deploying OpenTelemetry and running my own k8s cluster in aws, it's a complete no brainer to buy SAAS.
    - speedgoose 5y ago
      Yes almost everyone does that. But outside of the common SaaS services, all SaaS companies with a “contact us” price I contacted came back with a “fuck you” offer.
      - maccard 5y ago
        I've found the exact opposite. I work in games and many of the vendors we deal with don't publicly post their prices, but I've found their pricing fair and competitive. It's not "race to the bottom $4/user/month" but it's usually a fair quote.
        speedgoose 5y ago
        That’s nice. I work in the data and cloud industry and everyone thinks their customers are MoneyBagsInc, which is very annoying.
annoyingnoob 5y ago
I've been running infrastructure for a long time. I feel like all of the work is just part of my job, things I do regularly. You could replace git with some other service/software and each has its own requirements and considerations. I would not call my work trivial but running git and lots of other things well is completely reasonable.
- throwmeariver1 5y ago
  The problem is not with the services itself it's more in regards of infrastructure, backups etc. If you run the services yourself there is not a single point of failure as you would have with a SAAS.
  - xyzzy123 5y ago
    Also team considerations. When you add an important moving part (all your devs will complain bitterly when it breaks and their work will be significantly impacted) you need at least 2 people who know how to keep it running at a minimum, and more would be better.
erulabs 5y ago
It’s just a matter of tooling. Do you think AWS logs into the servers that back your service and care for them individually (as the article imagines caring for your own server)? Not even in the least - they treat them like cattle. If data center scale tools were easier to use for the average engineer - the gap between managed and self-hosted would start to close. Obviously, software, tooling, process, scale: there are plenty of huge challenges to making self-hosting viable. Personally, I see it as far more doable and less radical than trying to distribute the internet in any other fashion (blockchain or otherwise).
- WJW 5y ago
  I agree that it's a tooling issue, but I don't agree that the gap between managed and self-hosted will ever start to close again. IMO, modern IT tooling is having an industrial revolution moment where economies of scale really start to push out small scale operators. Back in the day every village would have their own blacksmith, but as steel mills scaled up fewer and fewer of them were needed. These days no hobbyist metalworker can ever hope to keep up with the accuracy and speed of professionals running multi-million CNC machines. Similarly, big tech companies will keep developing tools that provide total cost-of-ownership several orders cheaper than what hobbyists achieve, but will need to be operated by entire teams of people. The complexity of Kubernetes is only the start. No consumer refines their own petroleum or makes their own steel these days, except maybe as a hobby. Similarly I don't think the market share of consumers who host their own email, git servers or payment infrastructure will ever rise again.
  - spicyusername 5y ago
    > Similarly I don't think the market share of consumers who host their own email, git servers or payment infrastructure will ever rise again. Every IT shop I've ever interacted with has a sizable on-prem footprint. And an even larger self-managed footprint if we include things that are run in the cloud but are managed by the IT shop itself (i.e. Red Hat OpenShift on AWS, HashiCorp Vault in Azure, GitLab in Alibaba, etc). So I think we've yet to see the initial demise of that paradigm. Even if the trend is that most Enterprises are moving towards SaaS and PaaS products, I think we still have a long way to go until the majority of IT infrastructure is managed by a third-party.
mountainriver 5y ago
This is why Kubernetes is complex, people who shout that need haven’t internalized articles like this
johnklos 5y ago
"I'm new to systems administration, and this thing is hard for me and for many others, so we should all stop doing it." Does that about sum it up? Why are there even hackers? People should stop doing tricky things because others might find those things difficult. We're making them uncomfortable and should stop.
- wintermutestwin 5y ago
  Isn't the real answer: This thing is unnecessarily hard, so let's get together and fix it. One clear thing that needs fixing: Linux desperately needs definitive how tos for every common thing an admin/user might need to do posted and maintained on a distro specific/owned site. As it is now, when I want to learn how to do {thing1}, I have to sift through a complex maze of stack questions, blog posts, youtube videos, etc. Many are ancient, don't really apply to the distro I'm on, fail to mention that there are other ways of doing it (some of which might be better for a given situation), etc. Then, when I finally settle on a howto, it fails to work and then I burn hours troubleshooting and tweaking. Eventually, I get it to work but when I reflect on what it took to get there, I couldn't really follow the breadcrumbs of my frustrated efforts well enough to document it for posterity. Eventually, I run into problems with {thing2} and I realize that the fastest way to troubleshoot is to wipe the box and start clean, but I can't because recreating {thing1} is a multi-hour task.
  - LilBytes 5y ago
    Is this not where containers improve this a lot? Plus the use of Packer and Ansible with CI/CD, you'll see where your builds and deploys break quickly and see those failures before you need a server replacement now.
  - chlorion 5y ago
    The first source I always check are man pages, both for the program and for the config files for that program. Then anything else is normally on the Arch or Gentoo wiki. The are very few topics that aren't covered by one or more of these sources! I think most people don't consider checking those wikis because they aren't using those distros, but right now they are the largest and most comprehensive source of information that we have and most of the content is generic and works with any distro or even non-linux OSes.
cassepipe 5y ago
Fossil has built-in web interface and does not require a central server : "Fossil does not require a central server. Data sharing and synchronization can be entirely peer-to-peer. Fossil uses conflict-free replicated data types to ensure that (in the limit) all participating peers see the same content. " ...but still it is easy to set up on a central server : https://fossil-scm.org/home/doc/trunk/www/server/whyuseaserver.wiki https://fossil-scm.org/home/doc/trunk/www/server/whyuseaserv... https://fossil-scm.org/home/doc/trunk/www/webui.wiki https://fossil-scm.org/home/doc/trunk/www/webui.wiki https://fossil-scm.org/home/doc/trunk/www/server/ https://fossil-scm.org/home/doc/trunk/www/server/
Havoc 5y ago
I can see that for workplace. For home use I’ve found a local gitlab server to be a neat trick though.

117 points by rognjen 5y ago

they forgot "document everything you did so that your replacement can recreate everything exactly as it was from instructions and backups"

sokoloff 5y ago
That’s one of the appeals of infrastructure as code. For my own hobby work (where I might put down a project for months between serious work sessions, I often am that replacement). Some projects I have well enough structured or documented to pick right back inside of 30 minutes; others are more archeological in nature.
- cersa8 5y ago
  I wish there was a statically typed version of Ansible. There is Pulumi but it's mostly tied to cloud API's and not Linux system administration (setting up HAProxy, NGINX, PostgreSQL, users)
  - nijave 5y ago
    I wish Ansible just had better data handling. Adding variable assignment plays and jamming things in Jinja is pretty clunky (if you want to, say, pull some data from a REST API and loop over a subset of the info) You could always make an Ansible module but then there's the overhead to managing/installing that Data wrangling can also make idempotent playbooks a bit clunky, too. You get into this 2-4 play "run check, reshape results, conditionally run play" pattern
  - edude03 5y ago
    Me too, though nix (https://nixos.org https://nixos.org) and eventually nickel (https://nickel-lang.org https://nickel-lang.org) get you very close today.
  - baq 5y ago
    Not exactly that, but definitely take a look: https://dhall-lang.org/ https://dhall-lang.org/
- pid-1 5y ago
  Most software is hostile to IaC, unfortunately. Even cloud providers, who have APIs for everything, will fuck up at making services that can be deployed as code in a sane way.
  - sokoloff 5y ago
    If I’m being honest with myself, more of my side projects have a bin/publish script than are actually IaC.
    - maccard 5y ago
      bin/publish.sh is better than remembering whether the service is on kubernetes, containers or a raw ec2 box, and "do I need to restart the service after update or just wait"
- bezospen15 5y ago
  Lol you're overthinking it. We will just take a snapshot and call it a day
throwaway984393 5y ago
"If it's not written down it doesn't exist"
voiper1 5y ago
Not just your replacement! If I have to check or change something or recreate it even 6 months later I often have no clue what the heck I did.

goodpoint 5y ago

Sooner or later the pendulum will swing back, hopefully, and we'll have a new generation of secure and easy to maintain local services.

nijave 5y ago

Article leaves out cost of migration. If you're trying to get better uptime, you're likely going to incur a downtime migrating everything over so you've already taken a few step back uptime wise unless the platform you're moving off of is already horrible. The article touches on it, but there's also compliance concerns (access control is mentioned, also retention policies, DR, ability to redact/remove improperly added data) They briefly mentioned monitoring but, like auth, can be non trivial for a well built system. Something like email is easy until the entire system is down and the alerts don't go out anymore (so you need an external monitoring system) I also don't see labor cost mentioned very often. How many hours will it take to support?

kqr 5y ago

This is one thing feel intuitively but have trouble arguing. When a solution requires running a server, I mentally count it as much more expensive than if it's done locally and synchronously. I've noticed not everyone shares this bias, and I'm wondering if I'm unnecessarily conservative or other people are underestimating maintenance costs.

pid-1 5y ago
"just run your own git server" comments always make me scratch my head. IMO folks managing services for personal use vastly underestimate how much harder everything can get in an actual business environment.
- nijave 5y ago
  I don't think it's necessarily just "personal use". The complexity quickly scales with the company size. Replacing Github with Gitlab might be a week-long project at a small company but could easily turn into a multi-year project at a large one. It's not just a technical problem, either. Bigger companies tend to have more expertise-oriented teams (security, compliance, developer tooling, operations, internal infrastructure) which tends to make decisions more difficult than when a single person or team can do it themselves.
  - vlunkr 5y ago
    > Replacing Github with Gitlab might be a week-long project at a small company A week-long project initially. Now you have to install updates, set up and maintain secure access, reboot or troubleshoot when it dies, etc. Installing things is the easy part.
    - netizen-936824 5y ago
      What kind of a company doesn't have at least one sysadmin, I feel like that's kind of a critical position for maintaining systems?

krnlpnc 5y ago

Writing good code is not trivial either, so while at it just buy a SAAS.

speedgoose 5y ago
Buying SaaS is not trivial either.
- maccard 5y ago
  I do most of the service work for a team of 25 - half developers and half non technical product people (art/design). Saas is substantially easier than writing it yourself or even running it yourself. Github costs $4/user/month, sentry costs $25/month, and 2 digitalocean k8s clusters costs $20/month. For $175/month I can have a decelopment environment with basically 0 maintenance for a team of 25 people, including monitoring and alerting for my app. Compared to running a local gitlab instance, deploying OpenTelemetry and running my own k8s cluster in aws, it's a complete no brainer to buy SAAS.
  - speedgoose 5y ago
    Yes almost everyone does that. But outside of the common SaaS services, all SaaS companies with a “contact us” price I contacted came back with a “fuck you” offer.
    - maccard 5y ago
      I've found the exact opposite. I work in games and many of the vendors we deal with don't publicly post their prices, but I've found their pricing fair and competitive. It's not "race to the bottom $4/user/month" but it's usually a fair quote.
      - speedgoose 5y ago
        That’s nice. I work in the data and cloud industry and everyone thinks their customers are MoneyBagsInc, which is very annoying.

annoyingnoob 5y ago

I've been running infrastructure for a long time. I feel like all of the work is just part of my job, things I do regularly. You could replace git with some other service/software and each has its own requirements and considerations. I would not call my work trivial but running git and lots of other things well is completely reasonable.

throwmeariver1 5y ago
The problem is not with the services itself it's more in regards of infrastructure, backups etc. If you run the services yourself there is not a single point of failure as you would have with a SAAS.
- xyzzy123 5y ago
  Also team considerations. When you add an important moving part (all your devs will complain bitterly when it breaks and their work will be significantly impacted) you need at least 2 people who know how to keep it running at a minimum, and more would be better.

erulabs 5y ago

It’s just a matter of tooling. Do you think AWS logs into the servers that back your service and care for them individually (as the article imagines caring for your own server)? Not even in the least - they treat them like cattle. If data center scale tools were easier to use for the average engineer - the gap between managed and self-hosted would start to close. Obviously, software, tooling, process, scale: there are plenty of huge challenges to making self-hosting viable. Personally, I see it as far more doable and less radical than trying to distribute the internet in any other fashion (blockchain or otherwise).

WJW 5y ago
I agree that it's a tooling issue, but I don't agree that the gap between managed and self-hosted will ever start to close again. IMO, modern IT tooling is having an industrial revolution moment where economies of scale really start to push out small scale operators. Back in the day every village would have their own blacksmith, but as steel mills scaled up fewer and fewer of them were needed. These days no hobbyist metalworker can ever hope to keep up with the accuracy and speed of professionals running multi-million CNC machines. Similarly, big tech companies will keep developing tools that provide total cost-of-ownership several orders cheaper than what hobbyists achieve, but will need to be operated by entire teams of people. The complexity of Kubernetes is only the start. No consumer refines their own petroleum or makes their own steel these days, except maybe as a hobby. Similarly I don't think the market share of consumers who host their own email, git servers or payment infrastructure will ever rise again.
- spicyusername 5y ago
  > Similarly I don't think the market share of consumers who host their own email, git servers or payment infrastructure will ever rise again. Every IT shop I've ever interacted with has a sizable on-prem footprint. And an even larger self-managed footprint if we include things that are run in the cloud but are managed by the IT shop itself (i.e. Red Hat OpenShift on AWS, HashiCorp Vault in Azure, GitLab in Alibaba, etc). So I think we've yet to see the initial demise of that paradigm. Even if the trend is that most Enterprises are moving towards SaaS and PaaS products, I think we still have a long way to go until the majority of IT infrastructure is managed by a third-party.

mountainriver 5y ago

This is why Kubernetes is complex, people who shout that need haven’t internalized articles like this

johnklos 5y ago

"I'm new to systems administration, and this thing is hard for me and for many others, so we should all stop doing it." Does that about sum it up? Why are there even hackers? People should stop doing tricky things because others might find those things difficult. We're making them uncomfortable and should stop.

wintermutestwin 5y ago
Isn't the real answer: This thing is unnecessarily hard, so let's get together and fix it. One clear thing that needs fixing: Linux desperately needs definitive how tos for every common thing an admin/user might need to do posted and maintained on a distro specific/owned site. As it is now, when I want to learn how to do {thing1}, I have to sift through a complex maze of stack questions, blog posts, youtube videos, etc. Many are ancient, don't really apply to the distro I'm on, fail to mention that there are other ways of doing it (some of which might be better for a given situation), etc. Then, when I finally settle on a howto, it fails to work and then I burn hours troubleshooting and tweaking. Eventually, I get it to work but when I reflect on what it took to get there, I couldn't really follow the breadcrumbs of my frustrated efforts well enough to document it for posterity. Eventually, I run into problems with {thing2} and I realize that the fastest way to troubleshoot is to wipe the box and start clean, but I can't because recreating {thing1} is a multi-hour task.
- LilBytes 5y ago
  Is this not where containers improve this a lot? Plus the use of Packer and Ansible with CI/CD, you'll see where your builds and deploys break quickly and see those failures before you need a server replacement now.
- chlorion 5y ago
  The first source I always check are man pages, both for the program and for the config files for that program. Then anything else is normally on the Arch or Gentoo wiki. The are very few topics that aren't covered by one or more of these sources! I think most people don't consider checking those wikis because they aren't using those distros, but right now they are the largest and most comprehensive source of information that we have and most of the content is generic and works with any distro or even non-linux OSes.

cassepipe 5y ago

Fossil has built-in web interface and does not require a central server : "Fossil does not require a central server. Data sharing and synchronization can be entirely peer-to-peer. Fossil uses conflict-free replicated data types to ensure that (in the limit) all participating peers see the same content. " ...but still it is easy to set up on a central server : https://fossil-scm.org/home/doc/trunk/www/server/whyuseaserver.wiki https://fossil-scm.org/home/doc/trunk/www/server/whyuseaserv... https://fossil-scm.org/home/doc/trunk/www/webui.wiki https://fossil-scm.org/home/doc/trunk/www/webui.wiki https://fossil-scm.org/home/doc/trunk/www/server/ https://fossil-scm.org/home/doc/trunk/www/server/

Havoc 5y ago

I can see that for workplace. For home use I’ve found a local gitlab server to be a neat trick though.