7 ms·
As somebody who has his own colocated server (and has since Bubble 1.0), I definitely agree that the old-fashioned way still works just fine. On the other hand
by wpietri 8y ago
As somebody who has his own colocated server (and has since Bubble 1.0), I definitely agree that the old-fashioned way still works just fine.
On the other hand, I've been building a home Kubernetes cluster to check out the new hotness. And although I don't think Kubernetes provides huge benefits to small-scale operators, I would still probably recommend that newbs look at some container orchestration approach instead of investing in learning old-school techniques.
The problem for me with the old big-server-many-apps approach is the way it becomes hard to manage. 5 years on, I know that I did a bunch of things for a bunch of reasons, but I don't really remember what or why. It mixes intention with execution in a way that gets muddled over time. Moving to a new server or OS is more archaeology than engineering.
The rise of virtual servers and tools like Chef and Puppet provided some ways to manage that complexity. But "virtual server" is like "horseless carriage". The term itself indicates that some transition is happening, but that we don't really understand it yet.
I believe containers are at least the next step in that direction. Done well, I think containers are a much cleaner way of separating intent from implementation than older approaches. Something like Kubernetes strongly encourages patterns that make scaling easier, sure. But even if the scaling never happens, it makes people better prepared for operational issues that certainly will happen. Migrations, upgrades, hardware failures, transfers of control.
- heipei 8y agoThis. Kubernetes (or whatever other container scheduler) might feel like overkill, but if all they do is force you to adopt a container-centric / 12-factor way of building your applications it was worth trying them. And once you've adopted that workflow it's a no-brainer to go from a single node to a cluster which will dynamically allocate the workloads it runs. Running a small container cluster at work has even changed how I setup single-host projects in my spare time: I will build everything into a container, bind-mount whatever it might need, create a simple systemd unit that just runs / rms the docker container on start and stop. Bliss.
- 3pt14159 8y agoI've found it just pushes the complexity elsewhere or opens up (or silences) performance or security problems you wouldn't have had if you'd stuck to the old fashion way of doing things. Keep checklists and script what you can. I find it helpful to follow edge whenever I can so if I hit a snag the developers that made the change still have that change fresh in their mind. It really doesn't take that much time to keep things updated if you don't go hog-wild on different libraries and if your stuff is reasonably well tested. As an aside, I've been thinking that there should be a stack that is designed and built for the sole purpose of staying stable over decades. Something with a bunch of stripped down technology. As robust and stable as possible. Only allow security updates. Only allow certain character sets. Something built on a language that is just stupid simple and secure. A Swift or Rust subset maybe? Lua? Lisp without the macro insanity? If the future needs some sort of tech that we didn't anticipate (say, something to handle quantum computers breaking cryptography) then the stack should be setup in such a way to decouple the varying layers with minimal work. I liken it to building codes. We should have pre-setup combinations of technologies that are stable, simple, and combinable. Sure, go outside them for skyscrapers, but for the day-to-day buildings things are getting too complicated.
- jammygit 8y agoHow do you use checklists in your workflows - are they part of your repository alongside the code, in some documentation system, printed out? I'm most of the way through the checklist manifesto and I'd love some insight on how software engineers incorporate them into their work.
- 3pt14159 8y agoFunny. After keeping them in files and emails and docs for years I finally decided to systematize it by writing a CLI that I plan to open source one day. If you want, send me an email and we can talk about it further.
- jerf 8y ago"5 years on, I know that I did a bunch of things for a bunch of reasons, but I don't really remember what or why." For my home servers, I've settled on "a default install of distro $X and an idempotent shell script that sets everything up for me". You have to use discipline to do everything in the shell script rather than simply fix the problem, but if you can do that, you end up with documentation as to how your server differs from a default install, and the ability to recover it again reasonably well if you store it in git somewhere or something. It's only "reasonably" well because when you have one server running for years at a time, your script decays more quickly than you are going to fix it. If your server goes down three years later, and you decide to go with the latest $X instead of whatever you used last time, then your script will be out of date and need to be updated. It isn't nirvana. But it's the best bang for the buck when you're in a situation where chef/ansible/puppet/etc. is massive, massive overkill. If you're already an expert with Docker, go nuts, but IMHO it's a bit silly to run a server just to run two Docker containers, just so you can say you're running Docker or something. Plus no matter how slick Docker has gotten, it's still more of a pain that just setting a few things up.
- m_mueller 8y agodocker image definitions are idempotent as a matter of principle. creating an idempotent shell script is non trivial IMO - e.g. what return code is returned from package manager XY when something is already installed etc.
- TheDong 8y agoReally? What do your dockerfiles look like? Most of the ones I've seen in the wild look something like: FROM debian:jessie RUN apt-get install -y somepkg ... What happens when the "debian:jessie" image changes (as it does weekly on dockerhub)? What about "somepkg" in debian's repositories? The answer is that `docker build` will produce a different image. In fact, very few docker image builds I've seen are idempotent. They're not declarative, they're not reproducible, merely the output (the docker image itself) is itself able to be run reproducibly. The actual image definition, not so much. Creating idempotent shell scripts is no harder than creating an idempotent dockerfile. Both are the same problem. A dockerfile is almost entirely the same as a shell script; it copies files around, it runs commands in an environment, and that's all.
- vidarh 8y agoYou can put everything in containers and still not need much orchestration, though. My personal projects run in dozens of containers, and the "orchestration" consists of a Makefile include pulled into each project that creates a systemd service file based on some variables, and pushes it into the right place. The service files will pull down and set up a suitable Docker container. The full setup for a couple of dozen containers is 40-50 lines of makefile and a ~20 lines or so of a template service file. Of course it won't scale to massive projects, and for work, I occasionally use Kubernetes and other more "serious" orchestration alternatives, but frankly it takes fairly big projects before they start paying for themselves in complexity. Meanwhile, my docker containers has kept chugging along without needing any maintenance aside from auto-updating security updates for several years. I do agree with you that Kubernetes may encourage patterns that are useful, though. But really the most essential part is that you can find relatively inexperienced devops people who have picked up some Kubernetes skills. That availability make up for a lot of pain vs finding someone experienced enough to wire up a much leaner setup.
- ori_b 8y ago> Of course it won't scale to massive projects Most projects aren't massive -- at work, 2 years on, we're still using a single instance of a single node, with the only component that needs to be reliable stored as static files in S3.
- vidarh 8y agoAbsolutely. Which is one of the reasons I find things like Kubernetes overkill for most setups.
- ngrilly 8y agoWhen you deploy a new version of a container, how do you avoid downtime? Do you start a new container running the new version, wait for the new container to be ready, switch traffic to the new container, stop traffic to the old container and drain connections, and then stop the old container?
- 8y ago
- taeric 8y agoI'm curious how docker will help with the "5 years on" problem. I'd be willing to place money saying your docker setup for this week will have trouble running "as is" next month. Especially true for the vast majority of one-offs out there.
- timr 8y agoThis is already happening. People are using "containers" the way that they used AMIs, which is the way that they used VMs: as a black-box execution environment that magically abstracts all problems. Until something breaks. As soon as you have to upgrade or fix anything inside the container, you're back to the same old set of challenges, which remain unsolved. But this isn't really why people are using containers. Probably the biggest reason that folks are stuck with this complexity (outside of big server farms) is because containers are being foisted upon them by companies who want to sell software, but don't want to solve deployment issues. Why plan for execution in an uncertain environment when you can just require Docker?
- weberc2 8y agoI use Docker for my single server because it (and its ecosystem) offers a straightforward path for: 1. Deployment 2. Distribution (I don't have to build a package for every platform) 3. Supervisory 4. Standard logging 5. Configuration management 6. Infrastructure as code 7. Process isolation (not perfect, but I can get some reasonable protection without managing VMs or figuring out how to roll my own isolation and permissioning) 8. Networking Basically I don't have to be a professional sysadmin but a "mere" engineer (yes, yes, in a perfect world I would have time to learn everything "properly", but for all its faults, Docker lets me build something useful Right Now). EDIT: For downvoters, I'd really appreciate more elaborate feedback.
- Aeolun 8y agoSorry, no idea. Seems like a decent summary of why docker would be interesting to a dev instead of a sysadmin.
- 8y ago
- Jedd 8y ago> The problem for me with the old big-server-many-apps approach is the way it becomes hard to manage. 5 years on, I know that I did a bunch of things for a bunch of reasons, but I don't really remember what or why. I thought this was a solved problem. I use SaltStack for config management & orchestration on my own machines. (I suggest any config management tool becomes 'worth the effort' once you're managing more than a handful of machines, and/or want to rebuild or spin up new machines with minimal effort and surprises more than once a year.) Why I do something is described in a comment in the yaml that does the something. For more nuanced situations, I'll document it in my wiki. (With a yaml comment pointing to same -- I am extremely pessimistic about future-Jedd's ability to remember things.) If you're running a big-server-many-apps or many-servers-with-their-own-apps, I'd expect the same approach to work equally well. Though the whole idea of virtual servers & config management (but not necessarily docker or k8s) is that you don't have a bunch of disparate and potentially conflicting apps with potentially conflicting dependencies on a single server. > But "virtual server" is like "horseless carriage". The term itself indicates that some transition is happening, but that we don't really understand it yet. That's a challenging assertion to fully unpack. IT's undeniably in a constant state of transition -- and not always thoughtfully directed -- but the problem isn't _'virtual server'_. The general trend is obviously towards isolation -- but the tooling, performance, scaling, design, and security disparities make the arguments around what level you try to implement your isolation so interesting.
- wpietri 8y agoI agree that "virtual server" isn't a problem. Neither was "horseless carriage" or "radio with pictures". All of them were steps forward. But they're transitional states on the way to new paradigms. When servers were expensive things that had full-time staff, the old ways of installing software made a lot of sense. But as server power got cheaper, they became impractical. A virtual server was at least familiar; slicing big machines up let us turn the clock back to when servers were less powerful. But that didn't really solve the problem, as we now had to do something to manage the explosion in the number of servers, real and virtual. Things like chef and puppet jumped in to solve this, but they are IMHO clumsy; it's all the work of managing a lot of servers the old way, even though it may be a small number of physical boxes. Containerization says: Forget about installing apps on servers; just wrap the app up with what it needs. Things like Kubernetes takes that further, saying: Don't worry about which apps are running on which servers; it'll just work. The impedance mismatch between modern hardware and the 1970s-university-department paradigm that underlies Unix gets solved automatically. I'm not sure if that's the end state in the paradigm shift. But I'm convinced that the approach to sysadminning I learned in the 1980s is on its way out.
- deleted 8y ago[deleted]
- weberc2 8y ago> I've been building a home Kubernetes cluster to check out the new hotness I tried to do this for the same reason, but all of the writeups seem to stop at "getting a cluster running", but that's not enough to actually run apps since you need a load balancer / ingress, dns, and probably a number of other things (ultimately I was overwhelmed by the number of things I needed but didn't completely understand). I haven't had any luck finding a soup-to-nuts writeup, so if you have any recommendations, I'd love to hear them.
- londons_explore 8y agoI believe those chef and puppet scripts won't help you resurrect a project from 5 years ago. You'll practically have to rewrite all the scripts to get it up and running in whatever new hotness is around in 2023. Package names will have changed, new config systems will be invented, and previous workarounds for bugs will now cause bugs.