4 ms·
I worked at WhatsApp, prior to moving to Facebook infra, we had some jails for specific things, but mostly ran without containers. Stack looked like: FreeBSD
by toast0 5y ago
I worked at WhatsApp, prior to moving to Facebook infra, we had some jails for specific things, but mostly ran without containers.
Stack looked like:
FreeBSD on bare metal servers (host service provided a base image, our shell script would fetch source, apply patches, install a small handful of dependencies, make world, manage system users, etc)
OTP/BEAM (Erlang) installed via rsync etc from build machine
Application code rsynced and started via Makefile scripts
Not a whole lot else. Lighttpd and php for www. Jails for stud (a tls terminator, popular fork is called hitch) and ffmpeg (until end to end encrypted media made server transcoding unpossible).
No virtualized servers (I ran a freebsd vm on my laptop for dev work, though).
When WA moved to Facebook infra, it made sense to use their deployment methodology for the base system (Linux containers), for organizational reasons. There was no consideration for which methodology was technically superior; both are sufficient, but running a very different methodology inside a system that was designed for everyone to use one methodology is a recipie for operational headaches and difficulty getting things diagnosed and fixed as it's so tempting to jump to the conclusion that any problem found on a different setup is because of the difference and not a latent problem. We had enough differences without requiring a different OS.
- freeqaz 5y agoThat's a very cool deployment method to just use rsync like that! Simple and very composable. And now I feel self-conscious about my pile of AWS and Docker!
- toast0 5y agoIt's 'easy' to do it with rsync and Make as long as you don't have a lot of dependencies. Once you start pulling in dependencies, you have to figure out how to get them installed, hopefully before you roll code that uses them. If the dependencies are small, you can pull them into your source (more or less) and deploy them that way, but that may make tracking upstream harder. (otoh, a lot of things I personally used like that were more of pull this probably abandoned thing that's close to what we need, and modify it quite a bit to fit our needs; there wasn't a need or desire to follow upstream after that)
- tiffanyh 5y agoJust curious, would your deployment method be any different today? If so, how?
- toast0 5y agoIf I was going to do it again, I would probably work on a user management script a lot sooner (you don't really need it, but it would have been nice, and the earlier you do it, the less crufty differences need to be tolerated) and maybe make build on one host, deploy on many or at least push from workstation to colo once and fan out; might have been nice to have that around 100 hosts instead of much later. Of course, today, my fleet is down to less than ten hosts, depending on exactly how you count, and I'm usually the only user, so I can do whatever.
- dolmen 5y agorsync+make is the deployment+run. But this assumes the source of rsync is already static. Handling dependencies is done in a build process, upstream. Fetching dependencies at deployment time is a direct path to deployment failures (especially if the dependency management relies on external resources or non-locked versionning).
- motoboi 5y agoContainers are just tar.gz files, you know? The whole layers thing it’s just an optimization. You can actually very simply run those tar.gz files without docker involved, just cgroups. But then you’ll have to write some daemon scripts to start, stop, restart, etc Follow this path and soon you’ll have a (worst) custom docker. Try to create a network out of those containers and soon a (worst) SDN network appears. Try to expand that to optimal node usage and soon a (worst) Kubernetes appears. My point here is: it’s just software packaged with their dependencies. The rest seems inconsequential, but it’s actually the hard part.
- lamontcg 5y ago"worst" might not really be all that bad. K8s may be the "worst" orchestrator for you, and you may not actually need all or any of that complexity/functionality. But starting with containers viewed as an RPM/tarball with some extra sauce (union filesystem and cgroups/jail) is a way better mental model than the whole "immutable infrastructure" meme to me.
- z3t4 5y agoYou can do a lot with just SystemD today. Would not get surprised if SystemD would eat up Docker too.
- Art9681 5y agoI'm personally interested in Unikernels. That tech seems to be a good contender for replacing K8s. Every app an appliance running in a hypervisor.
- dolmen 5y agoI even imagine a GOOS runtime for compiling a Go program that would talk directly to the hypervisor... Many services I write use only network resources, no disk.
- treesknees 5y agoThis is essentially what fly.io does, they unpack the container image into a micro-VM and execute it directly. Calling it worse is subjective, you certainly need to account for things you would never worry about by using traditional Docker. But for some companies it ends up working much better than Docker. https://fly.io/blog/docker-without-docker/ https://fly.io/blog/docker-without-docker/