17 ms·

Docker without Docker

689 points by kwindla 6y ago

fcatus 6y ago
you cant even try it without giving them your credit card info.. Looked interesting but nah...
- kondro 6y ago
  They have a pretty generous free tier though: https://fly.io/docs/about/pricing/ https://fly.io/docs/about/pricing/
- tptacek 6y ago
  We're a hosting provider. Later The parent comment is getting downvoted into oblivion (because, again, we're a hosting provider) but it's an honest observation! It just happens to have a simple answer.
- xfer 6y ago
  Otherwise it will be used for crypto mining.
  - richardfey 6y ago
    It will be used for crypto mining regardless, using stolen credit cards.
    - tptacek 6y ago
      Yup. But without the credit cards, it will be used only for crypto mining. :)
kondro 6y ago
Fly are very generous with what they're willing to share with the world.
- tptacek 6y ago
  We love you all so much.
  - atonse 6y ago
    Honestly the more I read these posts, the more I want to use the product. :) (Full-time elixir dev running a SaaS and 8+ projects on elixir, including covid vaccine scheduling system that handles thousands of users with no issues. LOVE elixir.)
    - tptacek 6y ago
      WE LOVE ELIXIR. I don't even write Elixir and I love it, because we built a system that makes it super easy to take a Dockerfile of some random thing, and deploy 20 instances of it spread evenly around some bunch of regions that can talk privately to each other by default. That's a fun thing to be able to do in any language, but to exploit it in a Go program, I'd have to actually think, pull something like Serf in, whatever. But Phoenix LiveView apps literally ship with a dashboard where you can see all your instances running, and then first-class features for them to chat between themselves. LiveView is cool all on its own, but Elixir+Phoenix+LiveView is such a sweet application of a hosting environment like ours. Expect us to be weirdly chatty about Elixir in the coming months.
      - therein 6y ago
        Looking forward to it.
      - cpursley 6y ago
        I'm looking forward to this. Is it going to be possible to run local Postgres (or one of the distributed dbs that are PG compatable) read instances alongside the Elixir instances?
        mrkurt 6y ago
        Yes. Our beta postgres service lets you add read only replicas in any region you want. "Deploy any full stack app globally" has been a big goal for us over the last year. We're getting close.
coding123 6y ago
transmogrify = convert
- jmholla 6y ago
  Yup. It's a reference to Calvin and Hobbes: http://1.bp.blogspot.com/-DXzlDXdGjlM/UGSlryq02vI/AAAAAAAAYdc/OJo1GDMOZRU/s1600/transmogrifier.png http://1.bp.blogspot.com/-DXzlDXdGjlM/UGSlryq02vI/AAAAAAAAYd...
  - lloeki 6y ago
    Is it? I remember reading the word from sources that predate C&H by a lot (early 20th c.). Doing my homework, a quick search shows uses as far back as 17th c. (which I din't quite expect!) https://www.merriam-webster.com/dictionary/transmogrify https://www.merriam-webster.com/dictionary/transmogrify
deleted 6y ago
[deleted]
nunez 6y ago
Actually a Dockerfile parser that can spin up Firecracker VMs and handle volume/net_iface binding transparently would be really neat.
- tptacek 6y ago
  Our local dev environment does this, but it does it by running our driver (so we can test the driver) so it's a lot more mechanism than you'd need. You could get this done yourself pretty easily: just unpack the Docker image (there's a Docker command that does this, I forget what) onto a mounted loop device, then hand its block device off to Firecracker. Our driver does its own local address goop, but it's not complicated (it just dumps netlink and allocates the next available subnet in a particular range). Having said all that: there are projects that do this already; for instance, I think there's a Kata Firecracker somewhere.
- rad_gruchalski 6y ago
  I am working on something like this: https://github.com/combust-labs/firebuild https://github.com/combust-labs/firebuild. From a Dockerfile, it's not as simple without creating an image first. The command @tptacek means is most likely docker export for a running container and docker save for an image. First exports selected paths, second exports a complete image as a tgz archive.
mperham 6y ago
This is really cool infrastructure work and impressively open.
- mrkurt 6y ago
  We're happy Sidekiq Pro customers too.
hardwaresofton 6y ago
I'm really impressed by fly.io, and the candidness with which they share some of their really awesome technology. Being container-first is the next step for PaaS IMO and they are ahead of the pack. I aim to build a platform like theirs someday (probably not any time soon) but I don't think I'd do any of what they're doing -- it feels unnecessary, because I feel like I know lots of projects that have actually already done the hard work for this. I probably only think this because I haven't actually done it, but think it's worth sharing the tech anyway. Bear with me as I recently learned that they use nomad[0] and some of these suggestions are kubernetes projects but I'd love to hear why the following technologies were decided against (if they were): - kata-containers[1] (it does the whole container -> VM flow for you, automatically) with multiple VMM options[2] - linuxkit[3] (let's say you didn't go with kata-containers, this is another container->VM path) - firecracker-containerd[4] (very minimal keep-your-container-but-run-it-as-a-VM) - kubevirt[5] (if you just want to actually run VMs, regardless of how you built them) - Ceph[6] for storage -- make LVM pools and just give them to Ceph, you'll get blocks, distributed filesystems (CephFS), and object gateways (S3/Swift) out of it (in the k8s space Rook manages this) As an aside to all this, there's also LXD, which supports running "system" (user namespace isolated) containers, VMs (somewhat recent[7][8]), live migration via criu[9], management/migration of underlying filesystems, runs on LVM or zfs[10], it's basically all-in-one, but does fall behind in terms of ecosystem since everyone else is aboard the "cloud native"/"works-with-kubernetes" train. I've basically how I plan to run a service like fly.io if I ever did -- so maybe my secret is out, but I sure would like to know just how much of this fly.io got built on (if any of it), and/or what was turned down. [0]: https://news.ycombinator.com/item?id=26745514 https://news.ycombinator.com/item?id=26745514 [1]: https://github.com/kata-containers/kata-containers https://github.com/kata-containers/kata-containers [2]: https://github.com/kata-containers/kata-containers/blob/2fc7f75724ac9e18e60f63dcc9aa395dc51c184d/docs/design/virtualization.md https://github.com/kata-containers/kata-containers/blob/2fc7... [3]: https://github.com/linuxkit/linuxkit https://github.com/linuxkit/linuxkit [4]: https://github.com/firecracker-microvm/firecracker-containerd/ https://github.com/firecracker-microvm/firecracker-container... [5]: https://github.com/kubevirt/kubevirt https://github.com/kubevirt/kubevirt [6]: https://docs.ceph.com/ https://docs.ceph.com/ [7]: https://discuss.linuxcontainers.org/t/running-virtual-machines-with-lxd-4-0/7519 https://discuss.linuxcontainers.org/t/running-virtual-machin... [8]: https://github.com/lxc/lxd/issues/6205 https://github.com/lxc/lxd/issues/6205 [9]: https://criu.org/Main_Page https://criu.org/Main_Page [10]: https://linuxcontainers.org/lxd/docs/master/storage https://linuxcontainers.org/lxd/docs/master/storage
shrubble 6y ago
This post is exceptionally well written. I have rarely seen technical explanations done so clearly.
- JMTQp8lwXL 6y ago
  I wish I had this skill. I'd probably be twice as valuable to my employer if I could communicate technical concepts with this level of clarity.
  - mrkurt 6y ago
    I am lucky enough to have read the drafts. These articles are a huge grind. Most of the skill seems to be putting the time in and having enough experience to go as deep as necessary.
- keyle 6y ago
  I agree, I'm not even that much into docker stuff, but I couldn't stop reading. It was a fun read: took me on a journey, I learnt stuff and I feel more intelligent now.
- richardfey 6y ago
  I did not like the rhythm. Lots of consecutive short phrases. What works well for HN comments doesn't everywhere else? :)
  - tptacek 6y ago
    It's funny, if you look at my drafts, most of what I do is just shortening sentences. (I don't think I'm a particularly good writer, and I'm fascinated by good writing, so this is an interesting comment, and I appreciate it).
    - sneak 6y ago
      You are a particularly good writer and reading your writing over the last ten years has made me a better writer thereby.
adamsvystun 6y ago
Fly.io has one of the most interesting infrastructure blogs out there. Every article is gold.
addisonj 6y ago
Pretty clever. It is pretty neat how much a small team can do by intelligently "remixing" the powerful primitives that now exist at the linux and VM/container layers. The cool bit is how many interesting things are here/arriving like bpf, io_uring, and wireguard in linux and with wasm and all of the interesting things it opens up with fast, sandboxed execution. I fully expect from some of these techs will really come innovation that will shake-up infrastructure even more than cloud and kubernetes already have. (Also, hi Kurt and team, cool stuff!)
- mrkurt 6y ago
  Well hello addison! Fancy meeting you here. It is very cool what's available to mashup these days. Kind of a lot to keep up with sometimes.
  - cpach 6y ago
    ”Kind of a lot to keep up with sometimes.“ Indeed! It’s striking what a difference there is compared to the 70s/80s when all tech in utility computing was proprietary and owned by IBM. These days companies like Amazon, Google and Facebook provide a lot of building blocks as FOSS libraries. The innovation rate is amazing.
pm90 6y ago
TLDR: they run containerd instead of docker daemon to deploy docker images to firecracker VMs. GCP has a somewhat similar “container on VM” product (though not on firecracker VMs, just GCE VMs) https://cloud.google.com/compute/docs/containers/deploying-containers#deploying_a_container_on_a_new_vm_instance https://cloud.google.com/compute/docs/containers/deploying-c....
- mrkurt 6y ago
  Specifically, containerd with the device mapper plugin!
lmm 6y ago
This is the opposite of what I was hoping for from the title, because I'm working in kind of the opposite environment: everyone is trusted, we don't need strong isolation, but multiplexing processes onto hardware is tricky (and currently handled mainly through ansible which feels pretty hacky). Is anyone doing orchestration without containers? I'm working on the JVM so we already have well-isolated single-file deployables, but orchestrating which of them get run on which server is still significantly tricky. I'd love to be able to use the orchestration part of something like Kubernetes without having to worry about all the extra complexity of docker/containers.
- mrkurt 6y ago
  Check out the Nomad exec driver. It can orchestrate processes outside of containers, and even knows how to pull builds from a URL. Nomad is great. https://www.nomadproject.io/docs/drivers/exec https://www.nomadproject.io/docs/drivers/exec
  - lmm 6y ago
    Sounds like that's oriented towards using a chroot and a cgroups namespace which is exactly the kind of thing I'm trying to avoid (given how it complicates debugging etc.). But maybe I can use their "java driver" and make sure nomad doesn't run as root?
    - mrkurt 6y ago
      I haven't tried the Java driver, but it's worth a shot. They also have "raw_exec" which might be helpful.
    - jen20 6y ago
      Look at raw-exec too - that will avoid cgroups and friends.
- dorianmonnier 6y ago
  HashiCorp Nomad maybe ? I don't use it but I know it can orchestrate applications of any type (not only containers) https://www.nomadproject.io https://www.nomadproject.io
BikiniPrince 6y ago
I need docker in docker. I’ve already virtualized at the bios and I’m running xen within xen. The only thing left is to run virtual box and then dosbox.
- gear54rus 6y ago
  https://hub.docker.com/_/docker https://hub.docker.com/_/docker thank me later
  - michaeldwan 6y ago
    Our remote builders are just fly apps based on this image with an auth proxy in front. Works great https://github.com/superfly/rchab https://github.com/superfly/rchab
- deleted 6y ago
  [deleted]
- etaioinshrdlu 6y ago
  https://www.nestybox.com/ https://www.nestybox.com/
  - vxNsr 6y ago
    That name makes you think it’s a satirical website, but it’s not, appears to be a real product with quite a few customers.
sandGorgon 6y ago
This is very cool ! There is also nerdctl (now a part of containerd community tools) - https://medium.com/nttlabs/nerdctl-359311b32d0e https://medium.com/nttlabs/nerdctl-359311b32d0e
Huggernaut 6y ago
Super great write up. This really took me back to my days of working on the container platform behind CloudFoundry. In particular, we also used to use loop devices [1] but with AUFS mounted on them. Later we moved over to BTRFS and then Overlay on XFS [2] to help with our unprivileged (security) story. Also, this was a great piece of technical writing. Thanks for sharing! 1: https://github.com/cloudfoundry-attic/garden-shed/blob/6c5b0a6dc87a61a2f1c1fd8de17ca24535761439/docker_drivers/aufs/loop.go#L18 https://github.com/cloudfoundry-attic/garden-shed/blob/6c5b0... 2: https://github.com/cloudfoundry/grootfs https://github.com/cloudfoundry/grootfs
Annatar 6y ago
All these contortions and complications, instead of just learning how to make pkgsrc packages, going to SmartOS and running in zones. Unbelievable, how needlessly complex this is. Awful.
- reducesuffering 6y ago
  Have you read HN's most famous comment about Dropbox? Do you know why it's famous? https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
  - mixmastamyk 6y ago
    Annatar's saying there's a solution out there with fewer moving parts than the industry is moving towards. Rather than the dropbox situtaion that was the opposite. I'm not that familiar with any of these, but SmartOS sounded cool when I looked it up.
  - Annatar 6y ago
    No, I haven't. My point on the other hand is that all that flapping in the fly.io article is too complex and completely unnecessary because that problem can be solved with a better, simpler solution which has existed far longer.
    - mrkurt 6y ago
      I also dislike docker. But it's a de facto standard. We have exactly one thing we want to "innovate": running app servers close to users. Which means we support what people already use. Postgres is great but I wouldn't choose to run it for fun. Our customers use Postgres. Node is not my favorite runtime, but we have good support for Node because it's popular. Devs are willing to package their apps as docker images. So we run Docker images. Your better, simpler solution would cause our company to fail. I'm very familiar with Joyent and SmartOS. They lost the container battle, despite the elegance.
      - Annatar 5y ago
awild 6y ago
This is really just a curious question, but I've been playing a lot with overlay Filesystems at work. Have you tried mounting the tars as overlays directly? (I'm not sure which driver, but to my knowledge podman does it this way) this way you wouldn't have to unpack the tarballs and also have an immutable base for the container.
- dmw_ng 6y ago
  Tarballs must be compressed in a very particular way to support random access, and even that only works with on-demand decompression using a slow compressor (zlib). In Fly's model this would probably also need to happen inside the container's VM. All together it might save a little one-time startup cost, at the expense of making every other operation on the filesystem much slower for the life of each container.
querez 6y ago
Everyone is praising how well written this is, but I personally am confused: what do you actually GAIN by running a docker container without docker? Is it faster, or more secure? Why?
- tptacek 6y ago
  It is comparably fast (in practical terms; "of the same general order of startup cost") and much more secure. I didn't want to belabor the security thing, because we wrote extensively about that in a previous (linked) blog post. https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/ You can skip to the grafs immediately before and after the string "If you’re running someone else’s applications, you should probably care a lot". I don't think you can reasonably host general-purpose applications on a multi-tenant basis on shared hardware using container systems (ie: using directly shared kernels), for reasons that post gets into. It's for the same reason that AWS wrote Firecracker to run Fargate, which is also a container hosting service. It's my post and I don't totally get the writing thing either. My M.O. with these posts: write it like it was an HN comment, and then edit the sentences to be shorter. I'm glad people like it, though.
  - deepstack 6y ago
    > AWS wrote Firecracker to run Fargate thank you for mention that. Which projects like Firecracker, it really doesn't make sense to have Docker.
    - krageon 6y ago
      Given that this directly precedes the sentence you rewrote: > I don't think you can reasonably host general-purpose applications on a multi-tenant basis on shared hardware using container systems I'm really not sure what point you're trying to make
      - dsr_ 6y ago
        I interpreted this as: "don't think you can reasonably host" => I encourage our competitors to do this "general-purpose applications on a multi-tenant basis" => any program a black hat who signed up with our automated system wants to run, next to all the other programs random people want to run "on shared hardware" => one lump of iron owned by Fly "using container systems" => containers vs VMs, in this case, and especially a VM manager that pays some attention to security.
Cu3PO42 6y ago
Similarly, I recently discovered it is possible to run Docker containers in LXC relatively easily using the provided "lxc" template. We use this to run a few pieces of software whose only supported method of distribution is Docker. But there's a number of things that don't quite work how we want it to out of the box, for example, our DNS isn't configured. It seems fly solves this with their custom init system, so I'm really excited to play around with that! Thank you for sharing!
- Havoc 6y ago
  Could you share a link for this docker containers as lxc? Thought that isn’t possible. Or is this only for ones following a specific pattern?
  - Cu3PO42 6y ago
    I would be happy to, but I don't really have one. I can give you a really quick rundown of how to do it, though. 1. Install jq, umoci and skopeo on your host. 2. Run "lxc-create -n <name> -t oci -f <temp-config> -- -u docker://<link to container on registry> --no-cache" 3. Start the resulting container with "lxc-execute -d" rather than "lxc-start". The <temp-config> file should contain a uid and gid mapping for LXC. If you don't intend to run the containers rootless, you can omit it and "--no-cache" (the caching mechanism is broken for rootless containers). After container creation we patch the generated config file with our network configuration, etc. similarly to what we would do with a normal LXC container. This has worked with all containers we tried it with, but if it relies on interacting with the Docker domain directly via docker.sock it obviously won't. A colleague has had success adding the init system from a container (in that case tini) and changing the confict so the container can start with lxc-start, this was a much more manual process, though.
    - m463 6y ago
      This is really interesting. I wish I could upvote you more than once. I use proxmox with lxc, but there's nothing like the simplicity of a Dockerfile I think it would be nice to have say "dockerpull2lxc" or even better "dockerbuild2lxc".
lazyweb 6y ago
Nice writeup, but another detail I'm very positively surprised about - no external resources, no 3rd party JS libraries or external tracking, not even on their landing page. This is an exceedingly rare thing these days!
- mrkurt 6y ago
  I'm glad you noticed. It sure makes our lives simpler.
dmw_ng 6y ago
If you haven't tried Fly yet, where have you been? - Anycast IPv4 to VMs hosted near the edge - Raw TCP/UDP servers or external HTTP/HTTPS termination - $.02/GB egress, no per-request pricing You can build almost anything with this model (VoIP, video, gaming, Heroku-like, App Engine-like, Lambda-like, ..), and the bandwidth pricing is astounding. Fly have an obviously bright future, they aren't even on the same page as existing CDNs.
- Havoc 6y ago
  Wow. That summarises the benefits well. I shall have a look!
- tomjen3 6y ago
  I can't quite figure it if you pay the bandwidth for where your servers are, or where the people are (the first would make the most sense to me) and Indian trafic is very expensive by their measure. But do not that you get a lot of free bandwidth to play with.
  - dmw_ng 6y ago
    > Inflicting that complexity on you all would help with margin control, but ugh. What we've done instead is set a blended price that fits most apps running on Fly.io, and decided to just eat the extra cost from outliers. If you want to exploit that, run an app in Sydney with a whole bunch of users in India. We'll lose money on your app and you will win one round of capitalism. https://fly.io/blog/we-cut-bandwidth-prices-go-nuts/ https://fly.io/blog/we-cut-bandwidth-prices-go-nuts/
    - everybodyknows 6y ago
      Nice, but still one worry: A bug, or an attacker, sends outbound traffic through the roof, and newbie's fun experiment ends in a huge bill. More than one such story on HN recently (AWS). Skimming the pricing docs just now, I don't see protection against bandwidth overage: https://fly.io/docs/about/ https://fly.io/docs/about/
debarshri 6y ago
I have been following microvms and kubernetes lately. Nice thing about leveraging OCI is that you can use cri-o and orchestrate using kubernetes. I am pretty bullish on microvms. Weavework has been doing quite some good stuff around OCI interface around microvms. Ignite[1] which is more robust and productize version of what is mentioned in this blog. Other projects that should be watched out for is kata containers [2] [1] https://github.com/weaveworks/ignite https://github.com/weaveworks/ignite [2] https://gokulchandrapr.medium.com/kata-containers-on-kubernetes-and-kata-firecracker-vmm-support-28abb3a196e7 https://gokulchandrapr.medium.com/kata-containers-on-kuberne...
wiradikusuma 6y ago
I'm more interested with the main product, Fly. Is this like Linode or Digital Ocean?
- shoo_pl 6y ago
  Haven't heard about them, but its more like Heroku - not pure VMs but instead IaaS where you deploy apps using CLI and they manage/scale machines for you.
- mrkurt 6y ago
  It's somewhere between Heroku and DigitalOcean. We have a CLI for managing and deploying apps that's very similar to Heroku. But apps also get private networking, disks, load balancing for TCP/UDP. We do have people deploy single instance apps and use them just like you'd use a Droplet. "fly ssh console" will SSH you directly into them, and with a persistent volume you can do all kinds of fun stuff.
rad_gruchalski 6y ago
Fly is really nice. I’ve been getting my head around microvms recently to build some infrastructures around them. Even started building some tools to rebuild them directly from Dockerfiles and Docker images. If anybody is interested: https://github.com/combust-labs/firebuild https://github.com/combust-labs/firebuild It’s very early stages so many things could be improved but things are working.
- iampims 6y ago
  This is fantastic. Thanks for sharing!
  - rad_gruchalski 6y ago
    Happy to hear that you like it!
kgraves 6y ago
Just curious who uses Fly.io in production? This looks very impressive but i'm scared to use this instead of something battle tested like Heroku or App Engine. Is the experience similar?
- jkarneges 6y ago
  We use Fly to run our website & control panel, that was previously hosted on Heroku. Not a big workload but we were able to move it regionally closer to our backend services to improve performance. The deployment experience is just as easy as Heroku. Perhaps easier.
- austinpena 5y ago
  I’ve used fly for a while for my SaaS. Very little down time and very robust.
samuell 6y ago
A week ago we heard about Simplenetes [1] (in 17k lines of Bash) and now this. Seems like people are getting tired of some accidental complexity :) [1] https://news.ycombinator.com/item?id=26661223 https://news.ycombinator.com/item?id=26661223
- 1vuio0pswjnm7 6y ago
  Accidental? I reckon those responsible are likely incapable of avoiding unnecessary complexity. Further, I think any talk of "simplicity" may make them uncomfortable.
  - Niksko 6y ago
    Distributed systems at scale are hard. These technologies are coming out of the largest, most successful companies on the plant. To assume that things are complex because of some naivete or stupidity (rather than because they solve complex problems) is hubris in the highest.
    - mrkurt 6y ago
      It's definitely not naivete or stupidity, but projects from large companies inherit all the complexity of the organization that created them. This is sometimes because they're very broadly scoped, and sometimes because large groups of people build complex things by default. Simplicity is important for people who are in smaller groups. This is why many people who dislike kubernetes like Fly.io, and people who love kubernetes think we're building a toy.
      - Niksko 6y ago
        Conway's law, totally fair, it's inescapable
    - brandmeyer 6y ago
      On the contrary, I am convinced that the solutions produced by large successful companies are complex because of the size and wealth of the employer.
scrollaway 6y ago
Article aside, EVERYONE with a marketing/technical blog should have an intro like fly.io does: a one paragraph explanation of what the service does. It gives context for the article and immediately creates the possibility of converting people off nothing but the article, right off the gate. Blows my mind that it's one of the first times I see this done.
- hans_castorp 6y ago
  > EVERYONE with a marketing/technical blog should have an intro like fly.io does: a one paragraph explanation of what the service does. And do away with the stupid and annoying teaser pictures that force me to scroll down on a 27" monitor to even see some text.
  - signal11 6y ago
    > the stupid and annoying teaser pictures that force me to scroll down on a 27" monitor to even see some text I think they're called hero images[1]. I think it's okay to have them, but scaling them up with monitor size is a web-design pattern that needs to die IMHO. [1] https://en.wikipedia.org/wiki/Hero_image https://en.wikipedia.org/wiki/Hero_image
- kgraves 6y ago
  Didn't appreciate the advertising, I found the article a bit hard to read, wanted to know who is using this in prod, but found nothing.
  - cpach 6y ago
    Not all companies advertise who their customers are.
    - kgraves 6y ago
      Sounds like a huge red flag to me then, and loses my confidence that this service is reliable, plus it makes it a hard decision for most (even me) to go with them. It also signals to me it is not battle tested and results in lost sales. I'll find someone else.
anotherhue 6y ago
@tptacek: Was there any exploration of Gvisor before landing on the current implementation?
- hollerith 6y ago
  He wrote about gvisor here (July 2020): https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/ >As batshit as this plan is, it works surprisingly well; you can build gVisor and runsc, its container runtime, relatively easily. Once you have runsc installed, it will run Docker containers for you. After reading the code, I sort of couldn’t believe it was working as well as it did, or, if it was, that it was actually using the code I had read. But I scattered a bunch of panic calls across the codebase and, yup, that all that stuff is actually happening. It’s pretty amazing. >You are probably strictly better off with gVisor than you are with a tuned Docker configuration, and I like it a lot. The big downside is performance; you’ll be looking at a low-double-digits percentage hit, degrading with I/O load. Google runs this stuff at scale in GCE; you can probably get away with it too. If you’re running gVisor, you should brag about it, because, again, gVisor is pretty bananas.
  - anotherhue 6y ago
    Thank you!
nextaccountic 6y ago
Here's a question about your architecture: why do you store layers in LVM2 block-level snapshots instead of btrfs or zfs file-level snapshots?
- mrkurt 6y ago
  The simple answer is: because containerd + devicemapper makes it easy and we already use LVM. We use LVM so we can expose raw block devices to peoples' applications.
- deleted 6y ago
  [deleted]
chrisweekly 6y ago
Fantastic writeup: technical details, context and framing, all in clear, informal, plain-English prose. It's profoundly effective, and vanishingly rare. As for Fly's solution per se: like many great ideas it seems simple bordering on obvious, in retrospect. I'm impressed and more interested than ever in bringing fly.io into my wheelhouse. Final thoughts: CLI-first: flyctl reminds me of `vercel` (fka Zeit `now`), which is a big compliment > "VM build-and-boot process on a second deployment is faster than the logging that we do" ... ie, well-suited for lambdas / serverless functions at CDN edge -- which is of extreme relevance and interest to me and everyone else working on minimizing latency with modern webapp architecture... exciting times!
undecisive 6y ago
So I've had a question running around my brain for a while. Has anyone tried using docker-compose files to deploy direct to separate cloud machines? By which I mean, given some config with user creds to AWS or Digital Ocean or Hetzner or whatever, running: dockerless-compose some.address.com up could spin up servers (ec2 instances, droplets, maybe some kind of hinting in the docker-compose.yml file to show sizes), maybe set up a virtual network if the hosting provider supports it, set network aliases in the /etc/hosts, set some.address.com to be the gatekeeper (the system would need to install SSH on your boxes, but it means that it could set up an ssh authorized key from the gatekeeper box) And then, maybe it doesn't need to be actual cloud servers. It could be bare metal. It could be chroots. It could even be a single host with docker (a bit pointless, but maybe useful for proving parity with docker-compose) Essentially, suddenly we have a replacement for the likes of terraform in many organisations. Your local developer setup becomes exactly the same as your production setup. Am I missing something? Is that where we're headed with this, or would this be a complete nightmare?
- acdha 6y ago
  AWS has been moving in that direction: https://aws.amazon.com/blogs/containers/deploy-applications-on-amazon-ecs-using-docker-compose/ https://aws.amazon.com/blogs/containers/deploy-applications-... (See also https://aws.amazon.com/containers/copilot/ https://aws.amazon.com/containers/copilot/) The catch is that this doesn’t replace a tool like Terraform unless your projects are extremely simple and you don’t have much in the way of security requirements. Once you need anything more, Compose doesn’t have the information you’d need and you’d lose the veneer of simplicity and portability trying to add it in.
- jonesetc 6y ago
  I believe this was docker's (the company) direction for a while, but it was spread across multiple tools. docker-machine does provisioning and docker-swarm runs the compose file on remote hosts.
- e12e 6y ago
  Sounds like docker swarm is 80% of what you want. Other than that, I think there's a reason why things like lxd, k8s (maybe joyent/smartos belong in this list) - generally have one layer for the actual hw, and a higher level abstraction on top. Otoh, I suppose something like canonical metal-as-a-service might be able to do more with a little help. In fact I think maas+lxd is probably pretty close to being able to work with a compose-like tool and docker containers.
nimbius 6y ago
we already have this, its called podman. https://podman.io https://podman.io
- tptacek 6y ago
  Podman is very, very cool, but it's not quite what we're doing.
- cpach 6y ago
  AFAIK, Podman still uses cgroups to separate the containers. Fly uses Firecracker to separate containers. Those are two quite different methods and AFAICT Fly uses Firecracker because of the stronger security model of that solution.
breatheoften 6y ago
Who are the people behind fly? It looks to me like the first thing I've seen in awhile that actually might be "a better heroku". Anybody out there using it? Anybody who was happy with heroku for mostly everything other than price that migrated to fly and remained happy?
- austinpena 5y ago
  Never used heroku but I love fly
pkulak 6y ago
I've gotten a bit sick of running Docker on my dev systems and switched to Podman. I found a "podman-compose" script that works great for setting up my dev environments just like I'm used to, and I can run everything as my own, non-root user. When my laptop inevitably starts running out of space, there's a single directory of images in my home directory I can rm -rf to start over. It's been great.
hinkley 6y ago
And once again, the promise of multitenant processing on the same machine gets kicked down the road ten years. I wonder if my grandkids will have the things the OS vendors were promising us back in the early 90's.
echelon 6y ago
"Docker without Docker" had me thinking about the venture-funded company behind the Docker runtime. $330M with no sustainable path to revenue. Everyone ate up the open source pieces and there's nothing left to monetize. Docker without Docker.
mwcampbell 6y ago
Just curious, what kind of host OS is Fly running for the machines hosting the VMs? A custom immutable image built with something like LinuxKit? A stock OS designed to be a container host, like Flatcar Linux? Or a conventional distro?
- tptacek 6y ago
  Conventional distro, very slightly fussy kernel because of the BPF stuff.
replwoacause 6y ago
Love the illustrations on the site!

689 points by kwindla 6y ago

fcatus 6y ago

you cant even try it without giving them your credit card info.. Looked interesting but nah...

kondro 6y ago
They have a pretty generous free tier though: https://fly.io/docs/about/pricing/ https://fly.io/docs/about/pricing/
tptacek 6y ago
We're a hosting provider. Later The parent comment is getting downvoted into oblivion (because, again, we're a hosting provider) but it's an honest observation! It just happens to have a simple answer.
xfer 6y ago
Otherwise it will be used for crypto mining.
- richardfey 6y ago
  It will be used for crypto mining regardless, using stolen credit cards.
  - tptacek 6y ago
    Yup. But without the credit cards, it will be used only for crypto mining. :)

kondro 6y ago

Fly are very generous with what they're willing to share with the world.

tptacek 6y ago
We love you all so much.
- atonse 6y ago
  Honestly the more I read these posts, the more I want to use the product. :) (Full-time elixir dev running a SaaS and 8+ projects on elixir, including covid vaccine scheduling system that handles thousands of users with no issues. LOVE elixir.)
  - tptacek 6y ago
    WE LOVE ELIXIR. I don't even write Elixir and I love it, because we built a system that makes it super easy to take a Dockerfile of some random thing, and deploy 20 instances of it spread evenly around some bunch of regions that can talk privately to each other by default. That's a fun thing to be able to do in any language, but to exploit it in a Go program, I'd have to actually think, pull something like Serf in, whatever. But Phoenix LiveView apps literally ship with a dashboard where you can see all your instances running, and then first-class features for them to chat between themselves. LiveView is cool all on its own, but Elixir+Phoenix+LiveView is such a sweet application of a hosting environment like ours. Expect us to be weirdly chatty about Elixir in the coming months.
    - therein 6y ago
      Looking forward to it.
    - cpursley 6y ago
      I'm looking forward to this. Is it going to be possible to run local Postgres (or one of the distributed dbs that are PG compatable) read instances alongside the Elixir instances?
      - mrkurt 6y ago
        Yes. Our beta postgres service lets you add read only replicas in any region you want. "Deploy any full stack app globally" has been a big goal for us over the last year. We're getting close.

coding123 6y ago

transmogrify = convert

jmholla 6y ago
Yup. It's a reference to Calvin and Hobbes: http://1.bp.blogspot.com/-DXzlDXdGjlM/UGSlryq02vI/AAAAAAAAYdc/OJo1GDMOZRU/s1600/transmogrifier.png http://1.bp.blogspot.com/-DXzlDXdGjlM/UGSlryq02vI/AAAAAAAAYd...
- lloeki 6y ago
  Is it? I remember reading the word from sources that predate C&H by a lot (early 20th c.). Doing my homework, a quick search shows uses as far back as 17th c. (which I din't quite expect!) https://www.merriam-webster.com/dictionary/transmogrify https://www.merriam-webster.com/dictionary/transmogrify

deleted 6y ago

[deleted]

nunez 6y ago

Actually a Dockerfile parser that can spin up Firecracker VMs and handle volume/net_iface binding transparently would be really neat.

tptacek 6y ago
Our local dev environment does this, but it does it by running our driver (so we can test the driver) so it's a lot more mechanism than you'd need. You could get this done yourself pretty easily: just unpack the Docker image (there's a Docker command that does this, I forget what) onto a mounted loop device, then hand its block device off to Firecracker. Our driver does its own local address goop, but it's not complicated (it just dumps netlink and allocates the next available subnet in a particular range). Having said all that: there are projects that do this already; for instance, I think there's a Kata Firecracker somewhere.
rad_gruchalski 6y ago
I am working on something like this: https://github.com/combust-labs/firebuild https://github.com/combust-labs/firebuild. From a Dockerfile, it's not as simple without creating an image first. The command @tptacek means is most likely docker export for a running container and docker save for an image. First exports selected paths, second exports a complete image as a tgz archive.

mperham 6y ago

This is really cool infrastructure work and impressively open.

mrkurt 6y ago
We're happy Sidekiq Pro customers too.

hardwaresofton 6y ago

I'm really impressed by fly.io, and the candidness with which they share some of their really awesome technology. Being container-first is the next step for PaaS IMO and they are ahead of the pack. I aim to build a platform like theirs someday (probably not any time soon) but I don't think I'd do any of what they're doing -- it feels unnecessary, because I feel like I know lots of projects that have actually already done the hard work for this. I probably only think this because I haven't actually done it, but think it's worth sharing the tech anyway. Bear with me as I recently learned that they use nomad[0] and some of these suggestions are kubernetes projects but I'd love to hear why the following technologies were decided against (if they were): - kata-containers[1] (it does the whole container -> VM flow for you, automatically) with multiple VMM options[2] - linuxkit[3] (let's say you didn't go with kata-containers, this is another container->VM path) - firecracker-containerd[4] (very minimal keep-your-container-but-run-it-as-a-VM) - kubevirt[5] (if you just want to actually run VMs, regardless of how you built them) - Ceph[6] for storage -- make LVM pools and just give them to Ceph, you'll get blocks, distributed filesystems (CephFS), and object gateways (S3/Swift) out of it (in the k8s space Rook manages this) As an aside to all this, there's also LXD, which supports running "system" (user namespace isolated) containers, VMs (somewhat recent[7][8]), live migration via criu[9], management/migration of underlying filesystems, runs on LVM or zfs[10], it's basically all-in-one, but does fall behind in terms of ecosystem since everyone else is aboard the "cloud native"/"works-with-kubernetes" train. I've basically how I plan to run a service like fly.io if I ever did -- so maybe my secret is out, but I sure would like to know just how much of this fly.io got built on (if any of it), and/or what was turned down. [0]: https://news.ycombinator.com/item?id=26745514 https://news.ycombinator.com/item?id=26745514 [1]: https://github.com/kata-containers/kata-containers https://github.com/kata-containers/kata-containers [2]: https://github.com/kata-containers/kata-containers/blob/2fc7f75724ac9e18e60f63dcc9aa395dc51c184d/docs/design/virtualization.md https://github.com/kata-containers/kata-containers/blob/2fc7... [3]: https://github.com/linuxkit/linuxkit https://github.com/linuxkit/linuxkit [4]: https://github.com/firecracker-microvm/firecracker-containerd/ https://github.com/firecracker-microvm/firecracker-container... [5]: https://github.com/kubevirt/kubevirt https://github.com/kubevirt/kubevirt [6]: https://docs.ceph.com/ https://docs.ceph.com/ [7]: https://discuss.linuxcontainers.org/t/running-virtual-machines-with-lxd-4-0/7519 https://discuss.linuxcontainers.org/t/running-virtual-machin... [8]: https://github.com/lxc/lxd/issues/6205 https://github.com/lxc/lxd/issues/6205 [9]: https://criu.org/Main_Page https://criu.org/Main_Page [10]: https://linuxcontainers.org/lxd/docs/master/storage https://linuxcontainers.org/lxd/docs/master/storage

shrubble 6y ago

This post is exceptionally well written. I have rarely seen technical explanations done so clearly.

JMTQp8lwXL 6y ago
I wish I had this skill. I'd probably be twice as valuable to my employer if I could communicate technical concepts with this level of clarity.
- mrkurt 6y ago
  I am lucky enough to have read the drafts. These articles are a huge grind. Most of the skill seems to be putting the time in and having enough experience to go as deep as necessary.
keyle 6y ago
I agree, I'm not even that much into docker stuff, but I couldn't stop reading. It was a fun read: took me on a journey, I learnt stuff and I feel more intelligent now.
richardfey 6y ago
I did not like the rhythm. Lots of consecutive short phrases. What works well for HN comments doesn't everywhere else? :)
- tptacek 6y ago
  It's funny, if you look at my drafts, most of what I do is just shortening sentences. (I don't think I'm a particularly good writer, and I'm fascinated by good writing, so this is an interesting comment, and I appreciate it).
  - sneak 6y ago
    You are a particularly good writer and reading your writing over the last ten years has made me a better writer thereby.

adamsvystun 6y ago

Fly.io has one of the most interesting infrastructure blogs out there. Every article is gold.

addisonj 6y ago

Pretty clever. It is pretty neat how much a small team can do by intelligently "remixing" the powerful primitives that now exist at the linux and VM/container layers. The cool bit is how many interesting things are here/arriving like bpf, io_uring, and wireguard in linux and with wasm and all of the interesting things it opens up with fast, sandboxed execution. I fully expect from some of these techs will really come innovation that will shake-up infrastructure even more than cloud and kubernetes already have. (Also, hi Kurt and team, cool stuff!)

mrkurt 6y ago
Well hello addison! Fancy meeting you here. It is very cool what's available to mashup these days. Kind of a lot to keep up with sometimes.
- cpach 6y ago
  ”Kind of a lot to keep up with sometimes.“ Indeed! It’s striking what a difference there is compared to the 70s/80s when all tech in utility computing was proprietary and owned by IBM. These days companies like Amazon, Google and Facebook provide a lot of building blocks as FOSS libraries. The innovation rate is amazing.

pm90 6y ago

TLDR: they run containerd instead of docker daemon to deploy docker images to firecracker VMs. GCP has a somewhat similar “container on VM” product (though not on firecracker VMs, just GCE VMs) https://cloud.google.com/compute/docs/containers/deploying-containers#deploying_a_container_on_a_new_vm_instance https://cloud.google.com/compute/docs/containers/deploying-c....

mrkurt 6y ago
Specifically, containerd with the device mapper plugin!

lmm 6y ago

This is the opposite of what I was hoping for from the title, because I'm working in kind of the opposite environment: everyone is trusted, we don't need strong isolation, but multiplexing processes onto hardware is tricky (and currently handled mainly through ansible which feels pretty hacky). Is anyone doing orchestration without containers? I'm working on the JVM so we already have well-isolated single-file deployables, but orchestrating which of them get run on which server is still significantly tricky. I'd love to be able to use the orchestration part of something like Kubernetes without having to worry about all the extra complexity of docker/containers.

mrkurt 6y ago
Check out the Nomad exec driver. It can orchestrate processes outside of containers, and even knows how to pull builds from a URL. Nomad is great. https://www.nomadproject.io/docs/drivers/exec https://www.nomadproject.io/docs/drivers/exec
- lmm 6y ago
  Sounds like that's oriented towards using a chroot and a cgroups namespace which is exactly the kind of thing I'm trying to avoid (given how it complicates debugging etc.). But maybe I can use their "java driver" and make sure nomad doesn't run as root?
  - mrkurt 6y ago
    I haven't tried the Java driver, but it's worth a shot. They also have "raw_exec" which might be helpful.
  - jen20 6y ago
    Look at raw-exec too - that will avoid cgroups and friends.
dorianmonnier 6y ago
HashiCorp Nomad maybe ? I don't use it but I know it can orchestrate applications of any type (not only containers) https://www.nomadproject.io https://www.nomadproject.io

BikiniPrince 6y ago

I need docker in docker. I’ve already virtualized at the bios and I’m running xen within xen. The only thing left is to run virtual box and then dosbox.

gear54rus 6y ago
https://hub.docker.com/_/docker https://hub.docker.com/_/docker thank me later
- michaeldwan 6y ago
  Our remote builders are just fly apps based on this image with an auth proxy in front. Works great https://github.com/superfly/rchab https://github.com/superfly/rchab
deleted 6y ago
[deleted]
etaioinshrdlu 6y ago
https://www.nestybox.com/ https://www.nestybox.com/
- vxNsr 6y ago
  That name makes you think it’s a satirical website, but it’s not, appears to be a real product with quite a few customers.

sandGorgon 6y ago

This is very cool ! There is also nerdctl (now a part of containerd community tools) - https://medium.com/nttlabs/nerdctl-359311b32d0e https://medium.com/nttlabs/nerdctl-359311b32d0e

Huggernaut 6y ago

Super great write up. This really took me back to my days of working on the container platform behind CloudFoundry. In particular, we also used to use loop devices [1] but with AUFS mounted on them. Later we moved over to BTRFS and then Overlay on XFS [2] to help with our unprivileged (security) story. Also, this was a great piece of technical writing. Thanks for sharing! 1: https://github.com/cloudfoundry-attic/garden-shed/blob/6c5b0a6dc87a61a2f1c1fd8de17ca24535761439/docker_drivers/aufs/loop.go#L18 https://github.com/cloudfoundry-attic/garden-shed/blob/6c5b0... 2: https://github.com/cloudfoundry/grootfs https://github.com/cloudfoundry/grootfs

Annatar 6y ago

All these contortions and complications, instead of just learning how to make pkgsrc packages, going to SmartOS and running in zones. Unbelievable, how needlessly complex this is. Awful.

reducesuffering 6y ago
Have you read HN's most famous comment about Dropbox? Do you know why it's famous? https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
- mixmastamyk 6y ago
  Annatar's saying there's a solution out there with fewer moving parts than the industry is moving towards. Rather than the dropbox situtaion that was the opposite. I'm not that familiar with any of these, but SmartOS sounded cool when I looked it up.
- Annatar 6y ago
  No, I haven't. My point on the other hand is that all that flapping in the fly.io article is too complex and completely unnecessary because that problem can be solved with a better, simpler solution which has existed far longer.
  - mrkurt 6y ago
    I also dislike docker. But it's a de facto standard. We have exactly one thing we want to "innovate": running app servers close to users. Which means we support what people already use. Postgres is great but I wouldn't choose to run it for fun. Our customers use Postgres. Node is not my favorite runtime, but we have good support for Node because it's popular. Devs are willing to package their apps as docker images. So we run Docker images. Your better, simpler solution would cause our company to fail. I'm very familiar with Joyent and SmartOS. They lost the container battle, despite the elegance.
    - Annatar 5y ago

awild 6y ago

This is really just a curious question, but I've been playing a lot with overlay Filesystems at work. Have you tried mounting the tars as overlays directly? (I'm not sure which driver, but to my knowledge podman does it this way) this way you wouldn't have to unpack the tarballs and also have an immutable base for the container.

dmw_ng 6y ago
Tarballs must be compressed in a very particular way to support random access, and even that only works with on-demand decompression using a slow compressor (zlib). In Fly's model this would probably also need to happen inside the container's VM. All together it might save a little one-time startup cost, at the expense of making every other operation on the filesystem much slower for the life of each container.

querez 6y ago

Everyone is praising how well written this is, but I personally am confused: what do you actually GAIN by running a docker container without docker? Is it faster, or more secure? Why?

tptacek 6y ago
It is comparably fast (in practical terms; "of the same general order of startup cost") and much more secure. I didn't want to belabor the security thing, because we wrote extensively about that in a previous (linked) blog post. https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/ You can skip to the grafs immediately before and after the string "If you’re running someone else’s applications, you should probably care a lot". I don't think you can reasonably host general-purpose applications on a multi-tenant basis on shared hardware using container systems (ie: using directly shared kernels), for reasons that post gets into. It's for the same reason that AWS wrote Firecracker to run Fargate, which is also a container hosting service. It's my post and I don't totally get the writing thing either. My M.O. with these posts: write it like it was an HN comment, and then edit the sentences to be shorter. I'm glad people like it, though.
- deepstack 6y ago
  > AWS wrote Firecracker to run Fargate thank you for mention that. Which projects like Firecracker, it really doesn't make sense to have Docker.
  - krageon 6y ago
    Given that this directly precedes the sentence you rewrote: > I don't think you can reasonably host general-purpose applications on a multi-tenant basis on shared hardware using container systems I'm really not sure what point you're trying to make
    - dsr_ 6y ago
      I interpreted this as: "don't think you can reasonably host" => I encourage our competitors to do this "general-purpose applications on a multi-tenant basis" => any program a black hat who signed up with our automated system wants to run, next to all the other programs random people want to run "on shared hardware" => one lump of iron owned by Fly "using container systems" => containers vs VMs, in this case, and especially a VM manager that pays some attention to security.

Cu3PO42 6y ago

Similarly, I recently discovered it is possible to run Docker containers in LXC relatively easily using the provided "lxc" template. We use this to run a few pieces of software whose only supported method of distribution is Docker. But there's a number of things that don't quite work how we want it to out of the box, for example, our DNS isn't configured. It seems fly solves this with their custom init system, so I'm really excited to play around with that! Thank you for sharing!

Havoc 6y ago
Could you share a link for this docker containers as lxc? Thought that isn’t possible. Or is this only for ones following a specific pattern?
- Cu3PO42 6y ago
  I would be happy to, but I don't really have one. I can give you a really quick rundown of how to do it, though. 1. Install jq, umoci and skopeo on your host. 2. Run "lxc-create -n <name> -t oci -f <temp-config> -- -u docker://<link to container on registry> --no-cache" 3. Start the resulting container with "lxc-execute -d" rather than "lxc-start". The <temp-config> file should contain a uid and gid mapping for LXC. If you don't intend to run the containers rootless, you can omit it and "--no-cache" (the caching mechanism is broken for rootless containers). After container creation we patch the generated config file with our network configuration, etc. similarly to what we would do with a normal LXC container. This has worked with all containers we tried it with, but if it relies on interacting with the Docker domain directly via docker.sock it obviously won't. A colleague has had success adding the init system from a container (in that case tini) and changing the confict so the container can start with lxc-start, this was a much more manual process, though.
  - m463 6y ago
    This is really interesting. I wish I could upvote you more than once. I use proxmox with lxc, but there's nothing like the simplicity of a Dockerfile I think it would be nice to have say "dockerpull2lxc" or even better "dockerbuild2lxc".

lazyweb 6y ago

Nice writeup, but another detail I'm very positively surprised about - no external resources, no 3rd party JS libraries or external tracking, not even on their landing page. This is an exceedingly rare thing these days!

mrkurt 6y ago
I'm glad you noticed. It sure makes our lives simpler.

dmw_ng 6y ago

If you haven't tried Fly yet, where have you been? - Anycast IPv4 to VMs hosted near the edge - Raw TCP/UDP servers or external HTTP/HTTPS termination - $.02/GB egress, no per-request pricing You can build almost anything with this model (VoIP, video, gaming, Heroku-like, App Engine-like, Lambda-like, ..), and the bandwidth pricing is astounding. Fly have an obviously bright future, they aren't even on the same page as existing CDNs.

Havoc 6y ago
Wow. That summarises the benefits well. I shall have a look!
tomjen3 6y ago
I can't quite figure it if you pay the bandwidth for where your servers are, or where the people are (the first would make the most sense to me) and Indian trafic is very expensive by their measure. But do not that you get a lot of free bandwidth to play with.
- dmw_ng 6y ago
  > Inflicting that complexity on you all would help with margin control, but ugh. What we've done instead is set a blended price that fits most apps running on Fly.io, and decided to just eat the extra cost from outliers. If you want to exploit that, run an app in Sydney with a whole bunch of users in India. We'll lose money on your app and you will win one round of capitalism. https://fly.io/blog/we-cut-bandwidth-prices-go-nuts/ https://fly.io/blog/we-cut-bandwidth-prices-go-nuts/
  - everybodyknows 6y ago
    Nice, but still one worry: A bug, or an attacker, sends outbound traffic through the roof, and newbie's fun experiment ends in a huge bill. More than one such story on HN recently (AWS). Skimming the pricing docs just now, I don't see protection against bandwidth overage: https://fly.io/docs/about/ https://fly.io/docs/about/

debarshri 6y ago

I have been following microvms and kubernetes lately. Nice thing about leveraging OCI is that you can use cri-o and orchestrate using kubernetes. I am pretty bullish on microvms. Weavework has been doing quite some good stuff around OCI interface around microvms. Ignite[1] which is more robust and productize version of what is mentioned in this blog. Other projects that should be watched out for is kata containers [2] [1] https://github.com/weaveworks/ignite https://github.com/weaveworks/ignite [2] https://gokulchandrapr.medium.com/kata-containers-on-kubernetes-and-kata-firecracker-vmm-support-28abb3a196e7 https://gokulchandrapr.medium.com/kata-containers-on-kuberne...

wiradikusuma 6y ago

I'm more interested with the main product, Fly. Is this like Linode or Digital Ocean?

shoo_pl 6y ago
Haven't heard about them, but its more like Heroku - not pure VMs but instead IaaS where you deploy apps using CLI and they manage/scale machines for you.
mrkurt 6y ago
It's somewhere between Heroku and DigitalOcean. We have a CLI for managing and deploying apps that's very similar to Heroku. But apps also get private networking, disks, load balancing for TCP/UDP. We do have people deploy single instance apps and use them just like you'd use a Droplet. "fly ssh console" will SSH you directly into them, and with a persistent volume you can do all kinds of fun stuff.

rad_gruchalski 6y ago

Fly is really nice. I’ve been getting my head around microvms recently to build some infrastructures around them. Even started building some tools to rebuild them directly from Dockerfiles and Docker images. If anybody is interested: https://github.com/combust-labs/firebuild https://github.com/combust-labs/firebuild It’s very early stages so many things could be improved but things are working.

iampims 6y ago
This is fantastic. Thanks for sharing!
- rad_gruchalski 6y ago
  Happy to hear that you like it!

kgraves 6y ago

Just curious who uses Fly.io in production? This looks very impressive but i'm scared to use this instead of something battle tested like Heroku or App Engine. Is the experience similar?

jkarneges 6y ago
We use Fly to run our website & control panel, that was previously hosted on Heroku. Not a big workload but we were able to move it regionally closer to our backend services to improve performance. The deployment experience is just as easy as Heroku. Perhaps easier.
austinpena 5y ago
I’ve used fly for a while for my SaaS. Very little down time and very robust.

samuell 6y ago

A week ago we heard about Simplenetes [1] (in 17k lines of Bash) and now this. Seems like people are getting tired of some accidental complexity :) [1] https://news.ycombinator.com/item?id=26661223 https://news.ycombinator.com/item?id=26661223

1vuio0pswjnm7 6y ago
Accidental? I reckon those responsible are likely incapable of avoiding unnecessary complexity. Further, I think any talk of "simplicity" may make them uncomfortable.
- Niksko 6y ago
  Distributed systems at scale are hard. These technologies are coming out of the largest, most successful companies on the plant. To assume that things are complex because of some naivete or stupidity (rather than because they solve complex problems) is hubris in the highest.
  - mrkurt 6y ago
    It's definitely not naivete or stupidity, but projects from large companies inherit all the complexity of the organization that created them. This is sometimes because they're very broadly scoped, and sometimes because large groups of people build complex things by default. Simplicity is important for people who are in smaller groups. This is why many people who dislike kubernetes like Fly.io, and people who love kubernetes think we're building a toy.
    - Niksko 6y ago
      Conway's law, totally fair, it's inescapable
  - brandmeyer 6y ago
    On the contrary, I am convinced that the solutions produced by large successful companies are complex because of the size and wealth of the employer.

scrollaway 6y ago

Article aside, EVERYONE with a marketing/technical blog should have an intro like fly.io does: a one paragraph explanation of what the service does. It gives context for the article and immediately creates the possibility of converting people off nothing but the article, right off the gate. Blows my mind that it's one of the first times I see this done.

hans_castorp 6y ago
> EVERYONE with a marketing/technical blog should have an intro like fly.io does: a one paragraph explanation of what the service does. And do away with the stupid and annoying teaser pictures that force me to scroll down on a 27" monitor to even see some text.
- signal11 6y ago
  > the stupid and annoying teaser pictures that force me to scroll down on a 27" monitor to even see some text I think they're called hero images[1]. I think it's okay to have them, but scaling them up with monitor size is a web-design pattern that needs to die IMHO. [1] https://en.wikipedia.org/wiki/Hero_image https://en.wikipedia.org/wiki/Hero_image
kgraves 6y ago
Didn't appreciate the advertising, I found the article a bit hard to read, wanted to know who is using this in prod, but found nothing.
- cpach 6y ago
  Not all companies advertise who their customers are.
  - kgraves 6y ago
    Sounds like a huge red flag to me then, and loses my confidence that this service is reliable, plus it makes it a hard decision for most (even me) to go with them. It also signals to me it is not battle tested and results in lost sales. I'll find someone else.

anotherhue 6y ago

@tptacek: Was there any exploration of Gvisor before landing on the current implementation?

hollerith 6y ago
He wrote about gvisor here (July 2020): https://fly.io/blog/sandboxing-and-workload-isolation/ https://fly.io/blog/sandboxing-and-workload-isolation/ >As batshit as this plan is, it works surprisingly well; you can build gVisor and runsc, its container runtime, relatively easily. Once you have runsc installed, it will run Docker containers for you. After reading the code, I sort of couldn’t believe it was working as well as it did, or, if it was, that it was actually using the code I had read. But I scattered a bunch of panic calls across the codebase and, yup, that all that stuff is actually happening. It’s pretty amazing. >You are probably strictly better off with gVisor than you are with a tuned Docker configuration, and I like it a lot. The big downside is performance; you’ll be looking at a low-double-digits percentage hit, degrading with I/O load. Google runs this stuff at scale in GCE; you can probably get away with it too. If you’re running gVisor, you should brag about it, because, again, gVisor is pretty bananas.
- anotherhue 6y ago
  Thank you!

nextaccountic 6y ago

Here's a question about your architecture: why do you store layers in LVM2 block-level snapshots instead of btrfs or zfs file-level snapshots?

mrkurt 6y ago
The simple answer is: because containerd + devicemapper makes it easy and we already use LVM. We use LVM so we can expose raw block devices to peoples' applications.
deleted 6y ago
[deleted]

chrisweekly 6y ago

Fantastic writeup: technical details, context and framing, all in clear, informal, plain-English prose. It's profoundly effective, and vanishingly rare. As for Fly's solution per se: like many great ideas it seems simple bordering on obvious, in retrospect. I'm impressed and more interested than ever in bringing fly.io into my wheelhouse. Final thoughts: CLI-first: flyctl reminds me of `vercel` (fka Zeit `now`), which is a big compliment > "VM build-and-boot process on a second deployment is faster than the logging that we do" ... ie, well-suited for lambdas / serverless functions at CDN edge -- which is of extreme relevance and interest to me and everyone else working on minimizing latency with modern webapp architecture... exciting times!

undecisive 6y ago

So I've had a question running around my brain for a while. Has anyone tried using docker-compose files to deploy direct to separate cloud machines? By which I mean, given some config with user creds to AWS or Digital Ocean or Hetzner or whatever, running: dockerless-compose some.address.com up could spin up servers (ec2 instances, droplets, maybe some kind of hinting in the docker-compose.yml file to show sizes), maybe set up a virtual network if the hosting provider supports it, set network aliases in the /etc/hosts, set some.address.com to be the gatekeeper (the system would need to install SSH on your boxes, but it means that it could set up an ssh authorized key from the gatekeeper box) And then, maybe it doesn't need to be actual cloud servers. It could be bare metal. It could be chroots. It could even be a single host with docker (a bit pointless, but maybe useful for proving parity with docker-compose) Essentially, suddenly we have a replacement for the likes of terraform in many organisations. Your local developer setup becomes exactly the same as your production setup. Am I missing something? Is that where we're headed with this, or would this be a complete nightmare?

acdha 6y ago
AWS has been moving in that direction: https://aws.amazon.com/blogs/containers/deploy-applications-on-amazon-ecs-using-docker-compose/ https://aws.amazon.com/blogs/containers/deploy-applications-... (See also https://aws.amazon.com/containers/copilot/ https://aws.amazon.com/containers/copilot/) The catch is that this doesn’t replace a tool like Terraform unless your projects are extremely simple and you don’t have much in the way of security requirements. Once you need anything more, Compose doesn’t have the information you’d need and you’d lose the veneer of simplicity and portability trying to add it in.
jonesetc 6y ago
I believe this was docker's (the company) direction for a while, but it was spread across multiple tools. docker-machine does provisioning and docker-swarm runs the compose file on remote hosts.
e12e 6y ago
Sounds like docker swarm is 80% of what you want. Other than that, I think there's a reason why things like lxd, k8s (maybe joyent/smartos belong in this list) - generally have one layer for the actual hw, and a higher level abstraction on top. Otoh, I suppose something like canonical metal-as-a-service might be able to do more with a little help. In fact I think maas+lxd is probably pretty close to being able to work with a compose-like tool and docker containers.

nimbius 6y ago

we already have this, its called podman. https://podman.io https://podman.io

tptacek 6y ago
Podman is very, very cool, but it's not quite what we're doing.
cpach 6y ago
AFAIK, Podman still uses cgroups to separate the containers. Fly uses Firecracker to separate containers. Those are two quite different methods and AFAICT Fly uses Firecracker because of the stronger security model of that solution.

breatheoften 6y ago

Who are the people behind fly? It looks to me like the first thing I've seen in awhile that actually might be "a better heroku". Anybody out there using it? Anybody who was happy with heroku for mostly everything other than price that migrated to fly and remained happy?

austinpena 5y ago
Never used heroku but I love fly

pkulak 6y ago

I've gotten a bit sick of running Docker on my dev systems and switched to Podman. I found a "podman-compose" script that works great for setting up my dev environments just like I'm used to, and I can run everything as my own, non-root user. When my laptop inevitably starts running out of space, there's a single directory of images in my home directory I can rm -rf to start over. It's been great.

hinkley 6y ago

And once again, the promise of multitenant processing on the same machine gets kicked down the road ten years. I wonder if my grandkids will have the things the OS vendors were promising us back in the early 90's.

echelon 6y ago

"Docker without Docker" had me thinking about the venture-funded company behind the Docker runtime. $330M with no sustainable path to revenue. Everyone ate up the open source pieces and there's nothing left to monetize. Docker without Docker.

mwcampbell 6y ago

Just curious, what kind of host OS is Fly running for the machines hosting the VMs? A custom immutable image built with something like LinuxKit? A stock OS designed to be a container host, like Flatcar Linux? Or a conventional distro?

tptacek 6y ago
Conventional distro, very slightly fussy kernel because of the BPF stuff.

replwoacause 6y ago

Love the illustrations on the site!