8 ms·

Show HN: Hocus – self-hosted alternative to GitHub Codespaces using Firecracker

291 points by hugodutka 4y ago

enedil 4y ago
This is actually something I wish we had at my previous company. Setting up the environment was a pain, and very few people knew how to run all the tests (integration tests really). I am willing to give Hocus a go.
cube2222 4y ago
Ah, great to see this here! Disclaimer: Know the authors, who are extremely bright engineers. Based on my discussions with one of them, Hugo, re the architecture a couple weeks ago, I really think you should take a look at this if you want to do auto-provisioned dev environments in a self-hosted fashion, they know what they're doing.
notamy 4y ago
This looks excellent! I have to try spinning it up in my homelab to see how well it works for my needs.
yewenjie 4y ago
How does this compare to https://coder.com https://coder.com?
- hugodutka 4y ago
  Coder lets you describe a set of Terraform templates for your development environments. Your team members then manually spin up new VMs on e.g. AWS from these templates. With Hocus you specify your dev env right alongside your code in your Git repository with a hocus.yml file (https://hocus.dev/docs/hocus-yml https://hocus.dev/docs/hocus-yml). Hocus is deployed on a set of servers you manage (only 1 server during alpha). It has a built-in CI system that watches your repository for new commits and continuously builds the environment - we call this process a prebuild. You can specify that your dependencies should be installed and your code should be compiled during a prebuild. You can also describe tasks that should run when you open a dev env - for example automatically starting the app you are working on. Once a prebuild is ready you can spin up a new dev env - which we call a workspace - and connect to it automatically with your IDE. We also have a VSCode extension that automatically opens terminals connected to those tasks from hocus.yml. All this combined allows each team member to connect to a fresh, ready-to-code dev env anytime they want, whether they are reviewing a PR or writing code themselves.
  - donmcronald 4y ago
    I didn't look at it thoroughly, but it looks really cool and I'll probably give it a proper try at some point. Judging by your comment only... > watches your repository for new commits and continuously builds the environment That feels wrong to me. For example, what happens if I want to check out a 1 year old commit and edit it? Will the dev environment that was used at the time be available or will it be rebuilt? If it gets rebuilt, there's no way I get any guarantees it's going to work. IMHO most of the industry is getting it wrong when it comes to dev environments and build systems. I've always called it butterfly effect builds. You make a commit and it triggers a ton of non-repeatable builds. There's not much value in doing that for every commit IMO. What happens when something like an APT mirror fails and the whole system grinds to a halt? What happens when a runtime or dependency gets updated and breaks at exactly the worst time during the week? I'd rather see something on a (calendar) schedule with the potential to trigger a manual build if needed. For example, skim this [1] article. That's always made sense to me and, in the context of a dev environment, why not build the base dev environment once a week (or day) and only layer in the resources / code on-demand? IE: Build the things I don't control on a calendar schedule and only build on-demand for changes that are part of my project (using the base env). The way I've always envisioned it is to have a stable env build once per month or per week. It's fast enough to keep everything rolling in terms of being current, yet slow enough that I'm not spending time wondering if a problem I'm debugging is caused by a change in my dev environment. I try to do the same thing as dedicated build containers for projects, but it's hard because everything is tunnel visioned on per-commit re-builds these days. 1. https://phauer.com/2019/no-fat-jar-in-docker-image/ https://phauer.com/2019/no-fat-jar-in-docker-image/
    - hugodutka 4y ago
      > what happens if I want to check out a 1 year old commit and edit it? Your dev environment is version controlled too because it lives in your repository. You would run a prebuild for that old commit and get a dev env from that time. It would only break if your old dev env can no longer be built, for example if it depended on some external resource which is no longer available on the internet. > I'd rather see something on a (calendar) schedule with the potential to trigger a manual build if needed. That's actually on our todo list, but it didn't make it into the 0.1 release. This feature is especially useful for monorepos, where rebuilding everything on every commit is too much.
politelemon 4y ago
I see Docker listed as a requirement, and I am familiar with Docker. The repo mentions using Firecracker, a microVM that I've only ever heard of. What is the relation, if any, between these two? Is this a wrong question: if this uses Firecracker why is Docker involved?
- guywithabowtie 4y ago
  Firecracker is the VM powering AWS Lambda, it is maintained by Amazon. It is pretty decent and popular in certain areas. I believe Fly.io is using them behind them too.
  - guywithabowtie 4y ago
    It has special optimizations for reducing cold start.
- hugodutka 4y ago
  We use Firecracker to provider greater isolation and more capabilities in dev envs. For example, one thing you often want in your dev env is to run Docker containers, but to do that you need some permissions that usually only a root user has. If we used Docker to virtualize dev envs then we would have several options to enable that: - make dev envs privileged containers, effectively giving any random user root control over the machine where Hocus is hosted - mount the Docker agent from host into the container, but that also allows the user to gain root control over the host machine - use Linux user namespaces, and give you a root user with limited capabilities inside your dev env. That's actually what Gitpod - another Github Codespaces alternative - does. That root can't do everything though. It can't run Kubernetes, LocalStack, or the Elastic stack, because it can't modify some system limits. A dev env backed by Firecracker is a full-fledged virtual machine that has none of the aforementioned drawbacks. It has some performance tradeoffs though, but we are working to minimize them as much as possible.
  - politelemon 4y ago
    OK thanks for that. I am tinkering with it and noticed that it makes use of KVM and just found out that WSL2 has KVM support so I'm going to give it a try there and see what breaks, ha! https://serverfault.com/questions/1043441/how-to-run-kvm-nested-in-wsl2-or-vmware/1115773#1115773 https://serverfault.com/questions/1043441/how-to-run-kvm-nes...
guywithabowtie 4y ago
What would be the future plan here? Any hosted commercial offerings? I am asking to understand whether one could use and depend upon it for few years.
- hugodutka 4y ago
  A hosted offering is not on our radar right now. Currently we want to monetize by selling seat-based licenses for larger teams.
  - ezekg 4y ago
    I run a software licensing/distribution API (https://keygen.sh https://keygen.sh) that may be able to help you once your self-hosted licensing requirements expand over the next couple years. Reach out if you ever need anything or just want to talk licensing best practices. We're going to be going source-available under ELv2 soon as well. Good to see more companies lately using the Elastic license.
    - hugodutka 4y ago
      Thanks, this looks useful. We will consider using keygen in the future.
- capableweb 4y ago
  It seems the future plans (short and long-term) are already outlined in the README itself. - Ease of deployment and management are central goals of Hocus, with minimal assumptions made about the underlying OS and hardware. - Hocus should provide a native development experience with performance on par with bare-metal development. - Hocus should be scalable to accommodate large teams, with support for thousands of users and heavyweight repositories. We're not there yet. Roadmap - Add basic single node support - Optimize single node storage usage, performance, and reliability - Add permissioning for teams, admins, and regular users - Add multi-node support - Add support for more IDEs, particularly JetBrains
itay 4y ago
Pretty nice, I have two questions: 1. Comparing to something like GitPod (which lets you run on your own instances as well), where do you think Hocus shines? 2. Given you're leveraging Firecracker for isolation, and Firecracker doesn't support GPUs, I assume that adding GPU-enabled machines isn't on your near-term roadmap?
- hugodutka 4y ago
  Thanks! 1. GitPod doesn't officially support self-hosting anymore - https://news.ycombinator.com/item?id=33907897 https://news.ycombinator.com/item?id=33907897. When they did, it was extremely hard to set up. In fact we tried to do it at my previous company several times and failed. Hocus is designed with self-hosting in mind, and we want to make deployment and management straightforward. And from an end user's perspective there are 2 main advantages: - Hocus dev envs are VMs instead of containers, so you can run anything you want inside. For example Kubernetes or nested KVM. - Workspaces start in <5 seconds, while Gitpod can take a minute or longer. 2. Actually we are exploring moving to Cloud Hypervisor or QEMU, so we may support GPUs sooner than later. If you have a specific use case in mind, feel free to contact me - contact info is in my profile. I'd be happy to hear why you need them.
  - itay 4y ago
    Thanks - that's super helpful. 1. Good to know about GitPod - I haven't looked at it for a while so looks like I was outdated. The rest of what you said is good too. 2. This is mostly for ML development, where GPUs are sadly often required even for dev work.
0cf8612b2e1e 4y ago
Very intrigued. With the real VM isolation, I wonder if I could use this as a way of dipping my toe into the Nix pool. Have you given any thought to offering pre-baked development templates? For example, here is a standard Django build with poetry + js utilities + Postgres drivers + jq, etc.
- hugodutka 4y ago
  That's something we would love to do in the future. For now extending the base development template so that it has the most popular programming languages already installed is on top of our todo list.
cpa 4y ago
I believe this kind of service is the way forward when it comes to developer tooling. But paying $39/mo/user for having the right to deal with the deployment myself for a tool that is still alpha is pretty high. I understand the (tremendous) value prop, but that's the kind of price I'd be expecting from a SaaS version, not from a self-hosted app. Although not giving a discount for self-hosting and a price per user is the "modern" way to do open source (see gitlab or mattermost), it still feels a bit wrong. Anyway, excuse my old man rant, awesome work and good luck making a business out of it! I'll give it a try.
- hugodutka 4y ago
  Thanks! The pricing is there mostly to set expectations for the future. The current release works (for the most part) for individual users and it would not support a team well. Hocus is free for up to 3 users, so we expect people who deploy the alpha version not to pay a thing.
  - inhumantsar 4y ago
    For a self-hosted solution that pricing is a complete non-starter. I have a team of 30 devs, so it would be $1,200/mo. That's on par with GitHub's enterprise plan plus Codespaces instance usage. There's no way I could consider this for my company at that price.
    - hugodutka 4y ago
      Hocus is more cost-efficient when you need larger instances. If you need 8 CPU cores ($0.72/hour on Codespaces) and a single dev uses their IDE for 4 hours a day on average, Codespaces itself would come out to about $0.72 * 4 * 5 * 4 * 30 = $1728 a month. But that gives me the idea that maybe we should introduce cheaper pricing plans but with limits on VM resources - pay less if you have lower compute requirements.
      - inhumantsar 4y ago
        Maybe I'm confused here, but wouldn't I still be paying instance costs on top of your per-seat license? 256 cores would be about $2.50 to $5/hr in AWS depending on reservations and how much you're able to oversubscribe. That's another $1,500 to $3,000/mo on top of your $1,170/mo. If you're paying for the compute then cool, that's a good price. If this is self-hosted though then I can't really see why I would bother unless I had no choice but to self host and cobbling together a simpler if less ideal solution isn't allowed.
itissid 4y ago
This seems to be at the same level of how Meta manages their dev environments. You pick the environment right from VS code and it spins up an environment. I think > Add multi-node support will be important for non trivial project development where the 2 distinct environments might be needed to talk to each other like say for prototyping or "sharing configs"..
- hugodutka 4y ago
  By multi-node support we mean the ability of Hocus to manage multiple machines, so you can run dev envs on more than one server and handle more concurrent users. These environments would not communicate with one another. From the end user's perspective using a multi-node deployment vs a single-node deployment would not differ. If you needed 2 distinct environments I would suggest using virtualization within a single dev env.
throwawaaarrgh 4y ago
You can get close to this without licensing/fees by standing up a K8s cluster and running devspace
- gorbak25 4y ago
  Not really, if you develop inside containers in k8s then you will have a hard time: - Installing something into the container - Run docker/podman/k8s etc...
vanillax 4y ago
Why isn't this just wrapped in a single docker image or docker compose? Why do you need all these scripts? Just curious. I get its checking for docker and dependencies, but can you just get rid of the dependency problem by embedding in the docker file and have a single set of docker files that do all this work? Leverage compose after?
- hugodutka 4y ago
  We made these custom deployment scripts to walk the user through the installation process and make it more organized. We could just use plain docker compose but then we would have less control over what we show to the user - things like helpful error messages. For example we detect if you have KVM support enabled and if not, we suggest you should enable it in BIOS. Under the hood these scripts are just wrappers for docker compose with guardrails.
gravypod 4y ago
Are you planning on using any fancy tricks to optimize large repos? I have open sourced some code I was playing with for fun here which allows you to mount a git repo as a read only FUSE file system: https://github.com/gravypod/gitfs https://github.com/gravypod/gitfs You can really lower IOPS/memory/disk usage with an approach like this.
- hugodutka 4y ago
  We definitely want to optimize Hocus for large repos. We have some ideas how to reduce disk usage by stacking qcow2 files, but we haven't settled on anything specific yet. Thanks for the link!

291 points by hugodutka 4y ago

This is actually something I wish we had at my previous company. Setting up the environment was a pain, and very few people knew how to run all the tests (integration tests really). I am willing to give Hocus a go.

cube2222 4y ago

Ah, great to see this here! Disclaimer: Know the authors, who are extremely bright engineers. Based on my discussions with one of them, Hugo, re the architecture a couple weeks ago, I really think you should take a look at this if you want to do auto-provisioned dev environments in a self-hosted fashion, they know what they're doing.

notamy 4y ago

This looks excellent! I have to try spinning it up in my homelab to see how well it works for my needs.

yewenjie 4y ago

How does this compare to https://coder.com https://coder.com?

hugodutka 4y ago
Coder lets you describe a set of Terraform templates for your development environments. Your team members then manually spin up new VMs on e.g. AWS from these templates. With Hocus you specify your dev env right alongside your code in your Git repository with a hocus.yml file (https://hocus.dev/docs/hocus-yml https://hocus.dev/docs/hocus-yml). Hocus is deployed on a set of servers you manage (only 1 server during alpha). It has a built-in CI system that watches your repository for new commits and continuously builds the environment - we call this process a prebuild. You can specify that your dependencies should be installed and your code should be compiled during a prebuild. You can also describe tasks that should run when you open a dev env - for example automatically starting the app you are working on. Once a prebuild is ready you can spin up a new dev env - which we call a workspace - and connect to it automatically with your IDE. We also have a VSCode extension that automatically opens terminals connected to those tasks from hocus.yml. All this combined allows each team member to connect to a fresh, ready-to-code dev env anytime they want, whether they are reviewing a PR or writing code themselves.
- donmcronald 4y ago
  I didn't look at it thoroughly, but it looks really cool and I'll probably give it a proper try at some point. Judging by your comment only... > watches your repository for new commits and continuously builds the environment That feels wrong to me. For example, what happens if I want to check out a 1 year old commit and edit it? Will the dev environment that was used at the time be available or will it be rebuilt? If it gets rebuilt, there's no way I get any guarantees it's going to work. IMHO most of the industry is getting it wrong when it comes to dev environments and build systems. I've always called it butterfly effect builds. You make a commit and it triggers a ton of non-repeatable builds. There's not much value in doing that for every commit IMO. What happens when something like an APT mirror fails and the whole system grinds to a halt? What happens when a runtime or dependency gets updated and breaks at exactly the worst time during the week? I'd rather see something on a (calendar) schedule with the potential to trigger a manual build if needed. For example, skim this [1] article. That's always made sense to me and, in the context of a dev environment, why not build the base dev environment once a week (or day) and only layer in the resources / code on-demand? IE: Build the things I don't control on a calendar schedule and only build on-demand for changes that are part of my project (using the base env). The way I've always envisioned it is to have a stable env build once per month or per week. It's fast enough to keep everything rolling in terms of being current, yet slow enough that I'm not spending time wondering if a problem I'm debugging is caused by a change in my dev environment. I try to do the same thing as dedicated build containers for projects, but it's hard because everything is tunnel visioned on per-commit re-builds these days. 1. https://phauer.com/2019/no-fat-jar-in-docker-image/ https://phauer.com/2019/no-fat-jar-in-docker-image/
  - hugodutka 4y ago
    > what happens if I want to check out a 1 year old commit and edit it? Your dev environment is version controlled too because it lives in your repository. You would run a prebuild for that old commit and get a dev env from that time. It would only break if your old dev env can no longer be built, for example if it depended on some external resource which is no longer available on the internet. > I'd rather see something on a (calendar) schedule with the potential to trigger a manual build if needed. That's actually on our todo list, but it didn't make it into the 0.1 release. This feature is especially useful for monorepos, where rebuilding everything on every commit is too much.

politelemon 4y ago

I see Docker listed as a requirement, and I am familiar with Docker. The repo mentions using Firecracker, a microVM that I've only ever heard of. What is the relation, if any, between these two? Is this a wrong question: if this uses Firecracker why is Docker involved?

guywithabowtie 4y ago
Firecracker is the VM powering AWS Lambda, it is maintained by Amazon. It is pretty decent and popular in certain areas. I believe Fly.io is using them behind them too.
- guywithabowtie 4y ago
  It has special optimizations for reducing cold start.
hugodutka 4y ago
We use Firecracker to provider greater isolation and more capabilities in dev envs. For example, one thing you often want in your dev env is to run Docker containers, but to do that you need some permissions that usually only a root user has. If we used Docker to virtualize dev envs then we would have several options to enable that: - make dev envs privileged containers, effectively giving any random user root control over the machine where Hocus is hosted - mount the Docker agent from host into the container, but that also allows the user to gain root control over the host machine - use Linux user namespaces, and give you a root user with limited capabilities inside your dev env. That's actually what Gitpod - another Github Codespaces alternative - does. That root can't do everything though. It can't run Kubernetes, LocalStack, or the Elastic stack, because it can't modify some system limits. A dev env backed by Firecracker is a full-fledged virtual machine that has none of the aforementioned drawbacks. It has some performance tradeoffs though, but we are working to minimize them as much as possible.
- politelemon 4y ago
  OK thanks for that. I am tinkering with it and noticed that it makes use of KVM and just found out that WSL2 has KVM support so I'm going to give it a try there and see what breaks, ha! https://serverfault.com/questions/1043441/how-to-run-kvm-nested-in-wsl2-or-vmware/1115773#1115773 https://serverfault.com/questions/1043441/how-to-run-kvm-nes...

guywithabowtie 4y ago

What would be the future plan here? Any hosted commercial offerings? I am asking to understand whether one could use and depend upon it for few years.

hugodutka 4y ago
A hosted offering is not on our radar right now. Currently we want to monetize by selling seat-based licenses for larger teams.
- ezekg 4y ago
  I run a software licensing/distribution API (https://keygen.sh https://keygen.sh) that may be able to help you once your self-hosted licensing requirements expand over the next couple years. Reach out if you ever need anything or just want to talk licensing best practices. We're going to be going source-available under ELv2 soon as well. Good to see more companies lately using the Elastic license.
  - hugodutka 4y ago
    Thanks, this looks useful. We will consider using keygen in the future.
capableweb 4y ago
It seems the future plans (short and long-term) are already outlined in the README itself. - Ease of deployment and management are central goals of Hocus, with minimal assumptions made about the underlying OS and hardware. - Hocus should provide a native development experience with performance on par with bare-metal development. - Hocus should be scalable to accommodate large teams, with support for thousands of users and heavyweight repositories. We're not there yet. Roadmap - Add basic single node support - Optimize single node storage usage, performance, and reliability - Add permissioning for teams, admins, and regular users - Add multi-node support - Add support for more IDEs, particularly JetBrains

itay 4y ago

Pretty nice, I have two questions: 1. Comparing to something like GitPod (which lets you run on your own instances as well), where do you think Hocus shines? 2. Given you're leveraging Firecracker for isolation, and Firecracker doesn't support GPUs, I assume that adding GPU-enabled machines isn't on your near-term roadmap?

hugodutka 4y ago
Thanks! 1. GitPod doesn't officially support self-hosting anymore - https://news.ycombinator.com/item?id=33907897 https://news.ycombinator.com/item?id=33907897. When they did, it was extremely hard to set up. In fact we tried to do it at my previous company several times and failed. Hocus is designed with self-hosting in mind, and we want to make deployment and management straightforward. And from an end user's perspective there are 2 main advantages: - Hocus dev envs are VMs instead of containers, so you can run anything you want inside. For example Kubernetes or nested KVM. - Workspaces start in <5 seconds, while Gitpod can take a minute or longer. 2. Actually we are exploring moving to Cloud Hypervisor or QEMU, so we may support GPUs sooner than later. If you have a specific use case in mind, feel free to contact me - contact info is in my profile. I'd be happy to hear why you need them.
- itay 4y ago
  Thanks - that's super helpful. 1. Good to know about GitPod - I haven't looked at it for a while so looks like I was outdated. The rest of what you said is good too. 2. This is mostly for ML development, where GPUs are sadly often required even for dev work.

0cf8612b2e1e 4y ago

Very intrigued. With the real VM isolation, I wonder if I could use this as a way of dipping my toe into the Nix pool. Have you given any thought to offering pre-baked development templates? For example, here is a standard Django build with poetry + js utilities + Postgres drivers + jq, etc.

hugodutka 4y ago
That's something we would love to do in the future. For now extending the base development template so that it has the most popular programming languages already installed is on top of our todo list.

cpa 4y ago

I believe this kind of service is the way forward when it comes to developer tooling. But paying $39/mo/user for having the right to deal with the deployment myself for a tool that is still alpha is pretty high. I understand the (tremendous) value prop, but that's the kind of price I'd be expecting from a SaaS version, not from a self-hosted app. Although not giving a discount for self-hosting and a price per user is the "modern" way to do open source (see gitlab or mattermost), it still feels a bit wrong. Anyway, excuse my old man rant, awesome work and good luck making a business out of it! I'll give it a try.

hugodutka 4y ago
Thanks! The pricing is there mostly to set expectations for the future. The current release works (for the most part) for individual users and it would not support a team well. Hocus is free for up to 3 users, so we expect people who deploy the alpha version not to pay a thing.
- inhumantsar 4y ago
  For a self-hosted solution that pricing is a complete non-starter. I have a team of 30 devs, so it would be $1,200/mo. That's on par with GitHub's enterprise plan plus Codespaces instance usage. There's no way I could consider this for my company at that price.
  - hugodutka 4y ago
    Hocus is more cost-efficient when you need larger instances. If you need 8 CPU cores ($0.72/hour on Codespaces) and a single dev uses their IDE for 4 hours a day on average, Codespaces itself would come out to about $0.72 * 4 * 5 * 4 * 30 = $1728 a month. But that gives me the idea that maybe we should introduce cheaper pricing plans but with limits on VM resources - pay less if you have lower compute requirements.
    - inhumantsar 4y ago
      Maybe I'm confused here, but wouldn't I still be paying instance costs on top of your per-seat license? 256 cores would be about $2.50 to $5/hr in AWS depending on reservations and how much you're able to oversubscribe. That's another $1,500 to $3,000/mo on top of your $1,170/mo. If you're paying for the compute then cool, that's a good price. If this is self-hosted though then I can't really see why I would bother unless I had no choice but to self host and cobbling together a simpler if less ideal solution isn't allowed.

itissid 4y ago

This seems to be at the same level of how Meta manages their dev environments. You pick the environment right from VS code and it spins up an environment. I think > Add multi-node support will be important for non trivial project development where the 2 distinct environments might be needed to talk to each other like say for prototyping or "sharing configs"..

hugodutka 4y ago
By multi-node support we mean the ability of Hocus to manage multiple machines, so you can run dev envs on more than one server and handle more concurrent users. These environments would not communicate with one another. From the end user's perspective using a multi-node deployment vs a single-node deployment would not differ. If you needed 2 distinct environments I would suggest using virtualization within a single dev env.

throwawaaarrgh 4y ago

You can get close to this without licensing/fees by standing up a K8s cluster and running devspace

gorbak25 4y ago
Not really, if you develop inside containers in k8s then you will have a hard time: - Installing something into the container - Run docker/podman/k8s etc...

vanillax 4y ago

Why isn't this just wrapped in a single docker image or docker compose? Why do you need all these scripts? Just curious. I get its checking for docker and dependencies, but can you just get rid of the dependency problem by embedding in the docker file and have a single set of docker files that do all this work? Leverage compose after?

hugodutka 4y ago
We made these custom deployment scripts to walk the user through the installation process and make it more organized. We could just use plain docker compose but then we would have less control over what we show to the user - things like helpful error messages. For example we detect if you have KVM support enabled and if not, we suggest you should enable it in BIOS. Under the hood these scripts are just wrappers for docker compose with guardrails.

gravypod 4y ago

Are you planning on using any fancy tricks to optimize large repos? I have open sourced some code I was playing with for fun here which allows you to mount a git repo as a read only FUSE file system: https://github.com/gravypod/gitfs https://github.com/gravypod/gitfs You can really lower IOPS/memory/disk usage with an approach like this.

hugodutka 4y ago
We definitely want to optimize Hocus for large repos. We have some ideas how to reduce disk usage by stacking qcow2 files, but we haven't settled on anything specific yet. Thanks for the link!