7 ms·

Launch HN: LayerCI (YC S20) - Staging servers that act like (and replace) CI

174 points by colinchartier 6y ago

Hi HN, Lyn & Colin here. We’re co-founders of LayerCI (https://layerci.com https://layerci.com), which gives you a modern DevOps experience (CI/CD & staging environments) with as little work as writing a Dockerfile. Most teams need CI/CD (run the build and deploy every time a developer pushes) or staging (host a server with my app in it to share), but current approaches always have at least one of these problems: - Simplistic (only run unit tests) - Slow (wait 10 minutes to run the same repetitive setup steps like "npm install") - Complex (cache keys, base images, a slack channel to reserve staging servers, …) We’ve spent over a year iterating with our customers to build a product that solves all of these problems. Our configuration files (Layerfiles) look like Dockerfiles, so regular developers can write and maintain them. Here's one that creates a staging server for create-react-app: FROM vm/ubuntu:18.04 RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - && curl -fSsL https://deb.nodesource.com/setup_12.x https://deb.nodesource.com/setup_12.x | bash && apt-get install nodejs python3 make gcc build-essential COPY . . RUN npm install RUN npm test RUN BACKGROUND npm start EXPOSE WEBSITE http://localhost:3000 http://localhost:3000 We charge a flat $42/mo/developer on our paid plan. Because it's a flat fee and not usage based, we're incentivized to make things as fast as possible: Our current margins come from a custom-built hibernating hypervisor that lets us avoid running "npm install" thousands of times per day. We’ve upgraded the free tier to 5GB of memory for new installations this week. It’s perfect for personal projects or small MVPs where you’d like a powerful demo server that will build on every push and automatically hibernate when it’s not being used. The easiest way to try out LayerCI is to follow our interactive tutorial: https://layerci.com/ https://layerci.com/ or look at the docs: https://layerci.com/docs/ https://layerci.com/docs/ We would love to hear your thoughts about CI/CD, staging, and what we’ve built!

thruflo 6y ago
Congrats on the launch :) The pricing incentives sound smart. Do you see the ability to replicate more complex architecture as a differentiator? For example, one of your homepage quotes mentions "running a Kubernetes stack inside of a Layerfile". Can you elaborate more on how Layerfiles enable this?
- colinchartier 6y ago
  With Kubernetes, a typical workflow might look like this: 1. Start kubernetes cluster 2. Build docker images 3. Deploy docker images (helm, kubectl, argocd, ...) 4(a) Run unit tests (kubectl exec -l k8s-app=web rake test) 4(b) Run e2e tests (kubectl run cypress) 4(c) Create ephemeral environment (EXPOSE WEBSITE localhost:8000) Because we take memory snapshots after each step, you'd effectively get a fresh, fully-provisioned kubernetes cluster immediately after pushing instead of re-running all the steps every time (you'd skip to step 3) and then run 4{a,b,c} in parallel (they'd all "fork" the VM and get a separate copy of all of the resources) Here are a few links that go into more detail: - https://layerci.com/docs/tuning-performance/run-repeatable https://layerci.com/docs/tuning-performance/run-repeatable - https://layerci.com/blog/ci-at-layerci/ https://layerci.com/blog/ci-at-layerci/
  - thruflo22 6y ago
    Ok, so the more complex the infra, the more advantage there is in caching.
    - colinchartier 6y ago
      The idea is it starts as simple as a docker build (e.g., as easy or easier than GitHub actions, orbs, etc) but scales as your code/infra does.
deleted 6y ago
[deleted]
Lyn_layerci 6y ago
Colin's cofounder here :) Excited to hear from you all
mrkurt 6y ago
This is super cool. I would like to know more about VM hibernation!
- colinchartier 6y ago
  Hey Kurt, we're big fans of fly.io! Happy to chat about tech sometime, I think you folks are using firecracker for a similar use-case? Drop me a line at colin@layerci.com
  - ignoramous 6y ago
    Hi Colin, congrats on the (re-)launch! Fellow SUS 2019 graduate here. I know layerci's tech precedes snapshot support in Firecracker [0], but are snapshots similar to layerci's magic or there's still a significant way for snapshots to go before it can be on par with layerci's tech? I ask because I remember you mention layerci uses techniques (from how Linux hibernates PCs) which are good enough for tests and staging but not production; [1] whereas Firecracker obviously can't rely on such tech so there must be key differences? Thanks. [0] https://github.com/firecracker-microvm/firecracker/blob/master/docs/snapshotting/snapshot-support.md https://github.com/firecracker-microvm/firecracker/blob/mast... [1] https://news.ycombinator.com/item?id=23036776 https://news.ycombinator.com/item?id=23036776
    - colinchartier 6y ago
      Firecracker is definitely heading in the same direction we did with their "diff snapshots", though they don't let you create a COW chain yet, so there's still quite a bit of development to be done on that front.
      - mrkurt 6y ago
        Firecracker snapshots to device mapper could be pretty cool. We don't do much with snapshot/restore but we do a _lot_ with device mapper. It's pretty badass.
benkoller 6y ago
Great approach. I've been running ops teams for a long time, and eventually all teams have ended up spending some time optimising our CI tooling to involve some degree of caching - so you definitely are on to something. Before I clicked your pricing I'd really wished for an affordable pricing plan to run this on BYO resources, but only your enterprise plan seems to cover this. I always get an iffy feeling when I have to build my software on external resources I have 0 control over. Your downtime will prevent teams from shipping their code - but I guess that thought is part of your upsell to the enterprise plan. Anyway, it's one of these ideas I'd wish I had years ago, so congratz to you.
- colinchartier 6y ago
  BYO resources for CI sounds good in theory, but considering costs it's almost always a worse proposition: - 10 hours / month of maintenance + 200 hours of setup @ $60/hr = $13k in the first year, not including any infrastructure costs. If you just redirected that money to a hosted offering you'd get significantly more powerful servers without needing to allocate engineering / management resources - it also makes it significantly easier to push updates & monitor uptime on the host's end. Even traditionally hosted tools like Atlassian's suite are moving to cloud for the same reason: https://www.atlassian.com/migration/journey-to-cloud https://www.atlassian.com/migration/journey-to-cloud
  - zbruhnke 6y ago
    While this is likely true in many cases I'm in a very sensitive industry (banking) and we tend to self host things not for cost reasons but for security reasons. We spend a lot of time going through pen tests, getting SOC2 compliance, etc. Handing off something this critical can cause an even more painful audit in many cases so just a thought to consider cost is sometimes not the only factor. Looks like a really cool product though!
    - colinchartier 6y ago
      We're actually in the process of getting SOC2 and pen tests ourselves - another benefit of a hosted offering is it can (eventually) integrate into your compliance system (e.g., vanta) A lot of our customers are in fintech (payroll, banking, etc) so we've spent a lot of effort on our security model: https://layerci.com/security https://layerci.com/security
aloknnikhil 6y ago
Nice work! I played around with the demo and launched an SSH session for the last layer (hosting the website). And I was able to trigger a VM reboot from the shell. And now the website is not up anymore. Curious to know if there's any monitoring or alerting that I can see when something like this is happening? And how do I go about debugging it/restoring functionality?
- colinchartier 6y ago
  Hey Alok, RUN BACKGROUND basically does "./thecommand&" so it will stop after a restart, the onboarding example is ephemeral for simplicity. If you want things to persist across restarts you'd have to add a systemd script or docker container the same way you'd run it in production.
  - aloknnikhil 6y ago
    Makes sense. Thanks!
rumblecat 6y ago
I'm looking at your docs, and the RUN REPEATABLE command seems really powerful. But if the state is broken after a run, like if you have some pods stuck at Terminating, how would you recover things? Another question I have is how would you handle state that sometimes needs to update and sometimes doesn't? For example, it would be ideal to have a staging database that can keep having migrations and data added to it when new features are added, but we only want to checkpoint the changes to it from testing when the PR is actually merged.
- colinchartier 6y ago
  If the state is broken, you can "rerun without snapshots" via the dropdown in the top right - and since future runs load the latest snapshot, they'd use a "clean" one. For databases - usually users have a named S3 bucket and use a secret to authenticate, since we take memory snapshots, the top of your Layerfile can be "start the database and populate it from this specific anonymized dump" and then you can edit the file in S3 and re-run without snapshots if you'd like to reload it. Here's the doc page for that: https://layerci.com/docs/advanced-workflows/layerfiles-can-set-up-a-migrated-database-in-5-seconds https://layerci.com/docs/advanced-workflows/layerfiles-can-s...
mamcx 6y ago
This sound very tempting! I run Rust CI for Windows/Linux (on azure pipelines) and it need art least 1h to complete, with several tricks already done. When something broke the iteration is very slow to fix it (you wait 1h, then find the problem, then publish, another hour, you forget something...) So I have a local deployment so i can fix things faster, but is like negating the whole point. If yours can help in deploy rust faster chime in https://www.reddit.com/r/rust/ https://www.reddit.com/r/rust/ because is a major pain point! P.D: I assume this not include osx/windows?
- colinchartier 6y ago
  We definitely support rust - you could even use "RUN REPEATABLE" to reuse the state from prior builds, the same way you'd build locally. Happy to chat more and send you some swag if you try it out and write a blog post about your experience :) colin@layerci.com You can't 'FROM vm/osx' or 'FROM vm/windows' yet, it's relatively difficult to navigate licensing as a startup unfortunately.
  - mamcx 6y ago
    Yeah, lets try. If I can cut times for deployment the web sites is still a win. I could wait for the windows builds, them are for utilities... > You can't 'FROM vm/osx' or 'FROM vm/windows' yet, it's relatively difficult to navigate licensing as a startup unfortunately. Maybe allowing to connect to vultr/digitalocean? Them have cloud instances with windows.
malthejorgensen 6y ago
As an early user -- the speedups are very real! Feels great to just skip build step after build step, due to Docker-like layers.
- colinchartier 6y ago
  Thanks, Malthe - it's been great working with your team.
0xbadcafebee 6y ago
"LayerCI is an all-in-one devops platform, but we provide three main value propositions to our users:" - Run e2e tests on every commit. - Collaborate easier with per-PR staging environments. - Set up CI/CD to build and deploy to production Anyone can do this without your company. If that's the value you bring, then I don't need your company. What value do you bring that I can't get somewhere else? Also, red flag: mis-use of the term DevOps for marketing purposes. "There are a few ways of thinking about these files named ‘Layerfile’:" - Auto-discovered Dockerfiles that build entire virtual machines instead of containers, just as quickly. - Define a tree of virtual machines. Each subsequent layer can inherit all the running processes in its parents. - Trigger specific actions like build, push, test, and deploy in parallel every time you push new code. It's not explained well what benefit there is to using VMs rather than containers. Later on it briefly mentions possibly better caching than Docker containers. If it's actually better than Docker, you are severely burying the lede. "Better than Docker" should be on the front page. It's also not explaining whether I need to throw away all the time and money I might have already invested in containers, or what this is and isn't compatible with. A lot of this solution is really built around optimizing one specific problem, which is caching during re-running whole CI pipelines. This limits the viability of your business. As soon as your customers figure out a new way to solve this problem, they don't need your custom platform anymore. I would consider LayerCI more of a specific feature of a much larger offering. (In particular, the 'correc't way to solve this problem is to stand up an environment for your PR and structure your CI so you only re-run the parts that affect the particular change you want to make, rather than re-running the entire pipeline. It involves putting more thought into running your pipeline but often means the only tasks that get executed are copying a file and restarting your app) P.S. You're installing the yarn GPG key and then not doing anything with it. Your node setup script isn't installing the yarn repo or yarn itself, so this is a throwaway command.
- colinchartier 6y ago
  > In particular, the 'correct way to solve this problem is to stand up an environment for your PR and structure your CI so you only re-run the parts that affect the particular change you want to make, rather than re-running the entire pipeline. It involves putting more thought into running your pipeline but often means the only tasks that get executed are copying a file and restarting your app This is actually exactly what our platform helps with - instead of micromanaging this (and updating the configurations) you can rely on a friendly SaaS to do it for you. I wouldn't call us "better" than Docker, we're more like "Take Docker and make it work better for CI/CD and staging servers"
1337shadow 6y ago
> - Slow (wait 10 minutes to run the same repetitive setup steps like "npm install") That should happen any time you change dependencies if your Dockerfile is efficiently setup, not every time you change a line in runtime code ... Maybe that: COPY . . RUN npm install Is not what you should do to have an efficient docker build phase, did you try this instead ? COPY package.json RUN npm install COPY . . Multistage builds will help if you build in two languages in the same Dockerfile. Anyway, still slow when you add/remove a package because it redownloads everything, my best trick is to share ~/.npm cache directory, but I manage this with buildah instead of docker build.
- colinchartier 6y ago
  One of the benefits of Layerfiles vs Dockerfiles is that those two examples will do the same thing - we monitor which files are read so you don't have to micromanage copying specific steps. But I suppose "npm install" is a simplistic example, "npm run setup_our_ci_environment" would be a more likely step for a larger CI pipeline
  - Too 6y ago
    Impressive, and scary. How does it work? Some kind of ptrace wrapping of each command? If this genuinely works reliably, this alone could be a good product. As someone said below, you should promote this feature more specifically.
    - colinchartier 6y ago
      OS-level VFS wrapper (there are a few places you can put probes here, another approach might be eBPS) We used to sell the features, but that actually didn't work super well - maybe we can add a "technical details" page for the nerds ;)
      - ignoramous 6y ago
        Yes, please. If that impressive hypervisor tech can handle production, I see endless possibilities, including mitigating downtimes by teleporting workloads to different cloud vendors and different regions. Easier said than done, of course!
colinchartier 6y ago
If you'd like to chat with me (CEO @ LayerCI) personally, I'm at colin@layerci.com
old-gregg 6y ago
I see developer tools on HN. I upvote! Congrats on launching! I know this is a good idea because we actually invested time in building something similar internally.
- colinchartier 6y ago
  Thanks, we're big fans of teleport - funny how small the developer tools world actually is.
tekno45 6y ago
Can we get SSO for not a bajillion dollars? Seems like this functionality is really inflated across the entire SaaS space. Is SSO really that hard? that expensive?
- colinchartier 6y ago
  Most developers log in to LayerCI via GitHub or GitLab - so if you have a GitHub plan with SSO, you'll get SSO with Layer as well. We even support self-hosted GitLab, so if you use a custom SSO provider there, developers could use that oauth flow for authentication + authorization.
data_ders 6y ago
How does dependency caching configuration work? I get wanting to make things as fast as possible, but I struggle a bunch a Python dep caching in Circle CI... is there a way to ensure a 100% rebuild? Also interested to see how you plan to support open source
- colinchartier 6y ago
  There's a "CACHE" directive if you want traditional dep caching, but if you don't edit requirements.txt, a Layerfile will just entirely skip the pip install You can "SNAPSHOT disabled" if you'd like certain steps to always run
technics256 6y ago
What's the difference between this and something like hashicorp waypoint?
julienlafond 6y ago
Hi @colinchartier! From what I see your product looks great for running a single app with a ready-to-use staging environment. In practice, a stating environment is often not limited to one app, but at least 1 frontend inter-connected with at least 1 backend (and often, many frontends connected to many micro-services). Does your product respond to this kind of architecture?
- colinchartier 6y ago
  Yes, it's quite simple if you use something like docker compose. If your frontend/backend are in separate repositories, I'd reference this doc page: https://layerci.com/docs/advanced-workflows/routing https://layerci.com/docs/advanced-workflows/routing
gajus 6y ago
How is this different from the likes of https://vercel.com/ https://vercel.com/ ? It doesn't need to be – but understanding the difference / similarities with other tools that have a similar sounding proposition helps with user acquisition / defining your niche.
- colinchartier 6y ago
  We're halfway between CircleCI and Vercel - not for production hosting, but easily allowing for backends / databases to run (including a separate backend for every push)
  - gajus 6y ago
    Why wouldn't I just deploy with every commit to Vercel? Either using GitHub actions or GitLab CI/CD?
    - colinchartier 6y ago
      It takes significantly longer - we have advanced snapshotting logic to skip steps like "start the database" that you'd otherwise have to run every time you pushed. We also auto-hibernate the instances, so you don't have to pay for a bunch of separate databases running in parallel. Don't just listen to me though, try the free tier for yourself! A lot of our users combine LayerCI for developer tooling and vercel/netlify/GCP/AWS for deployment, they're quite complimentary.
- nrmitchi 6y ago
  Many production-hosting solutions offer similar functionality, but the catch is that they only really work if you're already using that system. If you're already deploying on Vercel, then you likely just want to use Vercel's solution. The same goes for Netlify Preview Deploys, Heroku Review Apps, Render Pull Request Previews, etc. If you are not using one of these PaaS's, trying to tap into one for just this functionality tends to end up rather convoluted, and somewhat defeats the purpose.
mcintyre1994 6y ago
This sounds really exciting. I love your live demo, really slick introduction and tutorial - and answered some questions I had about E2E! I've got a couple of other questions though. > 12 full stack preview environments per commit with up to 16GB of memory and 6 CPUs Quote from your pricing. If we had a monorepo with over 12 small test/integration jobs, but not 12 full preview environments, is this usable? Are they one and the same, or can we have eg. a suite of unit tests that don't count toward this full preview limit? Do some teams keep some unit tests off Layer and just use you for the more interesting pieces? Second, do you have any documentation about which databases you support? Concretely can you restore a MongoDB snapshot super-fast? Third, do you have any story around secrets if we want the staging server to hold some secret API keys? Currently we can do this with AWS (own account) CI machines in Gitlab deploying to ECS with AWS secrets - they stay end-to-end encrypted and nobody sees them. Is there any similar way we can get secrets onto a staging server without you having access to them? I suspect this would be a deal breaker for the staging use case for my team. Similar to above, our database snapshots are stored in non-public S3 buckets, how would that work? Again currently it's a case of giving the CI AWS role permission to access them, not sure what an equivalent would look like. One more, is it possible to access the built docker images? We deploy Docker images to ECS, and currently they're the exact same ones built and tested in CI which is a nice reassurance. Do your customers have an out-of-band process for building + deploying to production outside of Layer? Aside, I think this may be a typo on your pricing page? "We'll never increase your the terms of your bill once you start your subscription."
- andrejguran 6y ago
  I am curious about the third point of e2e encryption of the secrets. The problem is that the hosting provider would see plaintext secrets or that someone with access to AWS account could see them? You could also input the encrypted secrets theoretically into any service and then decrypt them in your application. Just curious on what kind of considerations do you have when choosing a hosting provider.
  - mcintyre1994 6y ago
    It's basically the same concern as having them in source code, we want to be able to control/restrict access to them. In AWS we can do that becase they're encrypted with a KMS key and unless a user/role has permission to decrypt with that key it can't decrypt the secret. This does assume that AWS aren't lying about encrypting things, and that their employees can't access our KMS keys. We currently use Gitlab for our CI/CD pipelines, but using our own runners in our own AWS account. So if we want to deploy a staging environment from there, it's actually deploying from an AWS role that we control, we're not leaking any secrets to Gitlab or anywhere else. I'm just wondering how people get around this in Layer/any hosted CI/CD setup where you can't have your own runners inside your AWS account. Especially because they don't replace the production deploy, so ultimately those secrets are staying in AWS - perhaps in addition to wherever they need to go with Layer. FWIW I did find something about secrets in the docs: https://layerci.com/docs/layerfile-reference/secret-env#secret-env https://layerci.com/docs/layerfile-reference/secret-env#secr... - but I'm not sure how you get secrets in from that.
pdeva1 6y ago
how do you know which steps to skip or rerun each run. in your react example, how do you know when to reinstall yarn to latest version and when to skip. https://layerci.com/docs/examples/react https://layerci.com/docs/examples/react
- colinchartier 6y ago
  We monitor which files are read by which steps and map that back to the changes you've made automatically!
danpalmer 6y ago
This looks great. I’m interested in your approach to databases and specifically the data in staging environments? I’ve found setting up the data in databases for staging environments can take as much time as CI, and often works in quite a different way. Shared databases can work, but then may limit staging changes that include migrations.
- colinchartier 6y ago
  We'll snapshot the state of the database after you set it up once, and then re-load that snapshot every following time you need the database to exist.

174 points by colinchartier 6y ago

thruflo 6y ago

Congrats on the launch :) The pricing incentives sound smart. Do you see the ability to replicate more complex architecture as a differentiator? For example, one of your homepage quotes mentions "running a Kubernetes stack inside of a Layerfile". Can you elaborate more on how Layerfiles enable this?

colinchartier 6y ago
With Kubernetes, a typical workflow might look like this: 1. Start kubernetes cluster 2. Build docker images 3. Deploy docker images (helm, kubectl, argocd, ...) 4(a) Run unit tests (kubectl exec -l k8s-app=web rake test) 4(b) Run e2e tests (kubectl run cypress) 4(c) Create ephemeral environment (EXPOSE WEBSITE localhost:8000) Because we take memory snapshots after each step, you'd effectively get a fresh, fully-provisioned kubernetes cluster immediately after pushing instead of re-running all the steps every time (you'd skip to step 3) and then run 4{a,b,c} in parallel (they'd all "fork" the VM and get a separate copy of all of the resources) Here are a few links that go into more detail: - https://layerci.com/docs/tuning-performance/run-repeatable https://layerci.com/docs/tuning-performance/run-repeatable - https://layerci.com/blog/ci-at-layerci/ https://layerci.com/blog/ci-at-layerci/
- thruflo22 6y ago
  Ok, so the more complex the infra, the more advantage there is in caching.
  - colinchartier 6y ago
    The idea is it starts as simple as a docker build (e.g., as easy or easier than GitHub actions, orbs, etc) but scales as your code/infra does.

deleted 6y ago

[deleted]

Lyn_layerci 6y ago

Colin's cofounder here :) Excited to hear from you all

mrkurt 6y ago

This is super cool. I would like to know more about VM hibernation!

colinchartier 6y ago
Hey Kurt, we're big fans of fly.io! Happy to chat about tech sometime, I think you folks are using firecracker for a similar use-case? Drop me a line at colin@layerci.com
- ignoramous 6y ago
  Hi Colin, congrats on the (re-)launch! Fellow SUS 2019 graduate here. I know layerci's tech precedes snapshot support in Firecracker [0], but are snapshots similar to layerci's magic or there's still a significant way for snapshots to go before it can be on par with layerci's tech? I ask because I remember you mention layerci uses techniques (from how Linux hibernates PCs) which are good enough for tests and staging but not production; [1] whereas Firecracker obviously can't rely on such tech so there must be key differences? Thanks. [0] https://github.com/firecracker-microvm/firecracker/blob/master/docs/snapshotting/snapshot-support.md https://github.com/firecracker-microvm/firecracker/blob/mast... [1] https://news.ycombinator.com/item?id=23036776 https://news.ycombinator.com/item?id=23036776
  - colinchartier 6y ago
    Firecracker is definitely heading in the same direction we did with their "diff snapshots", though they don't let you create a COW chain yet, so there's still quite a bit of development to be done on that front.
    - mrkurt 6y ago
      Firecracker snapshots to device mapper could be pretty cool. We don't do much with snapshot/restore but we do a _lot_ with device mapper. It's pretty badass.

benkoller 6y ago

Great approach. I've been running ops teams for a long time, and eventually all teams have ended up spending some time optimising our CI tooling to involve some degree of caching - so you definitely are on to something. Before I clicked your pricing I'd really wished for an affordable pricing plan to run this on BYO resources, but only your enterprise plan seems to cover this. I always get an iffy feeling when I have to build my software on external resources I have 0 control over. Your downtime will prevent teams from shipping their code - but I guess that thought is part of your upsell to the enterprise plan. Anyway, it's one of these ideas I'd wish I had years ago, so congratz to you.

colinchartier 6y ago
BYO resources for CI sounds good in theory, but considering costs it's almost always a worse proposition: - 10 hours / month of maintenance + 200 hours of setup @ $60/hr = $13k in the first year, not including any infrastructure costs. If you just redirected that money to a hosted offering you'd get significantly more powerful servers without needing to allocate engineering / management resources - it also makes it significantly easier to push updates & monitor uptime on the host's end. Even traditionally hosted tools like Atlassian's suite are moving to cloud for the same reason: https://www.atlassian.com/migration/journey-to-cloud https://www.atlassian.com/migration/journey-to-cloud
- zbruhnke 6y ago
  While this is likely true in many cases I'm in a very sensitive industry (banking) and we tend to self host things not for cost reasons but for security reasons. We spend a lot of time going through pen tests, getting SOC2 compliance, etc. Handing off something this critical can cause an even more painful audit in many cases so just a thought to consider cost is sometimes not the only factor. Looks like a really cool product though!
  - colinchartier 6y ago
    We're actually in the process of getting SOC2 and pen tests ourselves - another benefit of a hosted offering is it can (eventually) integrate into your compliance system (e.g., vanta) A lot of our customers are in fintech (payroll, banking, etc) so we've spent a lot of effort on our security model: https://layerci.com/security https://layerci.com/security

aloknnikhil 6y ago

Nice work! I played around with the demo and launched an SSH session for the last layer (hosting the website). And I was able to trigger a VM reboot from the shell. And now the website is not up anymore. Curious to know if there's any monitoring or alerting that I can see when something like this is happening? And how do I go about debugging it/restoring functionality?

colinchartier 6y ago
Hey Alok, RUN BACKGROUND basically does "./thecommand&" so it will stop after a restart, the onboarding example is ephemeral for simplicity. If you want things to persist across restarts you'd have to add a systemd script or docker container the same way you'd run it in production.
- aloknnikhil 6y ago
  Makes sense. Thanks!

rumblecat 6y ago

I'm looking at your docs, and the RUN REPEATABLE command seems really powerful. But if the state is broken after a run, like if you have some pods stuck at Terminating, how would you recover things? Another question I have is how would you handle state that sometimes needs to update and sometimes doesn't? For example, it would be ideal to have a staging database that can keep having migrations and data added to it when new features are added, but we only want to checkpoint the changes to it from testing when the PR is actually merged.

colinchartier 6y ago
If the state is broken, you can "rerun without snapshots" via the dropdown in the top right - and since future runs load the latest snapshot, they'd use a "clean" one. For databases - usually users have a named S3 bucket and use a secret to authenticate, since we take memory snapshots, the top of your Layerfile can be "start the database and populate it from this specific anonymized dump" and then you can edit the file in S3 and re-run without snapshots if you'd like to reload it. Here's the doc page for that: https://layerci.com/docs/advanced-workflows/layerfiles-can-set-up-a-migrated-database-in-5-seconds https://layerci.com/docs/advanced-workflows/layerfiles-can-s...

mamcx 6y ago

This sound very tempting! I run Rust CI for Windows/Linux (on azure pipelines) and it need art least 1h to complete, with several tricks already done. When something broke the iteration is very slow to fix it (you wait 1h, then find the problem, then publish, another hour, you forget something...) So I have a local deployment so i can fix things faster, but is like negating the whole point. If yours can help in deploy rust faster chime in https://www.reddit.com/r/rust/ https://www.reddit.com/r/rust/ because is a major pain point! P.D: I assume this not include osx/windows?

colinchartier 6y ago
We definitely support rust - you could even use "RUN REPEATABLE" to reuse the state from prior builds, the same way you'd build locally. Happy to chat more and send you some swag if you try it out and write a blog post about your experience :) colin@layerci.com You can't 'FROM vm/osx' or 'FROM vm/windows' yet, it's relatively difficult to navigate licensing as a startup unfortunately.
- mamcx 6y ago
  Yeah, lets try. If I can cut times for deployment the web sites is still a win. I could wait for the windows builds, them are for utilities... > You can't 'FROM vm/osx' or 'FROM vm/windows' yet, it's relatively difficult to navigate licensing as a startup unfortunately. Maybe allowing to connect to vultr/digitalocean? Them have cloud instances with windows.

malthejorgensen 6y ago

As an early user -- the speedups are very real! Feels great to just skip build step after build step, due to Docker-like layers.

colinchartier 6y ago
Thanks, Malthe - it's been great working with your team.

0xbadcafebee 6y ago

"LayerCI is an all-in-one devops platform, but we provide three main value propositions to our users:" - Run e2e tests on every commit. - Collaborate easier with per-PR staging environments. - Set up CI/CD to build and deploy to production Anyone can do this without your company. If that's the value you bring, then I don't need your company. What value do you bring that I can't get somewhere else? Also, red flag: mis-use of the term DevOps for marketing purposes. "There are a few ways of thinking about these files named ‘Layerfile’:" - Auto-discovered Dockerfiles that build entire virtual machines instead of containers, just as quickly. - Define a tree of virtual machines. Each subsequent layer can inherit all the running processes in its parents. - Trigger specific actions like build, push, test, and deploy in parallel every time you push new code. It's not explained well what benefit there is to using VMs rather than containers. Later on it briefly mentions possibly better caching than Docker containers. If it's actually better than Docker, you are severely burying the lede. "Better than Docker" should be on the front page. It's also not explaining whether I need to throw away all the time and money I might have already invested in containers, or what this is and isn't compatible with. A lot of this solution is really built around optimizing one specific problem, which is caching during re-running whole CI pipelines. This limits the viability of your business. As soon as your customers figure out a new way to solve this problem, they don't need your custom platform anymore. I would consider LayerCI more of a specific feature of a much larger offering. (In particular, the 'correc't way to solve this problem is to stand up an environment for your PR and structure your CI so you only re-run the parts that affect the particular change you want to make, rather than re-running the entire pipeline. It involves putting more thought into running your pipeline but often means the only tasks that get executed are copying a file and restarting your app) P.S. You're installing the yarn GPG key and then not doing anything with it. Your node setup script isn't installing the yarn repo or yarn itself, so this is a throwaway command.

colinchartier 6y ago
> In particular, the 'correct way to solve this problem is to stand up an environment for your PR and structure your CI so you only re-run the parts that affect the particular change you want to make, rather than re-running the entire pipeline. It involves putting more thought into running your pipeline but often means the only tasks that get executed are copying a file and restarting your app This is actually exactly what our platform helps with - instead of micromanaging this (and updating the configurations) you can rely on a friendly SaaS to do it for you. I wouldn't call us "better" than Docker, we're more like "Take Docker and make it work better for CI/CD and staging servers"

1337shadow 6y ago

> - Slow (wait 10 minutes to run the same repetitive setup steps like "npm install") That should happen any time you change dependencies if your Dockerfile is efficiently setup, not every time you change a line in runtime code ... Maybe that: COPY . . RUN npm install Is not what you should do to have an efficient docker build phase, did you try this instead ? COPY package.json RUN npm install COPY . . Multistage builds will help if you build in two languages in the same Dockerfile. Anyway, still slow when you add/remove a package because it redownloads everything, my best trick is to share ~/.npm cache directory, but I manage this with buildah instead of docker build.

colinchartier 6y ago
One of the benefits of Layerfiles vs Dockerfiles is that those two examples will do the same thing - we monitor which files are read so you don't have to micromanage copying specific steps. But I suppose "npm install" is a simplistic example, "npm run setup_our_ci_environment" would be a more likely step for a larger CI pipeline
- Too 6y ago
  Impressive, and scary. How does it work? Some kind of ptrace wrapping of each command? If this genuinely works reliably, this alone could be a good product. As someone said below, you should promote this feature more specifically.
  - colinchartier 6y ago
    OS-level VFS wrapper (there are a few places you can put probes here, another approach might be eBPS) We used to sell the features, but that actually didn't work super well - maybe we can add a "technical details" page for the nerds ;)
    - ignoramous 6y ago
      Yes, please. If that impressive hypervisor tech can handle production, I see endless possibilities, including mitigating downtimes by teleporting workloads to different cloud vendors and different regions. Easier said than done, of course!

colinchartier 6y ago

If you'd like to chat with me (CEO @ LayerCI) personally, I'm at colin@layerci.com

old-gregg 6y ago

I see developer tools on HN. I upvote! Congrats on launching! I know this is a good idea because we actually invested time in building something similar internally.

colinchartier 6y ago
Thanks, we're big fans of teleport - funny how small the developer tools world actually is.

tekno45 6y ago

Can we get SSO for not a bajillion dollars? Seems like this functionality is really inflated across the entire SaaS space. Is SSO really that hard? that expensive?

colinchartier 6y ago
Most developers log in to LayerCI via GitHub or GitLab - so if you have a GitHub plan with SSO, you'll get SSO with Layer as well. We even support self-hosted GitLab, so if you use a custom SSO provider there, developers could use that oauth flow for authentication + authorization.

data_ders 6y ago

How does dependency caching configuration work? I get wanting to make things as fast as possible, but I struggle a bunch a Python dep caching in Circle CI... is there a way to ensure a 100% rebuild? Also interested to see how you plan to support open source

colinchartier 6y ago
There's a "CACHE" directive if you want traditional dep caching, but if you don't edit requirements.txt, a Layerfile will just entirely skip the pip install You can "SNAPSHOT disabled" if you'd like certain steps to always run

technics256 6y ago

What's the difference between this and something like hashicorp waypoint?

julienlafond 6y ago

Hi @colinchartier! From what I see your product looks great for running a single app with a ready-to-use staging environment. In practice, a stating environment is often not limited to one app, but at least 1 frontend inter-connected with at least 1 backend (and often, many frontends connected to many micro-services). Does your product respond to this kind of architecture?

colinchartier 6y ago
Yes, it's quite simple if you use something like docker compose. If your frontend/backend are in separate repositories, I'd reference this doc page: https://layerci.com/docs/advanced-workflows/routing https://layerci.com/docs/advanced-workflows/routing

gajus 6y ago

How is this different from the likes of https://vercel.com/ https://vercel.com/ ? It doesn't need to be – but understanding the difference / similarities with other tools that have a similar sounding proposition helps with user acquisition / defining your niche.

colinchartier 6y ago
We're halfway between CircleCI and Vercel - not for production hosting, but easily allowing for backends / databases to run (including a separate backend for every push)
- gajus 6y ago
  Why wouldn't I just deploy with every commit to Vercel? Either using GitHub actions or GitLab CI/CD?
  - colinchartier 6y ago
    It takes significantly longer - we have advanced snapshotting logic to skip steps like "start the database" that you'd otherwise have to run every time you pushed. We also auto-hibernate the instances, so you don't have to pay for a bunch of separate databases running in parallel. Don't just listen to me though, try the free tier for yourself! A lot of our users combine LayerCI for developer tooling and vercel/netlify/GCP/AWS for deployment, they're quite complimentary.
nrmitchi 6y ago
Many production-hosting solutions offer similar functionality, but the catch is that they only really work if you're already using that system. If you're already deploying on Vercel, then you likely just want to use Vercel's solution. The same goes for Netlify Preview Deploys, Heroku Review Apps, Render Pull Request Previews, etc. If you are not using one of these PaaS's, trying to tap into one for just this functionality tends to end up rather convoluted, and somewhat defeats the purpose.

mcintyre1994 6y ago

This sounds really exciting. I love your live demo, really slick introduction and tutorial - and answered some questions I had about E2E! I've got a couple of other questions though. > 12 full stack preview environments per commit with up to 16GB of memory and 6 CPUs Quote from your pricing. If we had a monorepo with over 12 small test/integration jobs, but not 12 full preview environments, is this usable? Are they one and the same, or can we have eg. a suite of unit tests that don't count toward this full preview limit? Do some teams keep some unit tests off Layer and just use you for the more interesting pieces? Second, do you have any documentation about which databases you support? Concretely can you restore a MongoDB snapshot super-fast? Third, do you have any story around secrets if we want the staging server to hold some secret API keys? Currently we can do this with AWS (own account) CI machines in Gitlab deploying to ECS with AWS secrets - they stay end-to-end encrypted and nobody sees them. Is there any similar way we can get secrets onto a staging server without you having access to them? I suspect this would be a deal breaker for the staging use case for my team. Similar to above, our database snapshots are stored in non-public S3 buckets, how would that work? Again currently it's a case of giving the CI AWS role permission to access them, not sure what an equivalent would look like. One more, is it possible to access the built docker images? We deploy Docker images to ECS, and currently they're the exact same ones built and tested in CI which is a nice reassurance. Do your customers have an out-of-band process for building + deploying to production outside of Layer? Aside, I think this may be a typo on your pricing page? "We'll never increase your the terms of your bill once you start your subscription."

andrejguran 6y ago
I am curious about the third point of e2e encryption of the secrets. The problem is that the hosting provider would see plaintext secrets or that someone with access to AWS account could see them? You could also input the encrypted secrets theoretically into any service and then decrypt them in your application. Just curious on what kind of considerations do you have when choosing a hosting provider.
- mcintyre1994 6y ago
  It's basically the same concern as having them in source code, we want to be able to control/restrict access to them. In AWS we can do that becase they're encrypted with a KMS key and unless a user/role has permission to decrypt with that key it can't decrypt the secret. This does assume that AWS aren't lying about encrypting things, and that their employees can't access our KMS keys. We currently use Gitlab for our CI/CD pipelines, but using our own runners in our own AWS account. So if we want to deploy a staging environment from there, it's actually deploying from an AWS role that we control, we're not leaking any secrets to Gitlab or anywhere else. I'm just wondering how people get around this in Layer/any hosted CI/CD setup where you can't have your own runners inside your AWS account. Especially because they don't replace the production deploy, so ultimately those secrets are staying in AWS - perhaps in addition to wherever they need to go with Layer. FWIW I did find something about secrets in the docs: https://layerci.com/docs/layerfile-reference/secret-env#secret-env https://layerci.com/docs/layerfile-reference/secret-env#secr... - but I'm not sure how you get secrets in from that.

pdeva1 6y ago

how do you know which steps to skip or rerun each run. in your react example, how do you know when to reinstall yarn to latest version and when to skip. https://layerci.com/docs/examples/react https://layerci.com/docs/examples/react

colinchartier 6y ago
We monitor which files are read by which steps and map that back to the changes you've made automatically!

danpalmer 6y ago

This looks great. I’m interested in your approach to databases and specifically the data in staging environments? I’ve found setting up the data in databases for staging environments can take as much time as CI, and often works in quite a different way. Shared databases can work, but then may limit staging changes that include migrations.

colinchartier 6y ago
We'll snapshot the state of the database after you set it up once, and then re-load that snapshot every following time you need the database to exist.