6 ms·

It's interesting to see CI wait times as core engineering metric. I 100% agree, so much so that I'm building a product specifically to speed up CI. We have rede

It's interesting to see CI wait times as core engineering metric. I 100% agree, so much so that I'm building a product specifically to speed up CI. We have redesigned how hosted CI works focusing on speed as our north star. We don't have CI wait time - your test starts immediately and we run your tests super fast. How do we do it? We have many workers with your test environment pre-built so we can split your tests and start running your tests in seconds. If anyone is interested you can check it out at https://brisktest.com/ https://brisktest.com/, I'd love to hear any feedback from the community.

andreareina 4y ago
It looks like you're focused on ruby and javascript at the moment, is python on the roadmap and how far down?
- reillyse 4y ago
  Python is actually on the roadmap, it's next up. Is it something you are interested in? We are in the process of trialing our solution with one other company, I'd love to add you to our beta test if that was something you'd be interested in ?
  - andreareina 4y ago
    Sent you an email
corytheboyd 4y ago
Most slow test suites I have seen aren't wasting hours waiting for runners to free up, they're just plain inefficient. Most common causes seem to be (roughly in order): - using factorybot and the factories do WAY too much - churning away retrying flaky tests, usually caused by global state issues - no parallelization. use those damn resources you're paying tons of money for already!!! Just my experience though, working for a handful of web development orgs over the years. Rails and JavaScript.
- reillyse 4y ago
  I think you'd be surprised how long some tests queue waiting for a runner - but I do agree with you that it's not the longest delay running tests, and it is relatively easy to fix with budget. I would say our solution mostly breaks down to massive parallelization and in order to support this pre-built environments. The problem with massive parallelization with traditional hosted CI is that the first 5-6 minutes (maybe as many as 10 minutes) of your build is just creating your environment. So it is extremely inefficient to spend such a long time building for say 2 minutes of tests. Your total test time is 6 minutes setup + 2 minutes run time. Scaling that up by adding 100 machines is only going to bring the 2 minutes down, you are stuck with the 6 minutes build time. By maintaining the environment between builds we can add more workers and get that test run time down to the 10 seconds range (with zero build time on most builds). Obviously we add complexity around knowing when to rebuild - but that is the price we pay for getting super fast tests.
  - quesera 4y ago
    FWIW, my big Rails test suite spends about 2 minutes on setup, then about 40 minutes running with a single runner. It parallelizes well, but of course the 2 mins is the lower bound.
    - reillyse 4y ago
      Yea, that sounds somewhat normal. I see builds getting bigger when you add in a bunch of yarn packages, building assets and webpacker etc. As projects get larger these tend to bloat up. You might be interested in looking at this demo of a Rails Rspec/Capybara test suite (https://brisktest.com/demos#rails https://brisktest.com/demos#rails) it's over 60 minutes locally on my macbook, we get it down to 90 seconds on brisk. It's got a lot of Selenium Chrome tests which is why it's so long (90 seconds is long for us), and I think fixing their flakiness would shave a lot off those seconds (lots of rspec retry going on) but it's a good example of a real-world test suite. Often fixing flaky tests is not the priority and getting your tests run in 90 seconds is huge.
  - Aeolun 4y ago
quesera 4y ago
Tell us more about your prebuilt environments. Are they reused between customers? How do you protect your customers' code and secret tokens? What does the customer have to do to parallelize their tests? Separate them into individual (or groups of) small test (rspec etc) files? Or is there some automation magic? Pricing is reasonable, but there's a big gap between $0 (5x) and $129/mo (60x). I suspect you want to drive customers to the "Wow that's great" performance level, at the "Yeah that's reasonable" price, because this optimizes your revenue to costs, while making customers happier than they expected to be. (This is a good strategy!) But might there be some space for "Yeah that's really good" performance at a "Wow that's affordable" tier? 20x for $49/mo? Nice service BTW. The SaaS CI market is ossified and needs a good shakeup!
- reillyse 4y ago
  Thanks, the kind words really do mean a lot! So each prebuilt environment is basically a container that is exclusively accessed by your test runner. We set up the environment in the container and code and tests are run there. So there is no sharing of the build environment between customers. We are pretty paranoid about protecting people's code (as you might expect). We do all the "normal things", encrypt connections, etc. One novelish idea we use is that different customers' code never shares a binary. So, what that means is that the CLI connects through a load balancer directly to it's dedicated server (a binary running in it's own container) which connects to it's dedicated workers (more binaries in their own containers). Once we are finished with a server or worker we destroy the entire container. We have multiple layers of security, and for example once a customer has connected to a container, only a customer with the correct key can connect again (so it should be the same customer). We also offer on-prem for people who need more security wise. I feel we are a long way from getting the pricing right. At the moment most people are on the free tier just trying things out and getting started. I think as we grow we'll probably fill the space between the two tiers as you suggest. But to be honest, the details of exactly what this will cost us at scale is still very much open. Usage rates and test suite size are very variable and until we have a good idea on that we are just trying to make the best guesses we can and try not to shoot ourselves in the foot too much :)
  - 4y ago

andreareina 4y ago

It looks like you're focused on ruby and javascript at the moment, is python on the roadmap and how far down?

reillyse 4y ago
Python is actually on the roadmap, it's next up. Is it something you are interested in? We are in the process of trialing our solution with one other company, I'd love to add you to our beta test if that was something you'd be interested in ?
- andreareina 4y ago
  Sent you an email

corytheboyd 4y ago

Most slow test suites I have seen aren't wasting hours waiting for runners to free up, they're just plain inefficient. Most common causes seem to be (roughly in order): - using factorybot and the factories do WAY too much - churning away retrying flaky tests, usually caused by global state issues - no parallelization. use those damn resources you're paying tons of money for already!!! Just my experience though, working for a handful of web development orgs over the years. Rails and JavaScript.

reillyse 4y ago
I think you'd be surprised how long some tests queue waiting for a runner - but I do agree with you that it's not the longest delay running tests, and it is relatively easy to fix with budget. I would say our solution mostly breaks down to massive parallelization and in order to support this pre-built environments. The problem with massive parallelization with traditional hosted CI is that the first 5-6 minutes (maybe as many as 10 minutes) of your build is just creating your environment. So it is extremely inefficient to spend such a long time building for say 2 minutes of tests. Your total test time is 6 minutes setup + 2 minutes run time. Scaling that up by adding 100 machines is only going to bring the 2 minutes down, you are stuck with the 6 minutes build time. By maintaining the environment between builds we can add more workers and get that test run time down to the 10 seconds range (with zero build time on most builds). Obviously we add complexity around knowing when to rebuild - but that is the price we pay for getting super fast tests.
- quesera 4y ago
  FWIW, my big Rails test suite spends about 2 minutes on setup, then about 40 minutes running with a single runner. It parallelizes well, but of course the 2 mins is the lower bound.
  - reillyse 4y ago
    Yea, that sounds somewhat normal. I see builds getting bigger when you add in a bunch of yarn packages, building assets and webpacker etc. As projects get larger these tend to bloat up. You might be interested in looking at this demo of a Rails Rspec/Capybara test suite (https://brisktest.com/demos#rails https://brisktest.com/demos#rails) it's over 60 minutes locally on my macbook, we get it down to 90 seconds on brisk. It's got a lot of Selenium Chrome tests which is why it's so long (90 seconds is long for us), and I think fixing their flakiness would shave a lot off those seconds (lots of rspec retry going on) but it's a good example of a real-world test suite. Often fixing flaky tests is not the priority and getting your tests run in 90 seconds is huge.
- Aeolun 4y ago

quesera 4y ago

Tell us more about your prebuilt environments. Are they reused between customers? How do you protect your customers' code and secret tokens? What does the customer have to do to parallelize their tests? Separate them into individual (or groups of) small test (rspec etc) files? Or is there some automation magic? Pricing is reasonable, but there's a big gap between $0 (5x) and $129/mo (60x). I suspect you want to drive customers to the "Wow that's great" performance level, at the "Yeah that's reasonable" price, because this optimizes your revenue to costs, while making customers happier than they expected to be. (This is a good strategy!) But might there be some space for "Yeah that's really good" performance at a "Wow that's affordable" tier? 20x for $49/mo? Nice service BTW. The SaaS CI market is ossified and needs a good shakeup!

reillyse 4y ago
Thanks, the kind words really do mean a lot! So each prebuilt environment is basically a container that is exclusively accessed by your test runner. We set up the environment in the container and code and tests are run there. So there is no sharing of the build environment between customers. We are pretty paranoid about protecting people's code (as you might expect). We do all the "normal things", encrypt connections, etc. One novelish idea we use is that different customers' code never shares a binary. So, what that means is that the CLI connects through a load balancer directly to it's dedicated server (a binary running in it's own container) which connects to it's dedicated workers (more binaries in their own containers). Once we are finished with a server or worker we destroy the entire container. We have multiple layers of security, and for example once a customer has connected to a container, only a customer with the correct key can connect again (so it should be the same customer). We also offer on-prem for people who need more security wise. I feel we are a long way from getting the pricing right. At the moment most people are on the free tier just trying things out and getting started. I think as we grow we'll probably fill the space between the two tiers as you suggest. But to be honest, the details of exactly what this will cost us at scale is still very much open. Usage rates and test suite size are very variable and until we have a good idea on that we are just trying to make the best guesses we can and try not to shoot ourselves in the foot too much :)
- 4y ago