5 ms·

It was honestly a bit hard for me to understand what happened from this write-up. I read it a few times, and the terms are still bit confusing. Seems like the

by chubot 3y ago

It was honestly a bit hard for me to understand what happened from this write-up. I read it a few times, and the terms are still bit confusing. Seems like there are at least two separate issues - migration of the CI configuration -- from existing CI YAML (github/gitlab) to Earthly (Makefile/Dockerfile hybrid) - migration of the job runners -- from existing CI to a service hosted by Earthly I would have thought the first thing is the hard thing. Changing the language could take months or even years. But I don't understand what the blog post was saying. I thought they validated that part? People were able to switch? But then it wasn't validated at the end? The customers were faced with 2 migrations, not 1? But now what's happening? You're still keeping the Earthly syntax but abandoning the CI? Isn't that the hard part to migrate to? I'm still confused.

Terretta 3y ago
It's not about migrating the syntax. It's that people's CI over time become some kind of amalgamated model encapsulating how a firm makes every individual piece of software it makes and lands it places, think (ab)using a CI as Airflow (arbitrary automation job runner), and that the migration is first reverse engineering what people used to know, before even starting untangling all of that to express it some different way. Nobody really wants to stop the world and sort that.
- aftbit 3y ago
  I think the key is that nobody "wants" to spend any time at all on CI or their build system. It is one of those necessary evils that you have to do to have functioning software. Once you do it, even if it's a slow buggy pile of hacks, you will just ignore it until the pain gets very very bad. This was the same problem that caused our team to abandon CircleCI. They kept wanting us to rewrite our CI configs, touting various benefits that we just didn't care about. All we wanted was for our janky broken builds to keep working 90% of the time, like they did before Circle tried migrating this or that.
  - wpietri 3y ago
    Some people do want to spend time on that! It depends whether you see fast feedback as a competitive advantage. I once worked on a system where we had excellent test coverage and all our unit tests ran in under a minute. We worked hard to keep it that way, because it was a huge boon to productivity, especially when improving design issues. It also enabled frequent commits and merges, which made collaboration easier. It was a great experience. But yes, from the business perspective, the leading edge is a pretty small audience. (Circa 15%, per Moore.) And the bottom half of the audience by competence, which Moore called the Late Majority and the Laggards, can be well more than half your revenue in a SaaS context because they will create problems for themselves (like slow/buggy builds) that mean higher per-user revenue. And they may also have special feature needs where they'll pay more rather than clean up their messes. In that context, doing the right thing for your business can be very different than doing the right thing for users. Sort of like a physical therapist who makes most of their money from renting crutches and in-home meal delivery, and so doesn't push people to actually get better.
  - aeyes 3y ago
    What did they make you change? I have been using CircleCI since the beginning and the only breaking change I saw was when they introduced workflows. This didn't need any changes to the difficult parts of my own code and was mostly copy-paste. I believe that my configs are fairly complex using almost all existing features across many different projects and build targets.
    - aftbit 3y ago
      We started using CircleCI in early 2016, I believe. There have been two major migration points. The first time was when they went from circle.yml to .circleci/config.yml, around 2018, which might have corresponded with Workflows. The second time was when they got rid of their old converged docker containers and moved to custom container runtimes (and a new set of images, none of which did _quite_ the same thing as their old images). Neither of these took tons of effort to resolve, but both broke our builds and made us spend time thinking about CI when we would rather have been iterating on our products. When their later migration happened, Github Actions was just coming around, so we figured that if we had to rebuild and re-test all of our CI, we might as well move it to a more reliable and cheaper runner. Now we only use CircleCI on a handful of legacy repos.
- danudey 3y ago
  I started work in December at an awesome company as a build/release engineer. Our current release workflow for our open-source components: 1. Run the CI build/test pipeline 2. If it passes, run the 'Publish' promotion (manually, on any pipeline you want) 3. The pipeline runs a make target 4. The make target starts up a VM (from the VM it's running on) 5. The make target SSH'es to the VM to run a make target 4. The make target builds a binary 5. The binary runs more make targets 6. The make targets run various scripts in docker containers on the VM the other VM started 7. The scripts in the docker containers build the components 8. The binary from step five runs more make targets to publish the releases 9. We test everything with another promotion which runs make targets which build docker containers to run python scripts (pytest) This is also built by a complicated web of wildcarded makefile targets, which need to be interoperable and support a few if/else cases for specific components. My plan is to migrate all of this to something simpler and more straightforward, or at least more maintainable, which is honestly probably going to turn into taskfile[0] instead of makefiles, and then simple python scripts for the glue that ties everything together or does more complex logic. My hope is that it can be more straightforward and easier to maintain, with more component-ized logic, but realistically every step in that labyrinthine build process (and that's just the open-source version!) came from a decision made by a very talented team of engineers who know far more about the process and the product than I do. At this point I'm wondering if it would make 'more sense' to replace it with a giant python script of some kind and get access to all the logic we need all at once (it would not). [0] https://taskfile.dev/ https://taskfile.dev/
  - 392 3y ago
    Try buck2
  - pimlottc 3y ago
    HN collapses lists, you might want to add some newlines between the steps for better readability.
  - hedora 3y ago
    Taskfile looks worse than make to me in every possible dimension (except maybe windows compatibility, but I don't believe that it meaningfully supports that). You should at least read and understand the paper "Recursive Make Considered Harmful" before attempting to replace make with something "better": https://aegis.sourceforge.net/auug97.pdf https://aegis.sourceforge.net/auug97.pdf Most people use make incorrectly (and it sounds like the system you describe makes most of the classic mistakes). The paper I linked explains how to use Make's templating language to help it scale to complicated setups. Here are a few critiques of taskfile (from skimming their documentation): - The syntax is incredibly verbose and weird. For instance, you have to write a tree of yaml nodes to loop over a set of files. Make's syntax is weird, but at least it is not verbose. - The documentation makes no mention that I could find of deriving a set of tasks from a list of files (Make's $(wildcard) macro, or stemming rules like "%.o : %.c"). So, I guess the taskfile will end up being huge (like the size of autogenerated makefiles from autoconf or cmake), or you'll end up using additional ad-hoc build systems that are invoked by your taskfile. - From the documentation, they haven't thought through having more than one project per unix account "When running your global Taskfile with -g, tasks will run on $HOME by default, and not on your working directory!" - They seem to have partially reimplemented the bad ideas from recursive make: https://taskfile.dev/usage/#including-other-taskfiles https://taskfile.dev/usage/#including-other-taskfiles - I suggest taking an idiomatic but non-trivial bash script (which is 99% of what make is invoking, and that taskfiles support), and then trying to port it to python directly. There's usually a 10-100x line of code blowup from doing that, and the python thing usually bitrots every year or so (vs. every few decades for shell).
- Spivak 3y ago
  The space was destined to collapse these two products not because it makes sense conceptually but because their implementations are almost identical and maintaining fewer systems is great. One was always going to eat the other and CI systems are turnkey, hosted, and live close to the code that it will be running. I wish they had either remained separate or gone the other way but it's not worth swimming upstream for it.
- chubot 3y ago
  OK but my point is that should have been apparent at the step 1 validation. Seems like they are onto step 2 or 3, and figured out that step 1 wasn't actually validated -- people didn't want to migrate to Earthly files ??? The blog post is not clear about what happened
wpietri 3y ago
I'm glad it wasn't just me. I'm interested in both sides of this, the product development side and the dev tools side. But with all that somewhat confusing prose, I ended up in the neighborhood of: "I'm happy for u tho. Or sorry that happened." Was the heart of this that they assumed that radically faster builds was the killer feature, but they hadn't actually validated that assumption before building it? Or was it more they hadn't really segmented their users, so they hadn't realized that the high-dollar customers had different needs? Or that they just gave away the thing that was the real-problem solver for paying customers? And given that the somewhat chaotic article was written by the CEO, I have to wonder to what extent the chaos was just a lack of editing versus there being a lot of actual chaos in the company throughout the process. Long ago Steve Blank wrote an article called "Founders and dysfunctional families" [1] where he talked about how a lot of founders are really good at managing chaos because they grew up in it, something that certainly describes me. He added that something that can make the difference between success and failure is the extent to which founders can also handle non-chaos. Because the ones who can't tend to "throw organizational hand grenades into their own companies" to get things back to the level of chaos that they're good at. That observation has given me pause for thought many times in the years since. [1] https://steveblank.com/2009/05/18/founders-and-dysfunctional-families/ https://steveblank.com/2009/05/18/founders-and-dysfunctional...

by chubot 3y ago

Terretta 3y ago

It's not about migrating the syntax. It's that people's CI over time become some kind of amalgamated model encapsulating how a firm makes every individual piece of software it makes and lands it places, think (ab)using a CI as Airflow (arbitrary automation job runner), and that the migration is first reverse engineering what people used to know, before even starting untangling all of that to express it some different way. Nobody really wants to stop the world and sort that.

aftbit 3y ago
I think the key is that nobody "wants" to spend any time at all on CI or their build system. It is one of those necessary evils that you have to do to have functioning software. Once you do it, even if it's a slow buggy pile of hacks, you will just ignore it until the pain gets very very bad. This was the same problem that caused our team to abandon CircleCI. They kept wanting us to rewrite our CI configs, touting various benefits that we just didn't care about. All we wanted was for our janky broken builds to keep working 90% of the time, like they did before Circle tried migrating this or that.
- wpietri 3y ago
  Some people do want to spend time on that! It depends whether you see fast feedback as a competitive advantage. I once worked on a system where we had excellent test coverage and all our unit tests ran in under a minute. We worked hard to keep it that way, because it was a huge boon to productivity, especially when improving design issues. It also enabled frequent commits and merges, which made collaboration easier. It was a great experience. But yes, from the business perspective, the leading edge is a pretty small audience. (Circa 15%, per Moore.) And the bottom half of the audience by competence, which Moore called the Late Majority and the Laggards, can be well more than half your revenue in a SaaS context because they will create problems for themselves (like slow/buggy builds) that mean higher per-user revenue. And they may also have special feature needs where they'll pay more rather than clean up their messes. In that context, doing the right thing for your business can be very different than doing the right thing for users. Sort of like a physical therapist who makes most of their money from renting crutches and in-home meal delivery, and so doesn't push people to actually get better.
- aeyes 3y ago
  What did they make you change? I have been using CircleCI since the beginning and the only breaking change I saw was when they introduced workflows. This didn't need any changes to the difficult parts of my own code and was mostly copy-paste. I believe that my configs are fairly complex using almost all existing features across many different projects and build targets.
  - aftbit 3y ago
    We started using CircleCI in early 2016, I believe. There have been two major migration points. The first time was when they went from circle.yml to .circleci/config.yml, around 2018, which might have corresponded with Workflows. The second time was when they got rid of their old converged docker containers and moved to custom container runtimes (and a new set of images, none of which did _quite_ the same thing as their old images). Neither of these took tons of effort to resolve, but both broke our builds and made us spend time thinking about CI when we would rather have been iterating on our products. When their later migration happened, Github Actions was just coming around, so we figured that if we had to rebuild and re-test all of our CI, we might as well move it to a more reliable and cheaper runner. Now we only use CircleCI on a handful of legacy repos.
danudey 3y ago
I started work in December at an awesome company as a build/release engineer. Our current release workflow for our open-source components: 1. Run the CI build/test pipeline 2. If it passes, run the 'Publish' promotion (manually, on any pipeline you want) 3. The pipeline runs a make target 4. The make target starts up a VM (from the VM it's running on) 5. The make target SSH'es to the VM to run a make target 4. The make target builds a binary 5. The binary runs more make targets 6. The make targets run various scripts in docker containers on the VM the other VM started 7. The scripts in the docker containers build the components 8. The binary from step five runs more make targets to publish the releases 9. We test everything with another promotion which runs make targets which build docker containers to run python scripts (pytest) This is also built by a complicated web of wildcarded makefile targets, which need to be interoperable and support a few if/else cases for specific components. My plan is to migrate all of this to something simpler and more straightforward, or at least more maintainable, which is honestly probably going to turn into taskfile[0] instead of makefiles, and then simple python scripts for the glue that ties everything together or does more complex logic. My hope is that it can be more straightforward and easier to maintain, with more component-ized logic, but realistically every step in that labyrinthine build process (and that's just the open-source version!) came from a decision made by a very talented team of engineers who know far more about the process and the product than I do. At this point I'm wondering if it would make 'more sense' to replace it with a giant python script of some kind and get access to all the logic we need all at once (it would not). [0] https://taskfile.dev/ https://taskfile.dev/
- 392 3y ago
  Try buck2
- pimlottc 3y ago
  HN collapses lists, you might want to add some newlines between the steps for better readability.
- hedora 3y ago
  Taskfile looks worse than make to me in every possible dimension (except maybe windows compatibility, but I don't believe that it meaningfully supports that). You should at least read and understand the paper "Recursive Make Considered Harmful" before attempting to replace make with something "better": https://aegis.sourceforge.net/auug97.pdf https://aegis.sourceforge.net/auug97.pdf Most people use make incorrectly (and it sounds like the system you describe makes most of the classic mistakes). The paper I linked explains how to use Make's templating language to help it scale to complicated setups. Here are a few critiques of taskfile (from skimming their documentation): - The syntax is incredibly verbose and weird. For instance, you have to write a tree of yaml nodes to loop over a set of files. Make's syntax is weird, but at least it is not verbose. - The documentation makes no mention that I could find of deriving a set of tasks from a list of files (Make's $(wildcard) macro, or stemming rules like "%.o : %.c"). So, I guess the taskfile will end up being huge (like the size of autogenerated makefiles from autoconf or cmake), or you'll end up using additional ad-hoc build systems that are invoked by your taskfile. - From the documentation, they haven't thought through having more than one project per unix account "When running your global Taskfile with -g, tasks will run on $HOME by default, and not on your working directory!" - They seem to have partially reimplemented the bad ideas from recursive make: https://taskfile.dev/usage/#including-other-taskfiles https://taskfile.dev/usage/#including-other-taskfiles - I suggest taking an idiomatic but non-trivial bash script (which is 99% of what make is invoking, and that taskfiles support), and then trying to port it to python directly. There's usually a 10-100x line of code blowup from doing that, and the python thing usually bitrots every year or so (vs. every few decades for shell).
Spivak 3y ago
The space was destined to collapse these two products not because it makes sense conceptually but because their implementations are almost identical and maintaining fewer systems is great. One was always going to eat the other and CI systems are turnkey, hosted, and live close to the code that it will be running. I wish they had either remained separate or gone the other way but it's not worth swimming upstream for it.
chubot 3y ago
OK but my point is that should have been apparent at the step 1 validation. Seems like they are onto step 2 or 3, and figured out that step 1 wasn't actually validated -- people didn't want to migrate to Earthly files ??? The blog post is not clear about what happened

wpietri 3y ago

I'm glad it wasn't just me. I'm interested in both sides of this, the product development side and the dev tools side. But with all that somewhat confusing prose, I ended up in the neighborhood of: "I'm happy for u tho. Or sorry that happened." Was the heart of this that they assumed that radically faster builds was the killer feature, but they hadn't actually validated that assumption before building it? Or was it more they hadn't really segmented their users, so they hadn't realized that the high-dollar customers had different needs? Or that they just gave away the thing that was the real-problem solver for paying customers? And given that the somewhat chaotic article was written by the CEO, I have to wonder to what extent the chaos was just a lack of editing versus there being a lot of actual chaos in the company throughout the process. Long ago Steve Blank wrote an article called "Founders and dysfunctional families" [1] where he talked about how a lot of founders are really good at managing chaos because they grew up in it, something that certainly describes me. He added that something that can make the difference between success and failure is the extent to which founders can also handle non-chaos. Because the ones who can't tend to "throw organizational hand grenades into their own companies" to get things back to the level of chaos that they're good at. That observation has given me pause for thought many times in the years since. [1] https://steveblank.com/2009/05/18/founders-and-dysfunctional-families/ https://steveblank.com/2009/05/18/founders-and-dysfunctional...