3 ms·

It's not about migrating the syntax. It's that people's CI over time become some kind of amalgamated model encapsulating how a firm makes every individual piec

It's not about migrating the syntax. It's that people's CI over time become some kind of amalgamated model encapsulating how a firm makes every individual piece of software it makes and lands it places, think (ab)using a CI as Airflow (arbitrary automation job runner), and that the migration is first reverse engineering what people used to know, before even starting untangling all of that to express it some different way. Nobody really wants to stop the world and sort that.

aftbit 3y ago
I think the key is that nobody "wants" to spend any time at all on CI or their build system. It is one of those necessary evils that you have to do to have functioning software. Once you do it, even if it's a slow buggy pile of hacks, you will just ignore it until the pain gets very very bad. This was the same problem that caused our team to abandon CircleCI. They kept wanting us to rewrite our CI configs, touting various benefits that we just didn't care about. All we wanted was for our janky broken builds to keep working 90% of the time, like they did before Circle tried migrating this or that.
- wpietri 3y ago
  Some people do want to spend time on that! It depends whether you see fast feedback as a competitive advantage. I once worked on a system where we had excellent test coverage and all our unit tests ran in under a minute. We worked hard to keep it that way, because it was a huge boon to productivity, especially when improving design issues. It also enabled frequent commits and merges, which made collaboration easier. It was a great experience. But yes, from the business perspective, the leading edge is a pretty small audience. (Circa 15%, per Moore.) And the bottom half of the audience by competence, which Moore called the Late Majority and the Laggards, can be well more than half your revenue in a SaaS context because they will create problems for themselves (like slow/buggy builds) that mean higher per-user revenue. And they may also have special feature needs where they'll pay more rather than clean up their messes. In that context, doing the right thing for your business can be very different than doing the right thing for users. Sort of like a physical therapist who makes most of their money from renting crutches and in-home meal delivery, and so doesn't push people to actually get better.
- aeyes 3y ago
  What did they make you change? I have been using CircleCI since the beginning and the only breaking change I saw was when they introduced workflows. This didn't need any changes to the difficult parts of my own code and was mostly copy-paste. I believe that my configs are fairly complex using almost all existing features across many different projects and build targets.
  - aftbit 3y ago
    We started using CircleCI in early 2016, I believe. There have been two major migration points. The first time was when they went from circle.yml to .circleci/config.yml, around 2018, which might have corresponded with Workflows. The second time was when they got rid of their old converged docker containers and moved to custom container runtimes (and a new set of images, none of which did _quite_ the same thing as their old images). Neither of these took tons of effort to resolve, but both broke our builds and made us spend time thinking about CI when we would rather have been iterating on our products. When their later migration happened, Github Actions was just coming around, so we figured that if we had to rebuild and re-test all of our CI, we might as well move it to a more reliable and cheaper runner. Now we only use CircleCI on a handful of legacy repos.
- notatoad 3y ago
  >I think the key is that nobody "wants" to spend any time at all on CI or their build system. or if they do, they're hosting their own. if you're selling a hosted CI platform, your sales pitch is "pay us $$ instead of paying your staff $$$ to do devops". if your pitch is "pay us $$ and then also spend $$$ on staff hours to manage our service", it's very unlikely the CI is doing anything useful enough that the staff couldn't just run their own thing in-house. it doesn't take much configuration to be more difficult than an EC2 instance running a script on a webhook from your version control.
- larusso 3y ago
  I had a talk with a circle CI rep at a conference and I tried to explain to him that yes I’m not happy with our custom Jenkins setup (agents, CasC, pipeline library), but that I also want to spend time and money on a migration project to end up at the status co again. If I would go to my boss and say we can either spend some time and fix our setup or spend time plus running costs for this external solution then the answer will be clear. I think convincing teams with huge stacks of CI setup to just switch because of marketing promises isn’t working. I would think of using a new setup when I start from scratch. At the moment at least.
danudey 3y ago
I started work in December at an awesome company as a build/release engineer. Our current release workflow for our open-source components: 1. Run the CI build/test pipeline 2. If it passes, run the 'Publish' promotion (manually, on any pipeline you want) 3. The pipeline runs a make target 4. The make target starts up a VM (from the VM it's running on) 5. The make target SSH'es to the VM to run a make target 4. The make target builds a binary 5. The binary runs more make targets 6. The make targets run various scripts in docker containers on the VM the other VM started 7. The scripts in the docker containers build the components 8. The binary from step five runs more make targets to publish the releases 9. We test everything with another promotion which runs make targets which build docker containers to run python scripts (pytest) This is also built by a complicated web of wildcarded makefile targets, which need to be interoperable and support a few if/else cases for specific components. My plan is to migrate all of this to something simpler and more straightforward, or at least more maintainable, which is honestly probably going to turn into taskfile[0] instead of makefiles, and then simple python scripts for the glue that ties everything together or does more complex logic. My hope is that it can be more straightforward and easier to maintain, with more component-ized logic, but realistically every step in that labyrinthine build process (and that's just the open-source version!) came from a decision made by a very talented team of engineers who know far more about the process and the product than I do. At this point I'm wondering if it would make 'more sense' to replace it with a giant python script of some kind and get access to all the logic we need all at once (it would not). [0] https://taskfile.dev/ https://taskfile.dev/
- 392 3y ago
  Try buck2
- pimlottc 3y ago
  HN collapses lists, you might want to add some newlines between the steps for better readability.
- hedora 3y ago
  Taskfile looks worse than make to me in every possible dimension (except maybe windows compatibility, but I don't believe that it meaningfully supports that). You should at least read and understand the paper "Recursive Make Considered Harmful" before attempting to replace make with something "better": https://aegis.sourceforge.net/auug97.pdf https://aegis.sourceforge.net/auug97.pdf Most people use make incorrectly (and it sounds like the system you describe makes most of the classic mistakes). The paper I linked explains how to use Make's templating language to help it scale to complicated setups. Here are a few critiques of taskfile (from skimming their documentation): - The syntax is incredibly verbose and weird. For instance, you have to write a tree of yaml nodes to loop over a set of files. Make's syntax is weird, but at least it is not verbose. - The documentation makes no mention that I could find of deriving a set of tasks from a list of files (Make's $(wildcard) macro, or stemming rules like "%.o : %.c"). So, I guess the taskfile will end up being huge (like the size of autogenerated makefiles from autoconf or cmake), or you'll end up using additional ad-hoc build systems that are invoked by your taskfile. - From the documentation, they haven't thought through having more than one project per unix account "When running your global Taskfile with -g, tasks will run on $HOME by default, and not on your working directory!" - They seem to have partially reimplemented the bad ideas from recursive make: https://taskfile.dev/usage/#including-other-taskfiles https://taskfile.dev/usage/#including-other-taskfiles - I suggest taking an idiomatic but non-trivial bash script (which is 99% of what make is invoking, and that taskfiles support), and then trying to port it to python directly. There's usually a 10-100x line of code blowup from doing that, and the python thing usually bitrots every year or so (vs. every few decades for shell).
Spivak 3y ago
The space was destined to collapse these two products not because it makes sense conceptually but because their implementations are almost identical and maintaining fewer systems is great. One was always going to eat the other and CI systems are turnkey, hosted, and live close to the code that it will be running. I wish they had either remained separate or gone the other way but it's not worth swimming upstream for it.
chubot 3y ago
OK but my point is that should have been apparent at the step 1 validation. Seems like they are onto step 2 or 3, and figured out that step 1 wasn't actually validated -- people didn't want to migrate to Earthly files ??? The blog post is not clear about what happened

aftbit 3y ago

I think the key is that nobody "wants" to spend any time at all on CI or their build system. It is one of those necessary evils that you have to do to have functioning software. Once you do it, even if it's a slow buggy pile of hacks, you will just ignore it until the pain gets very very bad. This was the same problem that caused our team to abandon CircleCI. They kept wanting us to rewrite our CI configs, touting various benefits that we just didn't care about. All we wanted was for our janky broken builds to keep working 90% of the time, like they did before Circle tried migrating this or that.

wpietri 3y ago
Some people do want to spend time on that! It depends whether you see fast feedback as a competitive advantage. I once worked on a system where we had excellent test coverage and all our unit tests ran in under a minute. We worked hard to keep it that way, because it was a huge boon to productivity, especially when improving design issues. It also enabled frequent commits and merges, which made collaboration easier. It was a great experience. But yes, from the business perspective, the leading edge is a pretty small audience. (Circa 15%, per Moore.) And the bottom half of the audience by competence, which Moore called the Late Majority and the Laggards, can be well more than half your revenue in a SaaS context because they will create problems for themselves (like slow/buggy builds) that mean higher per-user revenue. And they may also have special feature needs where they'll pay more rather than clean up their messes. In that context, doing the right thing for your business can be very different than doing the right thing for users. Sort of like a physical therapist who makes most of their money from renting crutches and in-home meal delivery, and so doesn't push people to actually get better.
aeyes 3y ago
What did they make you change? I have been using CircleCI since the beginning and the only breaking change I saw was when they introduced workflows. This didn't need any changes to the difficult parts of my own code and was mostly copy-paste. I believe that my configs are fairly complex using almost all existing features across many different projects and build targets.
- aftbit 3y ago
  We started using CircleCI in early 2016, I believe. There have been two major migration points. The first time was when they went from circle.yml to .circleci/config.yml, around 2018, which might have corresponded with Workflows. The second time was when they got rid of their old converged docker containers and moved to custom container runtimes (and a new set of images, none of which did _quite_ the same thing as their old images). Neither of these took tons of effort to resolve, but both broke our builds and made us spend time thinking about CI when we would rather have been iterating on our products. When their later migration happened, Github Actions was just coming around, so we figured that if we had to rebuild and re-test all of our CI, we might as well move it to a more reliable and cheaper runner. Now we only use CircleCI on a handful of legacy repos.
notatoad 3y ago
>I think the key is that nobody "wants" to spend any time at all on CI or their build system. or if they do, they're hosting their own. if you're selling a hosted CI platform, your sales pitch is "pay us $$ instead of paying your staff $$$ to do devops". if your pitch is "pay us $$ and then also spend $$$ on staff hours to manage our service", it's very unlikely the CI is doing anything useful enough that the staff couldn't just run their own thing in-house. it doesn't take much configuration to be more difficult than an EC2 instance running a script on a webhook from your version control.
larusso 3y ago
I had a talk with a circle CI rep at a conference and I tried to explain to him that yes I’m not happy with our custom Jenkins setup (agents, CasC, pipeline library), but that I also want to spend time and money on a migration project to end up at the status co again. If I would go to my boss and say we can either spend some time and fix our setup or spend time plus running costs for this external solution then the answer will be clear. I think convincing teams with huge stacks of CI setup to just switch because of marketing promises isn’t working. I would think of using a new setup when I start from scratch. At the moment at least.

danudey 3y ago

I started work in December at an awesome company as a build/release engineer. Our current release workflow for our open-source components: 1. Run the CI build/test pipeline 2. If it passes, run the 'Publish' promotion (manually, on any pipeline you want) 3. The pipeline runs a make target 4. The make target starts up a VM (from the VM it's running on) 5. The make target SSH'es to the VM to run a make target 4. The make target builds a binary 5. The binary runs more make targets 6. The make targets run various scripts in docker containers on the VM the other VM started 7. The scripts in the docker containers build the components 8. The binary from step five runs more make targets to publish the releases 9. We test everything with another promotion which runs make targets which build docker containers to run python scripts (pytest) This is also built by a complicated web of wildcarded makefile targets, which need to be interoperable and support a few if/else cases for specific components. My plan is to migrate all of this to something simpler and more straightforward, or at least more maintainable, which is honestly probably going to turn into taskfile[0] instead of makefiles, and then simple python scripts for the glue that ties everything together or does more complex logic. My hope is that it can be more straightforward and easier to maintain, with more component-ized logic, but realistically every step in that labyrinthine build process (and that's just the open-source version!) came from a decision made by a very talented team of engineers who know far more about the process and the product than I do. At this point I'm wondering if it would make 'more sense' to replace it with a giant python script of some kind and get access to all the logic we need all at once (it would not). [0] https://taskfile.dev/ https://taskfile.dev/

392 3y ago
Try buck2
pimlottc 3y ago
HN collapses lists, you might want to add some newlines between the steps for better readability.
hedora 3y ago
Taskfile looks worse than make to me in every possible dimension (except maybe windows compatibility, but I don't believe that it meaningfully supports that). You should at least read and understand the paper "Recursive Make Considered Harmful" before attempting to replace make with something "better": https://aegis.sourceforge.net/auug97.pdf https://aegis.sourceforge.net/auug97.pdf Most people use make incorrectly (and it sounds like the system you describe makes most of the classic mistakes). The paper I linked explains how to use Make's templating language to help it scale to complicated setups. Here are a few critiques of taskfile (from skimming their documentation): - The syntax is incredibly verbose and weird. For instance, you have to write a tree of yaml nodes to loop over a set of files. Make's syntax is weird, but at least it is not verbose. - The documentation makes no mention that I could find of deriving a set of tasks from a list of files (Make's $(wildcard) macro, or stemming rules like "%.o : %.c"). So, I guess the taskfile will end up being huge (like the size of autogenerated makefiles from autoconf or cmake), or you'll end up using additional ad-hoc build systems that are invoked by your taskfile. - From the documentation, they haven't thought through having more than one project per unix account "When running your global Taskfile with -g, tasks will run on $HOME by default, and not on your working directory!" - They seem to have partially reimplemented the bad ideas from recursive make: https://taskfile.dev/usage/#including-other-taskfiles https://taskfile.dev/usage/#including-other-taskfiles - I suggest taking an idiomatic but non-trivial bash script (which is 99% of what make is invoking, and that taskfiles support), and then trying to port it to python directly. There's usually a 10-100x line of code blowup from doing that, and the python thing usually bitrots every year or so (vs. every few decades for shell).

Spivak 3y ago

The space was destined to collapse these two products not because it makes sense conceptually but because their implementations are almost identical and maintaining fewer systems is great. One was always going to eat the other and CI systems are turnkey, hosted, and live close to the code that it will be running. I wish they had either remained separate or gone the other way but it's not worth swimming upstream for it.

chubot 3y ago

OK but my point is that should have been apparent at the step 1 validation. Seems like they are onto step 2 or 3, and figured out that step 1 wasn't actually validated -- people didn't want to migrate to Earthly files ??? The blog post is not clear about what happened