3 ms·

I clicked into this because I was excited, but everything feels like it's missing the mark for me. And, that pricing. Woof. That's steep for being a fancy chang

I clicked into this because I was excited, but everything feels like it's missing the mark for me. And, that pricing. Woof. That's steep for being a fancy changelog interpreter. ---- My biggest pain isn't with the plan. It's with the actual upgrade process. Things _always_ break. Every single time we do a major upgrade, there's this long tail of things we need to fix. Even worse is the undocumented changes that break. This gives me nightmares. The problem with your product is four parts: * It needs to solve much more of the problem. It's not really valuable to know that I need to change stuff. I can figure that out by running my package managers update tool and seeing what breaks. * When things break, it's unbelievably time consuming to fix them. Even worse is the person tasked with doing the upgrade is likely not the subject matter expert on broken code. * I need to know what's broken that wasn't documented. This is by far the hardest part of our upgrade cycles. It keeps me up at night knowing a major upgrade could break things that aren't covered by unit tests. * Because dependency upgrades are so hard, we tend to do them in large swoops. We set aside a week or two of pain and power through them. Given you're charging $60/m, this suggests: * Your model is more aligned with small, continuous upgrades. Once you're "close" to the most recent version, it tends to be easier to upgrade. * Your model needs a better way to "gain pace" with the latest releases. If I can incrementally upgrade one package on every PR until I'm up to date, that'd be amazing. ----- I think this tool would be extremely value to me (likely $60/m/repo) if it: * It ran the repo's test suite against progressively newer versions of dependencies. Showing when and where unit tests fail. * Provide some means of "intelligent E2E" testing. Fire up our API/application. Tell us what changes when dependencies are upgraded. * Fan out broken code to the subject matter expert. If an upgrade from X to Y is broken, look at the `git blame` to figure out who knows how to fix the code. Ping them asking them for help.

stevepike 3y ago
Thanks for the feedback. We know we're tackling a big problem here. Our ultimate goal is that if you use Infield we keep you on the latest version of every dependency, automatically. The road to get there can go a number of ways. > My biggest pain isn't with the plan. It's with the actual upgrade process. Things _always_ break. Every single time we do a major upgrade, there's this long tail of things we need to fix. Even worse is the undocumented changes that break. This gives me nightmares. Planning is one way to make those major upgrades go wrong less often. Our idea is that we can break large upgrades down into incremental changes that are individually safe and mixed in to your regular dev cycles, so that when you go to do the large upgrade you're not making such a big change all at once. This means things like fixing every deprecation ahead of time and making sure all the other dependencies you use are compatible with the version you're upgrading to. We're calling this "upgrade paths" in our tool. Something like a major rails version upgrade is terrifying because of all the moving parts. Often you may have a dozen other gems that need to be upgraded in order to upgrade Rails. You said "If I can incrementally upgrade one package on every PR until I'm up to date, that'd be amazing.". We want to make that the flow for major framework upgrades, so you're merging in incremental improvements in the blocking dependencies ahead of time. Undocumented breakages are for sure on our roadmap. We have two types in mind today: - Changes that are missing from the changelog - Incompatibilities across gems (for instance there was a time recently where if you had both datadog and newrelic installed and used elsaticsearch upgrading the newrelic gem would break prod) We have some ideas on how to figure this out automatically (like reading code diffs with GPT rather than just changelogs), but the best way is to see upgrades out in the wild. As we see more and more upgrade experiences from our customers we'll be able to catch these issues and build this dataset. I'd love to hear other ideas for making major framework upgrades safer. > * It ran the repo's test suite against progressively newer versions of dependencies. Showing when and where unit tests fail. Would this be something like `git bisect` for upgrades? We run your CI to figure out where you can upgrade to without something breaking? We're trying to figure this out statically by reading changelogs and building up a database of breaking changes. Our roadmap is to make this database better over time by pulling in more and more undocumented changes (by seeing customer upgrade experiences and by sourcing information from places like github issues). In my experience it's more common (and much more dangerous) that things break when your dependency changes in a way that wouldn't be caught by CI.

stevepike 3y ago

Thanks for the feedback. We know we're tackling a big problem here. Our ultimate goal is that if you use Infield we keep you on the latest version of every dependency, automatically. The road to get there can go a number of ways. > My biggest pain isn't with the plan. It's with the actual upgrade process. Things _always_ break. Every single time we do a major upgrade, there's this long tail of things we need to fix. Even worse is the undocumented changes that break. This gives me nightmares. Planning is one way to make those major upgrades go wrong less often. Our idea is that we can break large upgrades down into incremental changes that are individually safe and mixed in to your regular dev cycles, so that when you go to do the large upgrade you're not making such a big change all at once. This means things like fixing every deprecation ahead of time and making sure all the other dependencies you use are compatible with the version you're upgrading to. We're calling this "upgrade paths" in our tool. Something like a major rails version upgrade is terrifying because of all the moving parts. Often you may have a dozen other gems that need to be upgraded in order to upgrade Rails. You said "If I can incrementally upgrade one package on every PR until I'm up to date, that'd be amazing.". We want to make that the flow for major framework upgrades, so you're merging in incremental improvements in the blocking dependencies ahead of time. Undocumented breakages are for sure on our roadmap. We have two types in mind today: - Changes that are missing from the changelog - Incompatibilities across gems (for instance there was a time recently where if you had both datadog and newrelic installed and used elsaticsearch upgrading the newrelic gem would break prod) We have some ideas on how to figure this out automatically (like reading code diffs with GPT rather than just changelogs), but the best way is to see upgrades out in the wild. As we see more and more upgrade experiences from our customers we'll be able to catch these issues and build this dataset. I'd love to hear other ideas for making major framework upgrades safer. > * It ran the repo's test suite against progressively newer versions of dependencies. Showing when and where unit tests fail. Would this be something like `git bisect` for upgrades? We run your CI to figure out where you can upgrade to without something breaking? We're trying to figure this out statically by reading changelogs and building up a database of breaking changes. Our roadmap is to make this database better over time by pulling in more and more undocumented changes (by seeing customer upgrade experiences and by sourcing information from places like github issues). In my experience it's more common (and much more dangerous) that things break when your dependency changes in a way that wouldn't be caught by CI.