9 ms·

A strong commitment to backwards compatibility means keeping your mistakes

75 points by pcr910303 5y ago

js8 5y ago
IBM, on zSeries mainframes and z/OS, has even bigger commitment to backward compatibility than Microsoft. Yes, it might take a long time to fix some mistakes, but usually, you get another better API added that doesn't have the old flaws. Over time, system accrues a byzantine set of features. Unfortunately, end result of these policies is that nobody wants to develop for those systems anymore. So the (dharmic?) wheel is reinvented elsewhere.
- yjftsjthsd-h 5y ago
  > Unfortunately, end result of these policies is that nobody wants to develop for those systems anymore. Is that the main reason? I always assumed "I can't buy these machines for less than a year's salary" was the major difficulty onboarding.
  - js8 5y ago
    That's an interesting question. In the late 90s, IBM had a competition in the same S/390 architecture, with Fujitsu and Hitachi mainframes. These could compete with IBM on price, so I don't think this would be a problem. It's hard to discern cause and effect here - maybe mainframes are expensive because IBM held to them and eventually became a monopoly. But I think the claims "mainframes are legacy and complicated" came sooner than that, and that was the real cause why people wanted to avoid them. Compare this with Unix in the early 90s, it had a very same problem, being somewhat expensive and inaccessible, even for home PCs. But this changed with Linux... and in general, home PCs changed the expectation of the onboarding, before that, the fact that you cannot buy a computer at home, didn't present a significant obstacle in employee training. I am almost wondering if any complex enough system is long-term commercially viable. Closed-sourcing somewhat inevitably means bleeding mindshare.
    - WJW 5y ago
      > if any complex enough system is long-term commercially viable I don't think you can determine this from first principles without taking the wider world into account. Things like the electricity grid or car manufacturing seem to have managed just fine over long periods of time, despite (almost) nobody having their own coal power plant or cylinder boring machinery. There are some niches in software that will survive just fine even as closed source software, SAP, Salesforce and industrial control systems spring to mind.
      - bryanrasmussen 5y ago
        >electricity grid or car manufacturing electricity grid became a commodity (I believe arguably car manufacturing did as well), not sure if the theory isn't that any system that lasts over the long term will either become commoditised or cease to be commercially viable.
        pjc50 5y ago
        Electricity feels like a commodity (although unlike the other commodity markets there's almost no "carry trade", nobody talks about contango power). The electric grid is a utility; on of those things that's either a state monopoly or a state-shielded private-ish company, because there can't be competing electric grids. Grid equipment itself is manufactured by unglamourous heavy engineering firms like Siemens or ABB.
    - tablespoon 5y ago
      > That's an interesting question. In the late 90s, IBM had a competition in the same S/390 architecture, with Fujitsu and Hitachi mainframes. These could compete with IBM on price, so I don't think this would be a problem. It's hard to discern cause and effect here - maybe mainframes are expensive because IBM held to them and eventually became a monopoly. I'm sure those were still really expensive though. IIRC, IBM made a big investment to migrate to a 64-bit architecture, but none of its competitors in the mainframe space wanted to do the same.
  - jnxx 5y ago
    > Is that the main reason? I think it is mostly the effect of them being legacy systems. Old systems become legacy systems if they are developed further. They can be difficult to work with but the mere fact that the language or platform breaks compatibility does not make that platform any better.
    - js8 5y ago
      True, and maybe it's the same effect that drives ageism. Young things show promise and potential, while the old things are whatever they are, unsurprising, set in their ways.
z0ltan 5y ago
Idealism vs money and staving off legal issues. Heh.
lugged 5y ago
Yes. That's the point. No it's not a real problem. Want to fix your mistakes? Make a fork and try harder next time. It's such a presumption to assume that I, as the user, gives a crap about what you think are your mistakes or not, or how important they are to you, or how much cleaner the code will be when you fix it. If you create work for me. I will walk.
mro_name 5y ago
lol, indeed. That's the point of engineering, to fence and mitigate errors - both current and future, isn't it? The future part especially makes it a hard problem.
madia_leva 5y ago
I'd say nonsense. Backwards compatibility is better than any other option. From time to time you need to provide new, clean versions of old APIs and that's it. Things get deprecated (as in "best not to use this unless you are aware of the consequences". Aggregate over time, don't replace. Garbage collect when not reachable. This works for APIs as weel as for functional programming data structures. Clojure(script) approach is a good example of how this can work beautifully well in practice. Java also used to be reasonably good in this regard. Javascript (the language) is good. Javascript (the ecosystem) is criminally bad.
- taffer 5y ago
  Suppose you create an API that allows users to enter nonsensical data or do things that are very bad for performance. Later, you realize your mistake, create a new, improved version of your API, and deprecate the old API. However, as long as you keep the old API for backward compatibility, you will have to deal with nonsensical data and poor performance.
  - pjc50 5y ago
    And not keeping the old API around incurs work, potentially a lot, for everyone using the thing - transitively. We've seen this with Python2->3 transition. If the new version isn't backwards compatible (requires non-trivial work to transition), what you've done isn't an upgrade: you've created a new product, and deprecated the old one. As we've seen from Google, people hate it when that happens (how many different eras of deprecated chat app are Google on now?) Other examples: Windows has, on a couple of occasions, attempted architecture transitions. Windows RT was the latest. They've not yet made the leap to ARM, despite having cutdown versions of Windows (CE) running on ARM for something like 20 years. Apple have managed multiple architecture transitions by providing emulators, bridges, "fat binaries" that work across the transition, and a history of being willing to tell both developers and customers to get stuffed if they don't like it.
    - selfhoster11 5y ago
      Windows ARM has an emulation layer for x86 binaries
Jyaif 5y ago
It may be possible to create a programming language where you express your intent at enough of a high level that the code can be automatically refactored into using new APIs.
WhyNotHugo 5y ago
The reality is that you have to be sensible and balanced. You can't keep BROKEN functions "because people rely on them", but you can't suddenly make a minor release (e.g.: from 2.3.4 to 2.3.5) completely changing the semantic of a feature you expose. You need to: - Avoid changing APIs just for the sake of change. - Announce breaking changes. - Document how users are intended to migrate from your old API to the new one. Very large projects like Django do things like: - v2.0: Adds a new API, but keeps the old one - v3.0: Removes the old API. This gives consumers of the API plenty of time and margin to experiment with both and move on. If a single release changes too many things, it's very hard for a developer to upgrade, since they have a completely broken codebase, and need to fix multiple breakage at once.
- MaxBarraclough 5y ago
  > This gives consumers of the API plenty of time and margin to experiment with both and move on. It's not a silver bullet though, as we saw with the move to Python3. It annoyed people with large Python2 codebases, giving them the choice between: • Laboriously reworking their code to transition it to Python3 (a language with only limited advantages over Python2) • Using an unsupported language for a significant codebase
  - WhyNotHugo 5y ago
    Python 2-3 is a great COUNTER example of what I'm saying. There was no intermediate release that could run both python2 and python3 code. It introduced new features AND removed the old ones all at once.
    - MaxBarraclough 5y ago
      > There was no intermediate release that could run both python2 and python3 code. Right, but would that have helped? Transitioning a large codebase from Python2 to Python3 is a significant undertaking, there's no way around it. As I understand it there are dedicated tools to help migrate Python code, but it's still no small thing if you're dealing with a large codebase. Perhaps it would have been helpful to have an interpreter that supported the union of both languages, but I don't see that this would have changed things from significantly costly down to no big deal. It would make the process more incremental, but it would still represent significant work. There are legitimate reasons people don't want to have to modify large stable codebases. (To be clear, I haven't had to do this personally, but that seems to be the sentiment of people who have.) C++, on the other hand, goes to considerable lengths to preserve backward compatibility, at the cost of accumulating complexity.
raverbashing 5y ago
Yeah this pushes developers of the platform between a rock and a hard place. It's ok to keep backwards compatibility, but the API needs to also move forward and get the bugs ironed out. The worse case is when developers start to rely on the bugs for their workarounds (yes, please, do test your API before releasing) Maybe shims are the way to go. Instead of keeping old functionality around you make a shim that only talks to the new API. And that will serve as a guide (because if you're taking functionality out that's usually not good). I've seen a couple of cases where the deprecated API was much more straight forward and easy to use than the new one. Not good.
- madia_leva 5y ago
  We are talking about APIs (Application Programming Interfaces), not implementations. Of course you are free to change the implementation of your API at any moment, as long as you keep the semantic and performance characteristics. If you have a much improved design which allows for a better implementation then it would be perfectly fine if you reimplement the old API on top of the new API. As long as this is strictly transparent (source and binary, if this makes sense), then why not. Linus attitude about not breaking user space is the correct one.
cecilpl2 5y ago
I'm reminded of a story told by Joel Sposky about Simcity on Windows 95. The DOS/Win3.x version of SimCity had a bug where it used memory after freeing it, which worked fine on DOS since the memory didn't get reallocated. The Windows 95 team found that it was broken, and so they added code to detect whether SimCity was running, and if so ran the memory manager in a special mode that didn't release memory immediately.
- skeeter2020 5y ago
  people love to credit MS's success to their ruthless monopolistic practices and there is truth to that, but they've also (a) treated developers very well and (b) put tremendous effort into backwards compatibility. Neither of these are trivial but I don't think there's another player out there who has done this as well.

75 points by pcr910303 5y ago

js8 5y ago

IBM, on zSeries mainframes and z/OS, has even bigger commitment to backward compatibility than Microsoft. Yes, it might take a long time to fix some mistakes, but usually, you get another better API added that doesn't have the old flaws. Over time, system accrues a byzantine set of features. Unfortunately, end result of these policies is that nobody wants to develop for those systems anymore. So the (dharmic?) wheel is reinvented elsewhere.

yjftsjthsd-h 5y ago
> Unfortunately, end result of these policies is that nobody wants to develop for those systems anymore. Is that the main reason? I always assumed "I can't buy these machines for less than a year's salary" was the major difficulty onboarding.
- js8 5y ago
  That's an interesting question. In the late 90s, IBM had a competition in the same S/390 architecture, with Fujitsu and Hitachi mainframes. These could compete with IBM on price, so I don't think this would be a problem. It's hard to discern cause and effect here - maybe mainframes are expensive because IBM held to them and eventually became a monopoly. But I think the claims "mainframes are legacy and complicated" came sooner than that, and that was the real cause why people wanted to avoid them. Compare this with Unix in the early 90s, it had a very same problem, being somewhat expensive and inaccessible, even for home PCs. But this changed with Linux... and in general, home PCs changed the expectation of the onboarding, before that, the fact that you cannot buy a computer at home, didn't present a significant obstacle in employee training. I am almost wondering if any complex enough system is long-term commercially viable. Closed-sourcing somewhat inevitably means bleeding mindshare.
  - WJW 5y ago
    > if any complex enough system is long-term commercially viable I don't think you can determine this from first principles without taking the wider world into account. Things like the electricity grid or car manufacturing seem to have managed just fine over long periods of time, despite (almost) nobody having their own coal power plant or cylinder boring machinery. There are some niches in software that will survive just fine even as closed source software, SAP, Salesforce and industrial control systems spring to mind.
    - bryanrasmussen 5y ago
      >electricity grid or car manufacturing electricity grid became a commodity (I believe arguably car manufacturing did as well), not sure if the theory isn't that any system that lasts over the long term will either become commoditised or cease to be commercially viable.
      - pjc50 5y ago
        Electricity feels like a commodity (although unlike the other commodity markets there's almost no "carry trade", nobody talks about contango power). The electric grid is a utility; on of those things that's either a state monopoly or a state-shielded private-ish company, because there can't be competing electric grids. Grid equipment itself is manufactured by unglamourous heavy engineering firms like Siemens or ABB.
  - tablespoon 5y ago
    > That's an interesting question. In the late 90s, IBM had a competition in the same S/390 architecture, with Fujitsu and Hitachi mainframes. These could compete with IBM on price, so I don't think this would be a problem. It's hard to discern cause and effect here - maybe mainframes are expensive because IBM held to them and eventually became a monopoly. I'm sure those were still really expensive though. IIRC, IBM made a big investment to migrate to a 64-bit architecture, but none of its competitors in the mainframe space wanted to do the same.
- jnxx 5y ago
  > Is that the main reason? I think it is mostly the effect of them being legacy systems. Old systems become legacy systems if they are developed further. They can be difficult to work with but the mere fact that the language or platform breaks compatibility does not make that platform any better.
  - js8 5y ago
    True, and maybe it's the same effect that drives ageism. Young things show promise and potential, while the old things are whatever they are, unsurprising, set in their ways.

z0ltan 5y ago

Idealism vs money and staving off legal issues. Heh.

lugged 5y ago

Yes. That's the point. No it's not a real problem. Want to fix your mistakes? Make a fork and try harder next time. It's such a presumption to assume that I, as the user, gives a crap about what you think are your mistakes or not, or how important they are to you, or how much cleaner the code will be when you fix it. If you create work for me. I will walk.

mro_name 5y ago

lol, indeed. That's the point of engineering, to fence and mitigate errors - both current and future, isn't it? The future part especially makes it a hard problem.

madia_leva 5y ago

I'd say nonsense. Backwards compatibility is better than any other option. From time to time you need to provide new, clean versions of old APIs and that's it. Things get deprecated (as in "best not to use this unless you are aware of the consequences". Aggregate over time, don't replace. Garbage collect when not reachable. This works for APIs as weel as for functional programming data structures. Clojure(script) approach is a good example of how this can work beautifully well in practice. Java also used to be reasonably good in this regard. Javascript (the language) is good. Javascript (the ecosystem) is criminally bad.

taffer 5y ago
Suppose you create an API that allows users to enter nonsensical data or do things that are very bad for performance. Later, you realize your mistake, create a new, improved version of your API, and deprecate the old API. However, as long as you keep the old API for backward compatibility, you will have to deal with nonsensical data and poor performance.
- pjc50 5y ago
  And not keeping the old API around incurs work, potentially a lot, for everyone using the thing - transitively. We've seen this with Python2->3 transition. If the new version isn't backwards compatible (requires non-trivial work to transition), what you've done isn't an upgrade: you've created a new product, and deprecated the old one. As we've seen from Google, people hate it when that happens (how many different eras of deprecated chat app are Google on now?) Other examples: Windows has, on a couple of occasions, attempted architecture transitions. Windows RT was the latest. They've not yet made the leap to ARM, despite having cutdown versions of Windows (CE) running on ARM for something like 20 years. Apple have managed multiple architecture transitions by providing emulators, bridges, "fat binaries" that work across the transition, and a history of being willing to tell both developers and customers to get stuffed if they don't like it.
  - selfhoster11 5y ago
    Windows ARM has an emulation layer for x86 binaries

Jyaif 5y ago

It may be possible to create a programming language where you express your intent at enough of a high level that the code can be automatically refactored into using new APIs.

WhyNotHugo 5y ago

The reality is that you have to be sensible and balanced. You can't keep BROKEN functions "because people rely on them", but you can't suddenly make a minor release (e.g.: from 2.3.4 to 2.3.5) completely changing the semantic of a feature you expose. You need to: - Avoid changing APIs just for the sake of change. - Announce breaking changes. - Document how users are intended to migrate from your old API to the new one. Very large projects like Django do things like: - v2.0: Adds a new API, but keeps the old one - v3.0: Removes the old API. This gives consumers of the API plenty of time and margin to experiment with both and move on. If a single release changes too many things, it's very hard for a developer to upgrade, since they have a completely broken codebase, and need to fix multiple breakage at once.

MaxBarraclough 5y ago
> This gives consumers of the API plenty of time and margin to experiment with both and move on. It's not a silver bullet though, as we saw with the move to Python3. It annoyed people with large Python2 codebases, giving them the choice between: • Laboriously reworking their code to transition it to Python3 (a language with only limited advantages over Python2) • Using an unsupported language for a significant codebase
- WhyNotHugo 5y ago
  Python 2-3 is a great COUNTER example of what I'm saying. There was no intermediate release that could run both python2 and python3 code. It introduced new features AND removed the old ones all at once.
  - MaxBarraclough 5y ago
    > There was no intermediate release that could run both python2 and python3 code. Right, but would that have helped? Transitioning a large codebase from Python2 to Python3 is a significant undertaking, there's no way around it. As I understand it there are dedicated tools to help migrate Python code, but it's still no small thing if you're dealing with a large codebase. Perhaps it would have been helpful to have an interpreter that supported the union of both languages, but I don't see that this would have changed things from significantly costly down to no big deal. It would make the process more incremental, but it would still represent significant work. There are legitimate reasons people don't want to have to modify large stable codebases. (To be clear, I haven't had to do this personally, but that seems to be the sentiment of people who have.) C++, on the other hand, goes to considerable lengths to preserve backward compatibility, at the cost of accumulating complexity.

raverbashing 5y ago

Yeah this pushes developers of the platform between a rock and a hard place. It's ok to keep backwards compatibility, but the API needs to also move forward and get the bugs ironed out. The worse case is when developers start to rely on the bugs for their workarounds (yes, please, do test your API before releasing) Maybe shims are the way to go. Instead of keeping old functionality around you make a shim that only talks to the new API. And that will serve as a guide (because if you're taking functionality out that's usually not good). I've seen a couple of cases where the deprecated API was much more straight forward and easy to use than the new one. Not good.

madia_leva 5y ago
We are talking about APIs (Application Programming Interfaces), not implementations. Of course you are free to change the implementation of your API at any moment, as long as you keep the semantic and performance characteristics. If you have a much improved design which allows for a better implementation then it would be perfectly fine if you reimplement the old API on top of the new API. As long as this is strictly transparent (source and binary, if this makes sense), then why not. Linus attitude about not breaking user space is the correct one.

cecilpl2 5y ago

I'm reminded of a story told by Joel Sposky about Simcity on Windows 95. The DOS/Win3.x version of SimCity had a bug where it used memory after freeing it, which worked fine on DOS since the memory didn't get reallocated. The Windows 95 team found that it was broken, and so they added code to detect whether SimCity was running, and if so ran the memory manager in a special mode that didn't release memory immediately.

skeeter2020 5y ago
people love to credit MS's success to their ruthless monopolistic practices and there is truth to that, but they've also (a) treated developers very well and (b) put tremendous effort into backwards compatibility. Neither of these are trivial but I don't think there's another player out there who has done this as well.