4 ms·

This is an incredibly bold assertion: > Developers of large or legacy systems that cannot already be run in their entirety during development often believe tha

This is an incredibly bold assertion: > Developers of large or legacy systems that cannot already be run in their entirety during development often believe that it is impractical to run the entire system during development. They'll talk about the many dependencies of their system, how it requires careful configuration of a large number of hosts, or how it's too complex to get reliable behavior. > In my experience, they're always wrong. These systems can be run locally during development with a relatively small investment of effort. Typically, these systems are just ultimately not as complicated as people think they are; once the system's dependencies are actually known and understood rather than being cargo-culted or assumed, running the system, and all its dependencies, is straightforward. In my 28 years of professional experience working in 9 different organizations, only one of them did fully run the core system in development, and another two of them could have done that with enough time and effort. For the other six, there was no reasonable way to run the full system in development. > Typically, these systems are just ultimately not as complicated as people think they are In fact, most long-running systems are quite a bit more complicated than people think they are. Granted, a variable but usually large chunk of that complexity can be removed/simplified, but at their core, these 'legacy' systems, the ones that make real companies work, do their job because they have, over many years, figured out how to successfully come up against all kinds of real world complications. To be clear: I get what (I think) this person is fundamentally trying to say: the more expansive/end to end a development environment is, the better. That it's better to have as wide a view into the system you're working on as possible. Good stuff!

whb07 6y ago
Aren’t you making the point of the author? Most places don’t have a way to run in development because they haven’t sat down and gotten the application to a running state. Every year it gets ignored, things get harder and more tedious but never impossible
- imoverclocked 6y ago
  I see both sides of this argument. Often legacy systems are tied to their deployment methodology/dogma and the world moves on from that model. Without a lot of resources poured into maintaining that methodology and upgrading it, the expected environment can age out of current practices far enough that running it locally does take a lot of effort. I think docker is an obvious enabler of this because people can just spin up an image from 6 years ago in a container and keep running against old libraries/dependencies. Even worse is depending on that container makes it difficult to upgrade dependencies to modern versions of software because the container is so old. Maybe a corollary to the article is, "without constant maintenance, software becomes un-runnable."
TheCoelacanth 6y ago
It seems tantamount to declaring that a system can never interact with the outside world. If I'm creating tax filing software, do I need to spin up a local instance of the IRS to test with?
- minitoar 6y ago
  You can possibly stub out the IRS with realistic test cases that you care about.
  - TheCoelacanth 6y ago
    > "Run an individual service against mocks" doesn't count. > "Run most services in a mostly-isolated development environment, calling out to a few hard-to-run external services" doesn't count.
    - minitoar 6y ago
      I think they're talking about services that you or your firm develops, or maintains, or ships. In this case the IRS is none of those things.
      - gfunk911 6y ago
        Yes, obviously this. People who don't understand this either just want to argue, or haven't thought about this at all.
- 0xEFF 6y ago
  > If I'm creating tax filing software, do I need to spin up a local instance of the IRS to test with? No, integrate directly with the test endpoints provided by the IRS [1]. [1]: https://www.irs.gov/pub/irs-schema/Portal%20Transition%20Guide%20for%20Modernized%20e-File%20v14r3.pdf https://www.irs.gov/pub/irs-schema/Portal%20Transition%20Gui...
- haroldl 6y ago
  In the spirit of the article, I think that the approach would be to ask your partners to provide a sandbox/test environment that your sandbox/test environment can interact with, or test accounts in production at least.
benlivengood 6y ago
> In my 28 years of professional experience working in 9 different organizations, only one of them did fully run the core system in development, and another two of them could have done that with enough time and effort. For the other six, there was no reasonable way to run the full system in development. Only ~20 years here but I've worked almost exclusively on three or four large systems with test environments despite their complexity. I think it may be the nature of the companies/business that makes the difference. Ancient (1980's code) hospital ADT+billing system (medseries 4 if anyone cares) on an AS/400. We kept the last generation of hardware around to run the test/demo environment which was end-to-end connected via HL7 to all the other test environments. That included the other thing I managed, Cerner Millennium, a hodgepodge of a medical records system on AIX and Oracle, where again the old hardware got used as the test and build systems. The test environment was critical because even though it was made up of a dozen commercial software packages talking to each other over HL7 and flat files (and serial for lab devices and who knows what else) it was critical to make sure the whole environment worked when doing upgrades or new development. We kept a full copy of prod data for final integration testing. The tooling for (most of) the software supported replicating environments. The largest example is Photon/Ubiq at Google; end-to-end test environment running with deterministic sampling to save resources at all stages of the pipeline so everything from injection through logjoining and aggregation to the final database updates were continuously running the latest code for testing before a release to prod. Thousands of prod machines devoted to it. That was in addition to the integrated tests and unit tests during the development process. At FB the statistical counter system I worked on also kept a continuous test pipeline running, again with event sampling to reduce cost, to ensure everything would work properly before a new release. My takeaway is that where it matters companies are willing to spend the money and people to have full end-to-end testing, often in realtime, and have some leeway to save on resources with subsampling, smaller datasets, etc.
- jrochkind1 6y ago
  > We kept the last generation of hardware around to run the test/demo environment which was end-to-end connected via HL7 to all the other test environments. If I understand right (and I may not, both your description and the OP), this may violate the OP's prohibition I've quoted below. What do you think? > "Run an individual service in a shared stateful development environment running all the other services" doesn't count. A shared development environment will be unreliable as it diverges more and more from the real system. The OP seems to be awfully strict.

whb07 6y ago

Aren’t you making the point of the author? Most places don’t have a way to run in development because they haven’t sat down and gotten the application to a running state. Every year it gets ignored, things get harder and more tedious but never impossible

imoverclocked 6y ago
I see both sides of this argument. Often legacy systems are tied to their deployment methodology/dogma and the world moves on from that model. Without a lot of resources poured into maintaining that methodology and upgrading it, the expected environment can age out of current practices far enough that running it locally does take a lot of effort. I think docker is an obvious enabler of this because people can just spin up an image from 6 years ago in a container and keep running against old libraries/dependencies. Even worse is depending on that container makes it difficult to upgrade dependencies to modern versions of software because the container is so old. Maybe a corollary to the article is, "without constant maintenance, software becomes un-runnable."

TheCoelacanth 6y ago

It seems tantamount to declaring that a system can never interact with the outside world. If I'm creating tax filing software, do I need to spin up a local instance of the IRS to test with?

minitoar 6y ago
You can possibly stub out the IRS with realistic test cases that you care about.
- TheCoelacanth 6y ago
  > "Run an individual service against mocks" doesn't count. > "Run most services in a mostly-isolated development environment, calling out to a few hard-to-run external services" doesn't count.
  - minitoar 6y ago
    I think they're talking about services that you or your firm develops, or maintains, or ships. In this case the IRS is none of those things.
    - gfunk911 6y ago
      Yes, obviously this. People who don't understand this either just want to argue, or haven't thought about this at all.
0xEFF 6y ago
> If I'm creating tax filing software, do I need to spin up a local instance of the IRS to test with? No, integrate directly with the test endpoints provided by the IRS [1]. [1]: https://www.irs.gov/pub/irs-schema/Portal%20Transition%20Guide%20for%20Modernized%20e-File%20v14r3.pdf https://www.irs.gov/pub/irs-schema/Portal%20Transition%20Gui...
haroldl 6y ago
In the spirit of the article, I think that the approach would be to ask your partners to provide a sandbox/test environment that your sandbox/test environment can interact with, or test accounts in production at least.

benlivengood 6y ago

> In my 28 years of professional experience working in 9 different organizations, only one of them did fully run the core system in development, and another two of them could have done that with enough time and effort. For the other six, there was no reasonable way to run the full system in development. Only ~20 years here but I've worked almost exclusively on three or four large systems with test environments despite their complexity. I think it may be the nature of the companies/business that makes the difference. Ancient (1980's code) hospital ADT+billing system (medseries 4 if anyone cares) on an AS/400. We kept the last generation of hardware around to run the test/demo environment which was end-to-end connected via HL7 to all the other test environments. That included the other thing I managed, Cerner Millennium, a hodgepodge of a medical records system on AIX and Oracle, where again the old hardware got used as the test and build systems. The test environment was critical because even though it was made up of a dozen commercial software packages talking to each other over HL7 and flat files (and serial for lab devices and who knows what else) it was critical to make sure the whole environment worked when doing upgrades or new development. We kept a full copy of prod data for final integration testing. The tooling for (most of) the software supported replicating environments. The largest example is Photon/Ubiq at Google; end-to-end test environment running with deterministic sampling to save resources at all stages of the pipeline so everything from injection through logjoining and aggregation to the final database updates were continuously running the latest code for testing before a release to prod. Thousands of prod machines devoted to it. That was in addition to the integrated tests and unit tests during the development process. At FB the statistical counter system I worked on also kept a continuous test pipeline running, again with event sampling to reduce cost, to ensure everything would work properly before a new release. My takeaway is that where it matters companies are willing to spend the money and people to have full end-to-end testing, often in realtime, and have some leeway to save on resources with subsampling, smaller datasets, etc.

jrochkind1 6y ago
> We kept the last generation of hardware around to run the test/demo environment which was end-to-end connected via HL7 to all the other test environments. If I understand right (and I may not, both your description and the OP), this may violate the OP's prohibition I've quoted below. What do you think? > "Run an individual service in a shared stateful development environment running all the other services" doesn't count. A shared development environment will be unreliable as it diverges more and more from the real system. The OP seems to be awfully strict.