4 ms·

> Is this common at other companies that you cannot run the whole app locally to develop and test against? Yes. I work in B2B banking software, so having an "a

> Is this common at other companies that you cannot run the whole app locally to develop and test against? Yes. I work in B2B banking software, so having an "authentic" environment to test against is extraordinarily complex. Even the big vendors in the space seem unable to maintain accurate facsimiles of production with their own decades-old core software, so smaller vendors stand no chance unless they adjust their entire strategy. What we ultimately had to do was negotiate a way to test in the production environment in such a way that all the activity could be tracked and backed out afterwards. A small group of important customer stakeholders were given access to this pre production environment for purposes of validating things. This is something that has never sat well with me. But, the way the business works the alternatives are usually worse (if you want to survive long-term, that is). We wasted 18 months waiting for a customer to have a test environment installed before we learned our lesson about how bad those environments are in reality.

nijave 4y ago
Worked with banking software before; it can be incredibly complicated. I worked on proof of concept infrastructure for a Websphere monolith that ran on AIX servers. It had at least 10-15 services it connected to and each one of those were huge Java monoliths, too. Some of those apps had 8+ copies of /each/ environment (8x dev, 8x test, 4-8x stage/uat, 1-2x prod) so you could do integrated tests without stepping on other people or something software (software B might have a dedicated test environment of software A)
- winrid 4y ago
  This works well. I've seen deployments where you have test1, 2, 3 etc, and you can "lock" the environment to yourself for a few hours.
- halfmatthalfcat 4y ago
  I haven't found a wholistic integration testing framework where you can effortlessly mock out upstreams as running services, so that you don't have to ham-fist integration testing code into your app or rely on actual, live upstreams which are probably prone to breakage anyway.
  - nerdponx 4y ago
    I've done things like this in ad-hoc setups with Docker Compose and environment variables, managed with shell scripts. I would be very interested in a more thoughtful alternative that was integrated with testing.
    - chupasaurus 4y ago
      I've done a thing automating whole environment setups with VMware cloning (undocumented feature because there is it's own product on top of it) and Docker via CI, in a bank. Basically we did golden images of VMs with databases having all accounts for test cases and prepared images for services to be deployed, while getting all necessary variables from CI and VSphere to write down in configs and Consul/Vault via scripts. Initial version control for the env could be done via same prefixes in the Git branches of services, after the deployment each env had a way to deploy whatever via CI's Web UI or API. Running automated integration tests on the whole env is just an added step in the pipeline.
  - nijave 4y ago
    One way to balance is contract testing where you run a test suite against your mock/stub and the same test suite against the real thing. Your mock/stub would usually just have hardcoded data that aligns with the test suite to pass. Another way is creating separate code libraries to abstract away external services. For instance, you might write ObjectStorage library that has real integration tests that use S3 but in your application you just stub out the ObjectStorage library. A lot of frameworks have verifying doubles or "magic mocks" or similar that will stub the method signatures so that can help make sure you're calling methods correctly (but wouldn't help catch data/logic issues). Live upstreams are also prone to inconsistent state and race conditions--especially when you try to run more than 1 instance of the test suite at once. For web services, I also like using "fixtures". You just manually run against a real service and copy the requests to a file (removing any sensitive data) than have your stub load the data from the file and return it. If you hit any bugs/regressions, you can just copy the results that caused them into a second file and add another test case.
  - mieubrisse 4y ago
- raffraffraff 4y ago
  It's funny, I worked in banking about 15 years ago and the technology stack was the same. Java monoliths, Websphere on AIX (plus mainframes naturally)
acwan93 4y ago
Same here in e-commerce. We end up testing in production environments with an element somewhere that's in a development environment that can be tracked and actively watched. Amazon notoriously does not have a production environment, and eBay's is just unusable.
- grogenaut 4y ago
  im pretty sure amazon has a production environment... just their staging is problematic. It's incredibly expensive to run a copy of prod and keep it up. You essentially have to get paged for it doubling your burden.
  - acwan93 4y ago
    Yep, I mistyped. Amazon MWS doesn't have a staging/sandbox environment that's available for third party developers to test on.
gumby 4y ago
My gf works in UX for a major bank and they simply issued her a bank account in her name (with her SSN etc -- it's a real account, with ATM and debit cards and the like). If she and some co workers want to test something out relating to accounts they actually use these accounts to minimize this kind of problem. They must have some special code that says that these accounts can have their fees zeroed and probably are subject to some special out of band audits. I don't really know the details of course but I was actually kind of impressed. (no she is not actually a customer of her employer)
- tixocloud 4y ago
  Previously led data teams in banks and you’re right. There are dozens of flags for test accounts. When auditing, we screen out these accounts. Of course, with a proper data cleaning process, analytics gets a lot easier
  - lobocinza 4y ago
    That reminds me of "Mostly Harmless" from Douglas Adams where Ford Prefect breaks into The Guide's offices and gets himself an infinite expense account from the computer system. That's a dangerous thing to have.
    - SheinhardtWigCo 4y ago
      Problem is, anyone with a good story to tell should really keep their mouth shut.
invalidname 4y ago
I worked at banks too but mostly segregated from the core business. Today I see observability tools as the solution for problems like that. Specifically tools like Lightrun where you can observe production, mitigate and fix directly from the IDE. I wonder if other developers are familiar with this approach?
- shardullavekar 4y ago
  We are working on Videobug - it records code execution on production and plays in the developer IDE. Our Show HN here: https://news.ycombinator.com/item?id=31286126 https://news.ycombinator.com/item?id=31286126
tluyben2 4y ago
Yep, work in b2b banking as well and there is a huge issue with banking ‘test’ environments. Our partners (100s of companies, mostly banks/brokers/insurers/etc) mostly don’t even have test envs and if they do, they don’t represent production at all (sometimes the api is a completely different version etc). Our software allows to spin up a test/dev env locally or where ever but nothing much works without the partners and to dev/test those, those, you will have to dev/test against their production systems with real money. In the beginning (more than 20 years ago), I insisted that we implement test systems ourselves so we didn’t have to test in production; that was naive; it was expensive and there is so much complexity that this does not work well.

nijave 4y ago

Worked with banking software before; it can be incredibly complicated. I worked on proof of concept infrastructure for a Websphere monolith that ran on AIX servers. It had at least 10-15 services it connected to and each one of those were huge Java monoliths, too. Some of those apps had 8+ copies of /each/ environment (8x dev, 8x test, 4-8x stage/uat, 1-2x prod) so you could do integrated tests without stepping on other people or something software (software B might have a dedicated test environment of software A)

winrid 4y ago
This works well. I've seen deployments where you have test1, 2, 3 etc, and you can "lock" the environment to yourself for a few hours.
halfmatthalfcat 4y ago
I haven't found a wholistic integration testing framework where you can effortlessly mock out upstreams as running services, so that you don't have to ham-fist integration testing code into your app or rely on actual, live upstreams which are probably prone to breakage anyway.
- nerdponx 4y ago
  I've done things like this in ad-hoc setups with Docker Compose and environment variables, managed with shell scripts. I would be very interested in a more thoughtful alternative that was integrated with testing.
  - chupasaurus 4y ago
    I've done a thing automating whole environment setups with VMware cloning (undocumented feature because there is it's own product on top of it) and Docker via CI, in a bank. Basically we did golden images of VMs with databases having all accounts for test cases and prepared images for services to be deployed, while getting all necessary variables from CI and VSphere to write down in configs and Consul/Vault via scripts. Initial version control for the env could be done via same prefixes in the Git branches of services, after the deployment each env had a way to deploy whatever via CI's Web UI or API. Running automated integration tests on the whole env is just an added step in the pipeline.
- nijave 4y ago
  One way to balance is contract testing where you run a test suite against your mock/stub and the same test suite against the real thing. Your mock/stub would usually just have hardcoded data that aligns with the test suite to pass. Another way is creating separate code libraries to abstract away external services. For instance, you might write ObjectStorage library that has real integration tests that use S3 but in your application you just stub out the ObjectStorage library. A lot of frameworks have verifying doubles or "magic mocks" or similar that will stub the method signatures so that can help make sure you're calling methods correctly (but wouldn't help catch data/logic issues). Live upstreams are also prone to inconsistent state and race conditions--especially when you try to run more than 1 instance of the test suite at once. For web services, I also like using "fixtures". You just manually run against a real service and copy the requests to a file (removing any sensitive data) than have your stub load the data from the file and return it. If you hit any bugs/regressions, you can just copy the results that caused them into a second file and add another test case.
- mieubrisse 4y ago
raffraffraff 4y ago
It's funny, I worked in banking about 15 years ago and the technology stack was the same. Java monoliths, Websphere on AIX (plus mainframes naturally)

acwan93 4y ago

Same here in e-commerce. We end up testing in production environments with an element somewhere that's in a development environment that can be tracked and actively watched. Amazon notoriously does not have a production environment, and eBay's is just unusable.

grogenaut 4y ago
im pretty sure amazon has a production environment... just their staging is problematic. It's incredibly expensive to run a copy of prod and keep it up. You essentially have to get paged for it doubling your burden.
- acwan93 4y ago
  Yep, I mistyped. Amazon MWS doesn't have a staging/sandbox environment that's available for third party developers to test on.

gumby 4y ago

My gf works in UX for a major bank and they simply issued her a bank account in her name (with her SSN etc -- it's a real account, with ATM and debit cards and the like). If she and some co workers want to test something out relating to accounts they actually use these accounts to minimize this kind of problem. They must have some special code that says that these accounts can have their fees zeroed and probably are subject to some special out of band audits. I don't really know the details of course but I was actually kind of impressed. (no she is not actually a customer of her employer)

tixocloud 4y ago
Previously led data teams in banks and you’re right. There are dozens of flags for test accounts. When auditing, we screen out these accounts. Of course, with a proper data cleaning process, analytics gets a lot easier
- lobocinza 4y ago
  That reminds me of "Mostly Harmless" from Douglas Adams where Ford Prefect breaks into The Guide's offices and gets himself an infinite expense account from the computer system. That's a dangerous thing to have.
  - SheinhardtWigCo 4y ago
    Problem is, anyone with a good story to tell should really keep their mouth shut.

invalidname 4y ago

I worked at banks too but mostly segregated from the core business. Today I see observability tools as the solution for problems like that. Specifically tools like Lightrun where you can observe production, mitigate and fix directly from the IDE. I wonder if other developers are familiar with this approach?

shardullavekar 4y ago
We are working on Videobug - it records code execution on production and plays in the developer IDE. Our Show HN here: https://news.ycombinator.com/item?id=31286126 https://news.ycombinator.com/item?id=31286126

tluyben2 4y ago

Yep, work in b2b banking as well and there is a huge issue with banking ‘test’ environments. Our partners (100s of companies, mostly banks/brokers/insurers/etc) mostly don’t even have test envs and if they do, they don’t represent production at all (sometimes the api is a completely different version etc). Our software allows to spin up a test/dev env locally or where ever but nothing much works without the partners and to dev/test those, those, you will have to dev/test against their production systems with real money. In the beginning (more than 20 years ago), I insisted that we implement test systems ourselves so we didn’t have to test in production; that was naive; it was expensive and there is so much complexity that this does not work well.