4 ms·

It has been in the making for at least ten years, the problem for me has been that that production environments and test environments are not the same when you

It has been in the making for at least ten years, the problem for me has been that that production environments and test environments are not the same when you use proxys. So you need to check both, and you need to have the same type of connection that your customers use. https://www.youtube.com/watch?v=B2qePLeI-s8 https://www.youtube.com/watch?v=B2qePLeI-s8 From the HTTP must die thread a month ago. https://news.ycombinator.com/item?id=44915090 https://news.ycombinator.com/item?id=44915090

radicalbyte 1y ago
The problem is that we have a culture of accepting mangled requests on the web. This happens in application code too - because web developers are sloppy it's common to either disable or not use strict input validation. In a pure .Net world it's the norm to use strict input validation and tell clients to fix their bad requests and this looks like one of those cultural blindspots. "We" wouldn't naturally consider a case where a server accepted a request which has not been strictly validated. With the move to .Net Core and a broadening of the scope to not only target enterprises and we'll find issues like this....
- jen20 1y ago
  I don't know about "not targeting enterprise" being the problem here - it's super common to find "enterprise" .NET APIs that return 200 for every possible condition and put some error text as a JSON blob in the response with "success" = "false" while setting caching headers. Mostly this stuff comes down to skill issues.
  - PantaloonFlames 1y ago
    If I transmit SOAP or JSONRPC over http, both of which use the response payload itself to contain whether the request was an error or not, what should the status be in case of error ? I jsonrpc I think 200 OK is correct with an error payload that says “you are not authorized” or similar.
    - jen20 1y ago
      > over http > what should the status be in case of error 400 to 500 range status code. You can't ignore the lower level protocol because you're implementing a higher level one.
  - j_w 1y ago
    At one point I interacted with an API that would return 200 for every condition, but with a "status" field that would have "OK" or "error", except on some browsers where it would use "OKAY" instead of "OK".
- immibis 1y ago
  We have to accept mangled requests when there are clients out there that send mangled requests, which they will continue to do as long as servers accept them. Postel's law was good for prototyping but created a security nightmare in production.
  - sfn42 1y ago
    First you create an API, then others start using it. So if you never allow mangled requests your clients will necessarily send proper requests. If you're maintaining an old api you can publish new versions of endpoints that don't accept mangled requests. If it's important you can give clients a time limit like let's say a few months to update their software to use your updated endpoints before you remove the old ones.
pixl97 1y ago
This tends to be huge in enterprise. Development, test/UAT, and production will all have different proxy methods and requirements. Devs may have a proxy with NTLM. Test may have something like proxy auto detect. Prod may be manually defined. It's really fun trying to test connectivity issues like this.
- giancarlostoro 1y ago
  In a high quality setup you have a staging server that is a carbon copy of PROD. Bonus points if you make it so staging and PROD are 100% interchangeable, to the level that you can point PROD to staging, and then turn PROD into staging, and do the same next deployment. If you can do that, you have a stronger change of at least reproducing production issues. Dev, UAT / QA, Staging, PROD. This is the ideal setup in my eyes. It lets QA / UAT hold changes that are maybe not 100% ready, while not blocking testing that is mean to go into PROD ASAP because it can sit in staging.
  - jen20 1y ago
    To be any use staging environments should be scientific tests. They should prove that a given change, if it goes to production, will work. You cannot do this if you're changing more than that one thing. The only way to make this work really is either dynamic environments that completely mirror everything, which tends to be time consuming or expensive or continuous delivery to a production-like environment via feature flags and so forth. Having a staging server that is a mirror of production[1] improves things a bit over doing nothing. You need the entire environment, including all your dependencies, to have a real test of anything, and that includes things that corporate IT departments typically hate. [1]: Why is it so common to see "PROD" written as if it were an acronym?
    - giancarlostoro 1y ago
      I always write it that way maybe for the same reason others do it, to emphasize how critical PROD is, so you don't overlook it if you just read prod, or production. If you see PRODUCTION you might slow down and go "oh crap" so it is definitely an emphasis I always add when talking about production in text. PROD is just shorter to write, but all caps makes the emphasis stick. If you staging environment is pointing to the exact same databases PROD is, and other similar dependencies, there's no reason you can't hotswap it with PROD itself, I mean I've done something like this before. It's much easier if your production deployment pipeline is setup for it though. You'd want to scale down drastically for staging, but in my eyes, if you're not going to have staging be as carbon copy of PROD as you humanely can have it, you might as well not have that fourth environment and just suffer when you cannot reproduce bugs. The real gem of staging is that if it would break in PROD, it would definitely break in staging. In the few companies where we had a carbon copy of PROD setup as a staging environment where key things are pulled from PROD itself, we've had way less bugs promoted to PROD when QA tests them in staging. In theory the ROI is worth it, if you care about quality. Sadly most places do not care about quality nearly enough.

radicalbyte 1y ago

The problem is that we have a culture of accepting mangled requests on the web. This happens in application code too - because web developers are sloppy it's common to either disable or not use strict input validation. In a pure .Net world it's the norm to use strict input validation and tell clients to fix their bad requests and this looks like one of those cultural blindspots. "We" wouldn't naturally consider a case where a server accepted a request which has not been strictly validated. With the move to .Net Core and a broadening of the scope to not only target enterprises and we'll find issues like this....

jen20 1y ago
I don't know about "not targeting enterprise" being the problem here - it's super common to find "enterprise" .NET APIs that return 200 for every possible condition and put some error text as a JSON blob in the response with "success" = "false" while setting caching headers. Mostly this stuff comes down to skill issues.
- PantaloonFlames 1y ago
  If I transmit SOAP or JSONRPC over http, both of which use the response payload itself to contain whether the request was an error or not, what should the status be in case of error ? I jsonrpc I think 200 OK is correct with an error payload that says “you are not authorized” or similar.
  - jen20 1y ago
    > over http > what should the status be in case of error 400 to 500 range status code. You can't ignore the lower level protocol because you're implementing a higher level one.
- j_w 1y ago
  At one point I interacted with an API that would return 200 for every condition, but with a "status" field that would have "OK" or "error", except on some browsers where it would use "OKAY" instead of "OK".
immibis 1y ago
We have to accept mangled requests when there are clients out there that send mangled requests, which they will continue to do as long as servers accept them. Postel's law was good for prototyping but created a security nightmare in production.
- sfn42 1y ago
  First you create an API, then others start using it. So if you never allow mangled requests your clients will necessarily send proper requests. If you're maintaining an old api you can publish new versions of endpoints that don't accept mangled requests. If it's important you can give clients a time limit like let's say a few months to update their software to use your updated endpoints before you remove the old ones.

pixl97 1y ago

This tends to be huge in enterprise. Development, test/UAT, and production will all have different proxy methods and requirements. Devs may have a proxy with NTLM. Test may have something like proxy auto detect. Prod may be manually defined. It's really fun trying to test connectivity issues like this.

giancarlostoro 1y ago
In a high quality setup you have a staging server that is a carbon copy of PROD. Bonus points if you make it so staging and PROD are 100% interchangeable, to the level that you can point PROD to staging, and then turn PROD into staging, and do the same next deployment. If you can do that, you have a stronger change of at least reproducing production issues. Dev, UAT / QA, Staging, PROD. This is the ideal setup in my eyes. It lets QA / UAT hold changes that are maybe not 100% ready, while not blocking testing that is mean to go into PROD ASAP because it can sit in staging.
- jen20 1y ago
  To be any use staging environments should be scientific tests. They should prove that a given change, if it goes to production, will work. You cannot do this if you're changing more than that one thing. The only way to make this work really is either dynamic environments that completely mirror everything, which tends to be time consuming or expensive or continuous delivery to a production-like environment via feature flags and so forth. Having a staging server that is a mirror of production[1] improves things a bit over doing nothing. You need the entire environment, including all your dependencies, to have a real test of anything, and that includes things that corporate IT departments typically hate. [1]: Why is it so common to see "PROD" written as if it were an acronym?
  - giancarlostoro 1y ago
    I always write it that way maybe for the same reason others do it, to emphasize how critical PROD is, so you don't overlook it if you just read prod, or production. If you see PRODUCTION you might slow down and go "oh crap" so it is definitely an emphasis I always add when talking about production in text. PROD is just shorter to write, but all caps makes the emphasis stick. If you staging environment is pointing to the exact same databases PROD is, and other similar dependencies, there's no reason you can't hotswap it with PROD itself, I mean I've done something like this before. It's much easier if your production deployment pipeline is setup for it though. You'd want to scale down drastically for staging, but in my eyes, if you're not going to have staging be as carbon copy of PROD as you humanely can have it, you might as well not have that fourth environment and just suffer when you cannot reproduce bugs. The real gem of staging is that if it would break in PROD, it would definitely break in staging. In the few companies where we had a carbon copy of PROD setup as a staging environment where key things are pulled from PROD itself, we've had way less bugs promoted to PROD when QA tests them in staging. In theory the ROI is worth it, if you care about quality. Sadly most places do not care about quality nearly enough.