4 ms·

Totally seconding this. Just trying to "quick fix" apparent bugs, without having understood the complex problem in its entirety beforehand, is a pretty bad app

by Slartie 10y ago

Totally seconding this. Just trying to "quick fix" apparent bugs, without having understood the complex problem in its entirety beforehand, is a pretty bad approach in quite some areas of the tech world. This is true for most server-side work, for example - you just can't put a band-aid type solution on a problem created by a weird race condition between different requests processed in parallel, because you most likely either ruin the performance or don't catch all the cases in which the bug appears, or even worse: create new bugs, deadlock situations or similar catastrophes as a result. I personally work on cash register software. In that area, it's basically this way for the server and client side of the equation, because if this highly business-critical piece of technology fails, the user really has a problem and typically can't do any business anymore. And the software release cycles on those machines also don't allow for continuous-deployment-we'll-just-test-the-fix-in-production-so-it-doesn't-matter-if-your-first-attempts-fail-miserably-style of work, so you have to get your bugs fixed thoroughly, which requires fully understanding the nature of the problem first.

Illniyar 10y ago
"You just can't put a band-aid type solution on a problem created by a weird race condition" Oh yes you can, and do, often, in payment processing code. I've done it (with a simple sleep for x seconds I think), and I can assure you the higher ups couldn't have been happier.
- Slartie 10y ago
  Oh, so maybe THAT is the reason why one of our customers had funny cases of thousands of his customers being double-billed on card payments by the payment processing provider! As a developer (of the cash register software, which wasn't at fault in this case), I was kinda amused by the whole thing. But I can assure you, the higher-up business people weren't amused at all.
  - Illniyar 10y ago
    Heh. We had tons of double billing from payment processors, we never could get a straight answer from the gateway as to why it happened. Most of them came from.a gateway called CyberSource. Also, lots of customers have weird banks where they see both an authorization (I.e. hold) and the actual capture, and they mistake it for two captures. Usually it helps to educate them about what is an authorization.
    - chris_wot 10y ago
      Man, a bank that doesn't know the difference is a bank I'd be withdrawing my funds from!
      - Illniyar 10y ago
        The bank knows the difference, the reports they send out to customers (or maybe what the customer's see on the site) don't make the distinction clear enough. Most banks don't even bother showing the customer authorizations of any kind.
  - deleted 10y ago
    [deleted]
  - Illniyar 10y ago
    If your using Java, there's a popular apache http framework[0] that automatically retries a failed request unless you specifically instruct it not to. That bit us in the A$$ with double payments a few times. 0- https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/DefaultHttpRequestRetryHandler.html https://hc.apache.org/httpcomponents-client-ga/httpclient/ap...
    - magnetic 10y ago
      If you rely on http to ensure the one-and-only-one type of paradigm, I have bad news for you (because you won't be able to distinguish failure vs success case when you hit a timeout: did it not receive the request or did I not receive the ack?). You should embed a transaction ID to ensure idempotence of your API call. Then you can make the request 1, 2 or 3 times without any risk. The receiving end will realize the transaction ID already transitioned state and won't cause multiple payments to occur.
      - Illniyar 10y ago
        Obviously. But youd be surprised, some gateways don't let you send your own transactionId. Or at least they ignored it for determining duplicate requests. Though that was several years ago.
- kalmi10 10y ago
  I always wondered why cards get processed so slowly in some countries...
- nathan_f77 10y ago
  You absolutely shouldn't need to do that often in payment processing code. Did you go back and fix it properly later?
  - Illniyar 10y ago
    The sleep was as good a solution as we were willing to do. The race condition resulted in an error syncing between two datacenters, so when it happens it can be dealt with manually. Also, to be clear, it did not affect customer waiting time as the delay was in a webhook which we used as a backup to the main payment process.
chris_wot 10y ago
I'm not entirely sure that she's right in tarring the whole industry with the same brush. Taking a bug out to tea is fantastic, because it really is about understanding the problem. But she conflates maintenance coding and program writing. There is a lot of half-baked software out there, but to say that this isn't like any other industry - like journalism - is really patently absurd. Despite best intentions, journalism is absolutely like programming. A sub-editor is really just like your QA department in many ways. And just like she made a mistake in a script on NPR unintentionally, so too do programmers. This piece is really a mass of contradictions. Take the following: But here’s the thing. No one will ever remember that number. No one remembers it now, and I’m sure no one noticed it when it happened. But I knew it happened, that it was an easily preventable mistake, and, in journalism, being wrong in that way is absolutely unacceptable. So... what is it? Was it unacceptable that the figure got into the story to the extent that she got fired? No, of course not - I'm assuming NPR ran a correction, and forgave her for her mistake. I really doubt most programmers are going out of their way to slap together code that has deliberate bugs. Yeah, they exist but to say any software development house that writes email clients, job seeking sites or any other software are intentionally encouraging this sort of behaviour seems to be an insult to the vast majority of founders and software developers who work hard to hone their craft and take care to produce quality software. This essay seems overly emotionally wrought and with a perspective that is out of kilter with reality. For me, despite her protestations that she is no better than all the others who work at software development, there's an implicit undertone of superiority masked by claims of victimisation and isolation. It's not particularly fair, and it seems rather manipulative to me.
- dcole2929 10y ago
  I think the difference is in how the two industries she are comparing here fundamentally treat mistakes. In journalism, you might not get fired immediately for a mistake like that but certainly you're going to get reprimanded and told how unacceptable that is. Your company is absolutely going to run a correction at the first opportunity they get. Any mistake is treated as a major embarrassment by staff of a respectable news company. The viewers/readers might not notice/care but the company does. This is totally not how things work in tech though. First of all there are a lot more mistakes/bugs, and because of that how bugs are handled is completely different. If you're not doing something where lives/money are at stake, it's ok to release with bugs so long as they aren't showstoppers. When I write something that has a bug in it, I don't even get talked to. We just write it up and toss it in the list with all the other bugs. There are of course situations in both industries where things are different but I think in general there is definitely a major difference in how tech treats mistakes as compare to how pretty much every other industry.
arca_vorago 10y ago
"Just trying to "quick fix" apparent bugs, without having understood the complex problem in its entirety beforehand, is a pretty bad approach in quite some areas of the tech world." And a million sysadmins across the world cried aloud in agreement and despair. The point is that I think there are business side problems more than there are tech problems in this industry as a whole. I think the phrase "the perfect is the enemy of the good" is a dangerous one for so many managers to latch onto. More on point to the article though, we need more magnum opuses, more results of this kind of long term thinking, put into action. Not more band-aids on problems. I do think though, that big picture thinkers of this ilk do find themselves trapped into a twist on the Bertrand Russell saying, "The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt." What we need then, are some of the intelligent to gain a bit of the cocksuredness, and the stupid to gain a bit of doubt.

by Slartie 10y ago

Illniyar 10y ago

"You just can't put a band-aid type solution on a problem created by a weird race condition" Oh yes you can, and do, often, in payment processing code. I've done it (with a simple sleep for x seconds I think), and I can assure you the higher ups couldn't have been happier.

Slartie 10y ago
Oh, so maybe THAT is the reason why one of our customers had funny cases of thousands of his customers being double-billed on card payments by the payment processing provider! As a developer (of the cash register software, which wasn't at fault in this case), I was kinda amused by the whole thing. But I can assure you, the higher-up business people weren't amused at all.
- Illniyar 10y ago
  Heh. We had tons of double billing from payment processors, we never could get a straight answer from the gateway as to why it happened. Most of them came from.a gateway called CyberSource. Also, lots of customers have weird banks where they see both an authorization (I.e. hold) and the actual capture, and they mistake it for two captures. Usually it helps to educate them about what is an authorization.
  - chris_wot 10y ago
    Man, a bank that doesn't know the difference is a bank I'd be withdrawing my funds from!
    - Illniyar 10y ago
      The bank knows the difference, the reports they send out to customers (or maybe what the customer's see on the site) don't make the distinction clear enough. Most banks don't even bother showing the customer authorizations of any kind.
- deleted 10y ago
  [deleted]
- Illniyar 10y ago
  If your using Java, there's a popular apache http framework[0] that automatically retries a failed request unless you specifically instruct it not to. That bit us in the A$$ with double payments a few times. 0- https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/DefaultHttpRequestRetryHandler.html https://hc.apache.org/httpcomponents-client-ga/httpclient/ap...
  - magnetic 10y ago
    If you rely on http to ensure the one-and-only-one type of paradigm, I have bad news for you (because you won't be able to distinguish failure vs success case when you hit a timeout: did it not receive the request or did I not receive the ack?). You should embed a transaction ID to ensure idempotence of your API call. Then you can make the request 1, 2 or 3 times without any risk. The receiving end will realize the transaction ID already transitioned state and won't cause multiple payments to occur.
    - Illniyar 10y ago
      Obviously. But youd be surprised, some gateways don't let you send your own transactionId. Or at least they ignored it for determining duplicate requests. Though that was several years ago.
kalmi10 10y ago
I always wondered why cards get processed so slowly in some countries...
nathan_f77 10y ago
You absolutely shouldn't need to do that often in payment processing code. Did you go back and fix it properly later?
- Illniyar 10y ago
  The sleep was as good a solution as we were willing to do. The race condition resulted in an error syncing between two datacenters, so when it happens it can be dealt with manually. Also, to be clear, it did not affect customer waiting time as the delay was in a webhook which we used as a backup to the main payment process.

chris_wot 10y ago

I'm not entirely sure that she's right in tarring the whole industry with the same brush. Taking a bug out to tea is fantastic, because it really is about understanding the problem. But she conflates maintenance coding and program writing. There is a lot of half-baked software out there, but to say that this isn't like any other industry - like journalism - is really patently absurd. Despite best intentions, journalism is absolutely like programming. A sub-editor is really just like your QA department in many ways. And just like she made a mistake in a script on NPR unintentionally, so too do programmers. This piece is really a mass of contradictions. Take the following: But here’s the thing. No one will ever remember that number. No one remembers it now, and I’m sure no one noticed it when it happened. But I knew it happened, that it was an easily preventable mistake, and, in journalism, being wrong in that way is absolutely unacceptable. So... what is it? Was it unacceptable that the figure got into the story to the extent that she got fired? No, of course not - I'm assuming NPR ran a correction, and forgave her for her mistake. I really doubt most programmers are going out of their way to slap together code that has deliberate bugs. Yeah, they exist but to say any software development house that writes email clients, job seeking sites or any other software are intentionally encouraging this sort of behaviour seems to be an insult to the vast majority of founders and software developers who work hard to hone their craft and take care to produce quality software. This essay seems overly emotionally wrought and with a perspective that is out of kilter with reality. For me, despite her protestations that she is no better than all the others who work at software development, there's an implicit undertone of superiority masked by claims of victimisation and isolation. It's not particularly fair, and it seems rather manipulative to me.

dcole2929 10y ago
I think the difference is in how the two industries she are comparing here fundamentally treat mistakes. In journalism, you might not get fired immediately for a mistake like that but certainly you're going to get reprimanded and told how unacceptable that is. Your company is absolutely going to run a correction at the first opportunity they get. Any mistake is treated as a major embarrassment by staff of a respectable news company. The viewers/readers might not notice/care but the company does. This is totally not how things work in tech though. First of all there are a lot more mistakes/bugs, and because of that how bugs are handled is completely different. If you're not doing something where lives/money are at stake, it's ok to release with bugs so long as they aren't showstoppers. When I write something that has a bug in it, I don't even get talked to. We just write it up and toss it in the list with all the other bugs. There are of course situations in both industries where things are different but I think in general there is definitely a major difference in how tech treats mistakes as compare to how pretty much every other industry.

arca_vorago 10y ago

"Just trying to "quick fix" apparent bugs, without having understood the complex problem in its entirety beforehand, is a pretty bad approach in quite some areas of the tech world." And a million sysadmins across the world cried aloud in agreement and despair. The point is that I think there are business side problems more than there are tech problems in this industry as a whole. I think the phrase "the perfect is the enemy of the good" is a dangerous one for so many managers to latch onto. More on point to the article though, we need more magnum opuses, more results of this kind of long term thinking, put into action. Not more band-aids on problems. I do think though, that big picture thinkers of this ilk do find themselves trapped into a twist on the Bertrand Russell saying, "The trouble with the world is that the stupid are cocksure and the intelligent are full of doubt." What we need then, are some of the intelligent to gain a bit of the cocksuredness, and the stupid to gain a bit of doubt.