5 ms·

You don't. Reproduce the issue on a development instance.

by bnr 14y ago

damncabbage 14y ago
I first took TazeTSchnitzel's comment to be sarcasm, but now I'm not so sure. (Poe's Law says hi.)
- TazeTSchnitzel 14y ago
  Not sarcasm. Of course you should reproduce in a development environment, but for some issues, looking at live DB is the only way to see what is wrong.
  - mgkimsal 14y ago
    A system I work on has grown a lot over the last few years (data for 2000 users is now data for 50,000, rules have been added, etc). Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email reports). Finally got a dump of live data to dev system (it's a lot of data, so I don't pull it all the time). Someone had updated their email address to something invalid, and the system threw an uncaught exception in the middle of the loop (yay java). So... half the people were getting their stuff, the second half didn't - guilty record was a user with last name starting with L. Yes, we should have prevented a bad email from going in with validation, but it slipped my testing (and the client's). It's just shocking to me today that professional people whose job it is to send and receive email can mistype their email. "john smith @yaho" is not valid, yes I should catch that, but someone typed that in. Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops. Multiple lessons learned from that one certainly (logging, validation, exception handling, etc) but... it would have been a lot longer for me to even consider putting in an invalid email address (it worked in dev - it was working for end users, etc) - pulling live data was the only thing that made it apparent.
    - wpietri 14y ago
      I agree totally with your broad point, but wanted to respond to the "I can't believe a user typed..." Even if your 50,000 people are all pretty smart, you're well out into the range of exceptional circumstances. If I do something once a day my whole life, that's still only 25,000 times. It's pretty easy to imagine somebody on their worst day doing something like that. E.g., You go to visit family, so you're jet lagged. You had a couple of late beers with your brother, so you're hung over. His baby is screaming with colic and the toddler is banging pots together. You're VPNing in from their kitchen table, trying to fill out some form, and in the middle somebody knocks over a glass of juice. You yank your laptop up, help clean up the mess, and then sit back down to finish. While the pot-banging proceeds at full volume. Madness, sure, but common madness.
    - gouranga 14y ago
      > Adding on top of that was my own dynamic language background not mapping well to the JVM There's your problem...
      - mgkimsal 14y ago
        It's a problem, but it certainly wasn't my only - groovy, to be precise.
    - tedunangst 14y ago
      Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops. Makes me wonder what dynamic language you're used to. An uncaught exception in both python and ruby aborts the program, not just the tiny inner loop that happens to be running.
      - mgkimsal 14y ago
        Primarily PHP, although I seem to remember classic ASP VBScript not dying if you passed a bad email to CDO (but that's been a long time, so i might not remember that correctly). PHP would not throw an exception if the mail function got a bad email address. well, using some mailing libraries might, but the core mailing function (basically just a wrapper for sendmail/postfix CLI stuff) wouldn't.
rimantas 14y ago
How do you do that, if issue is caused by some corner case with user data you did not foresee and hence not covered?
- Goladus 14y ago
  For some applications, the number of corner cases that can not be resolved by reading log files is minimal. (you are writing logs, right?) That is, too, assuming the problem is actually in the code itself and not the infrastructure.

You don't. Reproduce the issue on a development instance.

by bnr 14y ago

You don't. Reproduce the issue on a development instance.

damncabbage 14y ago

I first took TazeTSchnitzel's comment to be sarcasm, but now I'm not so sure. (Poe's Law says hi.)

TazeTSchnitzel 14y ago
Not sarcasm. Of course you should reproduce in a development environment, but for some issues, looking at live DB is the only way to see what is wrong.
- mgkimsal 14y ago
  A system I work on has grown a lot over the last few years (data for 2000 users is now data for 50,000, rules have been added, etc). Example story: We had an issue that was only coming up on production. I could not reproduce it in a dev environment. Worse, we didn't even notice it for a long time because it was a nightly job, and people were not reporting an absence of their notifications. (nightly job to email reports). Finally got a dump of live data to dev system (it's a lot of data, so I don't pull it all the time). Someone had updated their email address to something invalid, and the system threw an uncaught exception in the middle of the loop (yay java). So... half the people were getting their stuff, the second half didn't - guilty record was a user with last name starting with L. Yes, we should have prevented a bad email from going in with validation, but it slipped my testing (and the client's). It's just shocking to me today that professional people whose job it is to send and receive email can mistype their email. "john smith @yaho" is not valid, yes I should catch that, but someone typed that in. Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops. Multiple lessons learned from that one certainly (logging, validation, exception handling, etc) but... it would have been a lot longer for me to even consider putting in an invalid email address (it worked in dev - it was working for end users, etc) - pulling live data was the only thing that made it apparent.
  - wpietri 14y ago
    I agree totally with your broad point, but wanted to respond to the "I can't believe a user typed..." Even if your 50,000 people are all pretty smart, you're well out into the range of exceptional circumstances. If I do something once a day my whole life, that's still only 25,000 times. It's pretty easy to imagine somebody on their worst day doing something like that. E.g., You go to visit family, so you're jet lagged. You had a couple of late beers with your brother, so you're hung over. His baby is screaming with colic and the toddler is banging pots together. You're VPNing in from their kitchen table, trying to fill out some form, and in the middle somebody knocks over a glass of juice. You yank your laptop up, help clean up the mess, and then sit back down to finish. While the pot-banging proceeds at full volume. Madness, sure, but common madness.
  - gouranga 14y ago
    > Adding on top of that was my own dynamic language background not mapping well to the JVM There's your problem...
    - mgkimsal 14y ago
      It's a problem, but it certainly wasn't my only - groovy, to be precise.
  - tedunangst 14y ago
    Adding on top of that was my own dynamic language background not mapping well to the JVM - one bad address in the middle of a loop doesn't just get skipped and logged, but the entire process now stops. Makes me wonder what dynamic language you're used to. An uncaught exception in both python and ruby aborts the program, not just the tiny inner loop that happens to be running.
    - mgkimsal 14y ago
      Primarily PHP, although I seem to remember classic ASP VBScript not dying if you passed a bad email to CDO (but that's been a long time, so i might not remember that correctly). PHP would not throw an exception if the mail function got a bad email address. well, using some mailing libraries might, but the core mailing function (basically just a wrapper for sendmail/postfix CLI stuff) wouldn't.

rimantas 14y ago

How do you do that, if issue is caused by some corner case with user data you did not foresee and hence not covered?

Goladus 14y ago
For some applications, the number of corner cases that can not be resolved by reading log files is minimal. (you are writing logs, right?) That is, too, assuming the problem is actually in the code itself and not the infrastructure.