3 ms·
At a previous company, someone decided to experiment with putting a comma in a text field submitted to an internal service that for some godforsaken reason spat
by Twirrim 7y ago
At a previous company, someone decided to experiment with putting a comma in a text field submitted to an internal service that for some godforsaken reason spat out CSV files that a lot of other teams consumed. I forget the exact entry in the field but it was something like "I wonder, how does this handle commas".
Turns out it didn't handle them well. It didn't escape them or anything. Out spits this line in the csv format that has more columns than expected and everything from the that field onwards was off by one. Most teams were lucky, their code barfed when presented with a line with too many fields. Some teams were unlucky and it didn't. How unlucky _they_ were was somewhat varied based on how much data validation they did.
That caused a big headache for a lot of people trying to recover from the actions of the genius who thought doing that against the production endpoint vs the test endpoint.
That incident pretty much solidified my reputation in the team as some kind of weird voodoo type of chaos monkey, because my shifts _always_ had a significantly higher sev2 rate caused by things failing that I'd not touched or had any influence over.
- tmpz22 7y agoBetter an issue raised by a engineer then that same issue caused by an end user :shrug:
- Twirrim 7y agoThe engineer technically was an end user. It was a service that could get called as an indirect result of customer action, but without any customer supplied information passed along, which is part of the reason it didn't do any input clean-up (at least according to the post-incident report). They assumed they could trust people not to screw things up. To be fair, it had been running for a long time that way without anyone doing that to them.
- tmpz22 7y agoAll good, I hope my comment didn't come off as snarky because I have 110% been in similar environments!
- rusk 7y agoSounds like good system testing to me! EDIT or absence thereof
- jimnotgym 7y agoThis is actually frustratingly common with off-the-shelf ERP 'integrations'. Things like courier labelling software come to mind as culprits. And you are stuck between trying to get UPS to fix a system that they give away free, or training sales operators to write addresses properly. I love the ones that present a REST api but still can't handle commas, because they have wrapped some ancient importer in a front end that simply writes the request data to a csv! I had to write one once, but I double quoted all fields and had to translate UTF-8 back to some other standard the old ERP used! It was junk, but I believe is still chugging away years later...