19 ms·
The day my script killed 10k phones in South America
- TrackerFF 5y agoYou will own nothing, and you will be happy. No, but seriously, I can see situations where this is a legit service - company phones, leasing phones, etc. are some of them. But hopefully the author learned something about testing in production.
- RandomChimp 5y agoHoleup a second. Right at the beginning when writing your script you must have known that the random generator would end up creating valid numbers? Expecially when run for tens of thousands cases? There exist test ranges of numbers that are allocated in order to accommodate this type of testing and avoid exactly this problem.
- quenix 5y agoThis feels incredibly negligent. It seems obvious that generating thousands of random phone numbers and locking them without any other checks would lock some real phones. I'm puzzled that the author did not consider this when writing the script, let alone testing it or running it on live prod servers.
- heavenlyblue 5y agoI don't understand how people can defend this developer in the first place. I could understand if they were about 16 years old and actually abused by their employer, but generating random numbers to test the locking of phones? Basic maths should tell you that the probability of matching some real phone number is actually quite high. Moreover, what exactly were they testing? i.e. if they were testing nonexistent numbers, then how would they even know the phones were blocked in the end?
- shaan7 5y ago> Basic maths should tell you that the probability of matching some real phone number is actually quite high. Yeah well, the thing about pressure (of the sort the author has describing) is that you make mistakes - the kind where some times 3+3 = 9. Source: I queued jobs to nuke the whole production database (relax, we had automated backups+verification so we recovered fast enough; and then added some gating so make this more difficult in the future)
- throwdecro 5y agoBeing somewhat generous, I don't think it's because they didn't understand the probabilities, it's because they didn't think about the impact of what they were doing at all, beyond the problem they needed to solve. I.e., the only thing they were considering was whether the number entered at the start of the test would show up in the right places at the other end of their system. They simply didn't consider that an actual phone would be shut off along the way.
- jihadjihad 5y agoI pass no judgment, but merely take the developer at his word: https://new.pythonforengineers.com/blog/confessions-of-a-1x-programmer/ https://new.pythonforengineers.com/blog/confessions-of-a-1x-...
- polartx 5y agoWhy would he write and _publish_ this?!? Is this some kind of humiliation fetish? These are things that should maybe be admitted in a confessional, or therapy.
- svrourke 5y agoThis link is the cure to imposter syndrome, this is an actual imposter. I don't think they're even actually interested in programming and I cannot comprehend why they do it as a job or why they got hired.
- cycomanic 5y agoToo me this is another example showing how much behind "software engineering" is behind most otherl engineering disciplines. Processes in other disciplines are build very much into the practice that these things don't happen because people die if a plane crashes or a bridge collapses. Also that somehow software gets away with absolving itself of any liability (i.e. the usual "this software is provided as is...") is quite remarkable.
- treesknees 5y agoYou're just not looking in the right places, and your comment is a bit disingenuous. You're cherry-picking examples that don't represent an entire industry. NASA for example has an _extremely_ tight tolerance on software specifications, testing, and proofs [1]. Medical devices and other core infrastructure also has very robust development cycles to ensure bugs don't bring down the power grid or overcharge someones heart in a pacemaker. Meanwhile you can look at the auto industry and find example after example of shoddy engineering or manufacturing processes putting hundreds of thousands of drivers and families at risk. [2] [3] [4] 1 - https://www.nasa.gov/isd-robust-software-engineering https://www.nasa.gov/isd-robust-software-engineering 2 - https://news.yahoo.com/chevy-bolts-stressful-recall-161700057.html https://news.yahoo.com/chevy-bolts-stressful-recall-16170005... 3 - https://www.consumerreports.org/car-recalls-defects/takata-airbag-recall-everything-you-need-to-know-a1060713669/ https://www.consumerreports.org/car-recalls-defects/takata-a... 4 - https://www.cars.com/articles/explaining-the-toyota-floormat-recall-1420663233094/ https://www.cars.com/articles/explaining-the-toyota-floormat...
- 6gvONxR4sf7o 5y agoIt seems pretty clear that software is behind the auto industry in build quality/reliability too.
- travisgriggs 5y agoIt’s like trying to breathe underwater. Just because a person has the capability and know how to breathe, they can’t breathe if not given the proper environment to do so. Yelling at them “why didn’t you take a breathe?!?!” As you watched them drown underwater is just ridiculous and naive. But every time proper engineering doesn’t happen, we see a subset of cries like this, like it’s some moral failing in the engineer. There are surely some chronically stupid engineers out there, but 99% of the time it’s a culture that has subverted the doers below the discussers and the deciders that leads to this shitshow of “how did this happen??”
- treesknees 5y agoNot defending him, but he does state that unlike the other mobile providers he's worked with, this K-Pop company's backend system would lock _any_ phone number, not just the ones they're responsible for. Perhaps he didn't know this. I bet normally he'd generate random numbers not associated with the provider, check they show up as "locked" in the database, under the assumption that the backend actually doing the locking signal would reject the numbers due to them being under control of another operator. It's similar to our networking lab where we generate bogus traffic using random src/dst IP addresses under the assumption that our null routes and firewall will block them from the real Internet. In this case, the provider just decided no firewall was needed. We can't use "fake" IP addresses (yes there are IP ranges for internal use but that doesn't work for certain testing scenarios.)
- polartx 5y agoThe fact that he was running production code and not writing the result to any kind of log file, which would only take a couple more lines of code, one more dependency (if he wanted to skip some time rolling his own), and a couple of megs of storage, is so absurdly irresponsible, I’m surprised he wrote this post up at all—most would be too embarrassed.
- apcragg 5y agoThe author blames this on management pressure and deadline acceleration but it seems to me that they made an incredibly obvious and negligent mistake and is trying to pass the buck.
- nzmsv 5y agoShouldn't a company that has a process where a recently hired contractor can lock any phone in the world get some share of the blame?
- stuff4ben 5y agoExactly! There seemed to be a severe lack of any production control processes at this place. That's the fault of management and senior developers/architects. Contractors should never be given the keys to the kingdom like that. And if they are, then there should be some checks and balances to make sure they don't do something foolish like this. But I've worked at places like this where we coded "close to the wire" so to speak and it's just get the job done and who cares about quality. Management should absolutely get the blame when a screw-up happens.
- heavenlyblue 5y agoI am not sure how a non-technical director is supposed to hire a software engineer if software engineers themselves fail to hire good software engineers.
- axus 5y agoObviously he contributed the most to the problem, but doesn't a manager have some responsibility for the work they are managing?
- Iolaum 5y agoMore specifically for the deadline they are imposing without asking for feedback about feasibility and associated risks.
- l30n4da5 5y agoUsually not a great idea to run tests like that on production.
- swarnie 5y agoLook at this guy, implying he has a separate test environment. Probably has an adequate budget and manager buy in as well.
- stuff4ben 5y agoIf you haven't killed some production environment in your career, can you really call yourself a programmer? I kid of course, but I've been there, done that. It's lessons learned the hard way and I became a better developer because of it.
- kingcharles 5y agoI once ran a query on a live production web site thinking I was doing a SELECT but somehow ran an UPDATE which set all the millions of users' gender to 'M', regardless of their selection. Restoring from backups would have been a nightmare. And telling my supervisor would have been a nightmare. So I used everyone's selected title (Mr, Ms, Mrs etc) to reset their gender setting, and then put the keyboard down and went to bed.
- scollet 5y agoI think you made the wrong call here.
- inquist 5y agoOn the plus side, why are they storing gender info in the first place?
- kingcharles 5y agoBecause zero thought went into the account signup form. It just asked for whatever the coder who wrote it thought it needed. Shit, it might even have been me who wrote the signup. I don't remember. This was back in 1999. Now I try to ask for the absolute bare minimum of information to onboard someone. No information at all if I can! Times have changed. Damn, I don't even identify as a binary gender myself anymore and hate forms that ask me.
- jaywalk 5y agoWhy would you jump to the assumption that they don't need to store gender?
- mcintyre1994 5y agoWhat on Earth? This company just locks random people's phones as part of their testing, and nobody realised that was a problem until they did it to thousands at once?
- Ginden 5y agoWhat is even worse - this company can lock phones that aren't their property.
- jeroenhd 5y agoI think that's the real issue here. The Korean company (probably Samsung or LG based on the implications here) didn't validate that this company had permission to lock these phones. Things like these can happen by mistake (or incompetence, in this instance) and the unnamed company should have never let this happen.
- mrosett 5y agoHow did he not get that running a script to lock random phones would lock random phones?
- leephillips 5y ago“the main product was an app...that would lock the phone if it was stolen, or if the customer stopped paying. The app was built as part of the Android OS, so you couldn't uninstall it. It would lock the low-level features that allowed you to make calls, use Wifi...until you paid up. All good.” No, not all good. There’s the problem right there. If I stop paying you can stop providing me the service I’m paying for. Using WiFi has nothing to do with you. Breaking my property this way, without a court judgement allowing you to do it, is not only unethical, but should be a criminal offense, and is at least a civil tort. The potential liability here is literally unlimited. I believe responsible cell providers allow you to call 911, or the equivalent in your location, even if your account is delinquent. But you can’t do that if your phone is locked in this way. Did anyone die because of this product and this guy’s recklessness?
- tcskeptic 5y agoWhy do you assume the users own the phones? There are plenty of business models where the company owns the phones and the user gets to use it by paying a monthly fee. One of the services being provided may well be the use of the phone itself.
- cyxxon 5y agoThat would be something to discuss with the user and the company that then might own the phone. In this case though, someone completely unrelated to the contract locked the phone. Someone in another country. Someone working for some completely unrelated telco company. Someone who just randomly locked some phones by banging out a script to generate phone numbers, without checking if the phones were owned by the user, their telco, or my grandmother.
- junon 5y agoGreat so if my employer decides that I shouldn't have the phone anymore I can't even call 911 in an emergency.
- karteum 5y agoEven if/when they don't fully "own" the phones, users ought to always be able to access their data (at least for backup) and call emergency services (which is a legal requirement in many countries). More generally and regardless of time pressure, I can't believe someone can "play" with such test a script (used with a production tool) without even thinking it might fuck up and hit real phone numbers... (and if the developer was aware that his approach might fuck up, then he should make his boss aware that he had no time to fully test the feature rather than play russian roulette. Time pressure is not an excuse for everything...)
- helios_invictus 5y agoThe lack of judgement in both the articles content and publishing the article itself is horrifying.
- jeroenhd 5y agoI also get the feeling that the author didn't learn much from this experience. The lessons learnt here are "should do better testing" but even that didn't stick based on the paragraphs after that. The complete disregard for the victims of this company's incompetence ("the poor sod in Peru who couldn't make phone calls or post duck-face pictures to Instagram") makes this a very different kind of article from the usual "I fucked up production the other day" posts that you see here now and then. My conclusion after reading this is that the author didn't learn anything, the company didn't get punished enough to re-evaluate their testing and the Korean company's systems are still vulnerable to unauthorized locking. The real lessons I've learned from reading this are to avoid either LG or Samsung and any provider using Trustonic. And not to hire the author, I suppose.
- thih9 5y ago> Except for the poor sod in Peru who couldn't make phone calls or post duck-face pictures to Instagram, but no one asked him. There are countless situations where suddenly not having a working phone means a serious problem. The article presents an extremely negligent look at the issue.
- trutannus 5y ago> extremely negligent look at the issue Yes, the part about them overwriting the data from each cycle struck me as odd. I don't think that's good practice even if testing.
- Arnavion 5y agoIt doesn't seem unusual to me. You write a test script that generates random numbers and saves them to foo.txt, then reads them back from foo.txt for the purpose of running one kind of test. Then you wrap that in a loop for every kind of test. You end up testing everything, but foo.txt only has the numbers from the last test. To be clear, I'm not saying it's not a bad practice to do that. But the context is that it was a one-off job to do the testing, not a proper test framework, for which this kind of half-assed approach is reasonable.
- trutannus 5y agoTrue, I think I have something like that in multiple places for tests that are used to validate prod code. But our code won't lock people's devices down with no recourse. I think what made me scratch my head is that they didn't consider that generating random phone numbers might lead to collisions. I've been hesitant to publish fake datasets that have randomly generated and validly formatted SIN/SSN numbers in it for that exact reason. Even if the risk is low, I'd still think keeping indefinite logs of that would be a good idea.
- sundvor 5y agoThere are indeed large ranges of highly valid reasons why having phone/ Internet connectivity is highly critical. But sure, "post duck photos". I understand it was perhaps meant as a paraphrase, yet it is indicative of an almost callous level of arrogance at the author's company.
- brunooliv 5y agopigeonhole principle not strong with this one...
- otrahuevada 5y agoI've done this, but with manager-vetted data. Conversation (in writing) - "We NEED to run this incredibly risky thing yesterday" - "Ok, I'm going to use this data. Is this data OK to you?" - "Yes" - "Running" Great way to avoid getting thrown under the bus if it all goes south.
- jaywalk 5y ago"I validated the data to the best of my ability, but I had to rely on otrahuevada's experience in choosing the data. Unfortunately he really screwed up." If your manager is the type to throw people under the bus, there's always a way.
- otrahuevada 5y agoI mean yeah when there's a will there's a way, but keeping your manager and reports informed about your work and its progress is kind of the only way to cover your ass available to you as a non-manager.
- slingnow 5y agoI believe everything you need to know about who would be responsible for this kind of "mistake" is right here from the article: > Our own internal tool worked well (I would like to say it was all due to me, which would be an arrogant thing to say, but also 100% true, so I will say it: It was all or mostly due to me) This is a person who is incredibly overconfident in their limited skills and hasn't yet developed any sort of self-awareness or self-reflection. The fact that the rest of the article blames everyone else for his mistakes further proves the point. > I could have tested it better, but that would have meant working late into the night. No thanks. I had already worked a few late nights/weekends and I was done. Uh huh.
- Jiro 5y agoPeople in the industry tend to laugh at having to work late as some sort of standard practice in the industry. But if the time allotted to program it is not enough time to program it in a properly tested manner, that's a management problem, even if the programmer could have worked overtime and gotten it tested better. It's a bad practice for managers to schedule things such that they can only be completed with overtime, and managers bear a huge portion of the responsibility for any problems that result from that practice.
- oolonthegreat 5y agokinda unrelated but can someone explain to me how this one korean company has access to infrastructure to lock any phone anywhere in the the world? am I misreading something?
- albert_e 5y agoYes that is the most surprising and shocking thing I learned in this blog -- even more scary than the horrifying developer/tester tale told.
- alliao 5y agowonder if it's baked in android?
- FDSGSG 5y agoSamsung can remotely deactivate Samsung phones? Wow, shocking. Wait, this actually has been normal since Apple introduced the Activation Lock in 2013.
- ALittleLight 5y agoPutting aside the issue of locking real phones this has an issue of state persisting between test runs. The previous execution of a test should not change state for subsequent execution. This is a long way of asking why the phones weren't unlocked as part of a cleanup step in the same test. Although, at this company, if they did this, they might not notice they were locking and unlocking phones whenever this ran and then periodically disable and enable their customers phones.
- racuna 5y agoI'm from Chile, never heard of thousands of phones being blocked here.
- shantnutiwari 5y agoAuthor here. I shared this yesterday, didnt get much upvotes and quickly vanished. Thanks to /u/nixcraft for sharing it again! Looks like 2nd time lucky. Im seeing the same type of comments here and on Reddit, I'll try to answer a few common ones. 1. Yes, I know it was stupid testing on production. But we'd been told we needed to release the product on Monday (this was Friday), no objections. A previous project manager had been fired for not being fast enough, and the whole test department was at risk of redundancy. Vice president level executives were asking for updates in daily standups. Like I say in the post, sometimes, you do what you're told or you find another job. If it hadnt been the height of Covid , I might have chosen option 2. 2. Why didnt we test on a staging / dev server? We did have those, but they didnt work with the K-Pop companies servers. And we didnt have time (or enough people) to make it work. Lots of other engineers decided to just quit. Besides, we never thought K-pop would actually lock phones we didn't own. 3. Some people have asked about it: This type of app is perfectly legal. If you get a phone on contract, you don't actually own it till you pay it off, and the company is entitled to lock it. This is builtin on iPhones, Android need external apps to enforce this. 4. While it may seem stupid in retrospect (LOL, this guy calls himself an engineer, tests in production)-- at that time, it was just one firefight after another, and there was no time to think or plan. I can joke about it now, but at the time, it was one of the worst pressure cooker environments. Some updates: After we released the 3 products, the whole QA/test department was made redundant. Including my boss, who was fired when he was attending his mother's funeral. Funnily enough, I stayed because I knew details of a legacy product. So far the last 3-4 weeks, I had no boss or anyone to report to.
- macintux 5y ago> Including my boss, who was fired when he was attending his mother's funeral. And people wonder why big companies are generally loathed.
- systemvoltage 5y agoThis is a false generalization. It’s simply not true that this happens everywhere in big companies. Seems like a rare case.
- whitexn--g28h 5y ago+1(XXX)555-XXXX The 555 prefix for US phone numbers is reserved for fake phone numbers in the US and elsewhere. Use this block for testing.
- saithir 5y agoWhich would be excellent advice if he was using phone numbers. Which the phone's IMEI isn't.
- vodkapump 5y ago> Remember I told you the phones were "randomly" generated? The python script would randomly generate an 11 letter "phone number". Author says "phone number" and 11 letter(?) IMEI is 15 digits, so who even knows what the author means at this point.
- JanisL 5y agoI've noticed more as time has gone on people are working on critical software without the skills or organizational structures required to actually do so safely. What I'm wondering is if this is a change that's happened more broadly or is just something that I didn't notice before? Overall the average years of experience in the industry has dropped in the last decade due to the influx of new developers but what's the deal with inexperienced people working on critical systems without oversight from people with more experience? While I feel the failures from the developer here are pretty bad I feel the organizational factors that allows this situation to happen are far worse.
- JTbane 5y agoThe conclusions made in this blog seem wrong, it's perfectly fine to write these kinds of tests if you have to as part of a rush job, but WHY DO IT ON PROD? You know 100% that you are making calls to the 3rd partys' prod, assuming "random" numbers aren't real is incredibly negligent.
- diebeforei485 5y agoThis makes me curious if iPhones can be remotely disabled by a carrier.
- da39a3ee 5y agoThe main lesson here is do not, if you can possibly avoid it, accept a job at a company with a level of engineering quality as low as described in the article. You probably won't get fired or taken to court; the main problem is that you will have no-one good to learn from and you will lose terribly in career satisfaction and future earnings. If you're new to the software engineering job market, try to hold out for a company with a real team, and some colleagues that you look up to technically. It is hard, the jobs that are easier to get will be tiny companies where if you even have a colleague in engineering, they will be clueless.
- forgingahead 5y agoThis is a horror story, not because of the author (sorry no sympathy for you mate), but because of the dystopian nightmare that we've turned commercial transactions into. People presumably bought a phone, and because of some nonsense "business" plan, there is a recurring charge they are still liable for, and the manufacturer can arbitrarily lock and disable the phone? I know there are a bunch of "explainers" here on HN, happy to jump in and say that the phones were subsidised, consumers wouldn't get access to the phones if not for this scheme, and so on. But please stop that and take a step back. We've perverted the traditional expectation of any commercial transaction to own the item to instead be a permanent debt for regular consumers. You will own nothing, and you will be happy -- indeed.
- distantaidenn 5y agoAs others have pointed out, I wouldn't even have created this article, but to each their own. I'm assuming this client company was Samsung, due to scale and the author calling it the "K-Pop" company. They are known for using idols in their marketing. Still a little cringe to call it that. My take away from this article: 1. This act was extreme, bordering on criminal, negligence. Some peoples' livelihoods depend on access to their phone. I assume most people using this service are of a lower income. I wonder how many jobs may have been lost due to their not having access to their mobile for a day. 2. Samsung needs to hire better contractors -- but this is what happens when companies go with the lowest bidder, I suppose. I've seen it time and time again.
- miyuru 5y agoI think its LG, this a blog post from the said company. https://www.trustonic.com/news/lg-mobile-smartphones-to-implement-trustonic-security/ https://www.trustonic.com/news/lg-mobile-smartphones-to-impl...
- innodb1 5y agoImportance of testing has been told multiple times,even though we all ignore it largely. I vividly remember an incident ( this was in the early 2000's ) we had a couple of Linux machines both remote and local. The testing and development happens in Local and the code is pushed in a short time over dialup connection. One of my team member had a instance of remote open. He was deleting some cache files in his local , he by mistake typed in rm -rf / instead of rm -rf * in the remote machine ,assuming it was his local development machine. He didn't realize the mistake and finding it was taking much longer than expected he cancelled the command. went home happily. He was using a common login. Though he informed his lead , the tech lead also didn't realize the seriousness of the command that moment. What actually happened was every file which had 777 permission ,got deleted till he pressed Crtl-C. Now some of the files in production server was gone and some were still intact. This led to lots of confusion , the person who issued the command didn't realize his mistake , till the server admin who was about to loose his job for faulty permissions some how convinced the management that someone had issued a wrong command , and that he was somehow not at fault. Our Boss was super irritated when he found out what happened. The person who typed the wrong command , along with his tech lead left the job after a few months on their own. It taught very valuable lessons, Especially not to type rm -rf without checking multiple times. I still remember this incident every time I delete something.
- musiccog 5y ago> It taught very valuable lessons, Especially not to type rm -rf without checking multiple times. "Measure twice, cut once." One of the many uses I have for a mindmap in my daily use (Freemind 0.9.0!), is as a place to write out sysadmin/sql commands in a structured way before executing them. If I can test it on dev before production - great. If I have to run it directly on production, following this method (religiously!) has saved me from any catastrophic (read unrecoverable) issues. When you are under pressure it can be easy to say 'I don't have time to write out the commands'. When touching production servers, you need to spend the extra minute or so to write it out.
- VortexDream 5y agoCan you maybe go into a bit more detail about how you use Freemind? It's not something I've heard somebody do before and it sounds interesting.
- voiper1 5y ago"To err is human, to really screw up, you need a computer" Computers enable mistakes to propagate at tremendous speed...
- solarized 5y agoI always love when "TIL things" like this getting popular. Many peoples still threat it as a disgrace. Whereas other peoples really need it as a priceless lesson learned.
- dotancohen 5y agoI love learning from mistakes. Particularly from other peoples' mistakes. Articles like this are invaluable in high-technology fields, where one cannot afford the time nor consequences of making every possible mistake.
- lifthrasiir 5y agoBack in time I was working on an online trading card game, the integration test was running in the live server. (Don't ask me why. All I can say is that the server had to be hastily built for some reason and it was known to be suboptimal in several aspects.) In order to avoid inference the test created tons of placeholder accounts and corresponding environments and it actually ran mostly well given the circumstances. One day I received a report that several accounts were simply disappearing and had to be re-registered every few days or so. I had to go through lots of recorded logs to determine what was happening: one of these accounts was named "Card" and some test was using it as a placeholder account, so it got removed when the test finished. How apt it is, that someone would use "Card" or similarly rare handles in a TCG. The testing infrastructure was updated to ensure that placeholder accounts can't coincide with actual accounts and affected users were compensated.
- KronisLV 5y ago> Our company had been bought by an investment firm, and they wanted their pound of flesh. All projects deadlines were moved up, and at one time we were testing 3 products in parallel, all with different requirements. What are you supposed to do in such a situation? Jump ship? Or attempt to provide pushback against unreasonable practices and deadlines (which may either be listened to, or just get you fired), or simply ignore the deadlines altogether because they're made up and follow the best practices anyways? To me, it feels like there are no good options if you plan on staying at a company like that. > I could have tested it better, but that would have meant working late into the night. No thanks. I had already worked a few late nights/weekends and I was done. Regardless, it feels like you should put your health first, always. So, in a situation like that, consider the potential risks of what could happen due to badly written code, how likely those are to occur if you don't double check your work and if it feels like it's a good idea to check it but you don't have the time, just do it the next day. Of course, this may or may not play into the best practices and getting fired bits above. On an unrelated note: > b) Some project managers and Important sales managers had also been doing the same thing – making up phone numbers and locking real phones. Of course, they were doing it manually so only locked 3-4 phones, while we locked 10,000, but the final effect was the same. We were all pissing in the pool. Everyone pretended it was an honest mistake. Lol, look how silly we are. Not a big deal, live and learn ole chap, live and learn. Except for the poor sod in Peru who couldn't make phone calls or post duck-face pictures to Instagram, but no one asked him. This feels illegal. It probably should be illegal. Is it?