15 ms·

What is a coder's worst nightmare? (2014)

476 points by oskarth 11y ago

dalke 11y ago
That seems more like a fictionalization of "Reflections on Trusting Trust" than a true story.
- brudgers 11y ago
  Link to Reflections on Trusting Trust: https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomp...
  - cshimmin 11y ago
    Or for those who like explanations from secondary sources: http://c2.com/cgi/wiki?TheKenThompsonHack http://c2.com/cgi/wiki?TheKenThompsonHack
- auntienomen 11y ago
  Or, perhaps, the grad student antagonist got the idea from the essay.
- vidarh 11y ago
  Either that, or said student had read Reflections on Trusting Trust. It was published in 1984. The 3B2's and Tymnet were current around then, and the authors bio indicates he's been programming long enough for the timeline to fit too. It's not that unlikely that someone, somewhere read the paper and decided to have some fun at some point. The thing that makes me tend to agree with you is that he hasn't mentioned any "finally, years later, I found out where that guy got his idea" moment in it.
  - skybrian 11y ago
    He did sorta mention it, but not by name: "I suddenly realize it's in the compiler. It was the compiler. And every time you compile the original code and run it puts in the subliminal message code into the source code. I'd heard of this before."
    - vidarh 11y ago
      Ah, yes, I missed that... Thanks.
  - pdpi 11y ago
    There is an "I'd heard of this before" in there that I do attribute to the paper.
- joelbugarini 11y ago
  Mick (OP) responds to this assumption in the Quora comments: I just read that. This guy certainly wasn't Ken Thompson. But this happened to me in 1991, seven years after the date of publication you posted, and the "login" portion did the exact same thing Ken was describing as far as the secret password. This was probably Thompson's genius idea implemented by the grad student. The mechanism was very different though in the compiler, but the outcome was still a poisoned compiler. I had heard of self-modifying compiler before, too or I probably wouldn't have thought to look there myself. I'm not surprised to see Thompson behind the original idea.
  - keithpeter 11y ago
    I thought of the 'trusting trust' paper just as I reached the paragraph about the compiler. As an ex-PHB rather than a programmer, I think perhaps that Dr Phelps could have had taken a bit more interest in the project &c. Perhaps had a look at the code now and again as simply asked "why does this look so unlike other people's code?". Might have sent a shot across the bows of the bad apple.
    - Drdrdrq 11y ago
      This. I wonder what made the graduate so angry to go to such extreme lengths to protect his code. And why nobody noticed it - did they even have the right to fix the code?
  - anon4 11y ago
    I don't doubt the story's plausibility, but I'm left wondering how the student could have replaced the compiler with his own. Usually you need root in order to mess around with the system like that.
- jtheory 11y ago
  I assumed for the first half it was going to turn out to be a psych experiment (and there were several other compsci students getting similar "contracts") -- especially as the hurdles he was jumping were clearly intentionally set up.
crb002 11y ago
These days you can just run strace and see all the OS calls.
irascible 11y ago
This is pretty unbelievable.
- seanwilson 11y ago
  Yeah...I would have compiled the code on a different machine then wiped the original machine if it had clearly been hacked.
  - mytochar 11y ago
    I kept wondering that as well. I'm wondering if this was a different time, or if it's just a fun story
    - mwill 11y ago
      3B2, so probably like early-mid-80's. Likely didn't have many options.
  - vidarh 11y ago
    This was a university network in the 80's most likely based on machine model. Assuming the story is true, and not as mentioned elsewhere, a fictionalization of Reflections on Trusting Trust, the person in question most likely didn't have a standalone system that he could wipe, and didn't have copies of the OS media to install. Note the details of an AT&T tech coming in to fix the compiler install. That's quite believable for a university installation like in the story. When I went to university in '94, if we'd had to deal with the same thing I'd have been in largely the same situation: Sun and SGI workstations that largely booted via NFS (some of the SGI's were exceptions, I think, but most parts of the system were mounted via NFS there too) where anyone trying to re-install the compiler etc. without the approval of the computer department would be in big trouble (and have to hack the system; though the protections were relatively weak). And while we certainly could install compilers etc. locally in our accounts, getting hold of a cleanly compiled compiler for some of the systems we had would not have been all that trivial, even though the systems we had were all more mainstream than the 3B2 mentioned in the article. I wouldn't be surprised about an outside contractor having a hard time getting the computing department to believe that this problem was a hacked system. I'd expect the reaction would be to assume that he was just trying to make excuses for why it was taking him so long to find the actual problem.
SilasX 11y ago
This is funny, but it's actually more like a programmer's second worst nightmare -- "It doesn't work but it should". The true worst nightmare is, "it works, but it shouldn't", because that means your whole model of the domain was wrong, rather than some isolated, fixable component.
- Natsu 11y ago
  I remember having one of those moments with a function I hadn't implemented yet and being incredibly confused as to how it could possibly work in spite of there being no code yet.
  - moyix 11y ago
    Yes! I once started testing a class I had written, only to discover methods I hadn't finished writing yet were working perfectly! It turns out I was accidentally reimplementing a library function, and I had chosen the exact same names and method signatures for the (admittedly small and fairly obvious) class.
    - jacquesm 11y ago
      Hehe, I had more or less the opposite of this once. I spent way too much time tracing a function in ever more detail going on the assumption that it must have been the source of the problem. Only... it was never called in the first place, it was a library module that took the call and it was misbehaving...
      - Natsu 11y ago
        Just for the record, my test was a bit off and wasn't hitting the non-existent code at all.
- nkozyra 11y ago
  It's not a nightmare, but it's incomparably frustrating. I'm more motivated to figure out things that work when they shouldn't then the other way around.
0xCMP 11y ago
You know, I know everything posted on the internet is fact /s... but I really hope this story is true cause it's just crazy. I mean, who want to think that the main thing that would let you create code is the thing that is infected? And it seems like there weren't any other options for languages (which hopefully weren't compromised as well) so there would be no really getting around it.
- vezzy-fnord 11y ago
  I mean, who want to think that the main thing that would let you create code is the thing that is infected? I'd wager that people who work with esoteric embedded toolchains develop an instinctive distrust in that layer.
lunulata 11y ago
For the love of Zeus... it's called git, use it!
- davb 11y ago
  The 3B2 computer mentioned in the story was around in the mid-1980s. Git's initial release was in 2005.
  - iLoch 11y ago
    Not to mention the malicious code was in the compiler, not the source code.
- vidarh 11y ago
  After you invent time travel. At the time of this story, based on the details (3B2 computers and Tymnet) RCS was likely still current and CVS likely didn't exist. There were other alternatives like SCCS, but speaking as someone who wrote code back then: It wasn't until the late 90's it started being weird to not see source control systems used even in a corporate setting. The last time I worked in settings were source control was not universally applied was as late as '99.
  - plorkyeran 11y ago
    As a point of reference, the Joel Test was written in 2000 and even then "Do you use source control" was something you had to actually ask.
    - ddebernardy 11y ago
      You'd be surprised. One of the questions I systematically ask when interviewing developers is whether they're experienced using version control systems. Not all are. The same goes for automated testing, following coding standards, or (of all things) using a bug tracker.
matobago 11y ago
Once that you start using version control systems, this shouldn't be a problem. A real nightmare for me was when I inherit a bunch of code full of uncommented regular expressions
- greggyb 11y ago
  How does a version control system address a malicious compiler? Unless you're suggesting the entire system be under VCS?
martin1975 11y ago
Reverse scheduling. That's the worst one by far.
userbinator 11y ago
That's not a worst nightmare for someone who knows how to read assembly language and use a debugger. I would've narrowed that down to the compiler the first time and probably debugged into it too (providing whoever did this very clever hack didn't also do something to the debugger... but if the system is behaving this oddly, it's certainly better to bring one's own tools.) Seeing the CPU do something it shouldn't, being traced by a hardware ICE, now that is a worst nightmare - and one I've actually experienced.
- jacquesm 11y ago
  That sounds like a story worth telling.
- mcherm 11y ago
  Story time? (Please!)
  - userbinator 11y ago
    It was a long time ago, but I remember we were working on an embedded system controlling some industrial equipment, and it randomly crashed; the time between crashes was long enough that it'd take several days before it happened, so even getting a trace of the crash was an exercise in patience. Eventually we did get a trace, and it turned out the CPU would suddenly start fetching instructions and executing from a completely unexpected address, despite no interrupts or other things that might cause it. We collected several more traces (took around a month, because of its rarity) and the addresses at which it occurred, and the address it jumped to, were different every time. Replacing the CPU with a new one didn't fix it, and looking at the bus signals with an oscilloscope showed nothing unusual - everything was within spec. We asked the manufacturer and they were just as mystified and said it couldn't happen, so we resorted to implementing a workaround that involved resetting the system daily. Around a year after that, the CPU manufacturer released a new revision, and one of the errata it fixed was something like "certain sequences of instructions may rarely result in sudden arbitrary control transfer" - so we replaced the CPU with the new revision, and the problem disappeared. We never did find out what exactly was wrong with the first revision, other than the fact that it was silicon bug.
jimmeyotoole 11y ago
That's so messed up. I'm so glad I started my development career in an age where one could spin up a VM in minutes to isolate an issue like this.
forgottenacc56 11y ago
Programming.
rdtsc 11y ago
Other coders' nightmares: * Security : customers acccounts leaked, identities stolen, bank account depleted * Undetected data corruption : customers' data is corrupted and it has seeped into the backup stream, even the off-site one, and that is corrupted as well now. * Rarely occurring simingly random concurrency related bugs that manifest themselves once a month or longer. * Time : making assumptions about it, dealing with it distributed systems etc. Nobody thinks about it until it stops working or systems' time gets snapped back somehow.
- jacquesm 11y ago
  Anything involving interrupts and timing. Especially when the timing is so critical that it is time to break out the logic analyzer. Bugs like that can keep you busy for weeks.
- bm1362 11y ago
  Oh man, I used to work on a large service that had to aggregate data from a bunch of downstream services and submit it upstream. At the end of the month, billing would run off the data and send out invoices -- tracking down subtle data corruption was nightmarish and cost me my youth.
- Splines 11y ago
  * Works on my machine: You can't repro a customer bug in-house. Something about your internal setup is making a difference. Even worse: You work for a big-corp that is a platform for said customers, and so god knows what another team has running in your environment in the name of "dogfood all the things"
  - jschwartzi 11y ago
    I saw one once where the cell carrier actually made the difference. As in I had to fly to a foreign country before I was able to reproduce the bug.
  - lotharbot 11y ago
    Years ago, my wife had a bug that only happened in a non-debug build and only on one specific system. It was a recursive tree traversal algorithm, written in C, with an OB1 error in some pointer arithmetic. If memory was allocated in a particular order, the function would read off the end of level X's data and find level X+1's data, which was structured exactly as expected, so it would end up processing certain nodes twice. In the debug build there was a debug variable being initialized to null, which was in between the two blocks of data and therefore got interpreted as a terminator.
TerryADavis 11y ago
I made a dialect of C called HolyC. I made a compiler. There is nothing obfuscated. In planet of the apes, original, a white man told a monkey he made a flying machine. The monkey did not believe him.
av9 11y ago
Not having a job is my worst nightmare. Realizing your skills are being undercut by a fresh college grad who can get enough pats on the back while skidding along with code that works. Realizing the language you cradled has now become outdated and the other engineers that have found out the new next 9 years of hotness are too good for you to compete with. Having to interview for a new job and realizing that you can't pass the white board questions because you have fallen behind with the expectations of young blood. That's my nightmare, becoming irrelevant.
- Splines 11y ago
  > Having to interview for a new job and realizing that you can't pass the white board questions because you have fallen behind with the expectations of young blood. That's my nightmare, becoming irrelevant. I'd like to think that interviewing for positions that aren't entirely focused on new-hotness still leans on heavily trod areas of CS. Data structures, sorting and searching.
  - nikatwork 11y ago
    A large portion of modern dev jobs are either CRUD apps or data munging. Knowledge of algos are not necessary for those for those domains. It always amuses me when I'm interviewing for some vanilla web app job and they start pitching maths questions. It's filtering on the wrong criteria.
    - gaius 11y ago
      No, they are filtering on the criteria of "recently revised this stuff for an exam" i.e. "we want cheap and easy to exploit 23-year-olds but we can't say that in front of HR".
- vidarh 11y ago
  I've found the solution (for me at least), while I love spending time on new tech, is focusing on being a "solution provider" rather than focusing on specific technology. The moment you stop including the name of specific tech on your resume and start describing how you helped solve X problem that saved/made the company Y amount of money instead, you open the door to positions that may be as hands on (or not, depending on what you want), but that pay better and where the interviews are not focused on tech trivia at all.
joshu 11y ago
this seems like one of those stories that got slightly more intricate every time it was told. and it's been told a lot.
logicallee 11y ago
Here's a coder's worst nightmare: "I managed to get reflection working* in C++, by registering class functions in static variables. This lets you configure routes like in Ruby on Rails. Since C++ is somuch faster, can you port the rest of Rails over as well? You'll just be doing this for the next 4-5 months." * https://news.ycombinator.com/item?id=10607029 https://news.ycombinator.com/item?id=10607029
deleted 11y ago
[deleted]
devonkim 11y ago
Aside from the issues where all your assumptions of your tool chains, OS, and computational reality is melting around you this seems a lot related to lack of feeling of control and infomation. You have to keep calling people from vendors and other departments because your talent, knowledge, and skills hit a brick wall in the face of bureaucracy / compartmentalization. It is that you don't know what to trust anymore, you have to be resourceful with time, and you have limited access to the information / control needed. This is how I explain why my technical job is difficult in a typical enterprise organization and why you need to be far better than average to produce merely mediocre results.
macspoofing 11y ago
Make a technology, platform, architecture or framework choice. After a significant amount of effort realize you made the wrong choice.
- collyw 11y ago
  Worse when you make the right choice but everyone else flocks to the crappy choice.
DrScump 11y ago
Management that doesn't care about regressions then later also sticks you with placating the customers who get screwed by those regressions.
- NegativeK 11y ago
  Management that does care about regressions and you both still get stuck with placating the customer, because the customer always wanted features instead of dealing with debt.
jbombadil 11y ago
Users with ideas
shabbaa 11y ago
Coder's worst nightmare? When you are the last line before something gets pushed into production, and the fear that it will fail IN production.. have i missed something?
hashkb 11y ago
I was hired by a psychologist My money was on the psychologist the whole time. I still kind of think it was Dr. Phelps and maybe Mark the admin and the AT&T tech are grad students in disguise. So I guess my worst nightmare is finding myself in a similar situation and later finding out my boss set the whole thing up for funzies.
- balls187 11y ago
  Having been playing Fallout 4, I couldn't help but think this was some wacked-out psychological experiment.
- emcrazyone 11y ago
  I was hired by a bookie ;) I have a somewhat related story. Circa 1995 or so, I was working on software compiled on a 486DX computer. Back then the computers had L2 cache in SDRAM chips you plugged into the motherboard. I was moonlighting with a few guys to develop software and each night myself and two friends met up in office we were renting. There, on one of those fold out tables, we toiled away each night writing code. The code we worked on was for off shore gambling (aka a bookie). Back in the 90s you could walk into a convenient store and pick up an magazine; usually autotrader magazine. On the back was an 800 number you could call to place bets on football, hokey, or baseball. (Called HOBs bets in the industry). Anyway, because of the nature of the software and the people we dealt with, things were a bit hairy at times. The bookie we worked for paid us generously but, at the same time, expected perfection (i.e. software that just worked). One evening, as usual, the three of us met to work toward the next release of the software. The next release of the software was to include a feature for boxing bets. Mike Tyson was about to be release from jail and the bookie was anticipating various future business in taking boxing bets. So this one evening we were doing a software build to release as beta software. I did the software build and ran a battery of tests we normally run and all worked fine. To transfer the software to the offshore network we used a dial up connection. I pushed the software out which my bookie contact would run and make sure all the logic was correct. A few days go by and we get a phone call that the software is not functioning. Sometimes it crashes (which was rare) or sometimes just strange things would happen such as strange blinking characters on the screen. I collected enough information to try and replicate the steps necessary. You can imagine my customer wasn't too happy. So the three of us basically stopped working on the code to track down the bug. We spent a week looking for the bug we were sure was in the code somewhere. Back then we didn't really use a code repository but we did have a diff of the beta release and previous release. Pouring over the code changes we just could not find the problem. It even got to the point where I would recompile the beta version, run it, and could actually get the code to crash. Comparing to earlier release it would not crash. It was by chance that my fellow coder, sitting right next to me, started running similar tests on his machine while I was off making a food run. It was sort of normal for us to group up around my workstation when something needed a collective look. Long story short it turned out that one of the L2 cache chips was causing my compiles to become corrupt in just the right way to cause the code to be not as we typed it in. Anyway, thought I'd share. Although not a malicious act, it was one of the worst things to track down. Fast forward to my current life... I work in the embedded field and recently solved a bit-flip problem in one of the products my company produces that relies on NAND memory. That little incident from my past certainly has served me.... ;)
gesman 11y ago
Bottom line: insists on hourly rate.
- fishanz 11y ago
  I'm in this camp, but I keep hearing stories about the pot 'o gold. "Charge what it's worth to the client." This happens?!
musesum 11y ago
"Best Fit" memory management. Malloc three buffers, in order: 64K, 1K, 64K. Now free all three. Next allocate 1K. Guess where it resides in the heap? Yep: the middle! After a while, memory was fragmented with 1K detritus. Even though you were supposed to have 500K available, there wasn't room for a 64K buffer anymore. This was the default for Microsoft's C compiler, in 1990. How did I find out? There was a project that was crashing into a blue screen of death after a few hours. The crashing library had #defines that went 15 levels deep. The stack dump wasn't very instructive. Even though the library ISV was based in Boston, I had to fly out to Santa Cruz to talk to the developer directly. He was somewhat belligerent: "you should have figured it out yourself!". 15 levels of #defines. He was the perfect example of the other side of Dunning-Kruger thinking his arcane shit was obvious. But, that didn't solve the problem. His fix didn't. So, I flew back to Boston. I started to trace everything with -ahem- #defines of the main APIs. It was turning into a Zen Koan: the sound of one app crashing. Everything worked with the Borland compiler. It was just the MSC compiler that failed. The company was one of the top 3 ISVs back then. Meetings about what to do would last for hours. There were technical politics: "Can we use Borland?" No; the company had standardized on MS. I could talk to their engineers directly, though. But, they didn't have a clue. So, I read though the docs to see what compiler switches were available. And there it was: best-fit. Fixed! So, I wrote an analysis about why best-fit was absolutely the worst possible memory management and sent it to Microsoft. It took 8 years for MS to change the default. So, the reason why software would blow up? Best fit. Why early versions of Windows with go into blue-screen-of-death? Best fit. How many developers had to deal with this? Worst nightmare in aggregate.
- davidst 11y ago
  You just brought back old memories for me. I wrote the heap manager for Borland C. I chose the next-fit algorithm for its balance of speed and reasonable fragmentation under common use cases. Best-fit performs exactly as you described.
  - musesum 11y ago
    Man, oh man, BC was the bomb! In a good way; maybe I should exclaim that it was the anti-bomb!
deleted 11y ago
[deleted]
meshko 11y ago
Unicode and timezones.
Jupe 11y ago
COM. 'nuff said.
melted 11y ago
Leading a project where in order to just not fall behind on maintenance and user support you need 2 people, and 2 people is all you're gonna get for the foreseeable future, yet management demands forward progress out of you. Fucking demoralizing and degrading.
- vox_mollis 11y ago
  Nightmare? This is the status quo in the majority of small-but-not-tiny organizations.
  - melted 11y ago
    Doesn't make it any less of a nightmare. I wonder what percentage of effort is wasted in the industry this way.
OSButler 11y ago
As a contractor, the fear is more on the client side than with the code. Coding problems can be resolved or worked around eventually, but the client can be a wildcard. In developer meet-ups I usually hear about coding issues like they are a puzzle, whose solution is being shared with the group. But the scary stories are usually centered around clients, since problems can arise there even if you provide perfect code.
- ju-st 11y ago
  When I read the topic my first thought was "customers!"
shams93 11y ago
Being so sleep deprived on a new gig that you mistake myvar: for :myvar
bdotdub 11y ago
Time zones
cpeterso 11y ago
My "favorite" bug was debugging a crash in Macromedia's Flash Player for Windows Pocket PC. It was an ActiveX control running in the Pocket IE browser. The crash was intermittent and only happened after running a 15-minute automated test suite on the Pocket PC, crashing deep in Pocket IE code. After staring at code print outs for a couple weeks, I found the bug: a BSTR returned from a COM interface was freed using CoTaskMemFree() instead of SysFreeString(). ARGH! A one-line fix.
frik 11y ago
Software that crashes while presenting it to the CEO or on investors day.
- simula67 11y ago
  https://www.youtube.com/watch?v=IW7Rqwwth84 https://www.youtube.com/watch?v=IW7Rqwwth84
  - i336_ 11y ago
    How did I just know what video this was before I clicked it..
joenathan 11y ago
I'm not a programmer but a Sysadmin. I ran into an issue a few months ago that nearly drove me crazy. I was custom building a new server for a customer, very nice build, Xeon E5 10 Core, 64GB DDR ECC RAM, Windows Server 2012 R2, SSDs, etc... Everything is going well, update and configure bios, install Windows, install drivers and software. Then start configuring the server, sever needs to reboot, reboots to a blue screen of death. Can't get it to boot up normally. Ok must be bad software, driver, etc... Time for a clean install, everything is going fine and then reboot -> blue screen. Look up the bug check code, no help. Another clean install, this time no extra software, no drivers, same problem after a reboot. Finally figure out that it is only happening after I make the computer a domain controller. After Googling with this new information, I find one thread on some forum where people are having the same problem, turns out it was the solid state drive, if you use a Samsung 850 Evo SSD with Windows Server 2012, it will blue screen if you make it a domain controller. I never thought a problem like this was possible. Sure enough changed the installation drive and no more problems. Nearly drove me crazy, took me two days of troubleshooting.
- vl 11y ago
  Haha, these multi-hundred-thousand-line-source-code SSD firmwares need to do something. In particular they detect popular filesystems and try to optimize some operations (specifically if filesystem is recoverable by check disk after crash, it's considered to be acceptable optimization; i.e. it's possible to delay actual commit of free block info for example). Seems like detection glitched on the blocks that contained data related to being domain controller.
  - digi_owl 11y ago
    Thinking about SSD and HDD firmware can make a guy reach for tape drives as a last ditch hope that data stays just data. At least with those (and to some degree optical media, but the write process there is more laborious) the storage and the RW logic is different pieces.
- leonroy 11y ago
  That is insane. And also one of the reasons I try to avoid being a system integrator whenever possible. It just takes up too much time - so much simpler having my friendly Supermicro distributor build my server for me!
hacknat 11y ago
My team and I are in the middle of a bad one right now. Our site is allocating incredible amounts of memory per page request, and it took us forever to figure out we didn't have a memory leak, at least in the traditional sense. We're running .NET, and its GC scheduler has 3 tiers for objects 0,1, and 2. 0 and 1 are short lived objects, but after an object has been around long enough the GC assumes it will probably be around forever and the 2nd tier garbage collector gets almost no time to run. We proved this by forcing the GC-2 to run longer and it brings the memory under control, so we don't have a "leak", per se, but we still can't figure out why these ephemeral objects are sticking around long enough to get so promoted.
- grandinj 11y ago
  Increase the size of your Eden space. You probably have something allocating very large amounts of temp objects, which is exhausting Eden. You should optimism that, whatever it is, but bumping up Eden will give you some breathing room.
- i336_ 11y ago
  Would it be too insane of an idea to go do a local full rebuild of .NET, now that it's open source? If Microsoft can get "[Build|Passing]", surely you can too... :P Build .NET, prove the issue still exists with the local build, then add new instrumentation to the GC routines inside .NET's runtime. Instrumentation never killed anybody... and if this issue persists too much longer... I can't deny that it's a crazy idea though.
deleted 11y ago
[deleted]
datashovel 11y ago
"You don't have to pay me anymore, Dr. Phelps, I just want lab time." This is nerd war. LOL
- datashovel 11y ago
  This also came to mind while reading: https://pbs.twimg.com/media/Bp_KeZnCAAAijRe.jpg https://pbs.twimg.com/media/Bp_KeZnCAAAijRe.jpg
thathoo 11y ago
Apart from the obvious concurrency based or non-easily reproducible issues, some of my personal nightmare scenarios: 1. Uncommented code, but especially uncommented regular expressions 2. Unnecessary abstraction that runs too deep & so is hard to comprehend 3. Also, take any tractable problem and add timezones to it. Now you have an intractable problem!
facepalm 11y ago
I've heard another anecdote from university where somebody changed some late digit of PI in the compiler. Very hard to detect...
teddyh 11y ago
http://www.commitstrip.com/en/2014/10/31/a-coder-nightmare/ http://www.commitstrip.com/en/2014/10/31/a-coder-nightmare/
Drdrdrq 11y ago
Internet explorer. A custom CMS. And sometimes some of the posted data is missing... On a single POS. And because data ath end of form is optional, system can't even figure out that something went wrong, it just silently discards it. Never figured out what problems that particular IE had, but the workaround was simple... I added an extra hidden input field just to check if all was sent. The theory was that at least operator could repost the form if it went wrong - however, it never did again. Thank you IE for your obscure bugs and nonstandard behaviour! Rip. </rant>
dijit 11y ago
forgive me for saying, I'm not a coder, I'm a sysadmin. I'm scared of having developers with a little ops knowledge take over my job. /devops/
- ownagefool 11y ago
  You probably should be afraid. Those developers are going to replace your tools with better tools, that they'll know and you will no longer have years of arcane experience to fall back on. Developers spend a lot more time thinking about abstractions, where to place things and creating & controlling nThings. If you cannot do these things well, you need to learn them. The best way to do that will be to ride the bandwagon, work together with devs and learn from each other.
CM30 11y ago
That's a pretty messed up story, though I'm not sure it'd be a coder's worst nightmare. I'd think that'd be if some major project you were worked on got hacked... while you were enjoying the weekend off and had no access to your work computer. Or just if your project developed a show stopping bug at that point in general, since you'd get the feeling that 'oh god, we've been done by two days' or 'we've been serving malware for two days'. Or maybe if you'd found a breaking bug that needed fixed, yet no answers were available online, no one in the room could help and the site was due to go live that night. All that time worrying, refreshing your Stackoverflow question and trying in vain to find a solution would be torture.
- pjc50 11y ago
  The Knight Capital bug is probably most people's worst nightmare: http://pythonsweetness.tumblr.com/post/64740079543/how-to-lose-172222-a-second-for-45-minutes http://pythonsweetness.tumblr.com/post/64740079543/how-to-lo...
digi_owl 11y ago
Stories like these makes one wonder if "mere mortals" should just turn things off, scrap them in a industrial chipper, and go back to pen and paper.
mbrock 11y ago
I worked on software that encrypted entries with a custom C++ plugin to SQL Server. Nobody knew how it worked, it was implemented long ago and considered magic. Some entries on some installations sometimes would become totally garbled, which was a seriously dangerous issue. This was a pure Java shop and I was almost the only engineer who even knew C++ since the reorganization that happened before I joined. Turns out the affected installations all used custom compiled DLLs with hard coded specific crypto keys made up by someone long ago. We didn't really know how to deploy new versions of this. But I found some version of the code at least, on some old file server, so I could sort of debug it. It took me a while to figure out but after working on it on and off for a few weeks I found a crucial race condition in the code path that looked up the crypto key. Turns out if you were unlucky after a SQL Server reboot, a few entries could get encrypted with a "default" key. So I was able to decrypt the garbled entries on affected customers at least. We decided to change the whole method of encryption since it was obviously hacked together by some cowboy coder a decade ago. But in the meantime, the easiest way to fix the issue for the affected customers was a method I taught to one of the tech support guys. I told him to RDP into the affected server, open the crypto DLL in Notepad++ and do a string substitution on a particular ASCII sequence that would change a certain character into a tilde. And then reload the DLL with a SQL Server command. Because through disassembling the binary I found that this would sufficiently increase a certain value and fix the bogus comparison that caused the race condition. The nightmare here was mostly because of custom DLLs running semi-unknown code on remote servers.
cagenut 11y ago
being held responsible (on-call) for how it works in production
codingdave 11y ago
About 2 paragraphs in, I was thinking, "Tell me this is not a story about someone who spent weeks figuring out that his compiler had been compromised." Ah, well. The legends of self-modifying compilers are old at this point: http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomps...
- soyiuz 11y ago
  Yes, why did s/he not isolate and compile on another machine in step 1?
DblPlusUngood 11y ago
Here's another story: I was optimizing the performance of a small, in-house x86_64 OS on a specific benchmark. I spent many months finding and removing/improving bottlenecks. Once the performance was similar to competing OSes, the measurements from my performance experiments were varying by 30%! This was surprising because the benchmark was simple and the performance of competing OSes was completely stable. Well after three or four days of frustration, I discovered that the problem was our test hardware: for whatever reason, immediately after BIOS POST, any memory loads executed for the next 10 seconds were 30% slower than memory loads executed almost exactly 10 seconds after BIOS POST. Initially I was shocked, but this may make sense -- it seems plausible that hardware manufacturers would run initialization code in the background resulting in quicker boot times. Anyway, after we figured that out we simply waited 10 seconds after POST before we ran any experiments. I'm curious how common this slow-for-the-first-10-seconds thing is with other x86_64 machines. Edited for clarity.
ajarmst 11y ago
Yet another example of why all programmers should be familiar with "Reflections on Trusting Trust." (http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomps...). Clearly the malware author was.
nomercy400 11y ago
That's not a nightmare. Somebody else caused it, and you aren't to blame. It's not the code that causes the nightmare, it's the result of the code. Say, a live production system, where something went wrong due to your coding, causing a great irreversible loss. You realize it the moment it happened, and you can't do anything about it. Closest I got: data inconsistencies in a +5-year old, live core production database of a large company, caused by a coding error. You don't know where it is wrong, you don't have means to fix it (it was wrong from the start), you do know how much was lost because of it. Only 'good' thing was that it wasn't my fault.
todd8 11y ago
The OP is an interesting story. See Ken Thompson's Turning Award lecture[1] from about that time frame; it is strangely similar. [1] Reflections on Trusting Trust, Ken Thompson, Aug 1984. https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomp...

476 points by oskarth 11y ago

dalke 11y ago

That seems more like a fictionalization of "Reflections on Trusting Trust" than a true story.

brudgers 11y ago
Link to Reflections on Trusting Trust: https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomp...
- cshimmin 11y ago
  Or for those who like explanations from secondary sources: http://c2.com/cgi/wiki?TheKenThompsonHack http://c2.com/cgi/wiki?TheKenThompsonHack
auntienomen 11y ago
Or, perhaps, the grad student antagonist got the idea from the essay.
vidarh 11y ago
Either that, or said student had read Reflections on Trusting Trust. It was published in 1984. The 3B2's and Tymnet were current around then, and the authors bio indicates he's been programming long enough for the timeline to fit too. It's not that unlikely that someone, somewhere read the paper and decided to have some fun at some point. The thing that makes me tend to agree with you is that he hasn't mentioned any "finally, years later, I found out where that guy got his idea" moment in it.
- skybrian 11y ago
  He did sorta mention it, but not by name: "I suddenly realize it's in the compiler. It was the compiler. And every time you compile the original code and run it puts in the subliminal message code into the source code. I'd heard of this before."
  - vidarh 11y ago
    Ah, yes, I missed that... Thanks.
- pdpi 11y ago
  There is an "I'd heard of this before" in there that I do attribute to the paper.
joelbugarini 11y ago
Mick (OP) responds to this assumption in the Quora comments: I just read that. This guy certainly wasn't Ken Thompson. But this happened to me in 1991, seven years after the date of publication you posted, and the "login" portion did the exact same thing Ken was describing as far as the secret password. This was probably Thompson's genius idea implemented by the grad student. The mechanism was very different though in the compiler, but the outcome was still a poisoned compiler. I had heard of self-modifying compiler before, too or I probably wouldn't have thought to look there myself. I'm not surprised to see Thompson behind the original idea.
- keithpeter 11y ago
  I thought of the 'trusting trust' paper just as I reached the paragraph about the compiler. As an ex-PHB rather than a programmer, I think perhaps that Dr Phelps could have had taken a bit more interest in the project &c. Perhaps had a look at the code now and again as simply asked "why does this look so unlike other people's code?". Might have sent a shot across the bows of the bad apple.
  - Drdrdrq 11y ago
    This. I wonder what made the graduate so angry to go to such extreme lengths to protect his code. And why nobody noticed it - did they even have the right to fix the code?
- anon4 11y ago
  I don't doubt the story's plausibility, but I'm left wondering how the student could have replaced the compiler with his own. Usually you need root in order to mess around with the system like that.
jtheory 11y ago
I assumed for the first half it was going to turn out to be a psych experiment (and there were several other compsci students getting similar "contracts") -- especially as the hurdles he was jumping were clearly intentionally set up.

crb002 11y ago

These days you can just run strace and see all the OS calls.

irascible 11y ago

This is pretty unbelievable.

seanwilson 11y ago
Yeah...I would have compiled the code on a different machine then wiped the original machine if it had clearly been hacked.
- mytochar 11y ago
  I kept wondering that as well. I'm wondering if this was a different time, or if it's just a fun story
  - mwill 11y ago
    3B2, so probably like early-mid-80's. Likely didn't have many options.
- vidarh 11y ago
  This was a university network in the 80's most likely based on machine model. Assuming the story is true, and not as mentioned elsewhere, a fictionalization of Reflections on Trusting Trust, the person in question most likely didn't have a standalone system that he could wipe, and didn't have copies of the OS media to install. Note the details of an AT&T tech coming in to fix the compiler install. That's quite believable for a university installation like in the story. When I went to university in '94, if we'd had to deal with the same thing I'd have been in largely the same situation: Sun and SGI workstations that largely booted via NFS (some of the SGI's were exceptions, I think, but most parts of the system were mounted via NFS there too) where anyone trying to re-install the compiler etc. without the approval of the computer department would be in big trouble (and have to hack the system; though the protections were relatively weak). And while we certainly could install compilers etc. locally in our accounts, getting hold of a cleanly compiled compiler for some of the systems we had would not have been all that trivial, even though the systems we had were all more mainstream than the 3B2 mentioned in the article. I wouldn't be surprised about an outside contractor having a hard time getting the computing department to believe that this problem was a hacked system. I'd expect the reaction would be to assume that he was just trying to make excuses for why it was taking him so long to find the actual problem.

SilasX 11y ago

This is funny, but it's actually more like a programmer's second worst nightmare -- "It doesn't work but it should". The true worst nightmare is, "it works, but it shouldn't", because that means your whole model of the domain was wrong, rather than some isolated, fixable component.

Natsu 11y ago
I remember having one of those moments with a function I hadn't implemented yet and being incredibly confused as to how it could possibly work in spite of there being no code yet.
- moyix 11y ago
  Yes! I once started testing a class I had written, only to discover methods I hadn't finished writing yet were working perfectly! It turns out I was accidentally reimplementing a library function, and I had chosen the exact same names and method signatures for the (admittedly small and fairly obvious) class.
  - jacquesm 11y ago
    Hehe, I had more or less the opposite of this once. I spent way too much time tracing a function in ever more detail going on the assumption that it must have been the source of the problem. Only... it was never called in the first place, it was a library module that took the call and it was misbehaving...
    - Natsu 11y ago
      Just for the record, my test was a bit off and wasn't hitting the non-existent code at all.
nkozyra 11y ago
It's not a nightmare, but it's incomparably frustrating. I'm more motivated to figure out things that work when they shouldn't then the other way around.

0xCMP 11y ago

You know, I know everything posted on the internet is fact /s... but I really hope this story is true cause it's just crazy. I mean, who want to think that the main thing that would let you create code is the thing that is infected? And it seems like there weren't any other options for languages (which hopefully weren't compromised as well) so there would be no really getting around it.

vezzy-fnord 11y ago
I mean, who want to think that the main thing that would let you create code is the thing that is infected? I'd wager that people who work with esoteric embedded toolchains develop an instinctive distrust in that layer.

lunulata 11y ago

For the love of Zeus... it's called git, use it!

davb 11y ago
The 3B2 computer mentioned in the story was around in the mid-1980s. Git's initial release was in 2005.
- iLoch 11y ago
  Not to mention the malicious code was in the compiler, not the source code.
vidarh 11y ago
After you invent time travel. At the time of this story, based on the details (3B2 computers and Tymnet) RCS was likely still current and CVS likely didn't exist. There were other alternatives like SCCS, but speaking as someone who wrote code back then: It wasn't until the late 90's it started being weird to not see source control systems used even in a corporate setting. The last time I worked in settings were source control was not universally applied was as late as '99.
- plorkyeran 11y ago
  As a point of reference, the Joel Test was written in 2000 and even then "Do you use source control" was something you had to actually ask.
  - ddebernardy 11y ago
    You'd be surprised. One of the questions I systematically ask when interviewing developers is whether they're experienced using version control systems. Not all are. The same goes for automated testing, following coding standards, or (of all things) using a bug tracker.

matobago 11y ago

Once that you start using version control systems, this shouldn't be a problem. A real nightmare for me was when I inherit a bunch of code full of uncommented regular expressions

greggyb 11y ago
How does a version control system address a malicious compiler? Unless you're suggesting the entire system be under VCS?

martin1975 11y ago

Reverse scheduling. That's the worst one by far.

userbinator 11y ago

That's not a worst nightmare for someone who knows how to read assembly language and use a debugger. I would've narrowed that down to the compiler the first time and probably debugged into it too (providing whoever did this very clever hack didn't also do something to the debugger... but if the system is behaving this oddly, it's certainly better to bring one's own tools.) Seeing the CPU do something it shouldn't, being traced by a hardware ICE, now that is a worst nightmare - and one I've actually experienced.

jacquesm 11y ago
That sounds like a story worth telling.
mcherm 11y ago
Story time? (Please!)
- userbinator 11y ago
  It was a long time ago, but I remember we were working on an embedded system controlling some industrial equipment, and it randomly crashed; the time between crashes was long enough that it'd take several days before it happened, so even getting a trace of the crash was an exercise in patience. Eventually we did get a trace, and it turned out the CPU would suddenly start fetching instructions and executing from a completely unexpected address, despite no interrupts or other things that might cause it. We collected several more traces (took around a month, because of its rarity) and the addresses at which it occurred, and the address it jumped to, were different every time. Replacing the CPU with a new one didn't fix it, and looking at the bus signals with an oscilloscope showed nothing unusual - everything was within spec. We asked the manufacturer and they were just as mystified and said it couldn't happen, so we resorted to implementing a workaround that involved resetting the system daily. Around a year after that, the CPU manufacturer released a new revision, and one of the errata it fixed was something like "certain sequences of instructions may rarely result in sudden arbitrary control transfer" - so we replaced the CPU with the new revision, and the problem disappeared. We never did find out what exactly was wrong with the first revision, other than the fact that it was silicon bug.

jimmeyotoole 11y ago

That's so messed up. I'm so glad I started my development career in an age where one could spin up a VM in minutes to isolate an issue like this.

forgottenacc56 11y ago

Programming.

rdtsc 11y ago

Other coders' nightmares: * Security : customers acccounts leaked, identities stolen, bank account depleted * Undetected data corruption : customers' data is corrupted and it has seeped into the backup stream, even the off-site one, and that is corrupted as well now. * Rarely occurring simingly random concurrency related bugs that manifest themselves once a month or longer. * Time : making assumptions about it, dealing with it distributed systems etc. Nobody thinks about it until it stops working or systems' time gets snapped back somehow.

jacquesm 11y ago
Anything involving interrupts and timing. Especially when the timing is so critical that it is time to break out the logic analyzer. Bugs like that can keep you busy for weeks.
bm1362 11y ago
Oh man, I used to work on a large service that had to aggregate data from a bunch of downstream services and submit it upstream. At the end of the month, billing would run off the data and send out invoices -- tracking down subtle data corruption was nightmarish and cost me my youth.
Splines 11y ago
* Works on my machine: You can't repro a customer bug in-house. Something about your internal setup is making a difference. Even worse: You work for a big-corp that is a platform for said customers, and so god knows what another team has running in your environment in the name of "dogfood all the things"
- jschwartzi 11y ago
  I saw one once where the cell carrier actually made the difference. As in I had to fly to a foreign country before I was able to reproduce the bug.
- lotharbot 11y ago
  Years ago, my wife had a bug that only happened in a non-debug build and only on one specific system. It was a recursive tree traversal algorithm, written in C, with an OB1 error in some pointer arithmetic. If memory was allocated in a particular order, the function would read off the end of level X's data and find level X+1's data, which was structured exactly as expected, so it would end up processing certain nodes twice. In the debug build there was a debug variable being initialized to null, which was in between the two blocks of data and therefore got interpreted as a terminator.

TerryADavis 11y ago

I made a dialect of C called HolyC. I made a compiler. There is nothing obfuscated. In planet of the apes, original, a white man told a monkey he made a flying machine. The monkey did not believe him.

av9 11y ago

Not having a job is my worst nightmare. Realizing your skills are being undercut by a fresh college grad who can get enough pats on the back while skidding along with code that works. Realizing the language you cradled has now become outdated and the other engineers that have found out the new next 9 years of hotness are too good for you to compete with. Having to interview for a new job and realizing that you can't pass the white board questions because you have fallen behind with the expectations of young blood. That's my nightmare, becoming irrelevant.

Splines 11y ago
> Having to interview for a new job and realizing that you can't pass the white board questions because you have fallen behind with the expectations of young blood. That's my nightmare, becoming irrelevant. I'd like to think that interviewing for positions that aren't entirely focused on new-hotness still leans on heavily trod areas of CS. Data structures, sorting and searching.
- nikatwork 11y ago
  A large portion of modern dev jobs are either CRUD apps or data munging. Knowledge of algos are not necessary for those for those domains. It always amuses me when I'm interviewing for some vanilla web app job and they start pitching maths questions. It's filtering on the wrong criteria.
  - gaius 11y ago
    No, they are filtering on the criteria of "recently revised this stuff for an exam" i.e. "we want cheap and easy to exploit 23-year-olds but we can't say that in front of HR".
vidarh 11y ago
I've found the solution (for me at least), while I love spending time on new tech, is focusing on being a "solution provider" rather than focusing on specific technology. The moment you stop including the name of specific tech on your resume and start describing how you helped solve X problem that saved/made the company Y amount of money instead, you open the door to positions that may be as hands on (or not, depending on what you want), but that pay better and where the interviews are not focused on tech trivia at all.

joshu 11y ago

this seems like one of those stories that got slightly more intricate every time it was told. and it's been told a lot.

logicallee 11y ago

Here's a coder's worst nightmare: "I managed to get reflection working* in C++, by registering class functions in static variables. This lets you configure routes like in Ruby on Rails. Since C++ is somuch faster, can you port the rest of Rails over as well? You'll just be doing this for the next 4-5 months." * https://news.ycombinator.com/item?id=10607029 https://news.ycombinator.com/item?id=10607029

deleted 11y ago

[deleted]

devonkim 11y ago

Aside from the issues where all your assumptions of your tool chains, OS, and computational reality is melting around you this seems a lot related to lack of feeling of control and infomation. You have to keep calling people from vendors and other departments because your talent, knowledge, and skills hit a brick wall in the face of bureaucracy / compartmentalization. It is that you don't know what to trust anymore, you have to be resourceful with time, and you have limited access to the information / control needed. This is how I explain why my technical job is difficult in a typical enterprise organization and why you need to be far better than average to produce merely mediocre results.

macspoofing 11y ago

Make a technology, platform, architecture or framework choice. After a significant amount of effort realize you made the wrong choice.

collyw 11y ago
Worse when you make the right choice but everyone else flocks to the crappy choice.

DrScump 11y ago

Management that doesn't care about regressions then later also sticks you with placating the customers who get screwed by those regressions.

NegativeK 11y ago
Management that does care about regressions and you both still get stuck with placating the customer, because the customer always wanted features instead of dealing with debt.

jbombadil 11y ago

Users with ideas

shabbaa 11y ago

Coder's worst nightmare? When you are the last line before something gets pushed into production, and the fear that it will fail IN production.. have i missed something?

hashkb 11y ago

I was hired by a psychologist My money was on the psychologist the whole time. I still kind of think it was Dr. Phelps and maybe Mark the admin and the AT&T tech are grad students in disguise. So I guess my worst nightmare is finding myself in a similar situation and later finding out my boss set the whole thing up for funzies.

balls187 11y ago
Having been playing Fallout 4, I couldn't help but think this was some wacked-out psychological experiment.
emcrazyone 11y ago
I was hired by a bookie ;) I have a somewhat related story. Circa 1995 or so, I was working on software compiled on a 486DX computer. Back then the computers had L2 cache in SDRAM chips you plugged into the motherboard. I was moonlighting with a few guys to develop software and each night myself and two friends met up in office we were renting. There, on one of those fold out tables, we toiled away each night writing code. The code we worked on was for off shore gambling (aka a bookie). Back in the 90s you could walk into a convenient store and pick up an magazine; usually autotrader magazine. On the back was an 800 number you could call to place bets on football, hokey, or baseball. (Called HOBs bets in the industry). Anyway, because of the nature of the software and the people we dealt with, things were a bit hairy at times. The bookie we worked for paid us generously but, at the same time, expected perfection (i.e. software that just worked). One evening, as usual, the three of us met to work toward the next release of the software. The next release of the software was to include a feature for boxing bets. Mike Tyson was about to be release from jail and the bookie was anticipating various future business in taking boxing bets. So this one evening we were doing a software build to release as beta software. I did the software build and ran a battery of tests we normally run and all worked fine. To transfer the software to the offshore network we used a dial up connection. I pushed the software out which my bookie contact would run and make sure all the logic was correct. A few days go by and we get a phone call that the software is not functioning. Sometimes it crashes (which was rare) or sometimes just strange things would happen such as strange blinking characters on the screen. I collected enough information to try and replicate the steps necessary. You can imagine my customer wasn't too happy. So the three of us basically stopped working on the code to track down the bug. We spent a week looking for the bug we were sure was in the code somewhere. Back then we didn't really use a code repository but we did have a diff of the beta release and previous release. Pouring over the code changes we just could not find the problem. It even got to the point where I would recompile the beta version, run it, and could actually get the code to crash. Comparing to earlier release it would not crash. It was by chance that my fellow coder, sitting right next to me, started running similar tests on his machine while I was off making a food run. It was sort of normal for us to group up around my workstation when something needed a collective look. Long story short it turned out that one of the L2 cache chips was causing my compiles to become corrupt in just the right way to cause the code to be not as we typed it in. Anyway, thought I'd share. Although not a malicious act, it was one of the worst things to track down. Fast forward to my current life... I work in the embedded field and recently solved a bit-flip problem in one of the products my company produces that relies on NAND memory. That little incident from my past certainly has served me.... ;)

gesman 11y ago

Bottom line: insists on hourly rate.

fishanz 11y ago
I'm in this camp, but I keep hearing stories about the pot 'o gold. "Charge what it's worth to the client." This happens?!

musesum 11y ago

"Best Fit" memory management. Malloc three buffers, in order: 64K, 1K, 64K. Now free all three. Next allocate 1K. Guess where it resides in the heap? Yep: the middle! After a while, memory was fragmented with 1K detritus. Even though you were supposed to have 500K available, there wasn't room for a 64K buffer anymore. This was the default for Microsoft's C compiler, in 1990. How did I find out? There was a project that was crashing into a blue screen of death after a few hours. The crashing library had #defines that went 15 levels deep. The stack dump wasn't very instructive. Even though the library ISV was based in Boston, I had to fly out to Santa Cruz to talk to the developer directly. He was somewhat belligerent: "you should have figured it out yourself!". 15 levels of #defines. He was the perfect example of the other side of Dunning-Kruger thinking his arcane shit was obvious. But, that didn't solve the problem. His fix didn't. So, I flew back to Boston. I started to trace everything with -ahem- #defines of the main APIs. It was turning into a Zen Koan: the sound of one app crashing. Everything worked with the Borland compiler. It was just the MSC compiler that failed. The company was one of the top 3 ISVs back then. Meetings about what to do would last for hours. There were technical politics: "Can we use Borland?" No; the company had standardized on MS. I could talk to their engineers directly, though. But, they didn't have a clue. So, I read though the docs to see what compiler switches were available. And there it was: best-fit. Fixed! So, I wrote an analysis about why best-fit was absolutely the worst possible memory management and sent it to Microsoft. It took 8 years for MS to change the default. So, the reason why software would blow up? Best fit. Why early versions of Windows with go into blue-screen-of-death? Best fit. How many developers had to deal with this? Worst nightmare in aggregate.

davidst 11y ago
You just brought back old memories for me. I wrote the heap manager for Borland C. I chose the next-fit algorithm for its balance of speed and reasonable fragmentation under common use cases. Best-fit performs exactly as you described.
- musesum 11y ago
  Man, oh man, BC was the bomb! In a good way; maybe I should exclaim that it was the anti-bomb!

deleted 11y ago

[deleted]

meshko 11y ago

Unicode and timezones.

Jupe 11y ago

COM. 'nuff said.

melted 11y ago

Leading a project where in order to just not fall behind on maintenance and user support you need 2 people, and 2 people is all you're gonna get for the foreseeable future, yet management demands forward progress out of you. Fucking demoralizing and degrading.

vox_mollis 11y ago
Nightmare? This is the status quo in the majority of small-but-not-tiny organizations.
- melted 11y ago
  Doesn't make it any less of a nightmare. I wonder what percentage of effort is wasted in the industry this way.

OSButler 11y ago

As a contractor, the fear is more on the client side than with the code. Coding problems can be resolved or worked around eventually, but the client can be a wildcard. In developer meet-ups I usually hear about coding issues like they are a puzzle, whose solution is being shared with the group. But the scary stories are usually centered around clients, since problems can arise there even if you provide perfect code.

ju-st 11y ago
When I read the topic my first thought was "customers!"

shams93 11y ago

Being so sleep deprived on a new gig that you mistake myvar: for :myvar

bdotdub 11y ago

Time zones

cpeterso 11y ago

My "favorite" bug was debugging a crash in Macromedia's Flash Player for Windows Pocket PC. It was an ActiveX control running in the Pocket IE browser. The crash was intermittent and only happened after running a 15-minute automated test suite on the Pocket PC, crashing deep in Pocket IE code. After staring at code print outs for a couple weeks, I found the bug: a BSTR returned from a COM interface was freed using CoTaskMemFree() instead of SysFreeString(). ARGH! A one-line fix.

frik 11y ago

Software that crashes while presenting it to the CEO or on investors day.

simula67 11y ago
https://www.youtube.com/watch?v=IW7Rqwwth84 https://www.youtube.com/watch?v=IW7Rqwwth84
- i336_ 11y ago
  How did I just know what video this was before I clicked it..

joenathan 11y ago

I'm not a programmer but a Sysadmin. I ran into an issue a few months ago that nearly drove me crazy. I was custom building a new server for a customer, very nice build, Xeon E5 10 Core, 64GB DDR ECC RAM, Windows Server 2012 R2, SSDs, etc... Everything is going well, update and configure bios, install Windows, install drivers and software. Then start configuring the server, sever needs to reboot, reboots to a blue screen of death. Can't get it to boot up normally. Ok must be bad software, driver, etc... Time for a clean install, everything is going fine and then reboot -> blue screen. Look up the bug check code, no help. Another clean install, this time no extra software, no drivers, same problem after a reboot. Finally figure out that it is only happening after I make the computer a domain controller. After Googling with this new information, I find one thread on some forum where people are having the same problem, turns out it was the solid state drive, if you use a Samsung 850 Evo SSD with Windows Server 2012, it will blue screen if you make it a domain controller. I never thought a problem like this was possible. Sure enough changed the installation drive and no more problems. Nearly drove me crazy, took me two days of troubleshooting.

vl 11y ago
Haha, these multi-hundred-thousand-line-source-code SSD firmwares need to do something. In particular they detect popular filesystems and try to optimize some operations (specifically if filesystem is recoverable by check disk after crash, it's considered to be acceptable optimization; i.e. it's possible to delay actual commit of free block info for example). Seems like detection glitched on the blocks that contained data related to being domain controller.
- digi_owl 11y ago
  Thinking about SSD and HDD firmware can make a guy reach for tape drives as a last ditch hope that data stays just data. At least with those (and to some degree optical media, but the write process there is more laborious) the storage and the RW logic is different pieces.
leonroy 11y ago
That is insane. And also one of the reasons I try to avoid being a system integrator whenever possible. It just takes up too much time - so much simpler having my friendly Supermicro distributor build my server for me!

hacknat 11y ago

My team and I are in the middle of a bad one right now. Our site is allocating incredible amounts of memory per page request, and it took us forever to figure out we didn't have a memory leak, at least in the traditional sense. We're running .NET, and its GC scheduler has 3 tiers for objects 0,1, and 2. 0 and 1 are short lived objects, but after an object has been around long enough the GC assumes it will probably be around forever and the 2nd tier garbage collector gets almost no time to run. We proved this by forcing the GC-2 to run longer and it brings the memory under control, so we don't have a "leak", per se, but we still can't figure out why these ephemeral objects are sticking around long enough to get so promoted.

grandinj 11y ago
Increase the size of your Eden space. You probably have something allocating very large amounts of temp objects, which is exhausting Eden. You should optimism that, whatever it is, but bumping up Eden will give you some breathing room.
i336_ 11y ago
Would it be too insane of an idea to go do a local full rebuild of .NET, now that it's open source? If Microsoft can get "[Build|Passing]", surely you can too... :P Build .NET, prove the issue still exists with the local build, then add new instrumentation to the GC routines inside .NET's runtime. Instrumentation never killed anybody... and if this issue persists too much longer... I can't deny that it's a crazy idea though.

deleted 11y ago

[deleted]

datashovel 11y ago

"You don't have to pay me anymore, Dr. Phelps, I just want lab time." This is nerd war. LOL

datashovel 11y ago
This also came to mind while reading: https://pbs.twimg.com/media/Bp_KeZnCAAAijRe.jpg https://pbs.twimg.com/media/Bp_KeZnCAAAijRe.jpg

thathoo 11y ago

Apart from the obvious concurrency based or non-easily reproducible issues, some of my personal nightmare scenarios: 1. Uncommented code, but especially uncommented regular expressions 2. Unnecessary abstraction that runs too deep & so is hard to comprehend 3. Also, take any tractable problem and add timezones to it. Now you have an intractable problem!

facepalm 11y ago

I've heard another anecdote from university where somebody changed some late digit of PI in the compiler. Very hard to detect...

teddyh 11y ago

http://www.commitstrip.com/en/2014/10/31/a-coder-nightmare/ http://www.commitstrip.com/en/2014/10/31/a-coder-nightmare/

Drdrdrq 11y ago

Internet explorer. A custom CMS. And sometimes some of the posted data is missing... On a single POS. And because data ath end of form is optional, system can't even figure out that something went wrong, it just silently discards it. Never figured out what problems that particular IE had, but the workaround was simple... I added an extra hidden input field just to check if all was sent. The theory was that at least operator could repost the form if it went wrong - however, it never did again. Thank you IE for your obscure bugs and nonstandard behaviour! Rip. </rant>

dijit 11y ago

forgive me for saying, I'm not a coder, I'm a sysadmin. I'm scared of having developers with a little ops knowledge take over my job. /devops/

ownagefool 11y ago
You probably should be afraid. Those developers are going to replace your tools with better tools, that they'll know and you will no longer have years of arcane experience to fall back on. Developers spend a lot more time thinking about abstractions, where to place things and creating & controlling nThings. If you cannot do these things well, you need to learn them. The best way to do that will be to ride the bandwagon, work together with devs and learn from each other.

CM30 11y ago

That's a pretty messed up story, though I'm not sure it'd be a coder's worst nightmare. I'd think that'd be if some major project you were worked on got hacked... while you were enjoying the weekend off and had no access to your work computer. Or just if your project developed a show stopping bug at that point in general, since you'd get the feeling that 'oh god, we've been done by two days' or 'we've been serving malware for two days'. Or maybe if you'd found a breaking bug that needed fixed, yet no answers were available online, no one in the room could help and the site was due to go live that night. All that time worrying, refreshing your Stackoverflow question and trying in vain to find a solution would be torture.

pjc50 11y ago
The Knight Capital bug is probably most people's worst nightmare: http://pythonsweetness.tumblr.com/post/64740079543/how-to-lose-172222-a-second-for-45-minutes http://pythonsweetness.tumblr.com/post/64740079543/how-to-lo...

digi_owl 11y ago

Stories like these makes one wonder if "mere mortals" should just turn things off, scrap them in a industrial chipper, and go back to pen and paper.

mbrock 11y ago

I worked on software that encrypted entries with a custom C++ plugin to SQL Server. Nobody knew how it worked, it was implemented long ago and considered magic. Some entries on some installations sometimes would become totally garbled, which was a seriously dangerous issue. This was a pure Java shop and I was almost the only engineer who even knew C++ since the reorganization that happened before I joined. Turns out the affected installations all used custom compiled DLLs with hard coded specific crypto keys made up by someone long ago. We didn't really know how to deploy new versions of this. But I found some version of the code at least, on some old file server, so I could sort of debug it. It took me a while to figure out but after working on it on and off for a few weeks I found a crucial race condition in the code path that looked up the crypto key. Turns out if you were unlucky after a SQL Server reboot, a few entries could get encrypted with a "default" key. So I was able to decrypt the garbled entries on affected customers at least. We decided to change the whole method of encryption since it was obviously hacked together by some cowboy coder a decade ago. But in the meantime, the easiest way to fix the issue for the affected customers was a method I taught to one of the tech support guys. I told him to RDP into the affected server, open the crypto DLL in Notepad++ and do a string substitution on a particular ASCII sequence that would change a certain character into a tilde. And then reload the DLL with a SQL Server command. Because through disassembling the binary I found that this would sufficiently increase a certain value and fix the bogus comparison that caused the race condition. The nightmare here was mostly because of custom DLLs running semi-unknown code on remote servers.

cagenut 11y ago

being held responsible (on-call) for how it works in production

codingdave 11y ago

About 2 paragraphs in, I was thinking, "Tell me this is not a story about someone who spent weeks figuring out that his compiler had been compromised." Ah, well. The legends of self-modifying compilers are old at this point: http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomps...

soyiuz 11y ago
Yes, why did s/he not isolate and compile on another machine in step 1?

DblPlusUngood 11y ago

Here's another story: I was optimizing the performance of a small, in-house x86_64 OS on a specific benchmark. I spent many months finding and removing/improving bottlenecks. Once the performance was similar to competing OSes, the measurements from my performance experiments were varying by 30%! This was surprising because the benchmark was simple and the performance of competing OSes was completely stable. Well after three or four days of frustration, I discovered that the problem was our test hardware: for whatever reason, immediately after BIOS POST, any memory loads executed for the next 10 seconds were 30% slower than memory loads executed almost exactly 10 seconds after BIOS POST. Initially I was shocked, but this may make sense -- it seems plausible that hardware manufacturers would run initialization code in the background resulting in quicker boot times. Anyway, after we figured that out we simply waited 10 seconds after POST before we ran any experiments. I'm curious how common this slow-for-the-first-10-seconds thing is with other x86_64 machines. Edited for clarity.

ajarmst 11y ago

Yet another example of why all programmers should be familiar with "Reflections on Trusting Trust." (http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf http://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomps...). Clearly the malware author was.

nomercy400 11y ago

That's not a nightmare. Somebody else caused it, and you aren't to blame. It's not the code that causes the nightmare, it's the result of the code. Say, a live production system, where something went wrong due to your coding, causing a great irreversible loss. You realize it the moment it happened, and you can't do anything about it. Closest I got: data inconsistencies in a +5-year old, live core production database of a large company, caused by a coding error. You don't know where it is wrong, you don't have means to fix it (it was wrong from the start), you do know how much was lost because of it. Only 'good' thing was that it wasn't my fault.

todd8 11y ago

The OP is an interesting story. See Ken Thompson's Turning Award lecture[1] from about that time frame; it is strangely similar. [1] Reflections on Trusting Trust, Ken Thompson, Aug 1984. https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thompson.pdf https://www.ece.cmu.edu/~ganger/712.fall02/papers/p761-thomp...