20 ms·

The Servers Are Burning

229 points by tortilla 8y ago

JackFr 8y ago
Wow. 1. We don't test. 2. We don't code review (or rather if we do, we do it so poorly swallowed exceptions don't raise red flags.) That's an outrageously unprofessional software process.
- danesparza 8y ago
  I was immediately struck by the same set of thoughts. I don't know whether to be appalled or just ... sad.
- andrenotgiant 8y ago
  An outrageously unprofessional software process that created a product worth $50M! https://techcrunch.com/2011/02/02/match-com-acquires-online-dating-site-okcupid-for-50-million-in-cash/ https://techcrunch.com/2011/02/02/match-com-acquires-online-...
  - danesparza 8y ago
    The OP is about technical process in the software world. I'm not sure how a company's valuation comes into the conversation...
    - blacksmith_tb 8y ago
      I assume the implication is "if OKC can fly by the seat of their pants and still make millions, anyone could!" Which is true, but still not advisable...
      - stcredzero 8y ago
        Given that technical debt is quite hard for non-technical people to quantify, why wouldn't we expect that most startups would tend to accrue technical debt?
        theseatoms 8y ago
        Can anyone quantify technical debt? Best case is something resembling a medical diagnosis.
        stcredzero 8y ago
        I think that could be enough. Actuaries have a good idea about when you are likely to die in the next two years. Basically, when you are old and your medical costs suddenly double. Doctors use the same sort of inductive reasoning for their prognosis. The industry probably collectively has all the information to put together useful actuarial tables on technical debt, but it's all silo'd in individual companies.
        theseatoms 8y ago
        Interesting. Are there IT consulting firms that specialize in this field of study?
        adrianN 8y ago
        Startups accrue so much financial debt, adding some technical debt on top doesn't really make a difference.
        stcredzero 8y ago
        Technical debt doesn't accrue in a linear fashion. Instead, it's synergistic. Up to a certain point, it's a noticeable drag. Past that point, there is a change in the difficulty incurred. In the article, the combination of some "you're-supposed-to-just-not-do-that" with: >If (the database throws an error) { do nothing } resulted in a major mishap.
      - T2_t2 8y ago
        I think the implication is that they achieved the goal they had, to grow a company. Note that goal doesn't mention code at all. It depends what you think software engineering is - a means to an end, e.g. to grow a company, or as a craft you want to perfect and do for your whole career. I think there are more craft coders, because most jobs are at companies that have survived and there is more to lose from errors than gain from moving fast. The craft attitude may not be ideal when you want to go from zero to something, and avoid going back to zero very quickly. Or, perhaps more importantly, if you want to go from zero -> something wrong -> something wrong again -> something almost correct -> Product Market fit.
    - jschwartzi 8y ago
      The point is that engineering decisions need to involve both the business and the technical side of things, and they need to be pragmatic and not tied to any particular dogma from either world.
  - mabbo 8y ago
    That's not skill, that's luck. If the bug in question had instead leaked every customer's personal data there wouldn't be a $50m company, there'd be a multi-million dollar lawsuit.
    - TeMPOraL 8y ago
      I agree with your thinking, but with this particular example: you'd wish. See: every other data leak that happened over the last decade. It seems data leaks are harmless to companies.
      - laurent123456 8y ago
        I was going to say "not for a dating website", but I see that Ashley Madison still exists despite the massive data leak in 2015 including data from users who had paid to have their data deleted, so yes looks like there's no real fallback for data leaks.
  - JackFr 8y ago
    Created a product worth $50M because of their software process or despite there software process.
- merlincorey 8y ago
  I washed out of their interview process when I was fresh out of high school for not knowing bit twiddling hacks[1] in the technical phone screen with one of their engineers. After reading this, I can't help but feel like I dodged a bullet there. [1] https://graphics.stanford.edu/~seander/bithacks.html https://graphics.stanford.edu/~seander/bithacks.html
  - gwbas1c 8y ago
    Don't feel bad: I cut the phone screen short about 25 minutes into it. I just got a really bad vibe about all of the front-end complexity they were building into a dating site.
- swalsh 8y ago
  It's a free dating service. They have a responsibility to respect their users' data privacy, but the rest kind of doesn't matter. Eventual stability is more profitable than eventual release if you're not a Bank or making self driving cars.
  - skybrian 8y ago
    Yeah, I expect the devs at Ashley Madison told themselves that too. It's possible to build sandboxes where security or privacy bugs are contained, but the shops that don't care much about software quality are probably not doing that. It seems a little too easy for people building a social app to tell themselves they're just building a toy that doesn't matter. Their users might disagree after they get hacked.
- KingMob 8y ago
  I'm sure the OkCupid folk are nice people but it was: 1) founded by mathematicians and an academic, and 2) written in C++, with a custom templating language Even giving them the benefit of the doubt that the core matching algorithm warranted C++, the whole site really shouldn't have been built that way. I'm really not surprised there's poor technical decision-making there.
- ddebernardy 8y ago
  Outrageously unprofessional as it may be, they probably ended up laughing on their way to the bank when they got purchased by match.com.
- toast0 8y ago
  > That's an outrageously unprofessional software process. People who call me unprofessional forget that being a professional just means I'm paid to do it, it's not a pass time.
- megaman8 8y ago
  You have to judge it in context. If it's entirely a senior team then the value of code reviews is greatly diminished. You rarely catch a bug in a bug review when you're reviewing a Senior software engineer's code. those bugs aren't caught until you do testing.
  - nradov 8y ago
    I've seen plenty of defects in senior developers' code that were caught in reviews. Please the purpose of code reviews isn't just to catch defects, it's also to disseminate knowledge of that module across the team in order to reduce the "bus factor".
- chasd00 8y ago
  it worked well enough until they could do something better. Never let perfect be the enemy of good.
- tluyben2 8y ago
  In my experience this ‘outrageously unprofessional‘ process is very normal. Most companies are not software companies and they are not going to pay for whatever is not adding to the bottomline. And even in software companies that ofcourse happens. It is unfortunate but the shock and awe are a bit over the top; when I talk with partners (ctos is large corps; banks, insurance, retail), they generally glaze over and mumble about their dreams of doing tests and code reviews but unfortunately it is not possible ‘at the moment’.
mikestew 8y ago
(then-CEO Mike Maxim) thought of testing frameworks as somewhat academic, more lofty than practical. ... Mike the CEO, who was also OkCupid’s best engineer... Mystery of the Melting Servers, solved.
- plopz 8y ago
  Do testing frameworks normally catch memory leaks?
  - mikestew 8y ago
    Mine do.
    - fhood 8y ago
      You can't possibly say that with any confidence. I literally just finished reproducing a memory leak caused by an insane combination of circumstances. This was not something valgrind or any similar tool would find. It was not something that any rational human being would have thought to write tests for. Once code is no longer synchronous, and tasks are being juggled around and swapped between, it becomes virtually impossible to predict what might cause this sort of bug.
      - whatshisface 8y ago
        The parent is saying that their tests normally catch memory leaks, not that their tests catch all memory leaks.
      - tekstar 8y ago
        What's an example of a memory leak that can't be found by valgrind?
        viraptor 8y ago
        One happening in an embedded language with its own GC. Unless you can switch all allocations to explicit management for the tests, it's almost impossible to differentiate between live objects, free lists, cycles with deferred collection, and arenas that are supposed to be freed on exit.
1_800_UNICORN 8y ago
If you're an early-stage startup that's still determining product-market fit, then it's completely fair to eschew tests and accumulate tech debt; you're at a higher risk of running out of runway before you've even proven that your business works. As soon as you have enough users that an outage poses a significant risk to your business, you need to invest in either refactoring to reduce tech debt, or a rewrite. And you NEED to add tests, and a robust deployment process that tests changes at multiple stages with monitoring on the basics (server speed and memory usage, database error rate, etc). OKCupid got lucky in this case.
- nradov 8y ago
  Skipping tests generally doesn't allow for creating a working MVP any faster. It's just an illusion.
  - always_good 8y ago
    Only if you are referring to perfect tests which cost zero time to write/maintain and only the tests written that eventually will catch an issue. Otherwise, tests have a cost just like any other code. You can see this by looking at both extremes: perfect tests (described above) and useless tests. For example, tests that make your codebase too brittle, tests that don't actually test anything useful, spending too much time writing tests, spending a lot of time writing tests for precisely the code with high disposability, tests that are too coupled with the code, stupid tests that should be removed but won't be because tests tend to be append-only, etc. Testing is an advanced topic where every +1 unit you spend on testing doesn't mean your application is now +1 unit more robust. +1 unit of time spent in tests can even mean your application is -2 units worse because testing is a trade-off. So tests are more than capable of hamstringing your unlaunched MVP. They're also capable of being the reason you launched your MVP sooner than later. But the costs of testing are no illusion.
    - nradov 8y ago
      Tests save you time. They have a negative cost.
TheCapn 8y ago
(Off Topic -- Rant regarding Webpage) Can I take a short moment to bitch about "sidebar" frame or whatever the hell this domain is using? The webpage looks like a fluid single frame but the left side is static. The browser page has a scrollbar but I need to have my mouse over the correct div before my scroll wheel will move the page, its a bit aggravating since there's no visual break between the "sidebar" and the content.
- pwg 8y ago
  I just turned off CSS in Firefox (View menu -> Page Style -> No Style) and the fixed sidebar was no more and the odd scroll bar behavior was gone.
  - pritambaral 8y ago
    I used Firefox's Reader Mode.
- deleted 8y ago
  [deleted]
noelwelsh 8y ago
How could such a tiny change have such an outsized impact on the site? “That same story happened so many different times,” my old boss David told me. “Someone launched a small, relatively innocuous change that did one of the millions of unexpected things it could have done, which then happened to break some part of the site, and then bring it all down—sometimes bring it down to the point where we couldn’t recover it for hours.” Preventing this is one of the basic motivations of functional programming. If a function is "referentially transparent" or "pure" it can't do anything unexpected. It seems to me this is the only reasonable way we can reason about software at scale.
- jolmg 8y ago
  While I generally agree that functional programming does help immensely to make secure software, in this case, the function wasn't pure. It was meant to return data that didn't exist from a database, but the error was silenced. Also, the reason why the servers were "burning" was because this somehow resulted in a memory leak that consumed all the memory available to the system, which is why they also had trouble pushing fixes. Even in a pure function in Haskell, memory leaks are possible and the same scenario could've resulted.
- mreome 8y ago
  While I can understand some of the benefits of functional programming within a single sub-system, or within academic research, I struggle to understand how pure functional programming can address the need for the large amounts of state information and abstraction layers required for a large scale system. I work with systems where (large/numerous) sub components require very specialized domain knowledge to understand internally, so multiple levels of black-box abstraction are required to integrate/interact with the sub components. Additionally, both the client systems and the sub components may have huge amounts of persistent state information. Could you suggest any resources that might help me understand how this can be accomplished in a pure functional paradigm? Something that assumes only a casual understanding of functional programming would be especially helpful.
  - TheDong 8y ago
stcredzero 8y ago
Yet what I found even more troubling was that in order to write effective tests, a programmer had to know all of the ways that a piece of software could fail in order to write tests for those cases. So the programmer needs to think about all the ways the code can fail. Writing the tests concretely documents that thinking in the source code repository and automates re-checking that thinking. A really good programmer then tries to organize the code, such that it's easy to think about how any given unit of code can fail. Writing unit tests rewards such organization and penalizes bad organization. That said, TDD doesn't seem to work out that way in practice, most of the time.
hnruss 8y ago
After reading the introduction, I thought that the author was going to use a story of software failure as an example of why you should write unit tests, or at least why you shouldn't deploy untested software. However, the moral of the story was essentially: "software is so complicated that it is bound it break, so you have to be good at fixing it". While that is certainly true, I think that developers have a responsibility to use whatever tools they can to write high-quality software.
- jschwartzi 8y ago
  > However, the moral of the story was essentially: "software is so complicated that it is bound it break, so you have to be good at fixing it". While that is certainly true, I think that developers have a responsibility to use whatever tools they can to write high-quality software. I actually disagree that we have a responsibility to write high-quality software. As engineers our job is to create software that is good enough for the task at hand and that can be improved later. Sometimes that means that you invest in 100% coverage plus integration and system-level testing because "good enough" means that it has to never fail. Sometimes that means you don't spend much time automating tests and instead do informal development testing before you check something in. It's entirely up to your judgement based on what your code is going to be used for, and I don't think it was inappropriate for OkCupid's engineering staff to avoid testing in this case. After all, their business was really successful through the period where the error happened.
  - jhayward 8y ago
    > As engineers our job is to create software that is good enough for the task at hand This is the essence of engineering. As the saying goes, "Any fool can build a bridge that won't fall down. It takes an engineer to build a bridge that just barely won't fall down." In other words an engineer's contribution is not perfection, it is being able to know the difference between "good enough" and "needs more work" and to apply that knowledge to meet time, cost, and material constraints.
    - user5994461 8y ago
matt2000 8y ago
When making something new where you're not sure of the value yet, I've found that you can get 80% of the benefits of unit tests with around 20% of the tests you'd write to get "full coverage." My main goal is to at least have the code run in an expected way and produce an expected result. This doesn't catch everything, but it does seem to catch enough problems to be worth it for the time invested. Edit: I should mention that I also add tests to cover something when I experience a failure, so it at least won't happen again.
- sneak 8y ago
  Yes to the last! Whenever bughunting my first step is to reproduce the issue in a failing test. This has the benefit of me being sure I know what is failing and how, but also crystallizes then knowledge and experience into the repo for future engineers (or future me) to know the most common/practical ways things break in the real world. It is also marginally useful for catching regressions, if the test is clever/general enough.
- ddebernardy 8y ago
  The big problem I've noticed when reading unit tests in various projects is that they frequently test the implementation rather than the outcome: https://softwareengineering.stackexchange.com/a/304910/24932 https://softwareengineering.stackexchange.com/a/304910/24932
  - macintux 8y ago
    That's one reason I really like functional programming: it's easier to test pure functions, vs code with side effects that requires you to look at the internal state afterwards.
    - tobr 8y ago
      On the other hand, pure functions are also much easier to reason about, thus much less likely to cause bugs from "a tiny change" in the first place. It's in the messy imperative stuff things tend to break in incomprehensible ways, and where you really need tests.
drblast 8y ago
The reactions to the outage on Twitter are hilarious and make me glad this happened: “@okcupid how am I supposed to get my daily dose of crushing rejection and emotional humiliation if your site is down????” “Okcupid stops working over lunch hour, NYC wonders if we're committed to the people in our phones from now on, panic in the streets” “@okcupid How can I continue to be ignored by the females of the world if they don't know where I am to ignore me?! #panic #freakout”
ahallock 8y ago
> ...in order to write effective tests, a programmer had to know all of the ways that a piece of software could fail in order to write tests for those cases That's impossible. You reason about the problem as best as you can and create tests appropriately; and then if there are edge cases/failures in production, you go back and add those to the test suite.
3pt14159 8y ago
I appreciated this article because I so infrequently get to hear from people that hold this view of testing, which I do not share. To me, the importance of tests are a function of the consequence of failure and the likelihood that failure will happen. It's a question of hazard and risk. One thing to realize is that code that lasts longer is more likely to fail because the people and libraries that support it are prone to change and assumptions can break. So risk goes up over time. The other thing to realize is that code that's part of a growing product is going to impact more people if it fails. There might be only 400 users today, but 40k three years from now. So hazard goes up over time. When I heard the technical debt arguments I try to triage them into "grows with the company" vs "doesn't grow with the company" so I can figure out what to accept as debt and what to pay. A complex deploy infrastructure doesn't scale. Just SSHing into a box and doing the deploy manually is just as fine for 100 users as it is for 1000. But tests do. So I write lots of tests; especially for ACL and I punt on the infrastructure stuff until I absolutely need to. Note that this leaves aside the entire argument over whether tests slow stuff down. I think that on average they make refactors easier and features slower, so for the projects I'm on it's a wash. But even if I grant the point my conclusion wouldn't change. Tests are good.
- adrianN 8y ago
  Without automated deploys, how do you know that what you deploy is actually what you tested?
  - 3pt14159 8y ago
    Personally I have automated deploys, I was just reaching for an example that was available and relevant for a wide range of projects from inception. I think the thinking is that early on everything is on master so you just do the deploy by hand, run the migrations / tests, and bring the server back online.
  - rhizome 8y ago
    Use tags?
- ozim 8y ago
bunderbunder 8y ago
> in order to write effective tests, a programmer had to know all of the ways that a piece of software could fail in order to write tests for those cases No. In order to write effective tests, a programmer has to think of the piece of software's entire input domain, carve it up into a set of equivalence classes, and then determine what the expected behavior should be for a piece of input from each of those classes. With more careful testing at their boundaries, so that you can root out edge and corner cases. The results of those tests will then find the ways your software can fail for you. Thinking about it this way, it starts to become clear that a huge part of your job is finding ways to simplify that input space as much as possible. Simplifying at this stage makes getting it right so much easier. The fewer special, edge, and corner cases you allow in the first place, the fewer you have to code for, the fewer you have to test for, the fewer you might accidentally miss, and the fewer others might accidentally run afoul of. This, incidentally, is the key reason why it's good to take some time to define all your tests before you start writing your implementation. I'm not necessarily a huge advocate for "red green refactor", but I do think that at least identifying all your test cases before you ever start implementing can potentially save you boatloads of time, by helping you recognize opportunities to simplify your design before you get locked into an unnecessarily complicated one by a couple hours or days (or months) of sunk cost. It's also the real reason (IMO) why functional programming - as a style, not a kind of language - is such a valuable discipline. The challenge with programming in an imperative style is that it turns your module's entire past history into one of the inputs you need to consider, and that turns your test and specification surface into something that is just so much bigger. Considering that it opens up the possibility of injecting bugs into a routine without ever actually editing the routine itself, or even any of the functions it calls, it's possibly even fair to say that it's unbounded.
- pjc50 8y ago
  > think of the piece of software's entire input domain, carve it up into a set of equivalence classes, and then determine what the expected behavior should be for a piece of input from each of those classes This is an extremely good way of putting it. Doing some of this at the requirements stage if possible can be very helpful too, because you can detect nonsensical or conflicting requirements before you even write any code.
stephengillie 8y ago
> “Users don’t care.” This is the counter to every technical bikeshedding post. Either it creates revenue by being what users want, or it's a waste of time. And people seem to love wasting time instead of delivering.
- TeMPOraL 8y ago
  Programming would be trivial if things were so simple. The reality is, that "either it creates revenue by being what users want", or it enables creating more of that later on, or prevents the speed of creating more of that from dropping, or prevents the whole thing from catastrophic errors (like in this article), ... Those are the trade-offs engineers have to consider daily.
- nandemo 8y ago
  In a surprising turn of events, the user showed they did care. > “@okcupid how am I supposed to get my daily dose of crushing rejection and emotional humiliation if your site is down????”
Jeff_Brown 8y ago
Static typing helps. Higher order programming helps. These are two reasons I love Haskell. The compiler does a lot of work of understanding your code for you, and makes it easy to query that metadata interactively, and refer to it from other parts of static code. Haskell is of course not the only statically typed language, but its type system encompasses more, and more uniformly, than any other I am aware of. A lot of things that used to have to be part of the comments can now be first-class citizens.
gred 8y ago
> My first employer, the online dating site OkCupid, didn’t harp on testing either. In part, this was because our company was so small. Only seven fellow engineers and I maintained all the code running on our servers—and making tests work was time-consuming and error-prone. “We can’t sacrifice forward momentum for technical debt,” then-CEO Mike Maxim told me, referring to the cost of engineers building behind-the-scenes tech instead of user-facing features. “Users don’t care.” He thought of testing frameworks as somewhat academic, more lofty than practical. I found this paragraph interesting because my experience with my one-man side project is the opposite: I'm so small, and time is so scarce (I have a separate full-time job), that I can't afford to fix bugs on somebody else's schedule when they find them... instead I spend a large amount of time writing tests up front for new features so that I find the bugs on my schedule (i.e. late at night when I don't have other commitments).
- toss1 8y ago
  Yup, classic. "...no time/resources to do it right in the first place, but plenty of time/resources to fix it when the customers complain..." I always thought it better to find the bugs in-house before shipping, but so many others don't see it...
  - nitrogen 8y ago
    The logic kind of makes sense if you consider that customers will only find a small fraction of the bugs that your team will. Sometimes the cost of removing all bugs is greater than the cost of losing customers to the few unlucky bugs that get found. But you really need experienced people deciding where that tradeoff lies for each company and project.
    - cs02rm0 8y ago
      I think that's the bottom line. I've done virtually zero unit testing on a project before when there was barely time to write the core code. There wasn't much business logic, lots of I/O and no user inputs or browsers to worry about and I was leading a small team of four so it was relatively well suited to it. Still, I was nervous as it was quite high profile and unit test coverage was about the only metric management knew to ask for. It was replacing a critical third party system being turned off on a fixed date and we barely made it. 6 months after being switched on in production we'd had zero bugs reported and I moved on. I'm confident in hindsight that I made the right call for the circumstances, but having it funded properly would have made me sweat a bit less.
08-15 8y ago
Is this really about testing? > If (the database throws an error) { do nothing } It's paraphrased, the original was probably the C++ anti pattern of > try { /some code/ } catch {...} {} Whoever wrote this should be disciplined. This kind of code doesn't do anything, it just makes a bad situation worse. (It's also never tested, because who writes tests that provoke failures?) This kind of idiocy has no technical solution, unless the two-by-four counts. If you're ever tempted to write something like this... don't. Especially in C++, the correct approach is to do nothing. Someone else will handle the exception. Code that isn't written doesn't cause trouble and doesn't need to be tested either.
ww520 8y ago
Yep. I had caused a similar problem in the past that brought down a live site. It's a cascade failure on error handling that caused an avalanche of retrying requests piling up that eventually more and more servers failed under the load. Not fun. Luckily we had well defined deployment and rollback procedures and was able to roll back the change easily.
evancox100 8y ago
"If you wrote buggy software, why would the software you wrote to check that software be any less buggy?" I work in chip design, where bugs can be rather costly, e.g. $1mm for a new mask set, not to mention the months it takes to get back new hw. The situation described above is why we try to have one person do the design and a different person do the verification/testing. A lot of the time, the test writer will treat the design as a black box, not even look at the code, and instead verify to an external specification. (There's also white box testing, where you try to target specifc areas of the design that might be especially problematic.) This does protect against issues like misinterpreted requirements, invalid assumptions of valid input and/or operating scenarios. But I think it also ends up as a lot more work, to have two people get familiar with whatever it is you're designing. The best method is probably a combination of both.
- ereyes01 8y ago
  When you have a well thought-out external specification, it really gives you lots of good material from which to derive all the right tests while still treating your implementation as a black box. Unless you are implementing an established RFC or some other well-defined system, such a tight specification rarely exists in a software setting, and is very difficult to effectively design a priori. For this reason, techniques like test-driven development have arisen to try to make the programmer repeatedly and consistently assert their understanding of the specification via unit tests, while evolving those assumptions as needs and understandings change.
- spc476 8y ago
  When I was doing QA at work, I never used the official tools to generate the data files required for testing the application, but wrote my own. This brought to light a ton of issues and unwritten assumptions (both in my code, and in the application).
marcus_holmes 8y ago
This is one of the many reasons why I have an allergy to dependencies. I know other people's code is probably better than mine. But I understand mine. If something goes wrong (and something always goes wrong) I know how to fix it. When something goes wrong in someone else's code I either have to start working out how their code works, or report it and sit there like a lemon with a broken system until whoever wrote it has the time, energy and inclination to fix it. Also, testing. I wish I was better at testing. At least I've got into the habit of writing tests to cover whatever bug I just discovered in production.
- azeirah 8y ago
  All I know is that if enough people rely on the same dependency for long enough, the chance of encountering large bugs becomes smaller and smaller. Especially if the dependency has a stable interface. Good software gets better the more it gets used and abused, so I tend to stay away from small dependencies, they're usually not worth the time.
xupybd 8y ago
A very well written piece that sums up the state of play without any sense of panic. That’s pretty rare for this sort of article.
mirimir 8y ago
I'm reminded of the day that I discovered SQL cross joins. Some decades ago, working in Access on WinXP. So I executed, and nothing much happened. So I went for lunch, and when I got back, the machine was frozen. So I went WTF?, and hard reset. And then did it again, a few times. And the cool thing was that doing that, and running out of disk space, apparently didn't damage the system. Microsoft must have designed WinXP to tolerate that.
heisnotanalien 8y ago
You write tests not only to hopefully catch bugs but so that you can change the software reliably significantly in future and not lose forward momentum.
gonzo 8y ago
> If she forgot the square root of -1 was undefined, she’d never write a test for it. Sigh. sqrt(-1) is not “undefined”.
planxty 8y ago
Test your code, or stay the hell away from me. :)
franzwong 8y ago
Sometimes I want to see how a piece of code work, I can just run the unit test. Otherwise, I may need to start the server, prepare some unrelated data, send a request to the server.

229 points by tortilla 8y ago

Wow. 1. We don't test. 2. We don't code review (or rather if we do, we do it so poorly swallowed exceptions don't raise red flags.) That's an outrageously unprofessional software process.

danesparza 8y ago
I was immediately struck by the same set of thoughts. I don't know whether to be appalled or just ... sad.
andrenotgiant 8y ago
An outrageously unprofessional software process that created a product worth $50M! https://techcrunch.com/2011/02/02/match-com-acquires-online-dating-site-okcupid-for-50-million-in-cash/ https://techcrunch.com/2011/02/02/match-com-acquires-online-...
- danesparza 8y ago
  The OP is about technical process in the software world. I'm not sure how a company's valuation comes into the conversation...
  - blacksmith_tb 8y ago
    I assume the implication is "if OKC can fly by the seat of their pants and still make millions, anyone could!" Which is true, but still not advisable...
    - stcredzero 8y ago
      Given that technical debt is quite hard for non-technical people to quantify, why wouldn't we expect that most startups would tend to accrue technical debt?
      - theseatoms 8y ago
        Can anyone quantify technical debt? Best case is something resembling a medical diagnosis.
        stcredzero 8y ago
        I think that could be enough. Actuaries have a good idea about when you are likely to die in the next two years. Basically, when you are old and your medical costs suddenly double. Doctors use the same sort of inductive reasoning for their prognosis. The industry probably collectively has all the information to put together useful actuarial tables on technical debt, but it's all silo'd in individual companies.
        theseatoms 8y ago
        Interesting. Are there IT consulting firms that specialize in this field of study?
      - adrianN 8y ago
        Startups accrue so much financial debt, adding some technical debt on top doesn't really make a difference.
        stcredzero 8y ago
        Technical debt doesn't accrue in a linear fashion. Instead, it's synergistic. Up to a certain point, it's a noticeable drag. Past that point, there is a change in the difficulty incurred. In the article, the combination of some "you're-supposed-to-just-not-do-that" with: >If (the database throws an error) { do nothing } resulted in a major mishap.
    - T2_t2 8y ago
      I think the implication is that they achieved the goal they had, to grow a company. Note that goal doesn't mention code at all. It depends what you think software engineering is - a means to an end, e.g. to grow a company, or as a craft you want to perfect and do for your whole career. I think there are more craft coders, because most jobs are at companies that have survived and there is more to lose from errors than gain from moving fast. The craft attitude may not be ideal when you want to go from zero to something, and avoid going back to zero very quickly. Or, perhaps more importantly, if you want to go from zero -> something wrong -> something wrong again -> something almost correct -> Product Market fit.
  - jschwartzi 8y ago
    The point is that engineering decisions need to involve both the business and the technical side of things, and they need to be pragmatic and not tied to any particular dogma from either world.
- mabbo 8y ago
  That's not skill, that's luck. If the bug in question had instead leaked every customer's personal data there wouldn't be a $50m company, there'd be a multi-million dollar lawsuit.
  - TeMPOraL 8y ago
    I agree with your thinking, but with this particular example: you'd wish. See: every other data leak that happened over the last decade. It seems data leaks are harmless to companies.
    - laurent123456 8y ago
      I was going to say "not for a dating website", but I see that Ashley Madison still exists despite the massive data leak in 2015 including data from users who had paid to have their data deleted, so yes looks like there's no real fallback for data leaks.
- JackFr 8y ago
  Created a product worth $50M because of their software process or despite there software process.
merlincorey 8y ago
I washed out of their interview process when I was fresh out of high school for not knowing bit twiddling hacks[1] in the technical phone screen with one of their engineers. After reading this, I can't help but feel like I dodged a bullet there. [1] https://graphics.stanford.edu/~seander/bithacks.html https://graphics.stanford.edu/~seander/bithacks.html
- gwbas1c 8y ago
  Don't feel bad: I cut the phone screen short about 25 minutes into it. I just got a really bad vibe about all of the front-end complexity they were building into a dating site.
swalsh 8y ago
It's a free dating service. They have a responsibility to respect their users' data privacy, but the rest kind of doesn't matter. Eventual stability is more profitable than eventual release if you're not a Bank or making self driving cars.
- skybrian 8y ago
  Yeah, I expect the devs at Ashley Madison told themselves that too. It's possible to build sandboxes where security or privacy bugs are contained, but the shops that don't care much about software quality are probably not doing that. It seems a little too easy for people building a social app to tell themselves they're just building a toy that doesn't matter. Their users might disagree after they get hacked.
KingMob 8y ago
I'm sure the OkCupid folk are nice people but it was: 1) founded by mathematicians and an academic, and 2) written in C++, with a custom templating language Even giving them the benefit of the doubt that the core matching algorithm warranted C++, the whole site really shouldn't have been built that way. I'm really not surprised there's poor technical decision-making there.
ddebernardy 8y ago
Outrageously unprofessional as it may be, they probably ended up laughing on their way to the bank when they got purchased by match.com.
toast0 8y ago
> That's an outrageously unprofessional software process. People who call me unprofessional forget that being a professional just means I'm paid to do it, it's not a pass time.
megaman8 8y ago
You have to judge it in context. If it's entirely a senior team then the value of code reviews is greatly diminished. You rarely catch a bug in a bug review when you're reviewing a Senior software engineer's code. those bugs aren't caught until you do testing.
- nradov 8y ago
  I've seen plenty of defects in senior developers' code that were caught in reviews. Please the purpose of code reviews isn't just to catch defects, it's also to disseminate knowledge of that module across the team in order to reduce the "bus factor".
chasd00 8y ago
it worked well enough until they could do something better. Never let perfect be the enemy of good.
tluyben2 8y ago
In my experience this ‘outrageously unprofessional‘ process is very normal. Most companies are not software companies and they are not going to pay for whatever is not adding to the bottomline. And even in software companies that ofcourse happens. It is unfortunate but the shock and awe are a bit over the top; when I talk with partners (ctos is large corps; banks, insurance, retail), they generally glaze over and mumble about their dreams of doing tests and code reviews but unfortunately it is not possible ‘at the moment’.

mikestew 8y ago

(then-CEO Mike Maxim) thought of testing frameworks as somewhat academic, more lofty than practical. ... Mike the CEO, who was also OkCupid’s best engineer... Mystery of the Melting Servers, solved.

plopz 8y ago
Do testing frameworks normally catch memory leaks?
- mikestew 8y ago
  Mine do.
  - fhood 8y ago
    You can't possibly say that with any confidence. I literally just finished reproducing a memory leak caused by an insane combination of circumstances. This was not something valgrind or any similar tool would find. It was not something that any rational human being would have thought to write tests for. Once code is no longer synchronous, and tasks are being juggled around and swapped between, it becomes virtually impossible to predict what might cause this sort of bug.
    - whatshisface 8y ago
      The parent is saying that their tests normally catch memory leaks, not that their tests catch all memory leaks.
    - tekstar 8y ago
      What's an example of a memory leak that can't be found by valgrind?
      - viraptor 8y ago
        One happening in an embedded language with its own GC. Unless you can switch all allocations to explicit management for the tests, it's almost impossible to differentiate between live objects, free lists, cycles with deferred collection, and arenas that are supposed to be freed on exit.

1_800_UNICORN 8y ago

If you're an early-stage startup that's still determining product-market fit, then it's completely fair to eschew tests and accumulate tech debt; you're at a higher risk of running out of runway before you've even proven that your business works. As soon as you have enough users that an outage poses a significant risk to your business, you need to invest in either refactoring to reduce tech debt, or a rewrite. And you NEED to add tests, and a robust deployment process that tests changes at multiple stages with monitoring on the basics (server speed and memory usage, database error rate, etc). OKCupid got lucky in this case.

nradov 8y ago
Skipping tests generally doesn't allow for creating a working MVP any faster. It's just an illusion.
- always_good 8y ago
  Only if you are referring to perfect tests which cost zero time to write/maintain and only the tests written that eventually will catch an issue. Otherwise, tests have a cost just like any other code. You can see this by looking at both extremes: perfect tests (described above) and useless tests. For example, tests that make your codebase too brittle, tests that don't actually test anything useful, spending too much time writing tests, spending a lot of time writing tests for precisely the code with high disposability, tests that are too coupled with the code, stupid tests that should be removed but won't be because tests tend to be append-only, etc. Testing is an advanced topic where every +1 unit you spend on testing doesn't mean your application is now +1 unit more robust. +1 unit of time spent in tests can even mean your application is -2 units worse because testing is a trade-off. So tests are more than capable of hamstringing your unlaunched MVP. They're also capable of being the reason you launched your MVP sooner than later. But the costs of testing are no illusion.
  - nradov 8y ago
    Tests save you time. They have a negative cost.

TheCapn 8y ago

(Off Topic -- Rant regarding Webpage) Can I take a short moment to bitch about "sidebar" frame or whatever the hell this domain is using? The webpage looks like a fluid single frame but the left side is static. The browser page has a scrollbar but I need to have my mouse over the correct div before my scroll wheel will move the page, its a bit aggravating since there's no visual break between the "sidebar" and the content.

pwg 8y ago
I just turned off CSS in Firefox (View menu -> Page Style -> No Style) and the fixed sidebar was no more and the odd scroll bar behavior was gone.
- pritambaral 8y ago
  I used Firefox's Reader Mode.
deleted 8y ago
[deleted]

noelwelsh 8y ago

How could such a tiny change have such an outsized impact on the site? “That same story happened so many different times,” my old boss David told me. “Someone launched a small, relatively innocuous change that did one of the millions of unexpected things it could have done, which then happened to break some part of the site, and then bring it all down—sometimes bring it down to the point where we couldn’t recover it for hours.” Preventing this is one of the basic motivations of functional programming. If a function is "referentially transparent" or "pure" it can't do anything unexpected. It seems to me this is the only reasonable way we can reason about software at scale.

jolmg 8y ago
While I generally agree that functional programming does help immensely to make secure software, in this case, the function wasn't pure. It was meant to return data that didn't exist from a database, but the error was silenced. Also, the reason why the servers were "burning" was because this somehow resulted in a memory leak that consumed all the memory available to the system, which is why they also had trouble pushing fixes. Even in a pure function in Haskell, memory leaks are possible and the same scenario could've resulted.
mreome 8y ago
While I can understand some of the benefits of functional programming within a single sub-system, or within academic research, I struggle to understand how pure functional programming can address the need for the large amounts of state information and abstraction layers required for a large scale system. I work with systems where (large/numerous) sub components require very specialized domain knowledge to understand internally, so multiple levels of black-box abstraction are required to integrate/interact with the sub components. Additionally, both the client systems and the sub components may have huge amounts of persistent state information. Could you suggest any resources that might help me understand how this can be accomplished in a pure functional paradigm? Something that assumes only a casual understanding of functional programming would be especially helpful.
- TheDong 8y ago

stcredzero 8y ago

Yet what I found even more troubling was that in order to write effective tests, a programmer had to know all of the ways that a piece of software could fail in order to write tests for those cases. So the programmer needs to think about all the ways the code can fail. Writing the tests concretely documents that thinking in the source code repository and automates re-checking that thinking. A really good programmer then tries to organize the code, such that it's easy to think about how any given unit of code can fail. Writing unit tests rewards such organization and penalizes bad organization. That said, TDD doesn't seem to work out that way in practice, most of the time.

hnruss 8y ago

After reading the introduction, I thought that the author was going to use a story of software failure as an example of why you should write unit tests, or at least why you shouldn't deploy untested software. However, the moral of the story was essentially: "software is so complicated that it is bound it break, so you have to be good at fixing it". While that is certainly true, I think that developers have a responsibility to use whatever tools they can to write high-quality software.

jschwartzi 8y ago
> However, the moral of the story was essentially: "software is so complicated that it is bound it break, so you have to be good at fixing it". While that is certainly true, I think that developers have a responsibility to use whatever tools they can to write high-quality software. I actually disagree that we have a responsibility to write high-quality software. As engineers our job is to create software that is good enough for the task at hand and that can be improved later. Sometimes that means that you invest in 100% coverage plus integration and system-level testing because "good enough" means that it has to never fail. Sometimes that means you don't spend much time automating tests and instead do informal development testing before you check something in. It's entirely up to your judgement based on what your code is going to be used for, and I don't think it was inappropriate for OkCupid's engineering staff to avoid testing in this case. After all, their business was really successful through the period where the error happened.
- jhayward 8y ago
  > As engineers our job is to create software that is good enough for the task at hand This is the essence of engineering. As the saying goes, "Any fool can build a bridge that won't fall down. It takes an engineer to build a bridge that just barely won't fall down." In other words an engineer's contribution is not perfection, it is being able to know the difference between "good enough" and "needs more work" and to apply that knowledge to meet time, cost, and material constraints.
  - user5994461 8y ago

matt2000 8y ago

When making something new where you're not sure of the value yet, I've found that you can get 80% of the benefits of unit tests with around 20% of the tests you'd write to get "full coverage." My main goal is to at least have the code run in an expected way and produce an expected result. This doesn't catch everything, but it does seem to catch enough problems to be worth it for the time invested. Edit: I should mention that I also add tests to cover something when I experience a failure, so it at least won't happen again.

sneak 8y ago
Yes to the last! Whenever bughunting my first step is to reproduce the issue in a failing test. This has the benefit of me being sure I know what is failing and how, but also crystallizes then knowledge and experience into the repo for future engineers (or future me) to know the most common/practical ways things break in the real world. It is also marginally useful for catching regressions, if the test is clever/general enough.
ddebernardy 8y ago
The big problem I've noticed when reading unit tests in various projects is that they frequently test the implementation rather than the outcome: https://softwareengineering.stackexchange.com/a/304910/24932 https://softwareengineering.stackexchange.com/a/304910/24932
- macintux 8y ago
  That's one reason I really like functional programming: it's easier to test pure functions, vs code with side effects that requires you to look at the internal state afterwards.
  - tobr 8y ago
    On the other hand, pure functions are also much easier to reason about, thus much less likely to cause bugs from "a tiny change" in the first place. It's in the messy imperative stuff things tend to break in incomprehensible ways, and where you really need tests.

drblast 8y ago

The reactions to the outage on Twitter are hilarious and make me glad this happened: “@okcupid how am I supposed to get my daily dose of crushing rejection and emotional humiliation if your site is down????” “Okcupid stops working over lunch hour, NYC wonders if we're committed to the people in our phones from now on, panic in the streets” “@okcupid How can I continue to be ignored by the females of the world if they don't know where I am to ignore me?! #panic #freakout”

ahallock 8y ago

> ...in order to write effective tests, a programmer had to know all of the ways that a piece of software could fail in order to write tests for those cases That's impossible. You reason about the problem as best as you can and create tests appropriately; and then if there are edge cases/failures in production, you go back and add those to the test suite.

3pt14159 8y ago

I appreciated this article because I so infrequently get to hear from people that hold this view of testing, which I do not share. To me, the importance of tests are a function of the consequence of failure and the likelihood that failure will happen. It's a question of hazard and risk. One thing to realize is that code that lasts longer is more likely to fail because the people and libraries that support it are prone to change and assumptions can break. So risk goes up over time. The other thing to realize is that code that's part of a growing product is going to impact more people if it fails. There might be only 400 users today, but 40k three years from now. So hazard goes up over time. When I heard the technical debt arguments I try to triage them into "grows with the company" vs "doesn't grow with the company" so I can figure out what to accept as debt and what to pay. A complex deploy infrastructure doesn't scale. Just SSHing into a box and doing the deploy manually is just as fine for 100 users as it is for 1000. But tests do. So I write lots of tests; especially for ACL and I punt on the infrastructure stuff until I absolutely need to. Note that this leaves aside the entire argument over whether tests slow stuff down. I think that on average they make refactors easier and features slower, so for the projects I'm on it's a wash. But even if I grant the point my conclusion wouldn't change. Tests are good.

adrianN 8y ago
Without automated deploys, how do you know that what you deploy is actually what you tested?
- 3pt14159 8y ago
  Personally I have automated deploys, I was just reaching for an example that was available and relevant for a wide range of projects from inception. I think the thinking is that early on everything is on master so you just do the deploy by hand, run the migrations / tests, and bring the server back online.
- rhizome 8y ago
  Use tags?
ozim 8y ago

bunderbunder 8y ago

> in order to write effective tests, a programmer had to know all of the ways that a piece of software could fail in order to write tests for those cases No. In order to write effective tests, a programmer has to think of the piece of software's entire input domain, carve it up into a set of equivalence classes, and then determine what the expected behavior should be for a piece of input from each of those classes. With more careful testing at their boundaries, so that you can root out edge and corner cases. The results of those tests will then find the ways your software can fail for you. Thinking about it this way, it starts to become clear that a huge part of your job is finding ways to simplify that input space as much as possible. Simplifying at this stage makes getting it right so much easier. The fewer special, edge, and corner cases you allow in the first place, the fewer you have to code for, the fewer you have to test for, the fewer you might accidentally miss, and the fewer others might accidentally run afoul of. This, incidentally, is the key reason why it's good to take some time to define all your tests before you start writing your implementation. I'm not necessarily a huge advocate for "red green refactor", but I do think that at least identifying all your test cases before you ever start implementing can potentially save you boatloads of time, by helping you recognize opportunities to simplify your design before you get locked into an unnecessarily complicated one by a couple hours or days (or months) of sunk cost. It's also the real reason (IMO) why functional programming - as a style, not a kind of language - is such a valuable discipline. The challenge with programming in an imperative style is that it turns your module's entire past history into one of the inputs you need to consider, and that turns your test and specification surface into something that is just so much bigger. Considering that it opens up the possibility of injecting bugs into a routine without ever actually editing the routine itself, or even any of the functions it calls, it's possibly even fair to say that it's unbounded.

pjc50 8y ago
> think of the piece of software's entire input domain, carve it up into a set of equivalence classes, and then determine what the expected behavior should be for a piece of input from each of those classes This is an extremely good way of putting it. Doing some of this at the requirements stage if possible can be very helpful too, because you can detect nonsensical or conflicting requirements before you even write any code.

stephengillie 8y ago

> “Users don’t care.” This is the counter to every technical bikeshedding post. Either it creates revenue by being what users want, or it's a waste of time. And people seem to love wasting time instead of delivering.

TeMPOraL 8y ago
Programming would be trivial if things were so simple. The reality is, that "either it creates revenue by being what users want", or it enables creating more of that later on, or prevents the speed of creating more of that from dropping, or prevents the whole thing from catastrophic errors (like in this article), ... Those are the trade-offs engineers have to consider daily.
nandemo 8y ago
In a surprising turn of events, the user showed they did care. > “@okcupid how am I supposed to get my daily dose of crushing rejection and emotional humiliation if your site is down????”

Jeff_Brown 8y ago

Static typing helps. Higher order programming helps. These are two reasons I love Haskell. The compiler does a lot of work of understanding your code for you, and makes it easy to query that metadata interactively, and refer to it from other parts of static code. Haskell is of course not the only statically typed language, but its type system encompasses more, and more uniformly, than any other I am aware of. A lot of things that used to have to be part of the comments can now be first-class citizens.

gred 8y ago

> My first employer, the online dating site OkCupid, didn’t harp on testing either. In part, this was because our company was so small. Only seven fellow engineers and I maintained all the code running on our servers—and making tests work was time-consuming and error-prone. “We can’t sacrifice forward momentum for technical debt,” then-CEO Mike Maxim told me, referring to the cost of engineers building behind-the-scenes tech instead of user-facing features. “Users don’t care.” He thought of testing frameworks as somewhat academic, more lofty than practical. I found this paragraph interesting because my experience with my one-man side project is the opposite: I'm so small, and time is so scarce (I have a separate full-time job), that I can't afford to fix bugs on somebody else's schedule when they find them... instead I spend a large amount of time writing tests up front for new features so that I find the bugs on my schedule (i.e. late at night when I don't have other commitments).

toss1 8y ago
Yup, classic. "...no time/resources to do it right in the first place, but plenty of time/resources to fix it when the customers complain..." I always thought it better to find the bugs in-house before shipping, but so many others don't see it...
- nitrogen 8y ago
  The logic kind of makes sense if you consider that customers will only find a small fraction of the bugs that your team will. Sometimes the cost of removing all bugs is greater than the cost of losing customers to the few unlucky bugs that get found. But you really need experienced people deciding where that tradeoff lies for each company and project.
  - cs02rm0 8y ago
    I think that's the bottom line. I've done virtually zero unit testing on a project before when there was barely time to write the core code. There wasn't much business logic, lots of I/O and no user inputs or browsers to worry about and I was leading a small team of four so it was relatively well suited to it. Still, I was nervous as it was quite high profile and unit test coverage was about the only metric management knew to ask for. It was replacing a critical third party system being turned off on a fixed date and we barely made it. 6 months after being switched on in production we'd had zero bugs reported and I moved on. I'm confident in hindsight that I made the right call for the circumstances, but having it funded properly would have made me sweat a bit less.

08-15 8y ago

Is this really about testing? > If (the database throws an error) { do nothing } It's paraphrased, the original was probably the C++ anti pattern of > try { /some code/ } catch {...} {} Whoever wrote this should be disciplined. This kind of code doesn't do anything, it just makes a bad situation worse. (It's also never tested, because who writes tests that provoke failures?) This kind of idiocy has no technical solution, unless the two-by-four counts. If you're ever tempted to write something like this... don't. Especially in C++, the correct approach is to do nothing. Someone else will handle the exception. Code that isn't written doesn't cause trouble and doesn't need to be tested either.

ww520 8y ago

Yep. I had caused a similar problem in the past that brought down a live site. It's a cascade failure on error handling that caused an avalanche of retrying requests piling up that eventually more and more servers failed under the load. Not fun. Luckily we had well defined deployment and rollback procedures and was able to roll back the change easily.

evancox100 8y ago

"If you wrote buggy software, why would the software you wrote to check that software be any less buggy?" I work in chip design, where bugs can be rather costly, e.g. $1mm for a new mask set, not to mention the months it takes to get back new hw. The situation described above is why we try to have one person do the design and a different person do the verification/testing. A lot of the time, the test writer will treat the design as a black box, not even look at the code, and instead verify to an external specification. (There's also white box testing, where you try to target specifc areas of the design that might be especially problematic.) This does protect against issues like misinterpreted requirements, invalid assumptions of valid input and/or operating scenarios. But I think it also ends up as a lot more work, to have two people get familiar with whatever it is you're designing. The best method is probably a combination of both.

ereyes01 8y ago
When you have a well thought-out external specification, it really gives you lots of good material from which to derive all the right tests while still treating your implementation as a black box. Unless you are implementing an established RFC or some other well-defined system, such a tight specification rarely exists in a software setting, and is very difficult to effectively design a priori. For this reason, techniques like test-driven development have arisen to try to make the programmer repeatedly and consistently assert their understanding of the specification via unit tests, while evolving those assumptions as needs and understandings change.
spc476 8y ago
When I was doing QA at work, I never used the official tools to generate the data files required for testing the application, but wrote my own. This brought to light a ton of issues and unwritten assumptions (both in my code, and in the application).

marcus_holmes 8y ago

This is one of the many reasons why I have an allergy to dependencies. I know other people's code is probably better than mine. But I understand mine. If something goes wrong (and something always goes wrong) I know how to fix it. When something goes wrong in someone else's code I either have to start working out how their code works, or report it and sit there like a lemon with a broken system until whoever wrote it has the time, energy and inclination to fix it. Also, testing. I wish I was better at testing. At least I've got into the habit of writing tests to cover whatever bug I just discovered in production.

azeirah 8y ago
All I know is that if enough people rely on the same dependency for long enough, the chance of encountering large bugs becomes smaller and smaller. Especially if the dependency has a stable interface. Good software gets better the more it gets used and abused, so I tend to stay away from small dependencies, they're usually not worth the time.

xupybd 8y ago

A very well written piece that sums up the state of play without any sense of panic. That’s pretty rare for this sort of article.

mirimir 8y ago

I'm reminded of the day that I discovered SQL cross joins. Some decades ago, working in Access on WinXP. So I executed, and nothing much happened. So I went for lunch, and when I got back, the machine was frozen. So I went WTF?, and hard reset. And then did it again, a few times. And the cool thing was that doing that, and running out of disk space, apparently didn't damage the system. Microsoft must have designed WinXP to tolerate that.

heisnotanalien 8y ago

You write tests not only to hopefully catch bugs but so that you can change the software reliably significantly in future and not lose forward momentum.

gonzo 8y ago

> If she forgot the square root of -1 was undefined, she’d never write a test for it. Sigh. sqrt(-1) is not “undefined”.

planxty 8y ago

Test your code, or stay the hell away from me. :)

franzwong 8y ago

Sometimes I want to see how a piece of code work, I can just run the unit test. Otherwise, I may need to start the server, prepare some unrelated data, send a request to the server.