5 ms·
Finally, a few months into the job, I wrote “break” where I meant “continue,” and caused our payments servers to shutdown one-by-one over the course of several
by jrockway 15y ago
Finally, a few months into the job, I wrote “break” where I meant “continue,” and caused our payments servers to shutdown one-by-one over the course of several weeks.
So, they don't write tests at justin.tv, and they don't do automatic deployment? Sounds like a great place to work at...
- ankimal 15y agoThings are not quite as wild-west at Justin.tv as they once were. We now have QA (Thanks B!), monitoring and deploy management. I think most small teams start with minimal qa and testing (not the recommended approach) and thus things like this happen often. Rapid development becomes blitzkrieg.
- shadowmatter 15y agoThat's what caught my eye too. I like how it's soon followed by "Pushing code fast and often does have a cost, but the benefit in productivity is well worth it." That's certainly true if it's mostly _tested_ code, but there is nothing productive about spending a day trying to find and fix a bug like this in production. Been there, done that. I'm trying to bootstrap a startup now, and if it fails to gain traction then justin.tv would be one of the first places I'd interview. (Hey, I like watching competitive TF2, and they provide a hell of a platform for casts, can I say.) Now I know this blog post was sort of written for the purposes of recruitment, but it's sort of making me think twice about whether I'd want to interview there. Bleh.
- emmett 15y agoWe're not the wild west we once were, but neither are we a paragon of testing. When I started Justin.tv, I had 1.5 years of software development experience, never on a team larger than 2 developers; I've now been at it for almost 7 years and yeah I don't think I'd do everything the same from the beginning. Here are some things we now do at Justin.tv because of experiences like the one you mentioned: - Everything gets code reviewed - We have started to introduce tests (sorry Bill, they're really quite helpful in the right circumstance) - We have extensive monitoring for all systems in case something DOES slip through - We have fully automated deployment (and rollback) There's a cost to all of these things, either in setup time or in constant maintenance. When you have few users, maybe they're not worth it - as you grow they become essential.
- abstractbill 15y agosorry Bill, they're really quite helpful in the right circumstance I don't think I've ever claimed they're never useful ;) But I do still believe that, out of the things you listed, they have the lowest utility for the amount of effort involved. Personally, I would do fully-automated deploy and rollback first, closely followed by monitoring, then code reviews, and then if problems were still slipping through the net I'd tell people to start writing unit tests.
- aaronblohowiak 15y agoCode review has larger organizational benefits like reducing the "bus factor."
- abstractbill 15y agoThere seems to be a general assumption that writing [unit?] tests is always the "right thing" to do, even at a startup, and no matter what the code in question does. In my experience that's absolutely not the case.
- timr 15y agoJust to be clear: Bill's opinion on this isn't universally shared at JTV. When you're two guys in a garage, unit tests probably slow you down. But at some point, they begin to dramatically speed things up -- working on someone other person's code becomes orders of magnitude easier when there's a comprehensive test suite to validate your changes. The problem is that at some point between first employee and a dozen (or more) engineers, you start to wish you'd spent a bit more time on testing code. Writing those tests from scratch becomes "a project", and it only gets done with herculean effort.
- jdeseno 15y agoHaving been at JTV, looking back it almost seems insane to try to do as much work as we we're without good coverage. If it matters, it should get a test. I think a lot of people also see tests as slowing things down but, they also provide a lot of value in documenting intentions, make upgrades easier, etc.
- timr 15y agoIt also matters a lot more when you're hiring relatively new/inexperienced people, and throwing them in the deep end. I disagree with Bill mostly because I see the purpose of testing differently than he: the point is not to catch All Of The Bugs; the point is to make it easier to confidently work on code that you didn't write. There's a lot of counter-intuitive stuff that happens in production code for a big website, and it's not totally fair to throw inexperienced people at it without at least some safety belts in place.
- abstractbill 15y ago
- emmett 15y ago1) We've started introducing tests - as you grow I think they're essential. I would say though, not all bugs can be caught by tests (and in fact, I once went over the last 10 bugs we had fixed and discovered that none of them could have been caught by any testing framework). Here are some kinds of bugs you can't catch with testing: - The code assumes that an external service is highly available - The page looks glitchy in IE - If you use that flash api, computers with an nvidia graphics card will show a green screen - If you open and close a socket quickly, flash will hard-crash the entire browser...sometimes - Under peak load, when a vacuum is running, this new query will hose the DB That said, it sounds like this bug COULD have been caught by testing (though I don't know the details and can't be sure). And tests can be darn useful, when used in the right circumstance. But in the hierarchy of important things for running a production website, here's what I would list in priority order: - Extensive monitoring and alerting - Detailed, easily accessible logging - Code review required and taken seriously - Hiring good developers - Automated deployment and rollback - Integration testing - Unit testing We've been working our way down the hierarchy :-) 2) We have 100% automated deployment and rollback, and a neat internal tool called Brigade for managing it that I really should get around to open sourcing some day...
- ammon 15y agoFor what it's worth, the payment servers were one of the only parts of our system that did have (manual) tests in place at that time (because it took 24 hours to do a restart). The bug in question was in DB timeout code, and all tests passed (until several weeks later, when the DB had problems). Now, clearly coverage was insufficient, and I do agree that adding more testing (as we now have) is a good idea. But some bugs are always going to slip past. Code reviews are what could have caught this bug (which we also now take more seriously).
- justin_vanw 15y agoEvery hotshot thinks they don't need tests until they try writing tests for a while, then they usually become evangelists. My shock and amazement comes from the 24 hours to restart your payments system? Whaaaaat? If that is correct you guys have far, far more issues than not having tests.
- dools 15y agoThe goal of testing is not to never have bugs, but to never have the same bug twice.
- MPiccinato 15y agoI knew this post reminded me of another post from a while back, over 2 years ago now, on not writing unit tests for justin.tv: http://abstractstuff.livejournal.com/60388.html http://abstractstuff.livejournal.com/60388.html