18 ms·

Don’t Do This in Production

696 points by pattrn 8y ago

pattrn 8y ago
This blog post describes my motivation for writing a blog. It's mostly a written memoire for myself to remember in detail what motivated me to start writing, but I thought HN readers might have some good feedback, so here it is. Would love to hear any stories you have of production fiascos.
- deleted 8y ago
  [deleted]
quickben 8y ago
Summary: The author clickbaits the title with "don't do X". Proceeds with a life story of his struggles ommiting what X is.
- pattrn 8y ago
  Sorry about that. I guess I didn't really think much about the title sounding like clickbait. The phrase is what motivated me to write the post, so I used it as the title. Perhaps I should put quotes around it?
  - turdnagel 8y ago
    Quotes would probably be more "accurate," but I don't think they'd quell this person's (unwarranted) complaints. I believe sharing your story was valuable and I look forward to reading more of your blog.
    - pattrn 8y ago
      Updated. Thanks!
  - quickben 8y ago
    It would help, thank you for that. I was expecting an actual example, which what was my problem with the title. It was more subtle than that so I read the whole thing but kept wanting to know what the damn line was. The downvotes that I am receiving actually reflects the same conceptual problem that you are outlining in the article. People are nice. They don't call out bullshit early on, and companies end up with nice-but-incompetent that can't deliver quality (but apparently can downvote on HN thinking they are doing right in life following the echo chamber). This doesn't plague many other licensed industries. Either you reach a level to get your license (and lose it if you are incompetent), or won't be doing your professional job at all. There is no "nice" in law, medicine, architecture, etc, where professionalism is concerned. It's conflicting. Should we be altruistic, promote teaching, but also promote being not nice? Or shall we be selfish, keeping quiet and ensuring our personal job-security because we can fix messes afterward. Overall, I think your blog is in the right general direction. Best of luck.
- lmkg 8y ago
  X in this case is copy-pasting code into production from blogs that explicitly say not to copy-paste their code into production. I do not feel it was omitted from the article, it was stated rather directly more than once. Perhaps you were expecting a description of why that particular code was unsuitable for production? The blog is about the process of writing code, not the code itself.
- jasonlotito 8y ago
  > The author clickbaits the title with "don't do X". It's not. > Proceeds with a life story of his struggles ommiting what X is It does.
- Walkman 8y ago
  If you can't even read, why bother commenting?
  - dang 8y ago
    Please don't reply to a bad comment with a worse one. Personal attacks in particular will get you banned here. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
tlb 8y ago
There's no reasonable way to write code suitable for "production" in general, because there are so many conflicting requirements. Just among internet services, components need different kinds of robustness for content websites, communication, e-commerce, email, banking, social networks, internal corporate services, and more. There do exist components suitable for most environments, like Postgres, but usually at the cost of 10x complexity over what'd be needed for the pure functionality. That engineering is worth it for things like databases that get used everywhere, but not for any custom component.
ballenf 8y ago
There's a great book Apprenticeship Patterns which gives real answers and advice to developers. It addresses the situation described in a few different ways and offers different strategies. Most of them boil down to how to pick a team where you will learn the most quickly while not drowning. https://www.amazon.com/Apprenticeship-Patterns-Guidance-Aspiring-Craftsman/dp/0596518382 https://www.amazon.com/Apprenticeship-Patterns-Guidance-Aspi...
- pattrn 8y ago
  Thanks for the recommendation. I'll check it out.
- mazatta 8y ago
  Of all the books I recommend for junior/intermediate developers, this is the one I demand they read. I wish the the free online version was still available, but it doesn't appear to have survived O'Reilly's switch to the subscription model.
peterwwillis 8y ago
Bugs will always happen in production. The important part is not developing everything correctly. "Move fast and break stuff" is a decent philosophy, as long as you can learn from mistakes and fix recurring problems. When a bug in your application presents itself in production, something very specific should happen: 1) identify the symptoms, 2) isolate the causes, 3) address the issue in production, 4) perform a postmortem to identify the next steps, 5) fix the problem's causes throughout the entire supply chain so it cannot happen again. A lot of organizations get stuck only doing steps 1-3. That's not good. Operational processes are just as important, if not more so, than development processes. They're what ensure that when you find inefficiency or bugs, that they not only get fixed, but that they never happen again. This means actually making policy changes, training changes, etc in addition to just fixing code. This is the heart of lean manufacturing that supposedly influenced a lot of modern software development methodology. But some people may have forgotten that the lean manufacturing chain doesn't end at development.
- nordsieck 8y ago
  > 4) perform a postmortem to identify the next steps It takes an effective management team to make this step work. If the postmortem isn't blameless, this stem becomes completely ineffective as employees (not unreasonably) prioritize their own careers over the health of the company.
  - sidlls 8y ago
    Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task. I think there's a middle ground between never holding anyone accountable and the desperate situation you describe.
    - nordsieck 8y ago
      > Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task. It is true that some people are not good at their jobs. It's also true that the only way to discover that people are bad at their jobs is their performance. However, I think the process approach is applicable to a much larger domain than many people give it credit. One of the greatest software development quotes I've read comes from DJB [0]: > For many years I have been systematically identifying error-prone programming habits—by reviewing the literature, analyzing other people’s mistakes, and analyzing my own mistakes—and redesigning my programming environment to eliminate those habits. The difference in security bugs found in qmail and sendmail speaks for itself. This is an example of the superiority of process: it shows that what some people consider a "root cause" is actually only a proximate cause - that the root cause is the programming environment that allows the bugs to occur. [0] https://cr.yp.to/qmail/qmailsec-20071101.pdf https://cr.yp.to/qmail/qmailsec-20071101.pdf
ohiovr 8y ago
I usually leave a URL for stack overflow answers or other advice as a comment in what I'm working on. I know some areas of my code are more sensitive than others. It is good to have an understanding of what you are doing of course. Sometimes a stackoverflow link is removed simply because the mode of thinking is wide spread in the application and I no longer need the note to it. Might be worth searching for my links in the code when it is time for release incase there is any funny business. Stackoverflow usually tells you if you "shouldn't do it in production".
crazygringo 8y ago
Can someone give an example of a blog post that gives example code yet warns "not to do it in production"? I've programmed and read about programming for decades and can't recall ever seeing anything like this. It's hard for me to take this article seriously without even a single example -- I can't tell if this is a strawman or not. Is it about code that isn't thread-safe? That relies on undefined behavior? That doesn't scale? That has race conditions? "In dev" and "in prod" can mean so many different things (is it about scale? or running on different OS's? or running unattended? or running on different hardware? or running compiled?), out of any particular context they're essentially meaningless, and in case particular case, it would seem to be the particulars that matter...
- thinkingemote 8y ago
  First result from DDG. https://blogs.msdn.microsoft.com/canberrapfe/2013/02/16/powershell-web-access-very-quick-start-yep-powershell-in-your-web-browser/ https://blogs.msdn.microsoft.com/canberrapfe/2013/02/16/powe... the thing in that example is that a thing is not properly configured but good enough to get started with in development. Second result: https://stackoverflow.com/questions/1475297/phps-white-screen-of-death https://stackoverflow.com/questions/1475297/phps-white-scree... describes how to put debugging statements in the code which is good for local development but which shouldnt be kept for production.
- ISL 8y ago
  I searched for "don't do this in production". This article was the first hit, and several blog posts with example code followed.
- reificator 8y ago
  I've seen it dozens of times, and dozens more read articles where they should have. The entire Supercharged playlist on Chrome's Youtube channel does this, for instance.
- bartread 8y ago
  > It's hard for me to take this article seriously without even a single example -- I can't tell if this is a strawman or not. I felt similarly: it just rang slightly false to me. For one thing, a code review wouldn't be the first solution I'd reach for when diagnosing the kind of problems described (performance issues, memory leaks, random crashes). There's plenty of tooling available for most platforms that will generally get you to a solution more quickly and reliably (granted, the post mentions only spending half a day), and often highlight problems you might easily miss in a code review. I'm not saying code reviews aren't useful, but that measurement of real-life behaviour can be more useful in these circumstances. Still, the point about not copy-pasting code directly from the internet is well made: been bitten enough times over the years that I've become extremely wary of it, even - and perhaps especially - when in a hurry.
angersock 8y ago
"If it's stupid but it works, it isn't stupid". I liked the post, but I'm not sure that I agree.
- detaro 8y ago
  To take one of the classics of bad copy-past examples: "If the 'use-PHP-with-MySQL' example you copied is stupid, works, and also causes your production database to be hacked due to SQL-injection, it probably was stupid" The problem is to tell the difference between "ugly stupid" and "dangerous stupid"
- golergka 8y ago
  The problem is with defining "works". If your code "works" with ten test entries in a DB when you test the happy path, it doesn't mean that it really works in production, with real load and real attack attempts on it.
- nordsieck 8y ago
  "works" is a nuanced word. The saying can be wonderfully pithy or terribly destructive depending on people's understanding of the word.
- krallja 8y ago
  “If it’s stupid but it works, it’s still stupid and you got lucky.”
dsr_ 8y ago
"Move fast and break stuff" is a decent philosophy if and only if the consequences of breaking stuff are survivable. If breaking stuff means that your website looks weird, that's survivable. If breaking stuff means that performance sucks for a while, that's survivable. If breaking stuff causes unavailability during a critical period of end-user demand, a few incidents might be survivable. If breaking stuff causes your company to have a terrible reputation for privacy, security, or competency, that might not be survivable. If breaking stuff causes your company to divulge financial information, that might not be survivable. If breaking stuff ends up costing your customers any significant amount of money, that probably will not be survivable.
- credit_guy 8y ago
  If breaking stuff causes a pedestrian to be killed because "they came from the shadows"[1], that appears to be perfectly survivable... /s [1] https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg
  - ttty 8y ago
    Cars killed people... they didn't stop creating cars. Nor would be a good idea.
    - abraham_lincoln 8y ago
      No, we got jaywalking laws and our rights reduced.
      - anticensor 8y ago
        We got safer cars at the same time. It is a double-edged sword.
    - TheCoelacanth 8y ago
      Killing one person per 1 million miles driven like Uber's self-driving cars have done is drastically different from killing one person per 80 million miles like human drivers do.
wiz21c 8y ago
One thing I see amongst many of developpers, is that they write code like it will work in production. OTOH, I tend to write code that will crash nicely : good error reporting, ability to continue after a crash, write code that handle only the data it needs, make sure queries address the smallest possible set of data, log a lot. Interestingly, that's a lot of code I have to write anyway because it helps me to write my programs (I mostly don't use debuggers).
- pattrn 8y ago
  Completely agree. One of the first problems I found was that the server wasn't running via a process manager, so when it crashed it never rebooted. Simple enough to solve, but non-obvious if you've never run into it before. It's a checkmark for anyone who has hosted any application before, but it's one of many important ones.
  - orev 8y ago
    I can’t think of any well-known professional projects that operate like this. IMO, automatically restarting after a crash is a terrible crutch that ensures crashing issues never get fixed at the priority level they deserve. Can you imagine if Apache, postgres, postfix, haproxy, etc. operated this way? If you have so many crashing issues that you need something like this, you have serious problems.
    - zzzcpan 8y ago
      I encourage you to read Joe Armstrong's dissertation of how a simple idea of restarting on crash can make you an impossibly reliable system.
    - barrkel 8y ago
      Stateless servers can and should recover automatically after a crash. Erlang has built a whole philosophy of reliable software construction based on the idea of letting errors crash a process and be restarted by a supervisor - systems with higher reliability requirements than most database servers, e.g. upgrades with zero downtime.
wpietri 8y ago
I want to underline this bit: > they hired a team of developers without having a technical person on staff to vet them For any non-technical entrepreneurs reading this: please don't do this. If you aren't competent to hire developers, please borrow or rent a few trusted technical experts and use them as a hiring committee. Otherwise you are less likely to hire the best technologist, and more likely to get the most glib and appealing technologist. Over the years I have met far too many smooth-talking consultants and would-be employees. And I've seen them cause enormous garbage fires when managers hire beyond their ability to evaluate. The size of these garbage fires are a direct result of what got them hired: if someone is good at telling managers what they want to hear, they can go a very long time saying, "Yup, we're building it and it will be great! Just you wait and see!" And then, no matter who you hire, demand a "ship early and often" schedule. Ship to internal users. Ship to alpha users. Ship to external testers. Heck, ship to just person for just one narrow use case. The earlier you can start seeing real-world success or failure, the earlier you can course correct.
- brianpan 8y ago
  And how do you find the trusted technical experts? By using your trusted technical expert expert?
  - hyperpape 8y ago
    By not going in to a business that requires development unless you are one, or know one. I wouldn't assume I could start a restaurant or run a medical practice, why do so many people with no technical background assume that they can start a business centering around software?
    - jasonlotito 8y ago
      Because lots of people that haven't run a restaurant have successfully started a restaurant, and people who were in the restaurant business started a restaurant, but failed to keep it going. History is littered with people who ignored your rule here and have been successful. I guess we can flip that around: why do so many people with no business background assume that they can start a business?
paulie_a 8y ago
I work with a consultant that dealt with emergency situations that had the motto, "be brave, be bold, deploy" In a different situation three of us figured out a solution for a critical situation. We voted on deploying with the thought "well we can't guarantee it will work, but it won't make things worse"
- walshemj 8y ago
  They have not recently been working at the TSB :-) The TSB a medium size UK bank had a very well publicised disaster - when doing a one way no roll back possible switch from an legacy IBM system to a Java based one
Yokohiii 8y ago
Building a good team is hard. Yet companies think they can hire an engineer which had a senior position somewhere and be fine. Senior or 10y experience can mean literally nothing, it doesn't qualify for a central role in a team, nor does it tell anything about quality. Expecting something grand from a bunch of rookies is simply dumb. Writing better blog posts or smart essays won't change anything. A rookie needs to follow the wrong path and fail with it. Good mentoring can speed things up, but you still need to let them fail. Team building is vastly underdeveloped in software engineering, it is not even a serious thing. Yes if you start out with a good guy that carries the newcomers, then you're golden. But that is mostly just luck.
jseliger 8y ago
“Move fast and break things,” they said. It turns out that’s a pretty bad idea when your business relies on a small number of large customers. Broken products tend to scare them off, which in turn tanks your business. There’s a lot to be said for building things that work, but “move slowly and steadily towards a goal” just doesn’t have the same ring. In reality, there’s a balance between moving fast and and moving slow. It’s difficult to communicate that balance because every type of product demands a different balance. I suppose that intuition comes from experience, which is a terrible answer for someone trying to learn. I'm guessing this continuum also depends on the nature of the business one is in. In a fast-moving, consumer-oriented world like social networking, "move fast and break things" is probably very good advice. In a slower-moving, enterprise-oriented world, "move slowly and steadily without breaking things" is probably the right orientation. Those kinds of companies probably die in fast-moving consumer land. A while ago I read an article about the programming teams that work on the space station, on satellites, and that sort of thing. Alas, I cannot find the link right now, but the gist was that those teams move very slowly and work very hard towards correctness. Any bug found is generalized and their entire code base scoured for similar bugs. That's a different environment than social networking.
- hashkb 8y ago
  > In a fast-moving, consumer-oriented world like social networking, "move fast and break things" is probably very good advice. I'm not so sure about that particular example, given the privacy implications. (And beyond...) If your mantra specifically encourages breaking things in production, you've chosen to be flippant instead of thoughtful; or you actually want your product to be broken. It's likely that a more thoughtful mantra could result in better outcomes without discouraging urgency and innovation.
- tetha 8y ago
  Most enterprise customers I've dealt with don't even care about things breaking that much, they care about control. That's the much bigger thing. Things break. Let's just make sure we don't break things while our customer is dumping silly amounts of money into an EU-wide advertisement campaign. Or we tend to freeze updates during christmas, due to the christmas business of our customers. With those guys, it's fine to execute risky changes on productive systems, as long as those changes are tested, communicated, scheduled to a non-critical time of the customers and include backoff-plans. But yes, this naturally slows things down, or makes systems more complex and expensive due to redundancy and staged rollouts.
lmilcin 8y ago
I find this a result of pervasive command and control structures which inevitably lead to low trust environment. Something breaks, so the manager naturally thinks he must add a control that will prevent the person or a from ever being able to do it. Very soon nobody can do anything without approvals by managers so far removed from actual decision that they don't even know the names of components being subject to change. The change then must travel from the only people that know what the change is but are absolutely barred from ever seeing production environment to people that being so focused on operations don't have time to actually understand what the change is. Once you are being treated as untrustworthy it will show in the rest of your work. Why code correctly from the start when you are not deemed trustworthy to do it correctly and there is lengthy testing and approval process to make sure you are not going to bomb your company? The correct step is to recognize, that introducing more controls can very soon lead to low trust between employees which is self-fulfilling prophecy. When you are not allowed to do something because you are not deemed trustworthy to do something, then soon you have no experience and then this serves as a proof that the initial decision was correct.
- thinkingkong 8y ago
  Trust is important but if you trust people who arent capable of doing a job that trust is misplaced.
- tensor 8y ago
  On the flip side, it seems that many young devs think that they should be allowed to do anything they want, without oversight whatsoever. I don't know of any other industry where junior people have this sort of arrogance. Usually people recognize they are junior and work to both learn from and gain the trust of their seniors. In development land, the attitude is "seniors are old, useless, and outdated" and any form of oversight at all "removes trust." E.g. A young dev suggests building their own database. After being told politely that this is an enormous task full of seriously hard problems and that it's not feasible to try, they yell about their "ideas being shot down" and that the work environment "lacks psychological safety."
  - lmilcin 8y ago
hashrate 8y ago
> all of these developers lacked experience. This is nothing knew unfortunately. Working for large corporations I've often notice this : Either the devs on the project don't have the required experience and will deliver poor quality code creating immediate "legacy" , or either it'll be outsourced to another company with no governance plan to review the code quality creating a "black box". Because there is very little standard and regulation in software industry about competence, you can have two devs claiming they are "Software Developer" with one able to deliver a full project by himself the and the other not capable of creating a simple web page in HTML. Hence , I've noticed very little company understand the importance of "tech" , "IS Governance" and investing in their staff in general, they often see "tech" as a constraint like : "We have to do a website for our customers , otherwise we'll loose market share" and not "We have to seize the opportunity to create a platform for our customers, it will drive growth massively" This is sad , but this is pretty much the standard in corporation these days.
jahewson 8y ago
If you’re an amateur or a hobbyist then these excuses are all just fine. Stuff is hard. But a professional should not being doing this sort of thing - we’re literally talking about copying something labelled do not do this. The moment you ignore that warning is the moment you’ve forsaken your professional development. This problem only compounds. That said, a professional can still cut corners as and when appropriate (and clearly label and explain such things) but that’s not what’s happening here, is it? The advice I would give in this situation is simple: stop and think about what you’re doing. If you can’t do that, you’re not a professional, you certainly shouldn’t call yourself an engineer and without that skill you stand no chance of improving.
syndacks 8y ago
Stephen Mann: the subscribe button on your site is broken (at least on mobile web)
- pattrn 8y ago
  It should be a link to an RSS feed. I'll change it to link to my newsletter signup form. Thanks!
viburnum 8y ago
Naw, you can totally write garbage code and still get acquired for $100 million. Happens all the time.
hyporthogon 8y ago
Copypasta from other codebases happen to work better in non-production environments because the input space is smaller in virtually any non-prod runtime space than in prod. This makes for a larger intersection of inputs to the copypasta that produce outputs that are expected by the blogpost/googling developer and inputs to the copypasta that produce outputs that are generated at non-prod runtime. When you don't quickly see what's semantically wrong with the code (the problem mismatch) because you pasted it and it just worked (maybe with a few small tweaks), then you quickly forget about it and black-box the snippet in your head. Then, when input space gets real-world and things break, because the copypasta never had much cognitive gravity (because you didn't think much about it), it doesn't attract log dump elements in your log-scanning brain as much as stuff you had to think through yourself does. I guess you could automate some static checks for this by some massive trawl of snippets from a few websites (at least popular SO posts in the same language?) and finding verbatim matches in your codebase. Sounds massively impractical to do this at scale though. Does anyone know tools that do this?
z3t4 8y ago
When you publish some code you can be pretty sure someone will use it in production, whether you advice against it or not.
i000 8y ago
So a team of inexperienced developers made a system that mostly worked with some hiccups, could be debuged by an expert in 1 day, and fixed on a tight launch timeline. Looks like a success story!
- akavel 8y ago
  Plus they apparently also managed to validate existence & viability of a market.
- pattrn 8y ago
  It was definitely a success story. The company, their customers, and my company all benefited. It wasn't all rainbows (the story is a bit longer and more complicated), but it worked out well for everyone in the end.
- avinium 8y ago
  If a consultant could jet in, diagnose and fix up some issues in a couple of days, that means the team did a pretty awesome job to begin with. A truly inexperienced team would have created a spaghetti monstrosity that was practically unsalvageable.
platform 8y ago
Doesn't this highlight a continuous issue,that folks in our trade, do not have any meaningful and up-to-date professional certification?
- walshemj 8y ago
  Most professions don't eg when you have passed your bar exams taken silk or what have you - you do have cpd but these are self certified.
- alecbenzer 8y ago
  Is there a clear indication that certification helps in other fields? (Admittedly this seems like it'd be very hard to verify, since most fields either legally require certification or don't have it, and there's probably tons of other variables that would get in the way too.)
lisper 8y ago
> At the end of the day, the best way to get more money is to have other offers. Actually, the best way to get more money is to already have a job that you are willing to leave.
yeukhon 8y ago
> They hired the first developer, and he vetted the second developer, and so on until they had a development team This is hard for a startup in their early stage. Not sure how mature the company was, but let's assume it was young enough to have this very problem. Direct team hire is a double-edged sword. It works well when your team is very competent and good at weeding out false positive. Basically, when you have a bunch of senior people with maybe a couple junior and mid-level teammates. A normal distribution is actually perfect for a team. Having a direct contact with your future teammate allows you to know the person ahead of time, especially for a very team. Furthermore, direct hire means you are hiring not just a generalist. Your SWE may not know what to ask a network engineer, even from a coding point of view. The downside is what happened in the story. The way I'd like to go about it is to make sure we have two interviewers per round, even from the same team would be far better than one person vetting.
memset 8y ago
This is interesting - many of the comments here are correctly asking "what if you don't know an expert to help vet the engineers and the system?" I feel the same way when trying to find dentists or accountants. I simply don't know how to know who will perform completely! So, here is my pitch: if you don't have technical expertise, and you realize that you need someone to help understand just how the heck to get started with technology, what to build, and determine if you even need engineers, then hire me! I'll help! https://www.jaygoel.com/consulting/ https://www.jaygoel.com/consulting/
honkycat 8y ago
> If you read this far hoping for an answer, then I’m sorry: I don’t have a simple one. This is a difficult problem to solve. The solution is too large for a single blog post, changes every day, and differs subtly for every project. Crazy idea, what if we found one of these people who writes these blog posts, an especially well vetted and knowledgeable one. Maybe even one who writes the frameworks and software these people are using. cough https://pragprog.com/book/phoenix14/programming-phoenix-1-4 https://pragprog.com/book/phoenix14/programming-phoenix-1-4 cough What if we took this person, and paid them money to write a bunch of blog posts about on topic, in a way that it read in a linear fashion? We could even maybe hire someone to help them, to proof read and make helpful suggestions. We could give the knowledgeable developer a year or so to write the blog posts, then print out the blog posts, bind them together with glue, and SELL IT! People could pay money for the well researched, well edited series of blog posts about a specific topic written by the expert. I think I'll name this blog-printed-out-onto-paper... a honkycat. After it's inventor. --- Sarcasm aside, I find people's insistence on googling and sifting through half-thought-out blog posts infuriating. I am a HUGE proponent of a well written book about a software product. Compare "Programming Elixir" to their free online book and documentation. It's night and day. And I can always look in the same place for reference, the book. Written by Dave Thomas. A book has: ( At least ) one author, one editor, and a publisher. All of these people work hundreds of hours to produce a useful manuscript. A blog post is something a script kiddie shits out because they're trying to get their first programming job. Yet people say the same thing: "A book?!? I can just find it on the internet" and do the same thing: "I'm sure I'm smart enough to just start writing this software without any prior experience in the language, or forethought" and make the same mistakes over and over. Skip the blog. Read a book.
- aidenn0 8y ago
  Finding a good book is not particularly easier than finding a good set of blog posts. While the quality of books is on average significantly better, finding books where the material was written in the past year is much harder. Particulary in the trendy JS world, where there might not be a good book for the stack you are using, and even when there is, you ask questions online and get responses that are roughly "It hasn't worked like that for 6 months, why are you trying to do it this way?" You give the example of the Phoenix book; I don't know if that's what this team was using. If it was not, the fact that there is a good book for that stack is not particularly relevant. If they were using Phoenix, then the fact that there is not a good book for most stacks would explain why they wouldn't consider looking for one for the stack they are using.
janwillemb 8y ago
> Before I go any further, I’ll continue my story. It seems to me that that sentence has no significance at all and can be removed safely ;)
- pattrn 8y ago
  Point taken, and sentence removed. Thank you!
exabrial 8y ago
"Move fast and break things" only works if your company has a stranglehold on a monopoly and your user base is used to bugs in production. Terrible advice for just about everyone else. You want your users to have the best experience possible, rather than do your QA process for you. EDIT: I should say my point is: "moving fast" and "testing things" are goals that aren't contradictory. You can and should do both.
ddebernardy 8y ago
> If you read this far hoping for an answer, then I’m sorry: I don’t have a simple one. This is a difficult problem to solve. The solution is too large for a single blog post, changes every day, and differs subtly for every project. [Cough] - Hire a few more (expensive) experienced developers? It might not be a silver bullet, but it goes a long way towards avoiding the situation described in the blog post. Also, get help from a trusted technical friend to hire the initial one. FWIW I see something similar with almost every client I get in my own neck of the wood. Growth is flat. Turns out nobody is worrying about whether they're building a product anyone wants - they're doing it on paper but nobody is actually talking to end-users. Marketing is under-delivering. Turns out it's all juniors worrying about bringing traffic without wondering if it's the right type of traffic; or they're not measuring anything, plain and simple. Sales aren't selling enough. Turns out it's all juniors who aren't selective or systematic enough; or sales are doubling as pseudo-project managers and/or pseudo support-engineers because nobody is managing operations. Projects take forever to deliver. Turns out the project "manager" holds nobody accountable to deliver any specific task let alone by a certain deadline. The list could go on, and on, and on.
arendtio 8y ago
Some time ago I was writing some software in Go in my spare time (just for fun and I could spend my time how I liked it). As I wasn't completely sure what it should look like in the end, so I thought it would be reasonable to skip writing tests in the beginning and just write the code and add tests afterward where they would be necessary. Fast forward I had to write the whole code twice. The second time I wasn't completely sure how it should look like either, but I put a much greater emphasis on interfaces and modularity. That way I could guarantee that parts of the program were actually doing what they were supposed to do, which made the bug hunt so much more effective. So sometimes the longer way is actually the faster one and experience matters when it comes to recognizing such situations.
humzakh 8y ago
"most fast and break things" only if you have an idea about the implications of your actions and you are confident you will be able to fix the broken parts which in production without losing revenue or customers.
pytyper2 8y ago
So, what shouldn't i do in production? This article was unhelpful.
- pknopf 8y ago
  The snippet of code copied from a blog into production software was only a small detail. You obviously missed the entire point of the article.
stcredzero 8y ago
But what other option did these developers have? They had to deliver something, and they had never released a production application before. So they did what any sensible person would try to do, and they learned on the job. And yet, so many people push back on my complaints about the shallowness of the education many recent CS grads are getting. No, it's not an irrational fear of the next generation. It's an awareness that some large fraction of recent grads sporting high GPAs don't have enough basic background knowledge to know what they don't know and keep themselves out of trouble. What would we think if companies let non-engineer/architect managers put together teams of completely mechanical engineers and architects, who then went on to build faulty bridges and machines that started to call apart? People would be calling foul. Given the societal, privacy, and financial consequences, it's way past time for the programming field to grow up!
- user5994461 8y ago
  It's the same in every industry and every domain. A new grade needs a few years of practice before he can contribute well to any real project. However companies don't always have dummy projects for years, just to practice and drop afterwards.
  - stcredzero 8y ago
    It's the same in every industry and every domain. No. In most industries, groups of novices don't get to do something that gets pushed out to production.
mmt 8y ago
> helped mentor their developers This is typically missing from startup culture. Anecdotally, mentorship improves everyone involved, but I don't know if there's data out there showing if engineering mentorship within a startup improves its chances of success (or is otherwise beneficial [1]). To some extent, groups like YC participants and alumni end up being mentors to each other, but I doubt that extends beyond the founders. Might the OP have an opinion on something like a rent-a-mentor program/startup? [1] Not that certain benefits, like productivity, would be enough to motivate a change, as evidence by the open plan office situation
anon1253 8y ago
Even for an experienced programmer this stuff happens, and it's the most frustrating experience I've ever had. It's the stuff that happens when management and C-level just "wants stuff" to be buzzword compliant. And, every day/week/month the requirements change. This weeks feature X has "top priority" and then amnesia kicks in and now feature Y is the next best thing. As a programmer-now-manager I try to shield people from this, trying to find the gems among the rubble of business meetings. But it's tricky, so this really resonated with me: "I have an overwhelming empathy for developers in this position. They have more information than they will ever need, but it’s completely disorganized. It’s like trying to build a ten piece puzzle, except you have to find the ten pieces within a pile of 10,000,000,000 pieces, all of which are square, and you don’t know what it’s supposed to look like at the end. Good luck." Even with a team of highly experienced developers things quickly turn to a mess if it's not perfectly clear what to work on. I don't mean things like "button should go 1px north to align with header". That's trivial. In fact, I would go so far as to say: most production-ready programming is trivial. Trivial in the sense that someone, somewhere, solved it in a way that's correct. But, when it is not /crystal/ clear what the requirements are, what the business case is, or what the edge-cases look like: oh my. I've been in the incredibly unlucky position of only being in settings like this. Start-ups, academia, R&D positions. And, as a result even with the best intentions and the best teams you'll still produce production-quality garbage. Mostly because incentives were misaligned and there was no coherent vision. Agile story-driven development only works when there is leadership that has the guts to say "no", and when there is something well defined to work on. Sometimes I yearn for some CRUD webstore ... at least then the things are knowable. Anyway, just my rant. I'll probably go the meme-way of becoming a plumber/welder/etc at some point ;-)
Double_Cast 8y ago
> There’s a lot to be said for building things that work, but “move slowly and steadily towards a goal” just doesn’t have the same ring. I'm reminded of one of my math professors. When a student asked him "How do I do this problem?", the prof would always respond "Carefully and correctly". And then he'd laugh at his own joke and give the student a real answer. I thought it was funny. And I liked the alliteration.
Ftuuky 8y ago
The solution? Blockchain. /s
sattoshi 8y ago
This post was completely not what I expected it to be. The "do not use in production" is the programming equivalent of taking a strong stance on an issue but then appending "but hey, I don't know" or "don't take my word for it". They completely defeat the purpose of the entire statement and in the production comment's case, the code samples. Let's ask a different question, what am I supposed to use in production? I had an email discussion with some blogger who wrote about a very narrow problem which I needed to solve. Their code samples worked, but poorly. The entire thing was a dumpster fire but it solved the problem and were the only one I could find after days of Googling and Stackoverflowing. I did not care about their "do not use in production" comment. I needed their solution in my production system. What was I supposed to do? Sorry for ranting, but I believe it is irresponsible to put code out there which shouldn't be used except for when it is the point.
_hardwaregeek 8y ago
This something I don't understand about certain startup founders. If you're starting a company, you should have at least one absolute monster of a developer. Someone who is capable, communicative, and with a deep pool of knowledge. Someone who can attract, hire, and lead other good developers. Not your buddy Steve who made an iPhone app or two. Not that developer who has some PHP experience and built a React app once. A real, terrifyingly good developer. The aphorism "first class people hire first class people; second class people hire third class people" comes to mind. Granted, I'm a pretty inexperienced developer, but I see companies being founded by people who are even less competent and less knowledgeable than me. And sure, finding a competent person is hard. But it's like the movie Seven Samurai. Can't afford to pay a samurai? Find a hungry one. There are plenty of developers who are young, but still good. And if you can't even do that, then take any and all money you have, and pool it to find one or two (or seven) developers who can find other competent people.
- eropple 8y ago
  > There are plenty of developers who are young, but still good. I was young, and I was good. (Now I am moderately young, and I'm very good.) But I am a better talker than I was a developer. Only recently have I really become as good a developer as I am a talker. How are those startup founders supposed to evaluate me?
exmicrosoldier 8y ago
This was a problem with people. Specifically, people who force people to commit to a known deadline on something they dont know how to do. Someone got promoted for delivering business value, and they were smart, they left before it all blew up. Get 'er done without context and fire them for failure is bad management.
altitudinous 8y ago
Many developers are commenting in here trying to fix a problem. I am a developer but I now run a business. From a business point of view, there is no problem here. Code has been delivered and it is running in production successfully. It might not have arrived at that point in the most efficient manner, but the point is that the business has executed successfully here. There is no issue. This is a lot further than most businesses get. If you see a problem to be fixed here then you are looking at a small picture of the business.
- alecbenzer 8y ago
  Are you discounting that technical debt is a thing/matters? What gets delivered isn't just a set of features, it's a piece of software, which implements those features, but might have a whole bunch of baggage attached which can make it really hard (or easy) to change those features or add new features in the future, or to allow the software to adapt to changing external conditions. I don't think this is specific to software. E.g., I'd imagine it'd be possible to build a building a lot faster if you didn't need to worry about it not collapsing in five years. --- Though to be fair, the mere presence of technical debt on its own is not a problem; there is some amount of technical debt that's appropriate, and it's hard to know exactly where that is. You certainly can't get by just ignoring it completely.
- occams_chainsaw 8y ago
  A rotting foundation eventually catches up with you. Things don't have to be perfect, or even particularly great... but when things are working by accident rather than by design...
- draw_down 8y ago
  Sure, the problem comes later. And it will be very expensive to fix.
- tonyedgecombe 8y ago
  To some extent that was just luck, they happened to hire a consultant who happened to be able to resolve the visible problems. Hiring junior developers because they are cheap and keeping your fingers crossed isn't going to work most of the time. Not shipping will sink any business if it goes on long enough.
brian-armstrong 8y ago
Another angle to look at this through is that they hired a team in spite of only having business, non-technical minded people. It all worked out in the end when they added consulting to clear up production issues near launch. From a business perspective this is surely a win in all ways, even if not as efficient as it could be. Not every business is a Google.
jarym 8y ago
I dunno.. whenever I read ‘don’t do this in production’ I spend ages trying to really understand why - only when I feel I’ve got a real handle on what the pitfalls are would I go forward. I’ve done that ever since I started programming over 20 years ago and I still do it now. Sometimes it takes me longer to code a solution but I do feel I spend less time figuring out why stuff breaks. It’s not just experience. It’s about hiring curious and diligent people.
kristianp 8y ago
"... fix issues as I went along. Without unit tests or automated deployments, this took almost a year." One of the first things he should have done was to get them Joel-test compliant, otherwise he's wasting the money they pay him.
daveheq 8y ago
"The only clear issue was lack of experience" "In this case, however, it had a nefarious twist: all of these developers lacked experience." Am I missing the twist here? It was already mentioned as the primary problem.
- macns 8y ago
  try re-reading this with "all" in bold
- pattrn 8y ago
  Good catch! macns is correct: "all" should be bolded.
daveheq 8y ago
Copying and pasting internet code into production that says “Don’t do this in production“ is not a lack of experience problem, it's a lack-of-reading problem or a rushing-things problem; the latter is typically caused by management putting deployment dates on features without understanding the technical cost.
senderista 8y ago
MSDN used to make their code samples as simple as possible for educational reasons, until they realized everyone was just pasting them directly into production applications. So they started adding error handling, etc.

696 points by pattrn 8y ago

This blog post describes my motivation for writing a blog. It's mostly a written memoire for myself to remember in detail what motivated me to start writing, but I thought HN readers might have some good feedback, so here it is. Would love to hear any stories you have of production fiascos.

deleted 8y ago
[deleted]

quickben 8y ago

Summary: The author clickbaits the title with "don't do X". Proceeds with a life story of his struggles ommiting what X is.

pattrn 8y ago
Sorry about that. I guess I didn't really think much about the title sounding like clickbait. The phrase is what motivated me to write the post, so I used it as the title. Perhaps I should put quotes around it?
- turdnagel 8y ago
  Quotes would probably be more "accurate," but I don't think they'd quell this person's (unwarranted) complaints. I believe sharing your story was valuable and I look forward to reading more of your blog.
  - pattrn 8y ago
    Updated. Thanks!
- quickben 8y ago
  It would help, thank you for that. I was expecting an actual example, which what was my problem with the title. It was more subtle than that so I read the whole thing but kept wanting to know what the damn line was. The downvotes that I am receiving actually reflects the same conceptual problem that you are outlining in the article. People are nice. They don't call out bullshit early on, and companies end up with nice-but-incompetent that can't deliver quality (but apparently can downvote on HN thinking they are doing right in life following the echo chamber). This doesn't plague many other licensed industries. Either you reach a level to get your license (and lose it if you are incompetent), or won't be doing your professional job at all. There is no "nice" in law, medicine, architecture, etc, where professionalism is concerned. It's conflicting. Should we be altruistic, promote teaching, but also promote being not nice? Or shall we be selfish, keeping quiet and ensuring our personal job-security because we can fix messes afterward. Overall, I think your blog is in the right general direction. Best of luck.
lmkg 8y ago
X in this case is copy-pasting code into production from blogs that explicitly say not to copy-paste their code into production. I do not feel it was omitted from the article, it was stated rather directly more than once. Perhaps you were expecting a description of why that particular code was unsuitable for production? The blog is about the process of writing code, not the code itself.
jasonlotito 8y ago
> The author clickbaits the title with "don't do X". It's not. > Proceeds with a life story of his struggles ommiting what X is It does.
Walkman 8y ago
If you can't even read, why bother commenting?
- dang 8y ago
  Please don't reply to a bad comment with a worse one. Personal attacks in particular will get you banned here. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

tlb 8y ago

There's no reasonable way to write code suitable for "production" in general, because there are so many conflicting requirements. Just among internet services, components need different kinds of robustness for content websites, communication, e-commerce, email, banking, social networks, internal corporate services, and more. There do exist components suitable for most environments, like Postgres, but usually at the cost of 10x complexity over what'd be needed for the pure functionality. That engineering is worth it for things like databases that get used everywhere, but not for any custom component.

ballenf 8y ago

There's a great book Apprenticeship Patterns which gives real answers and advice to developers. It addresses the situation described in a few different ways and offers different strategies. Most of them boil down to how to pick a team where you will learn the most quickly while not drowning. https://www.amazon.com/Apprenticeship-Patterns-Guidance-Aspiring-Craftsman/dp/0596518382 https://www.amazon.com/Apprenticeship-Patterns-Guidance-Aspi...

pattrn 8y ago
Thanks for the recommendation. I'll check it out.
mazatta 8y ago
Of all the books I recommend for junior/intermediate developers, this is the one I demand they read. I wish the the free online version was still available, but it doesn't appear to have survived O'Reilly's switch to the subscription model.

peterwwillis 8y ago

Bugs will always happen in production. The important part is not developing everything correctly. "Move fast and break stuff" is a decent philosophy, as long as you can learn from mistakes and fix recurring problems. When a bug in your application presents itself in production, something very specific should happen: 1) identify the symptoms, 2) isolate the causes, 3) address the issue in production, 4) perform a postmortem to identify the next steps, 5) fix the problem's causes throughout the entire supply chain so it cannot happen again. A lot of organizations get stuck only doing steps 1-3. That's not good. Operational processes are just as important, if not more so, than development processes. They're what ensure that when you find inefficiency or bugs, that they not only get fixed, but that they never happen again. This means actually making policy changes, training changes, etc in addition to just fixing code. This is the heart of lean manufacturing that supposedly influenced a lot of modern software development methodology. But some people may have forgotten that the lean manufacturing chain doesn't end at development.

nordsieck 8y ago
> 4) perform a postmortem to identify the next steps It takes an effective management team to make this step work. If the postmortem isn't blameless, this stem becomes completely ineffective as employees (not unreasonably) prioritize their own careers over the health of the company.
- sidlls 8y ago
  Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task. I think there's a middle ground between never holding anyone accountable and the desperate situation you describe.
  - nordsieck 8y ago
    > Sometimes there is blame, though. Sometimes a root cause is simply that a person or group was not competent in their execution of a task. It is true that some people are not good at their jobs. It's also true that the only way to discover that people are bad at their jobs is their performance. However, I think the process approach is applicable to a much larger domain than many people give it credit. One of the greatest software development quotes I've read comes from DJB [0]: > For many years I have been systematically identifying error-prone programming habits—by reviewing the literature, analyzing other people’s mistakes, and analyzing my own mistakes—and redesigning my programming environment to eliminate those habits. The difference in security bugs found in qmail and sendmail speaks for itself. This is an example of the superiority of process: it shows that what some people consider a "root cause" is actually only a proximate cause - that the root cause is the programming environment that allows the bugs to occur. [0] https://cr.yp.to/qmail/qmailsec-20071101.pdf https://cr.yp.to/qmail/qmailsec-20071101.pdf

ohiovr 8y ago

I usually leave a URL for stack overflow answers or other advice as a comment in what I'm working on. I know some areas of my code are more sensitive than others. It is good to have an understanding of what you are doing of course. Sometimes a stackoverflow link is removed simply because the mode of thinking is wide spread in the application and I no longer need the note to it. Might be worth searching for my links in the code when it is time for release incase there is any funny business. Stackoverflow usually tells you if you "shouldn't do it in production".

crazygringo 8y ago

Can someone give an example of a blog post that gives example code yet warns "not to do it in production"? I've programmed and read about programming for decades and can't recall ever seeing anything like this. It's hard for me to take this article seriously without even a single example -- I can't tell if this is a strawman or not. Is it about code that isn't thread-safe? That relies on undefined behavior? That doesn't scale? That has race conditions? "In dev" and "in prod" can mean so many different things (is it about scale? or running on different OS's? or running unattended? or running on different hardware? or running compiled?), out of any particular context they're essentially meaningless, and in case particular case, it would seem to be the particulars that matter...

thinkingemote 8y ago
First result from DDG. https://blogs.msdn.microsoft.com/canberrapfe/2013/02/16/powershell-web-access-very-quick-start-yep-powershell-in-your-web-browser/ https://blogs.msdn.microsoft.com/canberrapfe/2013/02/16/powe... the thing in that example is that a thing is not properly configured but good enough to get started with in development. Second result: https://stackoverflow.com/questions/1475297/phps-white-screen-of-death https://stackoverflow.com/questions/1475297/phps-white-scree... describes how to put debugging statements in the code which is good for local development but which shouldnt be kept for production.
ISL 8y ago
I searched for "don't do this in production". This article was the first hit, and several blog posts with example code followed.
reificator 8y ago
I've seen it dozens of times, and dozens more read articles where they should have. The entire Supercharged playlist on Chrome's Youtube channel does this, for instance.
bartread 8y ago
> It's hard for me to take this article seriously without even a single example -- I can't tell if this is a strawman or not. I felt similarly: it just rang slightly false to me. For one thing, a code review wouldn't be the first solution I'd reach for when diagnosing the kind of problems described (performance issues, memory leaks, random crashes). There's plenty of tooling available for most platforms that will generally get you to a solution more quickly and reliably (granted, the post mentions only spending half a day), and often highlight problems you might easily miss in a code review. I'm not saying code reviews aren't useful, but that measurement of real-life behaviour can be more useful in these circumstances. Still, the point about not copy-pasting code directly from the internet is well made: been bitten enough times over the years that I've become extremely wary of it, even - and perhaps especially - when in a hurry.

angersock 8y ago

"If it's stupid but it works, it isn't stupid". I liked the post, but I'm not sure that I agree.

detaro 8y ago
To take one of the classics of bad copy-past examples: "If the 'use-PHP-with-MySQL' example you copied is stupid, works, and also causes your production database to be hacked due to SQL-injection, it probably was stupid" The problem is to tell the difference between "ugly stupid" and "dangerous stupid"
golergka 8y ago
The problem is with defining "works". If your code "works" with ten test entries in a DB when you test the happy path, it doesn't mean that it really works in production, with real load and real attack attempts on it.
nordsieck 8y ago
"works" is a nuanced word. The saying can be wonderfully pithy or terribly destructive depending on people's understanding of the word.
krallja 8y ago
“If it’s stupid but it works, it’s still stupid and you got lucky.”

dsr_ 8y ago

"Move fast and break stuff" is a decent philosophy if and only if the consequences of breaking stuff are survivable. If breaking stuff means that your website looks weird, that's survivable. If breaking stuff means that performance sucks for a while, that's survivable. If breaking stuff causes unavailability during a critical period of end-user demand, a few incidents might be survivable. If breaking stuff causes your company to have a terrible reputation for privacy, security, or competency, that might not be survivable. If breaking stuff causes your company to divulge financial information, that might not be survivable. If breaking stuff ends up costing your customers any significant amount of money, that probably will not be survivable.

credit_guy 8y ago
If breaking stuff causes a pedestrian to be killed because "they came from the shadows"[1], that appears to be perfectly survivable... /s [1] https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg
- ttty 8y ago
  Cars killed people... they didn't stop creating cars. Nor would be a good idea.
  - abraham_lincoln 8y ago
    No, we got jaywalking laws and our rights reduced.
    - anticensor 8y ago
      We got safer cars at the same time. It is a double-edged sword.
  - TheCoelacanth 8y ago
    Killing one person per 1 million miles driven like Uber's self-driving cars have done is drastically different from killing one person per 80 million miles like human drivers do.

wiz21c 8y ago

One thing I see amongst many of developpers, is that they write code like it will work in production. OTOH, I tend to write code that will crash nicely : good error reporting, ability to continue after a crash, write code that handle only the data it needs, make sure queries address the smallest possible set of data, log a lot. Interestingly, that's a lot of code I have to write anyway because it helps me to write my programs (I mostly don't use debuggers).

pattrn 8y ago
Completely agree. One of the first problems I found was that the server wasn't running via a process manager, so when it crashed it never rebooted. Simple enough to solve, but non-obvious if you've never run into it before. It's a checkmark for anyone who has hosted any application before, but it's one of many important ones.
- orev 8y ago
  I can’t think of any well-known professional projects that operate like this. IMO, automatically restarting after a crash is a terrible crutch that ensures crashing issues never get fixed at the priority level they deserve. Can you imagine if Apache, postgres, postfix, haproxy, etc. operated this way? If you have so many crashing issues that you need something like this, you have serious problems.
  - zzzcpan 8y ago
    I encourage you to read Joe Armstrong's dissertation of how a simple idea of restarting on crash can make you an impossibly reliable system.
  - barrkel 8y ago
    Stateless servers can and should recover automatically after a crash. Erlang has built a whole philosophy of reliable software construction based on the idea of letting errors crash a process and be restarted by a supervisor - systems with higher reliability requirements than most database servers, e.g. upgrades with zero downtime.

wpietri 8y ago

I want to underline this bit: > they hired a team of developers without having a technical person on staff to vet them For any non-technical entrepreneurs reading this: please don't do this. If you aren't competent to hire developers, please borrow or rent a few trusted technical experts and use them as a hiring committee. Otherwise you are less likely to hire the best technologist, and more likely to get the most glib and appealing technologist. Over the years I have met far too many smooth-talking consultants and would-be employees. And I've seen them cause enormous garbage fires when managers hire beyond their ability to evaluate. The size of these garbage fires are a direct result of what got them hired: if someone is good at telling managers what they want to hear, they can go a very long time saying, "Yup, we're building it and it will be great! Just you wait and see!" And then, no matter who you hire, demand a "ship early and often" schedule. Ship to internal users. Ship to alpha users. Ship to external testers. Heck, ship to just person for just one narrow use case. The earlier you can start seeing real-world success or failure, the earlier you can course correct.

brianpan 8y ago
And how do you find the trusted technical experts? By using your trusted technical expert expert?
- hyperpape 8y ago
  By not going in to a business that requires development unless you are one, or know one. I wouldn't assume I could start a restaurant or run a medical practice, why do so many people with no technical background assume that they can start a business centering around software?
  - jasonlotito 8y ago
    Because lots of people that haven't run a restaurant have successfully started a restaurant, and people who were in the restaurant business started a restaurant, but failed to keep it going. History is littered with people who ignored your rule here and have been successful. I guess we can flip that around: why do so many people with no business background assume that they can start a business?

paulie_a 8y ago

I work with a consultant that dealt with emergency situations that had the motto, "be brave, be bold, deploy" In a different situation three of us figured out a solution for a critical situation. We voted on deploying with the thought "well we can't guarantee it will work, but it won't make things worse"

walshemj 8y ago
They have not recently been working at the TSB :-) The TSB a medium size UK bank had a very well publicised disaster - when doing a one way no roll back possible switch from an legacy IBM system to a Java based one

Yokohiii 8y ago

Building a good team is hard. Yet companies think they can hire an engineer which had a senior position somewhere and be fine. Senior or 10y experience can mean literally nothing, it doesn't qualify for a central role in a team, nor does it tell anything about quality. Expecting something grand from a bunch of rookies is simply dumb. Writing better blog posts or smart essays won't change anything. A rookie needs to follow the wrong path and fail with it. Good mentoring can speed things up, but you still need to let them fail. Team building is vastly underdeveloped in software engineering, it is not even a serious thing. Yes if you start out with a good guy that carries the newcomers, then you're golden. But that is mostly just luck.

jseliger 8y ago

“Move fast and break things,” they said. It turns out that’s a pretty bad idea when your business relies on a small number of large customers. Broken products tend to scare them off, which in turn tanks your business. There’s a lot to be said for building things that work, but “move slowly and steadily towards a goal” just doesn’t have the same ring. In reality, there’s a balance between moving fast and and moving slow. It’s difficult to communicate that balance because every type of product demands a different balance. I suppose that intuition comes from experience, which is a terrible answer for someone trying to learn. I'm guessing this continuum also depends on the nature of the business one is in. In a fast-moving, consumer-oriented world like social networking, "move fast and break things" is probably very good advice. In a slower-moving, enterprise-oriented world, "move slowly and steadily without breaking things" is probably the right orientation. Those kinds of companies probably die in fast-moving consumer land. A while ago I read an article about the programming teams that work on the space station, on satellites, and that sort of thing. Alas, I cannot find the link right now, but the gist was that those teams move very slowly and work very hard towards correctness. Any bug found is generalized and their entire code base scoured for similar bugs. That's a different environment than social networking.

hashkb 8y ago
> In a fast-moving, consumer-oriented world like social networking, "move fast and break things" is probably very good advice. I'm not so sure about that particular example, given the privacy implications. (And beyond...) If your mantra specifically encourages breaking things in production, you've chosen to be flippant instead of thoughtful; or you actually want your product to be broken. It's likely that a more thoughtful mantra could result in better outcomes without discouraging urgency and innovation.
tetha 8y ago
Most enterprise customers I've dealt with don't even care about things breaking that much, they care about control. That's the much bigger thing. Things break. Let's just make sure we don't break things while our customer is dumping silly amounts of money into an EU-wide advertisement campaign. Or we tend to freeze updates during christmas, due to the christmas business of our customers. With those guys, it's fine to execute risky changes on productive systems, as long as those changes are tested, communicated, scheduled to a non-critical time of the customers and include backoff-plans. But yes, this naturally slows things down, or makes systems more complex and expensive due to redundancy and staged rollouts.

lmilcin 8y ago

I find this a result of pervasive command and control structures which inevitably lead to low trust environment. Something breaks, so the manager naturally thinks he must add a control that will prevent the person or a from ever being able to do it. Very soon nobody can do anything without approvals by managers so far removed from actual decision that they don't even know the names of components being subject to change. The change then must travel from the only people that know what the change is but are absolutely barred from ever seeing production environment to people that being so focused on operations don't have time to actually understand what the change is. Once you are being treated as untrustworthy it will show in the rest of your work. Why code correctly from the start when you are not deemed trustworthy to do it correctly and there is lengthy testing and approval process to make sure you are not going to bomb your company? The correct step is to recognize, that introducing more controls can very soon lead to low trust between employees which is self-fulfilling prophecy. When you are not allowed to do something because you are not deemed trustworthy to do something, then soon you have no experience and then this serves as a proof that the initial decision was correct.

thinkingkong 8y ago
Trust is important but if you trust people who arent capable of doing a job that trust is misplaced.
tensor 8y ago
On the flip side, it seems that many young devs think that they should be allowed to do anything they want, without oversight whatsoever. I don't know of any other industry where junior people have this sort of arrogance. Usually people recognize they are junior and work to both learn from and gain the trust of their seniors. In development land, the attitude is "seniors are old, useless, and outdated" and any form of oversight at all "removes trust." E.g. A young dev suggests building their own database. After being told politely that this is an enormous task full of seriously hard problems and that it's not feasible to try, they yell about their "ideas being shot down" and that the work environment "lacks psychological safety."
- lmilcin 8y ago

hashrate 8y ago

> all of these developers lacked experience. This is nothing knew unfortunately. Working for large corporations I've often notice this : Either the devs on the project don't have the required experience and will deliver poor quality code creating immediate "legacy" , or either it'll be outsourced to another company with no governance plan to review the code quality creating a "black box". Because there is very little standard and regulation in software industry about competence, you can have two devs claiming they are "Software Developer" with one able to deliver a full project by himself the and the other not capable of creating a simple web page in HTML. Hence , I've noticed very little company understand the importance of "tech" , "IS Governance" and investing in their staff in general, they often see "tech" as a constraint like : "We have to do a website for our customers , otherwise we'll loose market share" and not "We have to seize the opportunity to create a platform for our customers, it will drive growth massively" This is sad , but this is pretty much the standard in corporation these days.

jahewson 8y ago

If you’re an amateur or a hobbyist then these excuses are all just fine. Stuff is hard. But a professional should not being doing this sort of thing - we’re literally talking about copying something labelled do not do this. The moment you ignore that warning is the moment you’ve forsaken your professional development. This problem only compounds. That said, a professional can still cut corners as and when appropriate (and clearly label and explain such things) but that’s not what’s happening here, is it? The advice I would give in this situation is simple: stop and think about what you’re doing. If you can’t do that, you’re not a professional, you certainly shouldn’t call yourself an engineer and without that skill you stand no chance of improving.

syndacks 8y ago

Stephen Mann: the subscribe button on your site is broken (at least on mobile web)

pattrn 8y ago
It should be a link to an RSS feed. I'll change it to link to my newsletter signup form. Thanks!

viburnum 8y ago

Naw, you can totally write garbage code and still get acquired for $100 million. Happens all the time.

hyporthogon 8y ago

Copypasta from other codebases happen to work better in non-production environments because the input space is smaller in virtually any non-prod runtime space than in prod. This makes for a larger intersection of inputs to the copypasta that produce outputs that are expected by the blogpost/googling developer and inputs to the copypasta that produce outputs that are generated at non-prod runtime. When you don't quickly see what's semantically wrong with the code (the problem mismatch) because you pasted it and it just worked (maybe with a few small tweaks), then you quickly forget about it and black-box the snippet in your head. Then, when input space gets real-world and things break, because the copypasta never had much cognitive gravity (because you didn't think much about it), it doesn't attract log dump elements in your log-scanning brain as much as stuff you had to think through yourself does. I guess you could automate some static checks for this by some massive trawl of snippets from a few websites (at least popular SO posts in the same language?) and finding verbatim matches in your codebase. Sounds massively impractical to do this at scale though. Does anyone know tools that do this?

z3t4 8y ago

When you publish some code you can be pretty sure someone will use it in production, whether you advice against it or not.

i000 8y ago

So a team of inexperienced developers made a system that mostly worked with some hiccups, could be debuged by an expert in 1 day, and fixed on a tight launch timeline. Looks like a success story!

akavel 8y ago
Plus they apparently also managed to validate existence & viability of a market.
pattrn 8y ago
It was definitely a success story. The company, their customers, and my company all benefited. It wasn't all rainbows (the story is a bit longer and more complicated), but it worked out well for everyone in the end.
avinium 8y ago
If a consultant could jet in, diagnose and fix up some issues in a couple of days, that means the team did a pretty awesome job to begin with. A truly inexperienced team would have created a spaghetti monstrosity that was practically unsalvageable.

platform 8y ago

Doesn't this highlight a continuous issue,that folks in our trade, do not have any meaningful and up-to-date professional certification?

walshemj 8y ago
Most professions don't eg when you have passed your bar exams taken silk or what have you - you do have cpd but these are self certified.
alecbenzer 8y ago
Is there a clear indication that certification helps in other fields? (Admittedly this seems like it'd be very hard to verify, since most fields either legally require certification or don't have it, and there's probably tons of other variables that would get in the way too.)

lisper 8y ago

> At the end of the day, the best way to get more money is to have other offers. Actually, the best way to get more money is to already have a job that you are willing to leave.

yeukhon 8y ago

> They hired the first developer, and he vetted the second developer, and so on until they had a development team This is hard for a startup in their early stage. Not sure how mature the company was, but let's assume it was young enough to have this very problem. Direct team hire is a double-edged sword. It works well when your team is very competent and good at weeding out false positive. Basically, when you have a bunch of senior people with maybe a couple junior and mid-level teammates. A normal distribution is actually perfect for a team. Having a direct contact with your future teammate allows you to know the person ahead of time, especially for a very team. Furthermore, direct hire means you are hiring not just a generalist. Your SWE may not know what to ask a network engineer, even from a coding point of view. The downside is what happened in the story. The way I'd like to go about it is to make sure we have two interviewers per round, even from the same team would be far better than one person vetting.

memset 8y ago

This is interesting - many of the comments here are correctly asking "what if you don't know an expert to help vet the engineers and the system?" I feel the same way when trying to find dentists or accountants. I simply don't know how to know who will perform completely! So, here is my pitch: if you don't have technical expertise, and you realize that you need someone to help understand just how the heck to get started with technology, what to build, and determine if you even need engineers, then hire me! I'll help! https://www.jaygoel.com/consulting/ https://www.jaygoel.com/consulting/

honkycat 8y ago

> If you read this far hoping for an answer, then I’m sorry: I don’t have a simple one. This is a difficult problem to solve. The solution is too large for a single blog post, changes every day, and differs subtly for every project. Crazy idea, what if we found one of these people who writes these blog posts, an especially well vetted and knowledgeable one. Maybe even one who writes the frameworks and software these people are using. cough https://pragprog.com/book/phoenix14/programming-phoenix-1-4 https://pragprog.com/book/phoenix14/programming-phoenix-1-4 cough What if we took this person, and paid them money to write a bunch of blog posts about on topic, in a way that it read in a linear fashion? We could even maybe hire someone to help them, to proof read and make helpful suggestions. We could give the knowledgeable developer a year or so to write the blog posts, then print out the blog posts, bind them together with glue, and SELL IT! People could pay money for the well researched, well edited series of blog posts about a specific topic written by the expert. I think I'll name this blog-printed-out-onto-paper... a honkycat. After it's inventor. --- Sarcasm aside, I find people's insistence on googling and sifting through half-thought-out blog posts infuriating. I am a HUGE proponent of a well written book about a software product. Compare "Programming Elixir" to their free online book and documentation. It's night and day. And I can always look in the same place for reference, the book. Written by Dave Thomas. A book has: ( At least ) one author, one editor, and a publisher. All of these people work hundreds of hours to produce a useful manuscript. A blog post is something a script kiddie shits out because they're trying to get their first programming job. Yet people say the same thing: "A book?!? I can just find it on the internet" and do the same thing: "I'm sure I'm smart enough to just start writing this software without any prior experience in the language, or forethought" and make the same mistakes over and over. Skip the blog. Read a book.

aidenn0 8y ago
Finding a good book is not particularly easier than finding a good set of blog posts. While the quality of books is on average significantly better, finding books where the material was written in the past year is much harder. Particulary in the trendy JS world, where there might not be a good book for the stack you are using, and even when there is, you ask questions online and get responses that are roughly "It hasn't worked like that for 6 months, why are you trying to do it this way?" You give the example of the Phoenix book; I don't know if that's what this team was using. If it was not, the fact that there is a good book for that stack is not particularly relevant. If they were using Phoenix, then the fact that there is not a good book for most stacks would explain why they wouldn't consider looking for one for the stack they are using.

janwillemb 8y ago

> Before I go any further, I’ll continue my story. It seems to me that that sentence has no significance at all and can be removed safely ;)

pattrn 8y ago
Point taken, and sentence removed. Thank you!

exabrial 8y ago

"Move fast and break things" only works if your company has a stranglehold on a monopoly and your user base is used to bugs in production. Terrible advice for just about everyone else. You want your users to have the best experience possible, rather than do your QA process for you. EDIT: I should say my point is: "moving fast" and "testing things" are goals that aren't contradictory. You can and should do both.

ddebernardy 8y ago

> If you read this far hoping for an answer, then I’m sorry: I don’t have a simple one. This is a difficult problem to solve. The solution is too large for a single blog post, changes every day, and differs subtly for every project. [Cough] - Hire a few more (expensive) experienced developers? It might not be a silver bullet, but it goes a long way towards avoiding the situation described in the blog post. Also, get help from a trusted technical friend to hire the initial one. FWIW I see something similar with almost every client I get in my own neck of the wood. Growth is flat. Turns out nobody is worrying about whether they're building a product anyone wants - they're doing it on paper but nobody is actually talking to end-users. Marketing is under-delivering. Turns out it's all juniors worrying about bringing traffic without wondering if it's the right type of traffic; or they're not measuring anything, plain and simple. Sales aren't selling enough. Turns out it's all juniors who aren't selective or systematic enough; or sales are doubling as pseudo-project managers and/or pseudo support-engineers because nobody is managing operations. Projects take forever to deliver. Turns out the project "manager" holds nobody accountable to deliver any specific task let alone by a certain deadline. The list could go on, and on, and on.

arendtio 8y ago

Some time ago I was writing some software in Go in my spare time (just for fun and I could spend my time how I liked it). As I wasn't completely sure what it should look like in the end, so I thought it would be reasonable to skip writing tests in the beginning and just write the code and add tests afterward where they would be necessary. Fast forward I had to write the whole code twice. The second time I wasn't completely sure how it should look like either, but I put a much greater emphasis on interfaces and modularity. That way I could guarantee that parts of the program were actually doing what they were supposed to do, which made the bug hunt so much more effective. So sometimes the longer way is actually the faster one and experience matters when it comes to recognizing such situations.

humzakh 8y ago

"most fast and break things" only if you have an idea about the implications of your actions and you are confident you will be able to fix the broken parts which in production without losing revenue or customers.

pytyper2 8y ago

So, what shouldn't i do in production? This article was unhelpful.

pknopf 8y ago
The snippet of code copied from a blog into production software was only a small detail. You obviously missed the entire point of the article.

stcredzero 8y ago

But what other option did these developers have? They had to deliver something, and they had never released a production application before. So they did what any sensible person would try to do, and they learned on the job. And yet, so many people push back on my complaints about the shallowness of the education many recent CS grads are getting. No, it's not an irrational fear of the next generation. It's an awareness that some large fraction of recent grads sporting high GPAs don't have enough basic background knowledge to know what they don't know and keep themselves out of trouble. What would we think if companies let non-engineer/architect managers put together teams of completely mechanical engineers and architects, who then went on to build faulty bridges and machines that started to call apart? People would be calling foul. Given the societal, privacy, and financial consequences, it's way past time for the programming field to grow up!

user5994461 8y ago
It's the same in every industry and every domain. A new grade needs a few years of practice before he can contribute well to any real project. However companies don't always have dummy projects for years, just to practice and drop afterwards.
- stcredzero 8y ago
  It's the same in every industry and every domain. No. In most industries, groups of novices don't get to do something that gets pushed out to production.

mmt 8y ago

> helped mentor their developers This is typically missing from startup culture. Anecdotally, mentorship improves everyone involved, but I don't know if there's data out there showing if engineering mentorship within a startup improves its chances of success (or is otherwise beneficial [1]). To some extent, groups like YC participants and alumni end up being mentors to each other, but I doubt that extends beyond the founders. Might the OP have an opinion on something like a rent-a-mentor program/startup? [1] Not that certain benefits, like productivity, would be enough to motivate a change, as evidence by the open plan office situation

anon1253 8y ago

Even for an experienced programmer this stuff happens, and it's the most frustrating experience I've ever had. It's the stuff that happens when management and C-level just "wants stuff" to be buzzword compliant. And, every day/week/month the requirements change. This weeks feature X has "top priority" and then amnesia kicks in and now feature Y is the next best thing. As a programmer-now-manager I try to shield people from this, trying to find the gems among the rubble of business meetings. But it's tricky, so this really resonated with me: "I have an overwhelming empathy for developers in this position. They have more information than they will ever need, but it’s completely disorganized. It’s like trying to build a ten piece puzzle, except you have to find the ten pieces within a pile of 10,000,000,000 pieces, all of which are square, and you don’t know what it’s supposed to look like at the end. Good luck." Even with a team of highly experienced developers things quickly turn to a mess if it's not perfectly clear what to work on. I don't mean things like "button should go 1px north to align with header". That's trivial. In fact, I would go so far as to say: most production-ready programming is trivial. Trivial in the sense that someone, somewhere, solved it in a way that's correct. But, when it is not /crystal/ clear what the requirements are, what the business case is, or what the edge-cases look like: oh my. I've been in the incredibly unlucky position of only being in settings like this. Start-ups, academia, R&D positions. And, as a result even with the best intentions and the best teams you'll still produce production-quality garbage. Mostly because incentives were misaligned and there was no coherent vision. Agile story-driven development only works when there is leadership that has the guts to say "no", and when there is something well defined to work on. Sometimes I yearn for some CRUD webstore ... at least then the things are knowable. Anyway, just my rant. I'll probably go the meme-way of becoming a plumber/welder/etc at some point ;-)

Double_Cast 8y ago

> There’s a lot to be said for building things that work, but “move slowly and steadily towards a goal” just doesn’t have the same ring. I'm reminded of one of my math professors. When a student asked him "How do I do this problem?", the prof would always respond "Carefully and correctly". And then he'd laugh at his own joke and give the student a real answer. I thought it was funny. And I liked the alliteration.

Ftuuky 8y ago

The solution? Blockchain. /s

sattoshi 8y ago

This post was completely not what I expected it to be. The "do not use in production" is the programming equivalent of taking a strong stance on an issue but then appending "but hey, I don't know" or "don't take my word for it". They completely defeat the purpose of the entire statement and in the production comment's case, the code samples. Let's ask a different question, what am I supposed to use in production? I had an email discussion with some blogger who wrote about a very narrow problem which I needed to solve. Their code samples worked, but poorly. The entire thing was a dumpster fire but it solved the problem and were the only one I could find after days of Googling and Stackoverflowing. I did not care about their "do not use in production" comment. I needed their solution in my production system. What was I supposed to do? Sorry for ranting, but I believe it is irresponsible to put code out there which shouldn't be used except for when it is the point.

_hardwaregeek 8y ago

This something I don't understand about certain startup founders. If you're starting a company, you should have at least one absolute monster of a developer. Someone who is capable, communicative, and with a deep pool of knowledge. Someone who can attract, hire, and lead other good developers. Not your buddy Steve who made an iPhone app or two. Not that developer who has some PHP experience and built a React app once. A real, terrifyingly good developer. The aphorism "first class people hire first class people; second class people hire third class people" comes to mind. Granted, I'm a pretty inexperienced developer, but I see companies being founded by people who are even less competent and less knowledgeable than me. And sure, finding a competent person is hard. But it's like the movie Seven Samurai. Can't afford to pay a samurai? Find a hungry one. There are plenty of developers who are young, but still good. And if you can't even do that, then take any and all money you have, and pool it to find one or two (or seven) developers who can find other competent people.

eropple 8y ago
> There are plenty of developers who are young, but still good. I was young, and I was good. (Now I am moderately young, and I'm very good.) But I am a better talker than I was a developer. Only recently have I really become as good a developer as I am a talker. How are those startup founders supposed to evaluate me?

exmicrosoldier 8y ago

This was a problem with people. Specifically, people who force people to commit to a known deadline on something they dont know how to do. Someone got promoted for delivering business value, and they were smart, they left before it all blew up. Get 'er done without context and fire them for failure is bad management.

altitudinous 8y ago

Many developers are commenting in here trying to fix a problem. I am a developer but I now run a business. From a business point of view, there is no problem here. Code has been delivered and it is running in production successfully. It might not have arrived at that point in the most efficient manner, but the point is that the business has executed successfully here. There is no issue. This is a lot further than most businesses get. If you see a problem to be fixed here then you are looking at a small picture of the business.

alecbenzer 8y ago
Are you discounting that technical debt is a thing/matters? What gets delivered isn't just a set of features, it's a piece of software, which implements those features, but might have a whole bunch of baggage attached which can make it really hard (or easy) to change those features or add new features in the future, or to allow the software to adapt to changing external conditions. I don't think this is specific to software. E.g., I'd imagine it'd be possible to build a building a lot faster if you didn't need to worry about it not collapsing in five years. --- Though to be fair, the mere presence of technical debt on its own is not a problem; there is some amount of technical debt that's appropriate, and it's hard to know exactly where that is. You certainly can't get by just ignoring it completely.
occams_chainsaw 8y ago
A rotting foundation eventually catches up with you. Things don't have to be perfect, or even particularly great... but when things are working by accident rather than by design...
draw_down 8y ago
Sure, the problem comes later. And it will be very expensive to fix.
tonyedgecombe 8y ago
To some extent that was just luck, they happened to hire a consultant who happened to be able to resolve the visible problems. Hiring junior developers because they are cheap and keeping your fingers crossed isn't going to work most of the time. Not shipping will sink any business if it goes on long enough.

brian-armstrong 8y ago

Another angle to look at this through is that they hired a team in spite of only having business, non-technical minded people. It all worked out in the end when they added consulting to clear up production issues near launch. From a business perspective this is surely a win in all ways, even if not as efficient as it could be. Not every business is a Google.

jarym 8y ago

I dunno.. whenever I read ‘don’t do this in production’ I spend ages trying to really understand why - only when I feel I’ve got a real handle on what the pitfalls are would I go forward. I’ve done that ever since I started programming over 20 years ago and I still do it now. Sometimes it takes me longer to code a solution but I do feel I spend less time figuring out why stuff breaks. It’s not just experience. It’s about hiring curious and diligent people.

kristianp 8y ago

"... fix issues as I went along. Without unit tests or automated deployments, this took almost a year." One of the first things he should have done was to get them Joel-test compliant, otherwise he's wasting the money they pay him.

daveheq 8y ago

"The only clear issue was lack of experience" "In this case, however, it had a nefarious twist: all of these developers lacked experience." Am I missing the twist here? It was already mentioned as the primary problem.

macns 8y ago
try re-reading this with "all" in bold
pattrn 8y ago
Good catch! macns is correct: "all" should be bolded.

daveheq 8y ago

Copying and pasting internet code into production that says “Don’t do this in production“ is not a lack of experience problem, it's a lack-of-reading problem or a rushing-things problem; the latter is typically caused by management putting deployment dates on features without understanding the technical cost.

senderista 8y ago

MSDN used to make their code samples as simple as possible for educational reasons, until they realized everyone was just pasting them directly into production applications. So they started adding error handling, etc.