5 ms·
Two points: 1) Patio11 touches on a very good lesson, in passing, in an article about Japanese business[1]: While raw programming ability might not be highly
by NamTaf 10y ago
Two points:
1) Patio11 touches on a very good lesson, in passing, in an article about Japanese business[1]:
While raw programming ability might not be highly valued at many Japanese companies, and engineers are often not in positions of authority, there is nonetheless a commitment to excellence in the practice of engineering. I am an enormously better engineer for having had three years to learn under the more senior engineers at my former employer. We had binders upon binders full of checklists for doing things like e.g. server maintenance, and despite how chafing the process-for-the-sake-of-process sometimes felt, I stole much of it for running my own company. (For example, one simple rule is “One is not allowed to execute commands on production which one has not written into a procedural document, executed on the staging environment, and recorded the expected output of each command into the procedural document, with a defined fallback plan to terminate the procedure if the results of the command do not match expectations.” This feels crazy to a lot of engineers who think “I’ll just SSH in and fix that in a jiffy” and yet that level of care radically reduces the number of self-inflicted outages you’ll have.)
2) I once heard organisational 'red tape' described as 'the scar tissue of process failures' and it is absolutely true and I deeply regret not recording the source of it. Whenever you wonder why there's some tiresome, overly onerous process in place that is slowing you down, consider why it may have been put in place - chances are, there was a process failure that resulted in Bad Things. When you wonder why big orgs are glacially slow compared to more nimble startup competitors, understand that those startups have yet to experience the Bad Things that the big org has probably already endured. Like scar tissue, the processes they develop reduce their agility and performance but also serve to protect the wounds they experienced.
[1] http://www.kalzumeus.com/2014/11/07/doing-business-in-japan/ http://www.kalzumeus.com/2014/11/07/doing-business-in-japan/
- zwp 10y ago> regret not recording the source of it The googlable nugget is actually "organizational scar tissue" (it caught my attention too). It's from Jason Fried. On twitter: https://twitter.com/jasonfried/status/2758624714 https://twitter.com/jasonfried/status/2758624714 and also apparently in "Rework", quoted with more context here: https://www.goodreads.com/quotes/1012423-policies-are-organizational-scar-tissue-they-are-codified-overreactions-to https://www.goodreads.com/quotes/1012423-policies-are-organi...
- adrianN 10y agoThat is certainly true of some of the red tape, but in no way it's true for the majority. A lot of "process" is created because the people in charge of the process need to validate their existence.
- NamTaf 10y agoUndoubtably that's a problem, I agree. Competent management helps minimise that, though. And I would not agree that the majority is existance justification. Then again, I'll disclaim that by saying I work in a Mech Eng. role, where process for safety's sake has been established and engrainedi nto the culture for literal centuries. A good employment environment is one where you may ask why a process exists and receive valid justifications therein, but where the idea of not following it, no matter how bad, never crosses your mind. I acknowledge I'm really lucky to work in an industry that doesn't fall too far from that target.
- rmc 10y ago> When you wonder why big orgs are glacially slow compared to more nimble startup competitors, understand that those startups have yet to experience the Bad Things that the big org has probably already endured. Another explaination for big orgs vs teeny start ups: What level of failure is acceptable? For a teeny start up, a few hours being down is not so important. For (say) a bank, being down for a few hours might be mentioned in the national newspapers.
- unscaled 10y agoOne of thing that I learned the hard way about "Japanese companies" - despite western conceptions, every company in Japan has its own unique culture (and takes pride in having its own culture!). What's more, often departments and division inside the same companies work in very different way. Why am I saying that? Because in some of the Japanese companies I've worked with were the exact opposite of that. To be sure, lip service was duly paid to the aforementioned "commitment to excellence", and every release procedure had its own operational manual, sometimes 300 steps long. Repeated manually for every server. Out of 100-200. Configuration updates? Sure, let's log in to every server and vi the config file. How do we keep excellence? Just diff with prev and verify (with your eyes that is) that the result is the same as in your manual. After every "cd" you had to do a pwd and make sure that you moved to the directory you meant to. After every cp you diffed to the original file to make. Releases obviously took all day or often all night, and engineers were stressed and fatigued by Sisyphean manual with its 300 steps of red tape. They invariably made silly mistake, because this is what you get when you use human beings as glorified tty+diff. We had release issues and service outages all the time. We've fortunately managed to move away to modern DevOps practices with a lot of top down effort. But please don't tell me every Japanese company magically delivers top quality. Some of them do, some of them don't, even in the same industry. Insane levels of bureaucracy could be found all across the board, but whether that bureaucracy actually encourages or deters quality is an entirely different story.
- pmlnr 10y agoUnfortunately I had similar experiences as well; incredibly manual processes, frighteningly long manual procedure descriptions instead of scripted solutions. My opinion: script it. Always. It doesn't matter if it's ansible, bash, puppet, python, whatever, just make sure it's not an ad-hoc command. Test the script on a server which can be sacrificed. Test as long as there is a single glitch. Run it in production. It's to eliminate typos and to have a "log" to see what actually had been done.
- NamTaf 10y agoOh, absolutely. Where something can be scripted, script it. Why? Because scripting is a process development. You write something, validate it and then remove the human error element. For things that you can't script, you write abstracted processes that force the executor to write down the things that could cause Bad Things to happen, and use that writing down stage to verify that it's not going to cause a Bad Thing. That forces people to pause and consider what they're doing, which is 80% of the effort towards preventing these issues. eg: Forcing YP to write down which database they were scorching would've triggered an 'oh fuck' moment. Having a process that dodged naming databases as 'db1' and 'db2' would've prevented it. etc. etc. etc.
- quotemstr 10y agoI just wish big companies were more willing to remove this scar tissue.
- erelde 10y agoOr even not scarify themselves before potential Big Bad happens. Pushing the metaphor a bit too far.