7 ms·

I think there's something else that's necessary that doesn't align well with the things in the article, and which is currently lacking in my current work enviro

by elihu 5y ago

I think there's something else that's necessary that doesn't align well with the things in the article, and which is currently lacking in my current work environment, but which is kind of hard to describe succinctly. Basically, my job requires using a lot of tools and technology and processes created by other people, but none of those tools are easy to use or well documented. In order to get stuff done you have to have a social network of people who know how to do the things you need to do, and know all the secret tricks. The right environment variables to set. The right arguments to pass to simulators. How to find the important bit of data buried webapp developed by people who apparently like to invent new words for old concepts. I get a lot of terse autogenerated emails that probably mean something to somebody, but not to me, and don't have human return addresses or really any way to discern the context of where they came from or why. A lot of tools use domain-specific configuration languages that are usually just python scripts that call a bunch of functions that are defined elsewhere and have no documentation. These things might be reasonable in a startup with a dozen people, but we have over a hundred thousand. My opinion is that once an organization is a certain size, everything really needs to be written down and easily accessible. I shouldn't have to regularly ask people how to do basic things, it should be on a wiki. If it's complicated, it should be in a training video, or a user manual.

haney 5y ago
This sounds exactly like my experience at Meta.
bad_alloc 5y ago
> but none of those tools are easy to use or well documented Technological + organizational debt?
throwaway1293 5y ago
Note that I'm not saying we shouldn't do as best as we can with documentation, clean code, etc, BUT: In the open source world, I've developed a large low quality codebase that is very complex from when I was less experienced. This project has gotten plenty of contributions over the years, some from less experienced and some from more experienced. The experienced developers know how to get around the codebase, but the lesser experienced ones tend to complain about quality. I see this in the workplace as well. I had this exact same issue when I was less experienced. New codebases was very difficult to understand, but now that I've been through a lot of them in various languages and quality, it's not as bad as it used to be. I guess the argument here from the business point of view is that we write code for the less experienced? In a way this makes sense, but paradoxically some complex codebases that I thought was messy in the past have become more elegant today as I've gained more experience.
- pferde 5y ago
  It could be that the more experienced developers are just burnt out in trying to give that kind of feedback, and instead just grit their teeth and get through the code base as they need. Meanwhile the newer, less experienced developers probably still have a bit of... let's say naive optimism, and try to give feedback more.
  - throwaway1293 5y ago
    That could be the case for a job you get paid to do, but for open source there are people coming in to help because they like the solution and have ideas for new features or refactoring. The inexperienced developers in my case have not contributed much, but rather complained in other channels. (it's game related, the community is on discord, in game, etc)
- bdavis__ 5y ago
  it is probably because the new devs know so much more and are able to identify and communicate problems in code that others have written. write code for the center of the bell curve (at your organization). the bottom will never understand anything, the top understands it all. You need to make the majority in the middle be comfortable. that means slowly bringing in new language features, as they become widely known and practiced.
  - marcosdumay 5y ago
    That's an interesting point of view to explore. Is it worth more to attract a few highly capable developers or a lot of middle capable ones? Your code practices will probably resonate differently on each of those groups, so you will probably have to choose. And I don't think there's a generic correct choice like you imply. Also, by how much are people even capable of choosing the (required) expertise level of the code they create? I'm not sure we can deviate a lot from our natural level.
    - bdavis__ 5y ago
      yeah, absolutes never apply to every situation. it is possible to go down in expertise, but not up. sometimes "fortran in any language" can be a viable approach depending... note i didn't say a the best approach.
codethief 5y ago
I'm getting the same impression at my client and my client only employs ~150 devs, not a 100k. All in all, this strongly reminds me of the "Collapse of Civilization" described by Jonathan Blow: https://www.youtube.com/watch?v=ZSRHeXYDLko https://www.youtube.com/watch?v=ZSRHeXYDLko
- deleted 5y ago
  [deleted]
- 0x445442 5y ago
  Alan Kay has similar talks where he has some subtle condemnations of what the industry as a whole has devolved into. He uses terms like Cargo Cult and he calls out the wide spread hubris for not knowing and/or actively ignoring the history of the field.
austincheney 5y ago
As somebody who has been writing software for two decades the most important ingredient for me is maturity. I am not going to leave the employment I currently have if signals of maturity are weak. More money does not compensate for this. I thought it would when I interviewing a few months ago (50% salary increase). I was wrong. * https://news.ycombinator.com/item?id=29826152 https://news.ycombinator.com/item?id=29826152
jlkuester7 5y ago
> My opinion is that once an organization is a certain size, everything really needs to be written down and easily accessible. I shouldn't have to regularly ask people how to do basic things, it should be on a wiki. If it's complicated, it should be in a training video, or a user manual. This %100. Automate all the things, but for any part of the process that cannot be automated/scripted, write it down. And don't just put it in some random Google Doc or Sharepoint where it is lost forever in the ether. Put it in a README that lives with the code or have a shared Wiki for the team. It need to be somewhere that is generally available, searchable, version controlled, and easy to update.
- vianneychevalie 5y ago
  We call it Data governance and Enterprise architecture, those are fields and we're hiring like crazy
- mumblemumble 5y ago
  Write it down even if it can be automated. And make sure the documentation is also comprehensible to people who don't already know how the system works. Otherwise you're setting your company up for a miniature re-enactment of the Butlerian Jihad at some point in the future.
- metafunctor 5y ago
  Even if it's all documented with a README, with container images and Ansible playbooks or whatever to make it all happen… still people will be complaining that it's oh so complex and that's not how we did it in my previous job and what about secret management, or load scaling, or full disk encryption, or how would this be deployed to a moon-based server, or whatever they can think of how the probably less-than-average crap you've built isn't maximally convenient for them. So, even if you have everything 100% automated, most people will look at that and conclude that your automation is complex legacy bullshit and you should have automated in some other way. Just saying.
  - hoffspot 5y ago
    I second this emotion. We have spent years and countless hours investing in documentation and automation as we re-engineered a very large legacy application into cloud based technology (AWS). The application and supporting CI/CD pipeline is far better documented and supported than most and all of the underlying technology is within 5 years current. There are video based tutorials on nearly every aspect of the platform and machine images to accelerate development environment setup. Yet still, people new to the development pipeline will hand wring over "tribal knowledge" and "deployment complexity". News flash, anything constructed to support hundreds to thousands of business processes and millions of customers is more often than not going to be complex. Sometimes you just have to roll up your sleeves and try to improve the trail for those coming behind you.
deleted 5y ago
[deleted]
lc9er 5y ago
Sounds like you are describing “institutional knowledge”. In addition to good documentation, an org needs to maintain employee retention, so you have people that know the what and why’s of all the automated processes you’re seeing.
- elihu 5y ago
  Institutional knowledge is an okay thing, the problem is when it's passed down by oral tradition or actively kept secret unless you email exactly the right question to the right person.
tomxor 5y ago
> hard to describe succinctly. Maybe "Institutional Complexity"? Some degree of complexity is intrinsic to the problem domain, but it's the incidental complexity that emerges as a thoughtless side-effects of other efforts that is the most annoying, especially when it compounds. To flip your comment on it's head... Some of my most satisfying projects have been the ones that eliminated this type of complexity where I work, because they improved the quality of life of everyone I work with and gradually remove all but the necessary work. However because these things are usually difficult to isolate and deeply engrained, this is only usually possible once you've gained enough trust and autonomy, which I suppose makes it a hard sell for a new job.
loup-vaillant 5y ago
Sounds like a lack of repeatability to me: complex tools, combined with lack of documentation, makes for a fairly unstable work environment. It's for similar reason that I like to keep unreliable dependencies to a very strict minimum. I can rely on my compiler, my editor, my OS, and to a lesser extent a bunch of extremely popular libraries. But as soon as we get to internal tools & processes or obscure third party libraries, certainty goes downhill very quickly.
saiya-jin 5y ago
Sounds like every multinational org with 100k+ emplyees, certainly this fits my own. Fun part is, there is no realistic way out of this. It only takes one obscure critical legacy app to add so much integration mess into whole ecosystem that nobody ever will be re-doing those. Plus business won't put in the budget necessary for practically no change from their point of view.
- marginalia_nu 5y ago
  I've worked at a 500 employee-company that had many of these issues, mostly from misguided attempts at copying the way FAANG worked without any regard for our own needs or scale. The entire code was a tangled mess of git submodules, transitive dependencies and the build system was contingent on a domain-specific language that was partially developed in-house and partially based on some unstable open source project, incredibly janky, and only two people got to see the sources for it or even knew how it really worked. Dumpster fire doesn't even begin to describe the outcome. There was always this strange spooky-leftpad-at-a-distance thing where stuff just kept inexplicably breaking, and nobody ever quite knew what had gone wrong, who to talk to, or when it was gonna get fixed.
AuthorizedCust 5y ago
Corollary: the documentation must be written to speak to the stupidest person in your team/company who you expect to use the product it describes. Otherwise, you may end up with a documented system that you can’t use or fix. Fortunately, that’s often me (a supervisor), so I get to be self-deprecating when conveying this requirement.
- qchris 5y ago
  I also find that hypothetical "interns" are great for this. Something about "this person without a degree, who probably hasn't been working for very long, who doesn't know your organization or who to ask, and is probably very scared of looking or sounding stupid." It doesn't necessarily imply or require stupidity, but in my experience on both sides of it (writing docs now, and having been that intern), they actually make pretty great litmus tests for documentation and how accessible organizational knowledge is.
- geekbird 5y ago
  When I write process/operating docs, I write it for the person who is on call, at 02:00 Sunday morning, after overindulging in alcohol Friday night. I don't assume stupid, just not firing on all cylinders due to being half awake and distracted.
nhumrich 5y ago
> hard to describe succinctly To me, what it sounds like you are describing is the developer experience. Or "devX".
tablespoon 5y ago
> These things might be reasonable in a startup with a dozen people, but we have over a hundred thousand. My opinion is that once an organization is a certain size, everything really needs to be written down and easily accessible. I shouldn't have to regularly ask people how to do basic things, it should be on a wiki. If it's complicated, it should be in a training video, or a user manual. Here's the root cause of your bug: https://agilemanifesto.org/ https://agilemanifesto.org/ > Individuals and interactions over processes and tools > Working software over comprehensive documentation I remember when my employer adopted agile. One of the first things they did was lay off their team of technical writers. Also you wasting your time flailing about probably makes some manager's budget numbers look good, because they got to "save" the money that would have prevented you from flailing about.
- kodisha 5y ago
  I you could once a year do +100 votes instead of regular +1 vote, I would do in right here, even though it's not even half of January.
- commandlinefan 5y ago
  > Here's the root cause of your bug Hang on, don't blame the agile manifesto here. Blame the management that adopted the agile manifesto as "do all the exact same things we used to do that didn't work, but call it 'agile'".
  - tablespoon 5y ago
    > Hang on, don't blame the agile manifesto here. I totally can, and will. It's on them if they popularized slogans that can be too-easily misunderstood. That's meant that agile, in several respects, has been an exercise of throwing the baby out with the bathwater. > Blame the management that adopted the agile manifesto as "do all the exact same things we used to do that didn't work, but call it 'agile'". That's clearly not what my employer did.
    - headmelted 5y ago
PragmaticPulp 5y ago
> Basically, my job requires using a lot of tools and technology and processes created by other people, but none of those tools are easy to use or well documented. When I first moved into management roles I didn’t fully understand how much of my job would be simply saying “no” to a constant stream of engineers wanting to write or rewrite custom tools, libraries, frameworks and platforms when otherwise sufficiently good alternatives already existed. Not all engineers, but once you get into a big enough organization there are a lot of engineers who want to do meta-work and rework things instead of actually shipping things. The worst offender I worked for had meta-tools on top of everything. You couldn’t use GitHub, Slack, or even our cloud platform without first learning how to navigate all of the custom tools and bots that had been put together to gate access to everything and make it work. And like you said, it was poorly documented. It turned into little fiefdoms where you had to know who to talk to if you wanted to get past the systems to get things done. This can all be avoided with good management at an organizational level, but of course some managers want to get in on the situation because it puts them in a perceived stronger position in the company if their team is at the center of everything.
- rthomas6 5y ago
  It also helps to find the one extra weird engineer that loves to write documentation, then have them do that for a while for everything.
  - 0x445442 5y ago
    What are the acceptance criteria for that story and how many points would it be?
- Dave3of5 5y ago
  > You couldn’t use GitHub, Slack, or even our cloud platform without first learning how to navigate all of the custom tools and bots that had been put together to gate access to everything and make it work. First time in a really big org and I can see this as well. The worst part is trying to tie someone down to explain how to use their custom tool or bot or even entire systems. How do I use that custom BI tool that you setup ... oh no-one wants to even send me an email reply when I ask a simple question about these things. The way most stuff gets done in these orgs is various maverick people who find some way to work around all this crap. The problem is you have to have political clout to do that most employees don't have that. I think these "meta-tool" are initially well intentioned but as you say they become empire building tools. Worst still is that the teams that build them often leave or are disbanded. Then no one is responsible and no one wants to touch them and they sit in this weird limbo state.
yowlingcat 5y ago
I think you are touching on the right problem but your solution seems a bit more on the idealistic rather than realistic side. I've had this conversation before where documentation is proposed as a silver procedural bullet and it never works. Documentation gets stale and eventually out of date. Do you really want to be in a position where you have to /recompile/ all of those documentation/training video/user manual materials? Unless you're selling it, probably not. The solution you are looking for is not documentation but automation and training. The things which can be automated to make the process closer to obvious should absolutely be automated. The things which are intrinsically more challenging and can't be solved through automation should be covered through training -- and not a "training video" but a real set of sessions where someone (likely your manager or a tech lead responsible for ensuring clean consumption of their modules) is walking you through the right way to do things and leaving time for questions. I say this having worked at a place that touted a documentation centric culture at the level of organizational scale you suggested. If you're doing anything remotely challenging, documentation only gets you to the starting line. It doesn't get you to being useful and autonomous.
Mezzie 5y ago
Good luck convincing organizations to invest in this + finding people to do it. I have the background to organize this (I'm actually doing this for my own, smaller organization right now just because the situation is driving me insane) and the main problems are: 1.) The people who can do this aren't going to do it for 30k or $10/hr. Properly solving this problem requires an understanding of tech and the development process, pedagogy/communication/instructional design, and information management. Lacking any one of those would result in a system that's an unusable vanity project. I have a couple of decades of small-shop coding experience, experience as an instructional designer at an Ivy, and graduate education in information management, but if you want me to use all of those it's going to cost you and companies are resistant to that. (See: How a lot of non tech product companies treat their IT teams since they don't generate revenue). Now imagine if the project were led by someone more qualified, which it should be for a company of your size. 2.) Most companies don't have a culture which would allow this to be done well. This is the sort of work which requires both a lot of honesty and a lot of careful planning. A lot of information ends up hidden or inaccessible due to people's egos, or executives not being able to handle the information so it's buried. People need to be honest about their workflows and what they don't understand, and they need to feel comfortable being honest about such things. This gets really messy. On the dev side, a common cultural issue is "I had to learn it, they should too" or the urge to make understanding the complexities into a shibboleth for the in-group. So I agree with you, but it's unlikely to change soon in my opinion.
- fuzzer37 5y ago
  Can I ask how you got started in the field of instructional design? This is a field that I've not really heard much about until recently, and I'm curious how someone breaks into that field.
  - Mezzie 5y ago
    Unfortunately, I'm one of the old 'I fell into it because I had experience in both the field's components (tech and education)' stories: I've been coding (mostly but not exclusively front-end) since the 90s and been teaching techy things on and off about that long (I taught my first programming class in '99 when I was 11), and then I went and worked in libraries/studied how people learn outside of formal classroom contexts. So I already had all the skills needed once the field started. I cheated. If I were getting started now, my advice would be to remember that the human element and the tech element are of equal importance in Instructional Design. So you'd want to take stock of what skills you currently have and how best to a.) get the necessary knowledge you don't have and b.) take advantage of your particular skill set. Since you're on HN, I'm assuming you're of a techy bent vs. an educational/people bent. So if you have any teaching/tutoring/etc experience, I would lean on that. If not, I would get some. You can do this by volunteering somewhere + doing some basic research. I highly recommend this since a large portion of instructional design (like most front-end things) is planning for all the weird things humans do when interacting with your stuff and, more importantly for instructional design specifically, how people who can't computer at ALL think. When doing ID, you're often designing for people who can't figure out how an email works and are taking a work training, for example. Also since you're techy, I'd recommend spending some time with UI/UX and specifically focus on designing for different populations/audiences. From a tech perspective, your competing instructional designers are worst at the 'boring' things like accessibility guidelines, making sure things can run on terrible machines, coding fall-backs, etc. So a tech background is of great assistance as long as you communicate the problems it can help you avoid. As for the actual 'how to get a job', I literally applied for my first job off of Indeed out of grad school and I'm a first-generation student so trust me that was not the most polished resume or CV.
jensensbutton 5y ago
It's a problem of the incentives we've setup. We reward engineers for building things. Promotions, pay raises, etc. for engineers are typically based on impact and technical complexity, not how good their documentation was. I've been at a few places that handwave at such things in their level expectations or skills matrix or whatever, but having been in promo discussions at a variety of medium and large companies it always comes down to what was shipped. Problem is, at mature companies there aren't enough greenfield problems to match the large number of engineers who are looking to climb the ladder. So stuff is rewritten, then rewritten again since nobody understands the rewrite, and so on. Managers kind of look the other way because retention and the ecosystem becomes messier and messier over time as promoted folks leave and new folks come in that need to be promoted. This is the world of professional software development at pretty much every company not a startup.
sly010 5y ago
I admit I quickly searched my company directory to see if you work where I do :) The article mentions 3 things engineers care about: Technology, Intellection and Stability. Perhaps your pain fits into the 3rd category (if extrapolate a little). If so, ironically, the first 2 at scale is what causes the lack of the 3rd. When you make wider roads, the traffic will get worse, not better. Perhaps it's similar technology organizations. It's almost inevitable. Anecdotally Amazon figured this stuff out by mandating well defined organizational boundaries (kind of like solid white dividers on some section of the highway, eh?) and by making each team responsible for its own developer infrastructure. It seems wasteful at first, but maybe less so at scale. Maybe someone working there can confirm/deny whether this is the case.
Buttons840 5y ago
How many of us agree but haven't yet read our companies Confluence pages?
- Kon-Peki 5y ago
  My company is probably doing Confluence wrong. Because if you haven't bookmarked the page, you'll never find it again.

by elihu 5y ago

haney 5y ago

This sounds exactly like my experience at Meta.

bad_alloc 5y ago

> but none of those tools are easy to use or well documented Technological + organizational debt?

throwaway1293 5y ago

Note that I'm not saying we shouldn't do as best as we can with documentation, clean code, etc, BUT: In the open source world, I've developed a large low quality codebase that is very complex from when I was less experienced. This project has gotten plenty of contributions over the years, some from less experienced and some from more experienced. The experienced developers know how to get around the codebase, but the lesser experienced ones tend to complain about quality. I see this in the workplace as well. I had this exact same issue when I was less experienced. New codebases was very difficult to understand, but now that I've been through a lot of them in various languages and quality, it's not as bad as it used to be. I guess the argument here from the business point of view is that we write code for the less experienced? In a way this makes sense, but paradoxically some complex codebases that I thought was messy in the past have become more elegant today as I've gained more experience.

pferde 5y ago
It could be that the more experienced developers are just burnt out in trying to give that kind of feedback, and instead just grit their teeth and get through the code base as they need. Meanwhile the newer, less experienced developers probably still have a bit of... let's say naive optimism, and try to give feedback more.
- throwaway1293 5y ago
  That could be the case for a job you get paid to do, but for open source there are people coming in to help because they like the solution and have ideas for new features or refactoring. The inexperienced developers in my case have not contributed much, but rather complained in other channels. (it's game related, the community is on discord, in game, etc)
bdavis__ 5y ago
it is probably because the new devs know so much more and are able to identify and communicate problems in code that others have written. write code for the center of the bell curve (at your organization). the bottom will never understand anything, the top understands it all. You need to make the majority in the middle be comfortable. that means slowly bringing in new language features, as they become widely known and practiced.
- marcosdumay 5y ago
  That's an interesting point of view to explore. Is it worth more to attract a few highly capable developers or a lot of middle capable ones? Your code practices will probably resonate differently on each of those groups, so you will probably have to choose. And I don't think there's a generic correct choice like you imply. Also, by how much are people even capable of choosing the (required) expertise level of the code they create? I'm not sure we can deviate a lot from our natural level.
  - bdavis__ 5y ago
    yeah, absolutes never apply to every situation. it is possible to go down in expertise, but not up. sometimes "fortran in any language" can be a viable approach depending... note i didn't say a the best approach.

codethief 5y ago

I'm getting the same impression at my client and my client only employs ~150 devs, not a 100k. All in all, this strongly reminds me of the "Collapse of Civilization" described by Jonathan Blow: https://www.youtube.com/watch?v=ZSRHeXYDLko https://www.youtube.com/watch?v=ZSRHeXYDLko

deleted 5y ago
[deleted]
0x445442 5y ago
Alan Kay has similar talks where he has some subtle condemnations of what the industry as a whole has devolved into. He uses terms like Cargo Cult and he calls out the wide spread hubris for not knowing and/or actively ignoring the history of the field.

austincheney 5y ago

As somebody who has been writing software for two decades the most important ingredient for me is maturity. I am not going to leave the employment I currently have if signals of maturity are weak. More money does not compensate for this. I thought it would when I interviewing a few months ago (50% salary increase). I was wrong. * https://news.ycombinator.com/item?id=29826152 https://news.ycombinator.com/item?id=29826152

jlkuester7 5y ago

> My opinion is that once an organization is a certain size, everything really needs to be written down and easily accessible. I shouldn't have to regularly ask people how to do basic things, it should be on a wiki. If it's complicated, it should be in a training video, or a user manual. This %100. Automate all the things, but for any part of the process that cannot be automated/scripted, write it down. And don't just put it in some random Google Doc or Sharepoint where it is lost forever in the ether. Put it in a README that lives with the code or have a shared Wiki for the team. It need to be somewhere that is generally available, searchable, version controlled, and easy to update.

vianneychevalie 5y ago
We call it Data governance and Enterprise architecture, those are fields and we're hiring like crazy
mumblemumble 5y ago
Write it down even if it can be automated. And make sure the documentation is also comprehensible to people who don't already know how the system works. Otherwise you're setting your company up for a miniature re-enactment of the Butlerian Jihad at some point in the future.
metafunctor 5y ago
Even if it's all documented with a README, with container images and Ansible playbooks or whatever to make it all happen… still people will be complaining that it's oh so complex and that's not how we did it in my previous job and what about secret management, or load scaling, or full disk encryption, or how would this be deployed to a moon-based server, or whatever they can think of how the probably less-than-average crap you've built isn't maximally convenient for them. So, even if you have everything 100% automated, most people will look at that and conclude that your automation is complex legacy bullshit and you should have automated in some other way. Just saying.
- hoffspot 5y ago
  I second this emotion. We have spent years and countless hours investing in documentation and automation as we re-engineered a very large legacy application into cloud based technology (AWS). The application and supporting CI/CD pipeline is far better documented and supported than most and all of the underlying technology is within 5 years current. There are video based tutorials on nearly every aspect of the platform and machine images to accelerate development environment setup. Yet still, people new to the development pipeline will hand wring over "tribal knowledge" and "deployment complexity". News flash, anything constructed to support hundreds to thousands of business processes and millions of customers is more often than not going to be complex. Sometimes you just have to roll up your sleeves and try to improve the trail for those coming behind you.

deleted 5y ago

[deleted]

lc9er 5y ago

Sounds like you are describing “institutional knowledge”. In addition to good documentation, an org needs to maintain employee retention, so you have people that know the what and why’s of all the automated processes you’re seeing.

elihu 5y ago
Institutional knowledge is an okay thing, the problem is when it's passed down by oral tradition or actively kept secret unless you email exactly the right question to the right person.

tomxor 5y ago

> hard to describe succinctly. Maybe "Institutional Complexity"? Some degree of complexity is intrinsic to the problem domain, but it's the incidental complexity that emerges as a thoughtless side-effects of other efforts that is the most annoying, especially when it compounds. To flip your comment on it's head... Some of my most satisfying projects have been the ones that eliminated this type of complexity where I work, because they improved the quality of life of everyone I work with and gradually remove all but the necessary work. However because these things are usually difficult to isolate and deeply engrained, this is only usually possible once you've gained enough trust and autonomy, which I suppose makes it a hard sell for a new job.

loup-vaillant 5y ago

Sounds like a lack of repeatability to me: complex tools, combined with lack of documentation, makes for a fairly unstable work environment. It's for similar reason that I like to keep unreliable dependencies to a very strict minimum. I can rely on my compiler, my editor, my OS, and to a lesser extent a bunch of extremely popular libraries. But as soon as we get to internal tools & processes or obscure third party libraries, certainty goes downhill very quickly.

saiya-jin 5y ago

Sounds like every multinational org with 100k+ emplyees, certainly this fits my own. Fun part is, there is no realistic way out of this. It only takes one obscure critical legacy app to add so much integration mess into whole ecosystem that nobody ever will be re-doing those. Plus business won't put in the budget necessary for practically no change from their point of view.

marginalia_nu 5y ago
I've worked at a 500 employee-company that had many of these issues, mostly from misguided attempts at copying the way FAANG worked without any regard for our own needs or scale. The entire code was a tangled mess of git submodules, transitive dependencies and the build system was contingent on a domain-specific language that was partially developed in-house and partially based on some unstable open source project, incredibly janky, and only two people got to see the sources for it or even knew how it really worked. Dumpster fire doesn't even begin to describe the outcome. There was always this strange spooky-leftpad-at-a-distance thing where stuff just kept inexplicably breaking, and nobody ever quite knew what had gone wrong, who to talk to, or when it was gonna get fixed.

AuthorizedCust 5y ago

Corollary: the documentation must be written to speak to the stupidest person in your team/company who you expect to use the product it describes. Otherwise, you may end up with a documented system that you can’t use or fix. Fortunately, that’s often me (a supervisor), so I get to be self-deprecating when conveying this requirement.

qchris 5y ago
I also find that hypothetical "interns" are great for this. Something about "this person without a degree, who probably hasn't been working for very long, who doesn't know your organization or who to ask, and is probably very scared of looking or sounding stupid." It doesn't necessarily imply or require stupidity, but in my experience on both sides of it (writing docs now, and having been that intern), they actually make pretty great litmus tests for documentation and how accessible organizational knowledge is.
geekbird 5y ago
When I write process/operating docs, I write it for the person who is on call, at 02:00 Sunday morning, after overindulging in alcohol Friday night. I don't assume stupid, just not firing on all cylinders due to being half awake and distracted.

nhumrich 5y ago

> hard to describe succinctly To me, what it sounds like you are describing is the developer experience. Or "devX".

tablespoon 5y ago

> These things might be reasonable in a startup with a dozen people, but we have over a hundred thousand. My opinion is that once an organization is a certain size, everything really needs to be written down and easily accessible. I shouldn't have to regularly ask people how to do basic things, it should be on a wiki. If it's complicated, it should be in a training video, or a user manual. Here's the root cause of your bug: https://agilemanifesto.org/ https://agilemanifesto.org/ > Individuals and interactions over processes and tools > Working software over comprehensive documentation I remember when my employer adopted agile. One of the first things they did was lay off their team of technical writers. Also you wasting your time flailing about probably makes some manager's budget numbers look good, because they got to "save" the money that would have prevented you from flailing about.

kodisha 5y ago
I you could once a year do +100 votes instead of regular +1 vote, I would do in right here, even though it's not even half of January.
commandlinefan 5y ago
> Here's the root cause of your bug Hang on, don't blame the agile manifesto here. Blame the management that adopted the agile manifesto as "do all the exact same things we used to do that didn't work, but call it 'agile'".
- tablespoon 5y ago
  > Hang on, don't blame the agile manifesto here. I totally can, and will. It's on them if they popularized slogans that can be too-easily misunderstood. That's meant that agile, in several respects, has been an exercise of throwing the baby out with the bathwater. > Blame the management that adopted the agile manifesto as "do all the exact same things we used to do that didn't work, but call it 'agile'". That's clearly not what my employer did.
  - headmelted 5y ago

PragmaticPulp 5y ago

> Basically, my job requires using a lot of tools and technology and processes created by other people, but none of those tools are easy to use or well documented. When I first moved into management roles I didn’t fully understand how much of my job would be simply saying “no” to a constant stream of engineers wanting to write or rewrite custom tools, libraries, frameworks and platforms when otherwise sufficiently good alternatives already existed. Not all engineers, but once you get into a big enough organization there are a lot of engineers who want to do meta-work and rework things instead of actually shipping things. The worst offender I worked for had meta-tools on top of everything. You couldn’t use GitHub, Slack, or even our cloud platform without first learning how to navigate all of the custom tools and bots that had been put together to gate access to everything and make it work. And like you said, it was poorly documented. It turned into little fiefdoms where you had to know who to talk to if you wanted to get past the systems to get things done. This can all be avoided with good management at an organizational level, but of course some managers want to get in on the situation because it puts them in a perceived stronger position in the company if their team is at the center of everything.

rthomas6 5y ago
It also helps to find the one extra weird engineer that loves to write documentation, then have them do that for a while for everything.
- 0x445442 5y ago
  What are the acceptance criteria for that story and how many points would it be?
Dave3of5 5y ago
> You couldn’t use GitHub, Slack, or even our cloud platform without first learning how to navigate all of the custom tools and bots that had been put together to gate access to everything and make it work. First time in a really big org and I can see this as well. The worst part is trying to tie someone down to explain how to use their custom tool or bot or even entire systems. How do I use that custom BI tool that you setup ... oh no-one wants to even send me an email reply when I ask a simple question about these things. The way most stuff gets done in these orgs is various maverick people who find some way to work around all this crap. The problem is you have to have political clout to do that most employees don't have that. I think these "meta-tool" are initially well intentioned but as you say they become empire building tools. Worst still is that the teams that build them often leave or are disbanded. Then no one is responsible and no one wants to touch them and they sit in this weird limbo state.

yowlingcat 5y ago

I think you are touching on the right problem but your solution seems a bit more on the idealistic rather than realistic side. I've had this conversation before where documentation is proposed as a silver procedural bullet and it never works. Documentation gets stale and eventually out of date. Do you really want to be in a position where you have to /recompile/ all of those documentation/training video/user manual materials? Unless you're selling it, probably not. The solution you are looking for is not documentation but automation and training. The things which can be automated to make the process closer to obvious should absolutely be automated. The things which are intrinsically more challenging and can't be solved through automation should be covered through training -- and not a "training video" but a real set of sessions where someone (likely your manager or a tech lead responsible for ensuring clean consumption of their modules) is walking you through the right way to do things and leaving time for questions. I say this having worked at a place that touted a documentation centric culture at the level of organizational scale you suggested. If you're doing anything remotely challenging, documentation only gets you to the starting line. It doesn't get you to being useful and autonomous.

Mezzie 5y ago

Good luck convincing organizations to invest in this + finding people to do it. I have the background to organize this (I'm actually doing this for my own, smaller organization right now just because the situation is driving me insane) and the main problems are: 1.) The people who can do this aren't going to do it for 30k or $10/hr. Properly solving this problem requires an understanding of tech and the development process, pedagogy/communication/instructional design, and information management. Lacking any one of those would result in a system that's an unusable vanity project. I have a couple of decades of small-shop coding experience, experience as an instructional designer at an Ivy, and graduate education in information management, but if you want me to use all of those it's going to cost you and companies are resistant to that. (See: How a lot of non tech product companies treat their IT teams since they don't generate revenue). Now imagine if the project were led by someone more qualified, which it should be for a company of your size. 2.) Most companies don't have a culture which would allow this to be done well. This is the sort of work which requires both a lot of honesty and a lot of careful planning. A lot of information ends up hidden or inaccessible due to people's egos, or executives not being able to handle the information so it's buried. People need to be honest about their workflows and what they don't understand, and they need to feel comfortable being honest about such things. This gets really messy. On the dev side, a common cultural issue is "I had to learn it, they should too" or the urge to make understanding the complexities into a shibboleth for the in-group. So I agree with you, but it's unlikely to change soon in my opinion.

fuzzer37 5y ago
Can I ask how you got started in the field of instructional design? This is a field that I've not really heard much about until recently, and I'm curious how someone breaks into that field.
- Mezzie 5y ago
  Unfortunately, I'm one of the old 'I fell into it because I had experience in both the field's components (tech and education)' stories: I've been coding (mostly but not exclusively front-end) since the 90s and been teaching techy things on and off about that long (I taught my first programming class in '99 when I was 11), and then I went and worked in libraries/studied how people learn outside of formal classroom contexts. So I already had all the skills needed once the field started. I cheated. If I were getting started now, my advice would be to remember that the human element and the tech element are of equal importance in Instructional Design. So you'd want to take stock of what skills you currently have and how best to a.) get the necessary knowledge you don't have and b.) take advantage of your particular skill set. Since you're on HN, I'm assuming you're of a techy bent vs. an educational/people bent. So if you have any teaching/tutoring/etc experience, I would lean on that. If not, I would get some. You can do this by volunteering somewhere + doing some basic research. I highly recommend this since a large portion of instructional design (like most front-end things) is planning for all the weird things humans do when interacting with your stuff and, more importantly for instructional design specifically, how people who can't computer at ALL think. When doing ID, you're often designing for people who can't figure out how an email works and are taking a work training, for example. Also since you're techy, I'd recommend spending some time with UI/UX and specifically focus on designing for different populations/audiences. From a tech perspective, your competing instructional designers are worst at the 'boring' things like accessibility guidelines, making sure things can run on terrible machines, coding fall-backs, etc. So a tech background is of great assistance as long as you communicate the problems it can help you avoid. As for the actual 'how to get a job', I literally applied for my first job off of Indeed out of grad school and I'm a first-generation student so trust me that was not the most polished resume or CV.

jensensbutton 5y ago

It's a problem of the incentives we've setup. We reward engineers for building things. Promotions, pay raises, etc. for engineers are typically based on impact and technical complexity, not how good their documentation was. I've been at a few places that handwave at such things in their level expectations or skills matrix or whatever, but having been in promo discussions at a variety of medium and large companies it always comes down to what was shipped. Problem is, at mature companies there aren't enough greenfield problems to match the large number of engineers who are looking to climb the ladder. So stuff is rewritten, then rewritten again since nobody understands the rewrite, and so on. Managers kind of look the other way because retention and the ecosystem becomes messier and messier over time as promoted folks leave and new folks come in that need to be promoted. This is the world of professional software development at pretty much every company not a startup.

sly010 5y ago

I admit I quickly searched my company directory to see if you work where I do :) The article mentions 3 things engineers care about: Technology, Intellection and Stability. Perhaps your pain fits into the 3rd category (if extrapolate a little). If so, ironically, the first 2 at scale is what causes the lack of the 3rd. When you make wider roads, the traffic will get worse, not better. Perhaps it's similar technology organizations. It's almost inevitable. Anecdotally Amazon figured this stuff out by mandating well defined organizational boundaries (kind of like solid white dividers on some section of the highway, eh?) and by making each team responsible for its own developer infrastructure. It seems wasteful at first, but maybe less so at scale. Maybe someone working there can confirm/deny whether this is the case.

Buttons840 5y ago

How many of us agree but haven't yet read our companies Confluence pages?

Kon-Peki 5y ago
My company is probably doing Confluence wrong. Because if you haven't bookmarked the page, you'll never find it again.