10 ms·

Big company tale: six months for a list and a button

178 points by parsecs 5y ago

muuglay 5y ago
This pains me deeply. As one of the people that can just crank shut out, I find myself frustrated. I then meditate that these people exist for when things go wrong so I can check out. Just rest and vest. Rest and vest is the way.
- tluyben2 5y ago
  This is not really a tech thing though; it's process. There are probably enough people there who can 'crank stuff out' too, but they cannot do that because of the process in place. A lot of companies end up this way when they get big, especially if they were not software companies to start with. There will be many management layers and gatekeepers who will do anything they can to prevent you from cranking out your stuff.
  - divbzero 5y ago
    There are scenarios, e.g. something mission-critical or highly-regulated, where process/gatekeeping is appropriate. The scenario described by OP is not one of them.
- deleted 5y ago
  [deleted]
xyzzy123 5y ago
What impresses me most about this is that $BigCo's dashboard team didn't block them from doing it themselves...
- lodovic 5y ago
  That would be typical, take ownership of another team's pet project, block that team from contributing further, take all the credits from management and let the project die. Seen it happen multiple times at my current gig.
  - thecoppinger 5y ago
    Happened to my personally at the gig I recently quit at. I thought I was the only one.
- vander_elst 5y ago
  If news spreads out that you can do things on your own, the dashboard team would lose its power and budget. Dashboard team has probably done it for years, they have a safe job: everyone needs dashboard and everyone needs to go through them. They can keep their annual budget and since the upper management probably has no vision they don't have to invest time, effort and take the risk to create a self service platform. A global optimum might disrupt a local optimum and they will fight against it.
- mbrodersen 5y ago
  The trick is to make your own dashboard but not calling it a dashboard. When they try to stop you, claim that it is not a dashboard (even though everybody knows it is a dashboard). Insist that they define, in writing, exactly what a dashboard is. Continue rejecting their definition with counter examples. Insist that the definition is unclear and needs to be defined in a formal language. Reject their formal definition after spending months being “too busy” to review it. Change the background colour/font/title and claim that it is changed so it needs to go through the review process again. Ask for their written definition of their review process. Insist that it is ambiguous and needs reviewing/fixing. Repeat forever.
  - spikej 5y ago
    You've clearly played the game
    - mbrodersen 5y ago
      When in Rome behave like the Romans :)
tluyben2 5y ago
I have had similar experiences multiple times and you definitely learn not to really care and let it be as this is what it is. It took a bunch of teams of senior devs and other people 1 year to translate some simple excel sheet with formulas into a little Java powered site on top of a database. The cost was a few $100k and the end result was something most people here on HN can do in an afternoon (max a few days), but tech was never the issue; it was the process & politics between the IT dep (which was in another country (Germany), the head office, the hired consultants (I was one of them) etc. My job was to steer the tech people at a high level (they had their own project manager etc; that was not my job) and the consultants (from another company) that did the implementation kept disagreeing with my approach and everyone else kept disagreeing with everything else (like the color of a button). In the end it was implemented like I wanted it in the first place (and they took the credit) and I heard, much later, from one of the seniors that their PM told them to disagree with me to pump up the final bill. Which worked; luckily I never worked 'with' them again after.
- merrvk 5y ago
  > their PM told them to disagree with me to pump up the final bill This explains a lot of my experiences now
  - treis 5y ago
    IMHO that would a rare to almost non-existant thing. Any sort of Mega Corp has an unending stream of IT work. There's no real need or desire to pump up the final bill. You'll make way more money in the long run coming in on time and under budget.
    - tluyben2 5y ago
      These are often long term projects; consultancy firms, especially big ones, are very good at delivering over time and over budget and still remaining there 10+ years. They are trained to do this and do it well. Maybe it depends on the region but it's very normal in Europe anyway for the Cap Gemini's etc to do this. Sure they have an unending stream of work, but you rather still do far less work for far more money or rather; be able to bill far more hours for the same end result is preferable for profits. The art is then to still have a ravingly enthusiastic client at the end of the project. It's an art. Or a scam. As a techie I find it the latter but I admire how they keep the client happy, time and time again.
megablast 5y ago
Um so?? It was on the roadmap. They had other work to do. I get you want everything right now, but it first work that way.
- Smaug123 5y ago
  I think you may have missed the point: the team with other work to do also insisted on taking on this work, and that team prevented our protagonist from taking the requisite two days to do the work.
  - dudul 5y ago
    If they were the dashboard team, I understand that they wanted to own it. Otherwise they end up being seemingly accountable for something they didn't do. Now it becomes this snowflake dashboard that is not owned by the dashboard team. Not saying they were right, just that I can see an argument.
citrin_ru 5y ago
On a bright side this allows small companies (free of such problems) to compete with big corps, which benefit from economy of scale and can invest into R&D much more.
l0k3ndr 5y ago
This is too real. But not only big companies, even startups with good funding and mid-sized team can behave like this.
- Zababa 5y ago
  I can confirm this, I've seen this happen in a startup that has ~40 people. The cause seem to be that devs aren't sensitive to business needs, and product managers/owners don't fix this.
segfaultbuserr 5y ago
See also: a Web Server Installation story from The Daily WTF. https://thedailywtf.com/articles/web-server-installation https://thedailywtf.com/articles/web-server-installation It took 33 days for the sysadmin to get access permissions, before the project was cancelled one more month later. Comment: So, everything went smoothly?
- ArnoVW 5y ago
  Amazing reading. Though I wonder if the migration is canceled, or that the Client simply lost faith after 2 months and simply found another solution.
MrDresden 5y ago
This mimics my mobile product team's battle with the marketing team that is supposed to own the companies 'tone-of-voice' in all material (digital, printed etc). Have multiple stories on waiting for resources, such as images and text that took weeks to get and then were not used as they didn't follow the guidlines given ('text can be max x char long', 'image resolutions need to be w times v'). We have all but given up on trying to involve them, as it will slow everything down to a crawl if we do.
davedx 5y ago
Ha I once tried to write a dashboard for our fledgling ecommerce platform at a relatively small company and hit a solid brick wall trying to get it deployed for all sorts of internal politics reasons. After months of back and forth I gave up. After about a year an entire new team was spun up with back end and front end devs to build something like the dashboard but… better… it took months of course. This isn’t just something that happens at BigCorps unfortunately. Any company can have an anti-innovation, anti-hacker culture like this.
- cutemonster 5y ago
  What can be ways to change the culture into pro innovation and cooperation
siscia 5y ago
I keep reading from this author and it always struck me as a very capable technical person, but as an awful person to work with. There was a piece few days ago about picking co-founder using the army way. Beside being a well written piece, it made clear one thing. The role of leaders in an organisation is to get stuff done, despite rules, regulations, hierarchy and all the whistle and bells that an organisation need to create in order to exist and sustain itself. (Which doesn't means disregard rules, it means find a way to make stuff happens in the context of the rules.) In the article there are dozen of things that are just expected given the conditions. But there are also a lot of stuff that the author could have done differently in my opinion. For instance figure out the resources that was suppose to create the application, pass by its office, offer its a coffee and communicate clearly why the things is important and how important it is and what is the goal of the project and finally reassure the resources by putting in whatever work tracking system all the details. These kind of human work and relationships building is fundamental. It seems easy to you just ask another department "please do this" and when naturally things takes too long to look reasonable complaining on the internet. There is a world of difference between: 1. Some random guy wants a boring dashboard and it is not very clear the reason and the why. 2. That cool engineer has a real problem that really bother it and its whole team and I can easily fix it. Yeah it is not a scalable way to solve problems but dumping work on some oscure Jira board is not scalable neither.
- noisy_boy 5y ago
  > For instance figure out the resources that was suppose to create the application, pass by its office, offer its a coffee and communicate clearly why the things is important and how important it is and what is the goal of the project and finally reassure the resources by putting in whatever work tracking system all the details. You are assuming that just because the manager can swing by at author's desk, their team is also at the same location. Not necessarily true. They could be sitting across the world, in a another dysfunctional team, they have a bi-weekly dysfunctional meeting and nothing functional comes out of it. Even when engineers are in the same place, some of those can be very bureaucratic/defensive/lazy/CYA type because bigco allows such people to exist in the shadows. Based on past experience, I'm only slightly exaggerating. > These kind of human work and relationships building is fundamental. Having said that I said, I also agree with this in the current context - that personal connection, when established, speeds things up. But then again, this doesn't always work out/is possible due to reasons mentioned above.
deleted 5y ago
[deleted]
ABeeSea 5y ago
So much of this story doesn’t make sense to me. What is a “dashboard team”? Is that data engineers and report developers or is it a dev-ops reporting team? When I think the former, I think ETLs, data warehouses, tableau/powerBI and when I think the later I think uptime, memory, cpu, fleet metrics, etc. in the FAANGs I’ve worked, individual service teams are responsible for their devops. You needed a single page that talked to your service and a button that contacted you. Put it in your team’s roadmap. Why would you outsource your service’s alerting infrastructure to a team that doesn’t seem to have any expertise or broader ownership of other alerting infrastructure? How were other services at “big company” handling alerting? (And why would you rely on a manual process for altering anyway) This story left me confused more than anything.
leroman 5y ago
Had very similar experiences at not even a "big company" it's more of a mentality of a "Big Co.". People hired to just not care, sit at meetings to make managers feel important and provide some pseudo progress to higer-ups to make things look like they are moving somewhere. In reality, I have not doubt a half decent motivated engineer can replace a group of these unmotivated developers, which I have seen done and did my self in the past. The real reason unmotivated engineers are there and things work this way in a company, in my experience, is bad management. Good people leave, who ever is left does what ever they can, those who care, suffer.
- nicbou 5y ago
  The last line is accurate. You either give up and leave, or stay and become part of them. It's really hard to stay and effect change.
  - tmitchel2 5y ago
    Yup nailed it
  - TomDavey 5y ago
    You just described the thesis of one of the wisest books I've ever read, Albert O. Hirschman's 1970 classic "Exit, Voice, and Loyalty." https://en.wikipedia.org/wiki/Exit%2C_Voice%2C_and_Loyalty https://en.wikipedia.org/wiki/Exit%2C_Voice%2C_and_Loyalty
    - cutemonster 5y ago
      Thanks! I read the whole Wikipedia page and a bit more (this far), started thinking about the book.
      - cutemonster 5y ago
        Btw that book explains why sometimes a police force or the military in a country can be recklessly violent, and consist largely of men who happily do all sorts of atrocities
deleted 5y ago
[deleted]
question000 5y ago
I work in an organization where the developers pretty much run the show, it sounds like heaven but we bump into churn situations like this all the time. The thing the author doesn't really realize in this article is that building consensus and making sure everyone agrees with the priority of the project is part of the job. It kinda sounds like he had pet project that others disliked so they tried to kill it by claiming the "dashboard team" was handling it. Then he attempted to vicariously micromanage the project from another team.
amelius 5y ago
They need a project management system.
- chiph 5y ago
  This probably is their project management system. As someone else pointed out, a request for an internal tool is going to be sorted behind something that is revenue-generating.
deleted 5y ago
[deleted]
somebodythere 5y ago
I wonder what was wrong with the shell script thing really, other than it didn't have the panic button.
- lrem 5y ago
  One possibility: kept boiling the oceans for no good reason 24/7 to generate something with three views per week.
- Rastonbury 5y ago
  Yeah was the acutal business benefit I wonder? How much time was spent by the author over wrangling and learning to do it from scratch, it does not seem worth it
tester756 5y ago
>"this is the only way we can maintain it". So, when you don't have XP with X tech, but you wrote your hacky solution and the person who has XP with X tech tells you that other_solution is more maintainble, then probably... He/She's right.
- milesvp 5y ago
  I have mixed feelings about this. I’ve seen stuff built by different departments that weren’t core developers. They definitely needed help making sure they had something that could be sustained by the company going forward. Similarly, I’ve been handed unmaintainable trash handed to me by teammates who should have had enough experience to know better. On top of this, I’ve come to appreciate that maintainability is very much context sensitive. What’s unmaintable on an expensive team of engineers can be totally maintainable on a team of cheap interns. Is the solution maintainable by the engineers better? arguably. But the solution maintainable by the interns may well be good enough. So, yeah. I try to be responsive to the needs and capabilities of teams when they request help from me in an organization.
TrackerFF 5y ago
I've been in a small company (< 50 employers) with this exact culture. But instead of it being a small dashboard function, pretty much everything took minimum 3-6 months. How did they survive being so inefficient? Huge gov. contracts, where everything takes time. I mean, perfect place if you just enjoy a steady paycheck, while working on side-projects and personal stuff - but even that gets boring, real quick.
physicles 5y ago
You can also use this essay to examine opportunities and pitfalls of horizontal coordination across organizations. I used to work at a big tech company, but now I’m one of a few cloud devs at a startup. One day we realized we needed a dashboard, so I spun up a server and put a single binary on it that saves data to SQLite and renders HTML server-side. Dumb stuff. Took 2 days, problem solved, been humming along for a year, no plans to rewrite it. Now let’s say another dev decided that they wanted to build another dashboard to monitor a service they just built. Obviously I’m gonna insist that they add on to the stuff I already built. I show them where to add the function (literally one function that makes an HTTP request and returns an error), it takes them 10 minutes instead of 2 days, and doesn’t add another entry point for attackers. But now let’s say our company now has 100 devs. There’s now more status info than can fit on a single page. If you don’t do this right, the dashboard communication/management overhead at some point grows larger than the cost to develop a new dashboard. What the dashboard team in OP’s article should’ve done is to create a self-service solution for making a new dashboard: here’s a big template, fill in whatever health checks you need in one of N languages, and tell us when you’re done and we’ll spot check the code and deploy it for you. That way they maintain control of all the dashboards, and teams get their dashboards way faster. (Am I missing a downside, other than that the dashboard team will need to downsize?)
- cutemonster 5y ago
  > dashboard team ... create a self-service solution for making a new dashboard > Am I missing a downside I don't think so, I'm just wondering, what to do if the dashboard team creates a buggy & hard to use, worse than nothing, dashboard self service, and everyone is supposed/required to use it > 100 devs. ... If you don’t do this right, the dashboard communication/management overhead at some point grows larger than the cost to develop a new dashboard. That's an interesting point!
  - physicles 5y ago
    > I'm just wondering, what to do if the dashboard team creates a buggy & hard to use, worse than nothing, dashboard self service, and everyone is supposed/required to use it That sounds like the situation in the OP's article. Maybe it would be best to address that as a political problem ("don't make us use this thing") instead of a technical one.
chockablocker 5y ago
My personal record is 2 years for a text box to be added to a form. To be fair, that's harder than it may sound. The textbox must be in the web version, the app, the api. It needs to be localized, and data stored must be compliant with GDPR etc. The list goes on. It doesn't help task velocity if the PMs switch, the thing gets de-prioretized, re-prioretized and there still being ongoing discussions that question the purpose of the textbox (maybe we should do videos instead of text). After 2 years I'm happy to report the textbox launched... :-/
utunbu 5y ago
> April 8: it seems there's now a mock-up of sorts from the team. Inside the group, we start talking about that situation where if you mail the $open_source_project mailing list asking for help with a legitimate problem, nothing happens, but if you make up a shitty version of something and fire it off, then suddenly 50 million people show up and go OI! DO IT THIS WAY! But, three months earlier when you politely asked for help, zip, nothing, nada, zilch. This seems quite interesting to me. I haven't really got chances to interact with mailing lists, is this really the case?
- haukem 5y ago
  Yes I would say this is a case when someone wants to contribute something to an open source project. Often these initial questions are very unspecific and often as a maintainer you only get the question, answer it and then never see this person again. Time wasted. When someone already wrote code and want to contribute it you know that the person already invested some time and is seriously about implementing some stuff. Here the likelihood that your time is wasted is much less.
  - utunbu 5y ago
    I see, this makes a lot sense. I guess perhaps community forums and IRCs are better places for those initial and unspecific questions?
    - MrDOS 5y ago
      IMO there are very few places where it's appropriate to ask unspecific questions. IRC and forums are “better” for them in the sense that there's a tighter response loop on narrowing down what the asker is actually interested in, but it's still a waste of time for everyone involved. Being able to ask questions well is a skill, and everyone has to learn it somehow, but it's generally better for everyone if the question includes too much situation-specific detail than too little.

178 points by parsecs 5y ago

This pains me deeply. As one of the people that can just crank shut out, I find myself frustrated. I then meditate that these people exist for when things go wrong so I can check out. Just rest and vest. Rest and vest is the way.

tluyben2 5y ago
This is not really a tech thing though; it's process. There are probably enough people there who can 'crank stuff out' too, but they cannot do that because of the process in place. A lot of companies end up this way when they get big, especially if they were not software companies to start with. There will be many management layers and gatekeepers who will do anything they can to prevent you from cranking out your stuff.
- divbzero 5y ago
  There are scenarios, e.g. something mission-critical or highly-regulated, where process/gatekeeping is appropriate. The scenario described by OP is not one of them.
deleted 5y ago
[deleted]

xyzzy123 5y ago

What impresses me most about this is that $BigCo's dashboard team didn't block them from doing it themselves...

lodovic 5y ago
That would be typical, take ownership of another team's pet project, block that team from contributing further, take all the credits from management and let the project die. Seen it happen multiple times at my current gig.
- thecoppinger 5y ago
  Happened to my personally at the gig I recently quit at. I thought I was the only one.
vander_elst 5y ago
If news spreads out that you can do things on your own, the dashboard team would lose its power and budget. Dashboard team has probably done it for years, they have a safe job: everyone needs dashboard and everyone needs to go through them. They can keep their annual budget and since the upper management probably has no vision they don't have to invest time, effort and take the risk to create a self service platform. A global optimum might disrupt a local optimum and they will fight against it.
mbrodersen 5y ago
The trick is to make your own dashboard but not calling it a dashboard. When they try to stop you, claim that it is not a dashboard (even though everybody knows it is a dashboard). Insist that they define, in writing, exactly what a dashboard is. Continue rejecting their definition with counter examples. Insist that the definition is unclear and needs to be defined in a formal language. Reject their formal definition after spending months being “too busy” to review it. Change the background colour/font/title and claim that it is changed so it needs to go through the review process again. Ask for their written definition of their review process. Insist that it is ambiguous and needs reviewing/fixing. Repeat forever.
- spikej 5y ago
  You've clearly played the game
  - mbrodersen 5y ago
    When in Rome behave like the Romans :)

tluyben2 5y ago

I have had similar experiences multiple times and you definitely learn not to really care and let it be as this is what it is. It took a bunch of teams of senior devs and other people 1 year to translate some simple excel sheet with formulas into a little Java powered site on top of a database. The cost was a few $100k and the end result was something most people here on HN can do in an afternoon (max a few days), but tech was never the issue; it was the process & politics between the IT dep (which was in another country (Germany), the head office, the hired consultants (I was one of them) etc. My job was to steer the tech people at a high level (they had their own project manager etc; that was not my job) and the consultants (from another company) that did the implementation kept disagreeing with my approach and everyone else kept disagreeing with everything else (like the color of a button). In the end it was implemented like I wanted it in the first place (and they took the credit) and I heard, much later, from one of the seniors that their PM told them to disagree with me to pump up the final bill. Which worked; luckily I never worked 'with' them again after.

merrvk 5y ago
> their PM told them to disagree with me to pump up the final bill This explains a lot of my experiences now
- treis 5y ago
  IMHO that would a rare to almost non-existant thing. Any sort of Mega Corp has an unending stream of IT work. There's no real need or desire to pump up the final bill. You'll make way more money in the long run coming in on time and under budget.
  - tluyben2 5y ago
    These are often long term projects; consultancy firms, especially big ones, are very good at delivering over time and over budget and still remaining there 10+ years. They are trained to do this and do it well. Maybe it depends on the region but it's very normal in Europe anyway for the Cap Gemini's etc to do this. Sure they have an unending stream of work, but you rather still do far less work for far more money or rather; be able to bill far more hours for the same end result is preferable for profits. The art is then to still have a ravingly enthusiastic client at the end of the project. It's an art. Or a scam. As a techie I find it the latter but I admire how they keep the client happy, time and time again.

megablast 5y ago

Um so?? It was on the roadmap. They had other work to do. I get you want everything right now, but it first work that way.

Smaug123 5y ago
I think you may have missed the point: the team with other work to do also insisted on taking on this work, and that team prevented our protagonist from taking the requisite two days to do the work.
- dudul 5y ago
  If they were the dashboard team, I understand that they wanted to own it. Otherwise they end up being seemingly accountable for something they didn't do. Now it becomes this snowflake dashboard that is not owned by the dashboard team. Not saying they were right, just that I can see an argument.

citrin_ru 5y ago

On a bright side this allows small companies (free of such problems) to compete with big corps, which benefit from economy of scale and can invest into R&D much more.

l0k3ndr 5y ago

This is too real. But not only big companies, even startups with good funding and mid-sized team can behave like this.

Zababa 5y ago
I can confirm this, I've seen this happen in a startup that has ~40 people. The cause seem to be that devs aren't sensitive to business needs, and product managers/owners don't fix this.

segfaultbuserr 5y ago

See also: a Web Server Installation story from The Daily WTF. https://thedailywtf.com/articles/web-server-installation https://thedailywtf.com/articles/web-server-installation It took 33 days for the sysadmin to get access permissions, before the project was cancelled one more month later. Comment: So, everything went smoothly?

ArnoVW 5y ago
Amazing reading. Though I wonder if the migration is canceled, or that the Client simply lost faith after 2 months and simply found another solution.

MrDresden 5y ago

This mimics my mobile product team's battle with the marketing team that is supposed to own the companies 'tone-of-voice' in all material (digital, printed etc). Have multiple stories on waiting for resources, such as images and text that took weeks to get and then were not used as they didn't follow the guidlines given ('text can be max x char long', 'image resolutions need to be w times v'). We have all but given up on trying to involve them, as it will slow everything down to a crawl if we do.

davedx 5y ago

Ha I once tried to write a dashboard for our fledgling ecommerce platform at a relatively small company and hit a solid brick wall trying to get it deployed for all sorts of internal politics reasons. After months of back and forth I gave up. After about a year an entire new team was spun up with back end and front end devs to build something like the dashboard but… better… it took months of course. This isn’t just something that happens at BigCorps unfortunately. Any company can have an anti-innovation, anti-hacker culture like this.

cutemonster 5y ago
What can be ways to change the culture into pro innovation and cooperation

siscia 5y ago

I keep reading from this author and it always struck me as a very capable technical person, but as an awful person to work with. There was a piece few days ago about picking co-founder using the army way. Beside being a well written piece, it made clear one thing. The role of leaders in an organisation is to get stuff done, despite rules, regulations, hierarchy and all the whistle and bells that an organisation need to create in order to exist and sustain itself. (Which doesn't means disregard rules, it means find a way to make stuff happens in the context of the rules.) In the article there are dozen of things that are just expected given the conditions. But there are also a lot of stuff that the author could have done differently in my opinion. For instance figure out the resources that was suppose to create the application, pass by its office, offer its a coffee and communicate clearly why the things is important and how important it is and what is the goal of the project and finally reassure the resources by putting in whatever work tracking system all the details. These kind of human work and relationships building is fundamental. It seems easy to you just ask another department "please do this" and when naturally things takes too long to look reasonable complaining on the internet. There is a world of difference between: 1. Some random guy wants a boring dashboard and it is not very clear the reason and the why. 2. That cool engineer has a real problem that really bother it and its whole team and I can easily fix it. Yeah it is not a scalable way to solve problems but dumping work on some oscure Jira board is not scalable neither.

noisy_boy 5y ago
> For instance figure out the resources that was suppose to create the application, pass by its office, offer its a coffee and communicate clearly why the things is important and how important it is and what is the goal of the project and finally reassure the resources by putting in whatever work tracking system all the details. You are assuming that just because the manager can swing by at author's desk, their team is also at the same location. Not necessarily true. They could be sitting across the world, in a another dysfunctional team, they have a bi-weekly dysfunctional meeting and nothing functional comes out of it. Even when engineers are in the same place, some of those can be very bureaucratic/defensive/lazy/CYA type because bigco allows such people to exist in the shadows. Based on past experience, I'm only slightly exaggerating. > These kind of human work and relationships building is fundamental. Having said that I said, I also agree with this in the current context - that personal connection, when established, speeds things up. But then again, this doesn't always work out/is possible due to reasons mentioned above.

deleted 5y ago

[deleted]

ABeeSea 5y ago

So much of this story doesn’t make sense to me. What is a “dashboard team”? Is that data engineers and report developers or is it a dev-ops reporting team? When I think the former, I think ETLs, data warehouses, tableau/powerBI and when I think the later I think uptime, memory, cpu, fleet metrics, etc. in the FAANGs I’ve worked, individual service teams are responsible for their devops. You needed a single page that talked to your service and a button that contacted you. Put it in your team’s roadmap. Why would you outsource your service’s alerting infrastructure to a team that doesn’t seem to have any expertise or broader ownership of other alerting infrastructure? How were other services at “big company” handling alerting? (And why would you rely on a manual process for altering anyway) This story left me confused more than anything.

leroman 5y ago

Had very similar experiences at not even a "big company" it's more of a mentality of a "Big Co.". People hired to just not care, sit at meetings to make managers feel important and provide some pseudo progress to higer-ups to make things look like they are moving somewhere. In reality, I have not doubt a half decent motivated engineer can replace a group of these unmotivated developers, which I have seen done and did my self in the past. The real reason unmotivated engineers are there and things work this way in a company, in my experience, is bad management. Good people leave, who ever is left does what ever they can, those who care, suffer.

nicbou 5y ago
The last line is accurate. You either give up and leave, or stay and become part of them. It's really hard to stay and effect change.
- tmitchel2 5y ago
  Yup nailed it
- TomDavey 5y ago
  You just described the thesis of one of the wisest books I've ever read, Albert O. Hirschman's 1970 classic "Exit, Voice, and Loyalty." https://en.wikipedia.org/wiki/Exit%2C_Voice%2C_and_Loyalty https://en.wikipedia.org/wiki/Exit%2C_Voice%2C_and_Loyalty
  - cutemonster 5y ago
    Thanks! I read the whole Wikipedia page and a bit more (this far), started thinking about the book.
    - cutemonster 5y ago
      Btw that book explains why sometimes a police force or the military in a country can be recklessly violent, and consist largely of men who happily do all sorts of atrocities

deleted 5y ago

[deleted]

question000 5y ago

I work in an organization where the developers pretty much run the show, it sounds like heaven but we bump into churn situations like this all the time. The thing the author doesn't really realize in this article is that building consensus and making sure everyone agrees with the priority of the project is part of the job. It kinda sounds like he had pet project that others disliked so they tried to kill it by claiming the "dashboard team" was handling it. Then he attempted to vicariously micromanage the project from another team.

amelius 5y ago

They need a project management system.

chiph 5y ago
This probably is their project management system. As someone else pointed out, a request for an internal tool is going to be sorted behind something that is revenue-generating.

deleted 5y ago

[deleted]

somebodythere 5y ago

I wonder what was wrong with the shell script thing really, other than it didn't have the panic button.

lrem 5y ago
One possibility: kept boiling the oceans for no good reason 24/7 to generate something with three views per week.
Rastonbury 5y ago
Yeah was the acutal business benefit I wonder? How much time was spent by the author over wrangling and learning to do it from scratch, it does not seem worth it

tester756 5y ago

>"this is the only way we can maintain it". So, when you don't have XP with X tech, but you wrote your hacky solution and the person who has XP with X tech tells you that other_solution is more maintainble, then probably... He/She's right.

milesvp 5y ago
I have mixed feelings about this. I’ve seen stuff built by different departments that weren’t core developers. They definitely needed help making sure they had something that could be sustained by the company going forward. Similarly, I’ve been handed unmaintainable trash handed to me by teammates who should have had enough experience to know better. On top of this, I’ve come to appreciate that maintainability is very much context sensitive. What’s unmaintable on an expensive team of engineers can be totally maintainable on a team of cheap interns. Is the solution maintainable by the engineers better? arguably. But the solution maintainable by the interns may well be good enough. So, yeah. I try to be responsive to the needs and capabilities of teams when they request help from me in an organization.

TrackerFF 5y ago

I've been in a small company (< 50 employers) with this exact culture. But instead of it being a small dashboard function, pretty much everything took minimum 3-6 months. How did they survive being so inefficient? Huge gov. contracts, where everything takes time. I mean, perfect place if you just enjoy a steady paycheck, while working on side-projects and personal stuff - but even that gets boring, real quick.

physicles 5y ago

You can also use this essay to examine opportunities and pitfalls of horizontal coordination across organizations. I used to work at a big tech company, but now I’m one of a few cloud devs at a startup. One day we realized we needed a dashboard, so I spun up a server and put a single binary on it that saves data to SQLite and renders HTML server-side. Dumb stuff. Took 2 days, problem solved, been humming along for a year, no plans to rewrite it. Now let’s say another dev decided that they wanted to build another dashboard to monitor a service they just built. Obviously I’m gonna insist that they add on to the stuff I already built. I show them where to add the function (literally one function that makes an HTTP request and returns an error), it takes them 10 minutes instead of 2 days, and doesn’t add another entry point for attackers. But now let’s say our company now has 100 devs. There’s now more status info than can fit on a single page. If you don’t do this right, the dashboard communication/management overhead at some point grows larger than the cost to develop a new dashboard. What the dashboard team in OP’s article should’ve done is to create a self-service solution for making a new dashboard: here’s a big template, fill in whatever health checks you need in one of N languages, and tell us when you’re done and we’ll spot check the code and deploy it for you. That way they maintain control of all the dashboards, and teams get their dashboards way faster. (Am I missing a downside, other than that the dashboard team will need to downsize?)

cutemonster 5y ago
> dashboard team ... create a self-service solution for making a new dashboard > Am I missing a downside I don't think so, I'm just wondering, what to do if the dashboard team creates a buggy & hard to use, worse than nothing, dashboard self service, and everyone is supposed/required to use it > 100 devs. ... If you don’t do this right, the dashboard communication/management overhead at some point grows larger than the cost to develop a new dashboard. That's an interesting point!
- physicles 5y ago
  > I'm just wondering, what to do if the dashboard team creates a buggy & hard to use, worse than nothing, dashboard self service, and everyone is supposed/required to use it That sounds like the situation in the OP's article. Maybe it would be best to address that as a political problem ("don't make us use this thing") instead of a technical one.

chockablocker 5y ago

My personal record is 2 years for a text box to be added to a form. To be fair, that's harder than it may sound. The textbox must be in the web version, the app, the api. It needs to be localized, and data stored must be compliant with GDPR etc. The list goes on. It doesn't help task velocity if the PMs switch, the thing gets de-prioretized, re-prioretized and there still being ongoing discussions that question the purpose of the textbox (maybe we should do videos instead of text). After 2 years I'm happy to report the textbox launched... :-/

utunbu 5y ago

> April 8: it seems there's now a mock-up of sorts from the team. Inside the group, we start talking about that situation where if you mail the $open_source_project mailing list asking for help with a legitimate problem, nothing happens, but if you make up a shitty version of something and fire it off, then suddenly 50 million people show up and go OI! DO IT THIS WAY! But, three months earlier when you politely asked for help, zip, nothing, nada, zilch. This seems quite interesting to me. I haven't really got chances to interact with mailing lists, is this really the case?

haukem 5y ago
Yes I would say this is a case when someone wants to contribute something to an open source project. Often these initial questions are very unspecific and often as a maintainer you only get the question, answer it and then never see this person again. Time wasted. When someone already wrote code and want to contribute it you know that the person already invested some time and is seriously about implementing some stuff. Here the likelihood that your time is wasted is much less.
- utunbu 5y ago
  I see, this makes a lot sense. I guess perhaps community forums and IRCs are better places for those initial and unspecific questions?
  - MrDOS 5y ago
    IMO there are very few places where it's appropriate to ask unspecific questions. IRC and forums are “better” for them in the sense that there's a tighter response loop on narrowing down what the asker is actually interested in, but it's still a waste of time for everyone involved. Being able to ask questions well is a skill, and everyone has to learn it somehow, but it's generally better for everyone if the question includes too much situation-specific detail than too little.