6 ms·

I wonder what company they’re describing here. It sounds like so many self inflicted problems that that you could undo or set right in a couple of weeks if you

by jitl 2y ago

I wonder what company they’re describing here. It sounds like so many self inflicted problems that that you could undo or set right in a couple of weeks if you had the time and latitude to make changes across the system instead of being confined to a small area of team ownership.

bartread 2y ago
I worked somewhere that had a lot of this sort of thing going on once. You cannot overestimate how hard it is to get anything done: politics and organisational dysfunction, not to mention that you probably don’t have access to half of what you need to in order to fix any given problem and are even more unlikely to be able to get it, mean there are just huge scads of problems that, on the face of them, look relatively straightforward to solve but which, in practice, are organisationally impossible to solve.
- whstl 2y ago
  Yep. I don't work with data engineering, but from hearing war stories from them, this could perfectly describe the last four companies I worked at. :/ > you probably don’t have access to half of what you need to in order to fix any given problem and are even more unlikely to be able to get it I once sat down with a data engineer to try to fix a specific problem they had and that was 100% accurate. They were left to die by Ops and CISO.
- Terr_ 2y ago
  A framing/question I like to ask use is: "Look for a root problem that ought to be fixed through a change in policy, politics, or incentives, and the wasteful use of time/money is how the company tries to avoid or defer facing it." For example, Operations might demand that Engineering develops an increasingly-byzantine approvals process, to stop Sales from over-promising impossible or unprofitable projects.
- deergomoo 2y ago
  > you probably don’t have access to half of what you need to in order to fix any given problem I’ve found this to be one of my largest day-to-day problems even in a relatively functional organisation. Particularly when it involves something I can’t run on my own machine, like an AWS service. In a previous role I often found myself constructing elaborate hypotheses about what was going on inside systems I couldn’t see into. I’d then need to try to verify it with someone on another team, in another timezone, who had the access but not necessarily the development background. Which usually meant getting on a screen share and asking them to click various things I wasn’t allowed to. If I was wrong, back to the drawing board and start again.
deleted 2y ago
[deleted]
knallfrosch 2y ago
You're operating in the same company that let this happen in the first place. And — not to take away from the piece – If noone noticed the logs have been garbage for two years, they can't be that important.
- pavel_lishin 2y ago
  The logs are unimportant until suddenly they're critical.
- moregrist 2y ago
  It all seems easy to fix until you end up in a place where management process (and thus politics) has become more important than outcomes. The people with experience and knowledge get pushed aside in favor of someone who can talk management-speak and (often) looks the role. Suddenly meetings where Things Get Decided only include managers who don’t know what they’re doing. The best is when they make a mess and then get tremendous accolades for half-fixing it. I’ve even seen it happen within relatively small startups. It’s a sign of rotting culture, but sometimes you have a mortgage to pay and have to get comfortable with the situation until you can find a better gig.
- ludicity 2y ago
  I wrote a piece last year on how they were running a $500K Snowflake bill due to typing one number in wrong, and how I noticed within my first few weeks there by literally eyeballing something and going "queries take milliseconds so you really have to justify everywhere a minute appears in the configuration". The logs are stupid (sorta) but imagine how many other issues can exist if we uncover something like this every time we open something.
taberiand 2y ago
Irreducible complexity and load-bearing bugs. Kludge built upon kludge built upon kludge. An engineer with the go ahead and all the support required would still struggle because the knock on effects of any one change cascade out in unpredictable ways. Not to mention working in an active environment where, although the other engineers support the goals in theory, they still need to deliver business requirements for a fickle management that doesn't truly understand what lurks beneath the facade - they don't have time to do it the right way, and if they try they'll just break both the old system and the new.
tiew9Vii 2y ago
Sadly it sounds like most AU corporate companies I've worked at.
- chubs 2y ago
  Why are we so crummy? Is it because our talent finds it generally easy to migrate to the states for better opportunities, because of no language barrier and the ‘coalition of the willing’ visa (iirc)?
  - ykonstant 2y ago
    This must be one reason (not Australian, but have friends there), but more importantly it seems to me the "better opportunities" are much, much better. Australia seems to have worker rights that make most of the US look decent by comparison while wages remain far below the latter---and I don't mean just US hot spots: an Australian friend went from Sydney to Salt Lake City (!) to get a much better job with much higher compensation and better rights/environment. In Utah!
    - jpmoral 2y ago
      > Australia seems to have worker rights that make most of the US look decent by comparison This is the first I've heard anyone say that. Can you elaborate? We've got more leave (separate holiday and sick leave), none of the "at-will" stuff, right to disconnect, less hours, etc.
      - BrandonM 2y ago
        US software engineer. I have 24 days of PTO, 15 company holidays, and 9 sick days. 10 weeks for parental leave (16 for moms). $240K salary, $400-600K in annual vesting equity. That’s private paper equity, but I’ve already been able to cash out $700K and buy a house with cash. Fully remote. I can expense $120/month for phone and internet, and a few lunches each month, too. I can get a new laptop and/or monitor sent to me just by asking. When I do visit the office, the trip is fully expensed. Free daily lunch. Coffee and drinks and snacks everywhere, free. Private desks in a semi-open office with couches scattered around. Lounges with hundreds of board games, nearly all of which have seen table time during work hours. Primary projects are tracked in a knowledge-sharing system, but I can mostly work on what I want to. I’m encouraged to merge small fixes and refactors without any ticket-pushing at all. Yelling by managers or anyone would not be tolerated. “At-will” is more FUD than reality in my experience. Most companies, when firing or laying someone off, give something like 2 weeks of severance for every year of service.
lelandbatey 2y ago
The typical problem is a culture where if you say "I would like to make this systemically better by addressing our second order problems that are causing our very visible first order problems" You will be told that we absolutely do not have time for that. The only actions you're allowed to take are fighting the fires closest to you, not turning off the pumps that spray the gasoline everywhere. Typically this is only even possible because nothing that you're doing is actually used or scrutinized since if it was, someone would have immediately noticed that nothing works. Usually this is at places running on varying levels of investment dollar three-card-montey.
3eb7988a1663 2y ago
I inherited a pipeline like this. It is as if everything is a global variable. You cannot "just fix" one thing in isolation, because some spooky action at a distance of which you were unaware relies upon this insane behavior. Each and every hack is the expected input somewhere else in the chain. You have to carefully inspect everything downstream of any kind of minor adjustment because your cleanup is quite likely to break something else. Immensely frustrating and draining where you can have accomplished ~nothing in a full day of work to fix what should have been a five minute change.
- ludicity 2y ago
  I'm the author. You are exactly correct. Everything was so heavily interwoven that it was impossible to tell what would happen downstream without making an edit and then tracking the changes through dozens of steps through the architecture diagram.
  - baq 2y ago
    `The purpose of the system is what it does.’
  - jitl 2y ago
    In this situation I’d make a second copy of a bad thing and then try to make the copy good, instead of changing anything in place. But yeah, I totally get the huge upstream fight it must be and I’m not trying to backseat drive… just marveling at how the organizational fuckups & constraints make it hard to fix obvious problems
    - ludicity 2y ago
      You're right. I actually did propose this, but the other difficulty is getting stuff put onto the Jira board. My belief, yet to be verified, is that it was possible to deliver everything the executives promised AND perform the refactoring without any drop in service, but this would have required convincing management that a well-run team could 3x productivity. Eventually everyone got tired of talking to people that were just used to bad performance, so they couldn't envision smooth CI/CD and happy workers (as opposed to contented to get paid to hang around workers, which is what happy means in many cases).

by jitl 2y ago

bartread 2y ago

I worked somewhere that had a lot of this sort of thing going on once. You cannot overestimate how hard it is to get anything done: politics and organisational dysfunction, not to mention that you probably don’t have access to half of what you need to in order to fix any given problem and are even more unlikely to be able to get it, mean there are just huge scads of problems that, on the face of them, look relatively straightforward to solve but which, in practice, are organisationally impossible to solve.

whstl 2y ago
Yep. I don't work with data engineering, but from hearing war stories from them, this could perfectly describe the last four companies I worked at. :/ > you probably don’t have access to half of what you need to in order to fix any given problem and are even more unlikely to be able to get it I once sat down with a data engineer to try to fix a specific problem they had and that was 100% accurate. They were left to die by Ops and CISO.
Terr_ 2y ago
A framing/question I like to ask use is: "Look for a root problem that ought to be fixed through a change in policy, politics, or incentives, and the wasteful use of time/money is how the company tries to avoid or defer facing it." For example, Operations might demand that Engineering develops an increasingly-byzantine approvals process, to stop Sales from over-promising impossible or unprofitable projects.
deergomoo 2y ago
> you probably don’t have access to half of what you need to in order to fix any given problem I’ve found this to be one of my largest day-to-day problems even in a relatively functional organisation. Particularly when it involves something I can’t run on my own machine, like an AWS service. In a previous role I often found myself constructing elaborate hypotheses about what was going on inside systems I couldn’t see into. I’d then need to try to verify it with someone on another team, in another timezone, who had the access but not necessarily the development background. Which usually meant getting on a screen share and asking them to click various things I wasn’t allowed to. If I was wrong, back to the drawing board and start again.

deleted 2y ago

[deleted]

knallfrosch 2y ago

You're operating in the same company that let this happen in the first place. And — not to take away from the piece – If noone noticed the logs have been garbage for two years, they can't be that important.

pavel_lishin 2y ago
The logs are unimportant until suddenly they're critical.
moregrist 2y ago
It all seems easy to fix until you end up in a place where management process (and thus politics) has become more important than outcomes. The people with experience and knowledge get pushed aside in favor of someone who can talk management-speak and (often) looks the role. Suddenly meetings where Things Get Decided only include managers who don’t know what they’re doing. The best is when they make a mess and then get tremendous accolades for half-fixing it. I’ve even seen it happen within relatively small startups. It’s a sign of rotting culture, but sometimes you have a mortgage to pay and have to get comfortable with the situation until you can find a better gig.
ludicity 2y ago
I wrote a piece last year on how they were running a $500K Snowflake bill due to typing one number in wrong, and how I noticed within my first few weeks there by literally eyeballing something and going "queries take milliseconds so you really have to justify everywhere a minute appears in the configuration". The logs are stupid (sorta) but imagine how many other issues can exist if we uncover something like this every time we open something.

taberiand 2y ago

Irreducible complexity and load-bearing bugs. Kludge built upon kludge built upon kludge. An engineer with the go ahead and all the support required would still struggle because the knock on effects of any one change cascade out in unpredictable ways. Not to mention working in an active environment where, although the other engineers support the goals in theory, they still need to deliver business requirements for a fickle management that doesn't truly understand what lurks beneath the facade - they don't have time to do it the right way, and if they try they'll just break both the old system and the new.

tiew9Vii 2y ago

Sadly it sounds like most AU corporate companies I've worked at.

chubs 2y ago
Why are we so crummy? Is it because our talent finds it generally easy to migrate to the states for better opportunities, because of no language barrier and the ‘coalition of the willing’ visa (iirc)?
- ykonstant 2y ago
  This must be one reason (not Australian, but have friends there), but more importantly it seems to me the "better opportunities" are much, much better. Australia seems to have worker rights that make most of the US look decent by comparison while wages remain far below the latter---and I don't mean just US hot spots: an Australian friend went from Sydney to Salt Lake City (!) to get a much better job with much higher compensation and better rights/environment. In Utah!
  - jpmoral 2y ago
    > Australia seems to have worker rights that make most of the US look decent by comparison This is the first I've heard anyone say that. Can you elaborate? We've got more leave (separate holiday and sick leave), none of the "at-will" stuff, right to disconnect, less hours, etc.
    - BrandonM 2y ago
      US software engineer. I have 24 days of PTO, 15 company holidays, and 9 sick days. 10 weeks for parental leave (16 for moms). $240K salary, $400-600K in annual vesting equity. That’s private paper equity, but I’ve already been able to cash out $700K and buy a house with cash. Fully remote. I can expense $120/month for phone and internet, and a few lunches each month, too. I can get a new laptop and/or monitor sent to me just by asking. When I do visit the office, the trip is fully expensed. Free daily lunch. Coffee and drinks and snacks everywhere, free. Private desks in a semi-open office with couches scattered around. Lounges with hundreds of board games, nearly all of which have seen table time during work hours. Primary projects are tracked in a knowledge-sharing system, but I can mostly work on what I want to. I’m encouraged to merge small fixes and refactors without any ticket-pushing at all. Yelling by managers or anyone would not be tolerated. “At-will” is more FUD than reality in my experience. Most companies, when firing or laying someone off, give something like 2 weeks of severance for every year of service.

lelandbatey 2y ago

The typical problem is a culture where if you say "I would like to make this systemically better by addressing our second order problems that are causing our very visible first order problems" You will be told that we absolutely do not have time for that. The only actions you're allowed to take are fighting the fires closest to you, not turning off the pumps that spray the gasoline everywhere. Typically this is only even possible because nothing that you're doing is actually used or scrutinized since if it was, someone would have immediately noticed that nothing works. Usually this is at places running on varying levels of investment dollar three-card-montey.

3eb7988a1663 2y ago

I inherited a pipeline like this. It is as if everything is a global variable. You cannot "just fix" one thing in isolation, because some spooky action at a distance of which you were unaware relies upon this insane behavior. Each and every hack is the expected input somewhere else in the chain. You have to carefully inspect everything downstream of any kind of minor adjustment because your cleanup is quite likely to break something else. Immensely frustrating and draining where you can have accomplished ~nothing in a full day of work to fix what should have been a five minute change.

ludicity 2y ago
I'm the author. You are exactly correct. Everything was so heavily interwoven that it was impossible to tell what would happen downstream without making an edit and then tracking the changes through dozens of steps through the architecture diagram.
- baq 2y ago
  `The purpose of the system is what it does.’
- jitl 2y ago
  In this situation I’d make a second copy of a bad thing and then try to make the copy good, instead of changing anything in place. But yeah, I totally get the huge upstream fight it must be and I’m not trying to backseat drive… just marveling at how the organizational fuckups & constraints make it hard to fix obvious problems
  - ludicity 2y ago
    You're right. I actually did propose this, but the other difficulty is getting stuff put onto the Jira board. My belief, yet to be verified, is that it was possible to deliver everything the executives promised AND perform the refactoring without any drop in service, but this would have required convincing management that a well-run team could 3x productivity. Eventually everyone got tired of talking to people that were just used to bad performance, so they couldn't envision smooth CI/CD and happy workers (as opposed to contented to get paid to hang around workers, which is what happy means in many cases).