14 ms·

Get me out of data hell

698 points by pavel_lishin 2y ago

jaygreco 2y ago
I really like the author’s writing style here. The quips about the tea especially.
- remram 2y ago
  It reminds me of the classic series on /r/talesfromtechsupport by airz23. It's uncanny. They would almost always feature a "I sip my coffee. It tastes like <feelings at the time>" Example: https://www.reddit.com/r/talesfromtechsupport/comments/203qxp/securityis_it_a_thing/ https://www.reddit.com/r/talesfromtechsupport/comments/203qx...
- alex_suzuki 2y ago
  Same. For me the association was the Bastard Operator From Hell, with a darker, more cynical view of life. Just very enjoyable reading.
- ludicity 2y ago
  Very much appreciate the kind words! From you and the other commenters too.
ctippett 2y ago
> of course, we're serverless, because how can you hurt yourself without a cutting-edge? A beautiful epigram.
- storafrid 2y ago
  Yes, but I'm surprised that they attribute "cutting-edge" to Lambda. It's about as old as Docker.
  - deleted 2y ago
    [deleted]
  - zdragnar 2y ago
    I was part of a company that went all in on using lambdas for the majority of their web facing APIs. That was 7 years ago. The cutting edge bit is a nice quip, though I agree not exactly accurate anymore.
    - stackskipton 2y ago
      I worked for a company that went all in on Lambda as well. The knots they had to twist themselves into so that everything ran nice and smooth in Lambda environment was mindboggling. We have certain actions like orders that would pass through 8 Lambdas before completion because of execution time or just the big code base would result in 7 seconds start up time (node) so it would get broken down. If any of them failed, and it felt like failed a ton due to Amazon backend stuff, it was a nightmare to resolve. All of it could probably been handled by larger node application in docker container somewhere but AUTO SCALING, FAILOVER, SERVERLESS! Once I started as SRE for a new team, we built a larger monolith using Node and docker on EC2. We would get massive complements for our uptime and reliability but there were some architects extremely unhappy when I revealed in division presentation that it was just Docker + m4.xlarge running Ubuntu 18.04. When I left, more and more Lambdas were being broken down into docker running on EC2. They are probably on some container managed solution now.
  - sph 2y ago
    It sounds like you like to deal with much sharper edges than I am comfortable with. Or maybe I am too old with this shit. Still haven't found a use for "serverless" knives.
  - hinkley 2y ago
    For a bank, Lambda is brand new. You haven’t worked in conservative industries I take it? Late adopters, every one. OP is still trying to replace Cobol. I know an insurance company that started that process 15 years ago. Fifteen years.
- the_af 2y ago
  On one hand, yes. Beautiful. But on the other, the same sentence could be written about software deployed to traditional servers. "Because of course, how can you hurt yourself without the joys of badly configured servers?".
  - skydhash 2y ago
    You can hurt yourself with a badly held butter knife, and you can hurt yourself juggling katanas. Which situation would get people saying you're crazy?
    - zdragnar 2y ago
      Well, if you narrow down the metaphor to just knives, a dull knife is more dangerous to a chef than a sharp knife, because you need to apply more pressure and you get less control over the cutting action.
      - downut 2y ago
        The people juggling aren't chefs.
        zdragnar 2y ago
        Clearly, you haven't seen a good chef at work.
        the_af 2y ago
        Dull knives are dangerous to most people, not just chefs. Most beginner's cooking books/lessons will tell you to keep your knives very sharp, because dull knives are dangerous (for the reasons explained by grandparent comment). This affects amateurs just as much (if not more) as experienced chefs.
      - hinkley 2y ago
        Yes, but which one will people say you’re crazy?
- layoric 2y ago
  Came here to praise the same sentence, well done author, gave me a good laugh!
salt-thrower 2y ago
Beautifully written and fun to read. Blog posts like this give me a boost of mental strength to keep going during my worst episodes of burnout.
reverius42 2y ago
> I've even degraded team morale because I've convinced some of the engineers that things should be better, but not management, so now some of the engineers are upset. Oof, that hits a little close to home.
- ludicity 2y ago
  The good news is that, since the others are also looking for work elsewhere, there will be more engineers out in gen pop that actually thinks tests are useful, hah.
- hinkley 2y ago
  I have this illusion in my head that I stayed so long at my last company that almost all of my favorite people left, but one of my coworkers had my number. After a person I liaised with on another team left, I asked his superior if there was someone else I should build bridges with. We started talking about one of the team members and he said, “I don’t want you to talk to him. We like him, and if you talk to him he’ll leave.” This was on Slack so I don’t know if this was a jest or he was serious/mad. But it’s entirely true. I’ve convinced at least half a dozen people that we should expect better from a team environment and ourselves, and that this org (not the whole company, just this division) is a cult of stupidity. I was trying to recruit collaborators to fix the bullshit but apparently they decided it would be much easier to just start over.
  - refulgentis 2y ago
    There's some finer points to parse in your comment that I'm not 100% on (ex. if "this org" is your division or the partner division), so I'm out on a ledge a little bit here, might not relate to what you meant. I was lucky enough to get ~6 years running my own tech company after 6 years as a waiter. Then I sold it, yadda yadda, went to Google 6 months later, got ~7 years in there. It really, really, really, disturbed me how approximately every situation, in every division, with any people, ended up being boiling down to "how do we muddle through one more day without challenging anyones preconceptions", 95% of the time it was tribal antisocial stuff, and no one would speak up about it. Direct example, for posterity. I don't wanna speak too directly to it, so lets imagine Google Division A (hereafter, dApps). New division lead (ex-dApps) joins dBytes with apparent bias against partnering division (dConsumer). Despite the project being previously framed as top priority, new lead consistently undermines dConsumer in meetings and shows little interest in understanding their work. Team adopts leader's negative attitude, becoming obstructive and uncooperative. I ended up carrying a critical launch, virtually alone, for 6 months. At performance review time, my boss questions why I didn't get more team involvement - despite the hostile environment that prevented exactly that - and speaks glowingly about how we need to support peer going for promotion based on their excellent job on part Y...which they didn't do. They spent 2 days on it then said it was impossible. And they were definitively the most cooperative because at least they tried, and wouldn't actively be aggressive in meetings with the outgroup. Everything, always, came down to: A) don't cause conflict at all, at home, or you will be buried B) we'll bend over backwards to accomodate conflict you invent, as long as we can clearly define them as an out-group with 0 ability to affect us day to day.
SynasterBeiter 2y ago
I hate whining like these. Just do your job and get over it. No need to be theatrical about it.
- incognito124 2y ago
  A creative outlet is a need, to some. You sound like OP's employees.
- baq 2y ago
  Let the man live will you. Not everyone wants to be a drone for hire.
- hobs 2y ago
  If this was a job with a non-abstract input and output process they would have OSHA saying it was genuinely unsafe (and definitely stupid.) Many of us build systems to manage automatic actions to take care of this stuff, if I had an engineer I had to take 100+ steps to get something done I would definitely be considering 1) What the hell am I paying for and 2) Why the hell am I paying for it?
- hprotagonist 2y ago
  you know, he had a post about exactly this comment not too long ago! edit: https://ludic.mataroa.blog/blog/the-violent-role-of-relentless-positivity-in-the-workplace/ https://ludic.mataroa.blog/blog/the-violent-role-of-relentle...
  - tgv 2y ago
    That's pretty good. I'm somewhat curious now about his writing aspirations.
    - ludicity 2y ago
      I was offered a book deal and turned it down, hah. I read in spectacular volumes and books have a very special place in my heart. Growing up in Malaysia, most of my English was initially mastered from a gigantic pile of Enid Blyton books my family had lying around after the British occupation. We even give everyone at Hermit Tech a day off per week to study non-IT things and let the subconscious do some processing on work. Not because I'm a weirdo (though I am), but because I genuinely believe this produces a higher-quality experience for our clients, and no one can stop me from testing things like "five day work weeks are too long". In any case, the book deal had constraints like "no swearing", and it was implied they'd find their own artist for the cover. I didn't even intend to swear, but I care too much to let them assign an editor and impose arbitrary constraints. I ended up chatting a bit with Ed Zitron after the AI rant article went super viral, and he told me that I have the audience to just publish my own books. So I'm doing just that! I'm starting with ten short stories, at the advice of someone in the local Melbourne scene that has helped many people publish. Then I'm aiming to write one book containing a series of essays on IT work, and one fantasy/fiction book. I'm a reasonably good judge of popularity, and because neither will be particularly angry, I will probably only sell a few hundred copies. But I'll have a book with my name on it, which is very special to me.
brianhorakh 2y ago
Wonderful prose. I am in Melbourne also. I possibly used to work at the same place but I'm not sure. I resigned due to the night terrors caused by the cyber security issues I saw everywhere. The more I explored and understood the more sleep I lost.
- ludicity 2y ago
  Hah, yes, I'm not in cybersecurity but am very close to a few people that are. The incompetence is not evenly distributed and not as bad as it is in data, but some companies are in terrible states, and the stakes are much higher.
- hinkley 2y ago
  I worked with an Aussie who was in the US on H1B because the aerospace industry was even worse than the status quo in Australia. Last I heard he went back. I sincerely hope he switched industry verticals.
zombiwoof 2y ago
Data “engineering” is where all the cool kids go with no clue and create insane architectures to justify their incompetence
- hobs 2y ago
  As a data engineer I have seen absolutely bullshit pass for production, but it doesn't seem that different from all the other bullshit I have seen people deploy in my life. It is one of the few types of jobs I have worked were someone credulously offering adding five more layers to fix an issue with latency is a normal operating procedure though.
- parpfish 2y ago
  data engineer gives data science a run for their money when it comes to ambiguous job expectations. I’ve seen it mean anything from “distributed computing expert” to “knows SQL”
  - marcosdumay 2y ago
    > to “knows SQL” There are still ways to go. I've seen it means "spends the days filling excel spreadsheets".
  - sgarland 2y ago
    > knows SQL If by that you mean, “knows the commands to create, fill, and select from a table,” then yes. If you mean, “knows how to create a performant schema and queries that will serve them well into the future,” then no, absolutely not. OTOH, IME data folk are much more cheerful and willing to change things than devs when I point out the innumerable ways their DB choices are choking them. Devs more often fall on the side of “we don’t have time for that on our roadmap; can’t you just fix it?”
    - zelphirkalt 2y ago
      It is often not the devs telling you about the roadmap, but the management. Devs are more willing to fix things, if they are broken, but are not given the time.
jitl 2y ago
I wonder what company they’re describing here. It sounds like so many self inflicted problems that that you could undo or set right in a couple of weeks if you had the time and latitude to make changes across the system instead of being confined to a small area of team ownership.
- bartread 2y ago
  I worked somewhere that had a lot of this sort of thing going on once. You cannot overestimate how hard it is to get anything done: politics and organisational dysfunction, not to mention that you probably don’t have access to half of what you need to in order to fix any given problem and are even more unlikely to be able to get it, mean there are just huge scads of problems that, on the face of them, look relatively straightforward to solve but which, in practice, are organisationally impossible to solve.
  - whstl 2y ago
    Yep. I don't work with data engineering, but from hearing war stories from them, this could perfectly describe the last four companies I worked at. :/ > you probably don’t have access to half of what you need to in order to fix any given problem and are even more unlikely to be able to get it I once sat down with a data engineer to try to fix a specific problem they had and that was 100% accurate. They were left to die by Ops and CISO.
  - Terr_ 2y ago
    A framing/question I like to ask use is: "Look for a root problem that ought to be fixed through a change in policy, politics, or incentives, and the wasteful use of time/money is how the company tries to avoid or defer facing it." For example, Operations might demand that Engineering develops an increasingly-byzantine approvals process, to stop Sales from over-promising impossible or unprofitable projects.
  - deergomoo 2y ago
    > you probably don’t have access to half of what you need to in order to fix any given problem I’ve found this to be one of my largest day-to-day problems even in a relatively functional organisation. Particularly when it involves something I can’t run on my own machine, like an AWS service. In a previous role I often found myself constructing elaborate hypotheses about what was going on inside systems I couldn’t see into. I’d then need to try to verify it with someone on another team, in another timezone, who had the access but not necessarily the development background. Which usually meant getting on a screen share and asking them to click various things I wasn’t allowed to. If I was wrong, back to the drawing board and start again.
bartread 2y ago
I’m going to read the rest of this. I’m enjoying it. But, simultaneously, part II has me so triggered - it bears striking resemblance to repeated situations I’ve encountered where the meaning and content of columns in a relational database were overloaded in varying degrees of heaviness (which is a practice I absolutely detest) - that I need to take a short break.
- XorNot 2y ago
  People do this with system hostnames a lot. And it's almost impossible to get them to stop: the hostname should either be a random UUID or a random name from a pronounceable list depending on scale (or a syllabic UUID thing). Because every other factor has one answer: you look up the other data you need in your CMDB. If that's too hard, you fix that so it's easy (DNS TXT records can be surprisingly useful here).
- hermitcrab 2y ago
  I manage a database for a small local charity. I have set it up so that only I can add, delete or change the column structure. If someone wants a change, they have to email me and convince me (they are fine about this BTW). I'm sure the database would be an utter disaster zone by now if everyone was allowed to change it.
  - klysm 2y ago
    I think database schemas deserve to be protected with one’s life as the holy ground of the system. If the schema is fucked, everything else will be fucked too.
    - hermitcrab 2y ago
      With a database you can lock down the schema. In reality though, many data system are composed mainly of people emailing in Excel spreadsheets. Good luck enforcing any sort of schema there. My day job is writing a desktop/file-based ETL system. I have just added in a schema version feature to cover these sort of issues. It was one of the most requested features, because most people aren't able to control the schemas of the data they receive.
lifeisstillgood 2y ago
>>> pretending that any of this is more important than hiring competent people and treating them well. I could build something superior to this with an ancient laptop, an internet connection, and spreadsheets. Ow
iamthepieman 2y ago
I do not use this term to refer to myself. I respect those who do and respect the meaning behind it but am just old enough that it feels alien to me 99% of the time. But I am SO triggered by this piece. I had that intrusive feeling you sometimes get when driving where you think, "I could just close my eyes and see what happens", "Or that clif is so close and the guardrail doesn't really extend far enough" Only for my career. Like I should just not show up on Monday. I should get in the car and drive far away and change my name and work at a nice retail joint in a mid-sized town. I'm going to need to sit and stare into the distance for an hour and 3.
- Electricniko 2y ago
  > There has been a point in my life where I ended every day in the dark, staring at a wall for an hour or two straight, trying to figure out why everything felt awful. From his post about burnout and mental health. Also worth a read. https://ludic.mataroa.blog/blog/on-burnout-mental-health-and-not-being-okay/ https://ludic.mataroa.blog/blog/on-burnout-mental-health-and...
  - SanjayMehta 2y ago
    If on Monday morning you’re wishing it was Friday evening, it’s time to quit.
    - chubs 2y ago
      Many of us have kids to feed! The economy is not bursting with jobs anymore since rates rose post-COVID.
      - karel-3d 2y ago
        No need to quit immediately, just apply for jobs on the side.
        Aeolun 2y ago
jauntywundrkind 2y ago
The observability world still regards itself as a system for monitoring, but reading (and sometimes seeing) how these systems just go so bad continues to drive a conviction that perhaps their strategies and tools should become bigger. That they should converge with business pipines. We shouldn't just have wide events/big spans emitted... We should have those spans drive the pipeline. Rather than observability being a passive monitoring system, if we write code that reacts to events we are capturing, then we shuffle towards event sourcing. Given how badly coupled together with shoestring glue & good wishes so many systems are, how opaque these pain zones are, it feels like the centralization upon existing industry standard protocols to capture events (which imo include traces) is a clear win. (Obvious downside, these systems become mission critical, business process & monitoring both.)
- halfcat 2y ago
  What’s this look like in practice? Is this something like business process modeling and workflow engines, or something else?
- edejong 2y ago
  Totally agree. Observability is just another dataset and should be modeled, managed and governed as other datasets. Data quality controls should be equal or of higher standard than regular data sets. Monitoring, dashboarding and alerting should leverage other BI-class tooling.
halfcat 2y ago
What’s the solution to wrangling these data projects? The author’s experience is not far off from my own. 1. Any solution in place can only be understood by the person who created it 2. ”No, we can’t change that because then we’d have to validate everything from scratch again” And therefore, as the author says: > ”we'll continue with the work instead of fixing the critical production error” I’m honestly not sure how to address it either. With traditional software dev we’d write tests, incorporate those into CI/CD, and start to course correct. We can use sample data to validate the code does what we think it does and that we didn’t break it. But in these data projects, it’s not only the code that’s changing, but the data is also a moving target. You can write a test with sample data, but tomorrow your data might change because someone in sales added a custom field to the CRM, or IT upgraded the accounting software and all of the unique IDs changed, or someone upgraded their Excel version, or whatever. And your code that works on the sample data needs to handle all of this, which obviously it can’t. You can try to validate the data somehow, check the schema, check if the number of rows hasn’t doubled or halved, and so forth, and then stop it from importing until you look into it, but also you can’t stop inbound data because an exec has a meeting in a few hours and expects their report to be updated. I heard something about “data contracts” that’s supposed to address this, but it sounds like the next in a long line of buzz words intended to get management to buy another data product. Has anyone worked in this kind of project that went well?
- 3np 2y ago
  "We should add a single-source-of truth validation microservice that we put in front of all insertions, putting failing messages on a separate queue" And on it goes.
  - Liquix 2y ago
    https://xkcd.com/927/ https://xkcd.com/927/
- ludicity 2y ago
  Author here, and also executive director at Hermit Tech now where we do things like this. Your approach has the core of how I'd go about it. The contract stuff is legit, though you don't need to buy a product for it. The thing that is hard at big organizations isn't that executives need the data for meetings. The issue right now is that many organizations are already 3-4 years into building their analytics platforms, and Chief Data Officers worldwide are trying to prevent their role from disappearing. They're already very much "Mom, we have CTO at home" in many companies, as evidenced by the fact they're usually reporting to the CTO or CFO. So at this stage, they've already told the business that the platform is "ready", and they are onboarding data sources. With no way to measure data quality, the only thing visible at the organization level is number of data sources onboarded. The fastest way to onboard data sources is to have good CI/CD and a solid developer environment, but this would probably result in slowdown for 1 - 2 months even if you had executive backing to bulldoze all objections from the IT department. That's the sort of thing I can commit my team to as a business owner, but most executives don't have the nerve to slow delivery down and aren't losing money out of their pocket due to the inefficiency - I get to talk with a lot of them due to the blog's success these days, and many of them really are just employees with more status, with the same incentives. And to make it worse, the loss of nerve is actually understandable, because the type of team that would build something this bad will also waste those two months then still deliver slowly! But most people aren't thinking in terms this complex, and yes, I know it isn't that complex. I'm expecting to pick up some work in this area at larger orgs in a few years when these leaders rotate out and new leaders rotate in and go "what the hell IS this?", but for now we're mostly aimed at helping smaller places do it right from day one.
pxc 2y ago
This blog post rescheduled all my appointments, tucked me in, sang me a lullaby, then woke me up with coffee and breakfast late the next morning. I am healed. For real, a fun and refreshing read (if also a little haunting).
marcosdumay 2y ago
The link about Scrum has a link to this: https://agile2.net/ https://agile2.net/ Can someone, please, tell me this is a joke. Because I can't be certain, but it doesn't look like one.
- SSLy 2y ago
  This is the real agile2 https://scaledagileframework.com/ https://scaledagileframework.com/
  - marcosdumay 2y ago
    Oh, this is much worse satire. It didn't even put the effort to add details into The One Big Diagram. (But yeah, this one I know is for real. The other one that feels like a joke, but looks real is harder to decide.)
- ludicity 2y ago
  Author here. To the best of my knowledge, it is real. Sadly.
- bobnamob 2y ago
  Jeez, that’s almost as bad as the semantic versioning spec being semantically versioned
Muromec 2y ago
Great piece of writing from someone who truly cares about craft and suffers from the feeling that this craft is not what they are paid for. Add: for people who sharer the feeling -- you can work in a place where velocity isn't all, managers are not assholes and you can dedicate yourself to craft.
baazaa 2y ago
People always say this guy just has had bad luck with his employers but I live in Melbourne and work in data and reckon the whole industry is a scam. Like why didn't anyone catch the issue with the logs? Because it doesn't matter, every data team is a cost-centre that unscrupulous managers use to launch their careers by saying they're big on AI. So nothing works, no-one cares it doesn't work, most the data engineers are incapable of coding fizzbuzz but it doesn't matter. People always wonder why banks etc. use old mainframes. There's like a 0% success rate for new data projects. And that 0% includes projects which had launch parties etc. but no-one ever used the data or noticed how broken it was. I don't think a lot of orgs which use data as core-infra could modernize, the industry is just so broken at this point I don't think we can do what we did 30 years ago.
- le-mark 2y ago
  This jives with my experience at a financial services company. I once sat next to the “big data team” and the company 5 year plan was all about delivering analytics and ai to customers using their data the company housed. The team consisted of one guy (who had a business degree) and a lot of empty cubes they were trying to fill. A year later the company had been acquired and the big data initiative had evaporated.
- RangerScience 2y ago
  I have felt exactly this on regular full stack teams many, many times, so it’s also not just limited to data teams. IMO a major factor here is that software engineering is both opaque and esoteric - at least with physical engineering, there’s something people can look at and think they understand.
  - baazaa 2y ago
    My theory is that data is worse again because at least if you're making a website you're expected to end up with a website. The process is opaque and esoteric, but the end-product is somewhat tangible. A lot of data projects are moving and transforming data no-one cares about. They can fail completely silently, a manager can lie and say 'we've successfully built the data platform which is going to enable AI analytics' and it'll be like a misconfigured S3 or something. No-one's checking the end-product or even understands what it's meant to be.
djoldman 2y ago
This kind of thing is pretty typical. The older the company is, the more likely one finds this morass. It won't change absent powerful technical leadership.
holden_nelson 2y ago
I went down the rabbit hole of this blog after reading this post. This person's blog is amazing. I particularly appreciated this piece: https://ludic.mataroa.blog/blog/quitting-my-job-for-the-way-of-pain/ https://ludic.mataroa.blog/blog/quitting-my-job-for-the-way-...
- sph 2y ago
  "and still you will lay upon your death bed thinking that you may fend off the Reaper if you could but Estimate how long dying will take!" What would I give to be able to compose sentences like this one.
- sevensor 2y ago
  My man has a podcast as well, which is worth a listen.
  - ludicity 2y ago
    xoxo thanks! Although editing audio is one of my daily humility reminders. It's touuuugh.
akoboldfrying 2y ago
[flagged]
snidane 2y ago
Looks like the classic mistake of every data team. Every single office person works with data in one way or another. Having a team called 'data' just opens a blanch check for anyone in the organization to dump every issue and every piece of garbage to this team as long as they can identify it as data. That's why you build data platforms and name your team accordingly. This is much easier position to defend, where you and your team have a mandate to build tools for other to be efficient with data. If upstream provides funky logs or jsons where you expect strings, that's for your downstream to worry about. They need the data and they need to chase down the right people in the org to resolve that. Your responsibility should be only to provide a unified access to that external data and ideally some governance around the access like logging and lineage. Tldr; Open your 'data' mandate too wide and vague and you won't survive as a team. Build data platforms instead.
_jonas 2y ago
I'm excited for LLM applications that can setup, monitor/validate, and optimize data pipelines at scale. Seems possible soon given that SQL and most data records aren't intended to be human-friendly
- FridgeSeal 2y ago
  Once again. Because clearly this needs to be said loudly and repeatedly. That is a _technical solution_ to a fundamentally _social_ problem.
- sph 2y ago
  Ah, the hubris of youth.
- spencerchubb 2y ago
  When LLMs can do the following, they might be able to fix data hell: - Negotiate with different teams to figure out what a field means - Be told that a field should be converted from one format to another, but oh wait it's causing errors somewhere downstream because it was told the wrong instructions - People come to you with some issue about the code you maintain, and you dig enough to realize the root cause is another team's code
  - hinkley 2y ago
    - explain how they came up with “12” as the answer.
- hermitcrab 2y ago
  And now we have two problems...
  - hinkley 2y ago
    Ten problems. Regex is for wusses who fear danger. LLMs are where real men turn.
- sourcepluck 2y ago
codethief 2y ago
Largely unrelated, but "data hell" reminds me of this classic from Silicon Valley: https://m.youtube.com/watch?v=YPgkSH2050k https://m.youtube.com/watch?v=YPgkSH2050k
- hinkley 2y ago
  “I don’t want to live in a world where other people make the world a better place better than we do.” Mike Judge did such a great job of making that character into Dollar Store Steve Jobs.
nothingislost 2y ago
[flagged]
- isoprophlex 2y ago
  He didn't feel it was easy for him, and that's what matters. Then, he took corrective action to save his soul from something that corroded him. Pretty good outcome in my book.
  - ludicity 2y ago
    Hah, well, it was easy in some ways. Performance expectations were so low that I could have cruised for years if I didn't care about my job. Just not easy in other ways.
    - yxhuvud 2y ago
      There is a special hell for people that care about their jobs. Unfortunately we create that hell ourselves.
lucidguppy 2y ago
> because we don't have any tests,... Right there ^^^
lucidguppy 2y ago
> I could build something superior to this with an ancient laptop, an internet connection, and spreadsheets. It would take me a month tops. ^^^ Then do it... and then strangler fig the original.
- ahoka 2y ago
  You can’t as it will immediately draw out the leeches who will hurl “best practices”, strategic partnerships with scam software vendors and compliance check ticking bullshit at you until you back off.
  - ludicity 2y ago
    Author here. This is essentially the case for many clients, especially governemnt. But we're bootstrapped, so we can afford to consult and use what we've learned to build tooling that only targets a small handful of sane clients. I've sought advice from various people on this, some who are famous-ish or quiet sales powerhouses in the US. My question was "What are executives buying when they hire consultants?", and the answer is consistently "Comfort". No one is actually comforted by Deloitte, KPMG, whoever. The moment I had confidence in an ethical consulting practice is the moment someone said "I don't even know where I would hire good consultants". This was someone with 30 years or something absurd of industry experience, including mentoring people that went on to become staff engineers. After processing that, I realized I don't know where to hire a consultancy that isn't going to bait-and-switch me with mediocre talent. They obviously exist, but they probably can only support something like 1 to 10 clients each. Even Thoughtworks, a place that I used to hear mentioned positively, was flagged by the CTO of a >$1B company over lunch last week as "shifting to bait-and-switch" tactics. tl;dr Pretty sure you can absolutely do better and make a living off that, but you have to think very carefully, do research, read a bunch of sales/marketing books, have great communication skills and at least adequate engineering skills. I still don't know if I have some of those, but if I fail it'll be a skill issue, not because the problem is not tractable.
erulabs 2y ago
As a regular old “platform engineer” I fight to ignore “data platform” tasks. There’s no target to hit, it’s just moving sand around a sand box. If you want an answer to a specific question, we can spin up a read replica and a Metabase and write a query in an afternoon, cool. I’ll get you a chart, we’ll move on. If you want “a data analytics platform to enable blah blah blah” I’m out, I can’t do it. My eyes won’t focus, my hands stop moving. Developers sometimes tell me stuff like “Kubernetes is too complex”, “jeez React is a pain”. I send those quotes to my friends stuck writing 195 step DAGs to transform log files from s3 into s3 so they can eventually land in s3 - ah yes but they’re parquet somewhere in between, and that matters for some reason. We laugh together, but I can see it hurts them more than I intended. Life is too short to faff about doing nothing. Go join a company with less than 100 engineers and learn to be happy again. Let the enterprises burn, we’ll all be better for it. Anyways this was a fantastic piece, I hope this person writes their book after all.
- spencerchubb 2y ago
  > my friends stuck writing 195 step DAGs to transform log files from s3 into s3 so they can eventually land in s3 sounds similar to what I do! on my team, it does actually serve a purpose, it's just hard to see the forest from the trees sometimes
- hermitcrab 2y ago
  >Go join a company with less than 100 engineers and learn to be happy again. Let the enterprises burn, we’ll all be better for it. But but then I won't be able to lease a new panzerwankmobile every 2 year to impress people I don't like!
- hinkley 2y ago
  In police dramas there’s always the judge who refuses to sign a warrant and accuses the cops of going on a “fishing expedition”. That’s half of Big Data to me. Maybe more. No idea what we’re looking for but we will tell you after the checks clear.
LAC-Tech 2y ago
This has such strong Australian vibes. AU dev scene is not great. Really heavy with POs, and PMs and CTO's without the background.
0xbadcafebee 2y ago
I've worked for like a dozen companies full-time. Most people don't know what they're doing. I always thought 'impostor syndrome' was a projection of general insecurity. But I've started to think it's actually the subconscious saying "I'm not sure what's right or wrong, please consult an expert." I have a fantasy of quitting my job to write books on the [modern] theory and practice of information systems engineering. Not 'how to write software', that's been done; I mean all the forms of engineering around software/information systems. In my dream, I write the books, everyone reads them, and starts doing their jobs right. But then I remember, I, a person arrogant enough to believe he knows how to do things right, still can't get shit done right. Maybe if I were a one-man company, I could 'do everything right', and feel good about the result. But I depend on an entire company of people to do the right thing, in the right way, at the right time. That's hard even with the best people. No company is made up of the best people. It's always a mix of the best, worst, and in-between. Strangely, a company can put out a decent product, despite the company being a tire fire. This is some comfort when you get older. You realize that everything being shit is okay, as long as the bills are paid. I have PTSD from when the thing that paid the bills was on fire, every week, for years. Lately at every job I have, I internally panic and scream at how horrible everything is. Because I'm haunted by what might happen. But it's not happening yet. So I muffle the screams, smile and nod along with the stand-up-meeting-cum-status-update. The sad thing is, I forget that it's okay that the stand-up is shit. I forget that I'm still getting a fat paycheck just to sit in meetings that could have been an e-mail. I forget that, despite the company bleeding cloud costs [no savings plans, RIs, serverless, right-sizing, etc], we seem to be making a profit. Despite the terrible designs, bad process, ineffective leadership, absentee management, lack of security, and all the rest, the bottom line is fine. The shit is fine. Currently, and probably for the unforeseeable future. I get craftsmanship. I'm a crappy woodworker. I enjoy making things well, and getting better at it. But our jobs are not fine woodworking. Our jobs are construction. We are banging rusted nails into shitty, twisted, racked, cupped, knotty-ass studs. If we're lucky. Yeah, this building is going to be shit. But somebody's still going to pay for it. And there'll be another job after. If we really wanted fine woodworking, we never would have taken this job, and we know it. We'd be struggling to sell a cabinet that took us two 80+ hour weeks, too tired to appreciate its beauty, too defeated by flaws only we notice. So let's stop beating ourselves up. Let's stop beating each other up. We don't, can't, won't, find meaning in this monument to mediocrity. No comfort from the pain zone. No pride to take home. But we are paying the bills, with more left over than most have. No broken backs and long hours. No lack of health care, no abuse from customers or the public. Not even that big a worry about job security. We are the lucky ones. We are blessed with a golden shovel. So let's do like those blue collar laborers we often idolize, and get to this annoying, bloody awful work that we are blessed with.
tofflos 2y ago
> Like why didn't anyone catch the issue with the logs? I see questions like these a lot and every time I feel that people immensely underestimate the effort required for curating data. In my experience data can only ever be as good as what it's being used for and in this story the logs haven't been used for this purpose before so they're not going to be any good. It's some sort of data variation on the second law of thermodynamics - entropy is winning. Going in with the expectation that things should be better will only lead to frustration.
- epgui 2y ago
  This is not a data curation issue though, it’s a basic o11y issue.
  - andrewflnr 2y ago
    > o11y Seems to be "observability", for anyone else seeing that for the first time.
    - downrightmike 2y ago
      It pops up every 8 months or so for me
t420mom 2y ago
I feel like there's a whole new generation of tech workers that need to read _Zen and the art of motorcycle maintenance_
- ludicity 2y ago
  Author here. This is one of my favourite books of all time. In fact, my two favourite books are ZAMM and The Black Swan, both of which I hated on my first read when I was 19. I recently re-read it for the third time while on holiday and was taking notes on pages worth including in a book review. I ended up with about 1/3rd of all pages logged and decided no review was better than just telling people to read the book, but I might write a blog post on gumption traps.
GardenLetter27 2y ago
[flagged]
- ahoka 2y ago
  I assume as a data scientist, he is at the consumer side of things.
- deleted 2y ago
  [deleted]
- spencerchubb 2y ago
  When someone leaves have you ever had code dumped on you that you've never read before and has no tests? If you don't even know what the code is supposed to do, you can't write tests for it. The way I handled it is: when people come to me with a bug, then I fix it and add a test for it.
nathancspencer 2y ago
of course, we're serverless, because how can you hurt yourself without a cutting-edge? Brilliant
zbyforgotp 2y ago
Bullshit jobs once again. I don’t know. These companies are complex systems. He is writing as if the engineers all knew how to fix the systems, but were just powerless to do that. But I’ve also seen projects lead by engineers that only added to the overall complexity. There is a paradox in this - the people who seem the most confident about fixing the systems usually only make things worse. Chesterton fences and stuff. This article triggers me because everybody who reads it will always believe that they would fix the mess if only they got the power, but in practice when they get power they would only add new complexity to the whole mess.
- xena 2y ago
  People generally do have some idea how to fix complicated systems have have endemic problems. The reason they don't is that the company considers the capitalism going up faster NOW to be way more important than it going up faster in the future.
mrlonglong 2y ago
This makes me very glad I now write software that drives hardware.
paulsutter 2y ago
Understanding that IT projects are difficult gives us more empathy. Gartner says that 80% of corporate IT projects are considered failures. McKinsey says that 17% of large projects fail so badly that the companies existence is threatened. Standish group says only 10% of projects succeed.
- hinkley 2y ago
  All large successful projects start as small successful projects.
sethammons 2y ago
> Like why didn't anyone catch the issue with the logs? Because there were no automated tests. If the company needs something to work, that thing needs a, preferably automated, test.
pards 2y ago
> The word enterprise means that we do this in a way that makes people say "Dear God, why would anyone ever design it that way?" Thank you for this phrase; I'll quote it at every opportunity.
- hinkley 2y ago
  I’m working on an SDLC app that will end up with inspirational sayings as interstitials once I run out of bigger features/desperately want to procrastinate. I’ll stick heavy hitters from Goldratt, Fowler, Feynman et al in there, but there’s going to be a “dark humor” and “snarky” category and this will definitely go into one of them, along with some Ambrose Bierce.
tokinonagare 2y ago
> At two of the four businesses I've worked at, the most highly-performing engineers have resorted to something that I think of as Pain Zone navigation. It's the practice of never working unless pair programming [...] The fear and dread comes from a culture where people feel bad that they can't work quickly enough in the terrible codebase Exactly why I burned-out at work, worked at most 2 hours per day on a good day and finally was ejected from the project after a PM that graduated last year from school noticed and went after my head. Author is a wizard for describing the situation this well. It's been 3 days I've been free from the tyranny of Jira and project managers, and I worked more on my personal projets than I did in a week at my former workplace.
hinkley 2y ago
> The word enterprise means that we do this in a way that makes people say "Dear God, why would anyone ever design it that way?" I feel this comment in my bones.
- schnatterer 2y ago
  My search for a generic definition of the term enterprise finally comes to an end :D
kayo_20211030 2y ago
Good Grief! Prolix? > coated in grass which rends those who tread upon it like a legion of upraised spears, and there's more. Get to the point, whatever it is.
- dcminter 2y ago
  Some of us enjoy the journey.
hinkley 2y ago
This is the “I saved my company half a million dollars in about five minutes” person who got in trouble for saving them half a million dollars.
- ludicity 2y ago
  You are the only person in months that has called me anything other than "the AI rant guy" and I just wanted to say I appreciate you. <3
  - hinkley 2y ago
    I am an AI agnostic so even if I remembered that was you too, I’d only say it to our tribe. I came into the industry exactly as a previous AI Trough of Disillusionment was in full swing so I’ve been vaccinated.
  - sampo 2y ago
    The "AI rant": https://news.ycombinator.com/item?id=40725329 https://news.ycombinator.com/item?id=40725329
  - mildzebrataste 2y ago
    This snippet is so relatable, so great: "…what they actually needed to do was fire most of the staff in every team, leaving behind the two people who actually had good domain knowledge, then allow them to collaborate with good engineering teams to build sensible processes and systems. Instead, they hired a bunch of Big Firm Consultants. You can see where this is going already." I’m on month ten of my tech sabbatical and it’s been great. I’m no closer to wanting to return to the industry (former FAANG data jockey), not in a position where I can never go back but am in that middle-state where I want to use tech to contribute to the betterment of society, the community, nature and sustainability. I have a couple more months to try and figure out how.
deleted 2y ago
[deleted]
sourcepluck 2y ago
I had read and very much enjoyed the "AI silence or you'll be Piledriven into next week" post, without clicking on anything else on the author's blog. It was click link, read whole thing, love it, send it to one or two people, move on. Very happy to see this here, realise it's the same person and that this is "a thing", and then to rollick in the author's backlog. A joy! Raucous real-life laughter has exploded from me on numerous occasions along with most articles. I think I've read 5 in a row there, and my brain is buzzing happily. Thank you to the author for having the courage to write about real experiences. A breath of fresh air. I look forward to future books and articles, and reading more previous work, and cross my proverbial fingers hoping they can keep it real in the face of what will presumably be an avalanche of grifters looking to leech off the attention.
- refulgentis 2y ago
  for fellow readers: https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you-if-you-mention-ai-again/ https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you...
  - sourcepluck 2y ago
    And, adjacent, in my opinion, readings like this piece, entitled "Hallucinating sense in the era of infinity-content": https://www.documentjournal.com/2024/05/technical-images-film01-angelicism-art-showtime-true-detective-shein/ https://www.documentjournal.com/2024/05/technical-images-fil...
    - refulgentis 2y ago
      This is really, really, fucking good, thank you.
      - sourcepluck 2y ago
        Nice to see someone having a similar reaction to me upon reading it. I too found it astoundingly good. I couldn't say what exactly about ludic's blog reminded me of it so strongly, but in any case, thrilled to share Caroline Busta's work with someone.
jacobyoder 2y ago
"Suffice it to say that while people are sincerely trying their best, our leaders are not even remotely equipped to handle the volume of people just outright lying to them about IT." I've tried to come up with some heuristic to determine whether or not a team is competent, good, or doomed. I've been exposed to all over the last... 8-10 years, and one of the key things I've noticed is the ratio of competent/skilled developers to the unskilled ones is a big ... indicator(?). Predictor? Colleague of mine has been working with a team - dev team has ranged from 5-8 people over the last few years. Few people seem to have any grasp of programming at all. Only two people - my colleague and one other - have ever taken projects from ideas to delivery, or even taken features from requests to successful rollout of already functioning software. The arguments that people get in to there - days or weeks of people 'researching' whether or not OAUTH 'really' requires 'refresh tokens' or whether it's really supposed to be a JWT. Management has some notion of 'every voice is legitimate and should be heard - we don't support bullying' and so on. If you have a team of 10, and 1 or 2 people are simply bad at having the ability to think somewhat abstractly, you can survive. If that number hits, say, 4-5... the team will struggle. A lot. You can keep things going, but it will be slow. And everything becomes a battle. If that number becomes 7 or 8, and you only really have 1-2 developers who are actually competent developers... things will continue to spiral downward. On the other side - I worked with a team of about 8-10 people on a 6 month contract. The larger org had another 40 or so folks, handling other projects, and support. Onboarding was great - I pushed production code in the first week. Everyone on the team was competent, including the juniors. I had more development experience, but they had more company experience, and it was really a relatively enjoyable engagement overall. It was refreshing to be able to ask anyone on the team questions, and either get a workable answer, or an "I'm not sure, let's check with XYZ" to get working answers. The "oh, yeah, it's ABC" when ABC is clearly not the answer stuff never happened. People committing code and pushing to production without ever having run the code at all - I've experienced that - didn't happen - that's happening to my colleague. The problem with a plurality of tech-incompetent folks in a tech group is that they honestly can not determine that they aren't competent. The only examples of competence are in the minority, and tend to not be trusted (even though that minority is the only portion that turns out working/functional code). Leaving ends up being the only option in those cases. My colleague is only at his place part time, and has hung around because they've gone through some restructuring where new folks were brought in, and... you hope that things might get better in a few months, then realize they don't.
javajosh 2y ago
Tempting to write a response piece, "Get me into data hell, I need the money."
- ludicity 2y ago
  In Australia, data jobs have a high floor but a low-ish ceiling at most places. Salaries start at around A$100K if you know how to interview, but 30 years of diligent practice only nets you around A$200K, and this would be considered an extreme success. If you're already good, just have coffee with executives/leads at places that take engineering seriously and you'll earn well, but there's no need to enter data hell as an employee at the typical company.
  - pugz 2y ago
    I'm also based in Melbourne. I would add that all the above is true of maybe 95% of companies, but there are a few exceptions: big US-based tech companies. There are roles for devs that pay AU$250-$550K/year, including the data engineers. Unfortunately, there appears to be zero correlation with expertise and we have the exact same pain zone experience that I had when I was earning a third of this salary at an AU-headquartered company.
ramshanker 2y ago
Acting against your own better Judgement, I have to take that fight/ urge every few days. ;)
anitil 2y ago
I really enjoy this writing for a couple of reasons. Firstly, Australian. Secondly the acerbic wit just tickles my fancy > we're serverless, because how can you hurt yourself without a cutting-edge? Just perfect.

698 points by pavel_lishin 2y ago

jaygreco 2y ago

I really like the author’s writing style here. The quips about the tea especially.

remram 2y ago
It reminds me of the classic series on /r/talesfromtechsupport by airz23. It's uncanny. They would almost always feature a "I sip my coffee. It tastes like <feelings at the time>" Example: https://www.reddit.com/r/talesfromtechsupport/comments/203qxp/securityis_it_a_thing/ https://www.reddit.com/r/talesfromtechsupport/comments/203qx...
alex_suzuki 2y ago
Same. For me the association was the Bastard Operator From Hell, with a darker, more cynical view of life. Just very enjoyable reading.
ludicity 2y ago
Very much appreciate the kind words! From you and the other commenters too.

ctippett 2y ago

> of course, we're serverless, because how can you hurt yourself without a cutting-edge? A beautiful epigram.

storafrid 2y ago
Yes, but I'm surprised that they attribute "cutting-edge" to Lambda. It's about as old as Docker.
- deleted 2y ago
  [deleted]
- zdragnar 2y ago
  I was part of a company that went all in on using lambdas for the majority of their web facing APIs. That was 7 years ago. The cutting edge bit is a nice quip, though I agree not exactly accurate anymore.
  - stackskipton 2y ago
    I worked for a company that went all in on Lambda as well. The knots they had to twist themselves into so that everything ran nice and smooth in Lambda environment was mindboggling. We have certain actions like orders that would pass through 8 Lambdas before completion because of execution time or just the big code base would result in 7 seconds start up time (node) so it would get broken down. If any of them failed, and it felt like failed a ton due to Amazon backend stuff, it was a nightmare to resolve. All of it could probably been handled by larger node application in docker container somewhere but AUTO SCALING, FAILOVER, SERVERLESS! Once I started as SRE for a new team, we built a larger monolith using Node and docker on EC2. We would get massive complements for our uptime and reliability but there were some architects extremely unhappy when I revealed in division presentation that it was just Docker + m4.xlarge running Ubuntu 18.04. When I left, more and more Lambdas were being broken down into docker running on EC2. They are probably on some container managed solution now.
- sph 2y ago
  It sounds like you like to deal with much sharper edges than I am comfortable with. Or maybe I am too old with this shit. Still haven't found a use for "serverless" knives.
- hinkley 2y ago
  For a bank, Lambda is brand new. You haven’t worked in conservative industries I take it? Late adopters, every one. OP is still trying to replace Cobol. I know an insurance company that started that process 15 years ago. Fifteen years.
the_af 2y ago
On one hand, yes. Beautiful. But on the other, the same sentence could be written about software deployed to traditional servers. "Because of course, how can you hurt yourself without the joys of badly configured servers?".
- skydhash 2y ago
  You can hurt yourself with a badly held butter knife, and you can hurt yourself juggling katanas. Which situation would get people saying you're crazy?
  - zdragnar 2y ago
    Well, if you narrow down the metaphor to just knives, a dull knife is more dangerous to a chef than a sharp knife, because you need to apply more pressure and you get less control over the cutting action.
    - downut 2y ago
      The people juggling aren't chefs.
      - zdragnar 2y ago
        Clearly, you haven't seen a good chef at work.
      - the_af 2y ago
        Dull knives are dangerous to most people, not just chefs. Most beginner's cooking books/lessons will tell you to keep your knives very sharp, because dull knives are dangerous (for the reasons explained by grandparent comment). This affects amateurs just as much (if not more) as experienced chefs.
    - hinkley 2y ago
      Yes, but which one will people say you’re crazy?
layoric 2y ago
Came here to praise the same sentence, well done author, gave me a good laugh!

salt-thrower 2y ago

Beautifully written and fun to read. Blog posts like this give me a boost of mental strength to keep going during my worst episodes of burnout.

reverius42 2y ago

> I've even degraded team morale because I've convinced some of the engineers that things should be better, but not management, so now some of the engineers are upset. Oof, that hits a little close to home.

ludicity 2y ago
The good news is that, since the others are also looking for work elsewhere, there will be more engineers out in gen pop that actually thinks tests are useful, hah.
hinkley 2y ago
I have this illusion in my head that I stayed so long at my last company that almost all of my favorite people left, but one of my coworkers had my number. After a person I liaised with on another team left, I asked his superior if there was someone else I should build bridges with. We started talking about one of the team members and he said, “I don’t want you to talk to him. We like him, and if you talk to him he’ll leave.” This was on Slack so I don’t know if this was a jest or he was serious/mad. But it’s entirely true. I’ve convinced at least half a dozen people that we should expect better from a team environment and ourselves, and that this org (not the whole company, just this division) is a cult of stupidity. I was trying to recruit collaborators to fix the bullshit but apparently they decided it would be much easier to just start over.
- refulgentis 2y ago
  There's some finer points to parse in your comment that I'm not 100% on (ex. if "this org" is your division or the partner division), so I'm out on a ledge a little bit here, might not relate to what you meant. I was lucky enough to get ~6 years running my own tech company after 6 years as a waiter. Then I sold it, yadda yadda, went to Google 6 months later, got ~7 years in there. It really, really, really, disturbed me how approximately every situation, in every division, with any people, ended up being boiling down to "how do we muddle through one more day without challenging anyones preconceptions", 95% of the time it was tribal antisocial stuff, and no one would speak up about it. Direct example, for posterity. I don't wanna speak too directly to it, so lets imagine Google Division A (hereafter, dApps). New division lead (ex-dApps) joins dBytes with apparent bias against partnering division (dConsumer). Despite the project being previously framed as top priority, new lead consistently undermines dConsumer in meetings and shows little interest in understanding their work. Team adopts leader's negative attitude, becoming obstructive and uncooperative. I ended up carrying a critical launch, virtually alone, for 6 months. At performance review time, my boss questions why I didn't get more team involvement - despite the hostile environment that prevented exactly that - and speaks glowingly about how we need to support peer going for promotion based on their excellent job on part Y...which they didn't do. They spent 2 days on it then said it was impossible. And they were definitively the most cooperative because at least they tried, and wouldn't actively be aggressive in meetings with the outgroup. Everything, always, came down to: A) don't cause conflict at all, at home, or you will be buried B) we'll bend over backwards to accomodate conflict you invent, as long as we can clearly define them as an out-group with 0 ability to affect us day to day.

SynasterBeiter 2y ago

I hate whining like these. Just do your job and get over it. No need to be theatrical about it.

incognito124 2y ago
A creative outlet is a need, to some. You sound like OP's employees.
baq 2y ago
Let the man live will you. Not everyone wants to be a drone for hire.
hobs 2y ago
If this was a job with a non-abstract input and output process they would have OSHA saying it was genuinely unsafe (and definitely stupid.) Many of us build systems to manage automatic actions to take care of this stuff, if I had an engineer I had to take 100+ steps to get something done I would definitely be considering 1) What the hell am I paying for and 2) Why the hell am I paying for it?
hprotagonist 2y ago
you know, he had a post about exactly this comment not too long ago! edit: https://ludic.mataroa.blog/blog/the-violent-role-of-relentless-positivity-in-the-workplace/ https://ludic.mataroa.blog/blog/the-violent-role-of-relentle...
- tgv 2y ago
  That's pretty good. I'm somewhat curious now about his writing aspirations.
  - ludicity 2y ago
    I was offered a book deal and turned it down, hah. I read in spectacular volumes and books have a very special place in my heart. Growing up in Malaysia, most of my English was initially mastered from a gigantic pile of Enid Blyton books my family had lying around after the British occupation. We even give everyone at Hermit Tech a day off per week to study non-IT things and let the subconscious do some processing on work. Not because I'm a weirdo (though I am), but because I genuinely believe this produces a higher-quality experience for our clients, and no one can stop me from testing things like "five day work weeks are too long". In any case, the book deal had constraints like "no swearing", and it was implied they'd find their own artist for the cover. I didn't even intend to swear, but I care too much to let them assign an editor and impose arbitrary constraints. I ended up chatting a bit with Ed Zitron after the AI rant article went super viral, and he told me that I have the audience to just publish my own books. So I'm doing just that! I'm starting with ten short stories, at the advice of someone in the local Melbourne scene that has helped many people publish. Then I'm aiming to write one book containing a series of essays on IT work, and one fantasy/fiction book. I'm a reasonably good judge of popularity, and because neither will be particularly angry, I will probably only sell a few hundred copies. But I'll have a book with my name on it, which is very special to me.

brianhorakh 2y ago

Wonderful prose. I am in Melbourne also. I possibly used to work at the same place but I'm not sure. I resigned due to the night terrors caused by the cyber security issues I saw everywhere. The more I explored and understood the more sleep I lost.

ludicity 2y ago
Hah, yes, I'm not in cybersecurity but am very close to a few people that are. The incompetence is not evenly distributed and not as bad as it is in data, but some companies are in terrible states, and the stakes are much higher.
hinkley 2y ago
I worked with an Aussie who was in the US on H1B because the aerospace industry was even worse than the status quo in Australia. Last I heard he went back. I sincerely hope he switched industry verticals.

zombiwoof 2y ago

Data “engineering” is where all the cool kids go with no clue and create insane architectures to justify their incompetence

hobs 2y ago
As a data engineer I have seen absolutely bullshit pass for production, but it doesn't seem that different from all the other bullshit I have seen people deploy in my life. It is one of the few types of jobs I have worked were someone credulously offering adding five more layers to fix an issue with latency is a normal operating procedure though.
parpfish 2y ago
data engineer gives data science a run for their money when it comes to ambiguous job expectations. I’ve seen it mean anything from “distributed computing expert” to “knows SQL”
- marcosdumay 2y ago
  > to “knows SQL” There are still ways to go. I've seen it means "spends the days filling excel spreadsheets".
- sgarland 2y ago
  > knows SQL If by that you mean, “knows the commands to create, fill, and select from a table,” then yes. If you mean, “knows how to create a performant schema and queries that will serve them well into the future,” then no, absolutely not. OTOH, IME data folk are much more cheerful and willing to change things than devs when I point out the innumerable ways their DB choices are choking them. Devs more often fall on the side of “we don’t have time for that on our roadmap; can’t you just fix it?”
  - zelphirkalt 2y ago
    It is often not the devs telling you about the roadmap, but the management. Devs are more willing to fix things, if they are broken, but are not given the time.

jitl 2y ago

I wonder what company they’re describing here. It sounds like so many self inflicted problems that that you could undo or set right in a couple of weeks if you had the time and latitude to make changes across the system instead of being confined to a small area of team ownership.

bartread 2y ago
I worked somewhere that had a lot of this sort of thing going on once. You cannot overestimate how hard it is to get anything done: politics and organisational dysfunction, not to mention that you probably don’t have access to half of what you need to in order to fix any given problem and are even more unlikely to be able to get it, mean there are just huge scads of problems that, on the face of them, look relatively straightforward to solve but which, in practice, are organisationally impossible to solve.
- whstl 2y ago
  Yep. I don't work with data engineering, but from hearing war stories from them, this could perfectly describe the last four companies I worked at. :/ > you probably don’t have access to half of what you need to in order to fix any given problem and are even more unlikely to be able to get it I once sat down with a data engineer to try to fix a specific problem they had and that was 100% accurate. They were left to die by Ops and CISO.
- Terr_ 2y ago
  A framing/question I like to ask use is: "Look for a root problem that ought to be fixed through a change in policy, politics, or incentives, and the wasteful use of time/money is how the company tries to avoid or defer facing it." For example, Operations might demand that Engineering develops an increasingly-byzantine approvals process, to stop Sales from over-promising impossible or unprofitable projects.
- deergomoo 2y ago
  > you probably don’t have access to half of what you need to in order to fix any given problem I’ve found this to be one of my largest day-to-day problems even in a relatively functional organisation. Particularly when it involves something I can’t run on my own machine, like an AWS service. In a previous role I often found myself constructing elaborate hypotheses about what was going on inside systems I couldn’t see into. I’d then need to try to verify it with someone on another team, in another timezone, who had the access but not necessarily the development background. Which usually meant getting on a screen share and asking them to click various things I wasn’t allowed to. If I was wrong, back to the drawing board and start again.

bartread 2y ago

I’m going to read the rest of this. I’m enjoying it. But, simultaneously, part II has me so triggered - it bears striking resemblance to repeated situations I’ve encountered where the meaning and content of columns in a relational database were overloaded in varying degrees of heaviness (which is a practice I absolutely detest) - that I need to take a short break.

XorNot 2y ago
People do this with system hostnames a lot. And it's almost impossible to get them to stop: the hostname should either be a random UUID or a random name from a pronounceable list depending on scale (or a syllabic UUID thing). Because every other factor has one answer: you look up the other data you need in your CMDB. If that's too hard, you fix that so it's easy (DNS TXT records can be surprisingly useful here).
hermitcrab 2y ago
I manage a database for a small local charity. I have set it up so that only I can add, delete or change the column structure. If someone wants a change, they have to email me and convince me (they are fine about this BTW). I'm sure the database would be an utter disaster zone by now if everyone was allowed to change it.
- klysm 2y ago
  I think database schemas deserve to be protected with one’s life as the holy ground of the system. If the schema is fucked, everything else will be fucked too.
  - hermitcrab 2y ago
    With a database you can lock down the schema. In reality though, many data system are composed mainly of people emailing in Excel spreadsheets. Good luck enforcing any sort of schema there. My day job is writing a desktop/file-based ETL system. I have just added in a schema version feature to cover these sort of issues. It was one of the most requested features, because most people aren't able to control the schemas of the data they receive.

lifeisstillgood 2y ago

>>> pretending that any of this is more important than hiring competent people and treating them well. I could build something superior to this with an ancient laptop, an internet connection, and spreadsheets. Ow

iamthepieman 2y ago

I do not use this term to refer to myself. I respect those who do and respect the meaning behind it but am just old enough that it feels alien to me 99% of the time. But I am SO triggered by this piece. I had that intrusive feeling you sometimes get when driving where you think, "I could just close my eyes and see what happens", "Or that clif is so close and the guardrail doesn't really extend far enough" Only for my career. Like I should just not show up on Monday. I should get in the car and drive far away and change my name and work at a nice retail joint in a mid-sized town. I'm going to need to sit and stare into the distance for an hour and 3.

Electricniko 2y ago
> There has been a point in my life where I ended every day in the dark, staring at a wall for an hour or two straight, trying to figure out why everything felt awful. From his post about burnout and mental health. Also worth a read. https://ludic.mataroa.blog/blog/on-burnout-mental-health-and-not-being-okay/ https://ludic.mataroa.blog/blog/on-burnout-mental-health-and...
- SanjayMehta 2y ago
  If on Monday morning you’re wishing it was Friday evening, it’s time to quit.
  - chubs 2y ago
    Many of us have kids to feed! The economy is not bursting with jobs anymore since rates rose post-COVID.
    - karel-3d 2y ago
      No need to quit immediately, just apply for jobs on the side.
      - Aeolun 2y ago

jauntywundrkind 2y ago

The observability world still regards itself as a system for monitoring, but reading (and sometimes seeing) how these systems just go so bad continues to drive a conviction that perhaps their strategies and tools should become bigger. That they should converge with business pipines. We shouldn't just have wide events/big spans emitted... We should have those spans drive the pipeline. Rather than observability being a passive monitoring system, if we write code that reacts to events we are capturing, then we shuffle towards event sourcing. Given how badly coupled together with shoestring glue & good wishes so many systems are, how opaque these pain zones are, it feels like the centralization upon existing industry standard protocols to capture events (which imo include traces) is a clear win. (Obvious downside, these systems become mission critical, business process & monitoring both.)

halfcat 2y ago
What’s this look like in practice? Is this something like business process modeling and workflow engines, or something else?
edejong 2y ago
Totally agree. Observability is just another dataset and should be modeled, managed and governed as other datasets. Data quality controls should be equal or of higher standard than regular data sets. Monitoring, dashboarding and alerting should leverage other BI-class tooling.

halfcat 2y ago

What’s the solution to wrangling these data projects? The author’s experience is not far off from my own. 1. Any solution in place can only be understood by the person who created it 2. ”No, we can’t change that because then we’d have to validate everything from scratch again” And therefore, as the author says: > ”we'll continue with the work instead of fixing the critical production error” I’m honestly not sure how to address it either. With traditional software dev we’d write tests, incorporate those into CI/CD, and start to course correct. We can use sample data to validate the code does what we think it does and that we didn’t break it. But in these data projects, it’s not only the code that’s changing, but the data is also a moving target. You can write a test with sample data, but tomorrow your data might change because someone in sales added a custom field to the CRM, or IT upgraded the accounting software and all of the unique IDs changed, or someone upgraded their Excel version, or whatever. And your code that works on the sample data needs to handle all of this, which obviously it can’t. You can try to validate the data somehow, check the schema, check if the number of rows hasn’t doubled or halved, and so forth, and then stop it from importing until you look into it, but also you can’t stop inbound data because an exec has a meeting in a few hours and expects their report to be updated. I heard something about “data contracts” that’s supposed to address this, but it sounds like the next in a long line of buzz words intended to get management to buy another data product. Has anyone worked in this kind of project that went well?

3np 2y ago
"We should add a single-source-of truth validation microservice that we put in front of all insertions, putting failing messages on a separate queue" And on it goes.
- Liquix 2y ago
  https://xkcd.com/927/ https://xkcd.com/927/
ludicity 2y ago
Author here, and also executive director at Hermit Tech now where we do things like this. Your approach has the core of how I'd go about it. The contract stuff is legit, though you don't need to buy a product for it. The thing that is hard at big organizations isn't that executives need the data for meetings. The issue right now is that many organizations are already 3-4 years into building their analytics platforms, and Chief Data Officers worldwide are trying to prevent their role from disappearing. They're already very much "Mom, we have CTO at home" in many companies, as evidenced by the fact they're usually reporting to the CTO or CFO. So at this stage, they've already told the business that the platform is "ready", and they are onboarding data sources. With no way to measure data quality, the only thing visible at the organization level is number of data sources onboarded. The fastest way to onboard data sources is to have good CI/CD and a solid developer environment, but this would probably result in slowdown for 1 - 2 months even if you had executive backing to bulldoze all objections from the IT department. That's the sort of thing I can commit my team to as a business owner, but most executives don't have the nerve to slow delivery down and aren't losing money out of their pocket due to the inefficiency - I get to talk with a lot of them due to the blog's success these days, and many of them really are just employees with more status, with the same incentives. And to make it worse, the loss of nerve is actually understandable, because the type of team that would build something this bad will also waste those two months then still deliver slowly! But most people aren't thinking in terms this complex, and yes, I know it isn't that complex. I'm expecting to pick up some work in this area at larger orgs in a few years when these leaders rotate out and new leaders rotate in and go "what the hell IS this?", but for now we're mostly aimed at helping smaller places do it right from day one.

pxc 2y ago

This blog post rescheduled all my appointments, tucked me in, sang me a lullaby, then woke me up with coffee and breakfast late the next morning. I am healed. For real, a fun and refreshing read (if also a little haunting).

marcosdumay 2y ago

The link about Scrum has a link to this: https://agile2.net/ https://agile2.net/ Can someone, please, tell me this is a joke. Because I can't be certain, but it doesn't look like one.

SSLy 2y ago
This is the real agile2 https://scaledagileframework.com/ https://scaledagileframework.com/
- marcosdumay 2y ago
  Oh, this is much worse satire. It didn't even put the effort to add details into The One Big Diagram. (But yeah, this one I know is for real. The other one that feels like a joke, but looks real is harder to decide.)
ludicity 2y ago
Author here. To the best of my knowledge, it is real. Sadly.
bobnamob 2y ago
Jeez, that’s almost as bad as the semantic versioning spec being semantically versioned

Muromec 2y ago

Great piece of writing from someone who truly cares about craft and suffers from the feeling that this craft is not what they are paid for. Add: for people who sharer the feeling -- you can work in a place where velocity isn't all, managers are not assholes and you can dedicate yourself to craft.

baazaa 2y ago

People always say this guy just has had bad luck with his employers but I live in Melbourne and work in data and reckon the whole industry is a scam. Like why didn't anyone catch the issue with the logs? Because it doesn't matter, every data team is a cost-centre that unscrupulous managers use to launch their careers by saying they're big on AI. So nothing works, no-one cares it doesn't work, most the data engineers are incapable of coding fizzbuzz but it doesn't matter. People always wonder why banks etc. use old mainframes. There's like a 0% success rate for new data projects. And that 0% includes projects which had launch parties etc. but no-one ever used the data or noticed how broken it was. I don't think a lot of orgs which use data as core-infra could modernize, the industry is just so broken at this point I don't think we can do what we did 30 years ago.

le-mark 2y ago
This jives with my experience at a financial services company. I once sat next to the “big data team” and the company 5 year plan was all about delivering analytics and ai to customers using their data the company housed. The team consisted of one guy (who had a business degree) and a lot of empty cubes they were trying to fill. A year later the company had been acquired and the big data initiative had evaporated.
RangerScience 2y ago
I have felt exactly this on regular full stack teams many, many times, so it’s also not just limited to data teams. IMO a major factor here is that software engineering is both opaque and esoteric - at least with physical engineering, there’s something people can look at and think they understand.
- baazaa 2y ago
  My theory is that data is worse again because at least if you're making a website you're expected to end up with a website. The process is opaque and esoteric, but the end-product is somewhat tangible. A lot of data projects are moving and transforming data no-one cares about. They can fail completely silently, a manager can lie and say 'we've successfully built the data platform which is going to enable AI analytics' and it'll be like a misconfigured S3 or something. No-one's checking the end-product or even understands what it's meant to be.

djoldman 2y ago

This kind of thing is pretty typical. The older the company is, the more likely one finds this morass. It won't change absent powerful technical leadership.

holden_nelson 2y ago

I went down the rabbit hole of this blog after reading this post. This person's blog is amazing. I particularly appreciated this piece: https://ludic.mataroa.blog/blog/quitting-my-job-for-the-way-of-pain/ https://ludic.mataroa.blog/blog/quitting-my-job-for-the-way-...

sph 2y ago
"and still you will lay upon your death bed thinking that you may fend off the Reaper if you could but Estimate how long dying will take!" What would I give to be able to compose sentences like this one.
sevensor 2y ago
My man has a podcast as well, which is worth a listen.
- ludicity 2y ago
  xoxo thanks! Although editing audio is one of my daily humility reminders. It's touuuugh.

akoboldfrying 2y ago

[flagged]

snidane 2y ago

Looks like the classic mistake of every data team. Every single office person works with data in one way or another. Having a team called 'data' just opens a blanch check for anyone in the organization to dump every issue and every piece of garbage to this team as long as they can identify it as data. That's why you build data platforms and name your team accordingly. This is much easier position to defend, where you and your team have a mandate to build tools for other to be efficient with data. If upstream provides funky logs or jsons where you expect strings, that's for your downstream to worry about. They need the data and they need to chase down the right people in the org to resolve that. Your responsibility should be only to provide a unified access to that external data and ideally some governance around the access like logging and lineage. Tldr; Open your 'data' mandate too wide and vague and you won't survive as a team. Build data platforms instead.

_jonas 2y ago

I'm excited for LLM applications that can setup, monitor/validate, and optimize data pipelines at scale. Seems possible soon given that SQL and most data records aren't intended to be human-friendly

FridgeSeal 2y ago
Once again. Because clearly this needs to be said loudly and repeatedly. That is a _technical solution_ to a fundamentally _social_ problem.
sph 2y ago
Ah, the hubris of youth.
spencerchubb 2y ago
When LLMs can do the following, they might be able to fix data hell: - Negotiate with different teams to figure out what a field means - Be told that a field should be converted from one format to another, but oh wait it's causing errors somewhere downstream because it was told the wrong instructions - People come to you with some issue about the code you maintain, and you dig enough to realize the root cause is another team's code
- hinkley 2y ago
  - explain how they came up with “12” as the answer.
hermitcrab 2y ago
And now we have two problems...
- hinkley 2y ago
  Ten problems. Regex is for wusses who fear danger. LLMs are where real men turn.
sourcepluck 2y ago

codethief 2y ago

Largely unrelated, but "data hell" reminds me of this classic from Silicon Valley: https://m.youtube.com/watch?v=YPgkSH2050k https://m.youtube.com/watch?v=YPgkSH2050k

hinkley 2y ago
“I don’t want to live in a world where other people make the world a better place better than we do.” Mike Judge did such a great job of making that character into Dollar Store Steve Jobs.

nothingislost 2y ago

[flagged]

isoprophlex 2y ago
He didn't feel it was easy for him, and that's what matters. Then, he took corrective action to save his soul from something that corroded him. Pretty good outcome in my book.
- ludicity 2y ago
  Hah, well, it was easy in some ways. Performance expectations were so low that I could have cruised for years if I didn't care about my job. Just not easy in other ways.
  - yxhuvud 2y ago
    There is a special hell for people that care about their jobs. Unfortunately we create that hell ourselves.

lucidguppy 2y ago

> because we don't have any tests,... Right there ^^^

lucidguppy 2y ago

> I could build something superior to this with an ancient laptop, an internet connection, and spreadsheets. It would take me a month tops. ^^^ Then do it... and then strangler fig the original.

ahoka 2y ago
You can’t as it will immediately draw out the leeches who will hurl “best practices”, strategic partnerships with scam software vendors and compliance check ticking bullshit at you until you back off.
- ludicity 2y ago
  Author here. This is essentially the case for many clients, especially governemnt. But we're bootstrapped, so we can afford to consult and use what we've learned to build tooling that only targets a small handful of sane clients. I've sought advice from various people on this, some who are famous-ish or quiet sales powerhouses in the US. My question was "What are executives buying when they hire consultants?", and the answer is consistently "Comfort". No one is actually comforted by Deloitte, KPMG, whoever. The moment I had confidence in an ethical consulting practice is the moment someone said "I don't even know where I would hire good consultants". This was someone with 30 years or something absurd of industry experience, including mentoring people that went on to become staff engineers. After processing that, I realized I don't know where to hire a consultancy that isn't going to bait-and-switch me with mediocre talent. They obviously exist, but they probably can only support something like 1 to 10 clients each. Even Thoughtworks, a place that I used to hear mentioned positively, was flagged by the CTO of a >$1B company over lunch last week as "shifting to bait-and-switch" tactics. tl;dr Pretty sure you can absolutely do better and make a living off that, but you have to think very carefully, do research, read a bunch of sales/marketing books, have great communication skills and at least adequate engineering skills. I still don't know if I have some of those, but if I fail it'll be a skill issue, not because the problem is not tractable.

erulabs 2y ago

As a regular old “platform engineer” I fight to ignore “data platform” tasks. There’s no target to hit, it’s just moving sand around a sand box. If you want an answer to a specific question, we can spin up a read replica and a Metabase and write a query in an afternoon, cool. I’ll get you a chart, we’ll move on. If you want “a data analytics platform to enable blah blah blah” I’m out, I can’t do it. My eyes won’t focus, my hands stop moving. Developers sometimes tell me stuff like “Kubernetes is too complex”, “jeez React is a pain”. I send those quotes to my friends stuck writing 195 step DAGs to transform log files from s3 into s3 so they can eventually land in s3 - ah yes but they’re parquet somewhere in between, and that matters for some reason. We laugh together, but I can see it hurts them more than I intended. Life is too short to faff about doing nothing. Go join a company with less than 100 engineers and learn to be happy again. Let the enterprises burn, we’ll all be better for it. Anyways this was a fantastic piece, I hope this person writes their book after all.

spencerchubb 2y ago
> my friends stuck writing 195 step DAGs to transform log files from s3 into s3 so they can eventually land in s3 sounds similar to what I do! on my team, it does actually serve a purpose, it's just hard to see the forest from the trees sometimes
hermitcrab 2y ago
>Go join a company with less than 100 engineers and learn to be happy again. Let the enterprises burn, we’ll all be better for it. But but then I won't be able to lease a new panzerwankmobile every 2 year to impress people I don't like!
hinkley 2y ago
In police dramas there’s always the judge who refuses to sign a warrant and accuses the cops of going on a “fishing expedition”. That’s half of Big Data to me. Maybe more. No idea what we’re looking for but we will tell you after the checks clear.

LAC-Tech 2y ago

This has such strong Australian vibes. AU dev scene is not great. Really heavy with POs, and PMs and CTO's without the background.

0xbadcafebee 2y ago

I've worked for like a dozen companies full-time. Most people don't know what they're doing. I always thought 'impostor syndrome' was a projection of general insecurity. But I've started to think it's actually the subconscious saying "I'm not sure what's right or wrong, please consult an expert." I have a fantasy of quitting my job to write books on the [modern] theory and practice of information systems engineering. Not 'how to write software', that's been done; I mean all the forms of engineering around software/information systems. In my dream, I write the books, everyone reads them, and starts doing their jobs right. But then I remember, I, a person arrogant enough to believe he knows how to do things right, still can't get shit done right. Maybe if I were a one-man company, I could 'do everything right', and feel good about the result. But I depend on an entire company of people to do the right thing, in the right way, at the right time. That's hard even with the best people. No company is made up of the best people. It's always a mix of the best, worst, and in-between. Strangely, a company can put out a decent product, despite the company being a tire fire. This is some comfort when you get older. You realize that everything being shit is okay, as long as the bills are paid. I have PTSD from when the thing that paid the bills was on fire, every week, for years. Lately at every job I have, I internally panic and scream at how horrible everything is. Because I'm haunted by what might happen. But it's not happening yet. So I muffle the screams, smile and nod along with the stand-up-meeting-cum-status-update. The sad thing is, I forget that it's okay that the stand-up is shit. I forget that I'm still getting a fat paycheck just to sit in meetings that could have been an e-mail. I forget that, despite the company bleeding cloud costs [no savings plans, RIs, serverless, right-sizing, etc], we seem to be making a profit. Despite the terrible designs, bad process, ineffective leadership, absentee management, lack of security, and all the rest, the bottom line is fine. The shit is fine. Currently, and probably for the unforeseeable future. I get craftsmanship. I'm a crappy woodworker. I enjoy making things well, and getting better at it. But our jobs are not fine woodworking. Our jobs are construction. We are banging rusted nails into shitty, twisted, racked, cupped, knotty-ass studs. If we're lucky. Yeah, this building is going to be shit. But somebody's still going to pay for it. And there'll be another job after. If we really wanted fine woodworking, we never would have taken this job, and we know it. We'd be struggling to sell a cabinet that took us two 80+ hour weeks, too tired to appreciate its beauty, too defeated by flaws only we notice. So let's stop beating ourselves up. Let's stop beating each other up. We don't, can't, won't, find meaning in this monument to mediocrity. No comfort from the pain zone. No pride to take home. But we are paying the bills, with more left over than most have. No broken backs and long hours. No lack of health care, no abuse from customers or the public. Not even that big a worry about job security. We are the lucky ones. We are blessed with a golden shovel. So let's do like those blue collar laborers we often idolize, and get to this annoying, bloody awful work that we are blessed with.

tofflos 2y ago

> Like why didn't anyone catch the issue with the logs? I see questions like these a lot and every time I feel that people immensely underestimate the effort required for curating data. In my experience data can only ever be as good as what it's being used for and in this story the logs haven't been used for this purpose before so they're not going to be any good. It's some sort of data variation on the second law of thermodynamics - entropy is winning. Going in with the expectation that things should be better will only lead to frustration.

epgui 2y ago
This is not a data curation issue though, it’s a basic o11y issue.
- andrewflnr 2y ago
  > o11y Seems to be "observability", for anyone else seeing that for the first time.
  - downrightmike 2y ago
    It pops up every 8 months or so for me

t420mom 2y ago

I feel like there's a whole new generation of tech workers that need to read _Zen and the art of motorcycle maintenance_

ludicity 2y ago
Author here. This is one of my favourite books of all time. In fact, my two favourite books are ZAMM and The Black Swan, both of which I hated on my first read when I was 19. I recently re-read it for the third time while on holiday and was taking notes on pages worth including in a book review. I ended up with about 1/3rd of all pages logged and decided no review was better than just telling people to read the book, but I might write a blog post on gumption traps.

GardenLetter27 2y ago

[flagged]

ahoka 2y ago
I assume as a data scientist, he is at the consumer side of things.
deleted 2y ago
[deleted]
spencerchubb 2y ago
When someone leaves have you ever had code dumped on you that you've never read before and has no tests? If you don't even know what the code is supposed to do, you can't write tests for it. The way I handled it is: when people come to me with a bug, then I fix it and add a test for it.

nathancspencer 2y ago

of course, we're serverless, because how can you hurt yourself without a cutting-edge? Brilliant

zbyforgotp 2y ago

Bullshit jobs once again. I don’t know. These companies are complex systems. He is writing as if the engineers all knew how to fix the systems, but were just powerless to do that. But I’ve also seen projects lead by engineers that only added to the overall complexity. There is a paradox in this - the people who seem the most confident about fixing the systems usually only make things worse. Chesterton fences and stuff. This article triggers me because everybody who reads it will always believe that they would fix the mess if only they got the power, but in practice when they get power they would only add new complexity to the whole mess.

xena 2y ago
People generally do have some idea how to fix complicated systems have have endemic problems. The reason they don't is that the company considers the capitalism going up faster NOW to be way more important than it going up faster in the future.

mrlonglong 2y ago

This makes me very glad I now write software that drives hardware.

paulsutter 2y ago

Understanding that IT projects are difficult gives us more empathy. Gartner says that 80% of corporate IT projects are considered failures. McKinsey says that 17% of large projects fail so badly that the companies existence is threatened. Standish group says only 10% of projects succeed.

hinkley 2y ago
All large successful projects start as small successful projects.

sethammons 2y ago

> Like why didn't anyone catch the issue with the logs? Because there were no automated tests. If the company needs something to work, that thing needs a, preferably automated, test.

pards 2y ago

> The word enterprise means that we do this in a way that makes people say "Dear God, why would anyone ever design it that way?" Thank you for this phrase; I'll quote it at every opportunity.

hinkley 2y ago
I’m working on an SDLC app that will end up with inspirational sayings as interstitials once I run out of bigger features/desperately want to procrastinate. I’ll stick heavy hitters from Goldratt, Fowler, Feynman et al in there, but there’s going to be a “dark humor” and “snarky” category and this will definitely go into one of them, along with some Ambrose Bierce.

tokinonagare 2y ago

> At two of the four businesses I've worked at, the most highly-performing engineers have resorted to something that I think of as Pain Zone navigation. It's the practice of never working unless pair programming [...] The fear and dread comes from a culture where people feel bad that they can't work quickly enough in the terrible codebase Exactly why I burned-out at work, worked at most 2 hours per day on a good day and finally was ejected from the project after a PM that graduated last year from school noticed and went after my head. Author is a wizard for describing the situation this well. It's been 3 days I've been free from the tyranny of Jira and project managers, and I worked more on my personal projets than I did in a week at my former workplace.

hinkley 2y ago

> The word enterprise means that we do this in a way that makes people say "Dear God, why would anyone ever design it that way?" I feel this comment in my bones.

schnatterer 2y ago
My search for a generic definition of the term enterprise finally comes to an end :D

kayo_20211030 2y ago

Good Grief! Prolix? > coated in grass which rends those who tread upon it like a legion of upraised spears, and there's more. Get to the point, whatever it is.

dcminter 2y ago
Some of us enjoy the journey.

hinkley 2y ago

This is the “I saved my company half a million dollars in about five minutes” person who got in trouble for saving them half a million dollars.

ludicity 2y ago
You are the only person in months that has called me anything other than "the AI rant guy" and I just wanted to say I appreciate you. <3
- hinkley 2y ago
  I am an AI agnostic so even if I remembered that was you too, I’d only say it to our tribe. I came into the industry exactly as a previous AI Trough of Disillusionment was in full swing so I’ve been vaccinated.
- sampo 2y ago
  The "AI rant": https://news.ycombinator.com/item?id=40725329 https://news.ycombinator.com/item?id=40725329
- mildzebrataste 2y ago
  This snippet is so relatable, so great: "…what they actually needed to do was fire most of the staff in every team, leaving behind the two people who actually had good domain knowledge, then allow them to collaborate with good engineering teams to build sensible processes and systems. Instead, they hired a bunch of Big Firm Consultants. You can see where this is going already." I’m on month ten of my tech sabbatical and it’s been great. I’m no closer to wanting to return to the industry (former FAANG data jockey), not in a position where I can never go back but am in that middle-state where I want to use tech to contribute to the betterment of society, the community, nature and sustainability. I have a couple more months to try and figure out how.

deleted 2y ago

[deleted]

sourcepluck 2y ago

I had read and very much enjoyed the "AI silence or you'll be Piledriven into next week" post, without clicking on anything else on the author's blog. It was click link, read whole thing, love it, send it to one or two people, move on. Very happy to see this here, realise it's the same person and that this is "a thing", and then to rollick in the author's backlog. A joy! Raucous real-life laughter has exploded from me on numerous occasions along with most articles. I think I've read 5 in a row there, and my brain is buzzing happily. Thank you to the author for having the courage to write about real experiences. A breath of fresh air. I look forward to future books and articles, and reading more previous work, and cross my proverbial fingers hoping they can keep it real in the face of what will presumably be an avalanche of grifters looking to leech off the attention.

refulgentis 2y ago
for fellow readers: https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you-if-you-mention-ai-again/ https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you...
- sourcepluck 2y ago
  And, adjacent, in my opinion, readings like this piece, entitled "Hallucinating sense in the era of infinity-content": https://www.documentjournal.com/2024/05/technical-images-film01-angelicism-art-showtime-true-detective-shein/ https://www.documentjournal.com/2024/05/technical-images-fil...
  - refulgentis 2y ago
    This is really, really, fucking good, thank you.
    - sourcepluck 2y ago
      Nice to see someone having a similar reaction to me upon reading it. I too found it astoundingly good. I couldn't say what exactly about ludic's blog reminded me of it so strongly, but in any case, thrilled to share Caroline Busta's work with someone.

jacobyoder 2y ago

"Suffice it to say that while people are sincerely trying their best, our leaders are not even remotely equipped to handle the volume of people just outright lying to them about IT." I've tried to come up with some heuristic to determine whether or not a team is competent, good, or doomed. I've been exposed to all over the last... 8-10 years, and one of the key things I've noticed is the ratio of competent/skilled developers to the unskilled ones is a big ... indicator(?). Predictor? Colleague of mine has been working with a team - dev team has ranged from 5-8 people over the last few years. Few people seem to have any grasp of programming at all. Only two people - my colleague and one other - have ever taken projects from ideas to delivery, or even taken features from requests to successful rollout of already functioning software. The arguments that people get in to there - days or weeks of people 'researching' whether or not OAUTH 'really' requires 'refresh tokens' or whether it's really supposed to be a JWT. Management has some notion of 'every voice is legitimate and should be heard - we don't support bullying' and so on. If you have a team of 10, and 1 or 2 people are simply bad at having the ability to think somewhat abstractly, you can survive. If that number hits, say, 4-5... the team will struggle. A lot. You can keep things going, but it will be slow. And everything becomes a battle. If that number becomes 7 or 8, and you only really have 1-2 developers who are actually competent developers... things will continue to spiral downward. On the other side - I worked with a team of about 8-10 people on a 6 month contract. The larger org had another 40 or so folks, handling other projects, and support. Onboarding was great - I pushed production code in the first week. Everyone on the team was competent, including the juniors. I had more development experience, but they had more company experience, and it was really a relatively enjoyable engagement overall. It was refreshing to be able to ask anyone on the team questions, and either get a workable answer, or an "I'm not sure, let's check with XYZ" to get working answers. The "oh, yeah, it's ABC" when ABC is clearly not the answer stuff never happened. People committing code and pushing to production without ever having run the code at all - I've experienced that - didn't happen - that's happening to my colleague. The problem with a plurality of tech-incompetent folks in a tech group is that they honestly can not determine that they aren't competent. The only examples of competence are in the minority, and tend to not be trusted (even though that minority is the only portion that turns out working/functional code). Leaving ends up being the only option in those cases. My colleague is only at his place part time, and has hung around because they've gone through some restructuring where new folks were brought in, and... you hope that things might get better in a few months, then realize they don't.

javajosh 2y ago

Tempting to write a response piece, "Get me into data hell, I need the money."

ludicity 2y ago
In Australia, data jobs have a high floor but a low-ish ceiling at most places. Salaries start at around A$100K if you know how to interview, but 30 years of diligent practice only nets you around A$200K, and this would be considered an extreme success. If you're already good, just have coffee with executives/leads at places that take engineering seriously and you'll earn well, but there's no need to enter data hell as an employee at the typical company.
- pugz 2y ago
  I'm also based in Melbourne. I would add that all the above is true of maybe 95% of companies, but there are a few exceptions: big US-based tech companies. There are roles for devs that pay AU$250-$550K/year, including the data engineers. Unfortunately, there appears to be zero correlation with expertise and we have the exact same pain zone experience that I had when I was earning a third of this salary at an AU-headquartered company.

ramshanker 2y ago

Acting against your own better Judgement, I have to take that fight/ urge every few days. ;)

anitil 2y ago

I really enjoy this writing for a couple of reasons. Firstly, Australian. Secondly the acerbic wit just tickles my fancy > we're serverless, because how can you hurt yourself without a cutting-edge? Just perfect.