12 ms·

People always say this guy just has had bad luck with his employers but I live in Melbourne and work in data and reckon the whole industry is a scam. Like why

• le-mark 2y ago
  This jives with my experience at a financial services company. I once sat next to the “big data team” and the company 5 year plan was all about delivering analytics and ai to customers using their data the company housed. The team consisted of one guy (who had a business degree) and a lot of empty cubes they were trying to fill. A year later the company had been acquired and the big data initiative had evaporated.
• RangerScience 2y ago
  I have felt exactly this on regular full stack teams many, many times, so it’s also not just limited to data teams. IMO a major factor here is that software engineering is both opaque and esoteric - at least with physical engineering, there’s something people can look at and think they understand.
  • baazaa 2y ago
    My theory is that data is worse again because at least if you're making a website you're expected to end up with a website. The process is opaque and esoteric, but the end-product is somewhat tangible. A lot of data projects are moving and transforming data no-one cares about. They can fail completely silently, a manager can lie and say 'we've successfully built the data platform which is going to enable AI analytics' and it'll be like a misconfigured S3 or something. No-one's checking the end-product or even understands what it's meant to be.
    • RangerScience 2y ago
      Excellent point… and one I should know. I spent about 6mo as a data eng (only one at the startup) and long after, found out no one ever had a clue what I was talking about in standup. (To be fair, I was self-teaching, and no-one else knew anything so)
• tiew9Vii 2y ago
  Reluctantly worked on AU data projects for maybe the past decade. I don't classify myself as a data engineer, in fact I hate data engineering or data related work which is glorified ETL and SQL most of the time. They are the worst, broken projects I've done, not software engineering in the software engineering sense. I don't think I've worked on a good one yet despite the potential to be really interesting projects. I prefer general software projects doing a bit of everything as a generalist, data pays the bills though in AU. Not seen/heard of this person before but reading this specific blog post it all sounds very familiar, it's depressing. The "CTO" getting on stage taking a bunch of credit and everything being a mess or incomplete or lies is very familiar. Maybe not CTO but higher management. It's all smoke, mirrors, optics, self promoting, it works as these people end up making their way up the ladder when the lonely dev trying to do better work is just a dispensable cog in the wheel.
  • 6510 2y ago
    > Not seen/heard of this person before but reading this specific blog post it all sounds very familiar, it's depressing. Someone once told me he, as a form of therapy, rewrote the company he worked at in a few weekends. He never mentioned it to his coworkers, it was strictly a therapeutic effort. They apparently spend years "fixing" things without making any progress.
    • intelVISA 2y ago
      Most apps are trivial for a decent dev to reproduce, I'd wager the root problem is rarely the codebase: the org is rotting. Years of 'fixes' with no progress is like blaming the water for sinking a ship. Success attracts deadweight who (un)intentionally sandbag efforts to reverse this downward trend for their own self-preservation. I don't blame them, doubt there's a fix when the system requires most people work bullshit jobs instead of collecting UBI.
      • whstl 2y ago
        Bingo. The #1 thing I learned in consulting is that you can't build good software if the processes and structures are wrong in the first place. Ditto with off-the-shelf software. Something that takes a week in company 1 can take a year in company 2 purely because of organizational issues. Rotting organizations will produce rotting software.
• lelandbatey 2y ago
  I have seen data work well, but it only worked well in a situation where we had management focusing on two very tangible things that even the CEO could verify (since the CEO did know the product). Those tangible things were: 1. Accurate, auditable billing down to per-chargable entity/event 2. Dashboards for each customer that reflect THE SAME numbers as we generate in #1, so customers could see relevant info quicker than just waiting for the bill from #1 The only reason those things were valued and made a focus though was because a HUGE customer threatened to completely drop our company because that customer did an audit and noticed that we had overcharged them 3% because we were actually billing them on estimated numbers. That led to our CEO being personally yelled at by a much larger CEO, and our CEO (to his credit) didn't blame us (we'd raised the alarms that the bills were estimates and not auditable) but did say "this can never happen again, I trust you, do whatever needs to happen to make sure this never happens again." And once we had #1 solid and tight, we were able to leverage that solid auditable data to generate solid dashboard numbers that always squared with what showed on the bill.
  • RHSman2 2y ago
    A CEO worth working for
• denimnerd42 2y ago
  I lead a team on a large data project at an enormous bank, hundreds of devs on the project across 3 continents. My team took care of the integration and automation of the sdlc process. We moved from several generations of ETL applications (9 applications) netezza/teradata/mainframes/hive map reduce all to spark. The project was a huge cost savings and great success. Massive risk reduction by getting these systems all under 1 roof. We found a lot of issues with the original data. We automated the lineage generation, data quality, data integrity, etc. We developed a frame work that made everything batteries included. Transformations were done as linear set of SQL steps or a DAG of sql steps if required. You could do more complicated things in reusable plugins if needed. We had a rock solid old school scheduler application also. We had thousands of these jobs. We had an automated data comparison tool that cataloged old data and ran the original code vs the new code on the same set of data. I don't think it's impossible to pull off but it was a hard project for sure. Grew my career a ton.
  • deleted 2y ago
    [deleted]
  • RHSman2 2y ago
    What was the main reason for your success?
    • jonathanlydall 2y ago
      Not the person you’re replying to, but I would expect that a near universal answer to this across all kinds of projects (not just software) is effective collaboration and communication between stakeholders and teams. Despite no shortage of technical talent on large projects they can still often fail, and it’s because building a technically impressive thing doesn’t matter if it doesn’t do what business needs. So it’s about making sure you’re building the “right” thing that delivers on business’s actual needs, and the only way to find out what those are is through constant and ongoing good communication between technical and business people.
      • ozim 2y ago
• bob1029 2y ago
  > I don't think a lot of orgs which use data as core-infra could modernize I argue this is a happy conclusion, not a problem to be solved. What would "modern" bring to a bank except even more pain & suffering? Database technology invented in the 80-90s is more than sufficient for tracking information at the scale that 99% of financial institutions operate at today. Virtually every core conversion project I've ever heard of has been a failure or is currently a burning wreck on its way to the bottom. The only new bank projects that touch data and seem to succeed are LOB apps with highly curated experiences that are tightly integrated with the actual front/back office business. Having buy-in from staff regarding your UX is way more important than spinning out a 20 page AWS architecture diagram. The CTO can only take you so far through the vendor approval process at a bank. Retail operations (i.e. the people who are responsible for the brick & mortar branches) typically have substantially more pull in these organizations.
  • makeitdouble 2y ago
    > What would "modern" bring to a bank except even more pain & suffering? In the most simple term, a future. Except if your bank is literally too big to fail, at some point you have to either move on from 80s technology or at least bring in an adaptation layer, because your profit center have also moved on or you're facing harder competition. A typical example is banks getting merged: there will be a fight to see which system stays and which one disappears. If you froze your technology 4 decades ago it won't be your stack winning. [0] Another is the evolution legal frameworks: EU countries passed laws requiring interoperable APIs to perform standard banking operations. Being a customer of a decent bank or a fossilized one made a huge difference and the market grew a lot more competitive. People would start hedging their bets when legacy banks looked too far behind. [0] The most interesting and recent example of this is Mizuho bank just miserably failing at that task to the point the gov. intervened and anyone not married to them probably moved out. https://www.mizuhogroup.com/news/2021/06/20210615_2release_eng.html https://www.mizuhogroup.com/news/2021/06/20210615_2release_e...
    • iamacyborg 2y ago
      > If you froze your technology 4 decades ago it won't be your stack winning. I’m unclear whether this is bad for the business or just bad for folks hoping to keep their jobs.
• deleted 2y ago
  [deleted]
• atoav 2y ago
  The problem is that most orgs seem to do the wrong thing, because the incentives of the higher ups don't align with ehat is good for the org. E.g. if you are a bank ideally you'd like all your processes automated and streamlined with extremly transparent data flows etc — and you want as many of the banks employees to be proficient in these systems and constantly work on improving the systems within an controlled environment. In practise this is not the kind of thing that allows single managers to come across like heroes — so it doesn't happen that way and you get island solutions with duct-taped connections between.
  • ludicity 2y ago
    I also get to spend a lot of time with executives thanks to the blog's success, and part of it isn't just incentives, it's pure confusion. People have no idea what they're buying. I also get invitations to "sponsor events" now, since people see "director" on LinkedIn and think I have way more money than I do. Their business model seems to be flattering executives by inviting them to events where they can network with other rich people, then ask me for "sponsorship" money so that I can go into the room and brainwash them with my marketing material. I might even try it at some point to see if that's an accurate read.
    • disgruntledphd2 2y ago
      If you want to make your consultancy a success, you probably should attend some of these and see if they help you get business.
      • ludicity 2y ago
        We're bootstrapped so we unfortunately can't fling money at sponsorships. Or rather, we can, but it would cut into our runway quite a bit, and we have more promising avenues to pursue. If we acquire bad clients, the type that would let themselves be brainwashed and who status-seek by attending these events, that's just going to be as dumb as a regular job but without the luxuries afforded to employees.
• ludicity 2y ago
  Author here. I now know some places in Melbourne that have a good success rate on projects. Some of them are so small as to be invisible and rarely hire. One uses two specific independent recruiters or internal referrals. As far as I know, they are extremely profitable because the competition is a joke. For many organizations, the success rate is indeed 0%. A Group of Eight university (our top 8 universities nationwide), for example, sent me a job description a few months ago where they misspelled the word engineer, and left change tracking on in the Word document. This allowed me to walk through the profiles of the people running their data projects, and it was super obvious that many of the people involved aren't going to do a good job. They could have saved millions by having a random HN person eyeball the CVs of their chosen leadership team.
  • svilen_dobrev 2y ago
    hey. Congratulations on your decision. i think it all goes deeper in overall culture/attitude there. i was in Melbourne in 2012.. with idea to relocate wholesale, 2nd time. Worked 2 months at some "startup", that fired me when i finished the task given.. Seems it was cheaper to hire "permanent" then fire, rather than take someone on 2 months contract. So that's one red light on the dashboard.. There were other redlights from overall "society", feeling something-is-wrong but i did ignore them for quite a while - people have become evil, etc.. Then i started going around places and mailing my cv here or there (with 22y of experience making software, by that time),.. ibm, ernst&young, you-name-it.. to no avail, and more red-lights flashed on me.. And one day visited some kind of meetup, organised/held in some wellknown company. Seemingly it was kind-of "hiring" event or so, we grouped in teams of 3-4-5 people, with half from company, and other half outsiders.. and went solving some problems of theirs. Or that was the "label". Any solution that any of outsiders suggested, was shot down, with somewhat vague reasons, that at the end started to sound like "if we solve this there'll be no job tomorrow". And Smile :) Lots of smiles. Empty ones. That was one of the Last red lights on my dashboard. Whether it was a financial balloon pressing everyone so they only smiled and did nothing, in order to pay the mortgage, or something else, i don't know. Next day i watched Sacrifice/1986/Offret by Tarkovsky, and.. bought a ticket out. Discontinued my oz dream. For good. quoting meself, from 2007-8: "with time, places change people. Other way happens noticeably only while coming in - or switching on." have fun
• nyarlathotep_ 2y ago
  >> I live in Melbourne and work in data and reckon the whole industry is a scam. You needn't live in Australia to reach that conclusion.
• chubs 2y ago
  Do you think perhaps the problem is rooted in people being dishonest, and honest people are driven mad by it all and self-select out? The dead-sea effect?
  • ludicity 2y ago
    It's so many things. Dishonesty, lack of technical competence, political pressure, hype, organization structure, and incentives. If I had to summarize though, it's that the median performance in any field will be at much lower levels than outsiders expect, and some fields with hazier results have this level set very, very, very low, especially when they're hyped up. But also that the market is actually at least a little bit efficient, but over long time scales. I think there's a 50%+ chance that the role of Chief Data Officer begins to die off, but also that it'll be replaced by something silly.
• DeathArrow 2y ago
  >Because it doesn't matter, every data team is a cost-centre that unscrupulous managers use to launch their careers by saying they're big on AI. If you have lots of data flowing and have full teams "working" 24/7 on it, does it really matter if that data is junk and that is not processed in a meaningful way? You can still ask AI to generate some nice looking charts with big numbers to show to investors. Investors like nice charts and big numbers. Or so, some businesses people think. But in all reality the investors will ask questions like: how will this solve problems for customers, how do you intend to sell this to customers, how much does it cost, how will this generate me money. Unless those investors plan an early exit by finding other, more gullible investors than them, kind of like knowingly investing early in a Ponzi scheme.
• CalRobert 2y ago
  Ten years ago data engineering was another discipline in software engineering, like backend or frontend. Somewhere along the line the term was co-opted by “I can maybe barely string together some untested airflow pipelines” and it means something much different now.
  • WorldMaker 2y ago
    It is said that every major, still living COBOL program contains a bespoke, (sometimes poorly optimized) database engine with no standard query language, the only query tool was more program code. Perhaps the longevity of Mainframes points to there was some wizardry/safety lost in standardizing databases, giving people the impression that data itself was standard and too many tools to footgun data into foot pain, that we lost when databases were defined entirely as COBOL internals? (Not that we haven't gained a lot from modern database tools, just something to think about that maybe the data siloes were good sometimes, too.)
• akdor1154 2y ago
  Agree.. I can tell you at least one Melbourne-based Flybuys retailer calculates your points with an unholy daily-scheduled stored procedure in Snowflake SQL, because.. big business dysfunction reasons lead to the data team being assigned to do it, and the data team didn't actually have any software engineer roles in it. At least it has tests.
  • jpmoral 2y ago
    Do you mean the points earned in-store that are then sent to Flybuys to add to your total? Or, god forbid, do they do their own total?
    • akdor1154 2y ago
      The former, in-store and online (which is beginning to touch on those business dysfunction reasons).
• photonthug 2y ago
  > it doesn't matter, every data team is a cost-centre that unscrupulous managers use to launch their careers by saying they're big on AI. So nothing works, no-one cares it doesn't work Yes. Lots of times the most important asset for these companies is actually contractual obligations in terms of exclusive access to data or customers. It doesn’t matter if the product works, you’ll have to buy the company to build a different one that does. But the (broken or nonsensical) product pushes up the value of mergers and acquisitions. If leadership completely makes shit up then they might go to jail, so, they burn X million on “work” and cloud spend as part of an elaborate argument that it should sell for 10X. > the industry is just so broken at this point I don't think we can do what we did 30 years ago. Well no, it’s never been easier to do high quality engineering, but mbas are in charge. They don’t think like philosophers or scientists and don’t traffic in common sense. For anyone questioning their life / career choices because of this, it’s not about you. An individual working in an environment like this can still be a craftsman of integrity if they focus on small problems and solve them well, but you need to be able to get satisfaction from that, not from some overall mission (which again, is probably fake). If you’re most motivated to work directly on architecture, unification, etc, and want to change lots of things then you will probably be miserable. But if you’re feeling shitty about the whole thing, it might help to realize that the actual nuts and bolts of adtech/martech data pipelines are much the same as the ones for cancer research or particle physics or climate science, so one can at least try and get transferable skills if circumstances are currently holding you hostage. Data isn’t a bullshit job. Leadership and management that just want to play games is the problem.
• neumann 2y ago
  100% agree with most of this. Most large organisations in Australia are just clueless about their needs and ride hype cycles with execs acting on FOMO. And they bought the lie that you need all these separate distinct engineers who are professionals in a niche. They think they are building a high end kitchen by having specialists and think they are hiring sous-chefs etc, but most of the time they are hiring line cooks, and in fact most of the time they should just be hiring short order cooks. Most teams and businesses I know who are doing great things with data are mostly smaller companies and tech built up start-ups and hire one of two types of people. Generalists who can fill the gaps and are interested in everything, and staff software engineers who are amazing developers and asked to put together data pipelines with consideration of the whole infrastructure. When I was looking for work, all the good companies were not hiring data engineers or machine learning engineers. They were hiring Senior Software Engineers with a remit to build data infrastructure, or build machine learning models. And they immediately removes 90% of the noise applicants.
• fphhotchips 2y ago
  Melbourne is easily the worst city in the country for this. Most of the tech sector is in the very large enterprise space lead by the banks, and as a result it's who you know and whether you went to Melbourne Grammar or Geelong Grammar that will determine which company you work for once you reach a certain level. Sydney is better just because there's more smaller stuff going on, and because CBA is better than NAB and ANZ combined on tech. (I hate Sydney otherwise and am based out of Melbourne) Some places in Melbourne get real work done, even in the data sector. They're hard to find, but they exist.