4 ms·
What must be communicated to management: It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there ca
by 1vuio0pswjnm7 6y ago
What must be communicated to management: It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic.
Magicians will be magicians, always hustling (bullshitting), but they will never have the value and job security of the blacksmith. The blacksmith can see the fruits of her own labour, whilst the magician must lie to herself and others in order to claim the blacksmith's value as her own.
If the blacksmith is good enough, she will earn the trust of management and management may consult the blacksmith in the selection of magicians. Management may ask the blacksmith to interview magicians and seek her advice on the final hiring decision.
The blacksmith may not carry the "prestige" of the hustling, bullshitting magician but she can command a high salary and dictate her own working conditions. This is only if management understands her value. What the magician thinks of the blacksmith is irrelevant.
Reliable blacksmiths are hard to find. Magicians are a dime-a-dozen.
- legerdemain 6y ago> It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. What? That runs counter to my experience at every company where I've either seen data engineers or worked as one. My observations of how management treats the two groups is this: Data engineers ("blacksmiths"): Blacksmiths are paid less. People think of them as less highly educated. Their work is less creative. When they are successful, their work is mostly invisible. They are interchangeable. People think of what blacksmiths do as more like scripting than writing code. Blacksmiths mostly work on configuring systems they didn't build. Blacksmiths do more troubleshooting than building. Their roles are focused on support. Data scientists ("magicians"): Magicians are paid more. Much more. People think of them as more highly educated. By definition, what they do is magic. They work on prominent projects. Their successes are highly visible. They build large systems that only they can comprehend. They use support staff to clear away mundane obstacles so they can focus on unique, highly creative aspects of work. Saying that we need more data engineers than data scientists is like saying that we need more janitors than CEOs. That's true, but it's true because we made it true by structuring projects around one prominent, well-paid person supported by a staff of invisible drudges. This smacks of the positive self-talk that QA and software testers used to give each other: "We are indispensable! We take pride in our craft! Nothing can ship without our signoff!" And then lots of companies reduced their QA or eliminated it wholesale by focusing on continuous delivery and changing consumer expectations of what "broken" or "acceptable" means. The same fate awaits data engineers.
- sgt101 6y agoGood Data Engineering is what enables good Data Science. With a good infrastructure you can go 100 times faster. Getting rid of Data Engineers means killing Data Science.
- legerdemain 6y agoSure. So in a few years, an all-in-one 80% solution like Palantir Foundry will come along and there will suddenly be a lot less demand for data engineers. Anecdotally, the former head of QA for Palantir UK is now the head of data engineering for Palantir UK, and Palantir does have an out-of-the-box, end-to-end, it-just-works product that handles 80% of ML workflows. You're betting your career that they won't put it in a box and sell it at commodity software prices?
- oivey 6y agoIf something like that becomes mainstream, data scientists who just glue together canned ML algorithms from libraries should also be concerned. The kind of automation that enables eliding finding and cleaning the right data probably can also manage running PCA and basic ML.
- sgt101 6y agoI remember the same claims for ISL Clementine before IBM bought it and turned it into SPSS modeler.
- lumost 6y agoThe way the evolution in software went, platforms became more capable and allowed individuals to automate more common tasks. QA/DevOps/SRE teams were consolidated and replaced with smaller platform teams which empowered internal engineers to quickly write scalable and well tested services. If data management, instrumentation, and ML tooling become sufficient then perhaps the data engineers will be replaced by a science platform team. Caveat is that many scientists are expected to publish novel research to advance their career. "Infrastructure" and "data management" do not tend to produce the kinds of sexy projects which are attractive to publish.
- Rury 6y ago>Without the right blacksmith, there can be no magic. Magicians will be magicians, always hustling (bullshitting) >management may consult the blacksmith in the selection of magicians I mean when you put it like that, why hire a magician (bullshitter), if the magic relies on the blacksmith? And, if management needs to consult someone (a blacksmith) on hiring (another blacksmith or for whatever reason a magician), then arguably management is made of magicians. Don't get me wrong, I agree with the point your making. It's just the problem is with BS and BS is rampant, or like you say: a dime-a-dozen.
- red_hare 6y ago> I mean when you put it like that, why hire a magician (bullshitter), if the magic relies on the blacksmith? Because magicians are better at the smoke and mirrors that drives funding rounds and closing big sales deals.