23 ms·

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit the

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a way that's detrimental to the rest of the team. You get people who just don't appreciate that there's other work to do than finding models, and that models depend on that other work to function. You also get a huge prestige gap, because inevitably management will think that there's a magician and a blacksmith. One guy needs to be paid a lot, and the other guy needs to be paid enough. These effects feed each other. Magician will say "where's my data" and expect blacksmith to make it, promptly. He won't do it himself, because spending time on mundane stuff makes the magic disappear. And not doing it yourself, or taking the time to understand it, will eventually lead to problems with the magic.

chadash 6y ago
I worked in investment banking (as an analyst, not an engineer), so very different part of finance, but this was my take as well. Companies might love to talk about how important engineers are, but at the end of the day, if you can't directly link someone to revenue, they get viewed as a cost center and take on second tier status in the organization. Then the same companies complain that they can't find enough (or retain) engineering talent. Not many places get the balance right. Silicon valley treats engineers well because for the most part, the value they bring is more obvious (and also, they don't threaten the existing hierarchy in the company). Curious to hear if anyone has had the opposite experience.
- isolli 6y ago
  Yes, quite a few developers left our investment bank and went to work for our suppliers (of trading software), stating they'd rather work somewhere where they're seen as value creators rather than a cost center.
- AtomicOrbital 6y ago
  I worked for 15 years as a software engineer at Morgan Stanley where they valued the process of taking a 3 martini lunch idea into a production platform so value of engineers was recognized and rewarded as such ... its somewhat easier to whip up a new financial wrinkle its a whole other level of magic to design and implement that idea when it takes 60 software developers 3 years to get that idea to market before the rest of the street ... of course the IT department was/is the largest budgeted portion at the entire bank and for a good reason
- mushbino 6y ago
  Engineers get paid well in SV because they are in demand, have lots of employment opportunities, and therefore are more difficult to retain.
  - mywittyname 6y ago
    And because their contributions can be tied back to revenue. You need both, demand for talent, as well as the ability & justification to pay for it. Engineers are in high demand all over the world. But most companies do not profit enough from technology to justify similar paying SV salaries.
    - pmiller2 6y ago
      Not always. Frequently, the connection between code that’s written today and revenue tomorrow is tenuous and difficult to package in a way that says “look at me! I’m valuable!” And, then there are those somewhat rare occasions where a project is not intended to increase revenue, and may even decrease it. At my last employer, we guesstimated that a project I worked on for months could possibly have ended up costing us $2M per year in revenue. That was both accepted and expected, because we were doing it to gain goodwill with users, but in such a way that it might end up pissing off a small minority of our customers. I really wish, just once, I could work on a project and put underneath it on my resume “Increased revenue by X%,” because I’ve never worked on anything that was so easy to directly trace back to the top line. Cost savings are another story, because engineers can fairly easily quantify how much less money is being spent by doing $THING a bit more efficiently....
    - mushbino 6y ago
      My partner leads engineering talent programs for a large SV company and I can assure you they do not track value added by engineers. In the overwhelming majority of cases the value added cannot be tracked. Now, if you're talking Product Managers, that's a bit of a different story. It's simply a supply and demand issue.
    - dzonga 6y ago
      tell that to Mr Nugget from The Wire. How much you get paid is a reflection on the CEO | culture not on the value you provide.
inthewoods 6y ago
That's interesting - I just completed book on Jim Simon/Renaissance (The Man Solved The Market). One of their early advantages was having a person who was just focused on acquiring and cleaning data. I expect that advantage has largely gone away at this point due to wide availability of market data but I thought it was interesting in the context of this article.
- curiousgal 6y ago
  Same for CFM too, they have an entire team working on alternative data and they feed it to a modeling team.
marcinzm 6y ago
As I see it you need people who have shallow knowledge of many areas and deep knowledge of one area. That lets you have a group of experts but ones that know enough about other areas of expertise to work with those other experts.
hntrader 6y ago
To add, quants that can't do the data engineering work are always crappy quants. I haven't seen a counter-example to that. Profitable models aren't going to be delivered on a silver platter. They need to be able to process pretty low level data effectively and build ad-hoc custom tools and data pipelines around that to test out their ideas. Otherwise they're constrained to the tools others have built and that massively narrows the search space that they're capable of traversing. The best quants are 1/3 statistician, 1/3 developer and 1/3 trader, in my view.
- twic 6y ago
  I'm not sure about crappy quants. Some people of the "quantitatively inclined trader who has learned Python" variety are never going to be good at the engineering side - it takes years to learn to be a good software engineer, and that's not a good use of time, for them, or for their employer. But they can still do useful work. The trick is to figure out how to work effectively with those people. Build infrastructure that keeps them on the rails, refactor their code, push them in the right direction, tell them when they've fucked up, teach them little things with high leverage. As long as that doesn't turn into being their slave, that's fine.
  - proverbialbunny 6y ago
    If they're using a dynamically typed language to do monetary calculations, it's not going to be ideal. Researchers do not need to have deep programming experience, but they have to be comfortable enough to use an environment that can lend itself itself to the problem at hand. On the quant side, unlike on the data science side, the barrier of entry on the programming side is a bit higher. To solve this problem many firms have their own internal programming language.
    - oivey 6y ago
      This is dogmatism swung too far in the other direction, IMO. There are many, many successful production code bases written in dynamic languages. In my own experience as a vision scientist/engineer, there is tremendous value in being able to quickly whip up a concept in Python and then being able to easily visualize the results. Doing this exploration in C++ is wasteful. Implementation takes much longer, the correctness brought by static typing is dubious since the code isn’t in prod, and the canned CV/visualization libraries are fewer and frequently suck in at least some way. That said, there is also tremendous value in understanding how to map your Python prototype into production code, too. Someone strong in this field can do both.
LeifCarrotson 6y ago
I really like your magician/blacksmith analogy. I'm in industrial automation, but it's much the same. Projects where someone developed a strategy but has never been involved in the details of a machine are doomed to failure (or at best to be unreliable and producing low quality parts). Projects built by machine fabricators are over-engineered, frequently late, and sometimes unprofitable, but damn if they don't work well. The main trouble, I think, is that when a shiny new contraption is brought to the king, it's too often the magicians doing the talking - whether they're speaking words of power or Common, their job is to talk. Meanwhile, the blacksmith is probably busy at in his workshop some ornate scroll work for the next thing, or repairing the previous gizmo, because he'd rather be hammering away at his anvil than talking. The higher you go in an org chart, the fewer the number of people who understand the work their company actually does, and the more voices you have between the workers and the decision-makers to take some of the credit for work as it passes up the chain.
- hef19898 6y ago
  That seems to be true in every field I can think of. The smaller the gap, or rather the more practical experience the strategy people have, the better a given org seems to be. One common issue I run into is that when the blacksmiths start talking, nobody listens.
- Rury 6y ago
  The funny thing is, even when it's unintentional, people seem to attribute credit to the magician rather than the blacksmith. At my workplace, I even have situations where I explicitly tell people: "I am familiar with X and how (some) of it works, but not all of it. I did not create it nor was it even my idea; all credit goes to Jim. If you need anything to do with X, you're best off asking Jim. But if Jim's too busy to help, I may be able to provide some minor help. Yet even when I do this, I somehow become the arbiter and authority for all problems and questions on X. 5 years go by and everyone thinks X was all my genius. And I hate it, because personally I do not like X created by Jim - even if everyone else does...
  - musingsole 6y ago
noodlenotes 6y ago
Also, a lot of data scientists find the science fun and the engineering boring. But they have overlapping skill sets - if you aren't good at one, you're probably not good at the other either. Somebody who shows up to a team with the goal of only modeling and pushing all the dirty engineering work to their teammates is basically a worst case scenario because 1) They probably aren't going to produce good models since they're not sensitive to data nuances, but now they've taken over ALL the modeling work. 2) They bring down the job satisfaction of everyone else on the team who would like to be doing at least some modeling. 3) They're sucking up the prestige that should be distributed over the entire team and management thinks they should be paid more for work that it turns out everybody thinks is more fun anyway. My number one advice to entry level data scientists is to not be this guy. Don't give your interviewers the impression that you won't do your own engineering work because they won't want someone who brings negative value to the team.
- NikolaNovak 6y ago
  Here's the tricky thing: I love your post; I agree with your post; but it takes a 90 degree turn at the end: "My number one advice to entry level data scientists is to not be this guy. " Everything most people are saying here indicates it's GREAT to be that guy. You're paid, you're respected, you get the fun parts, you love your job and it's pretty safe. It just happens to suck for everybody else including team and business... but it feels that in a practical sense, gist of everybody's actual unwitting message is "BE that guy, if you can" :-<<<
  - proverbialbunny 6y ago
    It sucks being that guy because everyone else ends up hating you. Depending on the work environment it's not a stretch to see software engineers complaining to management, sometimes going as far to create rumors to get the jr data scientist fired. So, no the grass is not greener. It's best to not be that person. This is why I go out of my way to prevent that scenario when I lead a team.
    - RobRivera 6y ago
oivey 6y ago
I think this insight exists across a lot of fields. Basically, if you want to be a really excellent magician you also better be a decent blacksmith. More concretely in this case, if you’re unable to do the data “engineering” yourself then it will close a lot of doors for interesting and novel work on the “science” side. Beyond that, if the scientist’s job just involves gluing sklearn models together I think that job is more on the engineering side of things than the supposed scientist usually wants to admit.
zzzeek 6y ago
maybe hedge funds would be able to find more people if they didn't only hire "guys".
- collyw 6y ago
  Can you show me a job ad where it is specific to guys?
wpietri 6y ago
> Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a way that's detrimental to the rest of the team. My god, this. These people make me bonkers. Especially because I feel like I have a bit of this tendency myself, the desire just to think big thoughts and do no actual work. Happily, I long ago learned that ideas were approximately worthless without labor, and that I anyway had much better ideas when laboring because it forced me to engage with the details. And yes, those people can poison a team. My best working experiences have all been with people who a) all valued actual work and b) believed that everybody could have good ideas.
- jnwatson 6y ago
  "I'm the idea guy" out of someone's mouth is the stark red-flag warning that their net contribution is 0.
  - mumblemumble 6y ago
    Or even negative. I've seen situations where the idea person is so busy being Mr. Toad that everyone around them is regularly scrambling to clean up messes and it ends up being a constant distraction from actually pushing projects through to completion.
  - the-pigeon 6y ago
    Ideas are so cheap and easy. Implementation is a long hard road. And where you learn your idea was vague enough that it had almost no value. And only through painstaking iteration can you turn it into something with value.
    - mbeex 6y ago
      > Ideas are so cheap and easy. I doubt this.
      - eftychis 6y ago
        It is true however. Consider it was easy to bring an idea in this world, and the hard part was the initial first thought; writing a paper/article painstakingly rigorously would be unnecessary. Writing a book would be a breeze and no author would ever go through more than a single draft. The idea was born beforehand, was complete correct and perfect, so putting everything down with words is just a matter of transcribing. An organization would not usually hire engineers with multiple degrees, but simply writers or an automated system that would listen and transcribe the idea. An idea is truly born and exists through a lot of effort and iteration and redefinement and refinement. P.S. There is a hard and subjective issue of where the line is drawn between ideation and minor uninteresting and menial maintainance/get the money in the bank work. We have to recall however that a parent brings their child to life and tags along through all the effort and work. A child is the result of years of high to low level of unpleasant work. Drawing an arbitrary line of when you shall stop giving as a parent is naive and egotistical.
lumost 6y ago
This problem only grows as the company scales and the science and engineering pieces are formally split along some role guideline. Inevitably if you treat a job role as a support role, you'll attract weaker individuals into that role then you would get if it wasn't considered a support role. The problem with Science oriented teams is that all roles other than the science role morph into science support roles over time. The same pattern used to occur with Engineers and QA, or Engineers and ops.
deleted 6y ago
[deleted]
- z3ncyberpunk 6y ago
  Then you end up with terrible engineers with zero practical experience who make terrible designs due to said lack of practical experience. Engineers out of college today have no zero clue how to run machines or do anything but create drawings that half work which is pathetic
notretarded 6y ago
Please don't change the status quo. I love my cushy job.
1vuio0pswjnm7 6y ago
What must be communicated to management: It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. Magicians will be magicians, always hustling (bullshitting), but they will never have the value and job security of the blacksmith. The blacksmith can see the fruits of her own labour, whilst the magician must lie to herself and others in order to claim the blacksmith's value as her own. If the blacksmith is good enough, she will earn the trust of management and management may consult the blacksmith in the selection of magicians. Management may ask the blacksmith to interview magicians and seek her advice on the final hiring decision. The blacksmith may not carry the "prestige" of the hustling, bullshitting magician but she can command a high salary and dictate her own working conditions. This is only if management understands her value. What the magician thinks of the blacksmith is irrelevant. Reliable blacksmiths are hard to find. Magicians are a dime-a-dozen.
- legerdemain 6y ago
  > It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. What? That runs counter to my experience at every company where I've either seen data engineers or worked as one. My observations of how management treats the two groups is this: Data engineers ("blacksmiths"): Blacksmiths are paid less. People think of them as less highly educated. Their work is less creative. When they are successful, their work is mostly invisible. They are interchangeable. People think of what blacksmiths do as more like scripting than writing code. Blacksmiths mostly work on configuring systems they didn't build. Blacksmiths do more troubleshooting than building. Their roles are focused on support. Data scientists ("magicians"): Magicians are paid more. Much more. People think of them as more highly educated. By definition, what they do is magic. They work on prominent projects. Their successes are highly visible. They build large systems that only they can comprehend. They use support staff to clear away mundane obstacles so they can focus on unique, highly creative aspects of work. Saying that we need more data engineers than data scientists is like saying that we need more janitors than CEOs. That's true, but it's true because we made it true by structuring projects around one prominent, well-paid person supported by a staff of invisible drudges. This smacks of the positive self-talk that QA and software testers used to give each other: "We are indispensable! We take pride in our craft! Nothing can ship without our signoff!" And then lots of companies reduced their QA or eliminated it wholesale by focusing on continuous delivery and changing consumer expectations of what "broken" or "acceptable" means. The same fate awaits data engineers.
schanq 6y ago
I see this same attitude about TDD adoption - teams in my company say things like “testing is for lackeys / that work is beneath us”, I.e. they see that as the responsibility of QA testers who are less important in their view. This is short sighted, arrogant and encourages similar problems with superiority complexes. TDD is still controversial in some circles, but engineers who have a deep understanding of both tests and implementation are far more valuable than those who only understand one side. Anyway, sorry for the somewhat off topic rant, but a lot of what you said resonated with me
- collyw 6y ago
  TDD is fine when you have a specification to work to. A lot of software development in the real world is quite "experimental". Requirements are poor so devs need to provide what is essentially a prototype and receive feedback until it is good enough.
  - schanq 6y ago
    Agreed and that’s what spiking is for! Once you have an idea of the solution then you can implement properly with TDD, which generally results in a much better design than just going with the original spike. However, I find it bizarre that many engineers think unit testing is beneath them. I can understand that people find it difficult to stick to the test first discipline. But it’s crazy to me that you wouldn’t want to write tests at all.. I feel so much happier with the additional protection.
leapis 6y ago
How do you achieve people like this? From my limited experience (college senior joining a hft firm shortly, so I've recently been in several quant finance SWE interview loops), firms seem to vastly downplay the financial aspects of the job for software engineers. Compounded on top of that, firms don't expect or encourage financial backgrounds for engineers (at least new grads)- the expectation is that whatever limited financial background we'll need to work will be given to us when it becomes necessary. Is this because it's easier (obviously) to teach a quant engineering than it is to teach an engineer quant finance? Or rather because it's expected now that traders will become the bridge between researcher models and implementation, and engineers will simply provide the underlying infrastructure to power these implementations?
kyawzazaw 6y ago
How does compensation tends to differ? And the education levels?
p5a0u9l 6y ago
this perception of classes within engineering is the greatest frustration of my career. People with a PhD or “scientist” in their title are not more valuable than engineers who end up being the ones to get things to work.
- amznthrowaway5 6y ago
  The "scientists" are often far less valuable or have outright negative contribution.
Borlands 6y ago
From experience, the magician will take every chance to make this divide greater, and sell their expertise, rather than grow with (and help grow in domain) the blacksmith skills. You end up with magic: closed siloed knowledge. “How would the blacksmith ever understand magic?” was often something thrown around by magicians at meetings. The (repetitive) blacksmith role is not an interesting one, digital revolution needs to come into place. Architects that build tools, self service systems are much more interesting.

chadash 6y ago

I worked in investment banking (as an analyst, not an engineer), so very different part of finance, but this was my take as well. Companies might love to talk about how important engineers are, but at the end of the day, if you can't directly link someone to revenue, they get viewed as a cost center and take on second tier status in the organization. Then the same companies complain that they can't find enough (or retain) engineering talent. Not many places get the balance right. Silicon valley treats engineers well because for the most part, the value they bring is more obvious (and also, they don't threaten the existing hierarchy in the company). Curious to hear if anyone has had the opposite experience.

isolli 6y ago
Yes, quite a few developers left our investment bank and went to work for our suppliers (of trading software), stating they'd rather work somewhere where they're seen as value creators rather than a cost center.
AtomicOrbital 6y ago
I worked for 15 years as a software engineer at Morgan Stanley where they valued the process of taking a 3 martini lunch idea into a production platform so value of engineers was recognized and rewarded as such ... its somewhat easier to whip up a new financial wrinkle its a whole other level of magic to design and implement that idea when it takes 60 software developers 3 years to get that idea to market before the rest of the street ... of course the IT department was/is the largest budgeted portion at the entire bank and for a good reason
mushbino 6y ago
Engineers get paid well in SV because they are in demand, have lots of employment opportunities, and therefore are more difficult to retain.
- mywittyname 6y ago
  And because their contributions can be tied back to revenue. You need both, demand for talent, as well as the ability & justification to pay for it. Engineers are in high demand all over the world. But most companies do not profit enough from technology to justify similar paying SV salaries.
  - pmiller2 6y ago
    Not always. Frequently, the connection between code that’s written today and revenue tomorrow is tenuous and difficult to package in a way that says “look at me! I’m valuable!” And, then there are those somewhat rare occasions where a project is not intended to increase revenue, and may even decrease it. At my last employer, we guesstimated that a project I worked on for months could possibly have ended up costing us $2M per year in revenue. That was both accepted and expected, because we were doing it to gain goodwill with users, but in such a way that it might end up pissing off a small minority of our customers. I really wish, just once, I could work on a project and put underneath it on my resume “Increased revenue by X%,” because I’ve never worked on anything that was so easy to directly trace back to the top line. Cost savings are another story, because engineers can fairly easily quantify how much less money is being spent by doing $THING a bit more efficiently....
  - mushbino 6y ago
    My partner leads engineering talent programs for a large SV company and I can assure you they do not track value added by engineers. In the overwhelming majority of cases the value added cannot be tracked. Now, if you're talking Product Managers, that's a bit of a different story. It's simply a supply and demand issue.
  - dzonga 6y ago
    tell that to Mr Nugget from The Wire. How much you get paid is a reflection on the CEO | culture not on the value you provide.

inthewoods 6y ago

That's interesting - I just completed book on Jim Simon/Renaissance (The Man Solved The Market). One of their early advantages was having a person who was just focused on acquiring and cleaning data. I expect that advantage has largely gone away at this point due to wide availability of market data but I thought it was interesting in the context of this article.

curiousgal 6y ago
Same for CFM too, they have an entire team working on alternative data and they feed it to a modeling team.

marcinzm 6y ago

As I see it you need people who have shallow knowledge of many areas and deep knowledge of one area. That lets you have a group of experts but ones that know enough about other areas of expertise to work with those other experts.

hntrader 6y ago

To add, quants that can't do the data engineering work are always crappy quants. I haven't seen a counter-example to that. Profitable models aren't going to be delivered on a silver platter. They need to be able to process pretty low level data effectively and build ad-hoc custom tools and data pipelines around that to test out their ideas. Otherwise they're constrained to the tools others have built and that massively narrows the search space that they're capable of traversing. The best quants are 1/3 statistician, 1/3 developer and 1/3 trader, in my view.

twic 6y ago
I'm not sure about crappy quants. Some people of the "quantitatively inclined trader who has learned Python" variety are never going to be good at the engineering side - it takes years to learn to be a good software engineer, and that's not a good use of time, for them, or for their employer. But they can still do useful work. The trick is to figure out how to work effectively with those people. Build infrastructure that keeps them on the rails, refactor their code, push them in the right direction, tell them when they've fucked up, teach them little things with high leverage. As long as that doesn't turn into being their slave, that's fine.
- proverbialbunny 6y ago
  If they're using a dynamically typed language to do monetary calculations, it's not going to be ideal. Researchers do not need to have deep programming experience, but they have to be comfortable enough to use an environment that can lend itself itself to the problem at hand. On the quant side, unlike on the data science side, the barrier of entry on the programming side is a bit higher. To solve this problem many firms have their own internal programming language.
  - oivey 6y ago
    This is dogmatism swung too far in the other direction, IMO. There are many, many successful production code bases written in dynamic languages. In my own experience as a vision scientist/engineer, there is tremendous value in being able to quickly whip up a concept in Python and then being able to easily visualize the results. Doing this exploration in C++ is wasteful. Implementation takes much longer, the correctness brought by static typing is dubious since the code isn’t in prod, and the canned CV/visualization libraries are fewer and frequently suck in at least some way. That said, there is also tremendous value in understanding how to map your Python prototype into production code, too. Someone strong in this field can do both.

LeifCarrotson 6y ago

I really like your magician/blacksmith analogy. I'm in industrial automation, but it's much the same. Projects where someone developed a strategy but has never been involved in the details of a machine are doomed to failure (or at best to be unreliable and producing low quality parts). Projects built by machine fabricators are over-engineered, frequently late, and sometimes unprofitable, but damn if they don't work well. The main trouble, I think, is that when a shiny new contraption is brought to the king, it's too often the magicians doing the talking - whether they're speaking words of power or Common, their job is to talk. Meanwhile, the blacksmith is probably busy at in his workshop some ornate scroll work for the next thing, or repairing the previous gizmo, because he'd rather be hammering away at his anvil than talking. The higher you go in an org chart, the fewer the number of people who understand the work their company actually does, and the more voices you have between the workers and the decision-makers to take some of the credit for work as it passes up the chain.

hef19898 6y ago
That seems to be true in every field I can think of. The smaller the gap, or rather the more practical experience the strategy people have, the better a given org seems to be. One common issue I run into is that when the blacksmiths start talking, nobody listens.
Rury 6y ago
The funny thing is, even when it's unintentional, people seem to attribute credit to the magician rather than the blacksmith. At my workplace, I even have situations where I explicitly tell people: "I am familiar with X and how (some) of it works, but not all of it. I did not create it nor was it even my idea; all credit goes to Jim. If you need anything to do with X, you're best off asking Jim. But if Jim's too busy to help, I may be able to provide some minor help. Yet even when I do this, I somehow become the arbiter and authority for all problems and questions on X. 5 years go by and everyone thinks X was all my genius. And I hate it, because personally I do not like X created by Jim - even if everyone else does...
- musingsole 6y ago

noodlenotes 6y ago

Also, a lot of data scientists find the science fun and the engineering boring. But they have overlapping skill sets - if you aren't good at one, you're probably not good at the other either. Somebody who shows up to a team with the goal of only modeling and pushing all the dirty engineering work to their teammates is basically a worst case scenario because 1) They probably aren't going to produce good models since they're not sensitive to data nuances, but now they've taken over ALL the modeling work. 2) They bring down the job satisfaction of everyone else on the team who would like to be doing at least some modeling. 3) They're sucking up the prestige that should be distributed over the entire team and management thinks they should be paid more for work that it turns out everybody thinks is more fun anyway. My number one advice to entry level data scientists is to not be this guy. Don't give your interviewers the impression that you won't do your own engineering work because they won't want someone who brings negative value to the team.

NikolaNovak 6y ago
Here's the tricky thing: I love your post; I agree with your post; but it takes a 90 degree turn at the end: "My number one advice to entry level data scientists is to not be this guy. " Everything most people are saying here indicates it's GREAT to be that guy. You're paid, you're respected, you get the fun parts, you love your job and it's pretty safe. It just happens to suck for everybody else including team and business... but it feels that in a practical sense, gist of everybody's actual unwitting message is "BE that guy, if you can" :-<<<
- proverbialbunny 6y ago
  It sucks being that guy because everyone else ends up hating you. Depending on the work environment it's not a stretch to see software engineers complaining to management, sometimes going as far to create rumors to get the jr data scientist fired. So, no the grass is not greener. It's best to not be that person. This is why I go out of my way to prevent that scenario when I lead a team.
  - RobRivera 6y ago

oivey 6y ago

I think this insight exists across a lot of fields. Basically, if you want to be a really excellent magician you also better be a decent blacksmith. More concretely in this case, if you’re unable to do the data “engineering” yourself then it will close a lot of doors for interesting and novel work on the “science” side. Beyond that, if the scientist’s job just involves gluing sklearn models together I think that job is more on the engineering side of things than the supposed scientist usually wants to admit.

zzzeek 6y ago

maybe hedge funds would be able to find more people if they didn't only hire "guys".

collyw 6y ago
Can you show me a job ad where it is specific to guys?

wpietri 6y ago

> Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a way that's detrimental to the rest of the team. My god, this. These people make me bonkers. Especially because I feel like I have a bit of this tendency myself, the desire just to think big thoughts and do no actual work. Happily, I long ago learned that ideas were approximately worthless without labor, and that I anyway had much better ideas when laboring because it forced me to engage with the details. And yes, those people can poison a team. My best working experiences have all been with people who a) all valued actual work and b) believed that everybody could have good ideas.

jnwatson 6y ago
"I'm the idea guy" out of someone's mouth is the stark red-flag warning that their net contribution is 0.
- mumblemumble 6y ago
  Or even negative. I've seen situations where the idea person is so busy being Mr. Toad that everyone around them is regularly scrambling to clean up messes and it ends up being a constant distraction from actually pushing projects through to completion.
- the-pigeon 6y ago
  Ideas are so cheap and easy. Implementation is a long hard road. And where you learn your idea was vague enough that it had almost no value. And only through painstaking iteration can you turn it into something with value.
  - mbeex 6y ago
    > Ideas are so cheap and easy. I doubt this.
    - eftychis 6y ago
      It is true however. Consider it was easy to bring an idea in this world, and the hard part was the initial first thought; writing a paper/article painstakingly rigorously would be unnecessary. Writing a book would be a breeze and no author would ever go through more than a single draft. The idea was born beforehand, was complete correct and perfect, so putting everything down with words is just a matter of transcribing. An organization would not usually hire engineers with multiple degrees, but simply writers or an automated system that would listen and transcribe the idea. An idea is truly born and exists through a lot of effort and iteration and redefinement and refinement. P.S. There is a hard and subjective issue of where the line is drawn between ideation and minor uninteresting and menial maintainance/get the money in the bank work. We have to recall however that a parent brings their child to life and tags along through all the effort and work. A child is the result of years of high to low level of unpleasant work. Drawing an arbitrary line of when you shall stop giving as a parent is naive and egotistical.

lumost 6y ago

This problem only grows as the company scales and the science and engineering pieces are formally split along some role guideline. Inevitably if you treat a job role as a support role, you'll attract weaker individuals into that role then you would get if it wasn't considered a support role. The problem with Science oriented teams is that all roles other than the science role morph into science support roles over time. The same pattern used to occur with Engineers and QA, or Engineers and ops.

deleted 6y ago

[deleted]

z3ncyberpunk 6y ago
Then you end up with terrible engineers with zero practical experience who make terrible designs due to said lack of practical experience. Engineers out of college today have no zero clue how to run machines or do anything but create drawings that half work which is pathetic

notretarded 6y ago

Please don't change the status quo. I love my cushy job.

1vuio0pswjnm7 6y ago

What must be communicated to management: It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. Magicians will be magicians, always hustling (bullshitting), but they will never have the value and job security of the blacksmith. The blacksmith can see the fruits of her own labour, whilst the magician must lie to herself and others in order to claim the blacksmith's value as her own. If the blacksmith is good enough, she will earn the trust of management and management may consult the blacksmith in the selection of magicians. Management may ask the blacksmith to interview magicians and seek her advice on the final hiring decision. The blacksmith may not carry the "prestige" of the hustling, bullshitting magician but she can command a high salary and dictate her own working conditions. This is only if management understands her value. What the magician thinks of the blacksmith is irrelevant. Reliable blacksmiths are hard to find. Magicians are a dime-a-dozen.

legerdemain 6y ago
> It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. What? That runs counter to my experience at every company where I've either seen data engineers or worked as one. My observations of how management treats the two groups is this: Data engineers ("blacksmiths"): Blacksmiths are paid less. People think of them as less highly educated. Their work is less creative. When they are successful, their work is mostly invisible. They are interchangeable. People think of what blacksmiths do as more like scripting than writing code. Blacksmiths mostly work on configuring systems they didn't build. Blacksmiths do more troubleshooting than building. Their roles are focused on support. Data scientists ("magicians"): Magicians are paid more. Much more. People think of them as more highly educated. By definition, what they do is magic. They work on prominent projects. Their successes are highly visible. They build large systems that only they can comprehend. They use support staff to clear away mundane obstacles so they can focus on unique, highly creative aspects of work. Saying that we need more data engineers than data scientists is like saying that we need more janitors than CEOs. That's true, but it's true because we made it true by structuring projects around one prominent, well-paid person supported by a staff of invisible drudges. This smacks of the positive self-talk that QA and software testers used to give each other: "We are indispensable! We take pride in our craft! Nothing can ship without our signoff!" And then lots of companies reduced their QA or eliminated it wholesale by focusing on continuous delivery and changing consumer expectations of what "broken" or "acceptable" means. The same fate awaits data engineers.

schanq 6y ago

I see this same attitude about TDD adoption - teams in my company say things like “testing is for lackeys / that work is beneath us”, I.e. they see that as the responsibility of QA testers who are less important in their view. This is short sighted, arrogant and encourages similar problems with superiority complexes. TDD is still controversial in some circles, but engineers who have a deep understanding of both tests and implementation are far more valuable than those who only understand one side. Anyway, sorry for the somewhat off topic rant, but a lot of what you said resonated with me

collyw 6y ago
TDD is fine when you have a specification to work to. A lot of software development in the real world is quite "experimental". Requirements are poor so devs need to provide what is essentially a prototype and receive feedback until it is good enough.
- schanq 6y ago
  Agreed and that’s what spiking is for! Once you have an idea of the solution then you can implement properly with TDD, which generally results in a much better design than just going with the original spike. However, I find it bizarre that many engineers think unit testing is beneath them. I can understand that people find it difficult to stick to the test first discipline. But it’s crazy to me that you wouldn’t want to write tests at all.. I feel so much happier with the additional protection.

leapis 6y ago

How do you achieve people like this? From my limited experience (college senior joining a hft firm shortly, so I've recently been in several quant finance SWE interview loops), firms seem to vastly downplay the financial aspects of the job for software engineers. Compounded on top of that, firms don't expect or encourage financial backgrounds for engineers (at least new grads)- the expectation is that whatever limited financial background we'll need to work will be given to us when it becomes necessary. Is this because it's easier (obviously) to teach a quant engineering than it is to teach an engineer quant finance? Or rather because it's expected now that traders will become the bridge between researcher models and implementation, and engineers will simply provide the underlying infrastructure to power these implementations?

kyawzazaw 6y ago

How does compensation tends to differ? And the education levels?

p5a0u9l 6y ago

this perception of classes within engineering is the greatest frustration of my career. People with a PhD or “scientist” in their title are not more valuable than engineers who end up being the ones to get things to work.

amznthrowaway5 6y ago
The "scientists" are often far less valuable or have outright negative contribution.

Borlands 6y ago

From experience, the magician will take every chance to make this divide greater, and sell their expertise, rather than grow with (and help grow in domain) the blacksmith skills. You end up with magic: closed siloed knowledge. “How would the blacksmith ever understand magic?” was often something thrown around by magicians at meetings. The (repetitive) blacksmith role is not an interesting one, digital revolution needs to come into place. Architects that build tools, self service systems are much more interesting.