5 ms·

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with

by whack 8y ago

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to improve upon their ideas, but for the most part, I was happy to focus on implementing their ideas, in the most clean, elegant, robust and testable manner possible. I was happy to do the "plumbing" work of improving upon our tech stack and architecture, in order to make the entire system better functioning and easier to maintain. According to the author, I'm supposed to resent the fact that I'm a "doer/plumber", and not a "thinker". In reality, it was the opposite. Do I really want to spend my entire day reading the Bloomberg manual and figuring out which tables/columns will give us the data we want, and the nuances of what this dataset does and does not cover? Sorry, I have zero interest in doing that. I enjoy programming. I enjoy system design. I enjoy building stuff. I have zero interest in becoming an expert on how to interpret the Bloomberg symbology file. Besides, if I ever left the financial industry and joined a tech company, that knowledge will become completely useless. Did I or anyone consider myself to be a "menial" plumber? I don't think so. I was getting paid hundreds of thousands of dollars, because the "thinkers" recognized the value that I brought to the table. They appreciated that I could quickly and robustly implement the ideas that they had, and keep the system running smoothly without hiccups. They recognized anyone can do a "good enough" job, but it's much much harder to find someone who can do a great job. And for my part, I was perfectly happy to be that guy. If you're someone who wants to expand your breadth and take on more "thinker" responsibilities, more power to you. But just don't forget that there are people like me out there too. There's no shame in being an excellent "doer".

internet555 8y ago
What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything but spewing out whatever pops into their head. Half is nonsense, one-quarter would be actively destructive if you tried to implement it, they always seem to know everything about everything but whenever it’s something you know really well you can tell that they are very confused, etc. when I meet these people now I just think “oh, you’re one of those guys who is good at saying a lot of things” and then move on. Oh well
- matwood 8y ago
  Talking is easy, executing is hard. Executing requires discipline which so many people seem to lack. One of the reasons I love making people write down their idea (myself included) before talking about it is that writing forces a small initial execution step. Even a step this small can often filter the useless ideas away.
  - yeukhon 8y ago
    This is where design doc should be required. At my company we engineers are required to come up with a design doc and share with the whole org for feedback then a design meeting that takes place every week. At first I thought this was a step back because I felt it was a water fall but after writing my own design doc I quickly realized “talking is really fucking cheap.” Sitting down and write a doc that considers as many corner cases and implementation really produce high quality work. It’s all about discipline indeed.
    - Annatar 8y ago
      "At my company we engineers are required to come up with a design doc and share with the whole org for feedback then a design meeting that takes place every week." Wow, true system engineering! Your company sounds like a good place to do some professional work.
      - yeukhon 8y ago
        Feel free to visit oscar career page.
    - matwood 8y ago
      That's great. A design doc is even a step beyond just writing down the idea. Bezos is famous for making people write one pagers to pass out prior to talking about any idea. I see one pagers as way to quickly weed through a bunch of ideas, and a design doc as the next step to determine feasibility.
Dowwie 8y ago
Yeah, seems like an aging configuration among quants and various teams in IT at many finance companies. Finance may have been ahead with adopting data science but it seems that many others reached parity or even surpassed in capabilities.
jerglingu 8y ago
I'm wondering if anybody lives in the hypothetical perfect world scenario the author writes about. I'm at one of the larger tech companies and it's inconceivable that something like this could exist (though the churn here is extremely high - a mature shop with longstanding membership could implement the hypothetical in some form). Everything sounds nice when dreaming it up in one's head, but discounts the reality of things. You can only lead a horse to water so many times before recognizing that they just will not drink the water themselves - some people actively refuse to implement solutions, no matter how convenient the building process might be. And then the more you burden them with things like SLAs, performance, etc., the more of shit show things become. There are some forward-moving, solid "soft skill" analysts/data scientists that can make this happen. But by and large they shouldn't all be held to this standard. Maybe my standards/expectations have been soured too much and I'm too pessimistic, but as a whole they're just not cut out for this kind of stuff. Which is fine - being a "doer" is easy to begin with, and over time the more that you're able to automate as a data engineer, the more trivially easy ETL/everything else becomes.
- itronitron 8y ago
  I've experienced that hypothetical perfect world, it comprised teams of data scientists, developers, and system engineers working on making progress towards multiple distinct 'business insights' and operating within a big data ecosystem. People's job roles were tightly defined but their participation in tasks was largely determined through self-selection and reputation.
- sagichmal 8y ago
  I think you may be experiencing a local minimum, so to speak. Data scientist programmers who "simply refuse" to think about basic productionization criteria like SLAs shouldn't remain employed.
  - Annatar 8y ago
    They shouldn't, but they do because understanding a little bit of mathematics is at a premium these days and it's very much in vogue to crap all over computer science majors even though we had the same kind of mathematics curriculum as they at the university.
RhodesianHunter 8y ago
Completely agree with your take, and as an engineer in a similar role, the post rubbed me the wrong way. I don't find ETL work soul-sucking, and I certainly don't think that my colleagues or I are mediocre.
- ak39 8y ago
  “ETL” also involves data modeling. This is a thinking role beyond many people’s understanding. Quite often the “Thinkers” demand data but have very little understanding how to structure it when you get temporal aspects (slow changing dimensions etc), new types of data (rules for types) etc. These and more are quite often left/relegated to ETL guys as housekeeping. Except it isn’t. It’s architecting the house.
- Annatar 8y ago
  I was once accused by a Windows C++ programmer who just couldn't grok UNIX to be a "string cutter", but honestly, there is nothing so viscerally information technology as extract-transform-load. The irony of his accusation was that it was just one of many facets of things that I do and can/know how to do. ETL is really the core of IT by definition.
  - badpun 8y ago
    A lot of the hardcore C++ programmers look down on the whole IT area of programming (as opposed to systems, graphics, embedded etc.), maybe that was the reason.
    - Annatar 8y ago
      No, this guy doesn't know anything but Windows.
zimablue 8y ago
I do this job and I'd say it's only enjoyable for now because I have complete freedom and I'm still learning. Splitting up the parts: Design architecture Wire up pipelines (once architecture is decided, this tends to declarative, connected is choosing schemas). Data science I'd definitely not want to be stuck doing stage 2 forever, would prefer 3. I think you're saying that you enjoyed a job which was some 1 and some 2. I'm sure there's someone out there who wants to wire up pipelines with no engineering and no analysis all day but I'd imagine it's a rare breed. Edit: I think the important distinction is team/company size. Doing a bit of everything as a 1 man team is challenging, if you have a team where devops/engineering/reports/tools have been chosen/built/standardized by specialist and you really are just wiring pipelines up, I think that would be tough. On the other hand being in a small team condemns you to always be doing the same fractions of work because there's noone to hand off to.
carlmr 8y ago
Completely agree, especially as people that stop "doing" also often forget how to "think" about the actual problems properly.
dsclough 8y ago
I'm working for a small hedge fund in basically the same role, do you have any resources you could recommend for learning how to do these things the "right" way?
dv_dt 8y ago
> I was getting paid hundreds of thousands of dollars, because the "thinkers" recognized the value that I brought to the table. They appreciated... I think this is the key idea there. It's good that you found a situation where you're both appreciated, and compensated for it. It's too easy for engineers to be devalued as replaceable cogs in the pipeline of things that need to happen to bring in revenue.

by whack 8y ago

internet555 8y ago

What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything but spewing out whatever pops into their head. Half is nonsense, one-quarter would be actively destructive if you tried to implement it, they always seem to know everything about everything but whenever it’s something you know really well you can tell that they are very confused, etc. when I meet these people now I just think “oh, you’re one of those guys who is good at saying a lot of things” and then move on. Oh well

matwood 8y ago
Talking is easy, executing is hard. Executing requires discipline which so many people seem to lack. One of the reasons I love making people write down their idea (myself included) before talking about it is that writing forces a small initial execution step. Even a step this small can often filter the useless ideas away.
- yeukhon 8y ago
  This is where design doc should be required. At my company we engineers are required to come up with a design doc and share with the whole org for feedback then a design meeting that takes place every week. At first I thought this was a step back because I felt it was a water fall but after writing my own design doc I quickly realized “talking is really fucking cheap.” Sitting down and write a doc that considers as many corner cases and implementation really produce high quality work. It’s all about discipline indeed.
  - Annatar 8y ago
    "At my company we engineers are required to come up with a design doc and share with the whole org for feedback then a design meeting that takes place every week." Wow, true system engineering! Your company sounds like a good place to do some professional work.
    - yeukhon 8y ago
      Feel free to visit oscar career page.
  - matwood 8y ago
    That's great. A design doc is even a step beyond just writing down the idea. Bezos is famous for making people write one pagers to pass out prior to talking about any idea. I see one pagers as way to quickly weed through a bunch of ideas, and a design doc as the next step to determine feasibility.

Dowwie 8y ago

Yeah, seems like an aging configuration among quants and various teams in IT at many finance companies. Finance may have been ahead with adopting data science but it seems that many others reached parity or even surpassed in capabilities.

jerglingu 8y ago

I'm wondering if anybody lives in the hypothetical perfect world scenario the author writes about. I'm at one of the larger tech companies and it's inconceivable that something like this could exist (though the churn here is extremely high - a mature shop with longstanding membership could implement the hypothetical in some form). Everything sounds nice when dreaming it up in one's head, but discounts the reality of things. You can only lead a horse to water so many times before recognizing that they just will not drink the water themselves - some people actively refuse to implement solutions, no matter how convenient the building process might be. And then the more you burden them with things like SLAs, performance, etc., the more of shit show things become. There are some forward-moving, solid "soft skill" analysts/data scientists that can make this happen. But by and large they shouldn't all be held to this standard. Maybe my standards/expectations have been soured too much and I'm too pessimistic, but as a whole they're just not cut out for this kind of stuff. Which is fine - being a "doer" is easy to begin with, and over time the more that you're able to automate as a data engineer, the more trivially easy ETL/everything else becomes.

itronitron 8y ago
I've experienced that hypothetical perfect world, it comprised teams of data scientists, developers, and system engineers working on making progress towards multiple distinct 'business insights' and operating within a big data ecosystem. People's job roles were tightly defined but their participation in tasks was largely determined through self-selection and reputation.
sagichmal 8y ago
I think you may be experiencing a local minimum, so to speak. Data scientist programmers who "simply refuse" to think about basic productionization criteria like SLAs shouldn't remain employed.
- Annatar 8y ago
  They shouldn't, but they do because understanding a little bit of mathematics is at a premium these days and it's very much in vogue to crap all over computer science majors even though we had the same kind of mathematics curriculum as they at the university.

RhodesianHunter 8y ago

Completely agree with your take, and as an engineer in a similar role, the post rubbed me the wrong way. I don't find ETL work soul-sucking, and I certainly don't think that my colleagues or I are mediocre.

ak39 8y ago
“ETL” also involves data modeling. This is a thinking role beyond many people’s understanding. Quite often the “Thinkers” demand data but have very little understanding how to structure it when you get temporal aspects (slow changing dimensions etc), new types of data (rules for types) etc. These and more are quite often left/relegated to ETL guys as housekeeping. Except it isn’t. It’s architecting the house.
Annatar 8y ago
I was once accused by a Windows C++ programmer who just couldn't grok UNIX to be a "string cutter", but honestly, there is nothing so viscerally information technology as extract-transform-load. The irony of his accusation was that it was just one of many facets of things that I do and can/know how to do. ETL is really the core of IT by definition.
- badpun 8y ago
  A lot of the hardcore C++ programmers look down on the whole IT area of programming (as opposed to systems, graphics, embedded etc.), maybe that was the reason.
  - Annatar 8y ago
    No, this guy doesn't know anything but Windows.

zimablue 8y ago

I do this job and I'd say it's only enjoyable for now because I have complete freedom and I'm still learning. Splitting up the parts: Design architecture Wire up pipelines (once architecture is decided, this tends to declarative, connected is choosing schemas). Data science I'd definitely not want to be stuck doing stage 2 forever, would prefer 3. I think you're saying that you enjoyed a job which was some 1 and some 2. I'm sure there's someone out there who wants to wire up pipelines with no engineering and no analysis all day but I'd imagine it's a rare breed. Edit: I think the important distinction is team/company size. Doing a bit of everything as a 1 man team is challenging, if you have a team where devops/engineering/reports/tools have been chosen/built/standardized by specialist and you really are just wiring pipelines up, I think that would be tough. On the other hand being in a small team condemns you to always be doing the same fractions of work because there's noone to hand off to.

carlmr 8y ago

Completely agree, especially as people that stop "doing" also often forget how to "think" about the actual problems properly.

dsclough 8y ago

I'm working for a small hedge fund in basically the same role, do you have any resources you could recommend for learning how to do these things the "right" way?

dv_dt 8y ago

> I was getting paid hundreds of thousands of dollars, because the "thinkers" recognized the value that I brought to the table. They appreciated... I think this is the key idea there. It's good that you found a situation where you're both appreciated, and compensated for it. It's too easy for engineers to be devalued as replaceable cogs in the pipeline of things that need to happen to bring in revenue.