5 ms·

That's not universal. The lat thing I need about any of my colleagues is to learning all gotchas, at the same time as integrating into our team. Some tech I'l

That's not universal. The lat thing I need about any of my colleagues is to learning all gotchas, at the same time as integrating into our team. Some tech I'll argue against, other tech I will require. Some tech is fundamental to a domain. Some tech is a nice to have. Some tech is just generic. My current team will not hire anyone for a senior role, without Spark experience. No matter how good you are in C++...

morelisp 5y ago
> My current team will not hire anyone for a senior role, without Spark experience. An example close to my heart. The company I'm currently at had a totally ossified Spark monolith and data eng team because they only wanted Spark developers. Drop one C++ developer in there to start spreading the good news about data-oriented design and suddenly there are a dozen previously-intractable "big data" problems we can rip out of the ball of mud and run outside of the cluster. Drop in another SRE who doesn't know shit about Java but can actually write shell scripts, and hey alerts from bad orchestration from weekly to twice yearly. Yeah if you have a Spark job, you probably need a Spark dev. But if you have a team of Spark devs, suddenly everything that involves more than 1MB of input data looks like a Spark problem and you're in for a real bad time in a couple years. C++ programmers know map/reduce/filter too.
- JAlexoid 5y ago
  Well... Funny enough. The person that our team of Spark engineers is cleaning up after was - a C++ dev. For some reason we, Spark engineers, had to rewrite his all Spark application into different components with relevant tech. Thanks for generalizing Spark engineers to an insulting level, though.
  - hippich 5y ago
    I can't know for sure, but I think person you are replying to meant that in a team lacking diversity (in this particular case, diversity of expertise) all solutions tends to receive same, less than optimal treatment no matter what. If anything, it is insult of managers who create such bubbles, not engineers stuck in these bubbles.
  - morelisp 5y ago
    > Thanks for generalizing Spark engineers to an insulting level, though. I have no clue how you got that from what I wrote. I even said a Spark project needs a Spark dev! It really sounds to me like you're the one generalizing from "the one C++ dev we hired couldn't do data engineering right at all" to "senior programmers are dangerous to us unless they are Spark developers" (which imo, is more indicative of serious deficiencies in your architecture / design process), to "all senior programmers crossing subdomains are dangerous."

morelisp 5y ago

> My current team will not hire anyone for a senior role, without Spark experience. An example close to my heart. The company I'm currently at had a totally ossified Spark monolith and data eng team because they only wanted Spark developers. Drop one C++ developer in there to start spreading the good news about data-oriented design and suddenly there are a dozen previously-intractable "big data" problems we can rip out of the ball of mud and run outside of the cluster. Drop in another SRE who doesn't know shit about Java but can actually write shell scripts, and hey alerts from bad orchestration from weekly to twice yearly. Yeah if you have a Spark job, you probably need a Spark dev. But if you have a team of Spark devs, suddenly everything that involves more than 1MB of input data looks like a Spark problem and you're in for a real bad time in a couple years. C++ programmers know map/reduce/filter too.

JAlexoid 5y ago
Well... Funny enough. The person that our team of Spark engineers is cleaning up after was - a C++ dev. For some reason we, Spark engineers, had to rewrite his all Spark application into different components with relevant tech. Thanks for generalizing Spark engineers to an insulting level, though.
- hippich 5y ago
  I can't know for sure, but I think person you are replying to meant that in a team lacking diversity (in this particular case, diversity of expertise) all solutions tends to receive same, less than optimal treatment no matter what. If anything, it is insult of managers who create such bubbles, not engineers stuck in these bubbles.
- morelisp 5y ago
  > Thanks for generalizing Spark engineers to an insulting level, though. I have no clue how you got that from what I wrote. I even said a Spark project needs a Spark dev! It really sounds to me like you're the one generalizing from "the one C++ dev we hired couldn't do data engineering right at all" to "senior programmers are dangerous to us unless they are Spark developers" (which imo, is more indicative of serious deficiencies in your architecture / design process), to "all senior programmers crossing subdomains are dangerous."