4 ms·

My professional observation (as ml researcher at big tech): These companies hire a lot of engineers straight out of undergrad/master's degrees. The interviews

My professional observation (as ml researcher at big tech): These companies hire a lot of engineers straight out of undergrad/master's degrees. The interviews test leetcode knowledge, and today lots of degrees are heavy on Python-scripted ML homework. The result is companies with billion dollar funding and world-changing goals having a lot of their code look like complete spaghetti. And this is the engineers who are meant to clean up research scientist code. Scientists generally don't feel like it's their responsibility to write strong code. Systems-side teams/orgs have better code, but essentially as soon as you enter the 'ml engineer/research engineer/research scientist' layer, it's doomed.

gerash 7y ago
I don't know why you excluded PhDs but their code aren't any better necessarily. Hiring straight from school doesn't necessarily mean bad S/W engineering skills. But generally the research code is either made hastily during exploration or by people not having enough Software Engineering background. Many research scientist I've seen at large tech companies don't have good CS background and write code that's either unreadable, unscalable, unmaintainable, or buggy at times. They often have good ideas but don't do much software design. Having been both at infrastructure teams and research teams, there are certain individuals in research orgs joining straight from school who think they are responsible with coming with a new and sexy thing and other engineers are responsible to run in production. It's like computing eigenvectors in Matlab or Python over a toy dataset and thinking you've done the bulk of the work of computing PageRank in production and should receive all the credit for a full search engine. That attitude is a red flag to me.
- high_derivative 7y ago
  I don't see how what you are saying is contradicting me. The point is that the scientists generally don't view it as their job, whereas the ml engineers/research engineers typically have it as their job to do the software architecture/engineering side.
codingslave 7y ago
Yeah I'm waiting for the backlash when companies realize this
gdb 7y ago
(I work at OpenAI. Before that, I worked at Stripe. I've spent most of my software career thinking about how to build effective engineering cultures.) I think this code is actually well-written and maintainable. This is proven in practice because we've adopted it many places in OpenAI, and I've personally found it very easy to adapt to other use-cases (certainly much more so than the from-scratch Transformer implementations I've written!). As https://news.ycombinator.com/item?id=21456605 https://news.ycombinator.com/item?id=21456605 points out, the complexity of the code arises from the complexity of the underlying algorithm. Complexity due to software engineering concerns, like Tensorflow scopes, are elegantly handled. [edited for clarity:] Writing a Transformer in 174 lines of code requires a lot of deep thinking about the right underlying abstractions. > but essentially as soon as you enter the 'ml engineer/research engineer/research scientist' layer, it's doomed. We actually don't do this! Our only official technical title is "member of technical staff". (People sometimes choose to self-identify as an engineer or researcher, so you might see that on LinkedIn, but we don't have a distinction internally.) Everyone is responsible for their own code, and people care quite a bit about writing code that others can build on.
- garmaine 7y ago
  I’m very sorry to see someone who obviously cares so much to be defending this code. This does not follow best practices, and using complexity of the underlying algorithm is just an excuse. Complex code can be beautiful and well documented. Writing a complex method in 174 lines is not elegant nor beautiful. Writing a well documented file that can take an engineer in a different specialty and bring them up to speed in 1,000 lines is.
  - gdb 7y ago
    We also have code like that. For example, that's the explicit goal of the Spinning Up repo: https://github.com/openai/spinningup/blob/master/spinup/algos/ddpg/ddpg.py https://github.com/openai/spinningup/blob/master/spinup/algo... In practice, it's much harder to use that code, and we tend not to consume code like that internally. There's a real tradeoff!

gerash 7y ago

I don't know why you excluded PhDs but their code aren't any better necessarily. Hiring straight from school doesn't necessarily mean bad S/W engineering skills. But generally the research code is either made hastily during exploration or by people not having enough Software Engineering background. Many research scientist I've seen at large tech companies don't have good CS background and write code that's either unreadable, unscalable, unmaintainable, or buggy at times. They often have good ideas but don't do much software design. Having been both at infrastructure teams and research teams, there are certain individuals in research orgs joining straight from school who think they are responsible with coming with a new and sexy thing and other engineers are responsible to run in production. It's like computing eigenvectors in Matlab or Python over a toy dataset and thinking you've done the bulk of the work of computing PageRank in production and should receive all the credit for a full search engine. That attitude is a red flag to me.

high_derivative 7y ago
I don't see how what you are saying is contradicting me. The point is that the scientists generally don't view it as their job, whereas the ml engineers/research engineers typically have it as their job to do the software architecture/engineering side.

codingslave 7y ago

Yeah I'm waiting for the backlash when companies realize this

gdb 7y ago

(I work at OpenAI. Before that, I worked at Stripe. I've spent most of my software career thinking about how to build effective engineering cultures.) I think this code is actually well-written and maintainable. This is proven in practice because we've adopted it many places in OpenAI, and I've personally found it very easy to adapt to other use-cases (certainly much more so than the from-scratch Transformer implementations I've written!). As https://news.ycombinator.com/item?id=21456605 https://news.ycombinator.com/item?id=21456605 points out, the complexity of the code arises from the complexity of the underlying algorithm. Complexity due to software engineering concerns, like Tensorflow scopes, are elegantly handled. [edited for clarity:] Writing a Transformer in 174 lines of code requires a lot of deep thinking about the right underlying abstractions. > but essentially as soon as you enter the 'ml engineer/research engineer/research scientist' layer, it's doomed. We actually don't do this! Our only official technical title is "member of technical staff". (People sometimes choose to self-identify as an engineer or researcher, so you might see that on LinkedIn, but we don't have a distinction internally.) Everyone is responsible for their own code, and people care quite a bit about writing code that others can build on.

garmaine 7y ago
I’m very sorry to see someone who obviously cares so much to be defending this code. This does not follow best practices, and using complexity of the underlying algorithm is just an excuse. Complex code can be beautiful and well documented. Writing a complex method in 174 lines is not elegant nor beautiful. Writing a well documented file that can take an engineer in a different specialty and bring them up to speed in 1,000 lines is.
- gdb 7y ago
  We also have code like that. For example, that's the explicit goal of the Spinning Up repo: https://github.com/openai/spinningup/blob/master/spinup/algos/ddpg/ddpg.py https://github.com/openai/spinningup/blob/master/spinup/algo... In practice, it's much harder to use that code, and we tend not to consume code like that internally. There's a real tradeoff!