5 ms·

My last job was at an ML company. Most ML people there cannot build large robust systems and some struggled with the non-algorithmic bits of software. I am sur

My last job was at an ML company. Most ML people there cannot build large robust systems and some struggled with the non-algorithmic bits of software. I am sure that some can out there in the world, but for the most part our ML people were very good at creating models and not very good at the development part, especially as the program grew (part of the motivation to hire devs like me in the first place). Python gets rid of as much of the developmental complexity as possible. No types, no memory management, libraries for everything, No need to create a class to run "hello world." Pip makes it trivial to import things. Use PyCharm and you just need to click the run button, with no complicated JRE and JDK setup. It is the fastest way to start writing models.

z5h 5y ago
> Most ML people there cannot build large robust systems and some struggled with the non-algorithmic bits of software. I have experience working alongside ML people and interviewing them, and have to agree (anecdotally) that this is often the case. Not only that, but they often do not make it a priority to get good at these things.
gonab 5y ago
I don't think this is a fair assessment of "most ML people" Some of the biggest distributed systems built today are used for statistical inference or scientific computation Most "ML people" I know are highly versatile in software, networks and deep hardware knowledge, i.e., essentially they have a very good understanding of what a computer is and what is capable from Its very naive to think that you can assemble machine learning systems without having a solid understanding of computers and statistics You know who also likes python a lot? Hackers. I wonder why
- gonab 5y ago
  Python is used to make fast and dirty experiences. You haven't figured out the answer yet, so no point on building dedicated optimized code which might be useless in end Almost always, ML production models end up being a binary files of matricial weights. This file can be loaded in wtv language or device you decide to use
- dustintrex 5y ago
  Perhaps we're talking about different sets of people? You seem to be describing the people who build ML systems, while the previous poster was talking about the people who use them. Your average data scientist most definitely does not have (or, really, even need to have) deep hardware knowledge or any understanding of networking.
  - gonab 5y ago
    I personally consider the term "data scientist" a very successful creation by a clever marketer It's sexy and most of the times ...
- perth 5y ago
  The most insane thing about Python is how you can override single methods in classes and use the class like normal. One time I was working on getting FIFO working on Windows, and none of the Python built-ins were set up to handle any random process writing to a named pipe that wasn't within the same Python instance. So what I did was I took the closest implementation Python offered, which was in the multiprocessing module [1], and overrode a single one of the methods to do what I wanted it to do. The module still handled all of the cleanup so I was confident there weren't any memory leak issues, and I was able to make a simple change to the flags it was passing Windows to allow for the functionality I wanted. A language that has a hackable standard library itself is insane and I don't think I've seen it on any other language. In addition, I've found the C/C++ bindings for Python wonderful and intuitive to work with. The setup takes no effort at all and it "just works", batteries included, via ctypes. https://github.com/python/cpython/blob/main/Lib/multiprocessing/connection.py https://github.com/python/cpython/blob/main/Lib/multiprocess...
  - dragonwriter 5y ago
    > The most insane thing about Python is how you can override single methods in classes and use the class like normal. Isn't that true of basically every language supporting class-based OOP and inheritance?
    - danwills 5y ago
      It's called 'monkey patching' and python does make it particularly easy, simply: class.methodName = newMethod .. kinda thing, future callers now get your method instead of the original. This does seem a fair bit easier than other languages make it to do?
      - perth 5y ago
        You see exactly what I mean!
      - alanfranz 5y ago
        class.methodName = newMethod > .. kinda thing, future callers now get your method instead of the original. Which is as powerful as it is a problem, since doing such kind of monkeypatching will change the behaviour all other instances, including already-created ones, that know nothing about your trick. Any part of the program can modify any other part of the program in a significant way, making local reasoning and debugging very hard. So, great for quick-and-dirty single-file scripts/ipython notebooks. Terrible for large systems. That's the very issue with Python. The way it doesn't enforce sane, clean programming behaviour makes it easy for a beginner/non-programmer to work with it. But a large system with a lot of external libraries is very hard to maintain. Source: Python user since ~2004
- deleted 5y ago
  [deleted]
- exdsq 5y ago
  Ethical hackers have written some of the worst code I’ve ever seen. They have to know a ton about the security of frameworks, networking, etc… it’s a very complex role for sure, but they are not shining beacons of software engineering quality
  - gonab 5y ago
    I agree. Their objective is not to build beautiful code but to provide a working prototype I'm sure every single one of them is capable of writing world class code if they feel like it Their exploits are world class and their focus is to exploit I have seen stuff in JavaScript exploitation that I can't even scratch the surface. I feel like I have been playing piano for 15 years and I can't even understand if that a music that is playing
- Agingcoder 5y ago
  Most ml people (aka data scientists) I've met have little understanding of what a computer is, and it's fine. They understand stats. The tiny subset of people who build ml systems (say tensorflow core devs, write actual distributed systems, etc) are actually hpc specialists, and have all the qualities you describe. Of course, you may work somewhere where you're lucky enough to have everyone be good at everything!
mynameisash 5y ago
Your description is pretty much spot-on. I'm a data engineer at FAANG. I love the data scientists I work with. They are, generally speaking, crazy smart and simultaneously humble about their (in)ability to write code - they're highly specialized in ML, not so much SWE. I therefore have generally good job security working in operationalizing and optimizing their code. (Recently tweaked a script a scientist wrote and dropped runtime from 5hrs to 5min.) If I never saw another line of Python again, I'd probably be quite happy. But I think they - and even some ML engineers I work with - love them some Python precisely because they can mash the keyboard a bit and take the shortcut to the finish line. And I don't begrudge them this at all - good on them that they can get their job done quickly! But it's a pain to make things stable and efficient.
a-nikolaev 5y ago
Python is a kind of a new Fortran for this generation of scientists. (And Python makes things even easier, ofc, compared to Fortran.)
billfruit 5y ago
JRE/JDK is a rather simple thing to set up. Compared to the complexity in most other stacks Java is rather straightforward to setup and start building things with.

z5h 5y ago

> Most ML people there cannot build large robust systems and some struggled with the non-algorithmic bits of software. I have experience working alongside ML people and interviewing them, and have to agree (anecdotally) that this is often the case. Not only that, but they often do not make it a priority to get good at these things.

gonab 5y ago

I don't think this is a fair assessment of "most ML people" Some of the biggest distributed systems built today are used for statistical inference or scientific computation Most "ML people" I know are highly versatile in software, networks and deep hardware knowledge, i.e., essentially they have a very good understanding of what a computer is and what is capable from Its very naive to think that you can assemble machine learning systems without having a solid understanding of computers and statistics You know who also likes python a lot? Hackers. I wonder why

gonab 5y ago
Python is used to make fast and dirty experiences. You haven't figured out the answer yet, so no point on building dedicated optimized code which might be useless in end Almost always, ML production models end up being a binary files of matricial weights. This file can be loaded in wtv language or device you decide to use
dustintrex 5y ago
Perhaps we're talking about different sets of people? You seem to be describing the people who build ML systems, while the previous poster was talking about the people who use them. Your average data scientist most definitely does not have (or, really, even need to have) deep hardware knowledge or any understanding of networking.
- gonab 5y ago
  I personally consider the term "data scientist" a very successful creation by a clever marketer It's sexy and most of the times ...
perth 5y ago
The most insane thing about Python is how you can override single methods in classes and use the class like normal. One time I was working on getting FIFO working on Windows, and none of the Python built-ins were set up to handle any random process writing to a named pipe that wasn't within the same Python instance. So what I did was I took the closest implementation Python offered, which was in the multiprocessing module [1], and overrode a single one of the methods to do what I wanted it to do. The module still handled all of the cleanup so I was confident there weren't any memory leak issues, and I was able to make a simple change to the flags it was passing Windows to allow for the functionality I wanted. A language that has a hackable standard library itself is insane and I don't think I've seen it on any other language. In addition, I've found the C/C++ bindings for Python wonderful and intuitive to work with. The setup takes no effort at all and it "just works", batteries included, via ctypes. https://github.com/python/cpython/blob/main/Lib/multiprocessing/connection.py https://github.com/python/cpython/blob/main/Lib/multiprocess...
- dragonwriter 5y ago
  > The most insane thing about Python is how you can override single methods in classes and use the class like normal. Isn't that true of basically every language supporting class-based OOP and inheritance?
  - danwills 5y ago
    It's called 'monkey patching' and python does make it particularly easy, simply: class.methodName = newMethod .. kinda thing, future callers now get your method instead of the original. This does seem a fair bit easier than other languages make it to do?
    - perth 5y ago
      You see exactly what I mean!
    - alanfranz 5y ago
      class.methodName = newMethod > .. kinda thing, future callers now get your method instead of the original. Which is as powerful as it is a problem, since doing such kind of monkeypatching will change the behaviour all other instances, including already-created ones, that know nothing about your trick. Any part of the program can modify any other part of the program in a significant way, making local reasoning and debugging very hard. So, great for quick-and-dirty single-file scripts/ipython notebooks. Terrible for large systems. That's the very issue with Python. The way it doesn't enforce sane, clean programming behaviour makes it easy for a beginner/non-programmer to work with it. But a large system with a lot of external libraries is very hard to maintain. Source: Python user since ~2004
deleted 5y ago
[deleted]
exdsq 5y ago
Ethical hackers have written some of the worst code I’ve ever seen. They have to know a ton about the security of frameworks, networking, etc… it’s a very complex role for sure, but they are not shining beacons of software engineering quality
- gonab 5y ago
  I agree. Their objective is not to build beautiful code but to provide a working prototype I'm sure every single one of them is capable of writing world class code if they feel like it Their exploits are world class and their focus is to exploit I have seen stuff in JavaScript exploitation that I can't even scratch the surface. I feel like I have been playing piano for 15 years and I can't even understand if that a music that is playing
Agingcoder 5y ago
Most ml people (aka data scientists) I've met have little understanding of what a computer is, and it's fine. They understand stats. The tiny subset of people who build ml systems (say tensorflow core devs, write actual distributed systems, etc) are actually hpc specialists, and have all the qualities you describe. Of course, you may work somewhere where you're lucky enough to have everyone be good at everything!

mynameisash 5y ago

Your description is pretty much spot-on. I'm a data engineer at FAANG. I love the data scientists I work with. They are, generally speaking, crazy smart and simultaneously humble about their (in)ability to write code - they're highly specialized in ML, not so much SWE. I therefore have generally good job security working in operationalizing and optimizing their code. (Recently tweaked a script a scientist wrote and dropped runtime from 5hrs to 5min.) If I never saw another line of Python again, I'd probably be quite happy. But I think they - and even some ML engineers I work with - love them some Python precisely because they can mash the keyboard a bit and take the shortcut to the finish line. And I don't begrudge them this at all - good on them that they can get their job done quickly! But it's a pain to make things stable and efficient.

a-nikolaev 5y ago

Python is a kind of a new Fortran for this generation of scientists. (And Python makes things even easier, ofc, compared to Fortran.)

billfruit 5y ago

JRE/JDK is a rather simple thing to set up. Compared to the complexity in most other stacks Java is rather straightforward to setup and start building things with.