6 ms·

I've seen a lot of the software engineers on teams I've worked with complain about the way data science and machine learning professionals work with code, and T

I've seen a lot of the software engineers on teams I've worked with complain about the way data science and machine learning professionals work with code, and They. Just. Don't. Get. It. Stop trying to push the tools and processes that you developed for yourselves onto us! People I work with don't care that our notebook tools have rudimentary syntax highlighting or type-checking. We don't rely on our tools to make sure our code is written correctly. We just memorize the APIs and rely on experimentation to get things working. Please don't make us use git. Checking out and checking in files is too slow! Oftentimes, Slacking code to a colleague is the best way to collaborate, and the fastest. Don't make us use confusing and broken authentication schemes to access our data! I work best if I can just download live data to my machine and work locally. If it's too big or it's something like streaming, just give me an admin password that I can save in my notebook and use to hit some database clusters. Nothing wrong with that!

squarerootof-1 6y ago
Is this sarcasm?
- Areading314 6y ago
  I'm hoping this isn't even sarcasm, just GPT-3
- lumberjack 6y ago
  Yes.
- Jugurtha 6y ago
  It most definitely is, but there's a large part of truth in it. You can try and train people to write good code or follow best practices but from what I have seen, you're swimming against strong currents. What we have here is the: "I don't get it, why don't you read this 500 pages manual and type in those 10 commands to print your document" era of computing. People wondered why people had trouble "just" programming something. That is not the job to be done. There are huge frictions and broken interfaces between roles: developers, engineers, data scientists, domain experts with everyone expecting the other to "just" do X. If data scientists just wrote better code. If developers just were more familiar with machine learning. If ML engineers just could be more CICD aware. If domain experts just could get it right. This is not working.
  - barumi 6y ago
    > That is not the job to be done. There are huge frictions and broken interfaces between roles: developers, engineers, data scientists, domain experts with everyone expecting the other to "just" do X. The only friction I see with data analysis roles is that data analysts typically don't need (or even outright refuse) to do collaborative work where there are multiple people working on the same code and data. Once we take the collaborative aspect out, and we see data analysts who never had to check out someone else's work to apply minor changes or have someone else apply minor changes to their own code, then we get to the same naive criticism directed at basic organizational practices and tools as we see here.
    - Jugurtha 6y ago
      >The only friction I see with data analysis roles is that data analysts typically don't need (or even outright refuse) to do collaborative work where there are multiple people working on the same code and data. In your experience, what do you attribute this to? One way to look at it is that programmers tend to see notebooks as code, and think it makes sense to use the tooling used for code. Another way to look at it is as a document, and build around that. Would fiction writers benefit from learning Git ? Sure. Could it be more productive to reduce the activation energy to accomplish the job of tracking changes with another implementation ? I think yes. There are more writers who use Google Docs than there are who use Git. For example, we added collaborative editing on notebooks[0]. Multiple people can work simultaneously on the same notebook, see each other cursors and changes live. It's a much better experience. We also added multiple checkpoints behind the scenes[1]. I believe that as programmers, we have the opportunity to create better abstractions around workflows to simplify what we take for granted and do as second nature for others. One issue is cognitive load. People having to remember to do things. We strive to make it automatic. One example is for experiment tracking: sure, they 'just' have to track experiments and add the necessary code, but we do it automatically so it happens without them remembering. Models are detected are saved, parameters and metrics are logged, etc. This considerably reduces the barrier to entry as even a beginner who knows nothing about tracking gets their experiments tracked. This is what we're doing. Handle complexity on behalf of a user, and reduce the skills required to get things right. Same for deployment. Can they learn Docker? Sure. Why do they have to, though. So we automatically build model images for them, and they can deploy them themselves clicking a button. As I said, these are problems we've had in the past, for example a 'data scientist' having to sollicit the help of à colleague to deploy their model. This was a bottleneck for us. [0]: https://iko.ai/docs/notebook/#collaboration https://iko.ai/docs/notebook/#collaboration [1]: https://iko.ai/docs/notebook/#multiple-checkpoints https://iko.ai/docs/notebook/#multiple-checkpoints
- fouc 6y ago
  It's accurate though, so it's more ironic than sarcastic
  - newen 6y ago
    Lol, I’m a machine learning person and I was agreeing with it until more than halfway into it.
Jugurtha 6y ago
Yes, oh yes. From what I've seen, many products we've seen fall into one of several categories: - People thinking if Jupyter notebooks had just the right CSS and animations, it'd make machine learning unstuck. - People coming from web dev and thinking that ML folks are somehow unaware of Docker/CICD/Kubernetes/Git and if they could just be enlightened, they'd be much more productive. - People coming from a background of publishing blog posts/Youtube videos about machine learning, or who've worked with cute datasets who think that the actual workflow is someone giving you a CSV and a metric to optimize for à la Kaggle and building something around that workflow. This is for ML what Aikido is for fighting: it only works if your opponent knows the script and agrees to fall when you go Hayaaa! Disclosure: we're working on our machine learning platform[0] because we desperately need it. We do not come from a web dev or blogging background, and it shows. We're building it because we spent seven years actually delivering machine learning products to paying enterprise in the mid six figures contracts with actual money, trust, and careers on the line. We've done it OK enough to have companies trust us and become repeat customers, but, anyone we faced the problems that anyone who has worked on actual projects has faced. Just getting to formulate the problem is akin to extracting Ramses II's brain through his nostrils. Agreeing on the data to use and then getting the data to a usable form takes a long time. Defining the real world metric is not obvious [if only it just boiled down to F1/AUC/ROC/ETC]. Most things that matter are done by a team, and tools work great for one individual, but break for teams. Just getting results to the client makes the work an endangered specie in an "only one result out of 100 makes it to the client". It is unbelievable how much these problems impact the team. People can quit over this and we've suffered a lot. So we're building around our workflow that has generated actual revenues with very large organizations that were satisfied enough to come back for more projects. It is not a perfect workflow, but it is from the real world on real projects and real clients that paid real money to solve real problems. And we're building it for ourselves to use on actual projects. We aim for 10x improvement. [0]: https://iko.ai https://iko.ai
- PaulHoule 6y ago
  When it comes to the impact I believe it.
  - 6y ago
natalyarostova 6y ago
You should use git. — someone who shares your job.
- dognotdog 6y ago
  I do find this mildly amusing – and sad – as git is possibly one of the worst SCMs for non-programmers because its command set is so bottom-up, and therefore doesn't align with a naive mental model of how version control should work, while at the same time it exposes an interface that makes it all to easy to screw things up catastrophically. GUI tools only help with that to some extent, too, and only the right ones.
  - prepend 6y ago
    It’s funny to me because git seems like it would be the worst, but is the widest adopted scm by non-programmers from anything I’ve seen. It has more users, I think, than even the “visual” source control systems that are supposed to be more usable.
    - dognotdog 6y ago
      Oh, I've had the joy of working with Perforce, Jazz / Team Concert, etc. /s Git is widest adopted, period (EDIT: because of how it blossomed out of linux dev, essentially). I was able to stick to bzr or fossil for a lot of things until github and gitlab basically became where open source work (and even corporate transitioned to some extent!) needed to happen, and the friction of using anything else is just too great at this point in time. Which is a shame, really, as --fast-import/export is actually not a bad way to transition from one scm to another these days, and I'd personally welcome a little more active competition in the space to evolve beyond the current status quo.
  - icedistilled 6y ago
    I came from a chemE background and it wasn't even any computational heavy branch of it. Now I do data science in a different field. I feel like command line Git is fantastic and simple 95% of the time and much prefer it over GUIs. Is git really that hard? Although I really dislike GUIs in general so maybe that's it.
doliveira 6y ago
I didn't catch the sarcasm until the last paragraph. It was actually quite believable, specially coming from a Master's degree in Physics.
- wodenokoto 6y ago
  I didn’t catch it until this comment!
jrott 6y ago
This is painfully funny. It's interesting though I feel like it's harder to do data science right because the iterations are so much slower. If it takes all day to run a model it's hard to get in enough reps to really understand what good practices are for the domain.
- xapata 6y ago
  If it takes all day to run every model, you're doing it wrong.
FridgeSeal 6y ago
Sarcasm aside, I've worked with people who fulfil at least any 2 of these behaviours, often-times most of them. Choice excerpts: * co-ordinating code between team members by having everyone periodically copy their code to one team members personal google drive, where one of them would then spend a day fixing merge conflicts by hand. When asked why they were doing it like that, would they like any help, git can probably solve these problems, they asserted that this worked fine with them. * Spinning up machines in EC2, then keeping track of keys (new one for every machine obviously) by storing the Pem files in an S3 bucket when they remembered. This didn't happen, so instead we had a dozen machines nobody could get into anymore after they left... * Calling Python code in another file, by running os.spawn, calling a python interpreter with the given file, passing in parameters a cmd line arguments, and getting the result by waiting for a specifically named file to be written to disk. When asked "why are you doing this?" I was told "because it was easier". * All the python they wrote eschewed functions were possible, with everything at root-indentation & no import guards (so it couldn't be imported safely). Functions, when they were used relied heavily on global variables and other objects/imports/etc being present implicitly. There were also logging and json serialisation functions that were hand written. When I asked why this teammate did this they replied "it only took me a couple of minutes and it works? What's wrong?" Unsurprisingly, it didn't serialised datetime objects correctly... * Re-inventing the functionality offered by Pythons multiprocessing/ Joblib libraries-but worse- by shelling out (again) and manually calling a shell script to detect the number of CPU's and then using x-args to spawn the python processes. "Well we've written it now and it works but maybe next time". Unsurprisingly, these processes would fail, silently and lose data. * Using eval() in production code because they "couldn't figure out a better way of getting the value of the attribute" despite being told that getattr is safer and faster. * Maintaining a spark cluster full of Jupiter notebooks unhelpfully named things like STEP_301.ipynb, STEP_405.ipynb and not anything that told you _what_ they did. Figuring out which notebooks called which other notebooks and where any bit of code came from was, unsurprisingly literally kafkaesque. These eventually got checked into git (root credentials, embedded images and all), in a single commit with the message "add to repo". * Instead of reading the documentation, or using something like SqlAlchemy, code samples of how to connect to various different types of database (mysql, Postgres etc) were stored in a random google doc (once again, one of their personal accounts) and access was passed around.

squarerootof-1 6y ago

Is this sarcasm?

Areading314 6y ago
I'm hoping this isn't even sarcasm, just GPT-3
lumberjack 6y ago
Yes.
Jugurtha 6y ago
It most definitely is, but there's a large part of truth in it. You can try and train people to write good code or follow best practices but from what I have seen, you're swimming against strong currents. What we have here is the: "I don't get it, why don't you read this 500 pages manual and type in those 10 commands to print your document" era of computing. People wondered why people had trouble "just" programming something. That is not the job to be done. There are huge frictions and broken interfaces between roles: developers, engineers, data scientists, domain experts with everyone expecting the other to "just" do X. If data scientists just wrote better code. If developers just were more familiar with machine learning. If ML engineers just could be more CICD aware. If domain experts just could get it right. This is not working.
- barumi 6y ago
  > That is not the job to be done. There are huge frictions and broken interfaces between roles: developers, engineers, data scientists, domain experts with everyone expecting the other to "just" do X. The only friction I see with data analysis roles is that data analysts typically don't need (or even outright refuse) to do collaborative work where there are multiple people working on the same code and data. Once we take the collaborative aspect out, and we see data analysts who never had to check out someone else's work to apply minor changes or have someone else apply minor changes to their own code, then we get to the same naive criticism directed at basic organizational practices and tools as we see here.
  - Jugurtha 6y ago
    >The only friction I see with data analysis roles is that data analysts typically don't need (or even outright refuse) to do collaborative work where there are multiple people working on the same code and data. In your experience, what do you attribute this to? One way to look at it is that programmers tend to see notebooks as code, and think it makes sense to use the tooling used for code. Another way to look at it is as a document, and build around that. Would fiction writers benefit from learning Git ? Sure. Could it be more productive to reduce the activation energy to accomplish the job of tracking changes with another implementation ? I think yes. There are more writers who use Google Docs than there are who use Git. For example, we added collaborative editing on notebooks[0]. Multiple people can work simultaneously on the same notebook, see each other cursors and changes live. It's a much better experience. We also added multiple checkpoints behind the scenes[1]. I believe that as programmers, we have the opportunity to create better abstractions around workflows to simplify what we take for granted and do as second nature for others. One issue is cognitive load. People having to remember to do things. We strive to make it automatic. One example is for experiment tracking: sure, they 'just' have to track experiments and add the necessary code, but we do it automatically so it happens without them remembering. Models are detected are saved, parameters and metrics are logged, etc. This considerably reduces the barrier to entry as even a beginner who knows nothing about tracking gets their experiments tracked. This is what we're doing. Handle complexity on behalf of a user, and reduce the skills required to get things right. Same for deployment. Can they learn Docker? Sure. Why do they have to, though. So we automatically build model images for them, and they can deploy them themselves clicking a button. As I said, these are problems we've had in the past, for example a 'data scientist' having to sollicit the help of à colleague to deploy their model. This was a bottleneck for us. [0]: https://iko.ai/docs/notebook/#collaboration https://iko.ai/docs/notebook/#collaboration [1]: https://iko.ai/docs/notebook/#multiple-checkpoints https://iko.ai/docs/notebook/#multiple-checkpoints
fouc 6y ago
It's accurate though, so it's more ironic than sarcastic
- newen 6y ago
  Lol, I’m a machine learning person and I was agreeing with it until more than halfway into it.

Jugurtha 6y ago

Yes, oh yes. From what I've seen, many products we've seen fall into one of several categories: - People thinking if Jupyter notebooks had just the right CSS and animations, it'd make machine learning unstuck. - People coming from web dev and thinking that ML folks are somehow unaware of Docker/CICD/Kubernetes/Git and if they could just be enlightened, they'd be much more productive. - People coming from a background of publishing blog posts/Youtube videos about machine learning, or who've worked with cute datasets who think that the actual workflow is someone giving you a CSV and a metric to optimize for à la Kaggle and building something around that workflow. This is for ML what Aikido is for fighting: it only works if your opponent knows the script and agrees to fall when you go Hayaaa! Disclosure: we're working on our machine learning platform[0] because we desperately need it. We do not come from a web dev or blogging background, and it shows. We're building it because we spent seven years actually delivering machine learning products to paying enterprise in the mid six figures contracts with actual money, trust, and careers on the line. We've done it OK enough to have companies trust us and become repeat customers, but, anyone we faced the problems that anyone who has worked on actual projects has faced. Just getting to formulate the problem is akin to extracting Ramses II's brain through his nostrils. Agreeing on the data to use and then getting the data to a usable form takes a long time. Defining the real world metric is not obvious [if only it just boiled down to F1/AUC/ROC/ETC]. Most things that matter are done by a team, and tools work great for one individual, but break for teams. Just getting results to the client makes the work an endangered specie in an "only one result out of 100 makes it to the client". It is unbelievable how much these problems impact the team. People can quit over this and we've suffered a lot. So we're building around our workflow that has generated actual revenues with very large organizations that were satisfied enough to come back for more projects. It is not a perfect workflow, but it is from the real world on real projects and real clients that paid real money to solve real problems. And we're building it for ourselves to use on actual projects. We aim for 10x improvement. [0]: https://iko.ai https://iko.ai

PaulHoule 6y ago
When it comes to the impact I believe it.
- 6y ago

natalyarostova 6y ago

You should use git. — someone who shares your job.

dognotdog 6y ago
I do find this mildly amusing – and sad – as git is possibly one of the worst SCMs for non-programmers because its command set is so bottom-up, and therefore doesn't align with a naive mental model of how version control should work, while at the same time it exposes an interface that makes it all to easy to screw things up catastrophically. GUI tools only help with that to some extent, too, and only the right ones.
- prepend 6y ago
  It’s funny to me because git seems like it would be the worst, but is the widest adopted scm by non-programmers from anything I’ve seen. It has more users, I think, than even the “visual” source control systems that are supposed to be more usable.
  - dognotdog 6y ago
    Oh, I've had the joy of working with Perforce, Jazz / Team Concert, etc. /s Git is widest adopted, period (EDIT: because of how it blossomed out of linux dev, essentially). I was able to stick to bzr or fossil for a lot of things until github and gitlab basically became where open source work (and even corporate transitioned to some extent!) needed to happen, and the friction of using anything else is just too great at this point in time. Which is a shame, really, as --fast-import/export is actually not a bad way to transition from one scm to another these days, and I'd personally welcome a little more active competition in the space to evolve beyond the current status quo.
- icedistilled 6y ago
  I came from a chemE background and it wasn't even any computational heavy branch of it. Now I do data science in a different field. I feel like command line Git is fantastic and simple 95% of the time and much prefer it over GUIs. Is git really that hard? Although I really dislike GUIs in general so maybe that's it.

doliveira 6y ago

I didn't catch the sarcasm until the last paragraph. It was actually quite believable, specially coming from a Master's degree in Physics.

wodenokoto 6y ago
I didn’t catch it until this comment!

jrott 6y ago

This is painfully funny. It's interesting though I feel like it's harder to do data science right because the iterations are so much slower. If it takes all day to run a model it's hard to get in enough reps to really understand what good practices are for the domain.

xapata 6y ago
If it takes all day to run every model, you're doing it wrong.

FridgeSeal 6y ago

Sarcasm aside, I've worked with people who fulfil at least any 2 of these behaviours, often-times most of them. Choice excerpts: * co-ordinating code between team members by having everyone periodically copy their code to one team members personal google drive, where one of them would then spend a day fixing merge conflicts by hand. When asked why they were doing it like that, would they like any help, git can probably solve these problems, they asserted that this worked fine with them. * Spinning up machines in EC2, then keeping track of keys (new one for every machine obviously) by storing the Pem files in an S3 bucket when they remembered. This didn't happen, so instead we had a dozen machines nobody could get into anymore after they left... * Calling Python code in another file, by running os.spawn, calling a python interpreter with the given file, passing in parameters a cmd line arguments, and getting the result by waiting for a specifically named file to be written to disk. When asked "why are you doing this?" I was told "because it was easier". * All the python they wrote eschewed functions were possible, with everything at root-indentation & no import guards (so it couldn't be imported safely). Functions, when they were used relied heavily on global variables and other objects/imports/etc being present implicitly. There were also logging and json serialisation functions that were hand written. When I asked why this teammate did this they replied "it only took me a couple of minutes and it works? What's wrong?" Unsurprisingly, it didn't serialised datetime objects correctly... * Re-inventing the functionality offered by Pythons multiprocessing/ Joblib libraries-but worse- by shelling out (again) and manually calling a shell script to detect the number of CPU's and then using x-args to spawn the python processes. "Well we've written it now and it works but maybe next time". Unsurprisingly, these processes would fail, silently and lose data. * Using eval() in production code because they "couldn't figure out a better way of getting the value of the attribute" despite being told that getattr is safer and faster. * Maintaining a spark cluster full of Jupiter notebooks unhelpfully named things like STEP_301.ipynb, STEP_405.ipynb and not anything that told you _what_ they did. Figuring out which notebooks called which other notebooks and where any bit of code came from was, unsurprisingly literally kafkaesque. These eventually got checked into git (root credentials, embedded images and all), in a single commit with the message "add to repo". * Instead of reading the documentation, or using something like SqlAlchemy, code samples of how to connect to various different types of database (mysql, Postgres etc) were stored in a random google doc (once again, one of their personal accounts) and access was passed around.