6 ms·

I really like Jupyter notebooks to build a simple concept and then move to .py files. But what I observe, especially at the entry level or junior level jobs in

I really like Jupyter notebooks to build a simple concept and then move to .py files. But what I observe, especially at the entry level or junior level jobs in data science is that people spend huge amount of its work on jupyter, which did not focus on how to plan flow properly. What I meant is that there is very short path from usefullness to overkill.

BiteCode_dev 5y ago
That's because they are not programmers. One should not expect them to be experts in 2 fields. They do their work with the tool provided, and if we want a better output, we need to provide better tooling or accept what comes out. I want my physicists to spend their mental effort on physics, not on software architecture.
- pavlovskyi 5y ago
  I agree that they are not programmers but in my opinion it cannot become an explanation to write a code which become non repeatable, especially if they have big impact on how the flow will look on production environment.
  - BiteCode_dev 5y ago
    Scientists are not meant to create code that ends up on production. Once they have a working concept, they should team up with a programmer to make it live. Again, if it's not possible, then you accept the imperfection of the result, or provide better tooling. There is no blame to put on them whatsoever.
    - throwaway210222 5y ago
      I disagree, many, many scientists hire a professional statistician to do the stats for their papers. Similarly, they should hire experienced, qualified software engineers to write/check the software in their papers. They don't because 'everyone can code - its just logic'.
      - BiteCode_dev 5y ago
        Err... That's kinda my point ?
        throwaway210222 5y ago
        Mea culpa - I interpreted your comment to mean: only once its ready to go into industrial use.
    - jstx1 5y ago
      > Scientists are not meant to create code that ends up on production. You might be mixing up the terms. I don't think that the point was about "scientists" in general, it was about "data scientists". The first is a common term used to describe someone who does science in some professional capacity. The second one is a very broad job title within software which very often includes writing code that ends up in production - at some data science roles that might even be your main responsiblity.
      - BiteCode_dev 5y ago
        Even among devs with very close speciality, the difference in productivity is immense. Take a iOS team, make them dev a MacOS desktop app, and see their output plummet in their of productivity or quality. They will end up doing a good job, but it will take between 6 months and a year to catch up with a specialized team. A data scientist is not even a somebody trained as a programmer. Their strong suit is data analysis, and it turns out one of the tool to manipulate data today are programming languages so their do it. But I as a Python trainer, I train data analyst regularly, and they don't have a clue about language ecosystems, how the OS work, data formats or reliable software architecture. They mainly want to output their graph, pdf report or other media to serve their conclusion. They may want to create some reusable algo, or machine learning model, but that's the limit most of them hit. If one take their code and put it in prod (which I know happens, don't get me wrong), that's not the data scientist fault. They are doing their job, in which programming is just one of the many means to an end, and is not their specialty.
        jstx1 5y ago
        > But I as a Python trainer, I train data analyst regularly, and they don't have a clue about language ecosystems, how the OS work, data formats or reliable software architecture. This is like teaching some JavaScript to complete beginners at a bootcamp and then declaring that front-end developers aren't real programmers because they know so little.
        BiteCode_dev 5y ago
- jstx1 5y ago
  > That's because they are not programmers. Beginners not being aware of some best practices doesn't automatically make them not-programmers. I work as a data scientist and I see it as part of software development. It's just a different domain - some people do front-end, some do mobile or embedded, I do data science.
  - BiteCode_dev 5y ago
    Data scientists are not programmers. They are data scientists. Just like I'm not a data scientists, I'm a programmer. Now, I can use pandas in a pinch and makes pretty graphs, but my statistical analysis will never be on part with yours. Just like a pianist hobbyist will have a hard time to rival somebody who does that 40 hours a week, although he may be able to play a few fantastic pieces. Hell, even a web dev programmers, if ask to code a GUI desktop app, is not going to do a good job. IT is becoming a very large field. And scientists are not even from this field.
    - jstx1 5y ago
      As a data scientist I spend more time writing software and building things than I spend doing statistical analysis (and I like it that way). That's why I consider it part of software development. Don't be tripped up by the word "scientist" in the job title (I wrote another comment about this in this thread). Also, it's a very broad field and it varies a lot from company to company - if you've had some interactions with data scientists, don't assume that what they do applies across the industry. I'm basing what I'm writing on having worked at/with companies of different sizes and in different industries.
      - BiteCode_dev 5y ago
        If you want to play with names and semantics, I'll leave you to it. That's not my point.
        jstx1 5y ago
        Well, my overall point was that you got the names and semantics wrong.
- bluescarni 5y ago
  As a scientist myself, I strongly disagree. If you spend a non-negligible amount of your time telling a computer what to do, you are, indeed, a programmer. And as such you should be expected to become a decently proficient programmer. Physicists are not mathematicians, and yet they are required to acquire a relatively high degree of proficiency in maths because maths is a fundamental tool in their job, and nobody would argue otherwise. The attitude of considering programming a mundane craft to be picked up as-you-go is the main reason why the scientific software landscape is such a shitshow. /rant
  - BiteCode_dev 5y ago
    > If you spend a non-negligible amount of your time telling a computer what to do, you are, indeed, a programmer. And as such you should be expected to become a decently proficient programmer. Just like if you are standing on your two legs most of the day and sprint once in a while, you can be considered a runner. Sure, you can play with semantics, but most people cannot run a marathon. > The attitude of considering programming a mundane craft to be picked up as-you-go is the main reason why the scientific software landscape is such a shitshow. Err... That's kinda my point? > And as such you should be expected to become a decently proficient programmer. It's very, very hard to be good in 2 different fields. Most people won't have the ability or the context to do so. Even if they did, the time and energy spent to do so would be taken from their main activity, which is why we employ them in the first place. It's not reasonable to ask a data scientist, geographer, biologist or physicist to follow up with the right practices to deploy the latest sci-stack on a linux server, understand the trade off between GIL locked python thread, asyncio and multiprocessing or spell out what WSGI stands for. Hell, I know a lot of professional programmers that don't know those things > Physicists are not mathematicians, and yet they are required to acquire a relatively high degree of proficiency in maths The quantity of information required to be learned is of one or two orders of magnitude, because the field of maths required to perform physics is quite stable, and well understood. IT is a very young field, in constant flux. The scientific stack is a moving target, not to even mention the web one. Nobody can expect them to understand python, numpy, pandas, then a web framework, then css, then js, and html, probably some frameworks for them, a builder or two, how to deploy all that stuff in dev, in prod and architectural concerns for linking all that stuff. That's crazy talk.
- misja111 5y ago
  Maybe they are not programmers but they surely work in a team or company where everybody is supposed to work as efficiently as possible together. Simply throwing your unstructured and unreadable code over the wall, with the excuse that it's not your key responsibility so somebody else should fix it, is just bad. It doesn't mean of course that everybody is supposed to be an expert programmer, but a minimum effort to help your colleague is surely not too much to be asked.
  - BiteCode_dev 5y ago
    It's not a matter of responsability or will, but of ability. Expecting a fish to fly is a very bad working relationship.
    - misja111 5y ago
      Structuring and documenting your code is not rocket science. It is not expecting a fish to fly, any scientist should easily be able to pick up this skill.
- civilized 5y ago
  Nah, data science is an inherently multidisciplinary topic, and most data scientists aren't domain experts in anything. Data science is mostly valuable to the extent that data scientists learn the full spectrum of skills needed to do their jobs well.
  - BiteCode_dev 5y ago
    Just like devoos, this is a fallacy. Eventually you can't be an expert in everything. Lost devops are very good at either dev or sysadmin, and not too bad at the other. Same for data scientists.
    - civilized 5y ago
      I would call "can only use Jupyter notebooks" very bad at programming, not merely "not too bad". It's well below expectation in my opinion. Whether you choose to call it "good" or "not too bad", there is a minimum bar of competence that data scientists need to meet in statistics/ML/AI, programming, and their domain of application. And the ability to move from Jupyter notebooks to Python modules/packages is a basic.
- analog31 5y ago
  Maybe there's a confusion here between programming and software development. One is a tool for the other, but can be used by itself too. I'm a physicist (not "data scientist" though I work with plenty of data), and I've been programming since 1981. Anything I do, I want to do well, especially if I do it regularly or it could cause problems if done badly. I've made an effort throughout my career to keep up with good programming practices. I do that out of a combination of pride, curiosity, and professional ethics. But I'm not a software developer, meaning that I don't create software for widespread or long term use by others. We have an entire department for that, and many of their techniques are quite specialized. Naturally it wouldn't surprise me if further improving my skills also moves me closer to being capable of software development, and I'm happy to learn and apply their techniques at a pace that works for me. I think that a scientist who is capable of learning to program should receive guidance on how to do it better, but perhaps in stages, such as: 1. Writing code that has a better chance of working, even as it gets bigger and more complex. 2. Working with others on projects that involve sharing code, meaning that it has to be readable and conform to agreed upon standards. 3. Creating code that can be confidently "shipped" for widespread or long term use.

BiteCode_dev 5y ago

That's because they are not programmers. One should not expect them to be experts in 2 fields. They do their work with the tool provided, and if we want a better output, we need to provide better tooling or accept what comes out. I want my physicists to spend their mental effort on physics, not on software architecture.

pavlovskyi 5y ago
I agree that they are not programmers but in my opinion it cannot become an explanation to write a code which become non repeatable, especially if they have big impact on how the flow will look on production environment.
- BiteCode_dev 5y ago
  Scientists are not meant to create code that ends up on production. Once they have a working concept, they should team up with a programmer to make it live. Again, if it's not possible, then you accept the imperfection of the result, or provide better tooling. There is no blame to put on them whatsoever.
  - throwaway210222 5y ago
    I disagree, many, many scientists hire a professional statistician to do the stats for their papers. Similarly, they should hire experienced, qualified software engineers to write/check the software in their papers. They don't because 'everyone can code - its just logic'.
    - BiteCode_dev 5y ago
      Err... That's kinda my point ?
      - throwaway210222 5y ago
        Mea culpa - I interpreted your comment to mean: only once its ready to go into industrial use.
  - jstx1 5y ago
    > Scientists are not meant to create code that ends up on production. You might be mixing up the terms. I don't think that the point was about "scientists" in general, it was about "data scientists". The first is a common term used to describe someone who does science in some professional capacity. The second one is a very broad job title within software which very often includes writing code that ends up in production - at some data science roles that might even be your main responsiblity.
    - BiteCode_dev 5y ago
      Even among devs with very close speciality, the difference in productivity is immense. Take a iOS team, make them dev a MacOS desktop app, and see their output plummet in their of productivity or quality. They will end up doing a good job, but it will take between 6 months and a year to catch up with a specialized team. A data scientist is not even a somebody trained as a programmer. Their strong suit is data analysis, and it turns out one of the tool to manipulate data today are programming languages so their do it. But I as a Python trainer, I train data analyst regularly, and they don't have a clue about language ecosystems, how the OS work, data formats or reliable software architecture. They mainly want to output their graph, pdf report or other media to serve their conclusion. They may want to create some reusable algo, or machine learning model, but that's the limit most of them hit. If one take their code and put it in prod (which I know happens, don't get me wrong), that's not the data scientist fault. They are doing their job, in which programming is just one of the many means to an end, and is not their specialty.
      - jstx1 5y ago
        > But I as a Python trainer, I train data analyst regularly, and they don't have a clue about language ecosystems, how the OS work, data formats or reliable software architecture. This is like teaching some JavaScript to complete beginners at a bootcamp and then declaring that front-end developers aren't real programmers because they know so little.
        BiteCode_dev 5y ago
jstx1 5y ago
> That's because they are not programmers. Beginners not being aware of some best practices doesn't automatically make them not-programmers. I work as a data scientist and I see it as part of software development. It's just a different domain - some people do front-end, some do mobile or embedded, I do data science.
- BiteCode_dev 5y ago
  Data scientists are not programmers. They are data scientists. Just like I'm not a data scientists, I'm a programmer. Now, I can use pandas in a pinch and makes pretty graphs, but my statistical analysis will never be on part with yours. Just like a pianist hobbyist will have a hard time to rival somebody who does that 40 hours a week, although he may be able to play a few fantastic pieces. Hell, even a web dev programmers, if ask to code a GUI desktop app, is not going to do a good job. IT is becoming a very large field. And scientists are not even from this field.
  - jstx1 5y ago
    As a data scientist I spend more time writing software and building things than I spend doing statistical analysis (and I like it that way). That's why I consider it part of software development. Don't be tripped up by the word "scientist" in the job title (I wrote another comment about this in this thread). Also, it's a very broad field and it varies a lot from company to company - if you've had some interactions with data scientists, don't assume that what they do applies across the industry. I'm basing what I'm writing on having worked at/with companies of different sizes and in different industries.
    - BiteCode_dev 5y ago
      If you want to play with names and semantics, I'll leave you to it. That's not my point.
      - jstx1 5y ago
        Well, my overall point was that you got the names and semantics wrong.
bluescarni 5y ago
As a scientist myself, I strongly disagree. If you spend a non-negligible amount of your time telling a computer what to do, you are, indeed, a programmer. And as such you should be expected to become a decently proficient programmer. Physicists are not mathematicians, and yet they are required to acquire a relatively high degree of proficiency in maths because maths is a fundamental tool in their job, and nobody would argue otherwise. The attitude of considering programming a mundane craft to be picked up as-you-go is the main reason why the scientific software landscape is such a shitshow. /rant
- BiteCode_dev 5y ago
  > If you spend a non-negligible amount of your time telling a computer what to do, you are, indeed, a programmer. And as such you should be expected to become a decently proficient programmer. Just like if you are standing on your two legs most of the day and sprint once in a while, you can be considered a runner. Sure, you can play with semantics, but most people cannot run a marathon. > The attitude of considering programming a mundane craft to be picked up as-you-go is the main reason why the scientific software landscape is such a shitshow. Err... That's kinda my point? > And as such you should be expected to become a decently proficient programmer. It's very, very hard to be good in 2 different fields. Most people won't have the ability or the context to do so. Even if they did, the time and energy spent to do so would be taken from their main activity, which is why we employ them in the first place. It's not reasonable to ask a data scientist, geographer, biologist or physicist to follow up with the right practices to deploy the latest sci-stack on a linux server, understand the trade off between GIL locked python thread, asyncio and multiprocessing or spell out what WSGI stands for. Hell, I know a lot of professional programmers that don't know those things > Physicists are not mathematicians, and yet they are required to acquire a relatively high degree of proficiency in maths The quantity of information required to be learned is of one or two orders of magnitude, because the field of maths required to perform physics is quite stable, and well understood. IT is a very young field, in constant flux. The scientific stack is a moving target, not to even mention the web one. Nobody can expect them to understand python, numpy, pandas, then a web framework, then css, then js, and html, probably some frameworks for them, a builder or two, how to deploy all that stuff in dev, in prod and architectural concerns for linking all that stuff. That's crazy talk.
misja111 5y ago
Maybe they are not programmers but they surely work in a team or company where everybody is supposed to work as efficiently as possible together. Simply throwing your unstructured and unreadable code over the wall, with the excuse that it's not your key responsibility so somebody else should fix it, is just bad. It doesn't mean of course that everybody is supposed to be an expert programmer, but a minimum effort to help your colleague is surely not too much to be asked.
- BiteCode_dev 5y ago
  It's not a matter of responsability or will, but of ability. Expecting a fish to fly is a very bad working relationship.
  - misja111 5y ago
    Structuring and documenting your code is not rocket science. It is not expecting a fish to fly, any scientist should easily be able to pick up this skill.
civilized 5y ago
Nah, data science is an inherently multidisciplinary topic, and most data scientists aren't domain experts in anything. Data science is mostly valuable to the extent that data scientists learn the full spectrum of skills needed to do their jobs well.
- BiteCode_dev 5y ago
  Just like devoos, this is a fallacy. Eventually you can't be an expert in everything. Lost devops are very good at either dev or sysadmin, and not too bad at the other. Same for data scientists.
  - civilized 5y ago
    I would call "can only use Jupyter notebooks" very bad at programming, not merely "not too bad". It's well below expectation in my opinion. Whether you choose to call it "good" or "not too bad", there is a minimum bar of competence that data scientists need to meet in statistics/ML/AI, programming, and their domain of application. And the ability to move from Jupyter notebooks to Python modules/packages is a basic.
analog31 5y ago
Maybe there's a confusion here between programming and software development. One is a tool for the other, but can be used by itself too. I'm a physicist (not "data scientist" though I work with plenty of data), and I've been programming since 1981. Anything I do, I want to do well, especially if I do it regularly or it could cause problems if done badly. I've made an effort throughout my career to keep up with good programming practices. I do that out of a combination of pride, curiosity, and professional ethics. But I'm not a software developer, meaning that I don't create software for widespread or long term use by others. We have an entire department for that, and many of their techniques are quite specialized. Naturally it wouldn't surprise me if further improving my skills also moves me closer to being capable of software development, and I'm happy to learn and apply their techniques at a pace that works for me. I think that a scientist who is capable of learning to program should receive guidance on how to do it better, but perhaps in stages, such as: 1. Writing code that has a better chance of working, even as it gets bigger and more complex. 2. Working with others on projects that involve sharing code, meaning that it has to be readable and conform to agreed upon standards. 3. Creating code that can be confidently "shipped" for widespread or long term use.