10 ms·

> Jupyter exists mostly because analytics/math guys are too lazy to spend a day learning software development practices Bahahahaha, yup those dang lazy mathema

> Jupyter exists mostly because analytics/math guys are too lazy to spend a day learning software development practices Bahahahaha, yup those dang lazy mathematicians just shooting themselves in the foot and forcing us to deal with it! /s Your preferences for software development are irrelevant. The value is in delivering the math to the end user. Using GCP, kubernetes or JavaScript to do that is an implementation detail. Sorry to tell you this, but you're a servant to those dang lazy analysts and without their insights, you're worthless.

otabdeveloper4 5y ago
You misunderstood. The guy above correctly said that Jupyter is nice for ad-hoc analysis kind of work. The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore. Too often the math guys try to avoid responsibility by claiming they're doing "ad-hoc" work when they're clearly not anymore. It's convenient, yes, but leads to a bad place eventually.
- musingsole 5y ago
  I understood fine. The problem is trying to formalize "ad hoc" versus "production development" practices as if they're meaningful. There isn't some golden truth of software development that analysts are too lazy to learn and implement. There's the problem and then there's solutions. Complaining that Jupyter-based development doesn't adequately accomdate version control or some other whistle commonly used in software development is some peak developer entitlement.
  - otabdeveloper4 5y ago
    Your boss will eventually ask to make your "ad-hoc" stuff "production". At which point you'll dump the hot mess in somebody else's lap, and the whole thing will be rewritten from scratch. If that's your thing, then go for it.
    - throwaway894345 5y ago
      That’s fine. Why waste time productionizing something that may or may not ever go to production? We do this all the time. Do you think Tesla built a whole fully-automated factory to build its first proof-of-concept car?
- throwaway894345 5y ago
  > The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore. Not in any meaningful way, the code to explore a 50MB data set on a little machine looks the same as the code to explore a 50GB data set on a big machine, so of course one doesn’t need version control more than the other. > Too often the math guys try to avoid responsibility by claiming they're doing "ad-hoc" work when they're clearly not anymore. It's convenient, yes, but leads to a bad place eventually. This is a moralistic argument. The economic argument is that the primary artifact of research is insight, not code, so getting to that novel insight as fast as possible is paramount. Putting version control, tests, or other ceremony into the exploration loop is a pointless cost. Productionizing that insight can happen later in a more traditional software development workflow. It’s similar to how we write proofs of concept without the intensive testing effort that we would go into if we were writing production software.
- kbelder 5y ago
  >The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore. Oh, yes you are. "Ad-hoc" doesn't imply small. It implies one-off exploration and creation. You have no less of a need to do that with terabytes of data than you do with megabytes.

otabdeveloper4 5y ago

You misunderstood. The guy above correctly said that Jupyter is nice for ad-hoc analysis kind of work. The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore. Too often the math guys try to avoid responsibility by claiming they're doing "ad-hoc" work when they're clearly not anymore. It's convenient, yes, but leads to a bad place eventually.

musingsole 5y ago
I understood fine. The problem is trying to formalize "ad hoc" versus "production development" practices as if they're meaningful. There isn't some golden truth of software development that analysts are too lazy to learn and implement. There's the problem and then there's solutions. Complaining that Jupyter-based development doesn't adequately accomdate version control or some other whistle commonly used in software development is some peak developer entitlement.
- otabdeveloper4 5y ago
  Your boss will eventually ask to make your "ad-hoc" stuff "production". At which point you'll dump the hot mess in somebody else's lap, and the whole thing will be rewritten from scratch. If that's your thing, then go for it.
  - throwaway894345 5y ago
    That’s fine. Why waste time productionizing something that may or may not ever go to production? We do this all the time. Do you think Tesla built a whole fully-automated factory to build its first proof-of-concept car?
throwaway894345 5y ago
> The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore. Not in any meaningful way, the code to explore a 50MB data set on a little machine looks the same as the code to explore a 50GB data set on a big machine, so of course one doesn’t need version control more than the other. > Too often the math guys try to avoid responsibility by claiming they're doing "ad-hoc" work when they're clearly not anymore. It's convenient, yes, but leads to a bad place eventually. This is a moralistic argument. The economic argument is that the primary artifact of research is insight, not code, so getting to that novel insight as fast as possible is paramount. Putting version control, tests, or other ceremony into the exploration loop is a pointless cost. Productionizing that insight can happen later in a more traditional software development workflow. It’s similar to how we write proofs of concept without the intensive testing effort that we would go into if we were writing production software.
kbelder 5y ago
>The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore. Oh, yes you are. "Ad-hoc" doesn't imply small. It implies one-off exploration and creation. You have no less of a need to do that with terabytes of data than you do with megabytes.