3 ms·

You misunderstood. The guy above correctly said that Jupyter is nice for ad-hoc analysis kind of work. The problem is that when you've reached terabytes of dat

You misunderstood. The guy above correctly said that Jupyter is nice for ad-hoc analysis kind of work. The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore. Too often the math guys try to avoid responsibility by claiming they're doing "ad-hoc" work when they're clearly not anymore. It's convenient, yes, but leads to a bad place eventually.

musingsole 5y ago
I understood fine. The problem is trying to formalize "ad hoc" versus "production development" practices as if they're meaningful. There isn't some golden truth of software development that analysts are too lazy to learn and implement. There's the problem and then there's solutions. Complaining that Jupyter-based development doesn't adequately accomdate version control or some other whistle commonly used in software development is some peak developer entitlement.
- otabdeveloper4 5y ago
  Your boss will eventually ask to make your "ad-hoc" stuff "production". At which point you'll dump the hot mess in somebody else's lap, and the whole thing will be rewritten from scratch. If that's your thing, then go for it.
  - throwaway894345 5y ago
    That’s fine. Why waste time productionizing something that may or may not ever go to production? We do this all the time. Do you think Tesla built a whole fully-automated factory to build its first proof-of-concept car?
throwaway894345 5y ago
> The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore. Not in any meaningful way, the code to explore a 50MB data set on a little machine looks the same as the code to explore a 50GB data set on a big machine, so of course one doesn’t need version control more than the other. > Too often the math guys try to avoid responsibility by claiming they're doing "ad-hoc" work when they're clearly not anymore. It's convenient, yes, but leads to a bad place eventually. This is a moralistic argument. The economic argument is that the primary artifact of research is insight, not code, so getting to that novel insight as fast as possible is paramount. Putting version control, tests, or other ceremony into the exploration loop is a pointless cost. Productionizing that insight can happen later in a more traditional software development workflow. It’s similar to how we write proofs of concept without the intensive testing effort that we would go into if we were writing production software.
kbelder 5y ago
>The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore. Oh, yes you are. "Ad-hoc" doesn't imply small. It implies one-off exploration and creation. You have no less of a need to do that with terabytes of data than you do with megabytes.

musingsole 5y ago

I understood fine. The problem is trying to formalize "ad hoc" versus "production development" practices as if they're meaningful. There isn't some golden truth of software development that analysts are too lazy to learn and implement. There's the problem and then there's solutions. Complaining that Jupyter-based development doesn't adequately accomdate version control or some other whistle commonly used in software development is some peak developer entitlement.

otabdeveloper4 5y ago
Your boss will eventually ask to make your "ad-hoc" stuff "production". At which point you'll dump the hot mess in somebody else's lap, and the whole thing will be rewritten from scratch. If that's your thing, then go for it.
- throwaway894345 5y ago
  That’s fine. Why waste time productionizing something that may or may not ever go to production? We do this all the time. Do you think Tesla built a whole fully-automated factory to build its first proof-of-concept car?

throwaway894345 5y ago

> The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore. Not in any meaningful way, the code to explore a 50MB data set on a little machine looks the same as the code to explore a 50GB data set on a big machine, so of course one doesn’t need version control more than the other. > Too often the math guys try to avoid responsibility by claiming they're doing "ad-hoc" work when they're clearly not anymore. It's convenient, yes, but leads to a bad place eventually. This is a moralistic argument. The economic argument is that the primary artifact of research is insight, not code, so getting to that novel insight as fast as possible is paramount. Putting version control, tests, or other ceremony into the exploration loop is a pointless cost. Productionizing that insight can happen later in a more traditional software development workflow. It’s similar to how we write proofs of concept without the intensive testing effort that we would go into if we were writing production software.

kbelder 5y ago

>The problem is that when you've reached terabytes of data and tens of cores you're not "ad-hoc" anymore. Oh, yes you are. "Ad-hoc" doesn't imply small. It implies one-off exploration and creation. You have no less of a need to do that with terabytes of data than you do with megabytes.