5 ms·
> I said this already 10 years ago: We don't need more cloud computing but need to empower users end devices again. Jupyter is typically operated on powerful no
by throwaway894345 5y ago
> I said this already 10 years ago: We don't need more cloud computing but need to empower users end devices again. Jupyter is typically operated on powerful notebooks and not on mobile devices.
If you’re working with data of any significant size at all it then it doesn’t matter how fast your user device is—it’s so much cheaper (time and network egress costs) to send the computations from a user device to the cloud than to pull tens-thousands of GB of data to your local machine. Moreover, I don’t know of many local machines with tens of CPU cores, hundreds or thousands of GB of RAM, or tens-hundreds of TB of SSD for handling that computation quickly.
User devices are great for very small data, but I don’t see the point for larger datasets.
- OldTimeCoffee 5y agoMost users don't have 'tens-thousands' of GB of data as part of their use case. You're describing a business case, not an end user consumer case.
- throwaway894345 5y agoI don’t believe that the majority of Jupyterhub users are “consumers” rather than professionals, but more importantly, that doesn’t change the fact that the professional use case exist and isn’t amenable to the fat client approach. As an anecdote, I work on a Jupyterhub managed service offering with customers in both the private and public sectors and our data sizes are pretty much all in this range.
- deleted 5y ago[deleted]
- mrtesthah 5y ago> I work on a Jupyterhub managed service offering ... That sounds like selection bias.
- throwaway894345 5y agoIt might be. Like I said: anecdote.
- mrtesthah 5y agoIsn't it, though? The people most likely to need a cloud-managed service like that probably have too much data to crunch on a laptop, as you described.
- throwaway894345 5y agoAh, to be clear I’m not saying our users are a representative sample of Jupyterhub users. I am saying that there are a lot of people who use Jupyterhub for large datasets—it’s certainly not uncommon.
- spicyramen 5y agoMost of the users will not run Jupyter Notebooks locally, they expect a remote kernel in the cloud with GPU/multicore machines, only the web app will be running in browser (think of big query). Of course small users with you datasets will run locally.
- candiodari 5y agoThis depends greatly what you want to do. After all, this has incredible latency (compared to the cloud), limited throughput. Zero configuration. Free. You want to train the next imagenet model? Analyse 100Gig database? Probably not the correct tradeoff. You want 20 students to have a perfectly consistent instant-start-up python environment? Definitely the correct tradeoff. You want to try some python methods, write tests, ... Very short latency is going to help you more than throughput is.
- egeozcan 5y agoTo be fair, "we don't need more cloud computing" doesn't mean, "we don't need any cloud computing". I don't agree that we don't need more of cloud, IMO we do, but we need to focus on personal computing much more than now, which is the general theme of the GP comment. Not everything is big data.
- throwaway894345 5y agoTo be clear, I’m not talking about big data. More impotantly though, I didn’t say there was no place for client compute, only that it isn’t economical for datasets in excess of a few GB.
- deleted 5y ago[deleted]
- otabdeveloper4 5y agoYou definitely shouldn't be running this stuff from a Jupyter notebook.
- throwaway894345 5y agoWhy not? We make a lot of money offering it to people.
- otabdeveloper4 5y agoYes, the market in enabling people to do the "you shouldn't be doing this" stupid things is huge.
- throwaway894345 5y agoAny substance on why we shouldn’t be doing it or why it’s stupid? What’s the alternative? Should researchers all learn Kubernetes and AWS and deploy their own environments?
- otabdeveloper4 5y agoThe problem with Jupyter is that it impedes common-sense practices like version control, reproducibility, and automation. If you're spending the time and effort to rent these big servers, why not spend the 5 percent of the effort and do it right? Jupyter exists mostly because analytics/math guys are too lazy to spend a day learning software development practices. Must be some sort of us-vs-them point of misplaced pride.
- musingsole 5y ago> Jupyter exists mostly because analytics/math guys are too lazy to spend a day learning software development practices Bahahahaha, yup those dang lazy mathematicians just shooting themselves in the foot and forcing us to deal with it! /s Your preferences for software development are irrelevant. The value is in delivering the math to the end user. Using GCP, kubernetes or JavaScript to do that is an implementation detail. Sorry to tell you this, but you're a servant to those dang lazy analysts and without their insights, you're worthless.
- huac 5y agoNot only do cloud services offer better compute capabilities (and GPUs/TPUs etc), but they offer easier reproduciblity and sharing. Even when I hack on stuff myself, Colab is quick and easy to set up, no worrying about Docker or virtualenvs.