3 ms·
As some other commenters have said, this is really a self-inflicted problem. The author has chosen to do an EDA task which is very manageable on a laptop via a
by lordgrenville 4y ago
As some other commenters have said, this is really a self-inflicted problem. The author has chosen to do an EDA task which is very manageable on a laptop via a convoluted stack of cloud services - possibly just to illustrate a point. But even if this all worked smoothly, the fact that it is far removed from "software engineering" has more to do with the fact that it's a data analysis project. If it were about about writing firmware for audio hardware it would look very different.
Nitpick: ! for shell commands is an IPython feature, not Jupyter. It doesn't work with other kernels.
- scaredginger 4y agoIt's also well-known that pandas is memory-inefficient and Dask would probably do the memory estimation they described for them (after the author dismissed it). They're really just showing they don't really understand these tools that well https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/