2 ms·
Nice work. This looks like it could add a lot of value to a DS's toolbox. Exploratory data analysis of large (but not huge) datasets have always been a slow an
by wenc 8y ago
Nice work. This looks like it could add a lot of value to a DS's toolbox.
Exploratory data analysis of large (but not huge) datasets have always been a slow and frustrating experience.
In the enterprise, we have plenty of datasets that are 100s of millions to a few billion rows (and many columns), so big enough to make conventional tools sluggish but not quite big enough for distributed computing. It sounds like vaex can help with EDA of these types of datasets on a single machine. I'd be interested in exploring the out-of-core functionality, which I hope means it will continue chugging along without throwing "out of memory" errors.
- maartenbreddels 8y agoThat is exactly the sweet spot for vaex, and with a familiar DataFrame API (read pandas like) the transition does not hurt so much. It may sound cool to set up a cluster, but in many cases it is overkill, and vaex can get these kinds of jobs done.