3 ms·
Python quickly falls over with large datasets. Its great as a "glue" language or for POCs for large data processing or handling jobs - but at some point you nee
by anonu 7y ago
Python quickly falls over with large datasets. Its great as a "glue" language or for POCs for large data processing or handling jobs - but at some point you need to graduate to faster things...
- pletnes 7y agoWhat’s «large»? I’ve set up multi-TB analyses without issue, using dask. I don’t know of a more productive language for such analyses, either. What is even better?
- TuringNYC 7y ago+1 on dask-distributed. I'm surprised they dont get more coverage/discussion in the industry! Why do you think they are so overlooked?
- pletnes 7y agoYou can chew through TB even without distributed, as long as you have the disks and cores on the same system.
- nojito 7y agodata.table is leagues ahead of python in terms of code length and performance. https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/
- gdulli 7y agoLanguage speed becomes mostly irrelevant compared to the strategy, framework, toolset, and dataset design for parallelization. If you have a use case where a program in C runs in a reasonable time on your laptop but one in Python doesn't, that solution is only going to take you so far before you'll want to graduate from your laptop and take advantage of the Python ecosystem again in real big data contexts.
- nightski 7y agoWell that just isn't true. The only reason Python is remotely even usable in this context is because it's merely a wrapper around C the vast majority of the time. Language definitely matters.
- jacobr1 7y agoBetween things like pyspark or using a rdbms you can scale pretty far. Even with other languages like java this is the case. Once your dataset goes beyond a single machine you need some kind of data platform.
- pmiller2 7y agoYou know, actual databases are something you can "glue" onto a Python infrastructure.