4 ms·

Ahh, thw beauty and versatility of Python again. And the momentum doesn’t stop. Never regret jumping off of Perl, then Ruby ten years ago to fully commit to Pyt

Ahh, thw beauty and versatility of Python again. And the momentum doesn’t stop. Never regret jumping off of Perl, then Ruby ten years ago to fully commit to Python. I can see so many use-cases for it (with the rich ecosystem), on an almost daily basis.

anonu 6y ago
Python quickly falls over with large datasets. Its great as a "glue" language or for POCs for large data processing or handling jobs - but at some point you need to graduate to faster things...
- pletnes 6y ago
  What’s «large»? I’ve set up multi-TB analyses without issue, using dask. I don’t know of a more productive language for such analyses, either. What is even better?
  - TuringNYC 6y ago
    +1 on dask-distributed. I'm surprised they dont get more coverage/discussion in the industry! Why do you think they are so overlooked?
    - pletnes 6y ago
      You can chew through TB even without distributed, as long as you have the disks and cores on the same system.
  - nojito 6y ago
    data.table is leagues ahead of python in terms of code length and performance. https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/
- gdulli 6y ago
  Language speed becomes mostly irrelevant compared to the strategy, framework, toolset, and dataset design for parallelization. If you have a use case where a program in C runs in a reasonable time on your laptop but one in Python doesn't, that solution is only going to take you so far before you'll want to graduate from your laptop and take advantage of the Python ecosystem again in real big data contexts.
  - nightski 6y ago
    Well that just isn't true. The only reason Python is remotely even usable in this context is because it's merely a wrapper around C the vast majority of the time. Language definitely matters.
- jacobr1 6y ago
  Between things like pyspark or using a rdbms you can scale pretty far. Even with other languages like java this is the case. Once your dataset goes beyond a single machine you need some kind of data platform.
- pmiller2 6y ago
  You know, actual databases are something you can "glue" onto a Python infrastructure.
Scarbutt 6y ago
Any programming language with a http client and a sqlite driver won't have problems with this example. Although, this is so simple that even sqlite might be overkill for analyzing the data.

anonu 6y ago

Python quickly falls over with large datasets. Its great as a "glue" language or for POCs for large data processing or handling jobs - but at some point you need to graduate to faster things...

pletnes 6y ago
What’s «large»? I’ve set up multi-TB analyses without issue, using dask. I don’t know of a more productive language for such analyses, either. What is even better?
- TuringNYC 6y ago
  +1 on dask-distributed. I'm surprised they dont get more coverage/discussion in the industry! Why do you think they are so overlooked?
  - pletnes 6y ago
    You can chew through TB even without distributed, as long as you have the disks and cores on the same system.
- nojito 6y ago
  data.table is leagues ahead of python in terms of code length and performance. https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/
gdulli 6y ago
Language speed becomes mostly irrelevant compared to the strategy, framework, toolset, and dataset design for parallelization. If you have a use case where a program in C runs in a reasonable time on your laptop but one in Python doesn't, that solution is only going to take you so far before you'll want to graduate from your laptop and take advantage of the Python ecosystem again in real big data contexts.
- nightski 6y ago
  Well that just isn't true. The only reason Python is remotely even usable in this context is because it's merely a wrapper around C the vast majority of the time. Language definitely matters.
jacobr1 6y ago
Between things like pyspark or using a rdbms you can scale pretty far. Even with other languages like java this is the case. Once your dataset goes beyond a single machine you need some kind of data platform.
pmiller2 6y ago
You know, actual databases are something you can "glue" onto a Python infrastructure.

Scarbutt 6y ago

Any programming language with a http client and a sqlite driver won't have problems with this example. Although, this is so simple that even sqlite might be overkill for analyzing the data.