3 ms·

How would this compare to ingesting this in a locally running rdbms, like postgres, and applying some judicious indexing? Or maybe a local spark cluster? I'm t

How would this compare to ingesting this in a locally running rdbms, like postgres, and applying some judicious indexing? Or maybe a local spark cluster? I'm tempted to benchmark this against better known tech, maybe anyone has some insight to share?

maartenbreddels 7y ago
I've never benchmarked against postgres, but would be interested about the results. I once tried monetdb, and it was orders of magnitude slower for simple calculations, so I stopped looking at RDMS'es after that. I think they solve a different problem, smaller data, data/relational integrity, data mutations. Dataframes can take shortcuts here. If you want to do a serious benchmark, feel free to contact us. Github: https://github.com/vaexio/vaex/issues https://github.com/vaexio/vaex/issues Email: (I'm easy to google).
drblah 7y ago
I am currently doing an internship as part of my masters degree where I am analyzing ~30 GB of data. I'm using Postgres + Python and it is working quite well, even on my 2014 MacBook Air. It would indeed be interesting to see how this approach with Vaex compares to Postgres. Though, I would be quite sad giving up SQL in favour of Pandas DataFrame indexing and Python looping :)
- maartenbreddels 7y ago
  No Python looping happening in Vaex :), otherwise, we wouldn't get this performance. We are also working on GraphQL support, with a Hasura-like API: https://docs.vaex.io/en/latest/example_graphql.html https://docs.vaex.io/en/latest/example_graphql.html I think GraphQL is easier in combinations with front end development, and you can tab-complete your way out. Early days for this sub-project, but I think very promising.
  - drblah 7y ago
    Ah, of cause. It makes sense you don't loop in Python. This all seems pretty interesting. I will give it a go.
- mooneater 7y ago
  Ive been switching an reporting system that did analysis in postgres, to analysis in pandas (mostly business-stats type summaries). It feels like growing wings and a jetpack. Almost everthing is waaay easier and faster.
kthejoker2 7y ago
Think you'd want to compare with a columnar database since this is an analytic workload, not a transactional one. And a 4 node Spark cluster with Parquet or Arrow files and a Scala job will compare favorably to this because it's also a lazy evaluator and the benchmark problems here are embarrassingly parallel. PS this is a very cool project!

maartenbreddels 7y ago

I've never benchmarked against postgres, but would be interested about the results. I once tried monetdb, and it was orders of magnitude slower for simple calculations, so I stopped looking at RDMS'es after that. I think they solve a different problem, smaller data, data/relational integrity, data mutations. Dataframes can take shortcuts here. If you want to do a serious benchmark, feel free to contact us. Github: https://github.com/vaexio/vaex/issues https://github.com/vaexio/vaex/issues Email: (I'm easy to google).

drblah 7y ago

I am currently doing an internship as part of my masters degree where I am analyzing ~30 GB of data. I'm using Postgres + Python and it is working quite well, even on my 2014 MacBook Air. It would indeed be interesting to see how this approach with Vaex compares to Postgres. Though, I would be quite sad giving up SQL in favour of Pandas DataFrame indexing and Python looping :)

maartenbreddels 7y ago
No Python looping happening in Vaex :), otherwise, we wouldn't get this performance. We are also working on GraphQL support, with a Hasura-like API: https://docs.vaex.io/en/latest/example_graphql.html https://docs.vaex.io/en/latest/example_graphql.html I think GraphQL is easier in combinations with front end development, and you can tab-complete your way out. Early days for this sub-project, but I think very promising.
- drblah 7y ago
  Ah, of cause. It makes sense you don't loop in Python. This all seems pretty interesting. I will give it a go.
mooneater 7y ago
Ive been switching an reporting system that did analysis in postgres, to analysis in pandas (mostly business-stats type summaries). It feels like growing wings and a jetpack. Almost everthing is waaay easier and faster.

kthejoker2 7y ago

Think you'd want to compare with a columnar database since this is an analytic workload, not a transactional one. And a 4 node Spark cluster with Parquet or Arrow files and a Scala job will compare favorably to this because it's also a lazy evaluator and the benchmark problems here are embarrassingly parallel. PS this is a very cool project!