maartenbreddels

8 ms·

31.
by maartenbreddels 7y ago
Thanks :) The first commit was Jan 2014, when it between 50-80% of my time until 2018 I think where I mostly developed it myself. After that, it is more difficult to say how much time was spend on it, and it wasn't only my time. Althou
32.
by maartenbreddels 7y ago
Indeed, numba, Pythran or cupy can be really useful. In the example of the article, it is 'simple' vectorized math. But in general any function can be added in Vaex. Those that go through numba or Pythran usually release the GIL a
33.
by maartenbreddels 7y ago
Indeed, numpy for numerical calculations. For strings, we have our own data structure based on Apache Arrow, but we plan/hope to move to Apache Arrow (in combination with numpy), since that's kind of the numpy++ for data science w
34.
by maartenbreddels 7y ago
What do you mean by that?
35.
by maartenbreddels 7y ago
No Python looping happening in Vaex :), otherwise, we wouldn't get this performance. We are also working on GraphQL support, with a Hasura-like API: https://docs.vaex.io/en/latest/example_graphql.html I think
36.
by maartenbreddels 7y ago
I've never benchmarked against postgres, but would be interested about the results. I once tried monetdb, and it was orders of magnitude slower for simple calculations, so I stopped looking at RDMS'es after that. I think they solv
37.
by maartenbreddels 7y ago
It is not similar to Dask, but similar to dask.dataframe. Dask.dataframe is built on top of Pandas, but that also means it inherits its issue, like memory usage, and performance. (BTW, totally a fan of Pandas). Xarray is more about nd-array
38.
Show HN: How to analyse 100 GB of data on your laptop with Python(towardsdatascience.com)
264 points by maartenbreddels 7y ago|26 comments
39.
Flying high with Vaex: analysis of over 30 years of flight data in Python(towardsdatascience.com)
10 points by maartenbreddels 7y ago|1 comments
40.
Show HN: Vaex – A DataFrame with super strings (up to 1000x speedup)(towardsdatascience.com)
11 points by maartenbreddels 7y ago|0 comments
41.
by maartenbreddels 8y ago
Don't expect this performance on a 4GB machine. Most machines now would have 16GB, or more. Let us assume you have 32GB and take a 24GB dataset. Most libraries load this into memory (allocating 24GB, leaves max 8 GB for the OS, includi
42.
by maartenbreddels 8y ago
the 'convert' name is misleading perhaps, maybe we can agree the proof is in the execution time https://youtu.be/TlTcQJPUL3M?t=478 Anyway, let us celebrate a wider adoption of Arrow! :)
43.
by maartenbreddels 8y ago
thanks!
44.
by maartenbreddels 8y ago
Oh, and thanks for the kind words!
45.
by maartenbreddels 8y ago
There is an issue open for this: https://github.com/vaexio/vaex/issues/93 It should have been fixed, some more detailed report (version numbers installed) would be good to know.
46.
by maartenbreddels 8y ago
That is not correct, I just refer to the buffers/memory, 0 copying going on. Vaex is not really opinionated about the memory model actually. The only exception is the bitmasks that are being copied for now because of an incompatibility
47.
by maartenbreddels 8y ago
That is exactly the sweet spot for vaex, and with a familiar DataFrame API (read pandas like) the transition does not hurt so much. It may sound cool to set up a cluster, but in many cases it is overkill, and vaex can get these kinds of job
48.
by maartenbreddels 8y ago
You are right, I'm actually underselling it. 1 second is the typical performance for doing a 2d histogram (or other binned statistics) since it involves writing to memory as well. I just ran a quick benchmark: In [7]: %timeit -r3 -n3 d
49.
by maartenbreddels 8y ago
Thank you, yes they give much more flexibility: optimization (JIT), derivatives, checking your calculations afterwards, sending them to a remote server etc. Glad you like that :)
50.
by maartenbreddels 8y ago
Thank you. Memory mapping could be used for other data as well, and I have looked into zarr (even opened an issue for that https://github.com/zarr-developers/zarr/issues ). Memory mapping of contiguous data makes
51.
by maartenbreddels 8y ago
Absolutely, I think nowadays the question should be: 'does it still support Python2?' (it does btw) My question is to you is, would you be so kind to open an issue to decribe the failure on https://github.com/vaexi
52.
by maartenbreddels 8y ago
I have not looked into it, maybe datashader can do this, which is a package purely focussing on viz, while vaex is more allround (although there is overlap). If you think vaex can be useful here, feel free to ask question/open issues
53.
by maartenbreddels 8y ago
Dask and vaex are not 'competing', they are orthogonal. Vaex could use dask to do the computations, but when this part of vaex was built, dask didn't exist. I recently tried using dask, instead of vaex' internal computat
54.
Show HN: Vaex - Out of Core Dataframes for Python and Fast Visualization(medium.com)
126 points by maartenbreddels 8y ago|32 comments
55.
Multivolume rendering in Jupyter with ipyvolume: cross-language 3d visualization(medium.com)
5 points by maartenbreddels 8y ago|0 comments
56.
Show HN: Ipywebrtc – Video/Camera streaming and snapshot fun for Jupyter(github.com)
3 points by maartenbreddels 8y ago|0 comments
57.
by maartenbreddels 8y ago
np, yes it does do 3d selections (projected only for now), shown here https://twitter.com/maartenbreddels/status/96712877380723507... but not yet documented. Should be for the next version (0.5, to be released thi
58.
by maartenbreddels 8y ago
Excellent work! From the article: > I’ve worked with a lot of data visualization libraries over the years, both proprietary and open source, and I don’t know of any that can both display a million points this quickly, and support zooming
59.
by maartenbreddels 10y ago
Shown on HN before, but this new version of ipyvolume has support for axes, styling, but most important addition is animations (with interpolation), see also the github (with new screencast): https://github.com/maartenbredd
60.
Show HN: Ipyvolume 0.3.0: Smooth animated 3d plotting in the Notebook(ipyvolume.readthedocs.io)
3 points by maartenbreddels 10y ago|1 comments

More ›