4 ms·

RipTable – multi-threaded Python data analytics tools for numpy arrays/datasets

79 points by aldanor 6y ago

Tinyyy 6y ago
> RipTable has been in development for 3 years and tested by dozens of quants at a large financial firm. I guess it's Susquehanna.
- jamessb 6y ago
  That seems almost certainly true, as at least four of the contributors seem to work there (Amarildo Zeneli, Thomas McClintock, Thomas Dimitri, Jack Papas). I wonder why they don't just release it as a Susquehanna (rather than "RTOS Holdings") project, as their involvement isn't exactly secret.
abalaji 6y ago
Does anyone know how this compares to something like Dask?
- mattip 6y ago
  Riptable can only distribute at the low level of primitives it sees, so it must split tasks and collect results at the level of single function calls. Dask builds up graphs of operations on a higher level. There are advantages and disadvantages to both approaches, depending on the actual task.
antman 6y ago
Looks interesting, a benchmark page would help in promoting it
arbol 6y ago
Why would this be preferred over using a GPU for calculations?
- aldanor 6y ago
  Your GPU has a few hundred gb of ram?
  - arbol 6y ago
    Well, multiple GPUs can easily have this, yes. Surely the mining rigs they use in crypto can also be used in data science. Plus the new Nvidia GPU has 80gb.
    - aldanor 6y ago
      It's mostly not about "we need to mine these 100gb of data", but rather pandas-like comfort of messing around with humongous amount of data that is stored in ram, applying some common aggregations, filters, searches etc, and possibly calling some c functions if needed. This just not the use case for gpus.
smabie 6y ago
it's pretty crazy what people are willing to do to make Python faster: Pyston, Numba, Cython, RipTable. Instead of just biting the bullet and adandoning Python, they spend thousands of manhours trying to hack a fundamentally slow and broken language into something that is acceptable. I see these hacks as a cognitive bias and rarely think they are the rational and correct decision. Just use Julia and call it a day. It really is that easy.
- disgruntledphd2 6y ago
  So, perhaps it makes sense in your industry to go all-in on Julia. Personally, I'm still burned by the time they announced 1.0 mere days after announcing 0.7, and thus meaning that literally nothing in the ecosystem worked at 1.0. If that transition had been managed better, then I would be much more comfortable evangelising Julia now, but it didn't and the decisions behind this make me incredibly uncomfortable in adopting it anywhere where I'm on the hook for failures. Additionally, Python has libraries for everything, and is known (badly at least), by almost everyone in Data Science/programming. When your modelling is a part of a larger project, this matters a lot (and is why I no longer write my production code in R). That being said, I really like Julia, and look forward to checking it out again, hopefully with less breakages in the future.
- aldanor 6y ago
  It's not the Python itself but the ecosystem and tooling around it, especially in machine learning / data science area. I find it strange that many people don't understand it and instead compare languages is. "Just use Julia" is not really an option in enterprise world either, when you already have hundreds of kloc, hundreds/thousands of packages and hundreds of people (not all of them professional developers, e.g. various analysts and quants) working on / using that on daily basis in production, plus all that code almost certainly depends on external packages (eg machine learning libraries) that are simply not available in Julia. For personal projects - sure, go use Julia or ditch it and go straight for Rust.
  - smabie 6y ago
    They don't need to be available on Julia, calling Python libraries from Julia is incredibly painful. I'm not saying you should migrate your whole data analysis pipeline over, I'm saying you should write new services/analysis in Julia. There are production trading systems that are written in Python. And everyday, the quants and devs fruitlessly try and squeeze out more performance from the system instead of just accepting that Python is inappropriate.

79 points by aldanor 6y ago

> RipTable has been in development for 3 years and tested by dozens of quants at a large financial firm. I guess it's Susquehanna.

jamessb 6y ago
That seems almost certainly true, as at least four of the contributors seem to work there (Amarildo Zeneli, Thomas McClintock, Thomas Dimitri, Jack Papas). I wonder why they don't just release it as a Susquehanna (rather than "RTOS Holdings") project, as their involvement isn't exactly secret.

abalaji 6y ago

Does anyone know how this compares to something like Dask?

mattip 6y ago
Riptable can only distribute at the low level of primitives it sees, so it must split tasks and collect results at the level of single function calls. Dask builds up graphs of operations on a higher level. There are advantages and disadvantages to both approaches, depending on the actual task.

antman 6y ago

Looks interesting, a benchmark page would help in promoting it

arbol 6y ago

Why would this be preferred over using a GPU for calculations?

aldanor 6y ago
Your GPU has a few hundred gb of ram?
- arbol 6y ago
  Well, multiple GPUs can easily have this, yes. Surely the mining rigs they use in crypto can also be used in data science. Plus the new Nvidia GPU has 80gb.
  - aldanor 6y ago
    It's mostly not about "we need to mine these 100gb of data", but rather pandas-like comfort of messing around with humongous amount of data that is stored in ram, applying some common aggregations, filters, searches etc, and possibly calling some c functions if needed. This just not the use case for gpus.

smabie 6y ago

it's pretty crazy what people are willing to do to make Python faster: Pyston, Numba, Cython, RipTable. Instead of just biting the bullet and adandoning Python, they spend thousands of manhours trying to hack a fundamentally slow and broken language into something that is acceptable. I see these hacks as a cognitive bias and rarely think they are the rational and correct decision. Just use Julia and call it a day. It really is that easy.

disgruntledphd2 6y ago
So, perhaps it makes sense in your industry to go all-in on Julia. Personally, I'm still burned by the time they announced 1.0 mere days after announcing 0.7, and thus meaning that literally nothing in the ecosystem worked at 1.0. If that transition had been managed better, then I would be much more comfortable evangelising Julia now, but it didn't and the decisions behind this make me incredibly uncomfortable in adopting it anywhere where I'm on the hook for failures. Additionally, Python has libraries for everything, and is known (badly at least), by almost everyone in Data Science/programming. When your modelling is a part of a larger project, this matters a lot (and is why I no longer write my production code in R). That being said, I really like Julia, and look forward to checking it out again, hopefully with less breakages in the future.
aldanor 6y ago
It's not the Python itself but the ecosystem and tooling around it, especially in machine learning / data science area. I find it strange that many people don't understand it and instead compare languages is. "Just use Julia" is not really an option in enterprise world either, when you already have hundreds of kloc, hundreds/thousands of packages and hundreds of people (not all of them professional developers, e.g. various analysts and quants) working on / using that on daily basis in production, plus all that code almost certainly depends on external packages (eg machine learning libraries) that are simply not available in Julia. For personal projects - sure, go use Julia or ditch it and go straight for Rust.
- smabie 6y ago
  They don't need to be available on Julia, calling Python libraries from Julia is incredibly painful. I'm not saying you should migrate your whole data analysis pipeline over, I'm saying you should write new services/analysis in Julia. There are production trading systems that are written in Python. And everyday, the quants and devs fruitlessly try and squeeze out more performance from the system instead of just accepting that Python is inappropriate.