3 ms·

I think the most interesting part of this project is the fault tolerance. I can’t say I’ve seen any other projects do this, but it seems reasonable to want chec

by sakras 4y ago

I think the most interesting part of this project is the fault tolerance. I can’t say I’ve seen any other projects do this, but it seems reasonable to want checkpointing during a long computation. Another thing I like is that conceptually it seems like it would be simple to switch the underlying query engine (right now it’s Polars) in the future. Seems like a pretty general distributed system.

marsupialtail_2 4y ago
Perhaps you can write the underlying query engine :-) in something that's not Python lol
- punnerud 4y ago
  Even the self driving in Tesla use Python. If your Python is slow, you are using Python wrong. So it’s kind of a good thing that it’s natively slow, because you are not supposed to do your computations with native Python. It it was faster (added a lot of type hints and build processes), you would probably not be nudged in the right direction. https://cerfacs.fr/coop/fortran-vs-python https://cerfacs.fr/coop/fortran-vs-python
  - arthurcolle 4y ago
    Self driving for TSLA cars is not really the benchmark I'd use for "something that works reliably"
    - yunohn 4y ago
      I don’t think there’s any evidence that FSD’s reliability is affected by being written in Python.
- gilnaa 4y ago
  Polars is already not python
marsupialtail_2 4y ago
Checkpointing is actually a bit difficult for fault tolerance for long running batch computations and state tends to grow linearly as the computation progresses, unlike streaming applications where you typically use windows. This means periodic checkpointing leads to quadratic amount of writes to durable storage which is pretty bad. Quokka supports checkpointing but does not enable that by default to prevent this common problem from killing normal operation performance.
divbzero 4y ago
Fault tolerance is an important feature but, given Python’s popularity in data science, the most interesting part of this project to me is support for Python UDFs. In principle, the programming language should not be the greatest consideration because developers can learn and use different languages for different applications. In practice, being able to draw on familiar syntax and libraries can make a real difference in usability.

by sakras 4y ago

marsupialtail_2 4y ago

Perhaps you can write the underlying query engine :-) in something that's not Python lol

punnerud 4y ago
Even the self driving in Tesla use Python. If your Python is slow, you are using Python wrong. So it’s kind of a good thing that it’s natively slow, because you are not supposed to do your computations with native Python. It it was faster (added a lot of type hints and build processes), you would probably not be nudged in the right direction. https://cerfacs.fr/coop/fortran-vs-python https://cerfacs.fr/coop/fortran-vs-python
- arthurcolle 4y ago
  Self driving for TSLA cars is not really the benchmark I'd use for "something that works reliably"
  - yunohn 4y ago
    I don’t think there’s any evidence that FSD’s reliability is affected by being written in Python.
gilnaa 4y ago
Polars is already not python

marsupialtail_2 4y ago

Checkpointing is actually a bit difficult for fault tolerance for long running batch computations and state tends to grow linearly as the computation progresses, unlike streaming applications where you typically use windows. This means periodic checkpointing leads to quadratic amount of writes to durable storage which is pretty bad. Quokka supports checkpointing but does not enable that by default to prevent this common problem from killing normal operation performance.

divbzero 4y ago

Fault tolerance is an important feature but, given Python’s popularity in data science, the most interesting part of this project to me is support for Python UDFs. In principle, the programming language should not be the greatest consideration because developers can learn and use different languages for different applications. In practice, being able to draw on familiar syntax and libraries can make a real difference in usability.