I don't have very much background in ML or distributed systems, so forgive my naive questions... > After all, most ML in industry today seems to be lightweight

I don't have very much background in ML or distributed systems, so forgive my naive questions... > After all, most ML in industry today seems to be lightweight models applied to heavily engineered features I assume "lightweight models" are those that don't have too many parameters, and "heavily engineered features" mean that the data fed into the model has undergone significant pre-processing via potentially complicated UDFs -- hence the motivation for the project. Is that right? > Quokka is an open-source push-based vectorized query engine ... it is meant to be much more performant than blocking-shuffle based alternatives like SparkSQL Does anyone have pointers to what push-based vs blocking-shuffle engines are? Any good papers? > It should work on local machine no problem (and should be a lot faster than Pandas!) So I understand why Quokka is faster than Spark, but I'm a bit uncertain as to why the author is also making a comparison with Pandas on a single machine. Is it because the streaming pipeline design means that Quokka can better take advantage of multiple cores?