The issue with data processing with workers in both Node and the browser is paying for serialization costs. Until we have rich typed containers & structs on top

The issue with data processing with workers in both Node and the browser is paying for serialization costs. Until we have rich typed containers & structs on top of raw shared memory arrays (which necessitates manual memory management), all these parallel libraries are highly constrained to domains where the unit of work is very large, and the result is either a side-effect or very small. I tried to use workers to speed up a JS bundler/compiler system 6 years ago by farming out parsing & analysis work, and getting results back onto the main thread for my naive implementation took more CPU time than the work itself. Deserialization costs on the main thread is one of the biggest issues we have in the Notion back-end as well - no matter how fast the DB/cache are, we’re throughout limited by JSON.parse (or equivalent). I’m thinking seriously about Arrow because we could keep data in a Dataframe instead of as JS objects, and share freely between threads without copy, but that kind of strategy would require a large codebase rethink. Much more expensive than other languages where a bit of shared memory threading can truly speed up data processing.