5 ms·

My experience with Web Workers was terrible. It's not because of the Web Workers itself but there is no proper standard way to use modules inside of the worker.

My experience with Web Workers was terrible. It's not because of the Web Workers itself but there is no proper standard way to use modules inside of the worker. Because it requires a separated .js file to instantiate, bundlers like webpack tend to introduce weird and non-standard way to work with [1]. It is usually fine when you use it your own project since you are going to stick with a bundler you choose anyway, but the worst things happens when you try to publish a JS library that uses a web worker internally. You cannot just publish the source because bundlers won't recognize require() and imports syntax in the web worker source code out of the box. There is a ES standard: new Worker("worker.js", { type: "module" }); but no browsers implementations yet and some people aren't happy with this because it is not async [2]. [1]: https://github.com/webpack-contrib/worker-loader/blob/master/README.md#config https://github.com/webpack-contrib/worker-loader/blob/master... [2]: https://bugs.chromium.org/p/chromium/issues/detail?id=680046 https://bugs.chromium.org/p/chromium/issues/detail?id=680046

dassurma 7y ago
Agreed. I am quite annoyed that no browser has shipped modules in workers. There is little incentive as Workers are sorely underused. Since I mostly use rollup, I ended up writing a plugin[1] that allows you to pretend that modules are in workers. It compiles down to AMD under the hood and uses a very minimalistic worker. [1]: https://github.com/surma/rollup-plugin-loadz0r https://github.com/surma/rollup-plugin-loadz0r
bufferoverflow 7y ago
Can someone explain why the ES standard doesn't have something like this: new Worker(function(params) { ... compute heavy stuff here ... });
- dassurma 7y ago
  The problem is mostly in that all these functions are closures, and scope can’t easily be transferred. Domenic and I have been working on a proposal[1] called Blöcks to introduce transferable functions to JavaScript. In the meantime, I wrote Clooney[2] ontop of Comlink[3] that gives you almost that. [1]: https://github.com/domenic/proposal-blocks https://github.com/domenic/proposal-blocks [2]: https://github.com/GoogleChromeLabs/clooney https://github.com/GoogleChromeLabs/clooney [3]: https://github.com/GoogleChromeLabs/comlink https://github.com/GoogleChromeLabs/comlink
  - ajkjk 7y ago
    Oo, I'm really interested that you're working on that, although I don't love the syntax (particularly `worker<endpoint>` seems awkward). It seems like you should stick with function invocation syntax: `worker = (endpoint) => {| block |}` `worker(endpont) // does what you want` with the difference that the result isn't a closure. As a bonus this syntax would let you write 'pure functions' even if you weren't working with workers. Perhaps the worker version then is `worker = async (endpoint => {| block... |}`. Although considering the precedence for `async function` maybe the way to do this ought to be `pure function`.
    - dassurma 7y ago
      If you take a look at the issues on the repo, a lot of people agree with you. The syntax was not final at all, we need to convince TC39 first before we can start bikeshedding ^^
  - dbetteridge 7y ago
    Web workers would make a good http203 episode, uses/pitfalls etc
- mstade 7y ago
  I don't actually know the answer, but my guess would be it's because of scoping. Web workers aren't a language level feature, so they can't really dictate that the function body can't have access to the outer scope which functions in JS typically do. And if worker functions like that would have access, how do you ensure thread safety? Making it so you have to point to a different module altogether seems like a decent way to enforce that boundary.
- deleted 7y ago
  [deleted]
- rl3 7y ago
  >Can someone explain why the ES standard doesn't have something like this: new Worker(function(params) { ... compute heavy stuff here ... }); In 2015 I worked on a framework that functioned like this as a personal project. It was even context-aware and spun itself up differently depending on the environment it was in; workers and shared workers loaded the same .js file that the UI thread did. As such, it chose speed over memory use seeing that the code was effectively pre-loaded everywhere. It was never completed, and the reasons for that were: 1. SharedArrayBuffer hadn't landed yet. Transfer cost of computable data between workers was high, and transferables had shortcomings of their own. 2. Closures were disallowed, and AST transformations were one possible solution. It seemed like a very bad idea to go down that road. 3. Back then the differences in API behavior between even Chrome and Firefox were terrible to deal with. The APIs were heavily neglected across the board. These days, I don't see the need. If you need the performance, WASM is a better choice anyways and depending on what you're doing it can use SharedArrayBuffer much more efficiently via way of something like pthreads.
  - millstone 7y ago
    But wasm isn’t asynchronous right? It still blocks the main thread.
  - kbumsik 7y ago
    > WASM is a better choice anyways and depending on what you're doing it can use SharedArrayBuffer much more efficiently via way of something like pthreads. I think you misunderstand the concept of WASM. WASM does not have its own thread, it blocks the main thread and runs in the same context as JS. WASM still must spawn a Web Worker to emulate multithreading in a browser.
    - rl3 7y ago
      I do understand, was just mobile and in a hurry so I failed to fully qualify the statement. What I was saying is if you're in the WASM toolchain/ecosystem, something like pthreads would already be implemented by Emscripten[0]. In other words, spinning up web workers is handled for you seamlessly. It's all dealt with at a very low level of abstraction. You're literally using the SharedArrayBuffer as memory. Contrast that to JS, and you have to serialize/deserialize types before they can even be stored in SharedArrayBuffer. It's costly and far from seamless. As an aside, it's unfortunate SharedArrayBuffer was hit so hard by Spectre. The poor standard is cursed or something. [0] https://emscripten.org/docs/porting/pthreads.html https://emscripten.org/docs/porting/pthreads.html
- apatheticonion 7y ago
  This is actually similar to how it's done in Go Imagine you define a function: func hello(value string) { // compute heavy stuff } To run it sync hello("World") To run it async, just add 'go' go hello("World") IMO when it has generics, it'll be the best language I have worked with.
- franciscop 7y ago
  Since you seem not to like Clooney, what about uwork? I wrote it a while ago: https://github.com/franciscop/uwork https://github.com/franciscop/uwork const findPi = uwork((iterations = 10000) => { let inside = 0; for (var i = 0; i < iterations; i++) { let x = Math.random(), y = Math.random(); if (x * x + y * y <= 1) inside++; } return 4 * inside / iterations; }); // Run this inside an async context: const pi = await findPi(200000000);
zawerf 7y ago
I don't really publish libraries so I didn't run into your main gripe. Is there any other reason why you didn't like worker-loader? I experimented with it and found it pretty straightforward. For anyone who wants to try it, you can even get it working from a create-react-app project with the webpack inline loader syntax without ejecting: /* eslint import/no-webpack-loader-syntax: "warn" */ import Worker from 'worker-loader!./Worker.js'; My use case was to run tensorflow.js in the background. This actually mostly worked out of the box with only some awkwardness with serializing images as messages. The only thing that didn't work out was that you don't have access to webgl there which defeated my original use case (since there's no point in offloading computation if that computational is going to be several orders slower in the background).
- paulryanrogers 7y ago
  Would it be possible to draw off screen or canvas somehow? To preserve WebGL compatibility while still using the workers?
  - Veedrac 7y ago
    https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCa... https://caniuse.com/#feat=offscreencanvas https://caniuse.com/#feat=offscreencanvas
- kbumsik 7y ago
  Because if you use it in a library you immediately create a hard dependency to Webpack. Libraries should be designed to work with any bundlers, or even without a bundler at best. Besides, I personally don't feel good with Webpack's do-everything-with-import approach. It is straightforward of course, but is ES6 import statement supposed be used like this?
moron4hire 7y ago
I did a thing where I used data URIs instead of explicit files to load up web workers. Don't know if that still works, as this was a few years ago, but it definitely helped a lot with those sorts of issues at the time.
- kbumsik 7y ago
  Yes, it is still one way to workaround the problem. For those who don't know: function workerFn() { ...Do worker stuff } let worker = new Worker(URL.createObjectURL(new Blob(['('+workerFn.toString()+')()']))); Though this hack works but it still have problems to be addressed: 1. It still does not resolve import/require(). This introduces another hack that dumping output of a bundler to workerFn. 2. It will make it much harder to debug since you create worker out of data, not source code.

dassurma 7y ago

Agreed. I am quite annoyed that no browser has shipped modules in workers. There is little incentive as Workers are sorely underused. Since I mostly use rollup, I ended up writing a plugin[1] that allows you to pretend that modules are in workers. It compiles down to AMD under the hood and uses a very minimalistic worker. [1]: https://github.com/surma/rollup-plugin-loadz0r https://github.com/surma/rollup-plugin-loadz0r

bufferoverflow 7y ago

Can someone explain why the ES standard doesn't have something like this: new Worker(function(params) { ... compute heavy stuff here ... });

dassurma 7y ago
The problem is mostly in that all these functions are closures, and scope can’t easily be transferred. Domenic and I have been working on a proposal[1] called Blöcks to introduce transferable functions to JavaScript. In the meantime, I wrote Clooney[2] ontop of Comlink[3] that gives you almost that. [1]: https://github.com/domenic/proposal-blocks https://github.com/domenic/proposal-blocks [2]: https://github.com/GoogleChromeLabs/clooney https://github.com/GoogleChromeLabs/clooney [3]: https://github.com/GoogleChromeLabs/comlink https://github.com/GoogleChromeLabs/comlink
- ajkjk 7y ago
  Oo, I'm really interested that you're working on that, although I don't love the syntax (particularly `worker<endpoint>` seems awkward). It seems like you should stick with function invocation syntax: `worker = (endpoint) => {| block |}` `worker(endpont) // does what you want` with the difference that the result isn't a closure. As a bonus this syntax would let you write 'pure functions' even if you weren't working with workers. Perhaps the worker version then is `worker = async (endpoint => {| block... |}`. Although considering the precedence for `async function` maybe the way to do this ought to be `pure function`.
  - dassurma 7y ago
    If you take a look at the issues on the repo, a lot of people agree with you. The syntax was not final at all, we need to convince TC39 first before we can start bikeshedding ^^
- dbetteridge 7y ago
  Web workers would make a good http203 episode, uses/pitfalls etc
mstade 7y ago
I don't actually know the answer, but my guess would be it's because of scoping. Web workers aren't a language level feature, so they can't really dictate that the function body can't have access to the outer scope which functions in JS typically do. And if worker functions like that would have access, how do you ensure thread safety? Making it so you have to point to a different module altogether seems like a decent way to enforce that boundary.
deleted 7y ago
[deleted]
rl3 7y ago
>Can someone explain why the ES standard doesn't have something like this: new Worker(function(params) { ... compute heavy stuff here ... }); In 2015 I worked on a framework that functioned like this as a personal project. It was even context-aware and spun itself up differently depending on the environment it was in; workers and shared workers loaded the same .js file that the UI thread did. As such, it chose speed over memory use seeing that the code was effectively pre-loaded everywhere. It was never completed, and the reasons for that were: 1. SharedArrayBuffer hadn't landed yet. Transfer cost of computable data between workers was high, and transferables had shortcomings of their own. 2. Closures were disallowed, and AST transformations were one possible solution. It seemed like a very bad idea to go down that road. 3. Back then the differences in API behavior between even Chrome and Firefox were terrible to deal with. The APIs were heavily neglected across the board. These days, I don't see the need. If you need the performance, WASM is a better choice anyways and depending on what you're doing it can use SharedArrayBuffer much more efficiently via way of something like pthreads.
- millstone 7y ago
  But wasm isn’t asynchronous right? It still blocks the main thread.
- kbumsik 7y ago
  > WASM is a better choice anyways and depending on what you're doing it can use SharedArrayBuffer much more efficiently via way of something like pthreads. I think you misunderstand the concept of WASM. WASM does not have its own thread, it blocks the main thread and runs in the same context as JS. WASM still must spawn a Web Worker to emulate multithreading in a browser.
  - rl3 7y ago
    I do understand, was just mobile and in a hurry so I failed to fully qualify the statement. What I was saying is if you're in the WASM toolchain/ecosystem, something like pthreads would already be implemented by Emscripten[0]. In other words, spinning up web workers is handled for you seamlessly. It's all dealt with at a very low level of abstraction. You're literally using the SharedArrayBuffer as memory. Contrast that to JS, and you have to serialize/deserialize types before they can even be stored in SharedArrayBuffer. It's costly and far from seamless. As an aside, it's unfortunate SharedArrayBuffer was hit so hard by Spectre. The poor standard is cursed or something. [0] https://emscripten.org/docs/porting/pthreads.html https://emscripten.org/docs/porting/pthreads.html
apatheticonion 7y ago
This is actually similar to how it's done in Go Imagine you define a function: func hello(value string) { // compute heavy stuff } To run it sync hello("World") To run it async, just add 'go' go hello("World") IMO when it has generics, it'll be the best language I have worked with.
franciscop 7y ago
Since you seem not to like Clooney, what about uwork? I wrote it a while ago: https://github.com/franciscop/uwork https://github.com/franciscop/uwork const findPi = uwork((iterations = 10000) => { let inside = 0; for (var i = 0; i < iterations; i++) { let x = Math.random(), y = Math.random(); if (x * x + y * y <= 1) inside++; } return 4 * inside / iterations; }); // Run this inside an async context: const pi = await findPi(200000000);

zawerf 7y ago

I don't really publish libraries so I didn't run into your main gripe. Is there any other reason why you didn't like worker-loader? I experimented with it and found it pretty straightforward. For anyone who wants to try it, you can even get it working from a create-react-app project with the webpack inline loader syntax without ejecting: /* eslint import/no-webpack-loader-syntax: "warn" */ import Worker from 'worker-loader!./Worker.js'; My use case was to run tensorflow.js in the background. This actually mostly worked out of the box with only some awkwardness with serializing images as messages. The only thing that didn't work out was that you don't have access to webgl there which defeated my original use case (since there's no point in offloading computation if that computational is going to be several orders slower in the background).

paulryanrogers 7y ago
Would it be possible to draw off screen or canvas somehow? To preserve WebGL compatibility while still using the workers?
- Veedrac 7y ago
  https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCa... https://caniuse.com/#feat=offscreencanvas https://caniuse.com/#feat=offscreencanvas
kbumsik 7y ago
Because if you use it in a library you immediately create a hard dependency to Webpack. Libraries should be designed to work with any bundlers, or even without a bundler at best. Besides, I personally don't feel good with Webpack's do-everything-with-import approach. It is straightforward of course, but is ES6 import statement supposed be used like this?

moron4hire 7y ago

I did a thing where I used data URIs instead of explicit files to load up web workers. Don't know if that still works, as this was a few years ago, but it definitely helped a lot with those sorts of issues at the time.

kbumsik 7y ago
Yes, it is still one way to workaround the problem. For those who don't know: function workerFn() { ...Do worker stuff } let worker = new Worker(URL.createObjectURL(new Blob(['('+workerFn.toString()+')()']))); Though this hack works but it still have problems to be addressed: 1. It still does not resolve import/require(). This introduces another hack that dumping output of a bundler to workerFn. 2. It will make it much harder to debug since you create worker out of data, not source code.