4 ms·

As someone who will shortly need to do two very compute intensive functions in the browser (filtering large datasets (100k+ rows) via client specified queries,

As someone who will shortly need to do two very compute intensive functions in the browser (filtering large datasets (100k+ rows) via client specified queries, and parsing large sting blobs) what is my best option? I’m aware of server side rendering but I suspect the performance in the client may be better responsiveness, with a failover to the server if it’s faster to provide a query response, but is there anything I should _really_ consider for client JS performance at that level?

ficklepickle 5y ago
A WebWorker can free up the main thread so the UI doesn't grind to a halt. Unfortunately they might not help you here. Data in/out of the worker is string only[0], so the extra parsing/serializing usually ruins any perf benefit. You can access indexedDB from the worker, though. So you load the DB from the main thread, then you can do the filtering from the worker. You could return indexes then fetch those from indexDB on the main thread. None of this will help with parsing large strings though, you are probably stuck with doing that on the main thread. And it is also not going to reduce the total time to perform the work. But hopefully it will avoid pinning the main thread, because that is bad. I really enjoy this stuff, so feel free to contact me if you wanna talk it through. My email address is in my profile. [0] SharedArrayBuffers were recently re-enabled according to https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
- markdog12 5y ago
  > Data in/out of the worker is string only[0] I don't think this is correct. You can pass lots of types in via postMessage(), including Objects, ArrayBuffers, even a Canvas context. Some objects are transferrable as well, that is, zero copy. As far as your point about still being stuck on the main thread, that may indeed be the case depending on the work. More info: https://stackoverflow.com/questions/19152772/how-to-pass-large-data-to-web-workers https://stackoverflow.com/questions/19152772/how-to-pass-lar...
- hsbauauvhabzb 5y ago
  Thanks, I’ve got some other work to kick off first but I’ll reach out when I have time!
csande17 5y ago
For the specific case of doing a map or filter operation on a really huge list, I've found it helpful to yield the main thread every once in a while with a setTimeout(0) call. Usually this means setting your code up so it can do the processing in batches of, say, 10K items each. That way, you can do your processing without locking up other UI stuff on the page (like scrolling or animated loading spinners).

ficklepickle 5y ago

A WebWorker can free up the main thread so the UI doesn't grind to a halt. Unfortunately they might not help you here. Data in/out of the worker is string only[0], so the extra parsing/serializing usually ruins any perf benefit. You can access indexedDB from the worker, though. So you load the DB from the main thread, then you can do the filtering from the worker. You could return indexes then fetch those from indexDB on the main thread. None of this will help with parsing large strings though, you are probably stuck with doing that on the main thread. And it is also not going to reduce the total time to perform the work. But hopefully it will avoid pinning the main thread, because that is bad. I really enjoy this stuff, so feel free to contact me if you wanna talk it through. My email address is in my profile. [0] SharedArrayBuffers were recently re-enabled according to https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

markdog12 5y ago
> Data in/out of the worker is string only[0] I don't think this is correct. You can pass lots of types in via postMessage(), including Objects, ArrayBuffers, even a Canvas context. Some objects are transferrable as well, that is, zero copy. As far as your point about still being stuck on the main thread, that may indeed be the case depending on the work. More info: https://stackoverflow.com/questions/19152772/how-to-pass-large-data-to-web-workers https://stackoverflow.com/questions/19152772/how-to-pass-lar...
hsbauauvhabzb 5y ago
Thanks, I’ve got some other work to kick off first but I’ll reach out when I have time!

csande17 5y ago

For the specific case of doing a map or filter operation on a really huge list, I've found it helpful to yield the main thread every once in a while with a setTimeout(0) call. Usually this means setting your code up so it can do the processing in batches of, say, 10K items each. That way, you can do your processing without locking up other UI stuff on the page (like scrolling or animated loading spinners).