4 ms·

Looks promising and I find the Deno project very exciting! What is the recommended way of running Deno in production on a multi core machine? In node there is

Looks promising and I find the Deno project very exciting! What is the recommended way of running Deno in production on a multi core machine? In node there is the cluster module that allows one process to run a cpu intensive task while the other processes to continue answering web requests. This is good for stuff like excel exports as an example on larger datasets. Obviously you could run some kind of cronjob in the background but it seems like I may a bit confused about Denos architechture regarding this subject. I am afraid of switching to Deno because of this since I often want to do these kind of operations without having to do the hassle of breaking the app up for simple exporting/importing operations. Also, will Deno really utilize other cores on the machine? It seems like if you have a machine with multiple cores Deno won't use the full potential of the hardware or am I just confused about this subject?

tehbeard 5y ago
Deno does expose the "Web Workers" standard as a way to utilize multiple cores, where communication between them is handled by postMessage(...). However, afaik, things like sockets can't be passed through (something node can do iirc). There is a mechanism for this in the web workers spec; Transerables, that's part of the web API spec and is used to allow passing things like a large ArrayBuffer from one worker to another without needing to copy it by removing access from the sender and giving it to the receiver. So you would have to plumb the socket into postMessage for a worker, or, for the excel example, make use of promises/async+await to spin up a worker, feed through the min. data needed to kick off the export, let it process in the worker and then send the response back to the main thread.
- staticelf 5y ago
  Yes I am aware of that and sure it can be used for export/import like I described. But it won't answer the second part of my question, about Deno not utilizing all cpu cores on a multicore machine? If I use a web worker it will be for one specific thing and that thing only, not for the entire web request? Thus, if there is anything slowing down the request in form of CPU intensity, it will "kill" the application at least until the cpu intensive request completes. Sometimes it's hard to know beforehand what users do for example and surprising things can happen. I have had the same issues with Meteor for example where users killed the app because they tried to upload something unexpected that crashed the app and them spamming the upload button kept it crashing after restarts. It was of course an error from my part but it could be avoided if Meteor did support the cluster module. I have also had the exact same issue when unexpected large datasets occurred by user input (not file uploads), the app got so slow it stopped answering requests from other sources. It was after that I discovered that pm2 can run any node app in a clustered mode, which was very nice. But with that experience in my baggage, it am very hesitant to use Deno because of this reason. I can't host my app on a serverless platform because reasons and I am not sure I even want to even if I could.
  - tehbeard 5y ago
    Really what your asking is a framework level query rather than what low level primitives the runtime provides. Demo itself won't use all the cores for some JS you just slap in, that's not how JS works (it'll use the same as v8, which will be one main thread for the JS to be processed in and potentially several helper threads for IO etc) If you want multi core you either build it yourself using the web workers or find a framework that does it for you.
    - staticelf 5y ago
      Sure ok, but I don't see how that is possible if it holds the port busy? You can't start two deno processes on the same port number.
deleted 5y ago
[deleted]

tehbeard 5y ago

Deno does expose the "Web Workers" standard as a way to utilize multiple cores, where communication between them is handled by postMessage(...). However, afaik, things like sockets can't be passed through (something node can do iirc). There is a mechanism for this in the web workers spec; Transerables, that's part of the web API spec and is used to allow passing things like a large ArrayBuffer from one worker to another without needing to copy it by removing access from the sender and giving it to the receiver. So you would have to plumb the socket into postMessage for a worker, or, for the excel example, make use of promises/async+await to spin up a worker, feed through the min. data needed to kick off the export, let it process in the worker and then send the response back to the main thread.

staticelf 5y ago
Yes I am aware of that and sure it can be used for export/import like I described. But it won't answer the second part of my question, about Deno not utilizing all cpu cores on a multicore machine? If I use a web worker it will be for one specific thing and that thing only, not for the entire web request? Thus, if there is anything slowing down the request in form of CPU intensity, it will "kill" the application at least until the cpu intensive request completes. Sometimes it's hard to know beforehand what users do for example and surprising things can happen. I have had the same issues with Meteor for example where users killed the app because they tried to upload something unexpected that crashed the app and them spamming the upload button kept it crashing after restarts. It was of course an error from my part but it could be avoided if Meteor did support the cluster module. I have also had the exact same issue when unexpected large datasets occurred by user input (not file uploads), the app got so slow it stopped answering requests from other sources. It was after that I discovered that pm2 can run any node app in a clustered mode, which was very nice. But with that experience in my baggage, it am very hesitant to use Deno because of this reason. I can't host my app on a serverless platform because reasons and I am not sure I even want to even if I could.
- tehbeard 5y ago
  Really what your asking is a framework level query rather than what low level primitives the runtime provides. Demo itself won't use all the cores for some JS you just slap in, that's not how JS works (it'll use the same as v8, which will be one main thread for the JS to be processed in and potentially several helper threads for IO etc) If you want multi core you either build it yourself using the web workers or find a framework that does it for you.
  - staticelf 5y ago
    Sure ok, but I don't see how that is possible if it holds the port busy? You can't start two deno processes on the same port number.

deleted 5y ago

[deleted]