5 ms·
Piscina – The Node.js Worker Pool
- freeqaz 6y agoDoes anybody use this or anything similar? If so, what problems are you solving?
- deleted 6y ago[deleted]
- barefeg 6y agoI’m guessing the fact that node is single threaded
- nobleach 6y agoJavaScript is single threaded, Node is most certainly not.
- gbrits 6y agoYes it surely is
- zbentley 6y agoMy "htop" display surely disagrees. So do my node programs that lag out waiting for disk IO thread pool slots. Sounds does node's documentation: https://nodejs.org/api/worker_threads.html https://nodejs.org/api/worker_threads.html
- hmcdona1 6y agoIt's not exactly. Certain operations like I/O are threaded in Node. libuv has it's own threadpool that it uses for a lot of these types of tasks.
- jayflux 6y agoNode uses a thread pool for tasks like reading files async and crypto operations etc, you can read about it here https://nodejs.org/en/docs/guides/dont-block-the-event-loop/ https://nodejs.org/en/docs/guides/dont-block-the-event-loop/
- curben 6y agoAFAIK this is a wrapper of worker_threads API. https://github.com/tuananh/camaro https://github.com/tuananh/camaro utilises this for multi-threading processing of XML input. Hexo (static site generator) is attempting to utilise this library (https://github.com/hexojs/hexo/issues/4355 https://github.com/hexojs/hexo/issues/4355).
- maxrev17 6y agoMight be quite nice for keeping tokens refreshed?
- Etheryte 6y agoThe worker API in Javascript is quite a pain to use, but needless to say multithreading is invaluable in many contexts, both in the browser and in Node. I haven't used this library but it seems to solve a similar problem to other ones in the same space — make writing multithreaded code sane, allowing you to avoid writing a bunch of repetitive boilerplate.
- Skhalar 6y agoThis is useful for processing large chunks of data like audio files (look at Superpowered sdk) but breaking em down or when processing multiple files.
- ddoolin 6y agoWe use worker threads directly to process large unorganized (for the browser) datasets and do some deductions before it hits the store. We also have a worker thread blocked on a redis channel that acts as a queue.
- niftylettuce 6y agoYeah. I built Bree @ https://jobscheduler.net https://jobscheduler.net It has support for worker threads + cron + human strings GH: https://github.com/breejs/bree https://github.com/breejs/bree
- ddoolin 6y agoThis is cool. Our use case was very simple so we went with node-schedule but this may have more upsides. I've had trouble with the former. Thanks.
- niftylettuce 6y agoI helped maintain all the Node.js node/cron/agenda libraries. Just switch to Bree and your troubles will be gone.
- rektide 6y agoKnex, the SQL query builder, uses Tarn.js[1] for connection pooling to the db. I've been using Tarn a bunch at work recently. We're doing some batch jobs, and I'm queuing work at each stage in Tarn.js pools. I created my own enqueue function that waits until the pool is less than a high-water mark in size before enqueueing. Then the pool has however many workers running. Neither of these are off-thread pools. But they help a lot of for managing multiple async operations. [1] https://github.com/vincit/tarn.js/ https://github.com/vincit/tarn.js/
- searchableguy 6y agoI use workers in deno to evaluate user facing code which may take a lot of time to finish and need sandboxing. Another place it's used in where I want to keep context and reuse it for subsequent code execution for each user. In deno, you can restrict what the code inside worker can do by passing a map of allowed permissions. I have built a simple privilege system on top of this to allow users different access level. This is cheaper and faster than spinning up container.
- domenicd 6y agoI maintain a command-line utility which assembles an eBook from scraped .xhtml files. It uses a similar package, workerpool, to process multiple chapters in parallel. https://github.com/domenic/worm-scraper/blob/master/lib/convert.js https://github.com/domenic/worm-scraper/blob/master/lib/conv... https://www.npmjs.com/package/workerpool https://www.npmjs.com/package/workerpool
- vorticalbox 6y agoAt work I use https://github.com/pioardi/poolifier https://github.com/pioardi/poolifier We have an api used for generating reports from mongo dB to csv, this let's the report process in the background leaving the report api to still handle requests.
- cjdell 6y agoWhat would be nice is a way of preventing the same job running multiple times concurrently. Like if I start a job and a job with the same parameters was already started milliseconds ago then it automatically awaits the already running job rather than starting another.
- sbarre 6y agoYou could build a reasonably lightweight supervisor pattern that uses a parameter-derived hash for comparison to handle this kind of situation in your application too. Might be easier and more flexible than asking the library to do it?
- tomwojcik 6y agoThat's exactly what I do in python with celery. I order all dicts/lists and hash this final object. If hash is not found in the db, I set hash as the task id. With `track started` setting I never run the same task twice.
- fendy3002 6y agoIf you can use redis, redlock can be a nice implementation. Example: https://github.com/fendy3002/NodeLearn/blob/master/Redlock/src/index.ts https://github.com/fendy3002/NodeLearn/blob/master/Redlock/s...
- tekno45 6y agoAdd unique job id?
- hfktk4nrn 6y agoIs it just me, or do I see a trend in naming projects using romance language words (Italian/Spanish/France)? Does it sound more exotic? Are these words less crowded?
- andreynering 6y agoI'm curious on why you call these languages "romance" languages. "Piscina" is a Portuguese word for "pool", by the way.
- age008 6y agoTo quote Wikipedia, "Romance languages (less commonly Latin languages, or Neo-Latin languages) are the modern languages that evolved from Vulgar Latin between the third and eighth centuries." "Piscina" is also "pool" in Romanian, which is, you guessed, a Romance language.
- andreynering 6y agoGot it. In Portuguese they're only known for "Latin languages", which explains my question. ;-)
- mayank 6y ago> I'm curious on why you call these languages "romance" languages. That's what they're called: https://en.wikipedia.org/wiki/Romance_languages https://en.wikipedia.org/wiki/Romance_languages
- 29athrowaway 6y agoCommon mistake in stream code: -- .on('end') ++ .once('end')
- revskill 6y agoCould i use it on a serverless platform ?
- timmit 6y agoI assume u can, but how it performs, it really dependents on the virtual CPU of the, platform. I did something similar https://github.com/tim-hub/pambdajs https://github.com/tim-hub/pambdajs but I haven't done the comparison on aws lambda yet
- timmit 6y agou remind me my own researching project too. https://github.com/tim-hub/pambdajs https://github.com/tim-hub/pambdajs
- ericlewis 6y agoNeat! I’m friends with the creator of this and teased him a bit about the name (so many of these projects have weird names now a day) The reason is: don’t wanna be boring from what I could glean.
- kevinstubbs 6y ago"piscina" in Italian means "pool". A library for worker pools simply named "pool" in Italian doesn't seem that strange :)
- hakcermani 6y agoPiscine Molitor Patel in the Life of Pi was named after the pool in Paris !
- chmod775 6y agoHere's my issue with this kind of "magic" API design: It's not clear when or how serialization happens. Supposedly at some point the message I am sending to the worker is serialized, but this is not made clear to the user and it's not made clear when this happens. A well-designed library would synchronously serialize the given object the moment it is passed in or let the user explicitly handle serialization. But I don't think that's what is happening here. It appears messages are serialized only eventually and when they are finally sent off to a worker. If you accidentally pass mutable state in here, you're in for a really confusing and fun debugging session. Likely it'll be a production-only bug too, because during development and testing you're unlikely to have the kind of message volume required to run into some modified-before-sent condition. CTRL-F "mutable" and CTRL-F "serialize" gives no results, so I don't think the designers thought of this or thought to warn users.
- nimbix 6y agoI definitely ran into some serialization issues when trying Piscina a while ago. Returning generated images from the workers would keep getting slower and slower until the overhead of data exchange was 20x the time it took to generate it. I discovered through trial and error that returning image bytes directly would incur this ever increasing overhead, btu creating a binary buffer and returning that would not. It was just a pet weekend project, so I never did discover what was causing the issue.
- lyjackal 6y agoInteresting that it's written in typescript and the readme doesn't mention using it with typescript. I ran into annoyances with this recently trying out nodes worker threads in a typescript project. I was running with ts-node, but the worker thread didn't know how to load a typescript file. There are some workarounds but they're not elegant.