welder

29 ms·

361.
by welder 4y ago
I built a local test app that does have some integration tests, and then I just went all in on production. That's why I was surprised it worked so well. I was ready to lose a few nights sleep debugging it in prod, but thankfully didn&#
362.
by welder 4y ago
Why would there exist a way in Python to conditionally import types, for the purpose of preventing cyclic imports, if cyclic imports weren't a problem? Your comment makes it seem like you haven't experienced Python types enough, o
363.
by welder 4y ago
WakaQ re-forks workers that crash, but normally a crash means an exception was raised in application task code. I do a combination of a few things to handle exceptions in tasks: 1. `@wakaq.task(max_retries=X)` auto retries the task on soft
364.
by welder 4y ago
When workers claim tasks, TaskTiger moves tasks to a new queue instead of popping them like WakaQ. That means if your task fails with WakaQ then it's just gone. > how it compares in benchmarks WakaQ processes many tasks per fork, wh
365.
by welder 4y ago
> Stating "WakaQ is stable and ready to use in production" is irresponsible I've changed it to "WakaQ is still a new project, so use at your own risk."
366.
by welder 4y ago
Correct, this setup wouldn't survive chaos monkey.
367.
by welder 4y ago
Sorry about the confusion. Btw, we've met in person once in Mountain View some years back. Nice to see you here :)
368.
by welder 4y ago
Oh sorry, somehow I thought we were talking about pubsub.
369.
by welder 4y ago
Yes, I don't know any distributed task queue that stays online or backs up tasks when the broker dies. Maybe one could be built on CockroachDB? My implementation is designed to process tasks fast and if the Redis server dies (hasn&#x27
370.
by welder 4y ago
New: Streams are append only, but I push eta tasks onto the front of a queue when their time is ready to run. Looks like streams could work and might even be better suited instead of lists, but I would have to figure out some way to run eta
371.
by welder 4y ago
It's being used in production at https://wakatime.com and performing better than Celery was, but yes it's an internal project that was open sourced early stage.
372.
by welder 4y ago
Yes, if you're accepting the type as a param or returning the type. It happens very frequently, and without moving those imports behind "if typing.TYPE_CHECKING", you constantly run into cyclic imports. https://sta
373.
by welder 4y ago
https://github.com/wakatime/wakaq/blob/main/wakaq/__init__.p... and https://github.com/wakatime/wakaq/blob/main/wakaq/worker.py is the meat of it. The blog
374.
by welder 4y ago
> i made one as well called SAQ! https://github.com/tobymao/saq Awesome! I'll check out your implementation. > Looks like you use BLPOP, but there could be race conditions or lost jobs if your workers crash,
375.
by welder 4y ago
Goes away like you delete your Redis VM or restarting the Redis server? Redis persists to disk so your tasks aren't lost.
376.
by welder 4y ago
Tests? We don't need no stinking tests! [1] I didn't plan to open source this so quickly, if ever. I'm just happy it's working for WakaTime. The makefile has plans for pytest, but honestly my motivation for tests has wan
377.
by welder 4y ago
Distributed means a central broker (Redis) and many distributed workers (multi-process and multi-machine). It's the term used by Celery/RQ/Huey and others.
378.
by welder 4y ago
It does have some typehints... but I don't like Python types when they require importing the object only for the purpose of types. I only import objects when they're actually used in the non-type code, and I don't like to use
379.
by welder 4y ago
1. That's the part I'm not happy with, but currently I use the `@wakaq.after_worker_started` decorator to setup an error logging handler in each worker. It outputs to a file that gets aggregated for error reporting, but without ex
380.
by welder 4y ago
> Say I start two identical workers that have the same schedules set up. Workers don't run the scheduled tasks, only a scheduler does. You should start only one scheduler but as many workers as you want. wakaq-scheduler and wakaq-wo
381.
by welder 4y ago
Maybe I should have used Dramatiq instead of building my own, but it's already done. I'm surprised how well WakaQ is working in production actually. If any issues come up they should be faster to fix than making PRs to Dramatiq.
382.
by welder 4y ago
I was initially excited when Dramatiq was released, but I think the GNU license prohibits using it for commercial projects? Also, I really wanted something simple and easy to maintain after working around so many bugs. Dramatiq isn't a
383.
Show HN: WakaQ - a Python distributed task queue(wakatime.com)
147 points by welder 4y ago|68 comments
384.
by welder 4y ago
Wish it could filter to show cities or towns instead of middle of nowhere back roads.
385.
by welder 4y ago
From the 2 marketplace links you linked above, click "Download Extension" on the right side of the page. Rename that file from .vsix to .zip then extract the zip file to get a folder containing the source files for the extension.
386.
by welder 4y ago
Just download it without installing it.
387.
by welder 4y ago
Do you own the home you live in or are you renting? Renting is enabled by doing just that, someone owning a home they don't live in. So you want to outlaw renting?
388.
by welder 4y ago
Sorry, I meant you could see the source code not the source code's license.
389.
by welder 4y ago
vscode extensions are open source so the community should report and vscode should ban extensions behaving badly. That's the same way all plugin marketplaces work, nothing bad vscode is doing with their extension setup.
390.
by welder 4y ago
That's kinda the point! For ex, my wakatime extension sends metrics about vscode usage so the developer can see their code stats.

More ›