3 ms·

Ask HN: What are ways to schedule tasks for the future? (web dev)

2 points by ge96 10y ago

In this particular case it's for a tracking system. Namely what I want is a non-performance affecting, for example I may want to gather the ip, time, what a person did (what they clicked on) but instead of doing all of that. You'd just say "Hey task manager, this person did this" but not do all the computation that goes into your database eg. inline in your server-side scripting or multiple calls from the front end. What technology am I looking for here? I currently use the LAMP stack but am open to other solutions as I have to evolve eventually to keep up. To further elaborate: To do this right now I would have to create several inserts in the database that would happen when the person clicks on something that I'm tracking. Rather than doing that, I'd do a short thing like sending a JSON string that has the details I need to insert to a background process that then does this insert outside/away from the main part that handles front end use. Thanks for any suggestions. I don't mean crontab either, unless that is relevant.

andyklock 10y ago
Sounds like you need a queue of some sort. What I usually see is something like each web server will dump these details (json is fine) into a file. Then at some scheduled time these details can be inserted into your database in batch. (nightly, hourly, etc)
- ge96 10y ago
  Does that suggest Cron? I mean I'm looking for something like a background running process (CRON) or Daemon haha. But for some reason I'm thinking of something like Node. I don't know.
  - andyklock 10y ago
    It could be cron, but mostly I'd just recommend using whatever you are comfortable supporting. Cron is great for simple batch jobs (I would imagine this would fall into that category). Cron isn't ideal when you have to chain a bunch of jobs together. But, if you have a script that loads files, cron could do the job easily enough.
    - ge96 10y ago
      I actually figured out an alternative for this particular case. I would create a client-side JavaScript object and store stuff there, then when they leave, it makes one call to the server with the object passed as JSON. Problem then becomes cleaning the output. I was just thinking, if you can edit stuff in the browser console, can't you edit the contents of an array? I guess actually you wouldn't see that displayed in the code, it's a state maintained in browser memory I think. But I'm still considering this potential problem and making a known/limited pattern eg. numbers only separated by a known symbol like a dash to use for exploding/splitting.... anyway will try it out.
tmaly 10y ago
I built something that does this idea. I wanted to be able to do expensive computations, but I did not want the person on the website to wait on them. It is sort of like how the old school craigslist does it not updating a category till it gets around to batch processing it. One example I use it for is image processing of an image that is uploaded. Another is sending an email. And yet another is a ton of database inserts for metrics. It works in the same manner as a bank queue processing one thing at a time, so you can run it on a smaller server and not worry about overloading your system. I plan to open source the code in the next week or two if you are interested. I wrote it in Go, and the job queue part uses beanstalkd
- ge96 10y ago
  Wow, Go. I've been listening to a podcast about that "Go time on Change log" I hear a lot of great stuff about it "2000 servers to 2" for example haha, 25,000 concurrent requests per second or some crazy thing. I would have thought this was a common thing. I am interested thanks for offering however I can't really contribute/possibly even use it as I don't know how to use Go at this time. Good luck though with it, I think Go by the sound of it will be even more popular than it is now. I mean they even talked about hardware-control with Go. It sounds incredible. I'll keep it in mind though but yeah outside of my scope at this time, unless it's not difficult to integrate into what I use right now which is LAMP stack on a single-core VPS. Pretty pathetic I know but I domain map. Probably not related but I have a raspberry pi server with Cron (actually working cron) as well that I look to use for prototyping.

2 points by ge96 10y ago

andyklock 10y ago

Sounds like you need a queue of some sort. What I usually see is something like each web server will dump these details (json is fine) into a file. Then at some scheduled time these details can be inserted into your database in batch. (nightly, hourly, etc)

ge96 10y ago
Does that suggest Cron? I mean I'm looking for something like a background running process (CRON) or Daemon haha. But for some reason I'm thinking of something like Node. I don't know.
- andyklock 10y ago
  It could be cron, but mostly I'd just recommend using whatever you are comfortable supporting. Cron is great for simple batch jobs (I would imagine this would fall into that category). Cron isn't ideal when you have to chain a bunch of jobs together. But, if you have a script that loads files, cron could do the job easily enough.
  - ge96 10y ago
    I actually figured out an alternative for this particular case. I would create a client-side JavaScript object and store stuff there, then when they leave, it makes one call to the server with the object passed as JSON. Problem then becomes cleaning the output. I was just thinking, if you can edit stuff in the browser console, can't you edit the contents of an array? I guess actually you wouldn't see that displayed in the code, it's a state maintained in browser memory I think. But I'm still considering this potential problem and making a known/limited pattern eg. numbers only separated by a known symbol like a dash to use for exploding/splitting.... anyway will try it out.

tmaly 10y ago

I built something that does this idea. I wanted to be able to do expensive computations, but I did not want the person on the website to wait on them. It is sort of like how the old school craigslist does it not updating a category till it gets around to batch processing it. One example I use it for is image processing of an image that is uploaded. Another is sending an email. And yet another is a ton of database inserts for metrics. It works in the same manner as a bank queue processing one thing at a time, so you can run it on a smaller server and not worry about overloading your system. I plan to open source the code in the next week or two if you are interested. I wrote it in Go, and the job queue part uses beanstalkd

ge96 10y ago
Wow, Go. I've been listening to a podcast about that "Go time on Change log" I hear a lot of great stuff about it "2000 servers to 2" for example haha, 25,000 concurrent requests per second or some crazy thing. I would have thought this was a common thing. I am interested thanks for offering however I can't really contribute/possibly even use it as I don't know how to use Go at this time. Good luck though with it, I think Go by the sound of it will be even more popular than it is now. I mean they even talked about hardware-control with Go. It sounds incredible. I'll keep it in mind though but yeah outside of my scope at this time, unless it's not difficult to integrate into what I use right now which is LAMP stack on a single-core VPS. Pretty pathetic I know but I domain map. Probably not related but I have a raspberry pi server with Cron (actually working cron) as well that I look to use for prototyping.