2 ms·
I implemented Gearman at a previous startup and then recently have done a "clean room" re-implementation of it for a personal project. First: Gearman is foremo
by encoderer 16y ago
I implemented Gearman at a previous startup and then recently have done a "clean room" re-implementation of it for a personal project.
First: Gearman is foremost a job quque, as others have mentioned. Obviously most jobs have SOME data submitted along with them.
Second: After implementing it as a job queue, another team down the road needed a message queue. So we ran benchmarks. Turns out, Gearman was a faster Message Queue than the dedicated *MQ platforms. We were getting killer performance out of the Gearman Java client. Of course, we had to spend 2 weeks fixing bugs on the Java client to get it to work.
Which brings me to my last comment... Gearman is NOT a mature system. It's brilliantly fast and very good at what it does, when it all works. But you may find yourself, like I did, getting personal with the Gearman code itself to explain wonky behavior. Like, hey, why the heck isn't this job timeout being enforced? Fail.
But as for being awesome, let me give you a taste:
It has a MySQL client. So on our reporting database servers, we installed the gearman client. Which meant we could create UDFs in MySQL that would be actually backed-out by Gearman. In other words... we could do something like this:
Select user_id, UserRiskScore(user_id) from users where user_id = 123;
And UserRiskScore kicks off a Gearman job... which in this case could be just an entry point for a Gearman map-reduce: it dispatches many other jobs, aggregates the result, and then returns it to MySQL and your waiting query.
And, of course, the actual worker code can be written in languages far more useful than SQL.
Now, I'd never install a system like that on the live database. You don't need 200 queries waiting on a gearman response when you've got buckets of data innodb is trying to flush to disk.
But on our reporting db cluster it was brilliant. It made our code wonderfully centralized. A FRONT-END coder can get complex results, having kicked off a map-reduce job without ever having to know what, say, HDFS, or MongoDB, really is.