3 ms·
Unrelated question: how are you doing those cron jobs. I tried using the task queue, but it kept timing out on me, even when I broke my tasks into smaller ones
by bad_user 16y ago
Unrelated question: how are you doing those cron jobs.
I tried using the task queue, but it kept timing out on me, even when I broke my tasks into smaller ones (couldn't figure out a threshold as the behavior of the datastore was totally unpredictable).
- madflo 16y agoVia Cron Jobs ? http://code.google.com/appengine/docs/python/config/cron.html http://code.google.com/appengine/docs/python/config/cron.htm...
- bad_user 16y agoQuote from that article: A URL invoked by cron is subject to the same limits and quotas as a normal HTTP request, including the request time limit. And I'm looking for some tips and tricks to do heavy processing (which is really needed when working with non-relational datastores).
- mshafrir 16y agoThe key is to combine cron jobs with Tasks. Tasks are am easy to use and powerful feature of Google Appengine that let you deal with the standard limitations. For example, rather than doing some operation on 1000 entities in one request, you spin off 1000 tasks, each handling the operation on one entity. Read up about them at http://code.google.com/appengine/docs/python/taskqueue/ http://code.google.com/appengine/docs/python/taskqueue/ You can also explore the new Mapper API at http://blog.notdot.net/2010/05/Exploring-the-new-mapper-API http://blog.notdot.net/2010/05/Exploring-the-new-mapper-API
- mmastrac 16y agoAgreed, the Mapper API is the best way to deal with processing large numbers of entities from the datastore. AppEngine requires careful thought of application design at this point. Everything is an HTTP request and subject to the same restrictions. I've heard rumours that they'll be launching longer-lived tasks at some point in the future, but I have no idea when that will happen.
- mmastrac 16y agoWe have to break our cron jobs down into single datastore queries for them to succeed near 100% of the time. If you want more complex logic, use a cron job to trigger tasks that process smaller bits of your operation. Even our cron job to compute the top users on our site (a simple query + ordering) uses one second of CPU time between the application and the datastore. It's just something you get used to in AppEngine.