5 ms·

Why we aren’t using the Gmail API (yet)

16 points by saryant 12y ago

jessaustin 12y ago
Would they have preferred to just have someone at Google to contact about this?
- toomuchtodo 12y ago
  You can speak to a human at Google regarding their API limitations? I'd be shocked if that was the case.
  - esquid2 12y ago
    One can request an increase in quota as it says here: https://developers.google.com/gmail/api/v1/reference/quota https://developers.google.com/gmail/api/v1/reference/quota
  - jessaustin 12y ago
    Perhaps my comment was a bit opaque... b^)
    - saryant 12y ago
      Not sure if it was you, but someone from Google did reach out to us. If it was you, I'll be replying this morning once I get to the office! If not, thanks anyways!
      - jessaustin 12y ago
        No I'm not at Google. Congratulations on finding a contact, and good luck with a resolution for the issue you describe. It might be interesting if you did a followup after the whole process plays out.
        esquid2 12y ago
        I work at Google. We don't typically monitor HN, reddit, twitter, etc but we have plenty of people watching Stack Overflow (the one we direct people to from our docs). The default quotas are conservative during this Beta period but we're happy to increase when needed--the service is certainly not provisioned as high as IMAP at this point. Submitting the quota increase request (from the developers console) will send us an email and we'll either approve it or if we need to, we'll start an email thread to ask for more info/suggest improvements. Hope that helps!
amhoab 12y ago
I wonder if they use IMAP's EXAMINE or SELECT commands when iterating through folders. The former is read-only, although the end user has no control or guarantee that this is how they'll go through folders.
esquid2 12y ago
Did you ever try to contact someone at Google for higher quota? (There's a form linked from the developer site.) I'm not surprised the default quotas were low, it is still Beta...
- saryant 12y ago
  Our primary issue isn't the quota limits, it's the actual speed of the API. The best we were able to achieve with the Gmail API was 3000 messages/minute whereas IMAP easily lets us get 20k messages/minute.
  - esquid2 12y ago
    Are you limiting yourself to one thread or something? If you're not getting limited by quota then why not use batching + more threads until you do get to those limits?
    - saryant 12y ago
      We are getting limited by the quota. As the documentation says, quotas can be temporarily exceeded and in our tests we could fetch about 3000 messages in those bursts. After that, we would be 403'd for one to two minutes before requests would succeed again with quotas strictly enforced for an unknown amount of time past that.
sportanova 12y ago
How did you make javamail fast? My impression was that you could batch request meta data info, but not the actual body. Gmail's been way faster for me
- saryant 12y ago
  We don't request message bodies, just headers.
  - sportanova 12y ago
    gotcha - thanks!
  - esquid2 12y ago
    When you do your tests are you using the new-ish message.get(format=METADATA)?
    - saryant 12y ago
      No, I wasn't aware of that option. In our tests we did list requests for 100 messages and then bulk requests for each messages' headers (messages.get("me", message.getId).setFields("historyId,payload/headers")). I'll look into that option and re-run the tests.

16 points by saryant 12y ago

jessaustin 12y ago

Would they have preferred to just have someone at Google to contact about this?

toomuchtodo 12y ago
You can speak to a human at Google regarding their API limitations? I'd be shocked if that was the case.
- esquid2 12y ago
  One can request an increase in quota as it says here: https://developers.google.com/gmail/api/v1/reference/quota https://developers.google.com/gmail/api/v1/reference/quota
- jessaustin 12y ago
  Perhaps my comment was a bit opaque... b^)
  - saryant 12y ago
    Not sure if it was you, but someone from Google did reach out to us. If it was you, I'll be replying this morning once I get to the office! If not, thanks anyways!
    - jessaustin 12y ago
      No I'm not at Google. Congratulations on finding a contact, and good luck with a resolution for the issue you describe. It might be interesting if you did a followup after the whole process plays out.
      - esquid2 12y ago
        I work at Google. We don't typically monitor HN, reddit, twitter, etc but we have plenty of people watching Stack Overflow (the one we direct people to from our docs). The default quotas are conservative during this Beta period but we're happy to increase when needed--the service is certainly not provisioned as high as IMAP at this point. Submitting the quota increase request (from the developers console) will send us an email and we'll either approve it or if we need to, we'll start an email thread to ask for more info/suggest improvements. Hope that helps!

amhoab 12y ago

I wonder if they use IMAP's EXAMINE or SELECT commands when iterating through folders. The former is read-only, although the end user has no control or guarantee that this is how they'll go through folders.

esquid2 12y ago

Did you ever try to contact someone at Google for higher quota? (There's a form linked from the developer site.) I'm not surprised the default quotas were low, it is still Beta...

saryant 12y ago
Our primary issue isn't the quota limits, it's the actual speed of the API. The best we were able to achieve with the Gmail API was 3000 messages/minute whereas IMAP easily lets us get 20k messages/minute.
- esquid2 12y ago
  Are you limiting yourself to one thread or something? If you're not getting limited by quota then why not use batching + more threads until you do get to those limits?
  - saryant 12y ago
    We are getting limited by the quota. As the documentation says, quotas can be temporarily exceeded and in our tests we could fetch about 3000 messages in those bursts. After that, we would be 403'd for one to two minutes before requests would succeed again with quotas strictly enforced for an unknown amount of time past that.

sportanova 12y ago

How did you make javamail fast? My impression was that you could batch request meta data info, but not the actual body. Gmail's been way faster for me

saryant 12y ago
We don't request message bodies, just headers.
- sportanova 12y ago
  gotcha - thanks!
- esquid2 12y ago
  When you do your tests are you using the new-ish message.get(format=METADATA)?
  - saryant 12y ago
    No, I wasn't aware of that option. In our tests we did list requests for 100 messages and then bulk requests for each messages' headers (messages.get("me", message.getId).setFields("historyId,payload/headers")). I'll look into that option and re-run the tests.