8 ms·

Open Letter to Calendar Developers about HTTP

61 points by bpierre 13y ago

eksith 13y ago
I have a feeling many of these are from rubbish applications that have mostly been abandoned by their developers but still remain installed and running in the background for a lot of users.
- codeka 13y ago
  Except dataaccessd is how iOS's CalDAV sync process identifies itself, so I'd say it's not exactly a "rubbish" application that's been abandoned...
lazyjones 13y ago
What do we learn from this? If you provide a public-facing API or service for direct use by applications over the net, use a unique host name / DNS entry for it so you can point it elsewhere or delete it at leisure. DNS entries are cheap, millions of HTTP hits might cost a bit more.
- shabble 13y ago
  For even less fun, there have been numerous incidents of consumer accesspoints/routers hardcoding NTP servers by IP address[1][2]. [1] http://pages.cs.wisc.edu/~plonka/netgear-sntp/ http://pages.cs.wisc.edu/~plonka/netgear-sntp/ [2] http://web.archive.org/web/20060408150213/http://people.freebsd.org/~phk/dlink/ http://web.archive.org/web/20060408150213/http://people.free...
smackfu 13y ago
I would guess the problem is that most sites are going to serve up a 404 in this situation, and handling 404s is a bit trickier, since they might be transient (like a proxy gets in the way, or the hosting service has temporary issues). So they just don't handle it, since polling a server and getting a 404 is going to work well enough for most use cases. Maybe put a little exclamation point next to the calendar or something, but still poll it. And then they handle a 410 the same as a 404, even though a 410 has very little chance of being transient. It's just rare enough that they don't even realize they don't support it properly.
- Wilya 13y ago
  I'd wager 99% of codes have a big if(200) { // Do stuff } else { // Try again later } Actually, a bunch of them probably don't even check for response code. Edit: If dataaccessd is really the iOS native CALDAV integration, that's a bit annoying. Of all people, they should be able to handle that correctly.
  - pyre 13y ago
    if(200) { // Do stuff } else { // Try again later } That's like an "idiot-filter" interview question. "Is the following code valid?"
    - jimbobimbo 13y ago
      This is a very interesting observation! I imagine, there could be a great conversation derived from this seemingly easy question.
    - Groxx 13y ago
      Seems like it should be paired with a "Is the following code useful?" about code which specifically handles every error code listed here http://en.wikipedia.org/wiki/List_of_HTTP_status_codes http://en.wikipedia.org/wiki/List_of_HTTP_status_codes and contains code to handle unlisted ones. Then "where do you draw the line?". Sometimes if(200){}else{} is plenty.
      - pessimizer 13y ago
        This is a silly objection. Not every code merits specific handling because there are groups of codes that merit the same handling. if(200){...}else{retry} is never plenty.
        Groxx 13y ago
        Assume it only queries your server. Assume your server will only ever meaningfully return 200. Assume the method has no reason to ever move (so no 301 redirect (edit: and redirects are generally handled by your request library anyway)). Assume the only time it should fail is if your server falls over on itself, which should be temporary. What else would you do then? These are not crazy expectations. Exponential backoff might be reasonable, but I'd argue that's part of "else{retry}" since it's inherently retrying.
rav 13y ago
This is probably a case where an exponential backoff would be preferable. When the client is faced with a fetch error the first time, it should wait and try again in e.g. 10 minutes, and double the wait for each repeated error. This way, after a week of downtime at the source, each client would send less than four requests for the resource per month.
- ZoFreX 13y ago
  No, this is a case where obeying the HTTP specification would be preferable. 410 GONE means: Gone. Don't try again.
  - Millennium 13y ago
    Agreed. Incremental backoff makes sense for the 5xx-class errors, where the error is known to be server-side. But 404 is the only 4xx-class error that it makes any sense for: in any other case, it's pretty certain that your side is doing something wrong. Even when you're not -i.e. if you're absolutely certain that this is the right URL for what you want- a 404 should still trigger some kind of notification so that a person can look into it. Using incremental backoff for 410, in particular, is to miss the point entirely. 410 very explicitly means that you should not try again. The fact that it's so rare only lowers the chance that it's being used incorrectly: someone savvy enough to set up a 410 knows enough to mean it. That said, I'm starting to think that if you're going to implement 410 on your site, you should also start logging activity. When a host keeps looking, either incrementally or otherwise, start refusing connections from that host at the firewall level. A program that sets 4xx-class errors to fail silently probably isn't doing anything about refused connections, so an IP-ban should wake the developers up. Even if it doesn't, you won't have erroneous requests clogging your server logs, so you still benefit.
    - vidarh 13y ago
      > The fact that it's so rare only lowers the chance that it's being used incorrectly: someone savvy enough to set up a 410 knows enough to mean it. I don't see how your conclusion follows from your premise. I fairly often come across cases where obscure options are mainly used by those who don't understand them, and in the absence of data I wouldn't assume that the same couldn't be the case for HTTP responses. Exponential backoff is a quite reasonable compromise for those kind of situations.
jstanley 13y ago
I have exactly the same problem with a project I run. I generate iCalendar versions of the timetables for each course at my university and let students download them. The timetables are static and do not change, yet I still get hundreds of hits every day from dataaccessd, even in the middle of summer when nobody has any classes. EDIT: http://notlate.co.uk/ http://notlate.co.uk/ EDIT2: Not that it's enough load to be noticeable, but it's a little wasteful on the part of dataaccessd IMO.
ezxs 13y ago
This is another case for rate limiting
kogir 13y ago
Am I the only person who after about 3 months would have started returning random and numerous events? Then clients would voluntarily unsubscribe.
- biot 13y ago
  Nope. I'd be hard pressed to not do a little iCal trolling. I'm sure the moon would get destroyed by the Death Star at least a few times per year. Not to mention a new phase of the moon: Space station.
- alexwright 13y ago
  "I have a 2PM with 'Goatse.cx', I don't remember this, let me just click the info link. URGHWTF."
- smackfu 13y ago
  > Then clients would voluntarily unsubscribe. Or write nasty blog posts about you that pollute Google results forever.
evmar 13y ago
A long time ago I published an XML file on my website for some ill-fated Google attempt at providing extensible search results. I took the file down some time later, but the Google crawler continued to hit the URL. Amusingly, I asked around within the company to try to find who was responsible for the crawler repeatedly fetching the 404 but ultimately couldn't figure it out. (It seemed like the decommissioned service had maybe registered the URL with some still-running automated refetching service?) It eventually stopped on its own.
MBCook 13y ago
Fetching stuff over HTTP can be incredibly stupid. I there is a security list out there that some little firewall company decided to have their devices automatically download. The list doesn't change that often, but every single device that company made likes to check every 10 minutes or so. Nothing can stop them, they follow Wilya's algorithm [1]. [1] https://news.ycombinator.com/item?id=6268919 https://news.ycombinator.com/item?id=6268919 Side note: it's sad to see that Apple is a huge perpetrator in this one. I hope they fix that bug.
eli 13y ago
Google will continue to request 410 resources for quite a while too. As far as I can tell it treats them the same as 404.
charlesism 13y ago
I'd never even noticed there was such a thing as a 410 until this. And it's not surprising - what incentive is there for a team in the real world with a shipping product to honor it? There's a pretty obvious incentive to ignore it: it takes developer time to implement, and the result of the 'feature' is that your customer may get permanently locked out of a net resource. What happens if a website admin accidentally puts up a 410, or - worse yet - if a previous owner of the domain once put up a 410 for a page with the same URI. I suppose there's an RFC for the code somewhere, but it seems like a bad idea. I pity the tech support agent who has to explain why his/her customer's browser refuses to visit foo.com/bar when everyone else can view it and there's no longer any way of knowing that once, for an afternoon in 1997 the page threw a 410!

61 points by bpierre 13y ago

I have a feeling many of these are from rubbish applications that have mostly been abandoned by their developers but still remain installed and running in the background for a lot of users.

codeka 13y ago
Except dataaccessd is how iOS's CalDAV sync process identifies itself, so I'd say it's not exactly a "rubbish" application that's been abandoned...

lazyjones 13y ago

What do we learn from this? If you provide a public-facing API or service for direct use by applications over the net, use a unique host name / DNS entry for it so you can point it elsewhere or delete it at leisure. DNS entries are cheap, millions of HTTP hits might cost a bit more.

shabble 13y ago
For even less fun, there have been numerous incidents of consumer accesspoints/routers hardcoding NTP servers by IP address[1][2]. [1] http://pages.cs.wisc.edu/~plonka/netgear-sntp/ http://pages.cs.wisc.edu/~plonka/netgear-sntp/ [2] http://web.archive.org/web/20060408150213/http://people.freebsd.org/~phk/dlink/ http://web.archive.org/web/20060408150213/http://people.free...

smackfu 13y ago

I would guess the problem is that most sites are going to serve up a 404 in this situation, and handling 404s is a bit trickier, since they might be transient (like a proxy gets in the way, or the hosting service has temporary issues). So they just don't handle it, since polling a server and getting a 404 is going to work well enough for most use cases. Maybe put a little exclamation point next to the calendar or something, but still poll it. And then they handle a 410 the same as a 404, even though a 410 has very little chance of being transient. It's just rare enough that they don't even realize they don't support it properly.

Wilya 13y ago
I'd wager 99% of codes have a big if(200) { // Do stuff } else { // Try again later } Actually, a bunch of them probably don't even check for response code. Edit: If dataaccessd is really the iOS native CALDAV integration, that's a bit annoying. Of all people, they should be able to handle that correctly.
- pyre 13y ago
  if(200) { // Do stuff } else { // Try again later } That's like an "idiot-filter" interview question. "Is the following code valid?"
  - jimbobimbo 13y ago
    This is a very interesting observation! I imagine, there could be a great conversation derived from this seemingly easy question.
  - Groxx 13y ago
    Seems like it should be paired with a "Is the following code useful?" about code which specifically handles every error code listed here http://en.wikipedia.org/wiki/List_of_HTTP_status_codes http://en.wikipedia.org/wiki/List_of_HTTP_status_codes and contains code to handle unlisted ones. Then "where do you draw the line?". Sometimes if(200){}else{} is plenty.
    - pessimizer 13y ago
      This is a silly objection. Not every code merits specific handling because there are groups of codes that merit the same handling. if(200){...}else{retry} is never plenty.
      - Groxx 13y ago
        Assume it only queries your server. Assume your server will only ever meaningfully return 200. Assume the method has no reason to ever move (so no 301 redirect (edit: and redirects are generally handled by your request library anyway)). Assume the only time it should fail is if your server falls over on itself, which should be temporary. What else would you do then? These are not crazy expectations. Exponential backoff might be reasonable, but I'd argue that's part of "else{retry}" since it's inherently retrying.

rav 13y ago

This is probably a case where an exponential backoff would be preferable. When the client is faced with a fetch error the first time, it should wait and try again in e.g. 10 minutes, and double the wait for each repeated error. This way, after a week of downtime at the source, each client would send less than four requests for the resource per month.

ZoFreX 13y ago
No, this is a case where obeying the HTTP specification would be preferable. 410 GONE means: Gone. Don't try again.
- Millennium 13y ago
  Agreed. Incremental backoff makes sense for the 5xx-class errors, where the error is known to be server-side. But 404 is the only 4xx-class error that it makes any sense for: in any other case, it's pretty certain that your side is doing something wrong. Even when you're not -i.e. if you're absolutely certain that this is the right URL for what you want- a 404 should still trigger some kind of notification so that a person can look into it. Using incremental backoff for 410, in particular, is to miss the point entirely. 410 very explicitly means that you should not try again. The fact that it's so rare only lowers the chance that it's being used incorrectly: someone savvy enough to set up a 410 knows enough to mean it. That said, I'm starting to think that if you're going to implement 410 on your site, you should also start logging activity. When a host keeps looking, either incrementally or otherwise, start refusing connections from that host at the firewall level. A program that sets 4xx-class errors to fail silently probably isn't doing anything about refused connections, so an IP-ban should wake the developers up. Even if it doesn't, you won't have erroneous requests clogging your server logs, so you still benefit.
  - vidarh 13y ago
    > The fact that it's so rare only lowers the chance that it's being used incorrectly: someone savvy enough to set up a 410 knows enough to mean it. I don't see how your conclusion follows from your premise. I fairly often come across cases where obscure options are mainly used by those who don't understand them, and in the absence of data I wouldn't assume that the same couldn't be the case for HTTP responses. Exponential backoff is a quite reasonable compromise for those kind of situations.

jstanley 13y ago

I have exactly the same problem with a project I run. I generate iCalendar versions of the timetables for each course at my university and let students download them. The timetables are static and do not change, yet I still get hundreds of hits every day from dataaccessd, even in the middle of summer when nobody has any classes. EDIT: http://notlate.co.uk/ http://notlate.co.uk/ EDIT2: Not that it's enough load to be noticeable, but it's a little wasteful on the part of dataaccessd IMO.

ezxs 13y ago

This is another case for rate limiting

kogir 13y ago

Am I the only person who after about 3 months would have started returning random and numerous events? Then clients would voluntarily unsubscribe.

biot 13y ago
Nope. I'd be hard pressed to not do a little iCal trolling. I'm sure the moon would get destroyed by the Death Star at least a few times per year. Not to mention a new phase of the moon: Space station.
alexwright 13y ago
"I have a 2PM with 'Goatse.cx', I don't remember this, let me just click the info link. URGHWTF."
smackfu 13y ago
> Then clients would voluntarily unsubscribe. Or write nasty blog posts about you that pollute Google results forever.

evmar 13y ago

A long time ago I published an XML file on my website for some ill-fated Google attempt at providing extensible search results. I took the file down some time later, but the Google crawler continued to hit the URL. Amusingly, I asked around within the company to try to find who was responsible for the crawler repeatedly fetching the 404 but ultimately couldn't figure it out. (It seemed like the decommissioned service had maybe registered the URL with some still-running automated refetching service?) It eventually stopped on its own.

MBCook 13y ago

Fetching stuff over HTTP can be incredibly stupid. I there is a security list out there that some little firewall company decided to have their devices automatically download. The list doesn't change that often, but every single device that company made likes to check every 10 minutes or so. Nothing can stop them, they follow Wilya's algorithm [1]. [1] https://news.ycombinator.com/item?id=6268919 https://news.ycombinator.com/item?id=6268919 Side note: it's sad to see that Apple is a huge perpetrator in this one. I hope they fix that bug.

eli 13y ago

Google will continue to request 410 resources for quite a while too. As far as I can tell it treats them the same as 404.

charlesism 13y ago

I'd never even noticed there was such a thing as a 410 until this. And it's not surprising - what incentive is there for a team in the real world with a shipping product to honor it? There's a pretty obvious incentive to ignore it: it takes developer time to implement, and the result of the 'feature' is that your customer may get permanently locked out of a net resource. What happens if a website admin accidentally puts up a 410, or - worse yet - if a previous owner of the domain once put up a 410 for a page with the same URI. I suppose there's an RFC for the code somewhere, but it seems like a bad idea. I pity the tech support agent who has to explain why his/her customer's browser refuses to visit foo.com/bar when everyone else can view it and there's no longer any way of knowing that once, for an afternoon in 1997 the page threw a 410!