3 ms·

On top of that, Google has the right to ban them. For example: http://www.google.com/search?q=site:itunes.com+pink+floyd http://www.google.com/search?q=site:itu

by mojuba 18y ago

On top of that, Google has the right to ban them. For example: http://www.google.com/search?q=site:itunes.com+pink+floyd http://www.google.com/search?q=site:itunes.com+pink+floyd

Tichy 18y ago
I think you meant "scan", not "ban"? I was wondering about that, because the marketing of my iPhone app would not be very far reaching if it could not even be found by Google.
- mojuba 18y ago
  I mean "ban". Google requires that all pages make no distinction between the indexer and ordinary browsers and should return same data to both. In the case of that link I gave or any other link to itunes.com, what you see doesn't look the same as what Google sees (see Cached). They have the right to ban the site, I think.
  - silencio 18y ago
    Actually, I think that is the same thing that you would get in the iTunes store if you could see the raw XML. So what distinction does it make, exactly?
    - mojuba 18y ago
      The problem is, that page doesn't require Google to use iTunes for viewing its contents. In fact, as you can see the output of wget from my other comment, iTunes tricks both Google and us by sending Content-type: text/html.
raganwald 18y ago
How is that different from Google reaching into PDF documents, Word documents, PowerPoint presentations, or Flash applications? In all of those cases, end users must use a proprietary application to actually view the content. I clicked on your link and when it opened in iTunes, I saw the same text that Google's spider previewed. Here is Google's cache: http://209.85.141.104/search?q=cache:GYFPOvjhhfoJ:itunes.com/album%3Fp%3D684888%26i%3D684859+site:itunes.com+pink+floyd&hl=en&ct=clnk&cd=1 http://209.85.141.104/search?q=cache:GYFPOvjhhfoJ:itunes.com... As you can see, the spider saw the full text of the review. The preview in the search page might be some metadata Apple provided as a hint to Google or it may be that Google's spider is really good at extracting interesting sentences :-)
- mojuba 18y ago
  It is different in that iTunes.com returns text/html: HTTP/1.0 200 OK Last-Modified: Thu, 10 Jul 2008 15:46:08 GMT Content-Type: text/html; charset=utf-8 Content-Encoding: gzip x-apple-aka-ttl: Generated Thu Jul 10 08:46:08 PDT 2008, Expires Thu Jul 10 08:46:08 PDT 2008, TTL 0s x-apple-application-instance: 6105 x-apple-asset-version: 46436 x-apple-max-age: 0 x-apple-request-store-front: <null> x-apple-date-generated: Thu, 10 Jul 2008 15:46:08 GMT x-webobjects-loadaverage: 0 Content-Length: 1940 Vary: Accept-Encoding Expires: Thu, 10 Jul 2008 15:46:08 GMT Cache-Control: max-age=0, no-cache, no-store Pragma: no-cache Date: Thu, 10 Jul 2008 15:46:08 GMT Connection: keep-alive X-Apple-Partner: origin.0 Now, why should this be handled by a custom application? Isn't it called breaking the Web?
  - raganwald 18y ago
    Have you used iTMS? It does a bunch of stuff not handled by any browser without proprietary plug-ins. So either (a) they have to cut functionality, (b) they have to give you one or more proprietary plug-ins to give you the same functionality in another browser, or (c) provide an application that is also a site-specific web browser, which is what they have done with iTunes. If it is a choice between (a), (b), and (c), which choice would you make if you were a product manager for Apple? Which choice would you make if you were an iPod owner?
    - mojuba 18y ago
      raganwald, the only problem here is, that page does not require Google to use iTunes software for viewing the content. This alone is sufficient for banning the web site and removing it from Google's indexes. I used wget.
      - raganwald 18y ago
        "Banning the web site and removing it from Google's indexes." You and I have a very different understanding of why Google does and does not ban web sites. They ban web sites for SEO black hattery. In this case, Apple presents information that people searching Google want to find. Why on Earth would it benefit Google to ban this practice? Are you equating the iTunes application to some sort of drive-by-download malware? If so, please make that point and not frame this as a violation of Google's quasi terms of service. Do you think that Google is unaware of what is happening, that Apple is fooling them? Send them an email and let them know how Apple is tricking their spiders.

by mojuba 18y ago

On top of that, Google has the right to ban them. For example: http://www.google.com/search?q=site:itunes.com+pink+floyd http://www.google.com/search?q=site:itunes.com+pink+floyd

Tichy 18y ago

I think you meant "scan", not "ban"? I was wondering about that, because the marketing of my iPhone app would not be very far reaching if it could not even be found by Google.

mojuba 18y ago
I mean "ban". Google requires that all pages make no distinction between the indexer and ordinary browsers and should return same data to both. In the case of that link I gave or any other link to itunes.com, what you see doesn't look the same as what Google sees (see Cached). They have the right to ban the site, I think.
- silencio 18y ago
  Actually, I think that is the same thing that you would get in the iTunes store if you could see the raw XML. So what distinction does it make, exactly?
  - mojuba 18y ago
    The problem is, that page doesn't require Google to use iTunes for viewing its contents. In fact, as you can see the output of wget from my other comment, iTunes tricks both Google and us by sending Content-type: text/html.

raganwald 18y ago

How is that different from Google reaching into PDF documents, Word documents, PowerPoint presentations, or Flash applications? In all of those cases, end users must use a proprietary application to actually view the content. I clicked on your link and when it opened in iTunes, I saw the same text that Google's spider previewed. Here is Google's cache: http://209.85.141.104/search?q=cache:GYFPOvjhhfoJ:itunes.com/album%3Fp%3D684888%26i%3D684859+site:itunes.com+pink+floyd&hl=en&ct=clnk&cd=1 http://209.85.141.104/search?q=cache:GYFPOvjhhfoJ:itunes.com... As you can see, the spider saw the full text of the review. The preview in the search page might be some metadata Apple provided as a hint to Google or it may be that Google's spider is really good at extracting interesting sentences :-)

mojuba 18y ago
It is different in that iTunes.com returns text/html: HTTP/1.0 200 OK Last-Modified: Thu, 10 Jul 2008 15:46:08 GMT Content-Type: text/html; charset=utf-8 Content-Encoding: gzip x-apple-aka-ttl: Generated Thu Jul 10 08:46:08 PDT 2008, Expires Thu Jul 10 08:46:08 PDT 2008, TTL 0s x-apple-application-instance: 6105 x-apple-asset-version: 46436 x-apple-max-age: 0 x-apple-request-store-front: <null> x-apple-date-generated: Thu, 10 Jul 2008 15:46:08 GMT x-webobjects-loadaverage: 0 Content-Length: 1940 Vary: Accept-Encoding Expires: Thu, 10 Jul 2008 15:46:08 GMT Cache-Control: max-age=0, no-cache, no-store Pragma: no-cache Date: Thu, 10 Jul 2008 15:46:08 GMT Connection: keep-alive X-Apple-Partner: origin.0 Now, why should this be handled by a custom application? Isn't it called breaking the Web?
- raganwald 18y ago
  Have you used iTMS? It does a bunch of stuff not handled by any browser without proprietary plug-ins. So either (a) they have to cut functionality, (b) they have to give you one or more proprietary plug-ins to give you the same functionality in another browser, or (c) provide an application that is also a site-specific web browser, which is what they have done with iTunes. If it is a choice between (a), (b), and (c), which choice would you make if you were a product manager for Apple? Which choice would you make if you were an iPod owner?
  - mojuba 18y ago
    raganwald, the only problem here is, that page does not require Google to use iTunes software for viewing the content. This alone is sufficient for banning the web site and removing it from Google's indexes. I used wget.
    - raganwald 18y ago
      "Banning the web site and removing it from Google's indexes." You and I have a very different understanding of why Google does and does not ban web sites. They ban web sites for SEO black hattery. In this case, Apple presents information that people searching Google want to find. Why on Earth would it benefit Google to ban this practice? Are you equating the iTunes application to some sort of drive-by-download malware? If so, please make that point and not frame this as a violation of Google's quasi terms of service. Do you think that Google is unaware of what is happening, that Apple is fooling them? Send them an email and let them know how Apple is tricking their spiders.