3 ms·

Once Nearly Invisible To Search Engines, Flash Files Can Now Be Found And Indexed

35 points by nickb 18y ago

dmix 18y ago
I've been waiting about 10 years for this! Not really, I would never make a site in flash. It's not 1998 anymore and I don't plan on selling cars or clothes, so this is a little late. Good for whats left online, but I really hope this doesn't cause people to revive all-out flash sites.
- wallflower 18y ago
  Indexing of SWF/FLA files opens up some interesting searches of the now-not-so-deep web: Flash + game (mostly silly Flash games): http://www.google.com/search?hl=en&q=filetype%3Aswf+OR+filetype%3Afla+%2B+game&btnG=Search http://www.google.com/search?hl=en&q=filetype%3Aswf+OR+f... Flash + algorithm (finds some interesting CS and non-CS presentations): http://www.google.com/search?hl=en&q=filetype%3Aswf+OR+filetype%3Afla+%2B+algorithm&btnG=Search http://www.google.com/search?hl=en&q=filetype%3Aswf+OR+f... > Not really, I would never make a site in flash. There are a lot of photography portfolios online that use Flash (presumably to protect the photos and for presentation - e.g. first impressions for potential wedding clients). It would be nice if Yahoo/Flickr or Google image search could auto-tag (caption proximity) and index their photos. Though not specific to this announcement, it would be nice to have Flash-wrapped video screencasts/vlogs indexed by their audio content.
Tichy 18y ago
So they only provide that technology to Google and Yahoo!? That really sucks, should be available for everyone.
- marcus 18y ago
  Considering Adobe's track record with open standards I'm not really surprised. Their purpose isn't to foster innovation, it is to remove an obstacle from the decision path of people buying Adobe's products.
  - bdotdub 18y ago
    I completely agree. Their reason for releasing this isn't to allow people to look into their SWFs, but for the two largest search engines to be able index them. This is an attempt to bump up the number of flash sites because now "not indexable by Google" is not an issue.
  - deleted 18y ago
    [deleted]
- pmorici 18y ago
  Maybe they will, eventually... The article does say though that they open sourced their player software so you could presumably write your own flash decomposer to index the files w/o Adobes help.
  - jm4 18y ago
    As is typical of many articles on Tech Crunch this is one is also misleading. Adobe hasn't released source code for anything yet; however, they have opened file specifications. It's still unclear when or if they will release an open source player.
- jm4 18y ago
  It seems that way. As someone who develops a crawler that currently parses Flash files to find child files and links to external content I was excited when I first read that Adobe had made a special version of the Flash player that could be used by bots. I'm sure theirs has many more features than mine. That quickly turned to disappointment when I found out this is only for Google and Yahoo. This is totally anti-competetive. I wonder how Microsoft will react. In any case, unless they are willing to make this available for everyone my opinion of Flash and Adobe is pretty much the same as it was yesterday. [edit] It's probably worth noting that I'm making the big assumption that this isn't something that was solicited by Google and Yahoo. Afterall, Flash has never been indexed before and it never seemed to bother them. It seems to be much more a problem for Adobe since the major search engines are pretty much in a position where they can tell content providers to F- off if they've developed their whole site in something like Flash. It's possible Google and Yahoo chose to pay for this or work out some kind of agreement and in that case there's nothing to gripe about.
kajecounterhack 18y ago
Hmm this reminds me of another tool I found on YC News just a few weeks ago... http://www.flashprobe.com http://www.flashprobe.com They probably used the same methods.
apgwoz 18y ago
I never understood why they didn't just use `/usr/bin/strings` to pull out ascii, and keep the pronounceable text. This of course wouldn't help with data that's being pulled in such as XML and things, but it would have been a start...
- jm4 18y ago
  That doesn't work even in the simplest cases. I know because it's the first thing I tried when I needed to find child Flash files and external references in Flash files. This is actually a tricky problem to solve. It's easy enough to find external references if the entire URL is in a string or assigned to a variable. These go in the constant pool and it's trivial to pull everything from there and try to identify URLs. When URLs are built up using concatenation they are much more difficult to find. If/else logic is even more difficult. At this point you need to implement your own little Flash player. The problem here is that it's not enough to only implement the few instructions you're interested in. The Flash Player is a stack-based VM and if you don't implement all the instructions your stack will get hosed. The Flash files that pull in their content from XML are easy once you find the external references. You simply crawl the references like you would any other page. Text embedded in Flash is another story... If you're curious to see how this works Describe SWF is a great little tool and there's some good documentation on SWF at the Flasm site. http://www.flagstonesoftware.com/describe/index.html http://www.flagstonesoftware.com/describe/index.html http://flasm.sourceforge.net/ http://flasm.sourceforge.net/
  - apgwoz 18y ago
    Yeah, I realize it's not a trivial problem to solve, but I can't imagine that many of the swf files on the web today are nothing more than some keyframes and a bit of embedded text. I would think in this case strings would be suitable enough. Also, since you can't really specify which spot in the swf to load (assuming you're going to display the swf in it's original context), I can't imagine why it would matter if you took into play conditionals. I mean, obviously you have to interpret everything, but aren't you really just after all the blobs of text so you can do, for all intents and purposes, does this document contain this text? Thanks for the links!
dkasper 18y ago
Interesting that the Techcrunch article hit the front page here and the Google blog article http://googleblog.blogspot.com/2008/06/google-learns-to-crawl-flash.html http://googleblog.blogspot.com/2008/06/google-learns-to-craw... hit the front page of Reddit.

35 points by nickb 18y ago

dmix 18y ago

I've been waiting about 10 years for this! Not really, I would never make a site in flash. It's not 1998 anymore and I don't plan on selling cars or clothes, so this is a little late. Good for whats left online, but I really hope this doesn't cause people to revive all-out flash sites.

wallflower 18y ago
Indexing of SWF/FLA files opens up some interesting searches of the now-not-so-deep web: Flash + game (mostly silly Flash games): http://www.google.com/search?hl=en&q=filetype%3Aswf+OR+filetype%3Afla+%2B+game&btnG=Search http://www.google.com/search?hl=en&q=filetype%3Aswf+OR+f... Flash + algorithm (finds some interesting CS and non-CS presentations): http://www.google.com/search?hl=en&q=filetype%3Aswf+OR+filetype%3Afla+%2B+algorithm&btnG=Search http://www.google.com/search?hl=en&q=filetype%3Aswf+OR+f... > Not really, I would never make a site in flash. There are a lot of photography portfolios online that use Flash (presumably to protect the photos and for presentation - e.g. first impressions for potential wedding clients). It would be nice if Yahoo/Flickr or Google image search could auto-tag (caption proximity) and index their photos. Though not specific to this announcement, it would be nice to have Flash-wrapped video screencasts/vlogs indexed by their audio content.

Tichy 18y ago

So they only provide that technology to Google and Yahoo!? That really sucks, should be available for everyone.

marcus 18y ago
Considering Adobe's track record with open standards I'm not really surprised. Their purpose isn't to foster innovation, it is to remove an obstacle from the decision path of people buying Adobe's products.
- bdotdub 18y ago
  I completely agree. Their reason for releasing this isn't to allow people to look into their SWFs, but for the two largest search engines to be able index them. This is an attempt to bump up the number of flash sites because now "not indexable by Google" is not an issue.
- deleted 18y ago
  [deleted]
pmorici 18y ago
Maybe they will, eventually... The article does say though that they open sourced their player software so you could presumably write your own flash decomposer to index the files w/o Adobes help.
- jm4 18y ago
  As is typical of many articles on Tech Crunch this is one is also misleading. Adobe hasn't released source code for anything yet; however, they have opened file specifications. It's still unclear when or if they will release an open source player.
jm4 18y ago
It seems that way. As someone who develops a crawler that currently parses Flash files to find child files and links to external content I was excited when I first read that Adobe had made a special version of the Flash player that could be used by bots. I'm sure theirs has many more features than mine. That quickly turned to disappointment when I found out this is only for Google and Yahoo. This is totally anti-competetive. I wonder how Microsoft will react. In any case, unless they are willing to make this available for everyone my opinion of Flash and Adobe is pretty much the same as it was yesterday. [edit] It's probably worth noting that I'm making the big assumption that this isn't something that was solicited by Google and Yahoo. Afterall, Flash has never been indexed before and it never seemed to bother them. It seems to be much more a problem for Adobe since the major search engines are pretty much in a position where they can tell content providers to F- off if they've developed their whole site in something like Flash. It's possible Google and Yahoo chose to pay for this or work out some kind of agreement and in that case there's nothing to gripe about.

kajecounterhack 18y ago

Hmm this reminds me of another tool I found on YC News just a few weeks ago... http://www.flashprobe.com http://www.flashprobe.com They probably used the same methods.

apgwoz 18y ago

I never understood why they didn't just use `/usr/bin/strings` to pull out ascii, and keep the pronounceable text. This of course wouldn't help with data that's being pulled in such as XML and things, but it would have been a start...

jm4 18y ago
That doesn't work even in the simplest cases. I know because it's the first thing I tried when I needed to find child Flash files and external references in Flash files. This is actually a tricky problem to solve. It's easy enough to find external references if the entire URL is in a string or assigned to a variable. These go in the constant pool and it's trivial to pull everything from there and try to identify URLs. When URLs are built up using concatenation they are much more difficult to find. If/else logic is even more difficult. At this point you need to implement your own little Flash player. The problem here is that it's not enough to only implement the few instructions you're interested in. The Flash Player is a stack-based VM and if you don't implement all the instructions your stack will get hosed. The Flash files that pull in their content from XML are easy once you find the external references. You simply crawl the references like you would any other page. Text embedded in Flash is another story... If you're curious to see how this works Describe SWF is a great little tool and there's some good documentation on SWF at the Flasm site. http://www.flagstonesoftware.com/describe/index.html http://www.flagstonesoftware.com/describe/index.html http://flasm.sourceforge.net/ http://flasm.sourceforge.net/
- apgwoz 18y ago
  Yeah, I realize it's not a trivial problem to solve, but I can't imagine that many of the swf files on the web today are nothing more than some keyframes and a bit of embedded text. I would think in this case strings would be suitable enough. Also, since you can't really specify which spot in the swf to load (assuming you're going to display the swf in it's original context), I can't imagine why it would matter if you took into play conditionals. I mean, obviously you have to interpret everything, but aren't you really just after all the blobs of text so you can do, for all intents and purposes, does this document contain this text? Thanks for the links!

dkasper 18y ago

Interesting that the Techcrunch article hit the front page here and the Google blog article http://googleblog.blogspot.com/2008/06/google-learns-to-crawl-flash.html http://googleblog.blogspot.com/2008/06/google-learns-to-craw... hit the front page of Reddit.