5 ms·

Indexing the hidden web

56 points by J3L2404 14y ago

RawData 14y ago
Who has a leg up in this? Duck duck go maybe? The more I use them the more I love them...
- eternauta3k 14y ago
  I like DuckDuckGo for some uses, mainly those that let me specify the context of the search term (for example http://duckduckgo.com/?q=firefly http://duckduckgo.com/?q=firefly ). However it's inferior to Google search for local content (from my country) or understanding strange error strings I get while programming.
ricksta 14y ago
"It's time for a search engine that isn't all about ads." How does a search engine make money then?
- benologist 14y ago
  Why would ads be the only way they can make money?
  - chalmerj 14y ago
    I'm curious: What could another viable business model be for a search engine? Back in the days of something like the yellow pages, it was also an advertising supported model (or pay-for-placement, something that might damage the credibility of a search engine). What I also found very interesting is the term 'yellow pages' is in the top 5 highest revenue generating search terms. [1] [1]:http://en.wikipedia.org/wiki/Yellow_Pages http://en.wikipedia.org/wiki/Yellow_Pages
    - benologist 14y ago
      Lots of things come to mind ... 1) paying for fine-grained index controls, eg you publish something new, head over to search engine and tell it to spider your site, or you tell it to spider it between 2 am and 3am, whatever. You could also use this to test updates you're making ... imagine being able to do a dry run on your new version and see this is going to cripple your SE traffic. Or get your new article analyzed before you publish it. 2) dress listings up ala ebay ... not sure if they still do it but they used to do cheesy crap that let you make your listing stand out more than the other guys, if there's a tasteful way to do that on a CPM basis it would print money 3) charging for an api like bing etc are doing 4) charge for reports on phrases, websites, industries 5) charge for telling you why your competitors are outranking you 6) charge sites a subscription ... AOL probably gets most of their traffic straight from Google, and they probably deserve almost none of it, so make them pay for that traffic. The large, eyeball-driven sites could easily be discriminated against. 7) charge low quality sites to un-penalize them. This is not a pardon, it's just a reset and it'll eat into their margins but whatever, they need your traffic. This all revolves around two things: tax garbage sites, and provide tools for legitimate sites. These feel like low hanging fruits to me, there'd have to be much more interesting ways to monetize it than these. The only hard part really is getting people to give a shit that you made / have / are a search engine.
      - billpatrianakos 14y ago
        There are two big problems with this: 1. You can get a lot of that data and those tools for free already. Webmaster Tools and Google Analytics. 2. Many of the things you'd charge for are things people don't understand anyway. In our little hacker bubble we know how valuable this stuff is and see a fair amount of companies use it but the vast majority of websites are operated by mom and pop shops and mom and pop can barely figure out how to turn on their computer. Expecting them to have any interest in getting or interpreting those reports is like trying to get them to learn quantum theory. You'll end up with a very limited customer base. These paid options create an unfair advantage. It's the exact reason why Google was so successful. Google is trusted and popular because it isnt a pay-to-play system. People will quickly figure out that the rankings are biased and quit using the engine. This is a step backwards in search. Saying that charging to unpenalize a site isn't a pardon but a "reset" is disingenuous. Call it whatever you'd like but in the end it really is a pardon. The idea is to discourage sites from gaming the system and your whole idea is to encourage them to. What we'll end up with in the end is that what you call "garbage sites" are just sites without a lot of money and "legit sites" are those with money. I'm sorry but your plan just takes us back to the pre-google dark ages of search.
  - MisterMerkin 14y ago
    You didn't answer the question.
- nhebb 14y ago
  If you search for "HD monitor" [1] & [2], or any other highly competitive commercial product, the scale tips to more of an ad service than a search provider. Yeah, they've got to make money, but they also need to be careful about maintaining balance as to not alienate users. Without a strong competitor, there's not much impetus to keep that balance. [1] Search: https://www.google.com/search?btnG=1&pws=0&q=HD+monitor https://www.google.com/search?btnG=1&pws=0&q=HD+moni... [2] Hat tip: http://01100111011001010110010101101011.co.uk/2012/04/matt-cutts-masterplan/ http://01100111011001010110010101101011.co.uk/2012/04/matt-c...
  - notatoad 14y ago
    the ads are targeted, relevant ads. why shouldn't they show you ads for HD monitors when you search for HD monitor? if you search for an overly broad term like that, you're either going to see google's ads or you're going to see an advertisement from some other website masquerading as a top ten list or something.
    - nhebb 14y ago
      On my monitor, all the links above the fold are Google ads. I'm not suggesting they stop showing ads. That's how they make money. It's just a question of balance between ads and organic search. The amount of screen real estate dedicated to PPC has been increasing over the last few years, though, and a lot of people are taking notice. Plus, Google comes off a bit hypocritical when Brin criticizes Facebook for being a walled garden. Bing isn't too much better for this search phrase, but kudos to DDG for the Super User post atop the search results.
voxx 14y ago
I'd like to see the actual hidden web indexed. I'm not talking about data behind apps, I mean a browser with built-in onion support, and onion-google.
- DanBC 14y ago
  Something that ignores robots.txt?
  - voxx 14y ago
    What? No, I'm talking about torweb here, what are you even
- zalew 14y ago
  yeah, when I saw the title I thought it's about tor indexing (which technically would be a pain)
billpatrianakos 14y ago
Did anyone else read this as having a subtext that basically implies Google needs to be taken down? Why is it that when we talk about making search better many (seriously, like gobs of people) talk like the only way to improve it is to overthrow Google. Improvements in search can come from anywhere. If Google can deliver on what the author talks about then that's great. If someone else can then that's great too. The point is to improve search not overthrow Google's dominance, right? I'm all for the ideas in this article but I was totally turned off by the subtext that implied a need to take down Google. We don't really need a next Google. Google can be the next Google. It doesn't matter so long as the hidden web is indexed. Why is it that as soon as a company is no longer the underdog we immediately throw them under the bus. Microsoft made PCs the norm in US households and now we love to tear them down (rightly so in many cases, admittedly). Facebook used to be the coolest thing ever and now we love to hate them too. Same with Google and Apple. Why do we hate incumbents so badly? In any case, yes, let's index that hidden web. But let's focus on the indexing itself rather than who does it. If Google succeeds at doing this will it not count and will we still call for someone else to "disrupt" the new hidden web indexing industry?
- Swizec 14y ago
  I think we love hating incumbents because of the history. Modern incumbents arguably aren't that bad at all, but in the old days incumbents were always the ones putting a handbrake on progress. For instance, a whole city rioting to break new looms because it was putting "honest weavers" out of business. Or the publishing world rioting against anything that smells of sharing ... since forever. Google surprisingly doesn't act like an incumbent at all. And that's good. We shouldn't hate on them because they are incumbents since they're doing a damn good job at it. edit: Also the whole idea that "When a market is dominated by a single player. That market is ripe for disruption."
  - benologist 14y ago
    Google excels at some things but they are useless and should be replaced with others - you shouldn't have to come crying to HN after Google banned your account with years of email, or thousands in adsense revenue, or whatever, in the hope that a Google employee here might see it and act on it.
aslewofmice 14y ago
I think people are thinking too much about beating Google solely with technology, which users clearly don't give a crap about. The best play against Google lies in branding a search engine. I'd love to see someone to pony up and purchase a first-generation search engine (ie: Altavista, Lycos, Hotbot) that still has some nostalgic branding left. AOL, Yahoo, MSN have all been around since that generation but they've all gone through failed re-branding issues in a struggle to keep up with Google. Pair it with a solid technology backend (DuckDuckGo) and you might actually have a chance to pick up the biggest US internet user demographic (Gen-Y) that happened to grow up with those sites.
therobotking 14y ago
If it can be indexed then it's not hidden, right? Though I guess in this context hidden doesn't mean 'hidden on purpose', more that it's inaccessible.
huragok 14y ago
I think the next innovation in search is crowd-sourced search. Users contribute directly to the index through a browser extension or somesuch. That way, you can get the site itself, how popular a site is by how many people visit it, and you also get the referrers. I experimented with this idea about a month ago. You can grab the source here (https://github.com/SeditiousTech/Avina https://github.com/SeditiousTech/Avina) and visit the index here (avina.apphb.com). It's not a real search engine per-se, but it is/was a pretty cool experiment. One of the problems is that people will forget to turn the extension off when accessing personal information (banking, porn etc).
- nikatwork 14y ago
  Google already use social signals. This is why they want +1 buttons everywhere. They have many more social signals than just the buttons though.

56 points by J3L2404 14y ago

RawData 14y ago

Who has a leg up in this? Duck duck go maybe? The more I use them the more I love them...

eternauta3k 14y ago
I like DuckDuckGo for some uses, mainly those that let me specify the context of the search term (for example http://duckduckgo.com/?q=firefly http://duckduckgo.com/?q=firefly ). However it's inferior to Google search for local content (from my country) or understanding strange error strings I get while programming.

ricksta 14y ago

"It's time for a search engine that isn't all about ads." How does a search engine make money then?

benologist 14y ago
Why would ads be the only way they can make money?
- chalmerj 14y ago
  I'm curious: What could another viable business model be for a search engine? Back in the days of something like the yellow pages, it was also an advertising supported model (or pay-for-placement, something that might damage the credibility of a search engine). What I also found very interesting is the term 'yellow pages' is in the top 5 highest revenue generating search terms. [1] [1]:http://en.wikipedia.org/wiki/Yellow_Pages http://en.wikipedia.org/wiki/Yellow_Pages
  - benologist 14y ago
    Lots of things come to mind ... 1) paying for fine-grained index controls, eg you publish something new, head over to search engine and tell it to spider your site, or you tell it to spider it between 2 am and 3am, whatever. You could also use this to test updates you're making ... imagine being able to do a dry run on your new version and see this is going to cripple your SE traffic. Or get your new article analyzed before you publish it. 2) dress listings up ala ebay ... not sure if they still do it but they used to do cheesy crap that let you make your listing stand out more than the other guys, if there's a tasteful way to do that on a CPM basis it would print money 3) charging for an api like bing etc are doing 4) charge for reports on phrases, websites, industries 5) charge for telling you why your competitors are outranking you 6) charge sites a subscription ... AOL probably gets most of their traffic straight from Google, and they probably deserve almost none of it, so make them pay for that traffic. The large, eyeball-driven sites could easily be discriminated against. 7) charge low quality sites to un-penalize them. This is not a pardon, it's just a reset and it'll eat into their margins but whatever, they need your traffic. This all revolves around two things: tax garbage sites, and provide tools for legitimate sites. These feel like low hanging fruits to me, there'd have to be much more interesting ways to monetize it than these. The only hard part really is getting people to give a shit that you made / have / are a search engine.
    - billpatrianakos 14y ago
      There are two big problems with this: 1. You can get a lot of that data and those tools for free already. Webmaster Tools and Google Analytics. 2. Many of the things you'd charge for are things people don't understand anyway. In our little hacker bubble we know how valuable this stuff is and see a fair amount of companies use it but the vast majority of websites are operated by mom and pop shops and mom and pop can barely figure out how to turn on their computer. Expecting them to have any interest in getting or interpreting those reports is like trying to get them to learn quantum theory. You'll end up with a very limited customer base. These paid options create an unfair advantage. It's the exact reason why Google was so successful. Google is trusted and popular because it isnt a pay-to-play system. People will quickly figure out that the rankings are biased and quit using the engine. This is a step backwards in search. Saying that charging to unpenalize a site isn't a pardon but a "reset" is disingenuous. Call it whatever you'd like but in the end it really is a pardon. The idea is to discourage sites from gaming the system and your whole idea is to encourage them to. What we'll end up with in the end is that what you call "garbage sites" are just sites without a lot of money and "legit sites" are those with money. I'm sorry but your plan just takes us back to the pre-google dark ages of search.
- MisterMerkin 14y ago
  You didn't answer the question.
nhebb 14y ago
If you search for "HD monitor" [1] & [2], or any other highly competitive commercial product, the scale tips to more of an ad service than a search provider. Yeah, they've got to make money, but they also need to be careful about maintaining balance as to not alienate users. Without a strong competitor, there's not much impetus to keep that balance. [1] Search: https://www.google.com/search?btnG=1&pws=0&q=HD+monitor https://www.google.com/search?btnG=1&pws=0&q=HD+moni... [2] Hat tip: http://01100111011001010110010101101011.co.uk/2012/04/matt-cutts-masterplan/ http://01100111011001010110010101101011.co.uk/2012/04/matt-c...
- notatoad 14y ago
  the ads are targeted, relevant ads. why shouldn't they show you ads for HD monitors when you search for HD monitor? if you search for an overly broad term like that, you're either going to see google's ads or you're going to see an advertisement from some other website masquerading as a top ten list or something.
  - nhebb 14y ago
    On my monitor, all the links above the fold are Google ads. I'm not suggesting they stop showing ads. That's how they make money. It's just a question of balance between ads and organic search. The amount of screen real estate dedicated to PPC has been increasing over the last few years, though, and a lot of people are taking notice. Plus, Google comes off a bit hypocritical when Brin criticizes Facebook for being a walled garden. Bing isn't too much better for this search phrase, but kudos to DDG for the Super User post atop the search results.

voxx 14y ago

I'd like to see the actual hidden web indexed. I'm not talking about data behind apps, I mean a browser with built-in onion support, and onion-google.

DanBC 14y ago
Something that ignores robots.txt?
- voxx 14y ago
  What? No, I'm talking about torweb here, what are you even
zalew 14y ago
yeah, when I saw the title I thought it's about tor indexing (which technically would be a pain)

billpatrianakos 14y ago

Did anyone else read this as having a subtext that basically implies Google needs to be taken down? Why is it that when we talk about making search better many (seriously, like gobs of people) talk like the only way to improve it is to overthrow Google. Improvements in search can come from anywhere. If Google can deliver on what the author talks about then that's great. If someone else can then that's great too. The point is to improve search not overthrow Google's dominance, right? I'm all for the ideas in this article but I was totally turned off by the subtext that implied a need to take down Google. We don't really need a next Google. Google can be the next Google. It doesn't matter so long as the hidden web is indexed. Why is it that as soon as a company is no longer the underdog we immediately throw them under the bus. Microsoft made PCs the norm in US households and now we love to tear them down (rightly so in many cases, admittedly). Facebook used to be the coolest thing ever and now we love to hate them too. Same with Google and Apple. Why do we hate incumbents so badly? In any case, yes, let's index that hidden web. But let's focus on the indexing itself rather than who does it. If Google succeeds at doing this will it not count and will we still call for someone else to "disrupt" the new hidden web indexing industry?

Swizec 14y ago
I think we love hating incumbents because of the history. Modern incumbents arguably aren't that bad at all, but in the old days incumbents were always the ones putting a handbrake on progress. For instance, a whole city rioting to break new looms because it was putting "honest weavers" out of business. Or the publishing world rioting against anything that smells of sharing ... since forever. Google surprisingly doesn't act like an incumbent at all. And that's good. We shouldn't hate on them because they are incumbents since they're doing a damn good job at it. edit: Also the whole idea that "When a market is dominated by a single player. That market is ripe for disruption."
- benologist 14y ago
  Google excels at some things but they are useless and should be replaced with others - you shouldn't have to come crying to HN after Google banned your account with years of email, or thousands in adsense revenue, or whatever, in the hope that a Google employee here might see it and act on it.

aslewofmice 14y ago

I think people are thinking too much about beating Google solely with technology, which users clearly don't give a crap about. The best play against Google lies in branding a search engine. I'd love to see someone to pony up and purchase a first-generation search engine (ie: Altavista, Lycos, Hotbot) that still has some nostalgic branding left. AOL, Yahoo, MSN have all been around since that generation but they've all gone through failed re-branding issues in a struggle to keep up with Google. Pair it with a solid technology backend (DuckDuckGo) and you might actually have a chance to pick up the biggest US internet user demographic (Gen-Y) that happened to grow up with those sites.

therobotking 14y ago

If it can be indexed then it's not hidden, right? Though I guess in this context hidden doesn't mean 'hidden on purpose', more that it's inaccessible.

huragok 14y ago

I think the next innovation in search is crowd-sourced search. Users contribute directly to the index through a browser extension or somesuch. That way, you can get the site itself, how popular a site is by how many people visit it, and you also get the referrers. I experimented with this idea about a month ago. You can grab the source here (https://github.com/SeditiousTech/Avina https://github.com/SeditiousTech/Avina) and visit the index here (avina.apphb.com). It's not a real search engine per-se, but it is/was a pretty cool experiment. One of the problems is that people will forget to turn the extension off when accessing personal information (banking, porn etc).

nikatwork 14y ago
Google already use social signals. This is why they want +1 buttons everywhere. They have many more social signals than just the buttons though.