8 ms·
Applebot, the web crawler for Apple
- JDDunn9 11y agoPlease let this be the beginning of an Apple search engine. We really need some better alternatives to Google.
- amelius 11y agoFor the privacy-aware, Apple used to be a good alternative for Google, but I'm afraid that soon this will no longer be the case.
- RexRollman 11y agoWhat are you basing this statement on?
- amelius 11y agoGoogle's businessmodel is based on mining and "selling" your data (you are not the customer). Apple's businessmodel is based on selling real products.
- cromwellian 11y agoApple still collects data on you. Google knows what sites you browse on the Web and what Web sites you search for. Apple knows what apps you search for in the App Store and what you buy in the store, and probably usage statistics. At some point, if not already, that data will be used to profile you and generate revenue maximizing suggestions of goods and services to buy in the App Store.
- amelius 11y agoSorry, but I hope you agree that one's web browsing behavior is a gazillion times more interesting than one's app-usage behavior.
- grouponthink 11y agoMy search queries start with Siri and end in Safari Apple knows my web browsing behavior because Safari is my default mobile browser app syncing my history, reading list and bookmarks with desktop Safari. Apple knows where I work, live and my travels via Apple Maps and Passbook Apple knows a few of my purchases via ApplePay both in the real world transactions and via ApplePay in apps. I used MobileMe mail before switching to GMail and have considered switching to iCloud mail. Apple knows a lot more interesting things about me than Google.
- threeseed 11y agoYou're completely missing the point. Google makes 90% of their revenue from advertising. Apple makes <5% of their revenue from App Store sales.
- vilmosi 11y agoSo you're ok with your data being collected, as long as they don't make money of it?
- BozeWolf 11y agoMy question is: why do we need an alternative really? What is better when even more companies index everything and perhaps screw with your privacy? My intuition agrees with you though. More companies == more competition. At some point privacy might be a selling point for them. On the other hand: it is kind of crazy that when I read about "indexing the web" it starts ringing all kind of privacy bells, while apple's incentive might not even be to violate people's privacy.
- mrweasel 11y agoWhen one company have more than 90% market share then: Yes, we do need an alternative. (Google market in search has been reported to be as high as 94% in countries like Denmark). If a company manages to upset Google then that company ceases to exists online, regardless of the validity of Google reason to blacklist that company. Ideally no search engine would be above 20% market share (20% being a random low number I just made up). Right now websites and marketing material/money is directed at Google exclusively, making it continually harder for new search companies to succeed.
- PhilipA 11y agoI agree with you. Google has too much power over small webshops. If you somehow come in bad standing at Google you can just shut down your operation, because you won't make much money.
- Throwaway98123 11y agoIt's nice to have alternatives for something that's so fundamental to how we use the internet. Google works well today, but what about 5 years from now, or 10 years from now? What if you need to be signed into Google+ to Google search? What if your Google+ account needs to be tied to a mobile phone? What if Google has limited search, and you need to own a Chromebook or Android device for the full search experience and viewing more than 10 results? Competition keeps them in check from doing anything outrageous, and then if they decide to anyway, we have other choices we can migrate towards.
- 11y ago
- sschueller 11y agoI'd rather have a decentralized alternative ala bitcoin then another mega corp search engine. EDIT: on that point. It would be very neat but probably very difficult to build a search index based on a agreed upon algorithm like bitcoin. There would be a need for some sort of voting system to update the algorithm and moderate people gaming it.
- soong 11y agoYacy[1] is a decentralized search engine [1] http://www.yacy.net/en/ http://www.yacy.net/en/
- rodgerd 11y agoIn what way do you imagine Apple will be better?
- Stoo 11y agoI almost exclusively use Duck Duck Go (https://duckduckgo.com/ https://duckduckgo.com/) as an alternative to Google. The only time it doesn't give me the answers I need is occasionally for code / bug related searches, in which case I go back to Google.
- 616c 11y agoI have also noticed the more detailed a query for technical issues, bugs or combinations of software, I have to rely on Google as well. My workflow is similar to yours. I silently hope this is just a matter of DDG improving over time. But truth to be told, DDG is so handy with the shebang keywords, I search Google using it and only leave DDG in my Firefox search bar and specialize with !g !gv !gi as needed.
- Stoo 11y agoActually, yeah, whenever I need to search for something code related I don't go to Google, I just redo the DDG search with !g at the end. The shebangs are incredibly useful for that.
- _lce0 11y agosecond this. I've using duckduckgo for about two years so far, both in dekstop and mobile. This is a really google replacement, and we can search other sites directly with the shortcuts .. github? !gh wikipedia? !w youtube? !yt google? !g images? !i and there are tons of them!!
- venomsnake 11y agoWell with the control freaks that are apple, we know better than you approach and the love me in the curator/gatekeeper spot ideology, seems like the only search engine apple can produce is yahoo circa 1998. Also - no porn and torrents on it.
- tomelders 11y agoSearch is a dirty business - in the sense that you have to index everything, even the content you don't like, in order to be truly useful (with he common sense exceptions for scummy, abhorrent and/or illegal content). I could be wrong, but I think Apple might have an ideological problem with a lot of content. It's their call if they decide to filter that stuff out, but it's censorship, and I struggle to see how that would result in a "better" search engine. EDIT: Applebot still has uses beyond a search engine though. I think Apple are being straightforward in it's explanation. It's for Siri and Spotlight.
- mechazawa 11y ago>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. This is nice to see for a change. There are a lot of search engine bots out there and forgetting a a lot of them is easy to do.
- joelrunyon 11y agoThis is interesting. A while back, I think either Cook or Jobs mentioned that Apples makes PRODUCTS and doesn't sell ADS. If that's true (and stays true) AND this is the beginning of a search engine for them, it's going to be VERY interesting to see what it looks like.
- kekub 11y agoI think this is more about AI and offering you answers to questions like Siri does... It will not list any links. Launching a new search engine has no benefits for Apple from my point of view.
- esteer 11y agoThis is more into AI than Ads. AFAIK Siri uses Wolfram Alpha for its answers. Given the way Google's knowledge graph is advancing, it totally makes sense for them to build their own AI engine.
- castell 11y agoGoogle Knowledge Graph is based on Freebase.com. Google shut it down, last month. IBM relied on Freebase for Watson, so they acquired Blekko last month, a real web search engine & knowledge base startup. Microsoft already owns Powerset company that powers their Cortana. Apple needs a up-to-date knowledge base too, currently they rely on WolframAlpha and Freebase, afaik. Shutting down Freebase was a big hit for many projects in the AI space, Freebase had 2,903,361,537 facts in comparision Wikimedia's Wikidata has just 13,924,224 facts - that's still a huge difference. http://en.wikipedia.org/wiki/Freebase http://en.wikipedia.org/wiki/Freebase , http://en.wikipedia.org/wiki/Knowledge_Graph http://en.wikipedia.org/wiki/Knowledge_Graph , http://en.wikipedia.org/wiki/Blekko http://en.wikipedia.org/wiki/Blekko , http://en.wikipedia.org/wiki/Powerset_(company) http://en.wikipedia.org/wiki/Powerset_(company) , http://en.wikipedia.org/wiki/Knowledge_Vault http://en.wikipedia.org/wiki/Knowledge_Vault , http://venturebeat.com/2015/03/27/ibm-acquires-web-crawling-startup-blekko/ http://venturebeat.com/2015/03/27/ibm-acquires-web-crawling-...
- hahainternet 11y agoApple already sells ads, and in fact restricts some technology to advertising partners. WebGL, full screen ads etc. They also have a patent on unskippable ads. I don't know why people don't treat statements like that with enough cynicism.
- crapplesrch 11y agoIf you can't see the expected results, you are probably searching it the wrong way! But, it's so shiny. Yes, but all the results seem Apple specific?
- phn 11y agoApple is making a smart move by not wanting to depend on anyone else. Results may suck in the beginning, but well, competing in hard stuff is hard, this is another apple maps. Hopefully they won't get bashed so hard since this is not so user facing.
- onion2k 11y agoApple is making a smart move by not wanting to depend on anyone else. That's one way of looking at it. A different one would be that the wealthiest company in the world could work with practically anyone and get a better product than they could build themselves more quickly with more features. The idea that they have to do everything in-house to get the best is paranoid and stupid. For example, rather than build AppleBot, why couldn't they pump a few billion into DuckDuckGo to get use of DuckDuckBot? Or fund archive.org to access their index? Or buy in to commoncrawl.org? It would be possible for Apple to use its fortune to benefit both their customers and the world. Google and Bing are not the only options.
- bad_user 11y agoIs DuckDuckBot for real? While I'm sure they are diversifying, DDG's results are still based on Bing.
- thomasfoster96 11y agoAs far as I know DuckDuckGo has mostly been using Yandex's results lately. DuckDuckBot is a real thing (and has been for a long time), but it doesn't do much except power their anti-spam signals.
- phn 11y agoThey could, and they have done it in the past, even with google. But if they can afford it, why depend on anyone else? I'm pretty sure they're not interested in quickly, or more features. Their users can still use other products (and lets be honest, most products WANT to be on the apple platforms), but at least this way they make sure their users aren't left stranded if those products cease to exist and/or are not updated. Remember the maps situation? They are making sure their users have core features without having to depend on others' good will. They could buy another company, sure, but I wouldn't count on DDG being ready to sell out, and besides, getting new people to work on their new thing is probably easier/better organisation-wise than to on-board a different company/organisation with a lot of baggage.
- frankacter 11y ago>If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. So if I set in my robots.txt to disallow all bots except Googlebot, Applebot will index anyway? I don't think I like that precedent.
- Bulk70 11y agoSerious question, because I can't imagine your use case - under what circumstances would you wan't to block all bots except one?
- RexRollman 11y agoMaybe for ethical reasons or some kind of exclusive agreement with one particular search vendor?
- mbel 11y agoI'm not an expert but my guess is: limiting bot traffic, but keeping the site available for the most popular search engine.
- mrweasel 11y agoMy experience is that the worst bots don't respect robots.txt anyway. Getting crawled by the major search engines typically isn't that bad, they tend to know what they're doing. Getting hammered by some crappy local search engine is what's annoying. We don't limit any bots, except once where we completely blocked Eniro in our firewall. Google, Bing and a ton of other could index at the same time, with no issue. Eniro for some reason decided to just index way to much at once, no reaction to robots.txt and no reply from the email they so kindly included in the headers. But I see your point, it's just a bit sad when Google has become "The Internet".
- karmakaze 11y agoI thought FB was the internet. Googlebot is just the Kleenex of indexers.
- GameTheory8 11y agoWill this be more focused on improving search for digital products (App Store) or more toward creating a comprehensive catalog for digital/physical products to index + download/purchase?
- bambax 11y ago> please contact us at: Apple-NOC “at” apple.com Can't Apple build itself some spam protection??!? Search is harder than this.
- carlosrg 11y agoIntroducing Apple Web Search. Only available on iOS and OS X devices.
- beltex 11y ago“It looks like someone at Apple is running a web crawler written in Go.” Likely explains [1] from last year (see [2] for HN thread). [1] http://jan.moesen.nu/2014/11/06/apple-crawler.txt http://jan.moesen.nu/2014/11/06/apple-crawler.txt [2] https://news.ycombinator.com/item?id=8567205 https://news.ycombinator.com/item?id=8567205
- _lce0 11y agonice catch! I remember some thread from the Safari's early ages where they hid the user-agent, to avoid raising attention.
- abritishguy 11y agoBased on logs it would seem that the crawler, or at least parts of it are written in golang. When following some redirects the useragent would be cleared and would identify itself as golang.
- skrowl 11y agoI wonder if Apple Search will be as close to Google Search as Apple Maps is to Google Maps. That is to say, not very close at all.
- danieleggert 11y agoApple trying to get do what Google does faster, than Google doing what Apple does. In other words: Both Apple and Google in the mobile OS and device space. Both Apple and Google in the (mobile) search space.
- Retr0spectrum 11y agoHas anyone really been far even as decided to use even go want to do look more like?
- _lce0 11y agosmall-letter footprint: Risks are inherent in the use of the Internet