30 ms·
Applebot
- SethTro 6y agoAny idea what Apple is using this for? The bottom of the page says "Apple Search may take the following into account when ranking web search results:", is this another confirmation that Apple is developing a search engine?
- admax88q 6y ago"Products like Siri and Spotlight Suggestions use Applebot."
- deleted 6y ago[deleted]
- why_only_15 6y agoApple already has a search engine, called Parsec, that's used when you search in spotlight.
- deleted 6y ago[deleted]
- w-ll 6y agoI'm going to slow all traffic with "Applebot" in the user agent by 30%. Might do that same for all iOS traffic.
- nimchimpsky 6y agowhy ?
- sethaurus 6y agoWhat would be your desired outcome from this?
- w-ll 6y agoMy comment was made in jest. But. If Apple wants to embrace the openness of the web that they benefit from, I feel they should not cripple it on their mobile devices.
- tinus_hn 6y agoOf course to then complain that Apple and their ‘monopoly’ doesn’t get them the exposure they imagine their users are demanding
- jacobwilliamroy 6y agoRun slow loris directly on your server.
- kentf 6y agoI remember when I rolled my eyes at Apple making a Maps product. I thought it was a fruitless, dumb idea. I prefer Apple Maps to Google Maps now... don't underestimate Apple's ability to plant a flag and move it inch by inch each year.
- chadcmulligan 6y agoI used to say that but lately apple maps seems to be wrong a lot - back to how it was when it first started, probably a temporary regression (I hope). Yesterday for example it told me to do a U turn in a (sort of) one way street in the middle of the city, and not the first time its been dumb in the last few weeks.
- actuator 6y agoAny specific reason why? In my personal experience Google Maps has better navigation data, better place information and overall much better maps. The user contributed content makes the platform such a pleasure to use even if you are in remote parts of the world.
- cmckn 6y ago> better place information Probably 75% of my Google Maps usage is to check business hours or grab a phone number; they've really got that down. Apple is catching up, though!
- actuator 6y agoYeah, I really liked when Pixel integrated the phone number bit into dialer itself. I can start typing a business name and it just pulls in from Google maps. The iPhone I have now is a work phone, so don't use maps a lot, should see if it has improved.
- Polylactic_acid 6y agoI mostly use siri to call a business and it works every time for me.
- dvt 6y agoIs there any reason this is getting traction on HN? Applebot has been confirmed for at least 5 years[1][2]. [1] https://appleinsider.com/articles/15/05/06/apple-details-new-applebot-web-crawler-used-by-siri-and-spotlight https://appleinsider.com/articles/15/05/06/apple-details-new... [2] https://appleinsider.com/articles/15/05/06/apple-challenges-google-with-growing-web-search-program-fueled-by-topsy-acquisition https://appleinsider.com/articles/15/05/06/apple-challenges-...
- cmckn 6y agoThe page was updated in July with much more detail.
- dmix 6y agoA random older archive from last year: https://web.archive.org/web/20190429220555/https://support.apple.com/en-us/HT204683 https://web.archive.org/web/20190429220555/https://support.a... What seems to be new: - "About search rankings" section listing its purpose - Expanded bot identification and verification information
- jf 6y agoI think people are assuming that this confirms a rumor that Apple is launching a search engine. Note that the linked page mentions “Apple Search”
- spiderfarmer 6y agoFor me Applebot activity has increased by at least 500% YOY.
- Irishsteve 6y agoThe pívot podcast made a prediction that Apple will launch a search engine. Might be overlap in audience of hacker news and podcast
- judge2020 6y agoI don't see this crawling my company's site much, but it's there (67 crawls over the past month) https://i.judge.sh/well-off/Dash/chrome_yquafZlF2U.png https://i.judge.sh/well-off/Dash/chrome_yquafZlF2U.png
- TheEnder8 6y agoThe context for this is that Apple is rumored to be starting their own search engine https://appleinsider.com/articles/20/08/27/apple-may-launch-its-own-web-based-search-engine https://appleinsider.com/articles/20/08/27/apple-may-launch-...
- paxys 6y agoI can't believe this hasn't happened yet honestly. Apple Maps has been out for 7 years now. Also a bit troubling if it gains traction, considering Apple has been a lot more heavy handed in keeping content they don't approve of out from their ecosystem.
- sneak 6y agoI’m confused as to why it’s even happening now. A good index costs a fortune to build and maintain. How does this benefit Apple? They’re a hardware company.
- paxys 6y agoThe same reason why having a maps product benefits them. They want to have complete end-to-end control over a user's experience on their device, well beyond just the hardware. Having their biggest competitor dictate what happens whenever someone opens the web browser on an iPhone is a massive hole.
- yyy888sss 6y agoNo, Apple is a 'full-stack' tech company. People buy their products both for their hardware and software. If they were just a hardware company, why would they bother making macOS, iOS, iMessage, Safari etc?
- sneak 6y agoTo sell more hardware, obviously. You wouldn’t buy a mac without macOS. How does a search engine sell more hardware?
- sollewitt 6y agoGoogle needs the competition. Search it's getting more and more laden with ads, and I have found the kind of specific technical searches it used to excel at are increasingly on page 2 after articles and videos and product listings.
- karlshea 6y agoNot to mention the dozen poorly-formatted Stack Overflow and GitHub issue aggregators that show up before the actual Stack Overflow and GitHub results.
- jmnicolas 6y agoAh yes, I noticed that recently. To add insult to injury mine are poorly translated in French.
- fivre 6y agohell, it ain't just ads, it's outright malware at this point. i had to look up some shit about my printer, and hoo boy, searching for that churns up even more of a cesspool than i get when trying to search specialized windows topics. those don't turn up anything useful most of the time, but it's at least benign--a wall of posts on official MS forums where the top answer is some "top community contributor" yeeting out a "my friend, have you tried rebooting and reinstalling drivers?" copy-pasta response to any conceivable question the printer stuff is something else. i put some fault on canon here for pushing me to unofficial sites because their official shit requires you to download a fucking exe to deliver a PDF manual (cmon, wtf, how can your docs delivery pipeline be THAT bad), but the unofficial stuff is some ninth circle of hell the first was some bog-standard infini-redirect ad fraud crap, but the second was uh... well, i was impressed to the degree that the indian tech chop shop "YOU ARE INFECT DIAL 1-800-LEGIT-MICROSOFT TO CLEAN COMPUTER" site was able to shut down a fully up-to-date copy of firefox, somehow eating all my other tabs, grabbing hold of my mouse pointer in strange ways when the window was in focus AND preventing the windows "close window" button from killing it. task manager was still able to execute it but damn, that was some ARTISTRY put into making the fraud site as clingy as possible--puts 90s-era malware sites to shame, and this shit kicks around on right on the first page of google search results for a printer manual query, not some shady-ass porn site you'd drunkenly click through to via 3 levels of ever-sketchier ads people ranted and raved about goods of questionable quality gaming amazon reviews, but eh--honestly, most of those i've bought were fine, if not artisanal high-end shit--they just engaged in a lot of dubious "get free shit for a good review" growth hacking tactics that are A-OKAY THATS JUST HOW YA DO BUSINESS when it's some SF startup pulling the same crap in a different form. it's weird how seemingly nobody gives a fuck that a lot of google searches will boost things that are far worse--offering purestrain fraud on a platter is fine, selling mediocre flash drives and kitchen gadgets with inflated reviews is THE HIGHEST OF CRIMES
- protomyth 6y agoIf robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions. Is that pretty much normal?
- deleted 6y ago[deleted]
- spectramax 6y agoThis is one of the most exciting developments in the history of the internet. Apple has always sided with privacy and while DDG toots it, they’re too small to build an independent service. I am thrilled, excited and perhaps surprised why it took so long for the most used website for large majority of the people in the world to see some competition. Please don’t fuck this up, Apple.
- m0zg 6y agoThey're going to fail, yet pretend that they succeeded. Examples: Siri, Maps, Home.
- Polylactic_acid 6y agoSiri and maps work really well for me.
- m0zg 6y agoHave you compared them to Google counterparts? There's really no comparison.
- Polylactic_acid 6y agoI have used both and they both 100% cover my use cases. Gmaps could be 1000x better but it doesn't matter because there is nothing more I need that isn't in Apple maps.
- segmondy 6y agoApple can't built software for shit. They just want ad dollars. What does Apple care about providing search? They don't.
- bigdict 6y agoThis reminded me of the fact that all of 17.0.0.0/8 was assigned to Apple.
- xmprt 6y agoThat's crazy. So Apple owns 1/256 of all IP Addresses?
- bigdict 6y agoYes, and a few other companies/agencies enjoy the same privilege: https://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_address_blocks https://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_addre...
- xmprt 6y agoAT&T and Apple make sense but why is Ford on that list. And how did Comcast get theirs in 2005? I guess if you just ask very nicely, anything can happen.
- xenadu02 6y agoRemember: ARPANet was originally just a DOD research project to connect some universities and military sites. Back when it was opened to the public you could get a .com domain name for free just by asking. Same deal with IP addresses: in the very early days you just had to ask and you could have your own /8 (assuming you were a university, medium-to-large company, or government entity). There has never been any point in clawing back the large class A assignments. If that were done it wouldn't even give us an extra year of IPv4 address assignments before we exhaust the pool again... not that there is any legal mechanism to take them back anyway. The solution is IPv6. Despite the slow rollout it is finally happening now that even the regional RIRs have started to run out of addresses.
- vladoh 6y agoI find it even more surprising that Ford owns 19.0.0.0/8.
- fireattack 6y ago>Customizing robot.txt rules Shouldn't it be "robots.txt"? It even says so below.
- bcherny 6y agoI’m hearing a lot of theories in this thread about Apple building a search product. Yes, that’s possible, but Applebot is used in a lot of ways today that are wholly unrelated to search. (If search consists of crawling, indexing, ranking, retrieval, frontend, etc., Applebot only does the crawling part.) Applebot is used for generating attachment previews in iMessage (eg. Send someone a URL — the preview is from Applebot crawling it). From the docs, it sounds like it’s also used for similar previews in Siri.
- VWWHFSfQ 6y agoApple is 100% building a search engine. But do you remember when Apple launched their maps? A decade later and it's still not nearly as good as Google maps. But it's "good enough". I imagine that's what they're going for with this new search service
- eight_ender 6y agoI'm totally fine with good enough if it keeps Google in check
- exciteabletom 6y agoThe quality of the product is irrelvant. It will have usage as long as it is the default on Apple devices.
- GeekyBear 6y agoThe data that Apple previously paid TomTom and others to use was certainly subpar compared to Google's data. However, Apple started collecting their own data in 2015, and rolling out their own maps in 2018. >Apple is filling its map with so many [details] that Google now looks empty in comparison and all of these details create the impression that Apple hasn’t just closed the gap with Google—but has, in many ways, exceeded it. https://www.justinobeirne.com/new-apple-maps https://www.justinobeirne.com/new-apple-maps Apple's map data now covers the US and its territories and data for the UK and Ireland are currently in testing. https://www.justinobeirne.com/new-apple-maps-expansion-9 https://www.justinobeirne.com/new-apple-maps-expansion-9
- 30minAdayHN 6y ago> Aggregated user engagement with search results My bad experience is generally rooted in the above criteria. It's particularly problematic on news sites. For example, if I click a news article with certain political association, I keep seeing only news from that political side. This creates an echo chamber which makes uninformed believe that 'whole world' is thinking that way. It definitely make sense to increase more engagement, but might not be totally appropriate in all instances. Curious to see how it works in live (if the apple search does become a reality)
- drivebycomment 6y agoThe only serious study I know of on the information bubble says rather the opposite is true: https://www.pnas.org/content/117/6/2761 https://www.pnas.org/content/117/6/2761
- 30minAdayHN 6y agoThanks for the reference. It's good to know that there is research backing SNS (social network sites) exposes us to more diverse news contrary to popular belief. Size of the data set seems to be big enough to rely on the results. I'm wondering if the same holds true when that experiment is conducted in isolation. For example, if we take any particular news site, if a user is repeatedly visiting the same news site, then my hypothesis is that the kind of news one sees will converge onto a specific political view point. It might vary based on each news site - google news vs apple news based on how they are optimizing for users. In reality, the isolated experiment might be moot as users do end up on news site from SNS which ultimately leads to recommendation algos picking up diverse signals to show diverse results. Given that, whether the algo itself is diverse in isolation or not wouldn't matter as the environment that feeds it is diverse. Does it make sense?
- drivebycomment 6y agoGood questions. I suspect diverse news may not actually result in diverse perspective anyway. I wouldn't be surprised significant fraction of people simply don't change their opinion, even while consuming news from diverse source. So it may be doubly moot in the end.
- deleted 6y ago[deleted]
- jmull3n 6y agoI wish Apple would acquire Mozilla, use it to improve Safari and then create a privacy first search engine.
- thundermuffin 6y agoI had this same wish yesterday. I know it isn't the ideal scenario, but there are definitely some net positives that could come out of it in the form of keeping Mozilla's lights on for a long time to come and massive improvements to areas of Safari that (to me) are worse than competitors such as dev tools and extensions.
- infinityplus1 6y agoThen they would completely stop Firefox development in favor of Safari.
- deleted 6y ago[deleted]
- Polylactic_acid 6y agoThe most FOSS hostile company in the world acquiring the last properly open source browser does not sound good.
- jmull3n 6y agoBeing open source will be irrelevant if nobody uses it anymore. https://gs.statcounter.com/browser-market-share/desktop/worldwide/#monthly-201002-202008 https://gs.statcounter.com/browser-market-share/desktop/worl...
- sbahr001 6y agoThis isn't anything new. Apple has had this bot as far back as 2015. https://searchengineland.com/apple-confirms-their-web-crawler-applebot-220423 https://searchengineland.com/apple-confirms-their-web-crawle... Everyone appears to be nervous because of the supposed search engine. I believe this was part of their initiative to improve Siri results; when the service was even worse.
- dvduval 6y agoCan we just start putting "firewall" after the name? We have the Google firewall and the Apple firewall and the Facebook firewall.
- aembleton 6y agoHow is a crawler a firewall?
- mgh2 6y agoHere is a discussion from last week: https://news.ycombinator.com/item?id=24290613 https://news.ycombinator.com/item?id=24290613
- jd20 6y agoSome fun facts: - Applebot was originally written in Go (and uncovered a user agent bug on redirects, revealing it's Go origins to the world, which Russ Cox fixed the next day). - Up until the release of iOS 9, Applebot ran entirely on four Mac Pro's in an office. Those four Mac Pro's could crawl close to 1B web pages a day. - In it's first week of existence, it nearly took Apple's internal DNS servers offline. It was then modified to do it's own DNS resolution and caching, fond memories... Source: I worked on the original version.
- NiekvdMaas 6y agoCan you share some more details about the current state? Is it still written in Go?
- jd20 6y agoNo idea, it's been years since I last worked on it. It was also not the only Go service written at Apple (90% of cloud services at Apple were written in Java), though it may have been the first one used in production.
- etaioinshrdlu 6y agoAnd I sit here kind of shocked that Apple would use Java for anything, backend or not. I thought Apple had a strong preference for owning its own tech stacks, whether that be ObjC/WebObjects or later Swift...
- jd20 6y agoI think WebObjects was supporting Java even before it came to Apple from Next. In the early days, many of Apple's services built with WebObjects even ran on Sun server hardware, and XServe's. But nowadays it's all commodity Linux hardware, like you would find in any data center.
- Cthulhu_ 6y agoNot sure why you'd be shocked, it's a solid language for enterprise services like Apple offers, and their other languages - C/C++, Objective-C, Swift - aren't very kind for web services. Great use case for Go though, especially its concurrency features for web crawlers. I reckon Scala could work too, although it's a lot more complicated / clever.
- ricardo81 6y agoThink the summary from what I've read the past few days - It's established applebot is already used for Siri, but there is no general web search - Much commentary diverts to how poor Apple maps (is?) was. - Google pay apple a handsome amount of money to be the default search engine on Apple devices - If Apple were no longer to accept that payment, about 80% of people would still use Google anyway, given a choice [0] - Googles high earnings per search means they can offer more than any other search engine, like Bing, DDG et al - People would generally like more competition in the search space and Apple has the means to do it - Anecdotally, there isn't a drop in replacement for a search default other than Google that would meet the quality requirements for end users. In this case, the only other alternative would be an Apple search engine. - As per recent articles, there does seem to be more activity from Apple wrt applebot crawling and hiring [0] https://spreadprivacy.com/search-preference-menu-research/ https://spreadprivacy.com/search-preference-menu-research/
- simonh 6y agoIf they get a choice with no silent default yes, but if the default is set for them and they have to go into preferences to change it the results are almost completely reversed. This is why Apple Maps is so heavily used on iOS. Most people don't know or don't care. In the first year after the launch of Apple Maps it hit over 65% of maps usage on iPhones in the UK, even though back then the UK maps were awful.
- mattfrommars 6y agoI am mega curious how does Apple circumnavigate problem of getting flagged for crawling website. Doesn't CDN cloudfare eventually flag them? If proxies, what is their source for proxies.
- ricardo81 6y agoLikely whitelisted as a 'good bot' in Cloudflare. If a bot isn't well known, doesn't respect robots.txt or crawls too fast then much more likely to get flagged, but applebot are none of those.
- mcemilg 6y agoI was scared while writing crawlers to various sites since I don't know how far it's legal, but Apple says look, this is my crawler.
- achairapart 6y agoI think Applebot is nothing new (apart from the name). It's been years now since I found in my logs traces of crawlers from Apple IPs with an iOS 8 Safari-like user-agent. Maybe, well, since iOS 8 at least. Edit: see here @jd20 thread for insights and confirming this.
- simonh 6y agoI suspect this is all about Siri. They went from being top of the game in personal assistants to being a second stringer. Apple can't compete with Google Assistant without the data set and analytics. Personal assistants and web search are two sides of the same coin. There are reports that right now Applebot is being used to scrape the content used by Siri. If youre already scraping the web, indexing it and building search and analytics services on top of it for Siri, user facing search is just a short step to take.
- epaga 6y agoGiven that the App Store search has one of the worst search algorithms in a major search engine that I know of, I am not holding my breath for an Apple search product...on their hand, when Apple does get something right, they hit it out of the park.
- mrtksn 6y agoI keep hearing that but when I dug a bit into it to do some ASO, the impression I got is that it's simply not easy to game in name of Search Engine Optimisation because the consensus is that the search engine only looks into the App Title, subtitle, description and keywords and the ranking is affected only by the App's performance. SEO people seem annoyed by Apple that they cannot get their wallpapers app for the "stocks" keyword, so I have a very little sympathy. When they complain that App Store Search is dumb, they essentially mean that If I want to rank up my wallpaper app "stocks" and figure a way to people use it every day. They have all kinds of tricks for Google Play but the ASO tutorials are rather dull, recommending stuff like picking an appropriate title and doing A/B tests on the keywords. As a user, I tend to be quite happy with the search results in the App Store, I don't remember any time where I could not find the thing I am looking for.
- Jaruzel 6y agoI'm keeping an eye on siri.com - right now it redirects to a sort-of-holding page on apple.com... but I can totally envision that URL becoming the home of their new search engine.
- throwaway4good 6y agoSo when is this going to replace Google as default search engine in Safari?
- amelius 6y agoSince this was posted on "support.apple.com", I assumed this to be about a robot doing customer support.
- Razengan 6y agoOn a related note, since there are video podcasts, I've always wondered why they haven't been positioned as an alternative/competitor to YouTube by now. A lot of YouTube channels and types of content could be replaced by video podcasts, and be more convenient for viewers. As for monetization, Apple wouldn't even have to build an ads platform; content creators could embed ads themselves into the content, as they currently do for audio podcasts.