6 ms·
Apple might be running a web crawler written in Go
- jezfromfuture 12y agoWay to get someone sacked at apple.
- vetinari 12y agoWhy? This is exactly the situation, where the tool is a great match for a job.
- mseepgood 12y agoThey're looking for Go engineers: http://www.golangprojects.com/golang-go-job-in-Senior-CDN-DevOps-Software-Engineer-Santa-Clara-Valley-Apple.html http://www.golangprojects.com/golang-go-job-in-Senior-CDN-De...
- andreasklinger 12y agoWhy shouldnt someone at apple use Go. Servers/scripts/etc are not their usual focus and they dont really have an internal language that fits better. Go is pretty good at this kind of things.
- pjmlp 12y agoI like Go, but fail to see how Swift and C++14 don't fit the bill as well. EDIT: changed to a more constructive comment.
- boomlinde 12y ago"I like X, but i fail to see why Y and Z don't fit the bill as well." Try repeating your point for any order of X, Y and Z, and you might realize it isn't really a valid reason for them to have used Swift or C++ instead.
- pjmlp 12y agoApple owns C++ and Swift compilers.
- boomlinde 12y agoYour original argument was that C++ and Swift fit the bill as well. Now, as far as I can tell, you seem to be saying that C++ and Swift are better choices because Apple owns compilers for them. That is a different argument, which is in direct conflict with the one I originally responded to. My response was based on the assumption that you meant what you wrote. Anyway, onto your current stance. That Apple don't use Swift or C++ for their spider seems to indicate that the benefit of using Go outweighs the benefit of using a language they own the compiler for. Go has a BSD-style license, so as far as Apple implementing their own changes and doing whatever they want with the source code and their own binaries at will goes, they might as well have owned it. Go is also seems like a no-brainer choice for a web spider since it has concurrency modeled in such a straight forward and cruft free way.
- astrodust 12y agoAny Turing Complete language would "fit the bill", obviously. What's wrong with a company not just kicking the tires on another language, but using it on a big project? That's how you design better languages of your own. "Go was great, except for X and Y" leads to improvements on your own stuff.
- aikah 12y agolol,i'm not sure why anybody would be sacked because "Google product" ;) however,I wonder why they didnt use Swift,and it's a question.
- wyager 12y agoSwift was designed for coding user-facing applications, not internet utilities.
- melling 12y agoCan you explain what that means? Which language features is Swift missing?
- wyager 12y agoIt's a perfectly reasonable general purpose language. However, it doesn't have a low-overhead green thread mechanism, and the standard library has no particular emphasis on network tools. Even as an outspoken critic of go, I recognize that it has quite a strong networking toolset.
- antimagic 12y agoSwift was designed to do everything - one language to rule them all. To quote from Apple's Swift page: "Swift is a successor to the C and Objective-C languages." My guess would be that the webcrawler was started before Swift was allowed to escape from the Developer Tools enclave.
- wyager 12y agoDoing everything is not the same as doing everything well. Surely you agree that languages designed with one purpose in mind (a la Go and networking/server development) can reasonably do better at that purpose than an explicitly general-purpose language? Just to be clear, I prefer Swift to Go in general. However, between the two, I would personally prefer to use Go for the development of network utilities.
- 13 12y agoI've got several thousand requests for this in my logs with an IP address in the same /24. Two weeks ago it was using a different user agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Fetcher/0.1) This string is mostly from a beta build of iOS 6.0 beta 4, but with the same suffix the author discovered.
- unwind 12y agoNot at all sure why it's relevant which programming languages they use at Apple. Is this considered interesting since it somehow proves the mainstream adoption of Go? Or since Apple, a competitor of Google's in cell phones, are using one of Google's technologies? Or what? The takeaway for me was that Apple has a whole /8 subnet to themselves. That's just ... immense, for a single company. Gaah. EDIT: Mis-typed the netmask, I meant /8 but typed /24. Fixed.
- manacit 12y agoA /24 for a company of their size is nothing - they actually have the entire 17.0.0.0/8 to themselves: http://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_address_blocks http://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_addres...
- jrochkind1 12y agoLess about 'of their size' then about how long they've been around, since back in the days when a large company could get a /8, and just about anyone could get a /16.
- unwind 12y agoYeah, that was what I meant, brain error. Thanks, I edited my comment.
- runeks 12y agoInteresting. I know Google uses the IP 8.8.8.8 (and 8.8.4.4) for their public DNS servers. So it seems Level3 doesn't own the entire 8.0.0.0/8 block then.
- lmz 12y agoThey do, but they reassign some of them to other parties: http://whois.arin.net/rest/net/NET-8-0-0-0-1/children http://whois.arin.net/rest/net/NET-8-0-0-0-1/children
- wyager 12y ago
- hartator 12y agoI am doubtfull Apple wants to compete with Google in search. Or maybe it's a way for Steve Jobs to get a revenge post-mortem on Google for Android!
- yourad_io 12y ago> So far, I have seen requests from two IPs: 17.147.18.33 (7 on 2014-10-15) and 17.147.18.33 (7 on 2014-10-15) One of these should be .35, judging by the logs that follow ;)
- struct 12y agoPerhaps a proxy designed for iOS testing?
- 13 12y agoI don't think so, it's only requesting HTML.
- pkulak 12y agoGo is a fantastic language for writing a crawler. I wrote the Showyou crawler in Go and it's both one of our highest load processes (and it's very efficient, trust me) and most stable.
- brycekahle 12y agoThe Go http client definitely has a bug that doesn't maintain the User-Agent across redirected requests. https://code.google.com/p/go/source/browse/src/net/http/client.go#338 https://code.google.com/p/go/source/browse/src/net/http/clie...
- bgentry 12y agoLooks like issue #4800 is the place where it's tracked: https://code.google.com/p/go/issues/detail?id=4800 https://code.google.com/p/go/issues/detail?id=4800
- jdalgetty 12y agoI've been seeing this as well.
- bkeroack 12y agoI've seen golang mentioned in Apple job listings. I didn't think this was a secret.
- christop 12y agoI noticed this as well a couple of weeks ago. They're still actively visiting every page on the websites that are associated with our iOS apps. Today alone (starting shortly before 8am CET) they've crawled over 8000 pages on https://trails.io https://trails.io and http://offmaps.com http://offmaps.com — without sending conditional caching headers (our pages don't change too often). $ grep ^17\\. /var/log/nginx/*.log | grep 'Go|Fetcher' | wc -l 8254 Note that they don't appear to be scraping any URLs available from within the apps themselves, but rather the the company/support websites linked in our App Store listings. I guess they're automatically scanning for objectionable content, since these websites are linked from the App Store and the iTunes website?
- 0x0 12y agoOoh, that's interesting. They already increased the requirements for online privacy policies etc for ios8 apps, so perhaps we'll see some crackdown on appstore apps who are missing those.
- 0x0 12y agoSo they are "apple-maps'ing" search as well, hilariously using one of Google's own tools in the process. Would be interesting to know if/when any deals with bing/google/duckduckgo for search is expiring, like they were with google maps. That's probably when we will see an apple search engine. Makes sense for them to fully control another part of the backend for spotlight and siri.
- pvsnp 12y agoIt looks more likely that someone's learning Go and going through the crawler example in the tutorials. Of course I may be wrong here. The trigger for me was the term "Fetcher" here.. http://tour.golang.org/#73 http://tour.golang.org/#73 Fun exercise though.
- kngspook 12y agoAs a former employee, that was my first thought when I read the post too. Every once in a while, I would stay late there and work on a little personal project for fun there. Sometimes one of my friends would be working late, and I'd just keep him/her company. After all, if I'm just going to do the same thing at home, I might as well take advantage of coding on my big monitor and comfy chair...
- pyman 12y agoIMHO, a big couch is better than a comfy chair.
- lifeforms 12y agoIf Apple coders are reading, I noticed exactly the same issue of User-Agent header not being used on redirects. You can solve this by (ab?)using the CheckRedirect functionality of http.Client to set the User-Agent again. Here's an example: https://github.com/lifeforms/httpcheck/commit/07440d952d166002ab759873e2885e78e8fa5c61 https://github.com/lifeforms/httpcheck/commit/07440d952d1660... (The program is nothing special, just a little thing I use internally for monitoring)
- sean9999 12y agoThis just in: Someone at Apple was doing stuff