7 ms·
The Internet Map
- Wilya 14y agoIt's a bit strange to see wordpress.com alone in the top left of the map.
- littlemerman 14y agoAgreed. Maybe an abnormality?
- deleted 14y ago[deleted]
- mashmac2 14y agoI'd love to see a how this was made blog article... especially how all of the data was found/processed.
- dvanduzer 14y agoThe about page has a fair amount of detail, but a rough outline: Basically, they took a web crawler like Heritrix (archive.org) or Scrapy (a handy Python implementation good for prototyping) and just started fetching web pages. Eventually, they have a database of 350,000 websites, along with two million links between these domains. Any set of web pages within a given domain may have hundreds of hyperlinks to a dozen other domains, but a link from any page in one domain to any page in another domain becomes a relationship between two domain nodes in a graph. Presumably they used something like neo4j.org to store these relationships (cf. jokes about relational databases being bad at storing relationship information). Then the actual hard part comes in. They link to a high level paper on rendering a visualization of that much information, and then used a similar algorithm to determine placement of each node. The size of each node is presumably the number of links in/out and the color coding is geographic (and probably not considered in this algorithm). So now they have a database describing all these nodes and relationships, and an algorithm to draw a gigantic image of all of them in 2D space. They used GPU-based parallel processing techniques (probably with NVidia's CUDA language) to crunch all the numbers to generate the final image. Finally, the image ends up being pretty large at a reasonable zoom. A scaled map of the Earth zoomed a bit above street level would still be about 125 miles on a side. So they use Google Maps API to manage small chunks of the image at various zoom levels. (They also end up rerunning that algorithm a few times to generate smaller images for each zoom step, including one at good old 1024x768). Pretty neat. Would love to see their writeup.
- nzmsv 14y agoAccording to this: http://habrahabr.ru/post/148351/ http://habrahabr.ru/post/148351/ they are just using data from Alexa and visualizing it.
- pixie_ 14y agoVery awesome, it's cool how each color is a different country and you can see where countries integrate with other countries and where they clump together on their own.
- alanfang 14y agoInteresting what's next to ycombinator http://internet-map.net/#8-141.12828063964844-82.50301361083984 http://internet-map.net/#8-141.12828063964844-82.50301361083...
- cpr 14y agoThis is the Internet map a few months after I first got on from Harv-10 in 1972: http://www.flickr.com/photos/walkingsf/7257339850/lightbox/ http://www.flickr.com/photos/walkingsf/7257339850/lightbox/ . Brings back great memories of sitting by the Harvard IMP (router) late at night, and getting a call from BBN on the phone asking me (anyone) to reboot it manually...
- JakaJancar 14y agoWhat do the two different symbols represent?
- cpr 14y agoThe red circles are IMPs (interface message processors--full routers connected to hosts) and the circled T's are TIPs (terminal interface processors--dialup routers).
- axelav 14y agohere are a bunch of other ARPANET maps from 1969-77: http://som.csudh.edu/cis/lpress/history/arpamaps/ http://som.csudh.edu/cis/lpress/history/arpamaps/
- guard-of-terra 14y agoBy the way, I (as non-US) wonder why west coast and california had more nodes even back then? I mean, west coast is less populated and if you look at night lights map it's practically invisible compared to east coast. Why tech tend to be there anyway?
- bherms 14y agoBack then it was likely easier to lay the groundwork than in more populated cities. In addition there were great institutes of higher learning focusing on technology. Note this is just a small theory and in no way has any evidence to back it.
- 14y ago
- kkelly 14y agoA couple of these are very strange. Instagram is right beside yousendit as well as almost every Irish web property. Still an awesome visualization, but what is it visualizing?
- deleted 14y ago[deleted]
- richbradshaw 14y agoHey! My personal site is on their, right next to python.org! :)
- ansgri 14y agoApparently it uses Google Maps for display, and I wonder, does GMaps API allow use of custom map data, or does it mean that this visualization is made by Google?
- jlogsdon 14y agoIn their "About" > The Internet map is a non-commercial project. You can share our expenses and let more people see beauty of the Internet. I don't think it's a Google project.
- dvanduzer 14y agoThe API lets you load a custom set of map tiles, managing up to 16 (I think) levels of zoom. It doesn't have to be specifically tied to Earth's GPS/cartographic conventions.
- chrisbroadfoot 14y agoYou should be able to go to zoom level 23 or 24.
- Stratoscope 14y agoThe Maps API works with all sorts of custom map types [1]. You can augment or replace the standard Google map tiles. This map uses a google.maps.ImageMapType [2] along with a custom EuclideanProjection map projection [3] which replaces the standard spherical Mercator ("Google Mercator") projection. [1] https://developers.google.com/maps/documentation/javascript/maptypes#CustomMapTypes https://developers.google.com/maps/documentation/javascript/... [2] https://developers.google.com/maps/documentation/javascript/maptypes#ImageMapTypes https://developers.google.com/maps/documentation/javascript/... [3] View source of http://internet-map.net/ http://internet-map.net/
- ansgri 14y agoThank you for the detailed reply. I definitely agree with [3], and wouldn't ask if I've had enough experience with modern web technologies.
- ComputerGuru 14y agoWhat is it visualizing? Pages or links? Because if it's links, there's no way that Facebook is so large and Wikipedia is so small.
- delinka 14y agoAccording to other comments on HN, it seems they are counting outside links to a given domain.
- ComputerGuru 14y agoThat's what I'm saying, it's just not possible. Pageviews would explain Facebook being so much larger than Wikipedia, but think about it - how many links from outside Facebook point to Facebook vs how many links from outside Wikipedia point to Wikipedia?
- SonicSoul 14y agowhat about all the Facebook embedded widgets? wouldn't those be links to facebook.com ?
- brianfryer 14y agoNot necessarily. Most of those things are iframes, so technically it's FB linking back to FB.
- brennenHN 14y agoThe about section on the site says it's pageviews. The sites arrange themselves around one another according to links, but the size is about views.
- deleted 14y ago[deleted]
- sudhirj 14y agoThis is actually a very useful advertising tool - if you're considering placing ads on any sites, it's a great way to see how they compare to other sites in terms of traffic and visitor flow.
- sudhirj 14y agoFor instance it tells me that HN primarily receives it's non-direct traffic from Twitter, and that softwarebyrob and randsinrepose both have roughly the same traffic - both driven primarily from HN.
- lhtbws 14y agoAgreed, but it's also funny to find inexplicable little outliers in the graph. Amishamerica.com, for instance, seems to be nestled right in the middle of the pornoverse (http://internet-map.net/#12-157.30947875976562-183.96192932128906 http://internet-map.net/#12-157.30947875976562-183.961929321...).
- deleted 14y ago[deleted]
- revelation 14y agoThat is not the Internet, its the Web. I'm all for telling non-technical people that the internet ends with the Web, save we get politicians wanting to regulate TCP ports or establish police server patrols. A map of the internet would probably show AS and peerings between them.
- snambi 14y agoSome questions: [1] nice to have some details about the site when clicking on them. [2] Is there any meaning to the color? if so, what are they?
- deleted 14y ago[deleted]
- Wilya 14y agoColor is country, it's written somewhere in the about page. Russia is red, China is yellow-ish, etc.
- Trufa 14y agoI can't reddit is this[1] small! Also you are here[2]. [1]: http://i.imgur.com/impUR.jpg http://i.imgur.com/impUR.jpg [2]: http://i.imgur.com/7WKaF.jpg http://i.imgur.com/7WKaF.jpg
- SonicSoul 14y agoextra points for really tidy and readable JS and markup
- hhw 14y agoHere is a physical map of the Internet: http://www.cablemap.info/ http://www.cablemap.info/ And here is a logical map of the Internet: http://www.peer1.com/map-of-the-internet http://www.peer1.com/map-of-the-internet A map of websites doesn't really make that much sense as a website can easily be in multiple places
- dkd 14y agoThanks so much cablemap.info site.