9 ms·
Why the Internet in Sub-Saharan Africa Sucks
- zatkin 11y ago>Apple Company Huh? I thought they're Apple, Inc.
- noobie 11y agosub-Saharan* Africa
- dweekly 11y ago(fixed, ty)
- discardorama 11y agoSummary: Nairobi needs a datacenter, with Akamai and Cloudflare.
- dweekly 11y agoAlso, QUIC helps a lot.
- aaggarwal 11y agoIs QUIC protocol out of experimental phase?
- scurvy 11y agoIt's only in Chrome and only supported by Google's web server. They are using it though. You can see the stats by going to chrome://net-internals/#quic They are "working" to get it into Apache Traffic Server.
- aaggarwal 11y agoYeah, its communicating over port 443. Apache Traffic Server would sure be a big step forward, if and when that happens.
- dweekly 11y agoYes and no. It is a ways off from being solidified as an RFC AFAIK, but it serves a healthy portion of Google's traffic. So, more than an experiment but less than a standard.
- aaggarwal 11y agoI was looking for a reliable audio streaming method for a project, I came across QUIC and was in process of studying it. Its definitely seems to be the future, but I am still unsure though about using it for the project.
- TD-Linux 11y agoYou should take a look at the WebRTC stack. Opus => RTP => DTLS => SCTP => UDP. For simpler applications you can drop parts of that, for example just RTP over UDP if you don't need retransmissions or encryption.
- aaggarwal 11y agoYes, WebRTC is also a contender, almost final before I came across QUIC. I was also studying it, the application has to be reliable, so except for the DTLS, I can't skip any other layer of stack. I just have to make a final test to compare the reliability and speed characteristics of TCP vs SCTP (and maybe QUIC) for the application. WebRTC also has excellent P2P features, that's again a big plus. Thanks for the help!
- TD-Linux 11y agoIf it has to be reliable as in no loss is allowed, you're going to need slightly higher latency. If TCP-level latencies are acceptable, maybe you should look at icecast. It's super simple to set up, and works in most browsers directly.
- aaggarwal 11y agoThanks for sharing, icecast looks nice. However, it doesn't relate to the type of application I was trying to build. I was trying to improve over TCP latencies, that's why I was looking over reliable but faster protocols at transport level. I will probably benchmark everything later and share it with everyone.
- scurvy 11y agoQUIC is OK, but I'm much more interested in 0-RTT TLS 1.3
- scurvy 11y agoA local Cloudflare would definitely help with the TLS setup since that's their magic (fast SSL setup without requiring your private key). Akamai could help, depending on the use case. If the content isn't cacheable, there's not much you can do. They do have a service that keeps open TCP sessions from certain POPs back to your origin, thus cutting down on TCP 3-way handshake time. Clients connect quickly, locally and then send the request because the origin connection is already established. That service is pretty expensive compared to CDN and they don't push it very hard. I used it when I ran an ad network where latency mattered (and could pay for this feature in an explicit increase in conversions and impressions).
- bjt 11y agoBoth Cloudflare and Cloudfront (when used as a dynamic content CDN) offer persistent open connections to your origin, as a standard feature of their offering. https://blog.cloudflare.com/keep-alive/ https://blog.cloudflare.com/keep-alive/ https://support.cloudflare.com/hc/en-us/articles/200168346-Best-practices-for-Railgun-and-a-Load-Balancer https://support.cloudflare.com/hc/en-us/articles/200168346-B... https://aws.amazon.com/blogs/aws/amazon-cloudfront-support-for-dynamic-content/ https://aws.amazon.com/blogs/aws/amazon-cloudfront-support-f...
- davidu 11y agoWe have contemplated a POP in Kenya. We will get there. Johannesburg should be less than 200ms, so let me see if I can improve that... I would expect about 100ms.
- Havoc 11y agoJHB to Europe is generally round the 190ms mark. 170 if you're closer to the undersea cables. No way you're going to drop that to 100ms.
- tribaal 11y agoJHB is much further away from Europe than Kenya, though. JHB means usually going to Cape Town first, then up the west coast of the continent, wheras Mombassa uses SeaCom, going around the east of the continent. That saves a lot of time from a pure distance perspective alone.
- Havoc 11y ago>JHB is much further away from Europe than Kenya, though. Indeed. I was responding to a comment that was discussing JHB though. SA traffic is split - some of it goes round east (Seacom), some west (SAT3). Ends up being roughly 200ms east or west. Source: Got my interwebs over said cables for years
- davidu 11y agoWho is talking about Europe? This is about reaching Kenya. It should actually be ~60-70ms.
- davidu 11y agoWho is talking about Europe? This is about reaching Kenya. It should actually be ~60-70ms.
- higherpurpose 11y agoSolution: let's replace the "Internet" with only 100 faster-loading websites that we (and the local governments) approve. Win-win, right?
- x5n1 11y agoInternet freedom for all!
- dublinben 11y agoI can't help but notice that the writer is only checking US-based websites. I'm sure that local alternatives actually hosted in-country are much faster. Why should users in Kenya care about loading some Silicon Valley dating app?
- dweekly 11y agoWithout debating whether or not it's a good thing, it's inarguable that a healthy portion of the apps that are popular worldwide (outside China) are run by companies headquartered in the US: Facebook, WhatsApp, YouTube, Google, Twitter, Instagram. And now Uber. Given that each one of these has the vigorous majority of their users outside the US and most have substantial non-US infrastructure build, it gets a little strange to label them "US-based". But I think it's fair to point out that if performance lags to the extreme it may open market opportunities for local competitors to differentiate themselves with speed. Local hosting options are limited, however, making things more challenging for indigenous entrepreneurs. I was amused and appalled to find that many "Kampala web hosting" companies are actually themselves hosted in Los Angeles. So: not much is hosted in-country. Very little traffic terminates domestically outside caches.
- scurvy 11y agoElephant in the room here -> There's no money to be made there (yet)
- Epenthesis 11y agoThe question then: Is there sufficient first-mover advantage that a far-minded company setting up a datacenter there now could reap the benefits when there is money to be made?
- scarmig 11y agoI think this leaves out something critical: information infrastructure isn't merely something you deliver to a region to extract profits from it, but something that accelerates growth and helps build a civil society and flourishing middle class. If I had a couple billion dollars to blow, I'd be spending it on that. Though, maybe that's why I don't have a couple billion to blow =)
- scurvy 11y agoFar-minded and deep pocketed. Opening a POP in any new country requires a lot of capex, opex, and human capital (legal, coordination, ops, etc). This glosses over the fact that your service even can geo-scale well over high latency and high jitter networks.
- notahacker 11y agoJudging by how quickly certain African countries have taken to mobile payments - in the cases of much of the population, skipping banking altogether - there's plenty of money for those with the right vision. The profits from serving ads for consumer goods will be pretty slim though.
- Apocryphon 11y agoContinent with the fastest growing middle class, though.
- scurvy 11y ago
- MichaelCrawford 11y agoIn one of Google's lobby there is a 3-D animation of a spinning globe, with spikes that appear wherever and whenever searches are performed, with the height of the spikes being proportional to the rate of query submissions. The recruiter who showed me this was very proud of google's accomplishment and indeed it was impressive to see the spikes erupting from the east and west coast of the US, Europe, Japan and India. But I watched quietly, sadly and solemnly South America and Africa. There were modest spikes in south africa and brazil. Not many anywhere else. Both continents are quite populous. Among the reasons for the choices I've made in my life and my career the last few years is that I have come to regard the vast majority of the code I've written in my long career as a developer as being of no real lasting value to anyone. Instead I wander around downtown and oldtown portland seeking out crazy people then finding some way to lift their spirits somehow. It can be a little frightening as I specialize on the particularly violent sort. Those are also the most cruelly isolated: "LEAVE ME ALONE! LEAVE ME ALONE! WHY DON'T YOU LEAVE ME THE FUCK ALONE?" The gent he was accusing had no clue. "Hi! My name is Mike. What's yours?" "Jerry! Pleased to meet you." Does it have to be so hard? My point is that those google search spikes would be erupting all over South America and Africa were someone to take responsibility to enable them. I intend to contribute somehow, someday, in part by setting up some nameservers in cybercafes. There are actually lots of them, cybercafes are quite popular in the developing world.
- RyanZAG 11y agoI'm in Africa and... I'm sorry, but we have nameservers already. That's not really a problem. Are you really talking about DNS servers here? I wasn't sure. The issues which prevent those spikes you'd like to see on the maps at Google are education (primarily literacy, especially English literacy), access to devices, and culture. Access to devices is being solved by cheap smartphones becoming obtainable to some degree and constant improvements to data networks in African countries, but the other two have no easy solution. EDIT: To maybe make it a bit easier - consider someone in a village (with a cellphone and access to a data network) who wants to find out why something happened to the carrot he was growing at home. All of us would pop open Google and search about some information about carrots. A guy in a village would head up the road to talk to his friend who's been farming for 50 years and knows about carrots. If you tried to tell him "hey, you could ask Google that question" the only reply you'd get would be "I don't really know about that". I've had someone come to me to ask me to look up bus fares on the internet for them. They'd hand me a piece of paper with the URL for the bus companies tariff page. Not too interesting until that same person later pulled out his Windows Phone smartphone to make a phone call, and he had full internet access and a good mobile browser on that phone.
- MichaelCrawford 11y agoNorthern China talks to Southern China through Japan, to the extent that they talk at all. All of China at first had one state telecom monopoly, but then some party official set his son up with Northern China. What was left of the state monopoly won't route to it. My take on what to do about that is to set up servers in both north and south that have website mirrors, as well as a geographically-sensitive nameserver. I don't have a clue how to do the GeoDNS but surely it is a solved problem as big sites require it for load balancing.
- scurvy 11y agoChina Telecom also has a separate AS dedicated to high party officials, VIP's, etc. It still goes through GFW, but it goes through a dedicated, higher performant GFW.
- MichaelCrawford 11y agoThe US Government, at least at one time had something called FTS, for the federal telecommunications system. It was a private phone system for Uncle Sam.
- tribaal 11y agoThat is suprising to me. I lived in Tanzania for a year, and while the internet connection there was not perfect by any stretch of immagination, I had way less ping than this (100 ms to London, where my company's servers are, 120-150 to Boston). For reference, that is: aerial to my ISP two kilometers away, then fiber to Dar Es Salaam, then Seacom to Mombassa, then Seacom again to Marseille, then terrestrial fiber to London via the channel). This smells a lot like contention on the seacom link (the big fiber optics carrier that goes up the east coast of Africa through the Suez canal and accross the mediterranean to Marseille). I would be interested to see a traceroute. Most of the times ISPs in west Africa don't buy enough upstream traffic on undersea cables, creating contention. I can't blame them, it's expensive, but it's mostly an economic argument, not because the servers are too far away (also, most ISPs run pretty agressive caches of everything under the sun).
- batbomb 11y agoAre you sure that's round trip? It's at the extreme physical limit of being possible, as the distance to Boston, for example, is greater than 12k km as the crow flies, and the speed of light in fiber is about 70% that of light.
- rgbrenner 11y agoSeacom ends in Egypt (not Marseille). Here's the link on a map: http://www.submarinecablemap.com/#/submarine-cable/seacomtata-tgn-eurasia http://www.submarinecablemap.com/#/submarine-cable/seacomtat... the eassy link covers much of the same route: http://www.submarinecablemap.com/#/submarine-cable/eastern-africa-submarine-system-eassy http://www.submarinecablemap.com/#/submarine-cable/eastern-a...
- Dylan16807 11y agoTaking one second to establish the connection doesn't really explain a 12 second load time. TCP takes a little bit to ramp up, but not 30 round trips. Plus you have TCP preconnect as a lightweight way to avoid that initial delay.
- hstrauss 11y agoIt's that, along with local-loop congestion (and eventually packet loss). Then the retransmits begin. At least that's my experience with anything cellular-network based.
- Dylan16807 11y agoBut the article is talking about plentiful bandwidth over wires. Not a terrible cell link.
- Cameron_D 11y agoHowever, looking at Twitters front page I see it establishing connections to 6 different domains, that's one TLS session for each one.
- jgrahamc 11y agoThis is why CloudFlare is building out PoPs in Africa. Our goal is to be 10s of ms from people living there, not 100s, so that people who use CloudFlare for their sites provide a great experience in African countries. We are already live in Johannesburg and there will be many more.
- xrjn 11y agoWhat sorts of traffic are you seeing out of Africa? How does it compare to Europe and NA, and do you think it's worth for CDNs and large companies to host content there?
- jgrahamc 11y agoIt's definitely worth it for us. We plan to be _everywhere_.
- scurvy 11y agoAnd sniff/proxy _everything_ for economic gain. Your company's motives are not pure and altruistic. Don't pretend they are.
- jgrahamc 11y agoYour company's motives are not pure and altruistic. Huh? Let's be clear: CloudFlare is a for-profit company and hence wants to take in more money than it spends. We do that by getting people to pay us. Our revenue is from web site owners paying us. It's pretty clear what our business model is: pay is a little for bandwidth and hardware is possible, get web site owners to pay us.
- RyanZAG 11y agoThat's a nice straight forward business model most people can get behind. The problem comes later, as it does when any company gets big enough. Few years down the line, the board of directors has a look over the company and see's a big opportunity: all that data you're flowing around can be turned into money with some very clever thinking at 'no real loss to anyone'. People don't like to leave money on the table if they can just find a clever enough way to put it in their pockets, and more than enough very smart people saying "hey, pay me a little bit and I'll get all that nice money off the table for you - nobody will mind, privacy is dead anyway!"
- sitharus 11y ago250ms to Facebook and that 'sucks', glad to see New Zealand's opinion on our internet connectivity is validated! I routinely see 270-300ms pings to facebook. Anything on Amazon's CDN is at 150ms that as is CloudFlare. Akamai does have local nodes, as does Google but the lack of competition on the cable makes it slow. Guess I should move to Kenya?
- cldellow 11y agoAmazon does have POPs in Australia, but they cost more so many companies choose not to use them.
- sitharus 11y agoYep, that's why they're only 150ms a lot of the time. CloudFront has a few AU pops, no NZ ones though. The Singapore and Japan zones help, but most still goes to the US.
- viraptor 11y agoSame for Australia. (at least outside of state capitals) It's surprising how quickly you can get used to facts like github actually timing out on ssh connection... I really hope that something like ppsp / swarm / ipfs gets integrated into browsers. I'm pretty sure almost every non-user-content file I download every day has been downloaded somewhere in my town. I shouldn't have to go all the way to the US to get it.
- afarrell 11y agohttps://mosh.mit.edu/ https://mosh.mit.edu/ may be of use to you as an alternative to ssh
- WildUtah 11y agoHTTP 2.0 is going to be great for you Kiwis.
- sitharus 11y ago
- realusername 11y agoWe are currently building applications for our shops in Africa (Rwanda, Kenya and Uganda) at my current company. Kenya is not even the worst of all three, Uganda is really a disaster in terms of connectivity. In Uganda it's even difficult to get actual phone signal in most of the country. And you never have reliable Internet anyway, it works like half of the time even if you are in the few right places in the country. Building applications for this kind of environment has been quite challenging. In a few shops in Uganda, the only way to get data back is by USB keys, so you have an actual truck going there with data to send to the shop and data to get back from the shop. We are building connectivity out of trucks and USB keys.
- yxhuvud 11y agoThat sounds like a challenge! What line of business are you in that can survive that bad connectivity?
- realusername 11y agoWe are selling small solar panel kits in small towns on those three countries. The goal is to have a few lights in places where there is no electricity grid at all (it's a for-profit company, not charity). We handle pretty much everything ourselves there since there is not a lot of services pre-built to help you. The shops are selling the kits on payplan since it's quite expensive for the local purchase power. We are the equivalent of a car dealer locally to give a comparison. And the shops are running some software to handle all of that, so this is where my problems are starting ! :)
- allendoerfer 11y agoSounds like you could branch out and offer these infrastructure services to other businesses, too.
- scurvy 11y agoBummer about all those IPv4 traceroutes. I'd like to see the output of IPv6 traceroutes or mtr output.
- hstrauss 11y agoFor .ZA, I've only really seen native IPv6 at jupiter.is and on the TEnet (NREN) network. My university campus doesn't even have native IPv6 available for servers I maintain. From what I've seen running an HE tunnel through London, Youtube and Facebook make up the bulk of IPv6 traffic (as expected). At least those networks are the only thing routinely dedicating 10%+ of the total inbound traffic to IPv6 according to my home network flow analysis (I think it's about 18% IPv6 usage on a monthly basis).
- dweekly 11y agoSadly the majority of consumer traffic is still on IPv4; so if you want to understand what regular people are experiencing - it will be v4 traffic. The network I was on in this case for doing the testing was v4-only. All that said, I'd be quite keen to see where v6 peering is happening.
- xenophonf 11y agoIs there a way to read this posting without having to log into Facebook?
- witty_username 11y agoGoogle cache? https://webcache.googleusercontent.com/search?q=cache:Sv7aOTA2pjUJ:https://www.facebook.com/dweekly/posts/10101729307500703+&cd=1&hl=en&ct=clnk&gl=in&client=ubuntu https://webcache.googleusercontent.com/search?q=cache:Sv7aOT...
- zimtzucker 11y agoType Nairobi in the search box, and you'll see that Akamai has a presence in Nairobi. By the way also in Kampala (Uganda) and Kigali (Rwanda): http://wwwnui.akamai.com/gnet/globe/index.html http://wwwnui.akamai.com/gnet/globe/index.html
- witty_username 11y agoHmmm, I live in India and I get ~280 ms ping to Facebook and 180 ms ping to Twitter. Seems mostly an issue of "not-living-in-the-US".
- samuell 11y agoThis is interesting! It makes me wonder if it could also explain why AJAX based google apps like Gmail totally don't work in Addis Abeba in neighboring Ethiopia, even on the speediest of hotel wifis (really, the download speed was not bad, except for the initial lag). And I mean totally don't work ... gmail.com just gives a blank page, while the plain html version works. I have been wondering if it could be something with limits of the number of multiple connections, or even requests, affecting the AJAX requests in the background, or some kind of unfortunate timeout limit in the javascript layer. Not sure, but with this info, I'm of course thinking more about the lag. Would be interesting with more details on how it affects typical AJAX apps!
- chetanahuja 11y agoMy company provides mobile app content delivery for a bunch of apps. We use a mobile specific protocol (and backend smarts) that measures and records the latencies for each of these connections. Unfortunately, due to a lack of reasonably easy hosting or cloud solution in Africa, all African traffic is currently served out of Europe. As for sub-saharan countries, Nigeria Kenya and South Africa are the only countries where we see any significant traffic at all. And yes, the latencies are horrible, but if we look at our worldwide traffic, there are bad mobile connections everywhere (even in US and Europe) and we have specifically designed the protocol to minimize roundtrips and handle losses more cleverly than TCP, so we do ok under these conditions (though clearly not as well as we would if we had extensive server presence in the continent). There is some slight chance that Amazon or Google might introduce compute cloud nodes in Africa... as soon as that happens, we'd love to extend our presence there. And if that's not forthcoming in the near future, we'll definitely explore other hosting options (as we've done in India for now).
- Sami_Lehtinen 11y agoWhat's the problem with Hetzner? http://www.hetzner.co.za/ http://www.hetzner.co.za/
- chetanahuja 11y agoHmm... that's an intriguing possibility. I always thought of Hetzner as a Germany only hosting provider. Thanks for the pointer.
- chetanahuja 11y agoMy company provides mobile app content delivery for a bunch of apps. We use a mobile specific protocol (and backend smarts) that measures and records the latencies for each of these connections. Unfortunately, due to a lack of reasonably easy hosting or cloud solution in Africa, all African traffic is currently served out of Europe. As for sub-saharan countries, Nigeria Kenya and South Africa are the only countries where we see any significant traffic at all. And yes, the latencies are horrible, but if we look at our worldwide traffic, there are bad mobile connections everywhere (even in US and Europe) and we have specifically designed the protocol to minimize roundtrips and handle losses more cleverly than TCP, so we do ok under these conditions (though clearly not as well as we would if we had extensive server presence in the continent). There is some slight chance that Amazon or Google might introduce compute cloud nodes in Africa... as soon as that happens, we'd love to extend our presence there. And if that's not forthcoming in the near future, we'll definitely explore other hosting options (as we've done in India for now).
- pyvpx 11y agohaving spent a considerable amount of time in the Horn of Africa, the reason African internet connectivity "sucks" is mostly obvious: lack of local talent, substantially increased costs, graft/corruption, and minimal use of peering infrastructure (plenty of peering fabrics, little to no engagement or use) while I certainly love the excitement this has generated, it's not as easy as "we'll PoP Kenya! We'll PoP Tanzania! We'll PoP all of Africa! 50ms for everyone!" you would think a place like Djibout with five submarine cables and terrestrial routes into Ethiopia, Sudan, and others, would have a vibrant peering fabric or CDN nodes. Level3 just put a CDN node in there, and Telecom Italia Sparkle has been there for a while. AFAIK the CDN node is completely under-utilized and Sparkle provides high cost MPLS services for private and governmental outfits; not as much general IP transit. More work needs to be put in getting carriers to peer at existing points of presence, and costs have to be lowered any way they can.
- DocG 11y agoinstead of wires, even between citys the internet is transmitted by air. Im currently in southwest cameroon, and only means for access is over the air. At home, I use mobile network. But even in all the cyber cafes, offices, etc they use different over the air solutions. As far as I know, no one is wired directly to the network. And apparently even from this region, traffic goes to douala over the air, where it meets optical sea cable. Oh, and incoming connection is under monopoly. If it goes down, everyone go dark. There are startups whose plan is to create local network coverin city for high bandwith content, without relaying to the outside connection as this is mostly bandwith limited. So called alternative network of catche servers and wifi, as often even watching youtube is pain depending on the time and location.
- kaboro 11y agohttps://angani.co/blog/its-not-how-much-fibre-you-have-its-what-you-do-with-it/ https://angani.co/blog/its-not-how-much-fibre-you-have-its-w... a response to this post, highlighting our experience over the years in Africa
- tekesteg 11y agoI think Ethiopia is the worst!