3 ms·

Somewhat off topic, but since CloudFlare keeps mentioning "routing the nearest data center" as one of its benefits... I have a number of clients near ICN (Seou

by kijin 10y ago

Somewhat off topic, but since CloudFlare keeps mentioning "routing the nearest data center" as one of its benefits... I have a number of clients near ICN (Seoul) who have been happy with CloudFlare for some time, but now the vast majority of their users are being routed through LAX. Not Tokyo, nor Osaka, nor any of your other locations throughout East Asia, but LAX all the way across the Pacific Ocean! This detour adds about 300ms to every uncached request, and 150ms to every cached request. All the time I spent helping my clients shave a few milliseconds off their response times, now completely obliterated! When the Pro plan ($20/mo) users contacted CloudFlare to find out what was going on, they were told to upgrade to the Business plan ($200/mo). Some of them did, but their users were still being routed halfway around the world. When they contacted support again, they were told to consider upgrading to an Enterprise plan. What is this, a cheap webhosting shop that tells people to upgrade whenever they're having a problem? I don't know what happened around late summer that suddenly made Korean and Japanese locations off limits to all but the highest paying customers (local ISPs demanding more money?) but whatever it is, it has made CloudFlare barely usable where I live.

jgrahamc 10y ago
That experience with support doesn't sound right at all. Please email jgc @ cloudflare so I can follow up with you on this.
- kijin 10y ago
  Okay, I'll gather some information from people affected by this and get back to you.
- gagax 10y ago
  This is definitely correct. Every Cloudflare pro plan site I know of now routes to LAX for South Korean users. For business sites, going over a certain amount of traffic, they also route to LAX for South Korean users. I also have confirmed with Cloudflare support, that sites going over a certain amount of traffic have to upgrade to Business -> Enterprise plan to keep South Korean users to route to ICN (Seoul). This is causing huge continous performance problems for nearly all Cloudflare sites in South Korea. http://www.todayhumor.co.kr/cdn-cgi/trace http://www.todayhumor.co.kr/cdn-cgi/trace http://www.ilbe.com/cdn-cgi/trace http://www.ilbe.com/cdn-cgi/trace http://www.issuein.com/cdn-cgi/trace http://www.issuein.com/cdn-cgi/trace All show loc=KR, while using a slow LAX colocation. The first two from what I know, are business plan sites.
kyledrake 10y ago
It doesn't have anything to do with how much they're paying (if Cloudflare uses the same network for all customers). Most likely, an Asian IP transit provider found a peer in an IX in LA and decided it was better to "cold potato" the data over there than deliver it to a nearby connection. Or it decided to de-peer for some stupid reason, or a route became "AS best path" to LA despite being actually worse in a way BGP can't figure out, et cetera. Sometimes you can fix this with BGP communities, sometimes you have no control over it and are basically screwed unless you find another transit option that works better for everyone. I've been building an Anycast network all week. Right now my problem is Australia. I make Sydney work better by peering and half the world starts best path routing to it. Debugging these problems is really tricky, particularly in Asia/Pacific, which is full of expensive submarine cables and, frankly, crappy communication monopolies that love to fight with eachother over peering. Basically you need to get a contact to their NOC team and get them to check the BGP routes for those customers. I have no idea how they diagnose, but I myself would ask for some traceroute -A yoursite.com outputs from the people with the weird routing and go from there. Don't expect overnight results, the fix may require another transit hookup or peering agreement.
- kijin 10y ago
  Yep, I understand the difficulties. It's bad enough trying to get decent international transit in these parts as a local customer. But if the matter really is beyond CloudFlare's control, their support should not be encouraging people to upgrade to a plan that costs 10x as much.
Erwin 10y ago
I saw the same thing with China. We have something hosted on IBM Softlayer in HKG that takes a detour through LA. Apparently all China traffic for Cloudflare must go through LA and that's just now China works, we were told.

by kijin 10y ago

jgrahamc 10y ago

That experience with support doesn't sound right at all. Please email jgc @ cloudflare so I can follow up with you on this.

kijin 10y ago
Okay, I'll gather some information from people affected by this and get back to you.
gagax 10y ago
This is definitely correct. Every Cloudflare pro plan site I know of now routes to LAX for South Korean users. For business sites, going over a certain amount of traffic, they also route to LAX for South Korean users. I also have confirmed with Cloudflare support, that sites going over a certain amount of traffic have to upgrade to Business -> Enterprise plan to keep South Korean users to route to ICN (Seoul). This is causing huge continous performance problems for nearly all Cloudflare sites in South Korea. http://www.todayhumor.co.kr/cdn-cgi/trace http://www.todayhumor.co.kr/cdn-cgi/trace http://www.ilbe.com/cdn-cgi/trace http://www.ilbe.com/cdn-cgi/trace http://www.issuein.com/cdn-cgi/trace http://www.issuein.com/cdn-cgi/trace All show loc=KR, while using a slow LAX colocation. The first two from what I know, are business plan sites.

kyledrake 10y ago

It doesn't have anything to do with how much they're paying (if Cloudflare uses the same network for all customers). Most likely, an Asian IP transit provider found a peer in an IX in LA and decided it was better to "cold potato" the data over there than deliver it to a nearby connection. Or it decided to de-peer for some stupid reason, or a route became "AS best path" to LA despite being actually worse in a way BGP can't figure out, et cetera. Sometimes you can fix this with BGP communities, sometimes you have no control over it and are basically screwed unless you find another transit option that works better for everyone. I've been building an Anycast network all week. Right now my problem is Australia. I make Sydney work better by peering and half the world starts best path routing to it. Debugging these problems is really tricky, particularly in Asia/Pacific, which is full of expensive submarine cables and, frankly, crappy communication monopolies that love to fight with eachother over peering. Basically you need to get a contact to their NOC team and get them to check the BGP routes for those customers. I have no idea how they diagnose, but I myself would ask for some traceroute -A yoursite.com outputs from the people with the weird routing and go from there. Don't expect overnight results, the fix may require another transit hookup or peering agreement.

kijin 10y ago
Yep, I understand the difficulties. It's bad enough trying to get decent international transit in these parts as a local customer. But if the matter really is beyond CloudFlare's control, their support should not be encouraging people to upgrade to a plan that costs 10x as much.

Erwin 10y ago

I saw the same thing with China. We have something hosted on IBM Softlayer in HKG that takes a detour through LA. Apparently all China traffic for Cloudflare must go through LA and that's just now China works, we were told.