6 ms·

Japan's NTT DoCoMo to request that Google reduces Android's data usage.

42 points by bmunro 15y ago

objclxt 15y ago
Some of the blame here lies on programmers, not that I'm supporting NTT's approach. I lead a team of app developers for my day job, and one client once sent us some code from a previous development house to troubleshoot. The thing was polling for new data every 15 seconds. What was particularly surprising was the developer who had written it had quite a good reputation, and was highly though of - on the desktop. And perhaps that's the problem - that people are coming to mobile, writing apps and not considering that what's OK on the desktop is not going to fly on mobile. (...before someone chimes in here: polling every 15 seconds on a mobile device is unacceptable if you're on a cellular network. Firstly you'll kill the battery, as iOS and Android power down the radios when not transmitting, and secondly you'll eat up bandwidth)
- nolok 15y ago
  > iOS and Android power down the radios when not transmitting, and secondly you'll eat up bandwidth) Is that still valid with the apps like gmail@android which use push notifications instead of regular pulls ? For the server to push something, you must be listening somehow
  - pagekalisedown 15y ago
    Powered down doesn't mean turned off. It powers back up when it knows the server would like to push something.
    - SoftwareMaven 15y ago
      I would love to learn how the device learns the server has a message for it. I tried to find information on it, but I obviously don't know what to search for.
      - nknight 15y ago
        It's less magical than it sounds, but a lot of work goes into it. Depending on the system, hardware can partially wake at fixed intervals to check for incoming traffic, and fully wake to process that traffic. Notably, most of the stack can stay asleep almost all the time, and wake up only when actual traffic is coming in. Application-level polling can eat up the power savings as the whole hardware stack must wake up and process an application-level transaction taking anywhere from several tens of milliseconds to several full seconds. Compared to the "napping" state, this eats huge amounts of power.
- jsnell 15y ago
  The bandwidth argument is pretty weak, any halfway competent polling will not use significant amounts of data even if done fairly frequently. But a lot of operators pay license fees to the manufacturers of the network equipment based on the channel element usage. And to a first approximation the CE usage will be the same for a phone sends a single packet every 15 seconds as for one downloading at full speed.
  - objclxt 15y ago
    I admit with regards to bandwidth I'm often in the minority. Certainly battery life is a huge issue with any kind of polling. Personally, I think it's really important to minimise bandwidth as much as possibly, because you don't necessarily know how your user is paying for it. It would not be unknown for some (foolish?) users to be paying $10+ per megabyte in certain countries/carriers. In the case I mentioned, we were talking about a fairly hefty XML file (which I neglected to mention).
  - nknight 15y ago
    > competent polling In practice, that's basically an oxymoron. Any sort of application-level polling is problematic, and even where necessary, developers almost always take a naïve and highly sub-optimal approach.
  - olofsj 15y ago
    In a cellular network, even just establishing a connection is while use up resources. It's not just about licensing fees. There's a lot of low level control signaling required to keep a terminal connected and in sync, and a base station can only keep so many connected at once. So you really want to connect rarely and then download a lot at once and disconnect again. The more seldom they connect, the more terminals you can support.
    - jsnell 15y ago
      Indeed, initializing a session is a heavyweight operation. But something polling every 15 seconds is likely never going to tear down the session. From what I've heard the timeouts tend to be a bit longer than that (e.g. 30s), exactly for that reason. And yes, there could be physical limits as well causing an individual cell to be fully congested due to open but idle sessions. Either way the polling is causing problems disproportionate to the amount of actual traffic.
ge0rg 15y ago
With some Android applications polling for new data every three to five minutes (even when not in use), [...] This is actually the providers' own fault. If you want to keep a connection to a "push service" like gmail, imap, voip or jabber, you usually do so by maintaining an idle TCP connection, where the server can push notifications to you. However, thanks to the IP address shortage, almost all 3G providers deploy NAT and only give private addresses to their users. That means that their gateways are maintaining connection tracking tables, which usually have a certain timeout after which idle connections are kicked out. So as a developer, you are forced to regularly send keep-alive messages to prevent getting dropped. And because there is no documentation on the timeout, and it is better to be safe than sorry, most developers go somewhere below 5 minutes.
- sehugg 15y ago
  If Google supported C2DM without requiring users log in with a Google account, there would be far fewer apps having to fall back to their own (perhaps chatty or buggy, but definitely redundant) push mechanism. Not saying this is what happened in DoCoMo's case, but in our experience there are a great percentage of users that do not or cannot use C2DM, and this necessitates a separate connection for each app. There should be a way for the network provider to subsidize a default C2DM connection as it is in their interest. My guess here is that the software in question is using standard VoIP signaling protocols and not C2DM at all.
  - vetinari 15y ago
    We could go forever discussing "If X didn't". Telcos are getting whatever they started. If they wouldn't kill WAP OTA-Push (OMA-WAP-TS-PushOTA-V2_1-20051122-C) 10 years ago, they wouldn't have their network overloaded today, because you would not need long-living TCP connections. (Similarly, if they wouldn't kill E.164 ENUM, they wouldn't have problems with phone numbers migrating among providers. History is full of examples of telco shortsightness).
moylan 15y ago
souldn't be impossible to do. i use opera mini on android, ios and symbian and it's remote compression reduces data so that i can browse the web all day and get nowhere near my 50mb daily limit. it would be possible for google to create a service/api that does something similar for other apps.
- nknight 15y ago
  Opera Mini is extremely aggressive in its "optimization", which is lossy, quite specific to the web, creates a sub-par user experience, and is utterly inapplicable to the kind of data NTT is complaining about. At best, you could wrap the protocols in zlib, but if that actually helped, I'd say NTT's network is already hanging by a thread (and indeed it appears to be). Streaming a five minute video can account for hours, if not days, of polling by email, social media, and other 24/7 background-running clients
  - objclxt 15y ago
    The majority of carriers already compress data on-the-fly, to the point where it wouldn't be unknown for application developers to resort to tricks such as base-64 encoding and zipping images up to avoid automatic compression. There are pros and cons to this approach: some people (me!) appreciate the compression carriers provide, because it saves my bandwidth allowance. But if you had unlimited bandwidth and a fast 3G connection, perhaps you would feel differently.
    - nknight 15y ago
      I have no problem in general with the lossless compression of text or the transparent re-compression of images on websites, especially since most people are utterly clueless about how much bandwidth they waste on poorly-packed PNGs, obscenely-high-"quality" JPEGs, and even posting bizarrely high-resolution images and depending on img tag parameters to display it reasonably. Opera Mini is far more invasive, however. What actually gets transmitted to your device is not even HTML, and the JavaScript support is severely limited. It's not just "compression" in the sense we usually understand it, it significantly alters the nature of the data being transmitted.
shareme 15y ago
ahem why not push for $G sooner NTT? Solves most of those problems :)
- msh 15y ago
  you cant call NTT a laggard in that regard, they had the first 3G network in production and are apparently working on LTE from the beginning.
olofsj 15y ago
NTT DoCoMo are used to owning the whole end-to-end stack. They are one of the few, if not the only, operator that very thoroughly specifies both the network side and the terminal side of their network. They have a large R&D department that is very competent and actively engaged in standardization and development of new technologies. For instance, they were one of the drivers behind LTE from the very start. For them, it must be hard to no longer own the terminals on their network fully. I suppose they had to start offering Android terminals since their own smartphones were not competitive compared to the Iphone and Android. But without any control of the terminals' behavior it's very hard to tune and operate a cellular network. It's not so easy to increase capacity, since there's only so much spectrum available for use and the modern wireless technologies can't get much more efficient. Especially in crowded locations like downtown Tokyo the base stations are already about as close as you can have them. It's not something you can just throw money at to solve. So it's a tough problem. From the handset makers' and app developers' side it's easier to not restrict data usage since they are not the ones paying the price.
kogir 15y ago
The real problem here is that cell phone networks were originally (and to a large extent still are) designed with assumptions that clients will play nice. Even the carriers' contracts with other vendors are structured this way. As an example, on certain carriers I've worked with, the carrier paid per-request for a network assisted location lookup. Access to the location service was enforced with "secret" IP address and port pairs per allowed application. This "secret" information was compiled into every approved app and was trivially enumerable by brute force or discoverable via decompilation. If I had created a malicious app that used the same port as the carriers' branded navigation software, I could incur trivially incur costs in the $xxx/hour range. Their only recourse would have been to disable that port, and with it, navigation for all their customers (until they could issue an update, which would again be vulnerable to the same attack). They could also kick my phone off the network, but if a worm was every released they'd be screwed. Mobile networks weren't and still aren't built for general use. Like the internet at large, as time goes on they'll have to improve as false assumptions they've made are accidentally or maliciously violated. If (relatively) few phones with undesirable behavior can take you down, you're doing it wrong.

42 points by bmunro 15y ago

Some of the blame here lies on programmers, not that I'm supporting NTT's approach. I lead a team of app developers for my day job, and one client once sent us some code from a previous development house to troubleshoot. The thing was polling for new data every 15 seconds. What was particularly surprising was the developer who had written it had quite a good reputation, and was highly though of - on the desktop. And perhaps that's the problem - that people are coming to mobile, writing apps and not considering that what's OK on the desktop is not going to fly on mobile. (...before someone chimes in here: polling every 15 seconds on a mobile device is unacceptable if you're on a cellular network. Firstly you'll kill the battery, as iOS and Android power down the radios when not transmitting, and secondly you'll eat up bandwidth)

nolok 15y ago
> iOS and Android power down the radios when not transmitting, and secondly you'll eat up bandwidth) Is that still valid with the apps like gmail@android which use push notifications instead of regular pulls ? For the server to push something, you must be listening somehow
- pagekalisedown 15y ago
  Powered down doesn't mean turned off. It powers back up when it knows the server would like to push something.
  - SoftwareMaven 15y ago
    I would love to learn how the device learns the server has a message for it. I tried to find information on it, but I obviously don't know what to search for.
    - nknight 15y ago
      It's less magical than it sounds, but a lot of work goes into it. Depending on the system, hardware can partially wake at fixed intervals to check for incoming traffic, and fully wake to process that traffic. Notably, most of the stack can stay asleep almost all the time, and wake up only when actual traffic is coming in. Application-level polling can eat up the power savings as the whole hardware stack must wake up and process an application-level transaction taking anywhere from several tens of milliseconds to several full seconds. Compared to the "napping" state, this eats huge amounts of power.
jsnell 15y ago
The bandwidth argument is pretty weak, any halfway competent polling will not use significant amounts of data even if done fairly frequently. But a lot of operators pay license fees to the manufacturers of the network equipment based on the channel element usage. And to a first approximation the CE usage will be the same for a phone sends a single packet every 15 seconds as for one downloading at full speed.
- objclxt 15y ago
  I admit with regards to bandwidth I'm often in the minority. Certainly battery life is a huge issue with any kind of polling. Personally, I think it's really important to minimise bandwidth as much as possibly, because you don't necessarily know how your user is paying for it. It would not be unknown for some (foolish?) users to be paying $10+ per megabyte in certain countries/carriers. In the case I mentioned, we were talking about a fairly hefty XML file (which I neglected to mention).
- nknight 15y ago
  > competent polling In practice, that's basically an oxymoron. Any sort of application-level polling is problematic, and even where necessary, developers almost always take a naïve and highly sub-optimal approach.
- olofsj 15y ago
  In a cellular network, even just establishing a connection is while use up resources. It's not just about licensing fees. There's a lot of low level control signaling required to keep a terminal connected and in sync, and a base station can only keep so many connected at once. So you really want to connect rarely and then download a lot at once and disconnect again. The more seldom they connect, the more terminals you can support.
  - jsnell 15y ago
    Indeed, initializing a session is a heavyweight operation. But something polling every 15 seconds is likely never going to tear down the session. From what I've heard the timeouts tend to be a bit longer than that (e.g. 30s), exactly for that reason. And yes, there could be physical limits as well causing an individual cell to be fully congested due to open but idle sessions. Either way the polling is causing problems disproportionate to the amount of actual traffic.

ge0rg 15y ago

With some Android applications polling for new data every three to five minutes (even when not in use), [...] This is actually the providers' own fault. If you want to keep a connection to a "push service" like gmail, imap, voip or jabber, you usually do so by maintaining an idle TCP connection, where the server can push notifications to you. However, thanks to the IP address shortage, almost all 3G providers deploy NAT and only give private addresses to their users. That means that their gateways are maintaining connection tracking tables, which usually have a certain timeout after which idle connections are kicked out. So as a developer, you are forced to regularly send keep-alive messages to prevent getting dropped. And because there is no documentation on the timeout, and it is better to be safe than sorry, most developers go somewhere below 5 minutes.

sehugg 15y ago
If Google supported C2DM without requiring users log in with a Google account, there would be far fewer apps having to fall back to their own (perhaps chatty or buggy, but definitely redundant) push mechanism. Not saying this is what happened in DoCoMo's case, but in our experience there are a great percentage of users that do not or cannot use C2DM, and this necessitates a separate connection for each app. There should be a way for the network provider to subsidize a default C2DM connection as it is in their interest. My guess here is that the software in question is using standard VoIP signaling protocols and not C2DM at all.
- vetinari 15y ago
  We could go forever discussing "If X didn't". Telcos are getting whatever they started. If they wouldn't kill WAP OTA-Push (OMA-WAP-TS-PushOTA-V2_1-20051122-C) 10 years ago, they wouldn't have their network overloaded today, because you would not need long-living TCP connections. (Similarly, if they wouldn't kill E.164 ENUM, they wouldn't have problems with phone numbers migrating among providers. History is full of examples of telco shortsightness).

moylan 15y ago

souldn't be impossible to do. i use opera mini on android, ios and symbian and it's remote compression reduces data so that i can browse the web all day and get nowhere near my 50mb daily limit. it would be possible for google to create a service/api that does something similar for other apps.

nknight 15y ago
Opera Mini is extremely aggressive in its "optimization", which is lossy, quite specific to the web, creates a sub-par user experience, and is utterly inapplicable to the kind of data NTT is complaining about. At best, you could wrap the protocols in zlib, but if that actually helped, I'd say NTT's network is already hanging by a thread (and indeed it appears to be). Streaming a five minute video can account for hours, if not days, of polling by email, social media, and other 24/7 background-running clients
- objclxt 15y ago
  The majority of carriers already compress data on-the-fly, to the point where it wouldn't be unknown for application developers to resort to tricks such as base-64 encoding and zipping images up to avoid automatic compression. There are pros and cons to this approach: some people (me!) appreciate the compression carriers provide, because it saves my bandwidth allowance. But if you had unlimited bandwidth and a fast 3G connection, perhaps you would feel differently.
  - nknight 15y ago
    I have no problem in general with the lossless compression of text or the transparent re-compression of images on websites, especially since most people are utterly clueless about how much bandwidth they waste on poorly-packed PNGs, obscenely-high-"quality" JPEGs, and even posting bizarrely high-resolution images and depending on img tag parameters to display it reasonably. Opera Mini is far more invasive, however. What actually gets transmitted to your device is not even HTML, and the JavaScript support is severely limited. It's not just "compression" in the sense we usually understand it, it significantly alters the nature of the data being transmitted.

shareme 15y ago

ahem why not push for $G sooner NTT? Solves most of those problems :)

msh 15y ago
you cant call NTT a laggard in that regard, they had the first 3G network in production and are apparently working on LTE from the beginning.

olofsj 15y ago

NTT DoCoMo are used to owning the whole end-to-end stack. They are one of the few, if not the only, operator that very thoroughly specifies both the network side and the terminal side of their network. They have a large R&D department that is very competent and actively engaged in standardization and development of new technologies. For instance, they were one of the drivers behind LTE from the very start. For them, it must be hard to no longer own the terminals on their network fully. I suppose they had to start offering Android terminals since their own smartphones were not competitive compared to the Iphone and Android. But without any control of the terminals' behavior it's very hard to tune and operate a cellular network. It's not so easy to increase capacity, since there's only so much spectrum available for use and the modern wireless technologies can't get much more efficient. Especially in crowded locations like downtown Tokyo the base stations are already about as close as you can have them. It's not something you can just throw money at to solve. So it's a tough problem. From the handset makers' and app developers' side it's easier to not restrict data usage since they are not the ones paying the price.

kogir 15y ago

The real problem here is that cell phone networks were originally (and to a large extent still are) designed with assumptions that clients will play nice. Even the carriers' contracts with other vendors are structured this way. As an example, on certain carriers I've worked with, the carrier paid per-request for a network assisted location lookup. Access to the location service was enforced with "secret" IP address and port pairs per allowed application. This "secret" information was compiled into every approved app and was trivially enumerable by brute force or discoverable via decompilation. If I had created a malicious app that used the same port as the carriers' branded navigation software, I could incur trivially incur costs in the $xxx/hour range. Their only recourse would have been to disable that port, and with it, navigation for all their customers (until they could issue an update, which would again be vulnerable to the same attack). They could also kick my phone off the network, but if a worm was every released they'd be screwed. Mobile networks weren't and still aren't built for general use. Like the internet at large, as time goes on they'll have to improve as false assumptions they've made are accidentally or maliciously violated. If (relatively) few phones with undesirable behavior can take you down, you're doing it wrong.