7 ms·

I work for a telco (I don't speak for them, usual disclaimers apply). Everyone gets really ranty when operators delay firmware updates. Yeah, some of them del

by edent 12y ago

I work for a telco (I don't speak for them, usual disclaimers apply). Everyone gets really ranty when operators delay firmware updates. Yeah, some of them delay to force you to buy a new phone, or to stuff the ROM full of bloatware. Most, however, do extensive testing to make sure that customers don't ring up call centres making angry complaints. Carriers (by and large) test the phone to make sure that the network portion operates correctly. They make sure that it complies with local regulations and that it's not going to interfere with the network. They also test that basic cellular functionality works. Many go further and check all standard operations of the handset. It's been several years since I was responsible for device testing, but here's a list of problems that I remember from big name manufacturers (Nokia, HTC, Sagem) on their "final" firmware. - 999/112/911 features not working. - Reboots when receiving a text starting with a 0. - Factory reset when receiving an MMS with more than one attachment. - Inability to switch from 3G to 2G on handover. - Caller ID always picking the first name from the SIM. It was, frankly, an unending parade of shoddy workmanship. We'd file reports, wait for an updated firmware, test it again, see that it had more bugs, refuse to ship it. Frankly, a few "enthusiasts" whining that they don't have a 0.1 firmware upgrade is a sensible price to pay compared to millions of customers storming your shops and demanding refunds. Google has an atrocious attitude to customer service - it's part of their business model https://shkspr.mobi/blog/2013/02/googles-customer-contempt-conundrum/ https://shkspr.mobi/blog/2013/02/googles-customer-contempt-c...

baq 12y ago
thank you for this post. i always felt that the other side of the fence was greatly underrepresented here.
- TeMPOraL 12y ago
  I second that. We always get the same stream of negative comments towards telcos (not saying they're not deserved), but we don't get to see very often how this looks on the other side of the fence. It'll be wonderful to have more comments like GP's one.
glass- 12y ago
> compared to millions of customers storming your shops and demanding refunds And yet somehow Apple is able to push updates without carrier approval and this doesn't happen.
- mackey 12y ago
  Except for 8.0.1 which hosed cell service for everyone. However, Apple was able to pull the update within hours and release a fix the next day.
  - Alupis 12y ago
    Or 8.1.x that hosed most user's battery life (as in < 6 hours off a full charge) among a slurry of other issues.[1] [1] http://www.forbes.com/sites/gordonkelly/2014/12/09/apple-releases-ios-8-1-2/ http://www.forbes.com/sites/gordonkelly/2014/12/09/apple-rel...
  - glass- 12y ago
    And millions of customers stormed shops? Which is the rationale that is provided for carriers taking months to approve updates for everyone but Apple.
    - mackey 12y ago
      I am not defending carriers here. I just wanted to point out that Apple's approach isn't perfect either.
  - happyscrappy 12y ago
    >Except for 8.0.1 which hosed cell service for everyone. Only iPhone6 only if you updated a point release in the first hour, not that it is good.
- dguaraglia 12y ago
  What are you basing your statement on? Verizon and AT&T phones have been pegged at different minor versions of iOS many times. People want to believe Apple is some magic company that can do no wrong, but that ignores the extensive amount of QA (both in house and at the telcos) that happens before any release like this (as a longish time OS X user, I should know.)
joosters 12y ago
Surely these are issues to do with phone manufacturers and not with the carriers?
click170 12y ago
Wow, thanks for that interesting insight into the telco side of firmware upgrade delays. I humbly submit that the carriers could avoid a lot of ill-will if there was more transparency in that process. I hope there aren't contractual obligations in place preventing that because it really changes the way I personally see the problem.
- magicalist 12y ago
  They have always been fairly transparent in how this happens. The problem is they say they have to put these things through an extensive testing process before release...and then never release it. But hey, here's a new phone you can buy with the latest release!
- edent 12y ago
  Go to any carrier support forum / Twitter and they're usually pretty open about testing. But, yes, there are always contractual obligations. Not least, Manufacturer X doesn't want Carrier Y saying "Delays are due to..."
diminoten 12y ago
I had accidentally moved $1,000 from the wrong bank account into Google Wallet, and called customer service to try and figure out what I should do, because there wasn't going to be $1,000 in the account I had selected, but Google Wallet was giving me access to the funds already. The CS rep that responded was a nice guy, but had no clue what to do. We started spitballing together some possible ideas, and I suggested that we just let it go and Wallet will eventually figure out that the transfer didn't go through, and deduct the money back from my Wallet account. He thought that might be a good idea, but had no further ideas beyond that. That was Google Wallet, and that was $1,000 floating around. I can't imagine the level of customer service available to anyone who hasn't actually put money directly into the equation. Do they actually do that thing we all saw in The Internship, where they have random engineers field support calls? Because I felt like I was talking to a random engineer.
- danielvinson 12y ago
  I used to work at Google in support. You were not talking to a random engineer - the support people are all dedicated support contractors. Most of the teams are all non-technical and the only thing they know about how the product works is from a two week training where most of the time is spent on learning how to avoid lawsuits.
  - gcb0 12y ago
    support people at Google is there only to fill regulation minimums.
    - danielvinson 12y ago
      The teams I was a part of most definitely were not just for minimums (Chromebook/Glass). Google has incredible support for their hardware products.
      - gcb0 12y ago
        that's honestly awesome to hear. my experience in the past made me avoid newer Google products because support in the past was non existent. last thing i bought was a nexus one. i for the bad touch screen (registers ghost touches after screen is on for a while) and i could either complain on forums and be ignored or start a return with htc that from the forums histories ended with a device on worse condition and save touch screen bug sent back...
drzaiusapelord 12y ago
The real issue here is that GSM, MMS, etc are archaic and hard to code for. The industry's fight against VOIP seems to be the real issue here. Instead of all this complexity, why not just treat phones as data devices and voice is just another IP protocol that runs on them? I imagine the "shoddy workmanship" has a lot to do with carrier specific settings and vendor specific stuff that is difficult to code for properly. A modern telecom is a shitshow of legacy junk that should be thrown out. Granted, this is kinda happening with some carriers and google voice, but the move to LTE-only with VOIP needs to happen sooner than later. The carriers seem to be fighting this because they, then, can't sell "minutes" and other old fashioned concepts. They would become a dumb internet pipe and they fear what that means from a profit perspective.
- deleted 12y ago
  [deleted]
- noselasd 12y ago
  I'm not sure why you assume a switch to VoIP will help in that regard - in practical terms that means the SIP protocol for the voice (and SMS) signalling. While SIP is going to be the way forward, the specs are really full of ambiguity and not that well though out features. Most commercial vendors of SIP servers include full scripting languages in order for the admins to "normalize" the protocol in order to try to smooth over incompabilities - I can guarantee you that when phones are released in the wild with a SIP stack the telco hasn't tested, things will stop working until they've implemented workarounds for the SIP flavor of the day.
  - drzaiusapelord 12y ago
    SIP is software. What runs on my cellular modem and its stack is firmware written by yet another party. If SIP app #474 is screwing up, I switch to SIP app #475. This all can be done in userland as an application. Changing out my GSM stack? Yeah, I can't do that.
    - noselasd 12y ago
magicalist 12y ago
Apologies to the sibling posts praising this for a different perspective, but this post goes way too far in the other direction. Yes testing updates is very important. But by "operators delay firmware updates" do you mean never release updates? By "some of them" "stuff the ROM full of bloatware" do you mean nearly all of them? Telcos also love to write and ship their own OS UIs which are themselves full of shoddy workmanship, then never release updates for many of the phones running them. Listen, it is a shame the telcos get totally thrown under the bus when there's an entire supply chain full of issues, but I don't see what that has to do with this bug except in the sense that "there are bugs" not caused by the telcos.
- edent 12y ago
  It rather depends where you are in the world. UK operators are fairly good at releasing updates - but a lot depends on how many units the carrier sold. In the EU there's a trend away from bloatware. But, again, depends on your country / market.
xorcist 12y ago
On the other hand, by delaying an update containing important security fixes, you leave your customers sitting ducks for the next drive by download attack. We haven't seen any major botnets infecting mobile phones yet, but I think it is just a matter of time. Desktop browsers gets updated almost instantly world wide and they still fall prey for attacks, when mobile software lags for weeks or months at a time. The more people do their online banking on their phones instead of their desktops, it gets more valuable as a target. This problem needs a solution, and sooner rather than later.
gcb0 12y ago
Motorola and Google goes (dumbly, imho) completely opposite on this. they sell devices that are unlocked/developer edition, that are full of disclaimers that if you unlock boot loader some guaranties do no apply etc. and instead of providing h the new images first to this users to help then test, they usually roll out to the general population and weeks (Motorola, months) later make the images officially available. i guess there's two reasons, one is that they can get some money from operators to release first on one network or another, the second is that they have no consumer support so they couldn't care less. alternatively it's just honest incompetence.

by edent 12y ago

baq 12y ago

thank you for this post. i always felt that the other side of the fence was greatly underrepresented here.

TeMPOraL 12y ago
I second that. We always get the same stream of negative comments towards telcos (not saying they're not deserved), but we don't get to see very often how this looks on the other side of the fence. It'll be wonderful to have more comments like GP's one.

glass- 12y ago

> compared to millions of customers storming your shops and demanding refunds And yet somehow Apple is able to push updates without carrier approval and this doesn't happen.

mackey 12y ago
Except for 8.0.1 which hosed cell service for everyone. However, Apple was able to pull the update within hours and release a fix the next day.
- Alupis 12y ago
  Or 8.1.x that hosed most user's battery life (as in < 6 hours off a full charge) among a slurry of other issues.[1] [1] http://www.forbes.com/sites/gordonkelly/2014/12/09/apple-releases-ios-8-1-2/ http://www.forbes.com/sites/gordonkelly/2014/12/09/apple-rel...
- glass- 12y ago
  And millions of customers stormed shops? Which is the rationale that is provided for carriers taking months to approve updates for everyone but Apple.
  - mackey 12y ago
    I am not defending carriers here. I just wanted to point out that Apple's approach isn't perfect either.
- happyscrappy 12y ago
  >Except for 8.0.1 which hosed cell service for everyone. Only iPhone6 only if you updated a point release in the first hour, not that it is good.
dguaraglia 12y ago
What are you basing your statement on? Verizon and AT&T phones have been pegged at different minor versions of iOS many times. People want to believe Apple is some magic company that can do no wrong, but that ignores the extensive amount of QA (both in house and at the telcos) that happens before any release like this (as a longish time OS X user, I should know.)

joosters 12y ago

Surely these are issues to do with phone manufacturers and not with the carriers?

click170 12y ago

Wow, thanks for that interesting insight into the telco side of firmware upgrade delays. I humbly submit that the carriers could avoid a lot of ill-will if there was more transparency in that process. I hope there aren't contractual obligations in place preventing that because it really changes the way I personally see the problem.

magicalist 12y ago
They have always been fairly transparent in how this happens. The problem is they say they have to put these things through an extensive testing process before release...and then never release it. But hey, here's a new phone you can buy with the latest release!
edent 12y ago
Go to any carrier support forum / Twitter and they're usually pretty open about testing. But, yes, there are always contractual obligations. Not least, Manufacturer X doesn't want Carrier Y saying "Delays are due to..."

diminoten 12y ago

I had accidentally moved $1,000 from the wrong bank account into Google Wallet, and called customer service to try and figure out what I should do, because there wasn't going to be $1,000 in the account I had selected, but Google Wallet was giving me access to the funds already. The CS rep that responded was a nice guy, but had no clue what to do. We started spitballing together some possible ideas, and I suggested that we just let it go and Wallet will eventually figure out that the transfer didn't go through, and deduct the money back from my Wallet account. He thought that might be a good idea, but had no further ideas beyond that. That was Google Wallet, and that was $1,000 floating around. I can't imagine the level of customer service available to anyone who hasn't actually put money directly into the equation. Do they actually do that thing we all saw in The Internship, where they have random engineers field support calls? Because I felt like I was talking to a random engineer.

danielvinson 12y ago
I used to work at Google in support. You were not talking to a random engineer - the support people are all dedicated support contractors. Most of the teams are all non-technical and the only thing they know about how the product works is from a two week training where most of the time is spent on learning how to avoid lawsuits.
- gcb0 12y ago
  support people at Google is there only to fill regulation minimums.
  - danielvinson 12y ago
    The teams I was a part of most definitely were not just for minimums (Chromebook/Glass). Google has incredible support for their hardware products.
    - gcb0 12y ago
      that's honestly awesome to hear. my experience in the past made me avoid newer Google products because support in the past was non existent. last thing i bought was a nexus one. i for the bad touch screen (registers ghost touches after screen is on for a while) and i could either complain on forums and be ignored or start a return with htc that from the forums histories ended with a device on worse condition and save touch screen bug sent back...

drzaiusapelord 12y ago

The real issue here is that GSM, MMS, etc are archaic and hard to code for. The industry's fight against VOIP seems to be the real issue here. Instead of all this complexity, why not just treat phones as data devices and voice is just another IP protocol that runs on them? I imagine the "shoddy workmanship" has a lot to do with carrier specific settings and vendor specific stuff that is difficult to code for properly. A modern telecom is a shitshow of legacy junk that should be thrown out. Granted, this is kinda happening with some carriers and google voice, but the move to LTE-only with VOIP needs to happen sooner than later. The carriers seem to be fighting this because they, then, can't sell "minutes" and other old fashioned concepts. They would become a dumb internet pipe and they fear what that means from a profit perspective.

deleted 12y ago
[deleted]
noselasd 12y ago
I'm not sure why you assume a switch to VoIP will help in that regard - in practical terms that means the SIP protocol for the voice (and SMS) signalling. While SIP is going to be the way forward, the specs are really full of ambiguity and not that well though out features. Most commercial vendors of SIP servers include full scripting languages in order for the admins to "normalize" the protocol in order to try to smooth over incompabilities - I can guarantee you that when phones are released in the wild with a SIP stack the telco hasn't tested, things will stop working until they've implemented workarounds for the SIP flavor of the day.
- drzaiusapelord 12y ago
  SIP is software. What runs on my cellular modem and its stack is firmware written by yet another party. If SIP app #474 is screwing up, I switch to SIP app #475. This all can be done in userland as an application. Changing out my GSM stack? Yeah, I can't do that.
  - noselasd 12y ago

magicalist 12y ago

Apologies to the sibling posts praising this for a different perspective, but this post goes way too far in the other direction. Yes testing updates is very important. But by "operators delay firmware updates" do you mean never release updates? By "some of them" "stuff the ROM full of bloatware" do you mean nearly all of them? Telcos also love to write and ship their own OS UIs which are themselves full of shoddy workmanship, then never release updates for many of the phones running them. Listen, it is a shame the telcos get totally thrown under the bus when there's an entire supply chain full of issues, but I don't see what that has to do with this bug except in the sense that "there are bugs" not caused by the telcos.

edent 12y ago
It rather depends where you are in the world. UK operators are fairly good at releasing updates - but a lot depends on how many units the carrier sold. In the EU there's a trend away from bloatware. But, again, depends on your country / market.

xorcist 12y ago

On the other hand, by delaying an update containing important security fixes, you leave your customers sitting ducks for the next drive by download attack. We haven't seen any major botnets infecting mobile phones yet, but I think it is just a matter of time. Desktop browsers gets updated almost instantly world wide and they still fall prey for attacks, when mobile software lags for weeks or months at a time. The more people do their online banking on their phones instead of their desktops, it gets more valuable as a target. This problem needs a solution, and sooner rather than later.

gcb0 12y ago

Motorola and Google goes (dumbly, imho) completely opposite on this. they sell devices that are unlocked/developer edition, that are full of disclaimers that if you unlock boot loader some guaranties do no apply etc. and instead of providing h the new images first to this users to help then test, they usually roll out to the general population and weeks (Motorola, months) later make the images officially available. i guess there's two reasons, one is that they can get some money from operators to release first on one network or another, the second is that they have no consumer support so they couldn't care less. alternatively it's just honest incompetence.