9 ms·

> fighting Broadcom's old, god-awful bluetooth code Correction: god-awful host side bluetooth code. There is still the bluetooth firmware residing on the BCMx

> fighting Broadcom's old, god-awful bluetooth code Correction: god-awful host side bluetooth code. There is still the bluetooth firmware residing on the BCMxxx chip (or Qualcomm chip) - >1MB of god-awfulerer closed-source code, half of it is in ROM (with limited number of available patch slots), full of bugs. You can see it crash from time to time in the kernel debug logs (and auto-restart itself)

FredFS456 6y ago
For more illustrations of bugs/vulns in the firmware: https://www.youtube.com/watch?v=7tIQjPjjJQc&t=1986s https://www.youtube.com/watch?v=7tIQjPjjJQc&t=1986s
kevincox 6y ago
Do you have any more info about ROM patch slots? I have never heard of this before. I assume this is a small amount of r/w memory that is somehow overlaid over specific locations in the ROM?
- parwan98 6y ago
  Correct. It's a small table of: address1, 4 bytes overlay data address2, 4 bytes overlay data etc The data is overlayed over the specified addresses, in runtime. On some chips its 8 bytes instead of 4. On a typical Broadcom/Cypress chip you have 128 or 256 entries. By the time the chip is 2-3 years in the market and still getting firmware updates, ~98% of them are used by existing firmware, so there are only 5-10 free entries by the time the chip is considered "obsolete". Case in point: the Broadcom/Cypress BCM43455 chip on the raspberry pi is almost out of patch entries. Broadcom have switched to their usual tactic of stalling for years on known, reproducible bug reports.
  - ed25519FUUU 6y ago
    > Case in point: the Broadcom/Cypress BCM43455 chip on the raspberry pi is almost out of patch entries. Broadcom have switched to their usual tactic of stalling for years on known, reproducible bug reports. And it's still really buggy. I had to write a service on the RPI and the only way to reliably connect was to restart bluetooth before every attempt. That kind of fix makes a person feel dirty.
    - mondoshawan 6y ago
      Such is the sad world of Bluetooth. The dirty secret to this industry is that this, while seeming hacky, is the bare minimum de-facto standard in most cases.
    - parwan98 6y ago
      Sadly that's common in the hardware world. Step 1. Have a reliable hardware watchdog that restarts everytime there's a software problem. Step 2. There is no step 2.
  - eecc 6y ago
    So given these data points, isn’t it reasonable for Apple to refuse to play along this broken tune and just roll out their own dialect of a wireless protocol? Why, if not in the name of scarcely affirmed “standards”, drag suppliers through an endless contractual game, when you can direct your own capacity toward the quality standards that fit you?
    - realityking 6y ago
      I don’t follow the leap. The grandparent’s point was about the quality and terrible lack of long term support of Broadcom chips. How does that translate to issues of the standard itself? Nobody would complaining about Apple creating their own radio chips (which they seem to plan for 5G/6G). Apple creating their own standard protocols is an issue though.
      - eecc 6y ago
        Well, if the implementations of the standard are such a garbage fire what's the point chasing them? Just to check the box "standards compliant" and likely providing an abysmal UX and poor interoperability? I fixated on Apple because they're often picked on for taking the highway, but on the other hand what's the point doing otherwise? What's a common ground if it's just a pipe dream?
        azernik 6y ago
        Just because a chip is shitty doesn't mean it's worthless. In practice, Bluetooth is quite interoperable, and reliable enough for many use cases (especially the common, better-tested ones). Breaking compatibility with that ecosystem out of spite is not conducive to getting adoption for a better product.
        eecc 6y ago
        Well, the original posts report some frankly tragic scenarios - so bad that they “reboot to initialize” just to keep sane - in what are some pretty ubiquitous devices. Or not?
  - exikyut 6y ago
    The real brokenness here seems to be that the chips are not engineered with say ten times the patch capacity. And the root of the brokenness is that there isn't the end-to-end awareness and acceptance that ten times the capacity is obviously needed. Owww.
    - SwaraLink 6y ago
      Easy to say, and I can’t know for sure exactly what factored impacted Broadcom’s decisions here, but I can tell you that chip manufacturers are under extreme pressure to keep costs down, which means that they may under-spec systems at times. Also, with the long design cycles involved in chip design the patch capabilities may have been decided years in advance, before realizing how much would be needed. In general I agree with your comment, though it’s a lot easier to say this in hindsight.
mondoshawan 6y ago
On Glass, we actually went to Broadcom and had them on the ropes for fixing parts of their firmware. Sadly, we couldn't bring those fixes out of the closed source world of hardware, so it's still up to the system integrator to fight those battles...
- qbasic_forever 6y ago
  Serious question, why doesn't Google build its own bluetooth & BLE chips? Please put some competition on Broadcoam and the like and either push them entirely out of the market (good riddance), or force them to step up their game.
  - londons_explore 6y ago
    Building radio ASICs is a world rife with patents. Pretty much nobody new can enter the market.
    - mandelken 6y ago
      Care to expand? Is that why Software Define Radio is still so much a niche and expensive?
      - londons_explore 6y ago
        Nearly all modern radio chipsets are mostly software defined. That includes WiFi, LTE and GPS. The radio frontend is typically a downmixer and then straight into digital. Some of the typically "software" bits like FFT's, various encodings, checksums, clock recovery etc. are frequently done in digital hardware acceleration blocks for performance, and saving power. If you were writing the firmware of the device, you needn't use them though. With enough human years of effort, you could take almost any radio hardware for sale today and repurpose it to speak nearly any other radio protocol in similar frequency bands. Performance will probably be terrible though! It's rare people do this though - all the chips don't have their firmware documented (again mostly to avoid publishing documentation that proves they are violating someone elses patents), and many have various cryptographic elements that makes reverse engineering hard. The one exception to this is WiFi chips used in the Nexus 5 by Broadcom, which has had a reasonable amount of reverse engineering because Broadcom accidentally published the source code because the firmware code was in part shared with published Linux kernel driver source code.
deleted 6y ago
[deleted]
Abishek_Muthian 6y ago
Even if some courageous developer there fixes the bugs and updates the firmware; how many end-users would actually receive the update and actually apply the update? That's the problem Linux's LVFS[1] solves but it's unfortunate that not all manufacturers support it. I got update for my half a decade old Logitech's 2.4 GHz receiver (nRF24L) for wireless keyboard as soon as I plugged it on Linux, I've used the same keyboard on Mac and the official Logitech software doesn't even detect the device properly let alone update the receiver's firmware(no issues using the device though). [1] https://fwupd.org/ https://fwupd.org/
Twirrim 6y ago
Broadcom's firmware seems to be just absolutely terrible across the stack, NIC included. They seem to have solid product design / engineering chops, but firmware just defies them.
scoutt 6y ago
My question would be what were the motivations to move into a stack written in Rust, if the much of the bugs are in the closed source FW running in the peripheral? What would happen if consistency is lost outside the Rust domain but still in the BT stack?

FredFS456 6y ago

For more illustrations of bugs/vulns in the firmware: https://www.youtube.com/watch?v=7tIQjPjjJQc&t=1986s https://www.youtube.com/watch?v=7tIQjPjjJQc&t=1986s

kevincox 6y ago

Do you have any more info about ROM patch slots? I have never heard of this before. I assume this is a small amount of r/w memory that is somehow overlaid over specific locations in the ROM?

parwan98 6y ago
Correct. It's a small table of: address1, 4 bytes overlay data address2, 4 bytes overlay data etc The data is overlayed over the specified addresses, in runtime. On some chips its 8 bytes instead of 4. On a typical Broadcom/Cypress chip you have 128 or 256 entries. By the time the chip is 2-3 years in the market and still getting firmware updates, ~98% of them are used by existing firmware, so there are only 5-10 free entries by the time the chip is considered "obsolete". Case in point: the Broadcom/Cypress BCM43455 chip on the raspberry pi is almost out of patch entries. Broadcom have switched to their usual tactic of stalling for years on known, reproducible bug reports.
- ed25519FUUU 6y ago
  > Case in point: the Broadcom/Cypress BCM43455 chip on the raspberry pi is almost out of patch entries. Broadcom have switched to their usual tactic of stalling for years on known, reproducible bug reports. And it's still really buggy. I had to write a service on the RPI and the only way to reliably connect was to restart bluetooth before every attempt. That kind of fix makes a person feel dirty.
  - mondoshawan 6y ago
    Such is the sad world of Bluetooth. The dirty secret to this industry is that this, while seeming hacky, is the bare minimum de-facto standard in most cases.
  - parwan98 6y ago
    Sadly that's common in the hardware world. Step 1. Have a reliable hardware watchdog that restarts everytime there's a software problem. Step 2. There is no step 2.
- eecc 6y ago
  So given these data points, isn’t it reasonable for Apple to refuse to play along this broken tune and just roll out their own dialect of a wireless protocol? Why, if not in the name of scarcely affirmed “standards”, drag suppliers through an endless contractual game, when you can direct your own capacity toward the quality standards that fit you?
  - realityking 6y ago
    I don’t follow the leap. The grandparent’s point was about the quality and terrible lack of long term support of Broadcom chips. How does that translate to issues of the standard itself? Nobody would complaining about Apple creating their own radio chips (which they seem to plan for 5G/6G). Apple creating their own standard protocols is an issue though.
    - eecc 6y ago
      Well, if the implementations of the standard are such a garbage fire what's the point chasing them? Just to check the box "standards compliant" and likely providing an abysmal UX and poor interoperability? I fixated on Apple because they're often picked on for taking the highway, but on the other hand what's the point doing otherwise? What's a common ground if it's just a pipe dream?
      - azernik 6y ago
        Just because a chip is shitty doesn't mean it's worthless. In practice, Bluetooth is quite interoperable, and reliable enough for many use cases (especially the common, better-tested ones). Breaking compatibility with that ecosystem out of spite is not conducive to getting adoption for a better product.
        eecc 6y ago
        Well, the original posts report some frankly tragic scenarios - so bad that they “reboot to initialize” just to keep sane - in what are some pretty ubiquitous devices. Or not?
- exikyut 6y ago
  The real brokenness here seems to be that the chips are not engineered with say ten times the patch capacity. And the root of the brokenness is that there isn't the end-to-end awareness and acceptance that ten times the capacity is obviously needed. Owww.
  - SwaraLink 6y ago
    Easy to say, and I can’t know for sure exactly what factored impacted Broadcom’s decisions here, but I can tell you that chip manufacturers are under extreme pressure to keep costs down, which means that they may under-spec systems at times. Also, with the long design cycles involved in chip design the patch capabilities may have been decided years in advance, before realizing how much would be needed. In general I agree with your comment, though it’s a lot easier to say this in hindsight.

mondoshawan 6y ago

On Glass, we actually went to Broadcom and had them on the ropes for fixing parts of their firmware. Sadly, we couldn't bring those fixes out of the closed source world of hardware, so it's still up to the system integrator to fight those battles...

qbasic_forever 6y ago
Serious question, why doesn't Google build its own bluetooth & BLE chips? Please put some competition on Broadcoam and the like and either push them entirely out of the market (good riddance), or force them to step up their game.
- londons_explore 6y ago
  Building radio ASICs is a world rife with patents. Pretty much nobody new can enter the market.
  - mandelken 6y ago
    Care to expand? Is that why Software Define Radio is still so much a niche and expensive?
    - londons_explore 6y ago
      Nearly all modern radio chipsets are mostly software defined. That includes WiFi, LTE and GPS. The radio frontend is typically a downmixer and then straight into digital. Some of the typically "software" bits like FFT's, various encodings, checksums, clock recovery etc. are frequently done in digital hardware acceleration blocks for performance, and saving power. If you were writing the firmware of the device, you needn't use them though. With enough human years of effort, you could take almost any radio hardware for sale today and repurpose it to speak nearly any other radio protocol in similar frequency bands. Performance will probably be terrible though! It's rare people do this though - all the chips don't have their firmware documented (again mostly to avoid publishing documentation that proves they are violating someone elses patents), and many have various cryptographic elements that makes reverse engineering hard. The one exception to this is WiFi chips used in the Nexus 5 by Broadcom, which has had a reasonable amount of reverse engineering because Broadcom accidentally published the source code because the firmware code was in part shared with published Linux kernel driver source code.

deleted 6y ago

[deleted]

Abishek_Muthian 6y ago

Even if some courageous developer there fixes the bugs and updates the firmware; how many end-users would actually receive the update and actually apply the update? That's the problem Linux's LVFS[1] solves but it's unfortunate that not all manufacturers support it. I got update for my half a decade old Logitech's 2.4 GHz receiver (nRF24L) for wireless keyboard as soon as I plugged it on Linux, I've used the same keyboard on Mac and the official Logitech software doesn't even detect the device properly let alone update the receiver's firmware(no issues using the device though). [1] https://fwupd.org/ https://fwupd.org/

Twirrim 6y ago

Broadcom's firmware seems to be just absolutely terrible across the stack, NIC included. They seem to have solid product design / engineering chops, but firmware just defies them.

scoutt 6y ago

My question would be what were the motivations to move into a stack written in Rust, if the much of the bugs are in the closed source FW running in the peripheral? What would happen if consistency is lost outside the Rust domain but still in the BT stack?