8 ms·
Fixing an Elgato HD60 S HDMI capture device with the help of Ghidra
- deleted 2y ago[deleted]
- Topgamer7 2y agoWhat a well written reverse engineering project :)
- jwong_ 2y agoReally enjoyed this write up. You both had the skill and the patience to even buy a working version to fix the broken ones!
- sydbarrett74 2y agoWe need to encourage our kids to repair stuff. It aids the environment (less e-waste) and teaches them a valuable trade. I know that's it's typically cheaper to replace than repair, but to me that's a market failure.
- not_your_vase 2y agoMost devices nowadays are unfortunately not only not repair-friendly, they are straight impossible to take disassemble. They are welded, soldered, glued together, and any attempt at a repair is intentionally destructive :( This one is an odd exception, that usually happens only with the early versions of a product.
- addandsubtract 2y agoGermany is rolling out an incentive to get products repaired. For example, repairs that can be done by yourself or in a repair café will be subsidized by €200 and repairs that require a professional or sending in, will be subsidized by 50% up to €200.
- PUSH_AX 2y agoI agree with the sentiment, but in this example, how much domain expertise was required?
- pjc50 2y ago> market failure Industrialization has massively optimized the "hot path" for any kind of production, largely by taking out the humans. It is incredibly cheap to manufacture consumer electronics - because everything is standardized and predictable and amenable to machine operation. This is what market success looks like. The downside is that as soon as there is a deviation from the process, it gets more expensive. It's worth thinking about what happens to repair inside the factory. Some percentage of units will be coming off the production line defective and fail initial QA. This is usually a very low percentage, well below 1%, because the entire economics of the factory depends on not having to do any rework. Even there, the technicians will take a look, determine if it's something that can be fixed quickly, or is a novel kind of failure, and if not just throw it away right there. Why? Because while you're standing there looking at it, another hundred have come off the production line. The broken one in your hand is not a unique snowflake. At the start of a run though, this is an interesting and important job, because any failure is a novel failure. The first batch through the process should have high yield, but it might not, and then you stop the line ( https://mag.toyota.co.uk/andon-toyota-production-system/ https://mag.toyota.co.uk/andon-toyota-production-system/ ) and figure out what's happened (something misaligned? Defective inputs? Material problems? Design issue?) The first batch will often get reworked and sent out as demo units.
- KeplerBoy 2y agoWhat happens when a unit is defective and not easily repairable, maybe because the PCB itself was already defective? Do technicians salvage the high value components, like the CPLD in this case, off the board? That chip alone is probably worth 15$.
- ElectricalUnion 2y agoThe engineering time to "salvage" and test the 15$ chip, and the cost of downtime reintegrating the part to the production line is probably worth more that 15$, so unless the lack of that specific part is a bottleneck, probably no?
- 2y ago
- Razengan 2y agoI appreciate the intent, but.. Do you suppose time is free? and renewable? Time that could be spent doing other things instead of learning how to repair, buying the tools, and repairing something that you could just hire someone else to repair, or buy anew? As for environmental waste, figure out ways to make stuff out of more easily degradable stuff, or to reuse it, instead of _guilting_ people into repair.
- blitzar 2y agoA robust economy for parts and repairs would be ideal. Parts are (reasonably) obtainable, however, professional or even decent repair is about as reliable as a used car salesman.
- pjc50 2y agoTo repurpose a sentence from somewhere else, repair is only free if your time is worthless. (Imagine trying to contract someone to do the OP's level of thorough work; it would be a five-figure dollar sum)
- guadalupe4151 2y ago[dead]
- varispeed 2y agoUnfortunately, due to lobbying, it's now less of a trade but skill that corporations exploit for profit. In Western countries working on your own account is getting more and more restricted under guise of tackling tax avoidance (many tax authorities engage in anti-small business propaganda), so that the profit people generate is captured by corporations they have to work at if they want to pursue their "trade". edit: I find it fascinating that working class supports this, even though it is against their interest. It has a lot to do with crabs in the bucket mentality and the cultivated perception that venturing out of ones lane is wrong. Most people think they need to serve the rich masters and that is their calling and reject the idea that concept of class has been created to keep them in mental captivity.
- the_biot 2y agoIf the US gets into a full-blown trade war with China, i.e. massive tariffs on everything, I expect consumer electronics, clothing etc to become much more interesting to repair instead of replace.
- deleted 2y ago[deleted]
- XorNot 2y agoThis repair makes the case I've found more obvious though: we need microcontroller firmwares to be made available (I would argue as a public government archive honestly). I've had several Yamaha amps blow micro-processors or DSPs, and on the older ones there were no firmware upgrades so you just plain couldn't do anything afterwards. I've got the same problem with the controller for my heat exchange ventilation - microcontroller is dead, and while the chip is easy to get and replace, there's 0 chance I can source a replacement firmware for it.
- russdill 2y agoThe reason it likely goes into bootloader when the lock bit isn't set is that it's likely using this as a "flashed successful" signifier.
- echoangle 2y agoThats explicitely stated in the post: Interestingly I discovered that if the chip is set for unprotected mode, the bootloader actually waits around for update commands instead of booting the application firmware. So Elgato is using the locked/unlocked bit as a way of signifying whether it should stay in the bootloader or not, which I find to be a little bit weird. That completely explained why my second HD60 S stopped working after I installed my unlocked firmware — it was stuck in the bootloader waiting forever because the chip was unprotected.
- flimflamm 2y agoThanks for the writeup! It was great as you explained the tools and rabbit holes you went it to.
- mschuster91 2y agoElgato stuff is cheap but horribly unreliable in my experience. Thanks for the writeup, it's showing why that is very detailed...
- db48x 2y agoThe flashing and corruption problems point to a very poor software engineering culture.
- KeplerBoy 2y agoHow does the corruption point to a software engineering issue? What could cause corruption and how could it be avoided by better software engineering?
- mschuster91 2y agoIt's a culture issue when such cases don't get caught in testing and support staff either doesn't know about it (=the scripts are bad), doesn't get told that there is a workaround (because clearly there is), or (the worst of the possible options) gets told to act like everything is fine.
- crote 2y agoIt's a race condition which involves physical interaction with the product, which only occurs during a very rare operation, which looks pretty much identical to a genuine hardware defect. This isn't something you can just trivially unit test. If you don't see this happen multiple times during initial hardware development, you are never catching it. A single failure of a prototype can easily be attributed to a manufacturing defect - especially if it was hand-soldered. Once it's in the field replacing the 0.01% of units suffering from random issues under warranty is far cheaper than having an engineer spend weeks trying to diagnose every single weird failure mode. Unless it affects a number of units, it's just not worth the money. You have to consider that support doesn't get a "the LEDs stopped working when I unplugged the device immediately after flashing it" message, they just get "the LEDs don't work". Support scripts are made for horses, not zebras.
- KeplerBoy 2y agoWhat a humbling writeup. This is real fullstack engineering.
- beachy 2y agoHumbling all right. I think it would take me weeks to achieve this - and that's assuming I had this article to start from!
- Cthulhu_ 2y agoI suspect it may have taken the author weeks as well, given they indicated waiting for chips from China and that there was communications with Elgato's customer support. This writeup looks like it's the report of a few weeks if not months of on and off work.
- dougg3 2y agoI'm the author of this article and you're absolutely correct! This was a long, drawn-out project. For some context, I ordered the replacement regulators in February. The new LED driver chips were ordered in March, so that was around the time that I actually had the failed hardware fixed. Then everything sat idle for months. The firmware reverse engineering to figure out the LEDs was several weeks of on and off work in my spare time. This type of thing is definitely not something you can just figure out in a couple hours (or even days).
- robocat 2y agoYour writeup is inspiring. I most appreciate the honesty about mistakes and failures and dead-ends. Too often wasted effort and small failures are elided from articles and instead you get something that is written retrospectively as though everything went perfectly...
- dougg3 2y agoThanks! I wouldn't want to write it any other way. I think it helps convey just how much work it really was.
- rkachowski 2y agoamazing work! ultimately it comes down to a firmware bug - I would not have had the fortitude to continue when it came to debugging
- fhackenberger 2y agoWould love to see Elgatos response to your writeup :-)
- squarefoot 2y agoSame here. I can totally imagine the excitement in their engineers eyes after reading this fascinating writeup, paired with their managers anger while they ask for a better way to protect the company's IP.
- KeplerBoy 2y agoNow i am curious if they know what causes their LED troubles or what causes the underlying memory corruption.
- 1-6 2y agoIt’ll have to come from Corsair who now owns Elgato. IMHO, they’re a ‘conventional’ company with stock listed on the exchange. If they respond I will be pleasantly surprised.
- aftbit 2y agoBuy one share of their stock, then call investor relations and ask them why a hobbyist was able to fix the LED bug that their support claimed didn't exist while their whole expensive engineering team was not.
- carlmr 2y ago>while their whole expensive engineering team was not Question is maybe not if they could, but if they were allowed to. If the project is already out of development and a bug is detected, sometimes there's only a barebones team left that barely gets the time to do basic maintenance.
- ghoomketu 2y agoGrowing up in India about 20 years ago, we often repaired or renewed almost everything because our buying power was low and things were expensive. We used a lot of hacks, known as *jugaads*, to make things work. Even clothes were reused, with tailors doing *rafu* (patchwork) to extend their life. This was especially common in middle-class homes like mine. My dad, who worked in a garment export house, used to tell me stories about how people in the West preferred disposable items and often opted for newer stuff, whether it was cars, gadgets, or clothes. At the time, I didn't understand this mentality. But now, with increased buying power and lower costs (thanks to China), we too tend to just chuck things away and get replacements. I deeply admire people who don't give up midway and think it's easier to buy new. This type of persistence and resourcefulness is truly commendable.
- Cthulhu_ 2y agoThis was how things went for a long time in my region (western europe) as well, my parents grew up patching clothes and repairing stuff a lot. It's only in the past 50 years or so that consumerism has gone up and the quality and cost of e.g. clothing has gone down. I've been doing maintenance on my motorcycle myself recently, it does take some small investments in some tools to get started (like a tool to undo the oil filter, although in hindsight a strap and a stick would do the job) and you need to source some parts and replacements (fluids, copper washers, but also replacement screws for the weathered brake fluid reservoir ones), but it's in the region of €100-€150 instead of the €1000 the garage quoted me for.
- ghoomketu 2y agoYes Reflecting on it, making things last longer had some great side effects. For instance, almost every woman in my family knew how to *rafu* clothes (1), and people understood how things worked under the hood of a car (like you my father did all the maintenance too). These skills were passed down through generations, becoming a part of our everyday knowledge. I guess a lot of things aren't that simple or accessible as most of it is often a black box nowadays. But anyway, Skills like these not only saved money but also fostered a sense of self-reliance, resourcefulness and stuff your parents taught you as life skills. (1) https://m.youtube.com/results?search_query=Rafu+clothes https://m.youtube.com/results?search_query=Rafu+clothes
- londons_explore 2y agoIf the right to repair movement succeeds fully, OP would have had access to the complete source code and hardware schematics of this device, and the fix for his flashing lights probably would have been a 5 line shell script that someone else had already put on github.
- Aurornis 2y ago> would have had access to the complete source code and hardware schematics Forcing every company to open source their products is a pipe dream. This is never going to happen. Even if one country passed such a law, manufacturers would move their production to a different country. The products would then be imported and redistributed as foreign products, circumventing the law.
- londons_explore 2y agoPeople would have said the same 100 years ago about forcing food companies to reveal their ingredients list. Yet now it is required in almost every country, and doesn't appear to have been that detrimental to the food industry.
- dannyw 2y agoIngredients listing don't tell everything about how it is made, etc. A closer analogy might be detailed API specs.
- immibis 2y agoFor electronics it would probably show the part number and position of each component on the PCB, without showing how they are connected.
- brokenmachine 2y agoEven just knowing all the components would be a step in the right direction.
- userbinator 2y agoIt was hard to understand their purpose in the grand scheme of things though. Sometimes they were buried deep inside multiple layers of function calls. I must say this is often true even when I'm looking at the source code. I think the vast majority of software out there is heavily overabstracted, and MCUs have gotten big and cheap enough (two ARM cores in an HDMI capture device!?) that this is appearing in embedded stuff too. This device shouldn't have needed more than one MCU.
- ryukoposting 2y agoGreat work! I find it bizarre that they even bothered with this wild SPI Flash muxing thing if all it does is drive some LEDs. Seems massively overengineered.
- aftbit 2y agoGreat writeup, with very handy links!
- aftbit 2y ago>A light bulb went off in my head. Yes, these chunks of data in the flash chip were pictures! But not in the conventional sense. They were describing the various LED animations. Each set of 16 bytes is an animation frame written to the 14 LEDs. In the example above, a single red LED turns on and six of the seven white LEDs do stuff. The last two bytes in each row are a delay time before moving onto the next frame. The first 16 bytes looked kind of weird though. They looked like they possibly contained some header data for the animation or something. I have independently invented this format, albeit in text instead of binary, for animating my Hue lamps. Animations in my version look like this: bulb1 bulb2 dly R G B WW CW R G B WW CW 100 10 25 20 0 0 25 10 20 0 0 200 10 50 20 0 0 50 10 20 0 0 where the first value is the number of milliseconds to delay before moving to the next line, then each led channel's brightness (out of 100) is represented on each line. A line that starts with anything but a number is a comment, and any number of spaces (1+) are allowed between channels. I'm tickled to see a similar (binary) format in a commercial product. It really is quite obvious, but still interesting to see.
- sitkack 2y agoThis is like tracker software, or player piano rolls for LED animation. https://en.wikipedia.org/wiki/Piano_roll https://en.wikipedia.org/wiki/Piano_roll
- jauntywundrkind 2y agoThe ability to explore & chase down truths is what makes us divine, is the aspect of us that builds & enhances. This guy walked a deep deep spiritual path to exploring something of the mundane world. It's just so sad, so frustrating, so hellish & infernal that devices do resist us. Computers & devices should be amplifying & radiating the truths they are made of. But humanity is trapped in shells of their own making, in descending consumerism that rejects engagement & learn ability. Computers, alas, especially so, are often the prime obfuscater of truth and understanding! What a vast disuse of so much potential! I hope we see some break through currents, of machines that are open, emerge in my lifetime. Well done, this dude Doug.
- dpedu 2y agoIf you don't have a thermal camera, you can also paint rubbing alcohol onto the board and watch where it evaporates the quickest. Obviously, this only works for low voltage stuff...
- leloctai 2y agoHow do people maintain motivation on projects like this? Repeatedly having to order parts and wating for them to arrive, especially the one from China, make me abandoned an unfortunate amount of projects.
- carlmr 2y agoThe time, and also the cost of ordering these parts. I know each is like a few cents, but you don't know all the parts you will need beforehand. So if I counted correctly he ordered at least 3 times. It's fascinating nonetheless.