14 ms·
Exploiting the IKKO Activebuds “AI powered” earbuds (2024)
- mikeve 1y agoI love how run DOOM is listed first, over the possibility of customer data being stolen.
- reverendsteveii 1y agoI'm taking >run DOOM as the new >cat /etc/passwd It doesn't actually do anything useful in an engagement but if you can do it that's pretty much proof that you can do whatever you want
- bigiain 1y agoPopping Calc! (I'm showing my age here, aren't I?)
- jcul 1y agoTo be fair (or pedantic), in this post they didn't have root, so cat'ing etc/passwd would not have been possible, whereas installing a doom apk is trivial.
- rainonmoon 1y ago/etc/passwd is world readable by default.
- kaszanka 1y agoTo be even more pedantic, it's also not present on Android.
- jcul 1y agoGood points, I've been out pedantic-ed!
- reverendsteveii 1y agonot if you fork an open source os and add /etc/passwd you haven't been
- pvtmert 1y ago> What the fuck, they left ADB enabled. Well, this makes it a lot easier. Thinking that was all, but then; > Holy shit, holy shit, holy shit, it communicates DIRECTLY TO OPENAI. This means that a ChatGPT key must be present on the device! Oh my gosh. Thinking that is it? Nope! > SecurityStringsAPI which contained encrypted endpoints and authentication keys.
- rvnx 1y agoIt’s the best privacy protecting way to send directly data rather than a proxy
- JohnMakin 1y ago“decrypt” function just decoding base64 is almost too difficult to believe but the amount of times ive run into people that should know better think base64 is a secure string tells me otherwise
- pvtmert 1y agonot very much surprising given they left the adb debugging on...
- crtasm 1y ago>However, there is a second stage which is handled by a native library which is obfuscated to hell
- zihotki 1y agoThat native obfuscated crap still has to do an HTTP request, that's essentially a base64
- qoez 1y agoThey should have off-loaded security coding to the OAI agent.
- java-man 1y agothey probably did.
- jcul 1y agoThe raw crypt data is base64 encoded, probably just for ease of embedding the strings. There is a decryption function that does the actual decryption. Not to say it wouldn't be easy to reverse engineer or just run and check the return, but it's not just base64.
- _carbyau_ 1y agoSo easy a fancy webpage could do it. https://gchq.github.io/CyberChef/ https://gchq.github.io/CyberChef/ I mean, it's from gchq so it is a bit fancy. It's got a "magic" option! Cool thing being you can download it and run it yourself locally in your browser, no comms required.
- neya 1y agoI love how they tried to sponsor an empty YouTube channel hoping to put the whole thing under the carpet
- JumpCrisscross 1y agoIf they were smart they’d include anti-disparagement and confidentiality clauses in the sponsorship agreement. They aren’t, though, so maybe it’s just a pathetic attempt at bribery.
- neya 1y agoThat was my first thought too
- dylan604 1y agoif you don't have a bug bounty program but need to get creative to throw money at someone, this could be an interesting way of doing it.
- komali2 1y ago> "and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you." Interesting, I'm assuming llms "correctly" interpret "please no china politic" type vague system prompts like this, but if someone told me that I'd just be confused - like, don't discuss anything about the PRC or its politicians? Don't discuss the history of Chinese empire? Don't discuss politics in Mandarin? What does this mean? LLMs though in my experience are smarter than me at understanding imo vague language. Maybe because I'm autistic and they're not.
- williamscales 1y ago> Don't discuss anything about the PRC or its politicians? Don't discuss the history of Chinese empire? Don't discuss politics in Mandarin? In my mind all of these could be relevant to Chinese politics. My interpretation would be "anything one can't say openly in China". I too am curious how such a vague instruction would be interpreted as broadly as would be needed to block all politically sensitive subjects.
- rvnx 1y agoThere is no difference to other countries. In France if you say bad things about certain groups of people then you can literally go to jail (but the censorship is directly IN the models)
- komali2 1y agoYou don't feel there's a difference between a State banning criticism of the State, and a State passing anti-hate speech laws to protect people from, e.g., nazis?
- throw10920 1y agoNo, there isn't a difference. "Hate speech" has no meaning, and laws purporting to be combatting it are actively used to prevent criticism of the State (e.g. in Germany).
- throwawayoldie 1y ago[flagged]
- Cthulhu_ 1y agoI wish earning money was as easy as setting rules for yourself, unfortunately that doesn't work.
- throwawayoldie 1y agoOh, that's fine, the rule's for everyone else, not me. I would be more likely to cut my own head off than willingly describe something as "AI-powered".
- j16sdiz 1y agocutting your head off won't earn you any money either.
- throwawayoldie 1y ago[citation needed]
- mmaunder 1y agoThe system prompt is a thing of beauty: "You are strictly and certainly prohibited from texting more than 150 or (one hundred fifty) separate words each separated by a space as a response and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you.” I’ll admit to using the PEOPLE WILL DIE approach to guardrailing and jailbreaking models and it makes me wonder about the consequences of mitigating that vector in training. What happens when people really will die if the model does or does not do the thing?
- reactordev 1y agoThis is why AI can never take over public safety. Ever.
- sneak 1y agohttps://www.wired.com/story/wrongful-arrests-ai-derailed-3-mens-lives/ https://www.wired.com/story/wrongful-arrests-ai-derailed-3-m... Story from three years ago. You’re too late.
- reactordev 1y agoI’m not denying we tried, are trying, and will try again… That we shouldn’t. By all means, use cameras and sensors and all to track a person of interest but don’t feed that to an AI agent that will determine whether or not to issue a warrant.
- aspenmayer 1y agoIf it’s anything like the AI expert systems I’ve heard about in insurance, it will be a tool that is optimized for low effort, but will be used carelessly by end users, which isn’t necessary the fault of the AI. In automated insurance claims adjustment, the AI writes a report to justify appealing patient care already approved by a human doctor that has already seen the patient in question, and then an actual human doctor working for the insurance company clicks an appeal button, after reviewing the AI output one would hope. AI systems with a human in the loop are supposed to keep the AI and the decisions accountable, but it seems like it’s more of an accountability dodge, so that each party can blame the other with no one party actually bearing any responsibility because there is no penalty for failure or error to the system or its operators.
- psim1 1y agoIndeed, brace yourselves as the floodgates holding back the poorly-developed AI crap open wide. If anyone is thinking of a career pivot, now is the time to dive into all things cybersecurity. It's going to get ugly!
- 725686 1y agoThe problem with cybersecurity is that you only have to screw once, and you're toast.
- 8organicbits 1y agoIf that were true we'd have no cybersecurity professionals left. In my experience, the work is focused on weakening vulnerable areas, auditing, incident response, and similar activities. Good cybersecurity professionals even get to know the business and tailor security to fit. The "one mistake and you're fired" mentality encourages hiding mistakes and suggests poor company culture.
- ceejayoz 1y ago"One mistake can cause a breach" and "we should fire people who make the one mistake" are very different claims. The latter claim was not made. As with plane crashes and surgical complications, we should take an approach of learning from the mistake, and putting things in place to prevent/mitigate it in the future.
- 8organicbits 1y agoI believe the thread starts with cybersecurity as a job role, although perhaps I misunderstood. In either case, I agree with your learning-based approach. Blameless postmortem and related techniques are really valuable here.
- deleted 1y ago[deleted]
- immibis 1y ago
- aidos 1y ago> “Our technical team is currently working diligently to address the issues you raised” Oh now you’re going to be diligent. Why do I doubt that?
- brahyam 1y agoWhat a train wreck, there are thousand more apps in store that do exactly this because its the easiest way to use openAI without having to host your own backend/proxy. I have spend quite some time protecting my apps from this scenario and found a couple of open source projects that do a good job as proxys (no affiliation I just used them in the past): - https://github.com/BerriAI/litellm https://github.com/BerriAI/litellm - https://github.com/KenyonY/openai-forward/tree/main https://github.com/KenyonY/openai-forward/tree/main but they still lack other abuse protection mechanism like rate limitting, device attestation etc. so I started building my own open source SDK - https://github.com/brahyam/Gateway https://github.com/brahyam/Gateway
- memesarecool 1y agoCool post. One thing that rubbed me the wrong way: Their response was better than 98% of other companies when it comes to reporting vulnerabilities. Very welcoming and most of all they showed interest and addressed the issues. OP however seemed to show disdain and even combativeness towards them... which is a shame. And of course the usual sinophobia (e.g. everything Chinese is spying on you). Overall simple security design flaws but it's good to see a company that cares to fix them, even if they didn't take security seriously from the start. Edit: typo
- mmastrac 1y agoI agree they could have worked more closely with the team, but the chat logging is actually pretty concerning. It's not sinophobia when they're logging _everything_ you say. (in fairness pervasive logging by American companies should probably be treated with the same level of hostility these days, lest you be stopped for a Vance meme)
- rvnx 1y agoNo, it was only in DAN mode
- oceanplexian 1y agoThis might come as a weird take but I'm less concerned about the Chinese logging my private information than an American company. What's China going to do? It's a far away country I don't live in and don't care about. If they got an American court order they would probably use it as toilet paper. On the other hand, OpenAI would trivially hand out my information to the FBI, NSA, US Gov, and might even do things on behalf of the government without a court order to stay in their good graces. This could have a far more material impact on your life.
- dubcanada 1y agoThat's rather naive, considering China has a international police unit, that is stationed in several countries https://en.wikipedia.org/wiki/Chinese_police_overseas_service_stations https://en.wikipedia.org/wiki/Chinese_police_overseas_servic...
- computerthings 1y ago[dead]
- gbraad 1y agoStrongly suggest you to not buy, as the flex cable for the screen is easy to break/come loose. Mine got replaced three times, and my unit now still has this issue; touch screen is useless. https://youtube.com/shorts/1M9ui4AHXMo https://youtube.com/shorts/1M9ui4AHXMo Note: downvote?
- deleted 1y ago[deleted]
- add-sub-mul-div 1y agoSure let's start giving out participation trophies in security. Nothing matters anymore.
- jon_adler 1y agoThe humorous phrase “the S in IoT stands for security” can be applied to the wearable market too. I wonder if this rule applies to any market with fast release cycles, thin margins and low barriers to entry?
- thfuran 1y agoIt pretty much applies to every market where security negligence isn't an existential threat to the continued existence of its perpetrators.
- ixtli 1y agoThis is one of the best things ive read on here in a long time. Definitely one of the greatest "it runs doom" posts ever.
- jahsome 1y agoIt's always funny to me when people go to the trouble of editorializing a title, yet in doing so make the title even harder to parse.
- lxe 1y agoThat's some very amateur programming and prompting that you've exposed.
- throwawayoldie 1y agoWhen the ZIRP era ended, I thought it would turn out to be a good thing for the industry, since it would wash out a lot of lightweights and incompetents. Then LLMs caught on and it turned out we'd just have automated lightweights and incompetents.
- wedn3sday 1y agoI love the attempt at bribery by offering to "sponsor" their empty youtube channel.
- Jotalea 1y agoReally nice post, but I want to see Bad Apple next.
- Liquix 1y agogreat writeup! i love how it goes from "they left ADB enabled, how could it get worse"... and then it just keeps getting worse > After sideloading the obligatory DOOM > I just sideloaded the app on a different device > I also sideloaded the store app can we please stop propagating this slimy corporate-speak? installing software on a device that you own is not an arcane practice with a unique name, it's a basic expectation and right
- efilife 1y agoI agree. It's the same as calling a mobile OS a ROM
- userbinator 1y agoThat term at least has a history behind it, as many featurephones had their OS on a small XIP NOR flash ROM, and now the OS is usually (mostly) read-only. But "sideloading" is definitely a new term of anti-freedom hostility.
- efilife 1y agoI expected someone to say this. In my opinion if there's some history behind something stupid it doesn't make it not stupid Btw interesting stats here https://trends.google.com/trends/explore?date=all&q=%2Fm%2F04yljl https://trends.google.com/trends/explore?date=all&q=%2Fm%2F0...
- jekwoooooe 1y agoGood write up. At some point we have to just seize these Chinese malware adjacent crap at the borders already
- lysace 1y agoThis is marketing.
- 44za12 1y agoAbsolutely wild. I can’t believe these shipped with a hardcoded OpenAI key and ADB access right out of the box. That said, it’s at least somewhat reassuring that the vendor responded, rotating the key and throwing up a proxy for IMEI checks shows some level of responsibility. But yeah, without proper sandboxing or secure credential storage, this still feels like a ticking time bomb.
- lucasluitjes 1y agoHardcoded API keys and poorly secured backend endpoints are surprisingly common in mobile apps. Sort of like how common XSS/SQLi used to be in webapps. Decompiling an APK seems to be a slightly higher barrier than opening up devtools, so they get less attention. Since debugging hardware is an even higher threshold, I would expect hardware devices this to be wildly insecure unless there are strong incentive for investing in security. Same as the "security" of the average IoT device.
- bigiain 1y agoEventually someone is going to get a bill for the OpenAPI key usage. That will provide some incentive. (Incentive to just rotate the key and brick all the devices rather than fix the problem, most likely.
- eru 1y ago> (Incentive to just rotate the key and brick all the devices rather than fix the problem, most likely. But that at least turns it into something customers will notice. And companies already have existing incentives for dealing with that.
- bigiain 1y agoAt that stage you just rotate the company name or branding...
- 1y ago
- sim7c00 1y agoearbuds that run doom. achievement unlocked? (sure adb sideload, but doom is doom) nice writeup thanks!
- JumpCrisscross 1y agoA fair consumer protection imperative might be found in requiring system prompts and endpoints be disclosed. This is a good example to kick that off with, as it presents a national security issue.
- bytesandbits 1y agoPhenomenal write up I enjoyed every bit of it
- 1oooqooq 1y agomaking fun of a company amateur tech while posting screenshots of text is another level of lack of self awareness
- rvnx 1y agoIt’s also illegal to try to hack into their backend and access restricted data, so he should be happy actually that this company has little presence in the US
- p1necone 1y agoTheir email responses all show telltale signs of AI too which is pretty funny.
- paul-tharun 1y agoI think it has to do with language barrier and translation
- RataNova 1y agoHonestly, the most surprising part is that they eventually rotated the key
- sahil_sharma0 1y ago[dead]