7 ms·
Bankrupt Fisker says it can't migrate its EVs to a new owner's server
- sverhagen 2y agoIt would be nice to understand what "can't" means in this context. Are they not technically able to achieve it (which may be overcome by bringing in other technical experts), is it a DNS issue (ha-ha), or is there a licensing issue with some component they integrated with?
- mlhpdx 2y agoLikewise, is it strictly "we can't" or "you can't, either" -- the former would be a legal issue whereas the latter might be an tech/IP issue (ownership of PKI and, yes, DNS).
- jcgrillo 2y agoYeah... > Fisker informed American Lease that the Oceans "cannot, as a technical matter, be 'ported' from the Fisker server to which the vehicles are currently linked to a distinct server owned and/or controlled by" American Lease. "as a technical matter" is doing a lot of work in that sentence... I have questions.
- abound 2y agoThe first thing that came to my mind was "cert pinning and someone lost the private keys", but that seems pretty unlikely in the grand scheme of things that can go wrong. I'd actually be willing to pick one of these up for a 70%+ discount if it came with all the internal schematics and source code, server migrations be damned.
- zie 2y agoAs buggy as their software seems to be, I'd be a little surprised if they worked that hard and actually did it reliably.
- sam1r 2y agoRight, "technically" not theoretically, it is absolutely still possible. This almost always alludes politics.
- deleted 2y ago[deleted]
- randerson 2y agoI've can't imagine any technical situation short of "we lost the source to the in-car software, which is pinned to a certificate only installed on our server, which only one guy had the password to, and he kept it only in his memory, and he died".
- gjsman-1000 2y agoIronically, the best solution if that was the case, would be to hire some hackers and hack your own car. Find a kernel exploit, find a power glitching attack against the CPU, find a USB exploit, find a Bluetooth exploit, find a Recovery mode exploit, whatever it may be; and build a physical harness which consistently runs the exploit and modifies the signing certificate. Nasty but effective and probably cheap. Or, even, kills the main GUI and replaces it with your own. If I was American Lease, I would look into that, besides suing. Edit: Looking into it, it’s Android Automotive without Google Services. Who wants to bet that it’s an old version of Android on a recycled MediaTek tablet processor which is no longer getting security patches? Knowing MediaTek, completely reprogramming Secure Boot might not be off the table.
- jcgrillo 2y agoThere's also the option of the ethical version of the $5 wrench hack--hiring the (recently unemployed) people who actually built this stuff.
- gjsman-1000 2y agoAssuming they still have the signing keys, or knowledge of how the build servers worked… or even legal access to the source code, which would require a negotiation / lawsuit against Fisker that could take years. That’s assuming Fisker hasn’t already liquidated that material.
- jcgrillo 2y agoThose are good points, but even hiring someone with knowledge as a consultant could be valuable in framing that case--knowing what you actually need is important.
- davesmylie 2y agoI read that as "technically we can't do this as we don't have the money"
- Palomides 2y agoI strongly suspect everyone who knew their infrastructure was laid off (or quit), so there's no chance of a technical handoff
- deleted 2y ago[deleted]
- sundbry 2y agoSure there is, if you let an outside expert read the source code for a week.
- Our_Benefactors 2y agoOh sweet summer child.
- saulrh 2y agoIt's probably way more complicated than that. Ever seen the cold start disaster recovery procedure for a big system with identity and encryption-at-rest and message busses involved? You might be lucky if the bring-up doesnt have any individual stages that take a week to quiesce all by themselves. I know that this system probably isn't all that big, but if I assume their server-side software is as low-quality as their embedded software, I can easily imagine it being that complex and interdependent and poorly documented.
- chipdart 2y ago> It's probably way more complicated than that. I once saw a small team of FANG engineers, that included two well seasoned senior engineers, revive a project left unmaintained for two years after the owning team was disbanded. That small team took two weeks alone to get the project to build and run locally, with tons of bits missing. But hey, if a random anonymous internet expert says that all it takes to revive a project is a week of browsing through the source code then that must be true.
- 2y ago
- emmelaich 2y agoYes, 'technically' sounds like bullshit. It could be something as simple as the server ip address in each car's firmware. That's annoying but not insurmountable.
- saulrh 2y agoI mean, if there's no mechanism for deploying firmware updates, they took off the jtag headers or burned some fuses or blew a programmable-once ROM, there may be no way to deploy updates short of dismantling the car and swapping out a couple major boards. That might well count as "can't".
- gjsman-1000 2y agoReminder for anyone here who has imposter syndrome: Highly paid engineers built a system this way. I also have doubts they were specifically instructed to make the infrastructure non-transferable.
- mbreese 2y agoI’m also pretty sure they never expected to have to support transferring a car to a different infrastructure. Because if they ever needed to, that meant the company failed. Oh wait…
- ggm 2y agoShort of failure, was buy out and/or IPO with a single purchasor taking it private, or fusion with another company. There were scenarios down the track requiring porting the customer base into another data model that a competent design team would have in the brief, given a competent leadership. oh wait...
- whatshisface 2y ago"I'm going to need you to stay late this weekend, we're developing capabilities for a scenario where you've all been laid off."
- deleted 2y ago[deleted]
- steve_adams_86 2y agoAre we sure they were highly paid? Maybe that was part of the problem.
- cr3ative 2y agoI’m not overly surprised that in a startup, there was no mechanism built to split off a small defined portion of their fleet to a different, independent cloud. Bit different to changing the whole infrastructure out for every car in a disaster recovery situation.
- NotYourLawyer 2y agoI don’t understand the issue. Just transfer the domain names and the servers to the buyer. Problem solved.
- danpalmer 2y agoThey might have been licencing third party software that can't be migrated in this way due to contractual limitations. I imagine the simplest technical solution is for the leasing company to acquire the whole business entity. That may even be what they are trying to do in court.
- magicalhippo 2y agoThey're not buying the company though, just a bunch of cars as I understood it. At least several other people have bought Fisker cars as well, perhaps that's where the issue arises. At the very least, it sounds like a problematic privacy and legal issue for regular Joe to have his car suddenly send data to some "random" third party, without his consent.
- LegitShady 2y agocloud features are the enemy of consumers
- willcipriano 2y agoMy dream job is solving "impossible" technical problems like this rapidly, on short notice, and eating everything I kill (I'd imagine they could pay $500k+ for a fast [24 hours] solution here). Some sort of tech navy seal.
- fragmede 2y agoThat would be pretty fun. Joe Grand has made a career of it, I'm sure there are others.
- yieldcrv 2y agoYeah form an LLC and pretend there are a bunch of people involved to land that contract They wont typically pay an individual a rate equivalent to that, but an LLC can get a contract
- sneak 2y agoThis is exactly the same advice as you give an IC that is going after b2b work. It has nothing to do with who they will or won’t work with and everything to do with limiting your liability when playing in the big leagues.
- wmf 2y agoI would guess the root of this problem is that they aren't willing to pay.
- killingtime74 2y agoIndeed. If they did they would just pay their old employees on a contract basis, not this new person who's never seen the code before
- xvector 2y agoThat sounds like a fun company idea tbh
- neom 2y agoI think this must be an engineers wet dream? I'm not a SWE but over the years many engineers have told me that is their dream job.
- walrus01 2y agoA good reminder to never buy a high-dollar tech product that you don't actually own because its continued functioning is dependent upon phoning home to some corporate overlord. I'm happy with my shitty old non-tech-laden car, thanks.
- deely3 2y agoSo, just never use any online service. Its a good solution, and so simple!
- lykahb 2y agoI'd guess that making the cars depend on the cloud so much that they are unusable without it is intentional and was a part of the pitch deck.
- numpad0 2y agoIs it because someone fired hit [<] and [>] buttons on HSM on the way out, or is it because they’re trying to take control of fractions of the fleet?
- asteroidburger 2y agoThe latter one is the most plausible explanation I’ve read yet. Basically, they can’t give away the keys to the kingdom because they’re for the entire kingdom, not just the unsold inventory.
- Animats 2y ago"But those clouds and servers will not be maintained indefinitely, and once they go down, the cars that depend on them will lose features that owners may be relying upon." This may prevent cars being advertised for "sale" in California after January 1. AB 2426: Consumer protection: false advertising: digital goods.[1] “Digital application or game” means any application or game that a person accesses and manipulates using a specialized electronic gaming device, computer, mobile device, tablet, or other device with a display screen, including any add-ons or additional content for that application or game. That's a car with an infotainment system. This law makes it a crime to offer something "for sale" if it can be remotely disabled later, absent a separate acknowledgement that it's a lease. [1] https://legiscan.com/CA/text/AB2426/2023 https://legiscan.com/CA/text/AB2426/2023
- dceddia 2y agoSomehow this feels like it could backfire spectacularly. “Sorry, we can’t ‘sell’ our cars anymore. The law says we have to call them a ‘lease’ and that’ll be be $X,XXX/year”
- chipdart 2y ago> “Sorry, we can’t ‘sell’ our cars anymore. The law says we have to call them a ‘lease’ and that’ll be be $X,XXX/year” Wouldn't that be enough reason to take your business elsewhere?
- dceddia 2y agoI mean, that assumes there's anywhere else to go that's doing it differently. It feels like market forces can push things in either of 2 directions: (a) People hate practice X, so they shun companies that do it, and enough go to companies who don't do X, which leads to fewer companies doing X (b) One company gets away with doing X, and despite some complaints, they do just fine – and other companies realize they can get away doing X too, and soon every company is doing it
- 2y ago
- TheRealPomax 2y agoI guess they're handing over full control of those servers, then. Seems a pretty simple problem to solve, given the legal mandate Fisker is under?
- throwaway48476 2y agoIn a way I'm glad fisker went bankrupt if only to provide proof positive of the dangers of the 'connected car'.
- johnnyanmac 2y ago>The company filed for Chapter 11 bankruptcy protection in June after failing to reach a strategic agreement with another automaker that could provide it with more capital and domestic manufacturing capacity. I don't think software had anything to do with the filing. they couldn't make and sell cars fast enough and burned through their money over the pandemic.
- oniony 2y agoI don't believe the person you are replying to was suggesting that software had anything to do with their bankruptcy. I believe they are suggesting that it is illustrative to the general public as to why buying cars with heavy cloud integration is a bad idea.
- throwaway48476 2y agoThis is correct.
- deleted 2y ago[deleted]
- pabs3 2y agoProbably a bigger problem for modern cars (including EVs) is how long security updates will be provided for.
- Ekaros 2y agoThere should be some reasonable mandate how long security updates have to be provided from last manufacturing date. Say 25 years.
- dalai 2y agoIn the EU the CRA would have mandated something like that -- not 25 years but a support period defined by the manufacturer that reflects the time the product is expected to be used. The auto industry managed to get themselves excluded.
- beAbU 2y agoWhy would a car even need to have security updates in the first place. Or are we arguing that software updates and patches become like car parts, and you can go to your local parts dealer to get an oil filter, fan belt and a USB with the latest security updates?
- pabs3 2y agoCars are full of software now, they are basically smartphones with wheels, cameras, microphones etc. Even a 20-year-old car has an Engine Control Unit, a computer that controls the engine, handles the immobilizer etc.
- beAbU 2y agoI'm not questioning why cars have software, I'm questioning why they need software updates. Specifically OTA updates, which means the car has an attack vector. A car should be an embedded system, there is arguably no need for any of its core subsystems (ECU, BMS,immobilizer etc) to have a wireless connection to the outside world. If updates are needed, the dealer can handle it physically.
- sublinear 2y agoStop stacking the deck with incompetent opportunist scammers who don't belong in front of a computer let alone managing jack shit especially IT projects. Pure stupidity and downvote me all you want.
- h0l0cube 2y agoAre there yet any EVs out there where you actually have control over the software? Or are there any where you could conceivably root the firmware and use open source updates and 3rd party services when the support timeline ends or the manufacturer folds?
- frogblast 2y agoMy 2019 Kia Niro has some not-worthwhile connected service you can simply say No to. It is then a fully offline car. Assigning ownership involves handing someone the key fob.
- h0l0cube 2y agoI'm thinking that at least with phones you can get an Android and pretty much take it over fully and still have all the services^. I wonder if that will ever happen for cars. ^ Edit: Honestly I don't have any experience with this, but I assume you still need access to Google for notifications etc., so maybe not
- jve 2y agoNissan Leaf Visia owner here. The car is the most basic of 4 trim lines. I installed Nissan app and couldn't connect the car and found out that connectivity feature is not available for my basic car.
- pvillano 2y agoA blessing in disguise I guess
- beAbU 2y agoMy 2019 Hyundai Kona electric is a "dumb" car. Does not even have built-in nav. There is no connectivity, no app, no spying. Hyundai can vanish from existence tomorrow and the car would not even be aware of this happening. I don't care about it's software, because there is nothing to care about. It's just a dumb-ass car that goes when I need it to go. It comes with (wired) Android Auto, which is probably it's nicest and most advanced user feature.
- Habgdnv 2y agoJust in case someone from American Lease is reading this, I’d be willing to migrate their servers for less than a million. Jokes aside, after reading the comments here, I doubt anyone with technical knowledge would believe this. Even with certificate pinning, you can simply dump the firmware as a raw binary, replace the certificate with your own, and upload it back to the car. And even if the source code is lost, you can still sniff the traffic and implement an API. I did this for my previous employer, who had a collection of expensive, locked devices. It took me about a week, without any prior knowledge or experience. Imagine what someone with more experience could do...
- magicalhippo 2y ago> Even with certificate pinning, you can simply dump the firmware as a raw binary, replace the certificate with your own, and upload it back to the car. That's assuming they have access to the private key used to sign the firmware though...
- deleted 2y ago[deleted]
- joezydeco 2y agoExactly. This could be as simple as "we stopped paying for the AWS HSM instance and now it's been deleted, the keys are lost"
- DannyBee 2y agoMost implementations of this sort of thing in practice don't verify as hard you might think. A lot of it seems to do with wanting to be able to replace certs and have reasonable expiration times, but not really understanding how to do that (I don't mean it's not possible, i mean the manufacturers seem to not really understand how to do it effectively) As an example, the siemens CNC controller on my metal mill is totally signed. It has an FPGA with a secure element producing verification signatures to double check cert sigs haven't been modified, Every single file system with binaries is a read-only signed cramfs file signed with a secp521 ecc key. All read-write fsen are mounted noexec, nosuid, etc etc etc. The initial CA key is baked into secure hardware. However, in the end, they only verify the CA and signing certs have the right names and properties (various oem specific fields, etc), because the certs have 3-5 year expiration dates and these things are not connected to the internet or even updated often. So they accept expired certs for the signatures, and they also accept any root cert + signing cert that looks the same as the current ones. So you can replace the CA key and signing keys with something that looks exactly the same as their current one and resign everything, and it works fine. A whole lot of effort that can be defeated pretty quickly. I would be surprised if the cars were not similar - they look really secure, but in the end they made tradeoffs that defeat the system.
- hn72774 2y agoAre there any viable open source electric vehicle initiatives out there yet?
- bongodongobob 2y agoAs much as I love OSS I wouldn't want it in my literal daily driver. "My car shuts off when I open all the doors" "Why would you want it to run? You can't drive with your doors open! Marking as won't fix, working as intended." "It's open source! You can fix the code yourself!" No fucking thanks. Some things should cost money and have real stakes.
- hn72774 2y agoChances are there is OSS already running in commercially available vehicles. Owners tinkering with ICE vehicles was and is a thing and I don't see how an electric power train makes that too much different. Open standards and data formats would be a good middle ground to help avoid the type of problem with Fisker "unable" to migrate to a different provider. Although I wish that vehicles did not have to phone home to the mother ship at all.
- bongodongobob 2y agoOf course there is, but I don't want the whole thing to be OSS code run by people with no skin in the game.
- hn72774 2y agoThat's a good point. Hardware + software seems a lot harder than just software too.
- crest 2y agoJust because the source code is open source doesn't mean the upstream repo going into your car has to be run by a toxic community with a chip on their shoulder.
- kragen 2y agoTechnical incompetence, it sounds like? The risks of technical incompetence seem to be rising.
- jandrewrogers 2y agoAnalogues of this have been playing out in other industries for some time. The legal incentives strongly bias the outcomes toward it not being possible to own these products, only rent them. At a visceral level I don’t like it but at an intellectual level I understand why it is the only plausible way forward. There is no easy solution. In automotive, it is common for there to be a requirement that software is supported to some degree for 7-15 years. In practice, this is extremely expensive to guarantee but no one wants to pay for the cost of a reliable guarantee. The industry is at an impasse with consumers and it manifests in situations like this article.
- pas 2y agoI like renting cars, I hate to take them for some godforsaken garage to fix some issue with it. Sure, there's a premium, and so on, but similarly I hate looking for an overnight parking space, if I can simply hand it over to someone who takes/needs it for the night (going around selling drugs or going to fill it up and clean it, or all of the above) all the better. with onboard telemetry and ubiquitous HD image recording ability it's pretty easy to make a few shots to have some evidence of good and careful handling, and giving it to the next user in a known condition. of course if time and space are at a relative abundance then owning might make perfect sense, or if someone uses it so much that handover costs would start to be significant. 7-15 years is nothing for firmware. the hardware is fixed, requirements are also basically fixed. no need to support new codes, new protocols, etc.
- williamty689 2y ago[dead]