16 ms·

Car dealerships revert to pens and paper after cyberattacks on software provider

221 points by achristmascarl 2y ago

jwnin 2y ago
This illustrates the difference between a Business Continuity Plan and Disaster Recovery.
- thechao 2y ago
  I don’t know anything about operations management, securitization, etc. What’s the difference between these two? Also, I work for a very large HW company, and we’re locked down. From my point-of-view “operations” just works. What does this look like for a small not-in-computers company? Like, how could the Seattle Public Library have secured itself from ransomware with a turnkey, reasonably priced solution?
  - jl6 2y ago
    DR: bring IT services back online. BCP: keep the business running, possibly without using IT.
  - vel0city 2y ago
    At a high level, disaster recovery can often be "how do we get back to the way we were doing business?" A BCP is "what do we do if the way we did business before is not an option?" So in this case, a DR plan might be "how do we restore our CDK data if something important gets deleted?" A BCP might be "what do we do if CDK no longer exists?"
  - bombcar 2y ago
    The BCP is what you execute while the DRP is in process. If you have a BCP with no DRP you're probably going to survive, painfully. If you have a DRP and no BCP the company may be dead before the disaster is recovered.
spamizbad 2y ago
I know some (former) CDK alumni and I'm not surprised. Notorious for paying below market in engineering and IT. A place to cut your teeth but not to stay.
- lylejantzi3rd 2y ago
  Why don't they have any competition?
  - toomuchtodo 2y ago
    My hot take is auto dealers are a notoriously painful enterprise sales cycle due to network effects aka "good 'ol boys" club (anecdotally from folks in tech who have to deal in this industry).
    - mrfairrunheight 2y ago
      can confirm. also the high cost entailed to retrain folks from one system to a newer one.
      - deleted 2y ago
        [deleted]
    - SoftTalker 2y ago
      Many are also not tech-savvy. Many dealerships are inherited within a family that has owned them since the 1950s if not longer. They are cash cows for the owners and there's little motivation to try new things. When they are forced to, like when internet sales and marketing became a thing they had to deal with, they just go with the same system that all their fellow dealers are using.
      - mywittyname 2y ago
        There also isn't space to grow. It's not like running a dealership with hyper efficiency is going to lead to being able to open another dealership. The market is largely saturated and in the rare event that an existing dealership comes for sale, you'll be competing with massive PE-backed conglomerates that can pay nonsense prices for them.
josefresco 2y ago
For those looking for close coverage of this issue, "GuyDealership" is a good source. https://x.com/GuyDealership https://x.com/GuyDealership https://www.dealershipguy.com https://www.dealershipguy.com *not affiliated
- nikodunk 2y ago
  Wow these are great, thank you!
- imroot 2y ago
  He's def a good source, but, he said that "cybersecurity" is the new buzzword of the year, replacing "AI" and "Digital Retailing"[0]. When you talk security to most dealerships, it's about protecting the assets that are on their floor plan -- the vehicles -- not necessarily anything else. Most places have tried to take steps that prevent their sales folks from walking out with their customer book, but, that's the extent of most of their information security needs. -- 0: https://x.com/GuyDealership/status/1803421101713715290 https://x.com/GuyDealership/status/1803421101713715290
  - josefresco 2y ago
    I feel like GuyDealership is running about 4-5 years ahead of most dealerships. His guests, often successful dealership owners and related service providers are all pushing cutting-edge tools but yet the majority of the industry is creaking along using tactics that are at best decades old. All that being said, I will call him out on Twitter for some of the things he says that are quite ridiculous. He's a sales guy after all.
- HeyLaughingBoy 2y ago
  He was actually the first place I heard about this.
ryanmccullagh 2y ago
I would bet that a developer knew of the security issues, but was vetoed by Product.
- deleted 2y ago
  [deleted]
- neilv 2y ago
  The usual case for software development the last decade is developers who don't fully understand what they're doing, and are mostly focused on resume-driven-development (RDD) and looking good in Agile sprint standups. Noteworthy exceptions: FAANG promotion bid orientation, and VC growth startup alignment towards shipping something to look like growth towards exit. In a small minority of cases, you have developers who know what they are doing, and are thinking rigorously. The norm isn't big-meanie Product making (forthright, courageous, photogenic) developers ship negligent security vulnerabilities, against developers' protests. Developers are at least as much responsible as Product.
  - ryanmccullagh 2y ago
    Not getting time to do security because we need to ship v1 is a decision called by Product
    - neilv 2y ago
      If you know of a developer who would've done responsible security, but was pushed to ship instead, against their protest, I'd be happy to hear that.
      - ryanmccullagh 2y ago
        In my experience, Product is always the first one to suggest cutting corners.
        neilv 2y ago
CapitalistCartr 2y ago
I'm trying to buy two cars right now, and this has put a big wrench in the process. The dealership can't tell me what's on their lot compared to what's on their Website, except by going out and searching.
- wongarsu 2y ago
  Maybe offer to bring them more pens and paper so they can send someone out to make a list of what's on their lot.
  - bee_rider 2y ago
    This is mostly a site for programmers. I can’t imagine working in computers and not believing at least that the intent is to make people’s lives more convenient and make work efficient. If the dealers have the capacity to go around and run everything by pencil and paper, it seems like either the are really poorly run for the average case where the computers work and make life easier, or the programmers have done a really poor writing useful programs.
    - michaelt 2y ago
      Let's assume you're only selling what's on the lot, and you're not running some complex network operation where you care about other dealerships' stock. Googling tells me the average car salesman sells 8-10 cars per month. Maybe more for a 'high volume' dealership but certainly less than 1 car per day. I would think the paperwork to sell a single car would take substantially less than 8 hours for an experienced worker to complete.
      - bee_rider 2y ago
        The paperwork to do one transaction might not be too bad, but keeping all the inventory on pencil and paper seems like a pain.
        michaelt 2y ago
bilsbie 2y ago
I think every process needs a paper backup. There’s just no telling when something will be down. If you run a SAAS you could offer this as a feature. Even hire data entry people to input transactions later if needed.
- ethbr1 2y ago
  Is there anything relevant to recovery / graceful functional degradation in SoC2? If not, this feels like an obvious match there. E.g. "Company has a defined and demonstrated process by which they can (a) offer degraded but limited functionality in the event of full system outage & (b) accept data updates via a backup method, until system restoration." Sure would have helped with Change Healthcare...
  - quercusa 2y ago
    I think you have to have a tabletop exercise once a year. So all good, right?
    - ethbr1 2y ago
      I mean, better than not. Everytime I see something basic in a standard, it makes me realize there were businesses out there who weren't doing anything of the sort.
- codemac 2y ago
  You can price this relatively quickly, and see that it's probably not worth it. ~1 week outage once every ~5 years, let's say you lose 100% of revenue that week, that's 0.38% of revenue. Now most likely they're not losing 100% of revenue, demand for cars is not very elastic... it starts to get pointless all around. Which is frustrating for consumers, but the profit for perfection isn't usually there.
  - karaterobot 2y ago
    I think you'd have to compare that .38% of revenue to the cost of developing the paper process, as well comparing what revenue you'd lose with the paper process anyway. It doesn't seem crazy to me, unless your paper process is hard to develop. There's also an argument that developing a backup process is a great exercise to help people understand all the parts of your system and how they interact, as a first step to making them more efficient/redundant/secure/delightful.
n4r9 2y ago
This is what happens when a single player (CDK Global) has massive market share (90% when measured by number of vehicles sold [0] --- EDIT: this is actually the combined market share of CDK + Reynolds, who have a non-compete agreement ---). The entire industry becomes fragile to product issues and/or malicious attacks. Anti-trust is important. [0] https://casetext.com/case/loop-llc-v-cdk-global-llc-in-re-dealer-mgmt-sys-antitrust-litig https://casetext.com/case/loop-llc-v-cdk-global-llc-in-re-de...
- ethbr1 2y ago
  Also, the vendor becomes insulated from customer demands. Why patch things or work your ass off, when none of your customers have an alternative? (especially if PE buys the business or CEOs get a stock price focus and gut engineering to keep-the-lights-on levels)
  - mywittyname 2y ago
    There's a quiet irony in car dealership networks being fucked like this. They spend so much money ensuring their government-mandated monopoly, only to fall victim to a different kind of monopoly.
    - throwaway55533 2y ago
      > government-mandated monopoly Could you expand on this, aren't U.S. car dealerships private businesses competing in a (mostly) unregulated market?
      - PaywallBuster 2y ago
        > direct manufacturer auto sales are prohibited in many states by franchise laws requiring that new cars be sold only by independent dealers. https://en.wikipedia.org/wiki/Tesla_US_dealership_disputes https://en.wikipedia.org/wiki/Tesla_US_dealership_disputes not a monopoly per se, but "forced demand"
SSJPython 2y ago
Related note: car stealerships are a relic of the past. There is no reason why a middleman should exist when buying a car. Consumers should have the option to purchase directly from the manufacturer. I am personally not a fan of Tesla as a vehicle, but I admire the business model of buying directly from Tesla. Stealerships are a drain on the consumer. All that land they own can be used for productive economic activity or housing.
- hypeatei 2y ago
  I agree they're useless middlemen but I don't believe for a second that the direct-to-manufacturer model wouldn't also be enshitified eventually too.
  - kevingadd 2y ago
    It's what we had before the laws prohibiting direct-to-consumer/manufacturer-owned dealerships. It's hard for me to find any reliable sources that explain why we got those laws though (was it because direct-to-consumer was bad?) It seems like the EU has reined in the worst excesses of dealerships a lot, and allows direct-to-consumer. Maybe the US will follow suit in a few decades.
    - kube-system 2y ago
      In the early days of the automobile, auto manufacturers didn't have the resources to have stores in every town in America. People couldn't just get a vehicle in Atlanta from Detroit if they didn't have a way to get there. Dealers both organically popped up, and some manufacturers actively engaged with local dealers to get their cars out to a broader audience (e.g. Ford with the Model T). But as those manufacturers and dealers grew, they then had the resources to operate at broader geographies, and dealers lobbed for franchise laws to "protect local businesses", etc.
    - stackskipton 2y ago
      Dealerships came about because car manufacturers were demanding that all service go through their dealerships at crazy marked up prices. So dealerships started as way to "protect the consumer" but quickly because terrible places all on their own.
deleted 2y ago
[deleted]
ThalesX 2y ago
I was involved in the building of the 911 system for my country. When doing that, we had to design it with this in mind. If the system ever fell down, the operators had standard templated papers they would fill, and once the system would be back up they would add the case data back in.
- tonymet 2y ago
  Thank you for investing in that it will definitely pay off.
- ilamont 2y ago
  In many parts of the US, there are still red "call boxes" in working order and maintained as a backup to the 911 system, and was actually used when Massachusetts' 911 system went down last week: “The current 911 system is down statewide, if you have an emergency and need assistance pull your nearest Fire Box, or call the Boston Fire Department at 617-343-2880. You can also get assistance by going to your nearest Firehouse,” the post read. Police Commissioner and Boston Police Chief Michael Cox said earlier Tuesday that residents looking to contact officials while the 911 system was down could call local police departments for assistance, and pull the local red call box on streets and corners to alert fire/EMS. https://www.kake.com/story/50917965/statewide-911-system-outage-in-massachusetts-is-resolved-officials-say https://www.kake.com/story/50917965/statewide-911-system-out... The technology, patented by Gamewell in the 1880s, is based on telegraph technology from the mid 1800s - you pull the handle, it releases a notched disc which starts spinning and transmitting a number to the central fire command. The number corresponds to a street address and emergency services will be alerted. Old call box still in working order: https://www.instagram.com/reel/C5bwxFoJc56/ https://www.instagram.com/reel/C5bwxFoJc56/
- deleted 2y ago
  [deleted]
- iso8859-1 2y ago
  Why say "my country"? There are less than 300 countries world wide, people should know all of them.
blantonl 2y ago
I bought a new car this weekend and some of the symptoms of this were: * dealer didn't have a good handle on what was available in stock on the lot * anything related to a title for a tradein was massively hampered. * appears USAA's auto insurance online add/replace vehicle on your policy is broken * everything was literally hand written, down to the sales contract I would imagine the impact of this is in the tens of millions of dollars
- paxys 2y ago
  That has always been my experience at car dealerships regardless of the status of their computer system.
- rootusrootus 2y ago
  > appears USAA's auto insurance online add/replace vehicle on your policy is broken FWIW, USAA did some kind of upgrade recently and there have been several functions that broke utterly. They had a message for a while apologizing for it. Not sure if it's fixed now, it could be that which you are experiencing and not related to CDK. Or you may be right.
shagie 2y ago
One of the reports of this I heard was on Marketplace the other day... and the part that made me chuckle was on cursive. Pen and paper, but please no cursive: Widespread cyberattack sends car dealers back to 20th century - https://www.marketplace.org/2024/06/21/cdk-global-hack-software-auto-dealers-car-dealership/ https://www.marketplace.org/2024/06/21/cdk-global-hack-softw... > Car dealers are a pretty big part of this economy, doing some $1.2 trillion in sales last year, according to the National Automobile Dealers Association. In the wake of the CDK hack, there’s a new policy among the salespeople and mechanics at the Willis Automotive dealership in Des Moines, Iowa: no cursive. > “We have a lot of staff members that are younger than 30 that I’m not sure have seen cursive in their life. So we try not to go the cursive route just to make sure everybody understands what’s going on,” said Jason Willis, CEO of the dealership.
- mywittyname 2y ago
  I like how they blame "young people" for not being able read cursive instead of their own terrible hand writing.
  - 1-6 2y ago
    "Young people" are not being taught cursive in school. Schools are not teaching cursive to students any longer since 2010 if they're following the Common Core curriculum. https://en.wikipedia.org/wiki/Cursive_handwriting_instruction_in_the_United_States#:~:text=Recent%20events,in%20the%20respective%20state%20curriculum https://en.wikipedia.org/wiki/Cursive_handwriting_instructio....
    - mywittyname 2y ago
      You can learn to read cursive in a few hours. It's not like they are writing in Mandarin, most of the letters are identical. Complaining about it is culture war virtue signaling. Additionally, kids do still learn cursive. It's just not mandatory curriculum. My nephew is was learning it just before school ended this year.
  - jimt1234 2y ago
deleted 2y ago
[deleted]
2OEH8eoCRo0 2y ago
Chinese retaliation for EV tariffs?
tonymet 2y ago
In studying voting systems I’ve come to respect paper-based logs along the same quality parameters as databases. Resiliency, scalability, backup, readability, schema-flexibility , onboarding, power-efficiency are all very good with paper. So many of us were quick to move paper-based systems (voting, orders, kiosks, parking meters, journals, requests, etc) over to computers – without accounting for the e2e cost and flakiness of the software-based system.
- deleted 2y ago
  [deleted]
- metadat 2y ago
  How does the backup aspect work for paper? Scanning? Otherwise paper is quite fragile and susceptible to combustion. What if voting was all done via paper and you could later view the scan of your ballot as it existed at the time of count? Unfortunately there could still be funny business on the backend.
  - vel0city 2y ago
    I'm not a fan of being able to see the values of your ballot after it is cast. It greatly opens up the possibility of coercion or bribery. If I can show how I voted, it leaves it open for someone to demand/request I prove to them how I voted. And just seeing a picture of it tells you pretty much nothing about if the system actually counted it properly or not, you still need to trust the counting system at some level. Lets say someone offers me $500 to vote for candidate X. I don't like X, but I do like money. I vote for Y, I tell them I voted for X, there is no proof either way, would they still want to give me that $500? Or if someone says, "vote for X or I'll break your legs", how would they know if I voted for X or not?
    - metadat 2y ago
      Agreed, you raise some really good points. Thanks!
  - tonymet 2y ago
m3kw9 2y ago
Got took back to the stone ages
ChrisArchitect 2y ago
Some more updates from bleeping computer last week: https://news.ycombinator.com/item?id=40751754 https://news.ycombinator.com/item?id=40751754
- thecosas 2y ago
  More from them here too: https://www.bleepingcomputer.com/news/security/cdk-global-outage-caused-by-blacksuit-ransomware-attack/ https://www.bleepingcomputer.com/news/security/cdk-global-ou...
deleted 2y ago
[deleted]
blorenz 2y ago
Oh boy, has this made issues and opportunities for my automotive B2B SaaS. My product's customer are car dealerships and most of my customers are on CDK. This is definitely creating a void of data for them though fortunately my product won't fail without the expected DMS data. In terms of opportunity, dealers had paperwork that would be generated for the F/I department from CDK which I was able to port into my SaaS and prefill with data we can capture at the time of sale. All the praise is due for FOSS like `pdf-lib` that I was able to incorporate in a matter of a few hours. All the disdain for Adobe Acrobat Pro which fought me tooth and nail when converting their paperwork to fillable PDF forms ( - if you're the Product Manager on that then please get a UX audit of it!) Currently, we are supporting our CDK dealers in this critical time by getting them paperwork filled and printed to keep the car buying experience going. Also, we're hiring! I could use another solid dev and a solid designer that wants to create a delightful experience out of an awkward wait time in dealership. Challenges abound but persistence prevails. Primary stack includes React, Remix, Xstate, Remotion, Framer motion, Python, Django, Graphql. Contact devs@zipdeal.com if interested and we can go from there.
- brian-bk 2y ago
  What are cdk and dms in this context? I'm only familiar with them on AWS
  - _joel 2y ago
    Car Deployment Kit (I jest, I have no idea)
  - shrubble 2y ago
    CDK is the company that got cyberattacked; DMS = dealer management system.
  - rootusrootus 2y ago
    CDK is the company name. Stands for Cobalt (marketing firm that got bought and has since been spun off), Dealer Services (CDK used to be a subsidiary of ADP called ADP Dealer Services), and K (Kerrigan, the international arm of the business, it was spun off). Should just rename it back to Dealer Services at this point. DMS, as someone already mentioned, stands for Dealer Management System.
pjmorris 2y ago
For about ten years, I've theorized that we will see the rise of 'boutique' doctor, dentist, and law offices that keep only paper records for the sake of the privacy of their clients. There are a bunch of problems that connection and automation bring with them, the rise of digital monocultures seems to be one of them.
- hot_gril 2y ago
  I rented from a small landlord office, they were using Windows XP long after dropped support and got infected with ransomware.
- momojo 2y ago
  Interested in hearing more. I've heard the same sentiment from not just old (~30 yr old) practices but just-starting ones. Like any non-technical market, they just bought what was pitched, and there's often more gripes than complains. One example: When a patient transfers to another dental office, their records need to follow. Before you "had to" fax or snail-mail them. The software/cloud solutions promised streamlined, painless transfers, but those benefits never really materialized due to the amount of HIPA-compliancy issues (the details around this pain point are fuzzy). The office-manager explaining all of this was (half-jokingly) fantasizing about going back to snail mail.
  - silverquiet 2y ago
    I work in infrastructure for an EHR provider, so interoperability isn't really my area, but I do work with devs who must do this. I'd say the problem is approximately zero to do with HIPAA; we almost never even talk about it in my experience. It's just a lack of standards, and poor implementations of what standards do exist - every once in awhile something will crash because of a weird escape character or something in an HL7 message (though that could be on us no doubt). I believe last year was the first time that there was a mandated standard to export EHR data and one of the devs was complaining that it didn't even say if units should be imperial or metric. I'm told the labs are a complete mess and have no federally mandated standards; I know they've written code because doctor X wants to use lab Y. Obviously the big national labs win because you only have to write code for them once and all customers can use them. It's just another aspect of a broken American healthcare "system" - there's no incentive for anyone to interoperate in the best case, and in the worst case it allows you to create lock-in for your product.
xyst 2y ago
I worked at one of these companies catering to a large network of dealerships. Quit after 1 month. The one company I worked at was outsourcing all or most of the development work with an onshore mostly as management and a couple of devs in the US. It was a massive shit show. - Tests? Useless and garbage. Only used to pump their code coverage reports to show to management - Design? Non existent and a hacked together code base across a half dozen low bid contractors, probably junior engineers at best - CI? Dedicated test environments? Nah, costs too much money. Just “use docker to test on local and deploy directly to prod” - Documentation? Besides very basic instructions on how to deploy locally … good luck - backup and recovery? Nah who has time for that. Just work on the features that {big client} wants - code quality? I wish there was a “god” class, but in this code base they managed to create a universal class or “macro service” Management was a mess. Just a mouthpiece for sales or executives. No backbone. No vision. Just “alright we just need to get through the quarter” mentality.
- y-c-o-m-b 2y ago
  I've worked in many different companies (healthcare, real estate, investment research, payroll, semi-conductor, streaming media, and more). I am currently in FAANG. All of your points pretty much apply across the board. Especially the one about tests, that is nearly universal. Outsourcing is a recipe for amplifying all of those negatives though and always has been. It won't be long before they start bringing those jobs back here again for that reason. Happens every time.
- janee 2y ago
  "macro service" hilarious, will most certainly use this in some future conversation at work, thanks.
bilater 2y ago
Scary. I made a short story based on a massive AI attack that takes this up a notch: https://www.tiktok.com/@likearollingbot/video/7384110345201241386?is_from_webapp=1&sender_device=pc&web_id=7380557964720195118 https://www.tiktok.com/@likearollingbot/video/73841103452012...
StanislavPetrov 2y ago
Another commercial for eliminating cash and moving to 100% digital currency!
- yodon 2y ago
  >Another commercial for eliminating cash and moving to 100% digital currency! Because then it would be impossible to do transactions on paper when the computer system goes down? That doesn't sound like an advantage.
eagerpace 2y ago
I don't want to come off a loving Tesla too much, they're not perfect, but this is another big area where they are transforming the status quo. So much of the FUD being thrown at them is by this lobby, they still can't sell cars direct to consumer in Texas! Car dealers, especially used cars, in the US are one of the most hated consumer experiences. Pricing is unclear, consumers are talked into buying things they don't want or need, to the point that most women I know won't buy a car without a man joining them. Additionally, they're resistant to EVs because it is eating into their service margin. I saw a brand new Porsche dealer going up recently with only two Level 2 charges. Probably a 10M project, and they have only a token level of concern for planning for future infrastructure. They're out of touch, short sighted, businesses with little alignment to the corporate manufacturing brand.
- callalex 2y ago
  Getting repair parts can be impossible though, because they’d rather put the limited number of parts they have into new cars. I’ve had friends with broken windows and seats that took more than a year to fix. A whole year with a trash bag taped over the window in a luxury-priced car!
  - eagerpace 2y ago
    Is that still an issue? I know it happened in the past, but I don't believe it's the case anymore. The flip side is a dealer might have better parts availability, but often at a 2-3x mark up. I think it's a growing pain for Tesla than a profit strategy for dealers.
akskakskaksk 2y ago
Crazy how we are talking about this like a weather event, like it's just an unfortunate outage. Cyberattacked by whom? How? What vulnerabilities allowed the intrusion and what organizational processes created those vulnerabilities? Naturally the people who know these answers are very busy today but hopefully we will hear more soon.
- 93po 2y ago
  > Cyberattacked by whom? Whoever is politically convenient > How? Probably social engineering but possibly poor security practices > What vulnerabilities allowed the intrusion Probably poor security training on employees and poor safeguards, but possibly also poor security practices in software engineering > what organizational processes created those vulnerabilities? Leadership with poor priorities and shortsightedness for the sake of short-term profit
- ufmace 2y ago
  That probably is the better way to see it at this point. There's already enough bad actors out there that it's not really worth keeping track. I'd be willing to bet that they have comprehensively sloppy and terrible infosec practices and that we'll never hear any details about it - most of their customers wouldn't understand, and it would only expose how bad they are at all of this if they did.
- rootusrootus 2y ago
  > Naturally the people who know these answers are very busy today but hopefully we will hear more soon. Most of us don't actually know the answer, to be honest. They're not telling 6500 employees what really happened. I know more than you, but not that much more, and I'm definitely not saying anything about it.
proee 2y ago
What is the reality something like this could happen to all google accounts, apple accounts, etc?
nsxwolf 2y ago
I haven't bought a new car in 8 years. Do they still do that thing with the pen and paper where they write down a bunch of numbers while crossing other numbers out randomly for like 10 minutes?
- 0x0000000 2y ago
  The four square sheet? Yes they still do it because it's how they manipulate buyers. It's not random, it's very intentional.
- seattle_spring 2y ago
  Yes, all sorts of nonsense like that still happens.
- rootusrootus 2y ago
  Yes, but only if you foolishly walk in to the dealer cold without having already arranged some plan. Also, not all dealers. Just stand up and walk out as soon as they make the crossed lines. Those dealers aren't worth it at all, and there are actually some dealers that don't suck nearly that bad, believe it or not. I still choose to just do it all by email anyway and then show up only to grab the keys. Works fine. Next time I'm going to go a step further and have them bring the car out to my driveway. They absolutely will, even if it just means having a salesman drive it out.
afh1 2y ago
I like old doctors who still use pen and paper for medical history. I don't like the idea of my medical records being stored online. Unfortunately this is the exception and not the rule nowadays. The same goes for imagery, X-Rays aside I don't remember any exam that didn't go straight into an online system.
- hn_throwaway_99 2y ago
  Neither of these are feasible: 1. There are legal incentives (in the US at least) for doctors to use electronic health records, EHR. Providers that don't use EHR are penalized for it when it comes to Medicare/Medicaid. 2. Imaging PACS (Picture Archiving and Communication Systems) allow radiologists to read images from multiple different facilities from one location, including smaller/rural hospitals, and at all times. The alternative is to require physical staffing by radiologists at every single location, which is obviously much less efficient. I agree there are tons of issues with the quality of cybersecurity, especially in the health system, but saying "I just want it all on paper" is throwing out the baby with the bathwater, not to mention basically impossible at this point.
lolinder 2y ago
I'm reminded of the opening line of the second Mistborn novel: > I write these words in steel, for anything not set in metal cannot be trusted. The digital world feels increasingly dangerous and ephemeral. If I have something written with ink on paper I have certain guarantees: I know that it's accessible even if the power is completely out. I know that it won't randomly get deleted if my hard drive suddenly dies. I know that the only way someone can read its contents are if they get physical access to that piece of paper. I know that it's not being automatically scanned by a platform provider to comply with government surveillance laws. I know that any alteration to the document would require physical access to it and would most likely leave a visible trace. There are technologies that if applied would provide some or most of these guarantees and even provide increased safety, but I don't trust any existing SaaS providers to give me any of them out of the box, and the average person doesn't have the skill to string tools together to get them. And so here we are: in 2024, anything not printed on paper still cannot be fully trusted.
- rvba 2y ago
  I was recently looking at a 15 year old manual for an appliance. It had a sales recepit attached. Despite being put inside some papers, in a plastic case and then in a binder (kept in the closet)... the recepit just evaporated. The print is nearly invisible. Is that done on purpose?
  - ars 2y ago
    It's thermal printing, they heat the paper to turn it black, but if you store it somewhere warm the chemical reaction can reverse.
  - animex 2y ago
    Many of these thermal and carbon transfer receipt printers have poor ink/print life. I doubt it was done on purpose as most of these systems are sold by 3rd parties to retailers. I'm sure the issue has been raised but then they willing thought to themselves, yeah actually this benefits us, let's just "pretend to ignore" the issue. More modern retailers are offering receipt-email options which is the option I choose if available, that way I have a "permanent" record of the transaction. I recently did some old taxes from 7 years ago and this feature was invaluable as most online portals no longer offered the records.
Animats 2y ago
The AP article is terrible. It's basically CDK's press release. There's more useful stuff on Reddit.[1][2][3] - Some dealerships can still sell cars. Some can't. - Parts and service are in worse shape than new vehicle sales because their inventory info is in the CDK system. - The process by which new cars and parts are ordered and delivered to dealers is down. - Many dealerships can sell what's on the lot and do repairs with parts in inventory, but the supply chain has stopped. [1] https://www.reddit.com/r/partscounter/comments/1dmbmy7/the_cdk_outage_is_a_lot_bigger_than_people_realize/ https://www.reddit.com/r/partscounter/comments/1dmbmy7/the_c... [2] https://www.reddit.com/r/serviceadvisors/comments/1djisf5/cdk_down_everywhere/ https://www.reddit.com/r/serviceadvisors/comments/1djisf5/cd... [3] https://www.reddit.com/r/askcarsales/comments/1dkf0xv/how_prepared_was_your_dealership_for_the_total/ https://www.reddit.com/r/askcarsales/comments/1dkf0xv/how_pr...
BenderV 2y ago
Seems weird that to me that it make a front page on HN (from an European pov ; it's quite common)
- rootusrootus 2y ago
  It's a fairly large cyberattack, so it's halfway interesting to HN readers.
interludead 2y ago
In an age where digital transformation is the backbone of business operations, a cyberattack can bring entire industries to a standstill. And it's a little bit frightening
shrimp_emoji 2y ago
They upgraded to the Centurion package: https://youtu.be/gbyXfLSqveM https://youtu.be/gbyXfLSqveM
wormius 2y ago
I know it's not directly related to this, but I remember one time I went to a dealership in my area that was apparently known for being particularly skeezy (unbeknownst to me before I went in). They got sued for some sort of fraud. Anyways, I remember going in, and the guy pulling up a car in their system when the (too good to be true) car listed wasn't in stock. He turned the computer screen around, and there was one of those (I think) fake "you have a virus" pop ups in the IE window. It was either that or a real alert. Either way - clearly incompetent with tech, and once I saw that I noped out of there fast.
anthomtb 2y ago
This explains so much. I am in the market for a new vehicle and did multiple test drives this weekend. One salesperson casually mentioned "our software is down and things are a little crazy". But other than a lot of waiting, things proceeded as I expected. I figured that by 1000 this (Monday) morning I'd be fending off follow up texts and calls. Yet its now 1430 and not one outreach has appeared on my phone. This is not a complaint but it is a surprise compared with past buying experiences.
ryanmccullagh 2y ago
Someone needs to get fired over this. Probably an incompetent, non technical Product VP
- rootusrootus 2y ago
  I have several ideas in mind, but honestly I'd just go for the CEO and be done with it.

This illustrates the difference between a Business Continuity Plan and Disaster Recovery.

thechao 2y ago
I don’t know anything about operations management, securitization, etc. What’s the difference between these two? Also, I work for a very large HW company, and we’re locked down. From my point-of-view “operations” just works. What does this look like for a small not-in-computers company? Like, how could the Seattle Public Library have secured itself from ransomware with a turnkey, reasonably priced solution?
- jl6 2y ago
  DR: bring IT services back online. BCP: keep the business running, possibly without using IT.
- vel0city 2y ago
  At a high level, disaster recovery can often be "how do we get back to the way we were doing business?" A BCP is "what do we do if the way we did business before is not an option?" So in this case, a DR plan might be "how do we restore our CDK data if something important gets deleted?" A BCP might be "what do we do if CDK no longer exists?"
- bombcar 2y ago
  The BCP is what you execute while the DRP is in process. If you have a BCP with no DRP you're probably going to survive, painfully. If you have a DRP and no BCP the company may be dead before the disaster is recovered.

I know some (former) CDK alumni and I'm not surprised. Notorious for paying below market in engineering and IT. A place to cut your teeth but not to stay.

lylejantzi3rd 2y ago
Why don't they have any competition?
- toomuchtodo 2y ago
  My hot take is auto dealers are a notoriously painful enterprise sales cycle due to network effects aka "good 'ol boys" club (anecdotally from folks in tech who have to deal in this industry).
  - mrfairrunheight 2y ago
    can confirm. also the high cost entailed to retrain folks from one system to a newer one.
    - deleted 2y ago
      [deleted]
  - SoftTalker 2y ago
    Many are also not tech-savvy. Many dealerships are inherited within a family that has owned them since the 1950s if not longer. They are cash cows for the owners and there's little motivation to try new things. When they are forced to, like when internet sales and marketing became a thing they had to deal with, they just go with the same system that all their fellow dealers are using.
    - mywittyname 2y ago
      There also isn't space to grow. It's not like running a dealership with hyper efficiency is going to lead to being able to open another dealership. The market is largely saturated and in the rare event that an existing dealership comes for sale, you'll be competing with massive PE-backed conglomerates that can pay nonsense prices for them.

josefresco 2y ago

For those looking for close coverage of this issue, "GuyDealership" is a good source. https://x.com/GuyDealership https://x.com/GuyDealership https://www.dealershipguy.com https://www.dealershipguy.com *not affiliated

nikodunk 2y ago
Wow these are great, thank you!
imroot 2y ago
He's def a good source, but, he said that "cybersecurity" is the new buzzword of the year, replacing "AI" and "Digital Retailing"[0]. When you talk security to most dealerships, it's about protecting the assets that are on their floor plan -- the vehicles -- not necessarily anything else. Most places have tried to take steps that prevent their sales folks from walking out with their customer book, but, that's the extent of most of their information security needs. -- 0: https://x.com/GuyDealership/status/1803421101713715290 https://x.com/GuyDealership/status/1803421101713715290
- josefresco 2y ago
  I feel like GuyDealership is running about 4-5 years ahead of most dealerships. His guests, often successful dealership owners and related service providers are all pushing cutting-edge tools but yet the majority of the industry is creaking along using tactics that are at best decades old. All that being said, I will call him out on Twitter for some of the things he says that are quite ridiculous. He's a sales guy after all.
HeyLaughingBoy 2y ago
He was actually the first place I heard about this.

ryanmccullagh 2y ago

I would bet that a developer knew of the security issues, but was vetoed by Product.

deleted 2y ago
[deleted]
neilv 2y ago
The usual case for software development the last decade is developers who don't fully understand what they're doing, and are mostly focused on resume-driven-development (RDD) and looking good in Agile sprint standups. Noteworthy exceptions: FAANG promotion bid orientation, and VC growth startup alignment towards shipping something to look like growth towards exit. In a small minority of cases, you have developers who know what they are doing, and are thinking rigorously. The norm isn't big-meanie Product making (forthright, courageous, photogenic) developers ship negligent security vulnerabilities, against developers' protests. Developers are at least as much responsible as Product.
- ryanmccullagh 2y ago
  Not getting time to do security because we need to ship v1 is a decision called by Product
  - neilv 2y ago
    If you know of a developer who would've done responsible security, but was pushed to ship instead, against their protest, I'd be happy to hear that.
    - ryanmccullagh 2y ago
      In my experience, Product is always the first one to suggest cutting corners.
      - neilv 2y ago

CapitalistCartr 2y ago

I'm trying to buy two cars right now, and this has put a big wrench in the process. The dealership can't tell me what's on their lot compared to what's on their Website, except by going out and searching.

wongarsu 2y ago
Maybe offer to bring them more pens and paper so they can send someone out to make a list of what's on their lot.
- bee_rider 2y ago
  This is mostly a site for programmers. I can’t imagine working in computers and not believing at least that the intent is to make people’s lives more convenient and make work efficient. If the dealers have the capacity to go around and run everything by pencil and paper, it seems like either the are really poorly run for the average case where the computers work and make life easier, or the programmers have done a really poor writing useful programs.
  - michaelt 2y ago
    Let's assume you're only selling what's on the lot, and you're not running some complex network operation where you care about other dealerships' stock. Googling tells me the average car salesman sells 8-10 cars per month. Maybe more for a 'high volume' dealership but certainly less than 1 car per day. I would think the paperwork to sell a single car would take substantially less than 8 hours for an experienced worker to complete.
    - bee_rider 2y ago
      The paperwork to do one transaction might not be too bad, but keeping all the inventory on pencil and paper seems like a pain.
      - michaelt 2y ago

bilsbie 2y ago

I think every process needs a paper backup. There’s just no telling when something will be down. If you run a SAAS you could offer this as a feature. Even hire data entry people to input transactions later if needed.

ethbr1 2y ago
Is there anything relevant to recovery / graceful functional degradation in SoC2? If not, this feels like an obvious match there. E.g. "Company has a defined and demonstrated process by which they can (a) offer degraded but limited functionality in the event of full system outage & (b) accept data updates via a backup method, until system restoration." Sure would have helped with Change Healthcare...
- quercusa 2y ago
  I think you have to have a tabletop exercise once a year. So all good, right?
  - ethbr1 2y ago
    I mean, better than not. Everytime I see something basic in a standard, it makes me realize there were businesses out there who weren't doing anything of the sort.
codemac 2y ago
You can price this relatively quickly, and see that it's probably not worth it. ~1 week outage once every ~5 years, let's say you lose 100% of revenue that week, that's 0.38% of revenue. Now most likely they're not losing 100% of revenue, demand for cars is not very elastic... it starts to get pointless all around. Which is frustrating for consumers, but the profit for perfection isn't usually there.
- karaterobot 2y ago
  I think you'd have to compare that .38% of revenue to the cost of developing the paper process, as well comparing what revenue you'd lose with the paper process anyway. It doesn't seem crazy to me, unless your paper process is hard to develop. There's also an argument that developing a backup process is a great exercise to help people understand all the parts of your system and how they interact, as a first step to making them more efficient/redundant/secure/delightful.

n4r9 2y ago

This is what happens when a single player (CDK Global) has massive market share (90% when measured by number of vehicles sold [0] --- EDIT: this is actually the combined market share of CDK + Reynolds, who have a non-compete agreement ---). The entire industry becomes fragile to product issues and/or malicious attacks. Anti-trust is important. [0] https://casetext.com/case/loop-llc-v-cdk-global-llc-in-re-dealer-mgmt-sys-antitrust-litig https://casetext.com/case/loop-llc-v-cdk-global-llc-in-re-de...

ethbr1 2y ago
Also, the vendor becomes insulated from customer demands. Why patch things or work your ass off, when none of your customers have an alternative? (especially if PE buys the business or CEOs get a stock price focus and gut engineering to keep-the-lights-on levels)
- mywittyname 2y ago
  There's a quiet irony in car dealership networks being fucked like this. They spend so much money ensuring their government-mandated monopoly, only to fall victim to a different kind of monopoly.
  - throwaway55533 2y ago
    > government-mandated monopoly Could you expand on this, aren't U.S. car dealerships private businesses competing in a (mostly) unregulated market?
    - PaywallBuster 2y ago
      > direct manufacturer auto sales are prohibited in many states by franchise laws requiring that new cars be sold only by independent dealers. https://en.wikipedia.org/wiki/Tesla_US_dealership_disputes https://en.wikipedia.org/wiki/Tesla_US_dealership_disputes not a monopoly per se, but "forced demand"

SSJPython 2y ago

Related note: car stealerships are a relic of the past. There is no reason why a middleman should exist when buying a car. Consumers should have the option to purchase directly from the manufacturer. I am personally not a fan of Tesla as a vehicle, but I admire the business model of buying directly from Tesla. Stealerships are a drain on the consumer. All that land they own can be used for productive economic activity or housing.

hypeatei 2y ago
I agree they're useless middlemen but I don't believe for a second that the direct-to-manufacturer model wouldn't also be enshitified eventually too.
- kevingadd 2y ago
  It's what we had before the laws prohibiting direct-to-consumer/manufacturer-owned dealerships. It's hard for me to find any reliable sources that explain why we got those laws though (was it because direct-to-consumer was bad?) It seems like the EU has reined in the worst excesses of dealerships a lot, and allows direct-to-consumer. Maybe the US will follow suit in a few decades.
  - kube-system 2y ago
    In the early days of the automobile, auto manufacturers didn't have the resources to have stores in every town in America. People couldn't just get a vehicle in Atlanta from Detroit if they didn't have a way to get there. Dealers both organically popped up, and some manufacturers actively engaged with local dealers to get their cars out to a broader audience (e.g. Ford with the Model T). But as those manufacturers and dealers grew, they then had the resources to operate at broader geographies, and dealers lobbed for franchise laws to "protect local businesses", etc.
  - stackskipton 2y ago
    Dealerships came about because car manufacturers were demanding that all service go through their dealerships at crazy marked up prices. So dealerships started as way to "protect the consumer" but quickly because terrible places all on their own.

deleted 2y ago

[deleted]

ThalesX 2y ago

I was involved in the building of the 911 system for my country. When doing that, we had to design it with this in mind. If the system ever fell down, the operators had standard templated papers they would fill, and once the system would be back up they would add the case data back in.

tonymet 2y ago
Thank you for investing in that it will definitely pay off.
ilamont 2y ago
In many parts of the US, there are still red "call boxes" in working order and maintained as a backup to the 911 system, and was actually used when Massachusetts' 911 system went down last week: “The current 911 system is down statewide, if you have an emergency and need assistance pull your nearest Fire Box, or call the Boston Fire Department at 617-343-2880. You can also get assistance by going to your nearest Firehouse,” the post read. Police Commissioner and Boston Police Chief Michael Cox said earlier Tuesday that residents looking to contact officials while the 911 system was down could call local police departments for assistance, and pull the local red call box on streets and corners to alert fire/EMS. https://www.kake.com/story/50917965/statewide-911-system-outage-in-massachusetts-is-resolved-officials-say https://www.kake.com/story/50917965/statewide-911-system-out... The technology, patented by Gamewell in the 1880s, is based on telegraph technology from the mid 1800s - you pull the handle, it releases a notched disc which starts spinning and transmitting a number to the central fire command. The number corresponds to a street address and emergency services will be alerted. Old call box still in working order: https://www.instagram.com/reel/C5bwxFoJc56/ https://www.instagram.com/reel/C5bwxFoJc56/
deleted 2y ago
[deleted]
iso8859-1 2y ago
Why say "my country"? There are less than 300 countries world wide, people should know all of them.

blantonl 2y ago

I bought a new car this weekend and some of the symptoms of this were: * dealer didn't have a good handle on what was available in stock on the lot * anything related to a title for a tradein was massively hampered. * appears USAA's auto insurance online add/replace vehicle on your policy is broken * everything was literally hand written, down to the sales contract I would imagine the impact of this is in the tens of millions of dollars

paxys 2y ago
That has always been my experience at car dealerships regardless of the status of their computer system.
rootusrootus 2y ago
> appears USAA's auto insurance online add/replace vehicle on your policy is broken FWIW, USAA did some kind of upgrade recently and there have been several functions that broke utterly. They had a message for a while apologizing for it. Not sure if it's fixed now, it could be that which you are experiencing and not related to CDK. Or you may be right.

shagie 2y ago

One of the reports of this I heard was on Marketplace the other day... and the part that made me chuckle was on cursive. Pen and paper, but please no cursive: Widespread cyberattack sends car dealers back to 20th century - https://www.marketplace.org/2024/06/21/cdk-global-hack-software-auto-dealers-car-dealership/ https://www.marketplace.org/2024/06/21/cdk-global-hack-softw... > Car dealers are a pretty big part of this economy, doing some $1.2 trillion in sales last year, according to the National Automobile Dealers Association. In the wake of the CDK hack, there’s a new policy among the salespeople and mechanics at the Willis Automotive dealership in Des Moines, Iowa: no cursive. > “We have a lot of staff members that are younger than 30 that I’m not sure have seen cursive in their life. So we try not to go the cursive route just to make sure everybody understands what’s going on,” said Jason Willis, CEO of the dealership.

mywittyname 2y ago
I like how they blame "young people" for not being able read cursive instead of their own terrible hand writing.
- 1-6 2y ago
  "Young people" are not being taught cursive in school. Schools are not teaching cursive to students any longer since 2010 if they're following the Common Core curriculum. https://en.wikipedia.org/wiki/Cursive_handwriting_instruction_in_the_United_States#:~:text=Recent%20events,in%20the%20respective%20state%20curriculum https://en.wikipedia.org/wiki/Cursive_handwriting_instructio....
  - mywittyname 2y ago
    You can learn to read cursive in a few hours. It's not like they are writing in Mandarin, most of the letters are identical. Complaining about it is culture war virtue signaling. Additionally, kids do still learn cursive. It's just not mandatory curriculum. My nephew is was learning it just before school ended this year.
- jimt1234 2y ago

deleted 2y ago

[deleted]

2OEH8eoCRo0 2y ago

Chinese retaliation for EV tariffs?

tonymet 2y ago

In studying voting systems I’ve come to respect paper-based logs along the same quality parameters as databases. Resiliency, scalability, backup, readability, schema-flexibility , onboarding, power-efficiency are all very good with paper. So many of us were quick to move paper-based systems (voting, orders, kiosks, parking meters, journals, requests, etc) over to computers – without accounting for the e2e cost and flakiness of the software-based system.

deleted 2y ago
[deleted]
metadat 2y ago
How does the backup aspect work for paper? Scanning? Otherwise paper is quite fragile and susceptible to combustion. What if voting was all done via paper and you could later view the scan of your ballot as it existed at the time of count? Unfortunately there could still be funny business on the backend.
- vel0city 2y ago
  I'm not a fan of being able to see the values of your ballot after it is cast. It greatly opens up the possibility of coercion or bribery. If I can show how I voted, it leaves it open for someone to demand/request I prove to them how I voted. And just seeing a picture of it tells you pretty much nothing about if the system actually counted it properly or not, you still need to trust the counting system at some level. Lets say someone offers me $500 to vote for candidate X. I don't like X, but I do like money. I vote for Y, I tell them I voted for X, there is no proof either way, would they still want to give me that $500? Or if someone says, "vote for X or I'll break your legs", how would they know if I voted for X or not?
  - metadat 2y ago
    Agreed, you raise some really good points. Thanks!
- tonymet 2y ago

m3kw9 2y ago

Got took back to the stone ages

ChrisArchitect 2y ago

Some more updates from bleeping computer last week: https://news.ycombinator.com/item?id=40751754 https://news.ycombinator.com/item?id=40751754

thecosas 2y ago
More from them here too: https://www.bleepingcomputer.com/news/security/cdk-global-outage-caused-by-blacksuit-ransomware-attack/ https://www.bleepingcomputer.com/news/security/cdk-global-ou...

deleted 2y ago

[deleted]

blorenz 2y ago

Oh boy, has this made issues and opportunities for my automotive B2B SaaS. My product's customer are car dealerships and most of my customers are on CDK. This is definitely creating a void of data for them though fortunately my product won't fail without the expected DMS data. In terms of opportunity, dealers had paperwork that would be generated for the F/I department from CDK which I was able to port into my SaaS and prefill with data we can capture at the time of sale. All the praise is due for FOSS like `pdf-lib` that I was able to incorporate in a matter of a few hours. All the disdain for Adobe Acrobat Pro which fought me tooth and nail when converting their paperwork to fillable PDF forms ( - if you're the Product Manager on that then please get a UX audit of it!) Currently, we are supporting our CDK dealers in this critical time by getting them paperwork filled and printed to keep the car buying experience going. Also, we're hiring! I could use another solid dev and a solid designer that wants to create a delightful experience out of an awkward wait time in dealership. Challenges abound but persistence prevails. Primary stack includes React, Remix, Xstate, Remotion, Framer motion, Python, Django, Graphql. Contact devs@zipdeal.com if interested and we can go from there.

brian-bk 2y ago
What are cdk and dms in this context? I'm only familiar with them on AWS
- _joel 2y ago
  Car Deployment Kit (I jest, I have no idea)
- shrubble 2y ago
  CDK is the company that got cyberattacked; DMS = dealer management system.
- rootusrootus 2y ago
  CDK is the company name. Stands for Cobalt (marketing firm that got bought and has since been spun off), Dealer Services (CDK used to be a subsidiary of ADP called ADP Dealer Services), and K (Kerrigan, the international arm of the business, it was spun off). Should just rename it back to Dealer Services at this point. DMS, as someone already mentioned, stands for Dealer Management System.

pjmorris 2y ago

For about ten years, I've theorized that we will see the rise of 'boutique' doctor, dentist, and law offices that keep only paper records for the sake of the privacy of their clients. There are a bunch of problems that connection and automation bring with them, the rise of digital monocultures seems to be one of them.

hot_gril 2y ago
I rented from a small landlord office, they were using Windows XP long after dropped support and got infected with ransomware.
momojo 2y ago
Interested in hearing more. I've heard the same sentiment from not just old (~30 yr old) practices but just-starting ones. Like any non-technical market, they just bought what was pitched, and there's often more gripes than complains. One example: When a patient transfers to another dental office, their records need to follow. Before you "had to" fax or snail-mail them. The software/cloud solutions promised streamlined, painless transfers, but those benefits never really materialized due to the amount of HIPA-compliancy issues (the details around this pain point are fuzzy). The office-manager explaining all of this was (half-jokingly) fantasizing about going back to snail mail.
- silverquiet 2y ago
  I work in infrastructure for an EHR provider, so interoperability isn't really my area, but I do work with devs who must do this. I'd say the problem is approximately zero to do with HIPAA; we almost never even talk about it in my experience. It's just a lack of standards, and poor implementations of what standards do exist - every once in awhile something will crash because of a weird escape character or something in an HL7 message (though that could be on us no doubt). I believe last year was the first time that there was a mandated standard to export EHR data and one of the devs was complaining that it didn't even say if units should be imperial or metric. I'm told the labs are a complete mess and have no federally mandated standards; I know they've written code because doctor X wants to use lab Y. Obviously the big national labs win because you only have to write code for them once and all customers can use them. It's just another aspect of a broken American healthcare "system" - there's no incentive for anyone to interoperate in the best case, and in the worst case it allows you to create lock-in for your product.

xyst 2y ago

I worked at one of these companies catering to a large network of dealerships. Quit after 1 month. The one company I worked at was outsourcing all or most of the development work with an onshore mostly as management and a couple of devs in the US. It was a massive shit show. - Tests? Useless and garbage. Only used to pump their code coverage reports to show to management - Design? Non existent and a hacked together code base across a half dozen low bid contractors, probably junior engineers at best - CI? Dedicated test environments? Nah, costs too much money. Just “use docker to test on local and deploy directly to prod” - Documentation? Besides very basic instructions on how to deploy locally … good luck - backup and recovery? Nah who has time for that. Just work on the features that {big client} wants - code quality? I wish there was a “god” class, but in this code base they managed to create a universal class or “macro service” Management was a mess. Just a mouthpiece for sales or executives. No backbone. No vision. Just “alright we just need to get through the quarter” mentality.

y-c-o-m-b 2y ago
I've worked in many different companies (healthcare, real estate, investment research, payroll, semi-conductor, streaming media, and more). I am currently in FAANG. All of your points pretty much apply across the board. Especially the one about tests, that is nearly universal. Outsourcing is a recipe for amplifying all of those negatives though and always has been. It won't be long before they start bringing those jobs back here again for that reason. Happens every time.
janee 2y ago
"macro service" hilarious, will most certainly use this in some future conversation at work, thanks.

bilater 2y ago

Scary. I made a short story based on a massive AI attack that takes this up a notch: https://www.tiktok.com/@likearollingbot/video/7384110345201241386?is_from_webapp=1&sender_device=pc&web_id=7380557964720195118 https://www.tiktok.com/@likearollingbot/video/73841103452012...

StanislavPetrov 2y ago

Another commercial for eliminating cash and moving to 100% digital currency!

yodon 2y ago
>Another commercial for eliminating cash and moving to 100% digital currency! Because then it would be impossible to do transactions on paper when the computer system goes down? That doesn't sound like an advantage.

eagerpace 2y ago

I don't want to come off a loving Tesla too much, they're not perfect, but this is another big area where they are transforming the status quo. So much of the FUD being thrown at them is by this lobby, they still can't sell cars direct to consumer in Texas! Car dealers, especially used cars, in the US are one of the most hated consumer experiences. Pricing is unclear, consumers are talked into buying things they don't want or need, to the point that most women I know won't buy a car without a man joining them. Additionally, they're resistant to EVs because it is eating into their service margin. I saw a brand new Porsche dealer going up recently with only two Level 2 charges. Probably a 10M project, and they have only a token level of concern for planning for future infrastructure. They're out of touch, short sighted, businesses with little alignment to the corporate manufacturing brand.

callalex 2y ago
Getting repair parts can be impossible though, because they’d rather put the limited number of parts they have into new cars. I’ve had friends with broken windows and seats that took more than a year to fix. A whole year with a trash bag taped over the window in a luxury-priced car!
- eagerpace 2y ago
  Is that still an issue? I know it happened in the past, but I don't believe it's the case anymore. The flip side is a dealer might have better parts availability, but often at a 2-3x mark up. I think it's a growing pain for Tesla than a profit strategy for dealers.

akskakskaksk 2y ago

Crazy how we are talking about this like a weather event, like it's just an unfortunate outage. Cyberattacked by whom? How? What vulnerabilities allowed the intrusion and what organizational processes created those vulnerabilities? Naturally the people who know these answers are very busy today but hopefully we will hear more soon.

93po 2y ago
> Cyberattacked by whom? Whoever is politically convenient > How? Probably social engineering but possibly poor security practices > What vulnerabilities allowed the intrusion Probably poor security training on employees and poor safeguards, but possibly also poor security practices in software engineering > what organizational processes created those vulnerabilities? Leadership with poor priorities and shortsightedness for the sake of short-term profit
ufmace 2y ago
That probably is the better way to see it at this point. There's already enough bad actors out there that it's not really worth keeping track. I'd be willing to bet that they have comprehensively sloppy and terrible infosec practices and that we'll never hear any details about it - most of their customers wouldn't understand, and it would only expose how bad they are at all of this if they did.
rootusrootus 2y ago
> Naturally the people who know these answers are very busy today but hopefully we will hear more soon. Most of us don't actually know the answer, to be honest. They're not telling 6500 employees what really happened. I know more than you, but not that much more, and I'm definitely not saying anything about it.

proee 2y ago

What is the reality something like this could happen to all google accounts, apple accounts, etc?

nsxwolf 2y ago

I haven't bought a new car in 8 years. Do they still do that thing with the pen and paper where they write down a bunch of numbers while crossing other numbers out randomly for like 10 minutes?

0x0000000 2y ago
The four square sheet? Yes they still do it because it's how they manipulate buyers. It's not random, it's very intentional.
seattle_spring 2y ago
Yes, all sorts of nonsense like that still happens.
rootusrootus 2y ago
Yes, but only if you foolishly walk in to the dealer cold without having already arranged some plan. Also, not all dealers. Just stand up and walk out as soon as they make the crossed lines. Those dealers aren't worth it at all, and there are actually some dealers that don't suck nearly that bad, believe it or not. I still choose to just do it all by email anyway and then show up only to grab the keys. Works fine. Next time I'm going to go a step further and have them bring the car out to my driveway. They absolutely will, even if it just means having a salesman drive it out.

afh1 2y ago

I like old doctors who still use pen and paper for medical history. I don't like the idea of my medical records being stored online. Unfortunately this is the exception and not the rule nowadays. The same goes for imagery, X-Rays aside I don't remember any exam that didn't go straight into an online system.

hn_throwaway_99 2y ago
Neither of these are feasible: 1. There are legal incentives (in the US at least) for doctors to use electronic health records, EHR. Providers that don't use EHR are penalized for it when it comes to Medicare/Medicaid. 2. Imaging PACS (Picture Archiving and Communication Systems) allow radiologists to read images from multiple different facilities from one location, including smaller/rural hospitals, and at all times. The alternative is to require physical staffing by radiologists at every single location, which is obviously much less efficient. I agree there are tons of issues with the quality of cybersecurity, especially in the health system, but saying "I just want it all on paper" is throwing out the baby with the bathwater, not to mention basically impossible at this point.

lolinder 2y ago

I'm reminded of the opening line of the second Mistborn novel: > I write these words in steel, for anything not set in metal cannot be trusted. The digital world feels increasingly dangerous and ephemeral. If I have something written with ink on paper I have certain guarantees: I know that it's accessible even if the power is completely out. I know that it won't randomly get deleted if my hard drive suddenly dies. I know that the only way someone can read its contents are if they get physical access to that piece of paper. I know that it's not being automatically scanned by a platform provider to comply with government surveillance laws. I know that any alteration to the document would require physical access to it and would most likely leave a visible trace. There are technologies that if applied would provide some or most of these guarantees and even provide increased safety, but I don't trust any existing SaaS providers to give me any of them out of the box, and the average person doesn't have the skill to string tools together to get them. And so here we are: in 2024, anything not printed on paper still cannot be fully trusted.

rvba 2y ago
I was recently looking at a 15 year old manual for an appliance. It had a sales recepit attached. Despite being put inside some papers, in a plastic case and then in a binder (kept in the closet)... the recepit just evaporated. The print is nearly invisible. Is that done on purpose?
- ars 2y ago
  It's thermal printing, they heat the paper to turn it black, but if you store it somewhere warm the chemical reaction can reverse.
- animex 2y ago
  Many of these thermal and carbon transfer receipt printers have poor ink/print life. I doubt it was done on purpose as most of these systems are sold by 3rd parties to retailers. I'm sure the issue has been raised but then they willing thought to themselves, yeah actually this benefits us, let's just "pretend to ignore" the issue. More modern retailers are offering receipt-email options which is the option I choose if available, that way I have a "permanent" record of the transaction. I recently did some old taxes from 7 years ago and this feature was invaluable as most online portals no longer offered the records.

Animats 2y ago

The AP article is terrible. It's basically CDK's press release. There's more useful stuff on Reddit.[1][2][3] - Some dealerships can still sell cars. Some can't. - Parts and service are in worse shape than new vehicle sales because their inventory info is in the CDK system. - The process by which new cars and parts are ordered and delivered to dealers is down. - Many dealerships can sell what's on the lot and do repairs with parts in inventory, but the supply chain has stopped. [1] https://www.reddit.com/r/partscounter/comments/1dmbmy7/the_cdk_outage_is_a_lot_bigger_than_people_realize/ https://www.reddit.com/r/partscounter/comments/1dmbmy7/the_c... [2] https://www.reddit.com/r/serviceadvisors/comments/1djisf5/cdk_down_everywhere/ https://www.reddit.com/r/serviceadvisors/comments/1djisf5/cd... [3] https://www.reddit.com/r/askcarsales/comments/1dkf0xv/how_prepared_was_your_dealership_for_the_total/ https://www.reddit.com/r/askcarsales/comments/1dkf0xv/how_pr...

BenderV 2y ago

Seems weird that to me that it make a front page on HN (from an European pov ; it's quite common)

rootusrootus 2y ago
It's a fairly large cyberattack, so it's halfway interesting to HN readers.

interludead 2y ago

In an age where digital transformation is the backbone of business operations, a cyberattack can bring entire industries to a standstill. And it's a little bit frightening

shrimp_emoji 2y ago

They upgraded to the Centurion package: https://youtu.be/gbyXfLSqveM https://youtu.be/gbyXfLSqveM

wormius 2y ago

I know it's not directly related to this, but I remember one time I went to a dealership in my area that was apparently known for being particularly skeezy (unbeknownst to me before I went in). They got sued for some sort of fraud. Anyways, I remember going in, and the guy pulling up a car in their system when the (too good to be true) car listed wasn't in stock. He turned the computer screen around, and there was one of those (I think) fake "you have a virus" pop ups in the IE window. It was either that or a real alert. Either way - clearly incompetent with tech, and once I saw that I noped out of there fast.

anthomtb 2y ago

This explains so much. I am in the market for a new vehicle and did multiple test drives this weekend. One salesperson casually mentioned "our software is down and things are a little crazy". But other than a lot of waiting, things proceeded as I expected. I figured that by 1000 this (Monday) morning I'd be fending off follow up texts and calls. Yet its now 1430 and not one outreach has appeared on my phone. This is not a complaint but it is a surprise compared with past buying experiences.

ryanmccullagh 2y ago

Someone needs to get fired over this. Probably an incompetent, non technical Product VP

rootusrootus 2y ago
I have several ideas in mind, but honestly I'd just go for the CEO and be done with it.