4 ms·

I worked with it for 7 years and the software ran car dealerships. That codebase is 40 years old now and still had remnants from the 70s where it had been origi

by sqldba 9y ago

I worked with it for 7 years and the software ran car dealerships. That codebase is 40 years old now and still had remnants from the 70s where it had been originally built for the VAX. Migrating it to a new architecture would not be trivial. That software is extremely tied into the underlying inter-process messaging protocols and data file formats provided by the operating system. You're not just working with a single monolith and flat files or any known database system for example. Though I think some were available much later. The company did split and build a new piece of software from the ground up to replace it. That effort took nearly a decade and millions of dollars - way longer than anticipated - and so long that it was sold off to private investors for continued funding long before it ever became marketable. It does well now though and bears little resemblance to the original system except in the UI concepts and screens. OpenVMS was a pleasure to use though I was more on the support side and didn't have anything to do with the admin level maintenance. I did manage to accidentally shut down a production timeshare instance in the middle of the day once though and impact thousands of users. Which reminds me it had its own networking, and networking hardware, so a dealership would have VT terminal green screens and the DECnet protocol would route it over IP directly into the server. It was pretty awesome. You don't really hear of systems that "small" these days serving thousands of simultaneous connections and tonnes of back end processing. It had nightly batching for accounting which was a common source of problems when something went wrong because nobody really understood how it worked anymore and THAT code was mostly the very old shit.

cm2187 9y ago
I am always curious to understand what corporate process on earth can take a decade to be rebuilt. It can be complex but not that complex. Particularly something like car dealership. We are not talking about some super diverse supply chain and interfacing with lots of different hardware, etc.
- inopinatus 9y ago
  The key point in the GP is nobody really understood how it worked anymore. No matter how accessible the domain generally might seem, once a system becomes a mysterious black box sans documentation or continuing expertise then no CIO is prepared to sign off the migration, for fear of missing a business rule or special case. I've been the guy that left taking all the knowledge with him, years later learning that they've lost the documentation & the source code but the system is still running. I've seen this occur in other domains (finance and education sectors) and basically the black box hangs on until either the unit, or the entire business, ceases to exist. I did see one project that attempted a like-for-like reimplementation via a carve-out process and it was a decade-long gigadollar boondoggle that has basically crippled the enterprise that tried it.
- sqldba 9y ago
  I think you'd be surprised at what goes into running a dealership. The software is complex with a lot of dependencies. You have a full parts/warehouse inventory system and invoicing internally and to other mechanics. A service system (including rosters and timesheets) and it's own invoicing system to customers. You need a vehicle module to manage all the local laws and taxes as well as interfacing with manufacturer systems for models and options (which are God awful undocumented weird binary formats over long dead networks - sometimes hardwired through parts of the phone companies that even they don't know they have anymore) so you can sell to customers. So that's 3 different invoicing systems to start. It all needs to be underpinned with an extremely flexible accounting system in the back end because every dealer does stuff differently but all of those things are related. The parts department sells to the service department at a discount; the service department prepares new vehicles for sale (called pre-delivery) and bonuses and rebates occur through each step. You need to import a lot more data covering log book service information and parts catalogs. And lots of reports for those car companies and online sellers and more. Some of that isn't exactly a programmers problem but the breadth of the issue and lack of generalised knowledge sources (and how every car manufacturer does things completely differently), it's a minefield to keep one going let alone building a new one. Consequently most of the world runs on only 2 or 3 systems and they're all ancient and generally the companies that develop them are awful and treat customers like garbage. There's a lot of vendor lock in and manufacturers fighting each other. Car dealerships have a tough job. Also their accountants are near God level skills wise. Anyway I don't think I answered your question. Yes I also don't know exactly what happened but yeah it was a disaster that only happened to work out because of millions of dollars. Once it WAS ready though money started flowing in because dealers are desperate to get away from the existing awful vendors. On the other hand each wanted so much custom functionality that it led to work pipelines planned years in advance by the time I left - and that doesn't make for happy customers either.
- seasoned_geek 9y ago
  It takes a minimum of 7 years to design from scratch, develop, test then settle in a "simple" customized order and inventory processing system, sans a WMS and picking component. The "services" people subscribe to when setting up some kind of shopping cart on their Web site are neither complex nor highly customizable. Most importantly, they offer no opportunity what-so-ever to purchase a guaranteed delivery + guaranteed execution service. When an order from a dealer (doesn't matter what automotive area) leaves their Business System, it is no longer their responsibility. It is the responsibility of the badge (Freightliner, Harley, Toyota, etc.) That order, which can be hundreds of thousands of dollars each and every day, must completely pass through the badge system, routing out to warehouses, vendors, distributors and manufacturing schedules, without ever being lost no matter what fails. This is a completely different business model than Criminal-Net sales where people complain about the billions of dollars worth of "abandoned shopping carts" each year. By and large these are not "abandoned shopping carts." These are the symptom of a poorly designed system built with feeble tools. Something FAILED and the consumer went to a different site to place their order. Blade and rack mount servers are built with inferior hardware. Their entire premise is the swarm principle. Many will die, but some will survive. Each and every time one of the many die, the customer leaves. Some sites have tried to "fix" this problem by sending you emails every few hours for days about the things "you forgot in your cart." This isn't a fix. This is an alcoholic refusing to admit they have more than a drinking problem.

by sqldba 9y ago

cm2187 9y ago

I am always curious to understand what corporate process on earth can take a decade to be rebuilt. It can be complex but not that complex. Particularly something like car dealership. We are not talking about some super diverse supply chain and interfacing with lots of different hardware, etc.

inopinatus 9y ago
The key point in the GP is nobody really understood how it worked anymore. No matter how accessible the domain generally might seem, once a system becomes a mysterious black box sans documentation or continuing expertise then no CIO is prepared to sign off the migration, for fear of missing a business rule or special case. I've been the guy that left taking all the knowledge with him, years later learning that they've lost the documentation & the source code but the system is still running. I've seen this occur in other domains (finance and education sectors) and basically the black box hangs on until either the unit, or the entire business, ceases to exist. I did see one project that attempted a like-for-like reimplementation via a carve-out process and it was a decade-long gigadollar boondoggle that has basically crippled the enterprise that tried it.
sqldba 9y ago
I think you'd be surprised at what goes into running a dealership. The software is complex with a lot of dependencies. You have a full parts/warehouse inventory system and invoicing internally and to other mechanics. A service system (including rosters and timesheets) and it's own invoicing system to customers. You need a vehicle module to manage all the local laws and taxes as well as interfacing with manufacturer systems for models and options (which are God awful undocumented weird binary formats over long dead networks - sometimes hardwired through parts of the phone companies that even they don't know they have anymore) so you can sell to customers. So that's 3 different invoicing systems to start. It all needs to be underpinned with an extremely flexible accounting system in the back end because every dealer does stuff differently but all of those things are related. The parts department sells to the service department at a discount; the service department prepares new vehicles for sale (called pre-delivery) and bonuses and rebates occur through each step. You need to import a lot more data covering log book service information and parts catalogs. And lots of reports for those car companies and online sellers and more. Some of that isn't exactly a programmers problem but the breadth of the issue and lack of generalised knowledge sources (and how every car manufacturer does things completely differently), it's a minefield to keep one going let alone building a new one. Consequently most of the world runs on only 2 or 3 systems and they're all ancient and generally the companies that develop them are awful and treat customers like garbage. There's a lot of vendor lock in and manufacturers fighting each other. Car dealerships have a tough job. Also their accountants are near God level skills wise. Anyway I don't think I answered your question. Yes I also don't know exactly what happened but yeah it was a disaster that only happened to work out because of millions of dollars. Once it WAS ready though money started flowing in because dealers are desperate to get away from the existing awful vendors. On the other hand each wanted so much custom functionality that it led to work pipelines planned years in advance by the time I left - and that doesn't make for happy customers either.
seasoned_geek 9y ago
It takes a minimum of 7 years to design from scratch, develop, test then settle in a "simple" customized order and inventory processing system, sans a WMS and picking component. The "services" people subscribe to when setting up some kind of shopping cart on their Web site are neither complex nor highly customizable. Most importantly, they offer no opportunity what-so-ever to purchase a guaranteed delivery + guaranteed execution service. When an order from a dealer (doesn't matter what automotive area) leaves their Business System, it is no longer their responsibility. It is the responsibility of the badge (Freightliner, Harley, Toyota, etc.) That order, which can be hundreds of thousands of dollars each and every day, must completely pass through the badge system, routing out to warehouses, vendors, distributors and manufacturing schedules, without ever being lost no matter what fails. This is a completely different business model than Criminal-Net sales where people complain about the billions of dollars worth of "abandoned shopping carts" each year. By and large these are not "abandoned shopping carts." These are the symptom of a poorly designed system built with feeble tools. Something FAILED and the consumer went to a different site to place their order. Blade and rack mount servers are built with inferior hardware. Their entire premise is the swarm principle. Many will die, but some will survive. Each and every time one of the many die, the customer leaves. Some sites have tried to "fix" this problem by sending you emails every few hours for days about the things "you forgot in your cart." This isn't a fix. This is an alcoholic refusing to admit they have more than a drinking problem.