8 ms·

Something is missing here, why do batch jobs take 13 hours? If this thing was started on an old mainframe why isn't the downtime just 5 minutes at 3:39 AM? Exa

Something is missing here, why do batch jobs take 13 hours? If this thing was started on an old mainframe why isn't the downtime just 5 minutes at 3:39 AM? Exactly how much data is getting processed? Edit: Why does rebuilding take a decade or more? This is not a complex system. It doesn't need to solve any novel engineering challenges to operate efficiently. Article does not give much insight into why this particular task couldn't be fixed in 3 months.

shermantanktop 2y ago
It’s funny to me that I would never ask those questions. I’ve specialized in legacy rehab projects (among other things) and there seems to be no upper bound on how bad things can be or how many annoying reasons there are for why we can’t “just fix it.” Those “just” questions—which I ask too—end up being hopelessly naive. The answers will crush your soul if you let them, so you can’t let them, and you should always assume things are worse than you think. TFA is spot on - the way to make progress is to cut problems up and deliver value. The unfortunate consequence is that badness gets more and more concentrated into the systems that nobody can touch, sort of like the evolution of a star into an eventual black hole.
- abigail95 2y ago
  I made a lot of money moving mid size enterprises from legacy ERP systems to custom in house ones. The DVLA dataset and the computations that are run on it can be studied and replicated in 3 months by a competent team. From there it can be improved. There is no way that this system requires 13 hours of downtime. If it required two hours - even if the code was generated through automation it can be reverse engineered and optimized. It is absolute rubbish that this thing is still unavailable outside of 8am-7pm. I maintain my position that it could be replaced in 3 months. I got my start in this business when I was in university and they told us our online learning software was going offline for 3 days for an upgrade. Those are the gatekeepers and low achievers we fight against. Think bigger.
  - arccy 2y ago
    it's a gov agency, they don't quite pay enough for a motivated competent team....
    - gunian 2y ago
      even in this economy where people can't even work with their own SSN?
  - monkey_monkey 2y ago
    > The DVLA dataset and the computations that are run on it can be studied and replicated in 3 months by a competent team. From there it can be improved. Such an HN comment. Made me lol. Think funnier!
  - pixl97 2y ago
    Ya I don't think I'd let you in two miles of a system like this. Replacing legacy stuff always expands in scope far beyond the initial changes. When you have to come back and add wait() entries in your new program because it spits data back faster than the old program ever could which then causes peripheral devices/drivers to crash which then pulls a dev and testers off something else important for days figuring out what kind of fresh hell is occurring is just status quo for ancient systems.
    - gunian 2y ago
      idk much about dev much less legacy / enterprise dev but it seems like an A/B test type situation where you have 90% of the users running the legacy code and the remaining 10 on a new implementation would be feasible any idea why this is the case?
      - pixl97 2y ago
        This is what happens. All that testing with the required stakeholders takes way way more time than you'd expect. Gets even more fun in .gov where the work can change significantly at particular times of the year. Had one piece of Windows software required by the State of Texas used at year end like once a year. Seemingly nobody realized windows updates had stopped it from working until a few weeks before the deadline. I had to setup a box without updates for it to run for my customer. Lead to a lot of panic around the state.
that_guy_iain 2y ago
> Edit: Why does rebuilding take a decade or more? This is not a complex system. It doesn't need to solve any novel engineering challenges to operate efficiently. Article does not give much insight into why this particular task couldn't be fixed in 3 months. You do know the UK government has been cutting all their budgets to the bone for about 10 years? That means everywhere is pretty much understaffed. And how do you know it's not a complex system? I would think that a system like that would be somewhat complex. It's not just driving licenses but a whole bunch of other things that are handled by the DVLA.
- abigail95 2y ago
  The system may or may not be complex but the data is has to store and transform is not. Because it handles drivers licenses. A function that has been done on pen and paper and filing cabinets. Study the data, study the operations, reduce complexity. Since you imply you know more about UK budgets than I do - how much is the DVLA budgeted for IT operations like this and how much more would you give them to expect this problem solved? I can argue real numbers but vibes about bone dry budgets I cannot.
  - that_guy_iain 2y ago
    > The system may or may not be complex but the data is has to store and transform is not. Because it handles drivers licenses. A function that has been done on pen and paper and filing cabinets. It handles more than just driving licenses... The DVLA do more than just driving licenses. > Since you imply you know more about UK budgets than I do - how much is the DVLA budgeted for IT operations like this and how much more would you give them to expect this problem solved? It's not budgeted anything for this as far as I know. I believe it's handled by Government Digital Services which handles lots of the digital services for various departments. The budget for all of GDS is about 90 million most of which isn't for .gov.uk. A rewrite of that size I would expect to cost about 50-60 million in total but take several years.
    - skippyboxedhero 2y ago
ajnin 2y ago
The batch jobs don't take 13 hours. They're just scheduled to run some time at night where the old offices used to be closed and the jobs could be ran with some expectations regarding data stability over the period. There are probably many jobs scheduled to run at 1AM then 2AM, etc, all depending on the previous to be finished so there is some large delay to ensure that a job does not start before the previous one is finished. As to your "not a complex system" remark, when a system is built for 60 years, piling up new rules to implement new legislation and needs over time, you tend to end up with a tangled mess of services all interdependent that are very difficult to replace piece-wise with a new shiny architecturally pure one. This is closer to a distributed monolith than a microservices architecture. In my experience you can't rebuild such a thing "in 3 months". People who believe that are those that don't realize the complexity and the extraordinary amount of specifics, special cases, that are baked into the system, and any attempt to just rebuild from scratch in a few months hits that wall and ends up taking years.
- abigail95 2y ago
  The code will be spaghettified and hideous. The queries will be nonsense. That doesn't change the fact that the ultimate goal of the system is to manage drivers licenses. > In my experience you can't rebuild such a thing "in 3 months". Me and my team rebuilt the core stack for the central bank of a developing country. In 3 months. The tech started in the 70s just like this. Think bigger.
  - mattmanser 2y ago
    Yeah, I always raise an eyebrow at attitudes like that too. I've also reimplemented or gradually replaced several out-of-date systems. Albeit on a smaller scale. In my experience, when you start picking the programs apart you find 90% of the code is redundant or boilerplate. Much of it isn't even called from anywhere, abandoned code, and can be deleted en masse. A lot of programmers don't clean code up "just in case" and then no-one else deletes it. They can also often be vastly simplified because programmers back then didn't have the patterns and knowledge to write consisely. I often find myself simplifying the original code first, which gets rid of 50% of it. Then I can see what the code actually does and rewrite it which gets rid of the other 40%. On the other hand, many programmers don't have the patience, stubbornness or skill to do this kind of work. And the ability to get through the major panic you have when you're half way through and wondering if you were mad to even start.
firefoxd 2y ago
Our systems took 8 hours to back up. Then it grew to 12 hours [0]. The system was a side project by an intern fresh out of college. Over the years, it grew into a crucial software the company relied on. I joined over 10 years later and was able to bring it down to few minutes. [0]: https://news.ycombinator.com/item?id=38456429 https://news.ycombinator.com/item?id=38456429
jdietrich 2y ago
Per their own data, the DVLA are responsible for the records of 52 million drivers and 46 million vehicles. Those records are immensely complex, because they reflect decades of accumulated legislation, regulation and practice. Every edge case has an edge case. There's someone, somewhere in the bowels of the DVLA who understands the rules for drivers with visual field defects who use a bioptic device. There's someone who knows which date code applies to a vehicle that has been built with a brand new kit chassis but an old engine and drive train. There's someone who understands the special rates of tax that apply to goods vehicles that are solely used by showmen, or are based on certain offshore islands. God help any outsider who has to condense all of that institutional knowledge into a working piece of software. Government does not have a good track record of ground-up refactors of complex IT systems. The British government in particular does not have a good track record. Considering all that, the fact that most interactions with DVLA can be done entirely online is borderline miraculous. https://assets.publishing.service.gov.uk/media/675ad406fd753588dc3a5a18/DVLA_annual_report_and_accounts_2023_to_2024.pdf https://assets.publishing.service.gov.uk/media/675ad406fd753...
- IshKebab 2y ago
  I would be really really surprised if the database actually encodes all of these edge cases you are thinking about in a structured way. In other words, I really doubt there's code like `if engine_age > drivetrain_age` or whatever.
  - pixl97 2y ago
    The point is until you start ripping the application apart you have no idea what the internal logic looks like. When you look you can find terrors that will haunt you in the night where some ancient limit, say number of columns in a database end up holding multiple structures that are getting if/then'd later in the application. I would completely and totally believe there is code just like that.
    - MichaelZuo 2y ago
      Why can’t you draw a flowchart/diagram of the logic? Sure it may be so complex that a complete diagram would be dozens of feet across and tall on a real piece of paper. But with apporpriate software tools that is not a limitation.
knallfrosch 2y ago
The problem is that the full set of specifications accumulated over three decades of usage is exactly as complicated as the code that still runs. Just wait 10 more years and hope AI can solve it. In the meantime, people can't renew their driver's license at 3:36. So what? Is that a requirement?

shermantanktop 2y ago

It’s funny to me that I would never ask those questions. I’ve specialized in legacy rehab projects (among other things) and there seems to be no upper bound on how bad things can be or how many annoying reasons there are for why we can’t “just fix it.” Those “just” questions—which I ask too—end up being hopelessly naive. The answers will crush your soul if you let them, so you can’t let them, and you should always assume things are worse than you think. TFA is spot on - the way to make progress is to cut problems up and deliver value. The unfortunate consequence is that badness gets more and more concentrated into the systems that nobody can touch, sort of like the evolution of a star into an eventual black hole.

abigail95 2y ago
I made a lot of money moving mid size enterprises from legacy ERP systems to custom in house ones. The DVLA dataset and the computations that are run on it can be studied and replicated in 3 months by a competent team. From there it can be improved. There is no way that this system requires 13 hours of downtime. If it required two hours - even if the code was generated through automation it can be reverse engineered and optimized. It is absolute rubbish that this thing is still unavailable outside of 8am-7pm. I maintain my position that it could be replaced in 3 months. I got my start in this business when I was in university and they told us our online learning software was going offline for 3 days for an upgrade. Those are the gatekeepers and low achievers we fight against. Think bigger.
- arccy 2y ago
  it's a gov agency, they don't quite pay enough for a motivated competent team....
  - gunian 2y ago
    even in this economy where people can't even work with their own SSN?
- monkey_monkey 2y ago
  > The DVLA dataset and the computations that are run on it can be studied and replicated in 3 months by a competent team. From there it can be improved. Such an HN comment. Made me lol. Think funnier!
- pixl97 2y ago
  Ya I don't think I'd let you in two miles of a system like this. Replacing legacy stuff always expands in scope far beyond the initial changes. When you have to come back and add wait() entries in your new program because it spits data back faster than the old program ever could which then causes peripheral devices/drivers to crash which then pulls a dev and testers off something else important for days figuring out what kind of fresh hell is occurring is just status quo for ancient systems.
  - gunian 2y ago
    idk much about dev much less legacy / enterprise dev but it seems like an A/B test type situation where you have 90% of the users running the legacy code and the remaining 10 on a new implementation would be feasible any idea why this is the case?
    - pixl97 2y ago
      This is what happens. All that testing with the required stakeholders takes way way more time than you'd expect. Gets even more fun in .gov where the work can change significantly at particular times of the year. Had one piece of Windows software required by the State of Texas used at year end like once a year. Seemingly nobody realized windows updates had stopped it from working until a few weeks before the deadline. I had to setup a box without updates for it to run for my customer. Lead to a lot of panic around the state.

that_guy_iain 2y ago

> Edit: Why does rebuilding take a decade or more? This is not a complex system. It doesn't need to solve any novel engineering challenges to operate efficiently. Article does not give much insight into why this particular task couldn't be fixed in 3 months. You do know the UK government has been cutting all their budgets to the bone for about 10 years? That means everywhere is pretty much understaffed. And how do you know it's not a complex system? I would think that a system like that would be somewhat complex. It's not just driving licenses but a whole bunch of other things that are handled by the DVLA.

abigail95 2y ago
The system may or may not be complex but the data is has to store and transform is not. Because it handles drivers licenses. A function that has been done on pen and paper and filing cabinets. Study the data, study the operations, reduce complexity. Since you imply you know more about UK budgets than I do - how much is the DVLA budgeted for IT operations like this and how much more would you give them to expect this problem solved? I can argue real numbers but vibes about bone dry budgets I cannot.
- that_guy_iain 2y ago
  > The system may or may not be complex but the data is has to store and transform is not. Because it handles drivers licenses. A function that has been done on pen and paper and filing cabinets. It handles more than just driving licenses... The DVLA do more than just driving licenses. > Since you imply you know more about UK budgets than I do - how much is the DVLA budgeted for IT operations like this and how much more would you give them to expect this problem solved? It's not budgeted anything for this as far as I know. I believe it's handled by Government Digital Services which handles lots of the digital services for various departments. The budget for all of GDS is about 90 million most of which isn't for .gov.uk. A rewrite of that size I would expect to cost about 50-60 million in total but take several years.
  - skippyboxedhero 2y ago

ajnin 2y ago

The batch jobs don't take 13 hours. They're just scheduled to run some time at night where the old offices used to be closed and the jobs could be ran with some expectations regarding data stability over the period. There are probably many jobs scheduled to run at 1AM then 2AM, etc, all depending on the previous to be finished so there is some large delay to ensure that a job does not start before the previous one is finished. As to your "not a complex system" remark, when a system is built for 60 years, piling up new rules to implement new legislation and needs over time, you tend to end up with a tangled mess of services all interdependent that are very difficult to replace piece-wise with a new shiny architecturally pure one. This is closer to a distributed monolith than a microservices architecture. In my experience you can't rebuild such a thing "in 3 months". People who believe that are those that don't realize the complexity and the extraordinary amount of specifics, special cases, that are baked into the system, and any attempt to just rebuild from scratch in a few months hits that wall and ends up taking years.

abigail95 2y ago
The code will be spaghettified and hideous. The queries will be nonsense. That doesn't change the fact that the ultimate goal of the system is to manage drivers licenses. > In my experience you can't rebuild such a thing "in 3 months". Me and my team rebuilt the core stack for the central bank of a developing country. In 3 months. The tech started in the 70s just like this. Think bigger.
- mattmanser 2y ago
  Yeah, I always raise an eyebrow at attitudes like that too. I've also reimplemented or gradually replaced several out-of-date systems. Albeit on a smaller scale. In my experience, when you start picking the programs apart you find 90% of the code is redundant or boilerplate. Much of it isn't even called from anywhere, abandoned code, and can be deleted en masse. A lot of programmers don't clean code up "just in case" and then no-one else deletes it. They can also often be vastly simplified because programmers back then didn't have the patterns and knowledge to write consisely. I often find myself simplifying the original code first, which gets rid of 50% of it. Then I can see what the code actually does and rewrite it which gets rid of the other 40%. On the other hand, many programmers don't have the patience, stubbornness or skill to do this kind of work. And the ability to get through the major panic you have when you're half way through and wondering if you were mad to even start.

firefoxd 2y ago

Our systems took 8 hours to back up. Then it grew to 12 hours [0]. The system was a side project by an intern fresh out of college. Over the years, it grew into a crucial software the company relied on. I joined over 10 years later and was able to bring it down to few minutes. [0]: https://news.ycombinator.com/item?id=38456429 https://news.ycombinator.com/item?id=38456429

jdietrich 2y ago

Per their own data, the DVLA are responsible for the records of 52 million drivers and 46 million vehicles. Those records are immensely complex, because they reflect decades of accumulated legislation, regulation and practice. Every edge case has an edge case. There's someone, somewhere in the bowels of the DVLA who understands the rules for drivers with visual field defects who use a bioptic device. There's someone who knows which date code applies to a vehicle that has been built with a brand new kit chassis but an old engine and drive train. There's someone who understands the special rates of tax that apply to goods vehicles that are solely used by showmen, or are based on certain offshore islands. God help any outsider who has to condense all of that institutional knowledge into a working piece of software. Government does not have a good track record of ground-up refactors of complex IT systems. The British government in particular does not have a good track record. Considering all that, the fact that most interactions with DVLA can be done entirely online is borderline miraculous. https://assets.publishing.service.gov.uk/media/675ad406fd753588dc3a5a18/DVLA_annual_report_and_accounts_2023_to_2024.pdf https://assets.publishing.service.gov.uk/media/675ad406fd753...

IshKebab 2y ago
I would be really really surprised if the database actually encodes all of these edge cases you are thinking about in a structured way. In other words, I really doubt there's code like `if engine_age > drivetrain_age` or whatever.
- pixl97 2y ago
  The point is until you start ripping the application apart you have no idea what the internal logic looks like. When you look you can find terrors that will haunt you in the night where some ancient limit, say number of columns in a database end up holding multiple structures that are getting if/then'd later in the application. I would completely and totally believe there is code just like that.
  - MichaelZuo 2y ago
    Why can’t you draw a flowchart/diagram of the logic? Sure it may be so complex that a complete diagram would be dozens of feet across and tall on a real piece of paper. But with apporpriate software tools that is not a limitation.

knallfrosch 2y ago

The problem is that the full set of specifications accumulated over three decades of usage is exactly as complicated as the code that still runs. Just wait 10 more years and hope AI can solve it. In the meantime, people can't renew their driver's license at 3:36. So what? Is that a requirement?