8 ms·

Why some DVLA digital services don't work at night

124 points by edent 2y ago

pestatije 2y ago
DVLA - Driver & Vehicle Licensing Agency plus, since im already posting a comment: its because there is no batch window to process transactions
ForHackernews 2y ago
Unpopular opinion, but I think many systems would benefit from a regular "downtime window". Not everything needs to be 24/7 high availability. Maybe not every night, but if you get users accustomed to the idea that you're offline for 12 hours every Sunday morning, they will not be angry when you need to be offline for 12 hours on a Sunday morning to do maintenance. The stock market closes, more things should close. We are paying too high of a price for 99.999% uptime when 99.9% is plenty for most applications.
- kragen 2y ago
  Basically this happens because the DVLA and the stock market don't have any competition. Customers in a competitive market won't be angry when you need to be offline for 12 hours every Sunday morning; they'll just switch to your competitor some Sunday, because the competitor is providing them something they value that you don't provide.
  - ForHackernews 2y ago
    Maybe they should regulate Sunday trading hours, or unionized sysadmins should negotiate the end of on-call hours. The red queen's race that you describe for ever-greater scale, ever-greater availability is an example of the tragedy of the commons. Think how much money and many human minds have been wasted trying to squeeze out that last .0001% of "zero downtime" when they could have been creating something new. "Keep doing the same thing, but more of it, harder" is a recipe for a barren world of monoculture.
    - lifeoflejf 2y ago
      Bergen county NJ has blue laws that make it so non-grocery stores must be closed on Sunday’s. Maybe there’s some value in structuring a time where everybody is off? Just like at work the only time I really get off is when all of my customers are off. It’s nice when the industry sorta shuts off for a week or so around christmas
    - kragen 2y ago
      Something like that might plausibly be correct, though you've exaggerated it to a level where it's clearly false. If we steelman it to its most defensible essence, I think what you're saying is that the cost of the human effort needed to provide these higher uptimes exceeds the consumer benefit (the value of being able to buy a camera on Saturday), say. You could imagine, for example, that each incremental improvement in uptime wins over a proportion of the customer base providing a value that vastly exceeds its cost — but only until your competitors improve their own offering to match, so all the surplus from all this uptime improvement ultimately goes to the consumers, not the producers. There are two related holes in this idea. The first is that producing consumer surplus is what the economy is for, in a moral sense. The reason producing goods and services is a good thing to do is so that someone will benefit from using them! So if all the effort that sysadmins make goes into making services better for users, that's a good thing, not a bad thing. The second is that nothing is stopping a new entrant from offering a new, low-cost service that isn't as reliable. If the cost of providing all that extra reliability (bundled into the incumbents' pricing scheme) is higher than the actual benefit to users, the users will switch to the lower-cost, less-reliable service. This has happened many times, in fact: less-reliable minicomputers stole business from mainframes, less-reliable VoIP stole business from ATM and SONET and SDH, all kinds of less-reliable plastic goods have stolen business from all-metal versions, and now solar panels are stealing business from coal power plants even though solar panel "uptime" is like 30%. So the particular market dynamics we're talking about actually sensitively optimize the amount of effort given to uptime to the economic optimum. There do exist lots of market failures, but the particular dynamic we're discussing is the opposite extreme from something like a dollar auction.
      - ForHackernews 2y ago
  - ajnin 2y ago
    The stock markets definitely have competition. For instance Frankfurt, London, Paris or Amsterdam very much compete with each other to offer desirable conditions for investors, and companies will move their trading from one to another if it is their interest. I think the fact they close at night is a self-preservation mechanism, traders would become insane if they had to worry about their positions 24/7.
    - kragen 2y ago
      There's a very strong network effect, and most stocks are only listed on a single stock exchange, so in most contexts the competition is very minimal.
- OJFord 2y ago
  It only really works where the audience is already limited in country/timezone though. Sure a global service could just stagger the downtime around the world.. but (unless you've already equivalent partitioned the infrastructure) then you're just running 24/7 with arbitrary geofencing downtime on top.
- jmwilson 2y ago
  Who works Sunday morning then? The maintenance window will morph into a do-big-risky-changes window, which means everybody in engineering will have to be on-call. Many years ago, when I newly joined a FAANG, I asked, "shouldn't I run this migration after hours when load is low?" and the response was firm, "No, you'll run it when people are around to fix things". It may not always be the answer, but in general, I want to do maintenance when people are present and willing to respond, not nights and weekends when they're somewhere else and can't be found.
  - ForHackernews 2y ago
    Make your maintenance window Tuesday morning, then. The principle holds: not every stupid website needs to be up 99.999% of the time. You are not Amazon, and hell, even Amazon probably doesn't need to be online all the time. People can come back in a few hours to order dropshipped crap.
- crazygringo 2y ago
  > Not everything needs to be 24/7 high availability. If it makes you more money to be available 24/7 then why wouldn't you? > Maybe not every night, but if you get users accustomed to the idea that you're offline for 12 hours every Sunday morning Then I would use a competitor that was online, period. Imagine Sunday morning if the only time you have to complete a certain school assignment, but Wikipedia is offline? Or you need to send messages to a few folks that they need to see by the evening, but the platform won't come online until 3pm, which means you'll need to interrupt your afternoon family time instead? Maybe things closing works fine for your needs and your schedule. But it sure won't for everyone else. Having services that are reliable is one of the things that distinguishes developed countries from developing ones.
  - corint 2y ago
    > If it makes you more money to be available 24/7 then why wouldn't you? Agreed, but for a government service where you update your license, or tell them about selling a car or something, there's no real 'more' money. Being closed at 3am doesn't lose the opportunity in the way that it would if you were selling widgets. It instead forces the would-be users at 3am to wait until the morning.
  - ForHackernews 2y ago
    > If it makes you more money to be available 24/7 then why wouldn't you? This is the question, right? Is there anything we won't do to make money? We used to put 9 year olds down the mines. Those little kids are good in the small tunnels. Why wouldn't you use them most profitably? > you need to send messages to a few folks that they need to see by the evening, but the platform won't come online until 3pm, which means you'll need to interrupt your afternoon family time instead? I guess I'm old, but I remember when there were polite hours to call people on the phone and times that you just did not call. The world carried on fine for centuries before 24/7 instant messaging became a thing. > Having services that are reliable is one of the things that distinguishes developed countries from developing ones. Do you think that a business that has published hours and adheres to them is "unreliable"?
rozab 2y ago
I've often ran into this when using DVLA services and spluttered with indignation. But at the end of the day, these services are fantastically usable (during the daytime) and I appreciate Dafydd pushing to just get them out there! I got my license in 2015 so never in my life have I had the apparently ubiquitous American experience of queuing at the DMV and filling in paper forms. (is this still real? or limited to stand-up comedy?)
- nsxwolf 2y ago
  The queues have been mostly replaced with "take a number" systems where you can sit down and wait... with your... papers... that you had to fill out first...
  - fn-mote 2y ago
    > The queues have been mostly replaced with "take a number" systems where you can sit down and wait... My recent experience was: sign up online and get a 30 min window (9:00-9:30 say). Queue everyone for that 30 minute window outside the building. At exactly 9:30, enter and go through the usual queues inside. The advantage is that getting through those queues now takes 30 minutes or less because their length is limited. Presumably we/they traded volume of processing for certainty of time spent in the queue. A very familiar tradeoff for a computer scientist.
    - skippyboxedhero 2y ago
      The UK has a similar situation for hospitals. Your driving licence system sounds like a great embarrassment though.
      - nsxwolf 2y ago
        And the employees are mean! Really mean!
- AlotOfReading 2y ago
  Queuing at the DMV and filling out paperwork is very much a real thing that still happens. It's a pretty different experience in every state though.
mike_hearn 2y ago
tl;dr same reason other services go offline at night: concurrency is hard and many computations aren't thread safe, so need to run serially against stable snapshots of the data. If you don't have a database that can provide that efficiently you have no choice but to stop the flow of inbound transactions entirely. Sounds like Dafydd did the right thing in pushing them to deliver some value now and not try to rebuild everything right away. A common mistake I've seen some people make is assuming that overnight batch jobs that have to shut down the service are some side effect of using mainframes, and any new system that uses newer tech won't have that problem. In reality getting rid of those kinds of batch jobs is often a hard engineering project that requires a redesign of the algorithms or changes to business processes. A classic example is in banking where the ordering of these jobs can change real world outcomes (e.g. are interest payments made first and then cheques processed, or vice-versa?). In other cases it's often easier for users to understand a system that shuts down overnight. If the rule is "things submitted by 9pm will be processed by the next day" then it's easy to explain. If the rule is "you can submit at any time and it might be processed by the next day", depending on whether or not it happens to intersect the snapshot taken at the start of that particular batch job, then that can be more frustrating than helpful. Sometimes the jobs are batch just because of mainframe limitations and not for any other reason, those can be made incremental more easily if you can get off the mainframe platform to begin with. But that requires rewriting huge amounts of code, hence the popularity of emulators and code transpilers.
- abigail95 2y ago
  Do you know why the downtime window hasn't been decreasing over time as it gets deployed onto faster hardware over the years? Nobody would care or notice if this thing had 99.5% availability and went read only for a few minutes per day.
  - mike_hearn 2y ago
    It doesn't get deployed onto faster hardware. Mainframes haven't really got faster.
    - 2y ago
abigail95 2y ago
Something is missing here, why do batch jobs take 13 hours? If this thing was started on an old mainframe why isn't the downtime just 5 minutes at 3:39 AM? Exactly how much data is getting processed? Edit: Why does rebuilding take a decade or more? This is not a complex system. It doesn't need to solve any novel engineering challenges to operate efficiently. Article does not give much insight into why this particular task couldn't be fixed in 3 months.
- shermantanktop 2y ago
  It’s funny to me that I would never ask those questions. I’ve specialized in legacy rehab projects (among other things) and there seems to be no upper bound on how bad things can be or how many annoying reasons there are for why we can’t “just fix it.” Those “just” questions—which I ask too—end up being hopelessly naive. The answers will crush your soul if you let them, so you can’t let them, and you should always assume things are worse than you think. TFA is spot on - the way to make progress is to cut problems up and deliver value. The unfortunate consequence is that badness gets more and more concentrated into the systems that nobody can touch, sort of like the evolution of a star into an eventual black hole.
  - abigail95 2y ago
    I made a lot of money moving mid size enterprises from legacy ERP systems to custom in house ones. The DVLA dataset and the computations that are run on it can be studied and replicated in 3 months by a competent team. From there it can be improved. There is no way that this system requires 13 hours of downtime. If it required two hours - even if the code was generated through automation it can be reverse engineered and optimized. It is absolute rubbish that this thing is still unavailable outside of 8am-7pm. I maintain my position that it could be replaced in 3 months. I got my start in this business when I was in university and they told us our online learning software was going offline for 3 days for an upgrade. Those are the gatekeepers and low achievers we fight against. Think bigger.
    - arccy 2y ago
delta_p_delta_x 2y ago
Some DVLA services don't work in the day, too. Case in point, the 'get a share code' service: https://www.viewdrivingrecord.service.gov.uk/driving-record/licence-number https://www.viewdrivingrecord.service.gov.uk/driving-record/...
glonq 2y ago
This sounds a bit familiar. I used to work at a medium-sized company whose systems were based on COBOL code and Unisys mini/mainframe hardware from the 80's. We even had a person employed as a "tape ape"; thankfully not me. Throughout the next decade or two they tried various 4GL-generated facades and bolt-ons but could never escape from that COBOL core. Eventually I think they migrated the software to some kind of big box that emulated the Unisys environment but was slightly more civilized. I have no idea whether they ever eradicated all the COBOL though.
arjie 2y ago
While these explanations are plausible, certain other things I've encountered make me believe that deeper reasons underlie even these reasons. When I lived in the UK in 2017 as a foreigner, all applications for a driving licence as a foreigner on a T2-ICT visa had to be sent over for a couple of weeks and you had to include your passport and Biometric Residence Permit and everything. By comparison, I was able to get my driving licence at the California DMV pretty easily even as a foreigner and my passport and so on were photocopied and not retained. This drastic difference in service ability between the DVLA and a notoriously disliked American government service lead me to believe that the proximal technical causes for this are downstream from organizational choices for how to deliver service.
- robertlagrant 2y ago
  > downstream from organizational choices for how to deliver service 100000%. They're a monopoly service you must interact with or get fined and (eventually) locked up. They have zero incentive to do a particularly good job. Some orgs in this situation are just well run and do a good job, but there's no competitive pressure for them to do so.
  - Y_Y 2y ago
    And there are pressures other than competition, and some people just want to do a particularly job just because it's their job.
    - robertlagrant 2y ago
      Yeah, as I say. But the other pressures are often temporary and political.
IOT_Apprentice 2y ago
This seems weird to me. The number of records is minuscule compared to internet scale tech. The data model for this sounds like it would be simple. Exactly how many use cases are there to be implemented? Build this with modern tech on HA Linux backends. Eliminate the batch job nonsense. This could be written up as a project for bootcamps or even a YouTube series. I suspect some internal politics about moving forward and clinging to old methods is at hand. Perhaps someone could build an open source platform if the requirements were made public.
- dhosek 2y ago
  The thing is that a lot of internet scale stuff tends to be non-critical. It’s not a big deal if 1% of users don’t see a post to a social network site. It’ll show up later, maybe, or never, but nobody will care. On the other hand, with transactions like banking or licensing or health insurance, it’s absolutely essential that we definitely maintain ACID compliance for every single transaction, which is something that many “internet-scale” data solutions do not and often cannot promise. I have a vague recollection of some of the data issues at a large health insurance company where I worked a couple years ago that made it really clear why there would be an overnight period where the system would be offline—it was essential to make sure that systems could be brought to a consistent state. It also became clear why enrolling someone in a new plan was not simply a matter of adding a record to a database somewhere. Not to mention that I suspect that data such as bank transaction records or health insurance claims probably rival “internet scale” for being real big data operations.
  - mh- 2y ago
    The reason that these "internet scale" solutions are challenging to operate is because of their latency and availability targets. If you threw into the requirements "can go down nightly, for hours, for writes AND reads", they could absolutely provide the transactional guarantees you're looking for.
neuroelectron 2y ago
I'm sure the upgrade would have been trivial for a competent expert to do but instead they outsourced it to a big software firm and surprise, it went over-budget. Seriously, what could this database be doing that's so complicated?
simonbarker87 2y ago
Excellent bit of pragmatism and as a user of this service I’m happy with the trade off. People wondering why it’s not a simple switch and “there must be something else going on here” have clearly never worked with layers of legacy systems where the data actually matters. Sure it’s fixable and it’s a shame it hasn’t been but don’t assume there aren’t very good reasons why it’s not a quick fix. The gov.uk team have moved mountains over the past decade, members of it have earned the right to be believed when they say “it’s not simple”.
- NVHacker 2y ago
  [flagged]
  - simonbarker87 2y ago
    Yes I’d imagine the reason it still hasn’t been fixed after nearly a decade is management/politics etc. But it taking more than just 6 months will be technical. As a result it’s a job that falls into the area of being canned because it’s taking too long even though no one said it would be quick.
    - mjevans 2y ago
      There might be legal / compliance reasons. It can be incredibly difficult to replace a validated system that is known (or already accepted even if it's technically incorrect) to implement lawmaker dictated behavior. Otherwise, I think a new approach might be to ignore the specifics of the old system, implement a new system, and a separate translation layer that can run on an export of the old system (or the old system brought back online, but read only after the overnight maintenance) and completely cut over during an otherwise holiday weekend.
      - mgsouth 2y ago
        > I think a new approach might be to ignore the specifics of the old system, implement a new system It doesn't work like that. When you're revamping large, important, fingers-in-everything-and-everybody's-fingers-in-it systems you can't ignore anything. A (presumably) hypothetical example is sorting names. Simple, right? You just plop an ORDER-BY in the SQL, or call a library function. Except for a few niggling details: 1. This is an old IBM COBOL system. That means EBCDIC, not UTF or even ASCII. 1.A Fine, we'll mass-convert all the old data from EBCDIC to UTF. Done. 1.A.1 Which EBCDIC character set? There are multiple variants. Often based on nationality. Which ones are in use? Can you depend on all records in a dataset using the same one (hint: no.) Can you depend on all fields in a particular record using the same one? (hint: no.) Can you depend on all records using the same one for a particular field? (hint...) Can you depend on any sane method for figuring out what a particular field in a particular record in a particular dataset is using? Nope nope nope. 1.A.2 Looking at program A, you find it reads data from source B and merges it with source C. Source B, once upon a time, was from a region with lots of French names, and used code page 279 ('94 French). Except for those using 274 (old Belgium). And one really ancient set of data with what appears to be a custom code set only used by two parishes. Program A muddles through well enough to match up names with C, at least well enough for programs D, E, and F. 1.A.3 But it's not good enough for program G (when handling the Wednesday set of batches). G has to cross-reference the broken output from A with H to figure out what's what. 1.B You have now changed the output. It works for D and F, but now E is broken, and all the adhoc, painstakingly hand-crafted workarounds in G are completely clueless. 1.C Oh, and there's consumer J that wasn't properly documented, you don't know exists, and handles renewals for 60-70 year old pensioners who will be very vocal when their licenses are bungled. 2. Speaking of birth years, here's a mishmash of 2-, 4-, and even 3-digit years....
ajb 2y ago
This doesn't explain why the website can't just validate the input, queue up the transaction, and execute it later when the dB is available. It's not like your record changes that often. If the transaction fails they can then just send you an email to try again during working hours. All these processes used to work by paper forms, so most of them don't need interactive processing - that's why they have a batch job in the first place. My GP surgery has the same issue with non-urgent requests. It's entirely input, it's definitely not looking in a dB because it doesn't even ask who you are until the last step. And yet it won't accept an input except during working hours. Madness.
- mjevans 2y ago
  It does. That was a planned option as the next step in the phase of rebuilding / replacing things, but they opted to 'save money' and go for a full back-end replacement; which likely failed.
  - ajb 2y ago
    Ah thanks, I missed that somehow.
robertlagrant 2y ago
The cause is easy: the people specifying the problem to be solved don't ask for an independent set of tests that are human-readable and can be run automatically to verify the system. If they did that, then the system would be verifiable and so would changes to it - the tests would simply need to be adapted to talk to the new version of the system. Too late now, of course. But that's what should be done.
JSTrading 2y ago
[flagged]

124 points by edent 2y ago

DVLA - Driver & Vehicle Licensing Agency plus, since im already posting a comment: its because there is no batch window to process transactions

ForHackernews 2y ago

Unpopular opinion, but I think many systems would benefit from a regular "downtime window". Not everything needs to be 24/7 high availability. Maybe not every night, but if you get users accustomed to the idea that you're offline for 12 hours every Sunday morning, they will not be angry when you need to be offline for 12 hours on a Sunday morning to do maintenance. The stock market closes, more things should close. We are paying too high of a price for 99.999% uptime when 99.9% is plenty for most applications.

kragen 2y ago
Basically this happens because the DVLA and the stock market don't have any competition. Customers in a competitive market won't be angry when you need to be offline for 12 hours every Sunday morning; they'll just switch to your competitor some Sunday, because the competitor is providing them something they value that you don't provide.
- ForHackernews 2y ago
  Maybe they should regulate Sunday trading hours, or unionized sysadmins should negotiate the end of on-call hours. The red queen's race that you describe for ever-greater scale, ever-greater availability is an example of the tragedy of the commons. Think how much money and many human minds have been wasted trying to squeeze out that last .0001% of "zero downtime" when they could have been creating something new. "Keep doing the same thing, but more of it, harder" is a recipe for a barren world of monoculture.
  - lifeoflejf 2y ago
    Bergen county NJ has blue laws that make it so non-grocery stores must be closed on Sunday’s. Maybe there’s some value in structuring a time where everybody is off? Just like at work the only time I really get off is when all of my customers are off. It’s nice when the industry sorta shuts off for a week or so around christmas
  - kragen 2y ago
    Something like that might plausibly be correct, though you've exaggerated it to a level where it's clearly false. If we steelman it to its most defensible essence, I think what you're saying is that the cost of the human effort needed to provide these higher uptimes exceeds the consumer benefit (the value of being able to buy a camera on Saturday), say. You could imagine, for example, that each incremental improvement in uptime wins over a proportion of the customer base providing a value that vastly exceeds its cost — but only until your competitors improve their own offering to match, so all the surplus from all this uptime improvement ultimately goes to the consumers, not the producers. There are two related holes in this idea. The first is that producing consumer surplus is what the economy is for, in a moral sense. The reason producing goods and services is a good thing to do is so that someone will benefit from using them! So if all the effort that sysadmins make goes into making services better for users, that's a good thing, not a bad thing. The second is that nothing is stopping a new entrant from offering a new, low-cost service that isn't as reliable. If the cost of providing all that extra reliability (bundled into the incumbents' pricing scheme) is higher than the actual benefit to users, the users will switch to the lower-cost, less-reliable service. This has happened many times, in fact: less-reliable minicomputers stole business from mainframes, less-reliable VoIP stole business from ATM and SONET and SDH, all kinds of less-reliable plastic goods have stolen business from all-metal versions, and now solar panels are stealing business from coal power plants even though solar panel "uptime" is like 30%. So the particular market dynamics we're talking about actually sensitively optimize the amount of effort given to uptime to the economic optimum. There do exist lots of market failures, but the particular dynamic we're discussing is the opposite extreme from something like a dollar auction.
    - ForHackernews 2y ago
- ajnin 2y ago
  The stock markets definitely have competition. For instance Frankfurt, London, Paris or Amsterdam very much compete with each other to offer desirable conditions for investors, and companies will move their trading from one to another if it is their interest. I think the fact they close at night is a self-preservation mechanism, traders would become insane if they had to worry about their positions 24/7.
  - kragen 2y ago
    There's a very strong network effect, and most stocks are only listed on a single stock exchange, so in most contexts the competition is very minimal.
OJFord 2y ago
It only really works where the audience is already limited in country/timezone though. Sure a global service could just stagger the downtime around the world.. but (unless you've already equivalent partitioned the infrastructure) then you're just running 24/7 with arbitrary geofencing downtime on top.
jmwilson 2y ago
Who works Sunday morning then? The maintenance window will morph into a do-big-risky-changes window, which means everybody in engineering will have to be on-call. Many years ago, when I newly joined a FAANG, I asked, "shouldn't I run this migration after hours when load is low?" and the response was firm, "No, you'll run it when people are around to fix things". It may not always be the answer, but in general, I want to do maintenance when people are present and willing to respond, not nights and weekends when they're somewhere else and can't be found.
- ForHackernews 2y ago
  Make your maintenance window Tuesday morning, then. The principle holds: not every stupid website needs to be up 99.999% of the time. You are not Amazon, and hell, even Amazon probably doesn't need to be online all the time. People can come back in a few hours to order dropshipped crap.
crazygringo 2y ago
> Not everything needs to be 24/7 high availability. If it makes you more money to be available 24/7 then why wouldn't you? > Maybe not every night, but if you get users accustomed to the idea that you're offline for 12 hours every Sunday morning Then I would use a competitor that was online, period. Imagine Sunday morning if the only time you have to complete a certain school assignment, but Wikipedia is offline? Or you need to send messages to a few folks that they need to see by the evening, but the platform won't come online until 3pm, which means you'll need to interrupt your afternoon family time instead? Maybe things closing works fine for your needs and your schedule. But it sure won't for everyone else. Having services that are reliable is one of the things that distinguishes developed countries from developing ones.
- corint 2y ago
  > If it makes you more money to be available 24/7 then why wouldn't you? Agreed, but for a government service where you update your license, or tell them about selling a car or something, there's no real 'more' money. Being closed at 3am doesn't lose the opportunity in the way that it would if you were selling widgets. It instead forces the would-be users at 3am to wait until the morning.
- ForHackernews 2y ago
  > If it makes you more money to be available 24/7 then why wouldn't you? This is the question, right? Is there anything we won't do to make money? We used to put 9 year olds down the mines. Those little kids are good in the small tunnels. Why wouldn't you use them most profitably? > you need to send messages to a few folks that they need to see by the evening, but the platform won't come online until 3pm, which means you'll need to interrupt your afternoon family time instead? I guess I'm old, but I remember when there were polite hours to call people on the phone and times that you just did not call. The world carried on fine for centuries before 24/7 instant messaging became a thing. > Having services that are reliable is one of the things that distinguishes developed countries from developing ones. Do you think that a business that has published hours and adheres to them is "unreliable"?

rozab 2y ago

I've often ran into this when using DVLA services and spluttered with indignation. But at the end of the day, these services are fantastically usable (during the daytime) and I appreciate Dafydd pushing to just get them out there! I got my license in 2015 so never in my life have I had the apparently ubiquitous American experience of queuing at the DMV and filling in paper forms. (is this still real? or limited to stand-up comedy?)

nsxwolf 2y ago
The queues have been mostly replaced with "take a number" systems where you can sit down and wait... with your... papers... that you had to fill out first...
- fn-mote 2y ago
  > The queues have been mostly replaced with "take a number" systems where you can sit down and wait... My recent experience was: sign up online and get a 30 min window (9:00-9:30 say). Queue everyone for that 30 minute window outside the building. At exactly 9:30, enter and go through the usual queues inside. The advantage is that getting through those queues now takes 30 minutes or less because their length is limited. Presumably we/they traded volume of processing for certainty of time spent in the queue. A very familiar tradeoff for a computer scientist.
  - skippyboxedhero 2y ago
    The UK has a similar situation for hospitals. Your driving licence system sounds like a great embarrassment though.
    - nsxwolf 2y ago
      And the employees are mean! Really mean!
AlotOfReading 2y ago
Queuing at the DMV and filling out paperwork is very much a real thing that still happens. It's a pretty different experience in every state though.

mike_hearn 2y ago

tl;dr same reason other services go offline at night: concurrency is hard and many computations aren't thread safe, so need to run serially against stable snapshots of the data. If you don't have a database that can provide that efficiently you have no choice but to stop the flow of inbound transactions entirely. Sounds like Dafydd did the right thing in pushing them to deliver some value now and not try to rebuild everything right away. A common mistake I've seen some people make is assuming that overnight batch jobs that have to shut down the service are some side effect of using mainframes, and any new system that uses newer tech won't have that problem. In reality getting rid of those kinds of batch jobs is often a hard engineering project that requires a redesign of the algorithms or changes to business processes. A classic example is in banking where the ordering of these jobs can change real world outcomes (e.g. are interest payments made first and then cheques processed, or vice-versa?). In other cases it's often easier for users to understand a system that shuts down overnight. If the rule is "things submitted by 9pm will be processed by the next day" then it's easy to explain. If the rule is "you can submit at any time and it might be processed by the next day", depending on whether or not it happens to intersect the snapshot taken at the start of that particular batch job, then that can be more frustrating than helpful. Sometimes the jobs are batch just because of mainframe limitations and not for any other reason, those can be made incremental more easily if you can get off the mainframe platform to begin with. But that requires rewriting huge amounts of code, hence the popularity of emulators and code transpilers.

abigail95 2y ago
Do you know why the downtime window hasn't been decreasing over time as it gets deployed onto faster hardware over the years? Nobody would care or notice if this thing had 99.5% availability and went read only for a few minutes per day.
- mike_hearn 2y ago
  It doesn't get deployed onto faster hardware. Mainframes haven't really got faster.
  - 2y ago

abigail95 2y ago

Something is missing here, why do batch jobs take 13 hours? If this thing was started on an old mainframe why isn't the downtime just 5 minutes at 3:39 AM? Exactly how much data is getting processed? Edit: Why does rebuilding take a decade or more? This is not a complex system. It doesn't need to solve any novel engineering challenges to operate efficiently. Article does not give much insight into why this particular task couldn't be fixed in 3 months.

shermantanktop 2y ago
It’s funny to me that I would never ask those questions. I’ve specialized in legacy rehab projects (among other things) and there seems to be no upper bound on how bad things can be or how many annoying reasons there are for why we can’t “just fix it.” Those “just” questions—which I ask too—end up being hopelessly naive. The answers will crush your soul if you let them, so you can’t let them, and you should always assume things are worse than you think. TFA is spot on - the way to make progress is to cut problems up and deliver value. The unfortunate consequence is that badness gets more and more concentrated into the systems that nobody can touch, sort of like the evolution of a star into an eventual black hole.
- abigail95 2y ago
  I made a lot of money moving mid size enterprises from legacy ERP systems to custom in house ones. The DVLA dataset and the computations that are run on it can be studied and replicated in 3 months by a competent team. From there it can be improved. There is no way that this system requires 13 hours of downtime. If it required two hours - even if the code was generated through automation it can be reverse engineered and optimized. It is absolute rubbish that this thing is still unavailable outside of 8am-7pm. I maintain my position that it could be replaced in 3 months. I got my start in this business when I was in university and they told us our online learning software was going offline for 3 days for an upgrade. Those are the gatekeepers and low achievers we fight against. Think bigger.
  - arccy 2y ago

delta_p_delta_x 2y ago

Some DVLA services don't work in the day, too. Case in point, the 'get a share code' service: https://www.viewdrivingrecord.service.gov.uk/driving-record/licence-number https://www.viewdrivingrecord.service.gov.uk/driving-record/...

glonq 2y ago

This sounds a bit familiar. I used to work at a medium-sized company whose systems were based on COBOL code and Unisys mini/mainframe hardware from the 80's. We even had a person employed as a "tape ape"; thankfully not me. Throughout the next decade or two they tried various 4GL-generated facades and bolt-ons but could never escape from that COBOL core. Eventually I think they migrated the software to some kind of big box that emulated the Unisys environment but was slightly more civilized. I have no idea whether they ever eradicated all the COBOL though.

arjie 2y ago

While these explanations are plausible, certain other things I've encountered make me believe that deeper reasons underlie even these reasons. When I lived in the UK in 2017 as a foreigner, all applications for a driving licence as a foreigner on a T2-ICT visa had to be sent over for a couple of weeks and you had to include your passport and Biometric Residence Permit and everything. By comparison, I was able to get my driving licence at the California DMV pretty easily even as a foreigner and my passport and so on were photocopied and not retained. This drastic difference in service ability between the DVLA and a notoriously disliked American government service lead me to believe that the proximal technical causes for this are downstream from organizational choices for how to deliver service.

robertlagrant 2y ago
> downstream from organizational choices for how to deliver service 100000%. They're a monopoly service you must interact with or get fined and (eventually) locked up. They have zero incentive to do a particularly good job. Some orgs in this situation are just well run and do a good job, but there's no competitive pressure for them to do so.
- Y_Y 2y ago
  And there are pressures other than competition, and some people just want to do a particularly job just because it's their job.
  - robertlagrant 2y ago
    Yeah, as I say. But the other pressures are often temporary and political.

IOT_Apprentice 2y ago

This seems weird to me. The number of records is minuscule compared to internet scale tech. The data model for this sounds like it would be simple. Exactly how many use cases are there to be implemented? Build this with modern tech on HA Linux backends. Eliminate the batch job nonsense. This could be written up as a project for bootcamps or even a YouTube series. I suspect some internal politics about moving forward and clinging to old methods is at hand. Perhaps someone could build an open source platform if the requirements were made public.

dhosek 2y ago
The thing is that a lot of internet scale stuff tends to be non-critical. It’s not a big deal if 1% of users don’t see a post to a social network site. It’ll show up later, maybe, or never, but nobody will care. On the other hand, with transactions like banking or licensing or health insurance, it’s absolutely essential that we definitely maintain ACID compliance for every single transaction, which is something that many “internet-scale” data solutions do not and often cannot promise. I have a vague recollection of some of the data issues at a large health insurance company where I worked a couple years ago that made it really clear why there would be an overnight period where the system would be offline—it was essential to make sure that systems could be brought to a consistent state. It also became clear why enrolling someone in a new plan was not simply a matter of adding a record to a database somewhere. Not to mention that I suspect that data such as bank transaction records or health insurance claims probably rival “internet scale” for being real big data operations.
- mh- 2y ago
  The reason that these "internet scale" solutions are challenging to operate is because of their latency and availability targets. If you threw into the requirements "can go down nightly, for hours, for writes AND reads", they could absolutely provide the transactional guarantees you're looking for.

neuroelectron 2y ago

I'm sure the upgrade would have been trivial for a competent expert to do but instead they outsourced it to a big software firm and surprise, it went over-budget. Seriously, what could this database be doing that's so complicated?

simonbarker87 2y ago

Excellent bit of pragmatism and as a user of this service I’m happy with the trade off. People wondering why it’s not a simple switch and “there must be something else going on here” have clearly never worked with layers of legacy systems where the data actually matters. Sure it’s fixable and it’s a shame it hasn’t been but don’t assume there aren’t very good reasons why it’s not a quick fix. The gov.uk team have moved mountains over the past decade, members of it have earned the right to be believed when they say “it’s not simple”.

NVHacker 2y ago
[flagged]
- simonbarker87 2y ago
  Yes I’d imagine the reason it still hasn’t been fixed after nearly a decade is management/politics etc. But it taking more than just 6 months will be technical. As a result it’s a job that falls into the area of being canned because it’s taking too long even though no one said it would be quick.
  - mjevans 2y ago
    There might be legal / compliance reasons. It can be incredibly difficult to replace a validated system that is known (or already accepted even if it's technically incorrect) to implement lawmaker dictated behavior. Otherwise, I think a new approach might be to ignore the specifics of the old system, implement a new system, and a separate translation layer that can run on an export of the old system (or the old system brought back online, but read only after the overnight maintenance) and completely cut over during an otherwise holiday weekend.
    - mgsouth 2y ago
      > I think a new approach might be to ignore the specifics of the old system, implement a new system It doesn't work like that. When you're revamping large, important, fingers-in-everything-and-everybody's-fingers-in-it systems you can't ignore anything. A (presumably) hypothetical example is sorting names. Simple, right? You just plop an ORDER-BY in the SQL, or call a library function. Except for a few niggling details: 1. This is an old IBM COBOL system. That means EBCDIC, not UTF or even ASCII. 1.A Fine, we'll mass-convert all the old data from EBCDIC to UTF. Done. 1.A.1 Which EBCDIC character set? There are multiple variants. Often based on nationality. Which ones are in use? Can you depend on all records in a dataset using the same one (hint: no.) Can you depend on all fields in a particular record using the same one? (hint: no.) Can you depend on all records using the same one for a particular field? (hint...) Can you depend on any sane method for figuring out what a particular field in a particular record in a particular dataset is using? Nope nope nope. 1.A.2 Looking at program A, you find it reads data from source B and merges it with source C. Source B, once upon a time, was from a region with lots of French names, and used code page 279 ('94 French). Except for those using 274 (old Belgium). And one really ancient set of data with what appears to be a custom code set only used by two parishes. Program A muddles through well enough to match up names with C, at least well enough for programs D, E, and F. 1.A.3 But it's not good enough for program G (when handling the Wednesday set of batches). G has to cross-reference the broken output from A with H to figure out what's what. 1.B You have now changed the output. It works for D and F, but now E is broken, and all the adhoc, painstakingly hand-crafted workarounds in G are completely clueless. 1.C Oh, and there's consumer J that wasn't properly documented, you don't know exists, and handles renewals for 60-70 year old pensioners who will be very vocal when their licenses are bungled. 2. Speaking of birth years, here's a mishmash of 2-, 4-, and even 3-digit years....

ajb 2y ago

This doesn't explain why the website can't just validate the input, queue up the transaction, and execute it later when the dB is available. It's not like your record changes that often. If the transaction fails they can then just send you an email to try again during working hours. All these processes used to work by paper forms, so most of them don't need interactive processing - that's why they have a batch job in the first place. My GP surgery has the same issue with non-urgent requests. It's entirely input, it's definitely not looking in a dB because it doesn't even ask who you are until the last step. And yet it won't accept an input except during working hours. Madness.

mjevans 2y ago
It does. That was a planned option as the next step in the phase of rebuilding / replacing things, but they opted to 'save money' and go for a full back-end replacement; which likely failed.
- ajb 2y ago
  Ah thanks, I missed that somehow.

robertlagrant 2y ago

The cause is easy: the people specifying the problem to be solved don't ask for an independent set of tests that are human-readable and can be run automatically to verify the system. If they did that, then the system would be verifiable and so would changes to it - the tests would simply need to be adapted to talk to the new version of the system. Too late now, of course. But that's what should be done.

JSTrading 2y ago

[flagged]