5 ms·

This particular incident comes after another rail-related "cyber" incident, affecting TfL who run services in London. The previous incident was perpetrated by a

by lol768 2y ago

This particular incident comes after another rail-related "cyber" incident, affecting TfL who run services in London. The previous incident was perpetrated by a 17-year old, and TfL have still yet to re-enable the systems they turned off back at the end of August. This means customers can no longer check their contactless journey history or claim refunds online. My understanding is that staff were also shut off from being able to access internal systems. There is no ETA for restoration of service. I think there's a growing problem with digital competency in some of these organisations. TfL in particular have not kept up with the times, and their once revolutionary ticketing infrastructure and software (I say "their", but it's really all just outsourced to Cubic, there's very little in-house expertise or day-to-day ownership when it comes to this stuff) feels quite dated now.

Etheryte 2y ago
Not familiar with this particular system, but if the issue is that their whole webapp is pwnable spaghetti, then I doubt it's gonna come back online anytime soon. Those types of issues are usually endemic and no amount of whack-a-mole will fix it.
- deleted 2y ago
  [deleted]
- bugtodiffer 2y ago
  Really depends on the devs. I've seen it go both ways. But you are right, most retests I have done after the pentest issues were fixed is weird. They actively engineer around our suggested fixes instead of actually doing the work they are told to do
fidotron 2y ago
> I think there's a growing problem with digital competency in some of these organisations. TfL in particular have not kept up with the times, and their once revolutionary ticketing infrastructure and software (I say "their", but it's really all just outsourced to Cubic, there's very little in-house expertise or day-to-day ownership when it comes to this stuff) feels quite dated now. I've noticed this too. Tend to visit the UK annually or so over the last 15 years and the general quality of the machine interfaces goes down over time. The good ones are almost no different from how they were in 2005. The part they have improved on is the contactless integration with phones which does just work so dramatically reducing the need to use the rest.
- matt-p 2y ago
  "worse" doesn't ring true, they have really just stayed the same. IMO this is fine, as I've not needed to use a machine in a year or two. You buy rail tickets on your phone with QR code tickets and tap in and out of TFL with your choice of phone/watch/card/oyster. We don't need to revamp the buying a physical ticket machines, why? Who do they serve, only really tourists who don't know they just need to use thier card to tap in or use a website or app to buy either QR code or pick up train tickets.
  - fidotron 2y ago
    No, they are absolutely worse. I can still recall the GWR machines glitching out by themselves on my last visit, even without anyone near them. The absence of quality control was immediately obvious. The other thing that is obvious is the attention to detail in signage has declined. The electronic displays on the Elizabeth Line are flat out amateurish. I can understand how if you live there you wouldn’t notice as it is like the boiling frog.
    - matt-p 2y ago
      Oh yes, to be fair those are bad - they're the ones where they've removed the physical keyboard if I remember correctly. Elizabeth Line - the move to LCD screens was probably a move to make it feel more modern, no doubt at great cost. I never really have an issue with them though, maybe boiling a frog thing :)
switch007 2y ago
There's a slight perverse incentive in that not having access to Contactless history makes it harder for the customer to see overcharging and to make claims for a refund.
- avianlyric 2y ago
  The TfL contactless system doesn’t generally overcharge, as long as you properly tap-in and tap-out. The system as a whole seems to be designed to prevent overcharging at all costs. In situations where is ambiguous what you should be charged (there’s a number of interesting edge-cases), the system always charges you the lowest possible amount. Even in the event of user error (such as forgetting to tap-out), the system does it best the guess what you did, such as looking at your history and assuming you meant to tapped out at a station you normally use, or looking for a station where the gates where left open (due to a major event, overcrowding concern, evacuation etc), and assuming you left the system there. The contactless system is substantially more capable for handling these issues correctly, and generally better at charging less, than the older Oyster system.
  - reddalo 2y ago
    >In situations where is ambiguous what you should be charged (there’s a number of interesting edge-cases) I'm very curious to know more about those edge cases.
    - avianlyric 2y ago
      There are some interesting issues with the old oyster system when handling certain routes that involve out of station interchanges. Out of station interchanges effect make two sets of barriers behave like they’re part of the same physical station, allowing you to temporarily exit the system, re-enter, but only get charged once, as if you never exited the system at all. On the old oyster system this plays very badly with how the system handles you tapping in and out at the same station, at the start and end of a single journey. The Oyster systems assumes that you’ve made two trips, and not tapped out one way, and not tapped in the other way, and thus charges you two maximum fairs. Normally this isn’t an issue, but with out-of-station interchanges, it’s possible for you make a journey, leave the station, re-enter the station 20-30mins later, for your return journey (say you went to pick something up, and then went straight home), get caught within the out-of-station interchange, which effectively connects your return journey with your outbound journey. On the Oyster system, the out-of-station interchange is implemented by the re-entry barrier basically updating your card so it looks like you never left the system at all, as a consequence, when you tap out at your home station, the tap-out barrier thinks you’ve made two trips without properly tapping in and out, and charges you two maximum fairs. With the contactless system, every single tap in and out is recorded by the barriers in their backend system, and an end of day batch system then looks at all of your tap ins and outs, and computes your final fair. Because it can see every single tap, it’s capable of recognising that you made two legitimate journeys, that shouldn’t be linked to by the out of station interchange, and thus charge you two correct fairs. Unlike the Oyster system, which has to make fair charging decisions at each tap, and due to limited memory on the card, can’t accurately track every single tap, as a result it has to be more aggressive in it fair calculation approach, otherwise it would be trivial to exploit its limitations to defraud TfL. Although even the Oyster system will try and correct these issues. There are batch jobs that run on all the uploaded tap data collected by the barriers themselves, and those jobs do their best to try and spot errors like this, and perform automatic refunds. But it harder to perform those refunds because somehow the system has to the get the data on to the physical Oyster card, which uploading the patch into the barriers the system thinks you’re likely to use, so your card can be patched on the tap there.
avianlyric 2y ago
> I say "their", but it's really all just outsourced to Cubic, there's very little in-house expertise or day-to-day ownership when it comes to this stuff The original Oyster system was outsourced to Cubic. But the newer contactless system was built in-house by TfL, and is licensed to Cubic for sale to other transport systems around the world, including cities like New York. https://en.wikipedia.org/wiki/Oyster_card#Contactless_payment https://en.wikipedia.org/wiki/Oyster_card#Contactless_paymen...
- lol768 2y ago
  For various reasons, the contactless system sadly still lags behind Oyster, with basic functionality like NR discount entitlements missing and showing no signs of emerging anytime soon. The physical infrastructure, like the gatelines, are all still controlled by Cubic as well. Even in terms of security, you have TfL deciding to adopt SMS for multi-factor authentication (no TOTP/HOTP/push-based MFA) for their contactless portal. Which they only started enforcing a year ago. I don't get how anybody can defend decisions like these. We've known SMS is unsuitable for this use-case for years.
  - inferiorhuman 2y ago
    SMS is the de facto standard for MFA. Everyone's favorite ISP darling Sonic recently enabled SMS MFA on all accounts with no warning. Pretty much every financial institution I deal with offers SMS MFA (few offer any way to disable it). Retailers like Home Depot eschewed passwords in favor of SMS OTP. I wish it weren't so, but that's just how things are these days.
    - paulryanrogers 2y ago
      SMS MFA is better than no MFA. SMS as SFA is bad since it's easily spoofed.
      - ikekkdcjkfke 2y ago
        SMS MFA is usually used to reset password so it's almost always de facto SFA. Recent Veritasium video says overtaking anyones phone number costs just hundreds of dollars
KaiserPro 2y ago
The level of advancement isn't the issue here. What happened is that some kid phoned up helpdesk and said they were x and they had lost their phone, can you please help me to reset my password and 2fa (because the phone was nicked, its standard operating procedure right?) From there they managed to SSO into critical shit. Now, they've had to reset _everyone's_ password and 2fa (after an ID check) They also now need to go through _everything_ to make sure there aren't any backdoors, boobytraps or any unknown exfiltration events. that shit is _hard_, even harder for a safety critical place like TFL. They have something like 40k employees, and a whole bunch of disparate systems.
- lol768 2y ago
  > What happened is that some kid phoned up helpdesk and said they were x and they had lost their phone, can you please help me to reset my password and 2fa (because the phone was nicked, its standard operating procedure right?) Then the policies that they had in place for these eventualities were insufficient. > From there they managed to SSO into critical shit. We're all speculating here without many details of what actually happened, but the questions I'd be asking are: - Was the person whose account was compromised somebody that needed to routinely have access to Oyster card refund information? - Was the person whose account was compromised somebody that needed to routinely have access to Oyster card refund information for a handful of customers, as part of working in a support role? - Was the person whose account was compromised somebody that needed to have access to Oyster card refund for five thousand distinct customers, accessed in a very short timeframe? Why didn't security controls exist that prevented exfiltration of those volumes of data? - Why were TfL directly storing bank account details in the first place when there are better-equipped partners who have experience securing data that could have handled this? - Why is TfL so awful at providing Oyster refunds in the first place? They could very easily make it so you could get refunds back onto a Debit/Credit card (without needing to store the PAN), but instead the only options they give you are bank account/sort code (which have to be given over the phone) or back onto the Oyster card - needing to be physically collected within 3 to 4 days or the refund gets "lost" - completely useless if you don't live in London. > that shit is _hard_, even harder for a safety critical place like TFL. They have something like 40k employees, and a whole bunch of disparate systems. I don't care, it's not good enough. Everybody in London has no choice but to share some amounts of data with TfL if they want to use the transport system. They've shown they're incompetent time and time again. They can't do the work they've committed to do on-time either, and we pay them shedloads of public funds for this level of service.
0xDEAFBEAD 2y ago
I imagine big tech companies and startups in the UK pay better salaries, and are better at evaluating talent. That creates adverse selection in the remaining employee pool, which the public sector is fishing in. Same as in the US. What puzzles me is how Estonia managed to avoid this problem. https://www.youtube.com/watch?v=I5krZBe0Dck https://www.youtube.com/watch?v=I5krZBe0Dck
- lol768 2y ago
  I'd agree with this. At some point in the past, a final-salary defined benefit pension and the job security may have gone some way to levelling the playing field a bit.
- matt-p 2y ago
  The theory is right but in practice none of this is really run by the public sector anything remotely difficult is just contracted out to the private sector and they go with basically the cheapest bid from a "big" company. In this case it's telent and they probably just brought a captive portal product (the cheapest, of course ;)) and it got hacked.
  - matt-p 2y ago
    I'm just thinking about this. If they were to give it to a "public sector" style org who would it be? It would be interesting just to give the job to Jisc actually, eduroam at all mainline stations and for normal users a captive portal w/wpa3 for fist time registration/use then option of kicking you to 802.11x would be a fine solution and one they're enormously experienced at. Another option is giving it to TFL to run or some kind of joint procurement with them as TFL for whatever reason are vastly more agile and competent then national rail. Final option GDS get involved and come up with a standard solution, open source of course and managed in house.
ifwinterco 2y ago
Even before that cyberattack I was never able to get TfL online services working properly. Luckily it's not needed, you can just buy an Oyster from a corner shop and top it up at the machines in the station (all with cash if you're concerned about privacy, which is nice).

by lol768 2y ago

Etheryte 2y ago

Not familiar with this particular system, but if the issue is that their whole webapp is pwnable spaghetti, then I doubt it's gonna come back online anytime soon. Those types of issues are usually endemic and no amount of whack-a-mole will fix it.

deleted 2y ago
[deleted]
bugtodiffer 2y ago
Really depends on the devs. I've seen it go both ways. But you are right, most retests I have done after the pentest issues were fixed is weird. They actively engineer around our suggested fixes instead of actually doing the work they are told to do

fidotron 2y ago

> I think there's a growing problem with digital competency in some of these organisations. TfL in particular have not kept up with the times, and their once revolutionary ticketing infrastructure and software (I say "their", but it's really all just outsourced to Cubic, there's very little in-house expertise or day-to-day ownership when it comes to this stuff) feels quite dated now. I've noticed this too. Tend to visit the UK annually or so over the last 15 years and the general quality of the machine interfaces goes down over time. The good ones are almost no different from how they were in 2005. The part they have improved on is the contactless integration with phones which does just work so dramatically reducing the need to use the rest.

matt-p 2y ago
"worse" doesn't ring true, they have really just stayed the same. IMO this is fine, as I've not needed to use a machine in a year or two. You buy rail tickets on your phone with QR code tickets and tap in and out of TFL with your choice of phone/watch/card/oyster. We don't need to revamp the buying a physical ticket machines, why? Who do they serve, only really tourists who don't know they just need to use thier card to tap in or use a website or app to buy either QR code or pick up train tickets.
- fidotron 2y ago
  No, they are absolutely worse. I can still recall the GWR machines glitching out by themselves on my last visit, even without anyone near them. The absence of quality control was immediately obvious. The other thing that is obvious is the attention to detail in signage has declined. The electronic displays on the Elizabeth Line are flat out amateurish. I can understand how if you live there you wouldn’t notice as it is like the boiling frog.
  - matt-p 2y ago
    Oh yes, to be fair those are bad - they're the ones where they've removed the physical keyboard if I remember correctly. Elizabeth Line - the move to LCD screens was probably a move to make it feel more modern, no doubt at great cost. I never really have an issue with them though, maybe boiling a frog thing :)

switch007 2y ago

There's a slight perverse incentive in that not having access to Contactless history makes it harder for the customer to see overcharging and to make claims for a refund.

avianlyric 2y ago
The TfL contactless system doesn’t generally overcharge, as long as you properly tap-in and tap-out. The system as a whole seems to be designed to prevent overcharging at all costs. In situations where is ambiguous what you should be charged (there’s a number of interesting edge-cases), the system always charges you the lowest possible amount. Even in the event of user error (such as forgetting to tap-out), the system does it best the guess what you did, such as looking at your history and assuming you meant to tapped out at a station you normally use, or looking for a station where the gates where left open (due to a major event, overcrowding concern, evacuation etc), and assuming you left the system there. The contactless system is substantially more capable for handling these issues correctly, and generally better at charging less, than the older Oyster system.
- reddalo 2y ago
  >In situations where is ambiguous what you should be charged (there’s a number of interesting edge-cases) I'm very curious to know more about those edge cases.
  - avianlyric 2y ago
    There are some interesting issues with the old oyster system when handling certain routes that involve out of station interchanges. Out of station interchanges effect make two sets of barriers behave like they’re part of the same physical station, allowing you to temporarily exit the system, re-enter, but only get charged once, as if you never exited the system at all. On the old oyster system this plays very badly with how the system handles you tapping in and out at the same station, at the start and end of a single journey. The Oyster systems assumes that you’ve made two trips, and not tapped out one way, and not tapped in the other way, and thus charges you two maximum fairs. Normally this isn’t an issue, but with out-of-station interchanges, it’s possible for you make a journey, leave the station, re-enter the station 20-30mins later, for your return journey (say you went to pick something up, and then went straight home), get caught within the out-of-station interchange, which effectively connects your return journey with your outbound journey. On the Oyster system, the out-of-station interchange is implemented by the re-entry barrier basically updating your card so it looks like you never left the system at all, as a consequence, when you tap out at your home station, the tap-out barrier thinks you’ve made two trips without properly tapping in and out, and charges you two maximum fairs. With the contactless system, every single tap in and out is recorded by the barriers in their backend system, and an end of day batch system then looks at all of your tap ins and outs, and computes your final fair. Because it can see every single tap, it’s capable of recognising that you made two legitimate journeys, that shouldn’t be linked to by the out of station interchange, and thus charge you two correct fairs. Unlike the Oyster system, which has to make fair charging decisions at each tap, and due to limited memory on the card, can’t accurately track every single tap, as a result it has to be more aggressive in it fair calculation approach, otherwise it would be trivial to exploit its limitations to defraud TfL. Although even the Oyster system will try and correct these issues. There are batch jobs that run on all the uploaded tap data collected by the barriers themselves, and those jobs do their best to try and spot errors like this, and perform automatic refunds. But it harder to perform those refunds because somehow the system has to the get the data on to the physical Oyster card, which uploading the patch into the barriers the system thinks you’re likely to use, so your card can be patched on the tap there.

avianlyric 2y ago

> I say "their", but it's really all just outsourced to Cubic, there's very little in-house expertise or day-to-day ownership when it comes to this stuff The original Oyster system was outsourced to Cubic. But the newer contactless system was built in-house by TfL, and is licensed to Cubic for sale to other transport systems around the world, including cities like New York. https://en.wikipedia.org/wiki/Oyster_card#Contactless_payment https://en.wikipedia.org/wiki/Oyster_card#Contactless_paymen...

lol768 2y ago
For various reasons, the contactless system sadly still lags behind Oyster, with basic functionality like NR discount entitlements missing and showing no signs of emerging anytime soon. The physical infrastructure, like the gatelines, are all still controlled by Cubic as well. Even in terms of security, you have TfL deciding to adopt SMS for multi-factor authentication (no TOTP/HOTP/push-based MFA) for their contactless portal. Which they only started enforcing a year ago. I don't get how anybody can defend decisions like these. We've known SMS is unsuitable for this use-case for years.
- inferiorhuman 2y ago
  SMS is the de facto standard for MFA. Everyone's favorite ISP darling Sonic recently enabled SMS MFA on all accounts with no warning. Pretty much every financial institution I deal with offers SMS MFA (few offer any way to disable it). Retailers like Home Depot eschewed passwords in favor of SMS OTP. I wish it weren't so, but that's just how things are these days.
  - paulryanrogers 2y ago
    SMS MFA is better than no MFA. SMS as SFA is bad since it's easily spoofed.
    - ikekkdcjkfke 2y ago
      SMS MFA is usually used to reset password so it's almost always de facto SFA. Recent Veritasium video says overtaking anyones phone number costs just hundreds of dollars

KaiserPro 2y ago

The level of advancement isn't the issue here. What happened is that some kid phoned up helpdesk and said they were x and they had lost their phone, can you please help me to reset my password and 2fa (because the phone was nicked, its standard operating procedure right?) From there they managed to SSO into critical shit. Now, they've had to reset _everyone's_ password and 2fa (after an ID check) They also now need to go through _everything_ to make sure there aren't any backdoors, boobytraps or any unknown exfiltration events. that shit is _hard_, even harder for a safety critical place like TFL. They have something like 40k employees, and a whole bunch of disparate systems.

lol768 2y ago
> What happened is that some kid phoned up helpdesk and said they were x and they had lost their phone, can you please help me to reset my password and 2fa (because the phone was nicked, its standard operating procedure right?) Then the policies that they had in place for these eventualities were insufficient. > From there they managed to SSO into critical shit. We're all speculating here without many details of what actually happened, but the questions I'd be asking are: - Was the person whose account was compromised somebody that needed to routinely have access to Oyster card refund information? - Was the person whose account was compromised somebody that needed to routinely have access to Oyster card refund information for a handful of customers, as part of working in a support role? - Was the person whose account was compromised somebody that needed to have access to Oyster card refund for five thousand distinct customers, accessed in a very short timeframe? Why didn't security controls exist that prevented exfiltration of those volumes of data? - Why were TfL directly storing bank account details in the first place when there are better-equipped partners who have experience securing data that could have handled this? - Why is TfL so awful at providing Oyster refunds in the first place? They could very easily make it so you could get refunds back onto a Debit/Credit card (without needing to store the PAN), but instead the only options they give you are bank account/sort code (which have to be given over the phone) or back onto the Oyster card - needing to be physically collected within 3 to 4 days or the refund gets "lost" - completely useless if you don't live in London. > that shit is _hard_, even harder for a safety critical place like TFL. They have something like 40k employees, and a whole bunch of disparate systems. I don't care, it's not good enough. Everybody in London has no choice but to share some amounts of data with TfL if they want to use the transport system. They've shown they're incompetent time and time again. They can't do the work they've committed to do on-time either, and we pay them shedloads of public funds for this level of service.

0xDEAFBEAD 2y ago

I imagine big tech companies and startups in the UK pay better salaries, and are better at evaluating talent. That creates adverse selection in the remaining employee pool, which the public sector is fishing in. Same as in the US. What puzzles me is how Estonia managed to avoid this problem. https://www.youtube.com/watch?v=I5krZBe0Dck https://www.youtube.com/watch?v=I5krZBe0Dck

lol768 2y ago
I'd agree with this. At some point in the past, a final-salary defined benefit pension and the job security may have gone some way to levelling the playing field a bit.
matt-p 2y ago
The theory is right but in practice none of this is really run by the public sector anything remotely difficult is just contracted out to the private sector and they go with basically the cheapest bid from a "big" company. In this case it's telent and they probably just brought a captive portal product (the cheapest, of course ;)) and it got hacked.
- matt-p 2y ago
  I'm just thinking about this. If they were to give it to a "public sector" style org who would it be? It would be interesting just to give the job to Jisc actually, eduroam at all mainline stations and for normal users a captive portal w/wpa3 for fist time registration/use then option of kicking you to 802.11x would be a fine solution and one they're enormously experienced at. Another option is giving it to TFL to run or some kind of joint procurement with them as TFL for whatever reason are vastly more agile and competent then national rail. Final option GDS get involved and come up with a standard solution, open source of course and managed in house.

ifwinterco 2y ago

Even before that cyberattack I was never able to get TfL online services working properly. Luckily it's not needed, you can just buy an Oyster from a corner shop and top it up at the machines in the station (all with cash if you're concerned about privacy, which is nice).