5 ms·
This is what happens when optimists win and the realists are cut out of the conversation. As a taxi service, I believe I would want to know if I'm about to hav
by hinkley 4y ago
This is what happens when optimists win and the realists are cut out of the conversation.
As a taxi service, I believe I would want to know if I'm about to have a shortage of taxis in any one area of town, and I'd better only have a concentration in one area of town for an event the entire world is talking about, like a reunion tour or a championship game.
Even with the hack, the moment all of the taxis started converging on one area of town, alarms should have been going off and managers should have been asking questions. But that's not what happened, because we say yes the moment money enters the conversation, without bothering to ask what it says about you as a person if you'll do anything for money, or for that matter if the money is even real or just a trick to get our attention.
- monksy 4y agoThis is more of what happens when you do the least effort to build a product to make a buck. They're probably optimized for the average happy path, however flooding isn't a concern until someone gets upset.
- lbriner 4y agoNot necessarily. Despite us armchair critics, it is also very easy to miss an attack vector when building your software. We find stuff after years that we can't believe we missed like a missing auth check. Not that unusual at all when you are talking about 10s of 1000s of lines of code written by different people over the span of about 8 years.
- hinkley 4y agoThat’s why I favored detection at the top. I’ve worked on complex code signing apps that the blockchain people would recognize. Shit is hard. You can’t stop many things and still make money. But if you figure out what the boundaries are of the nominally running system, you can chart or earn when you start to lose the plot. I prefer charts over alerts, because as the company grows we keep forgetting to update the alerts. But then you need people who look at the charts between other tasks or you won’t catch anything and have to go back to alerts.
- tenebrisalietum 4y agoThere are always going to be individuals that say yes the moment money enters the conversation, as long as food and housing cost money and there is the possibility of going without.
- mike_hock 4y agoUniversal Basic Income now!
- tjs8rj 4y agoIt’s already so hard to build a large company, you just don’t have the resources to chase super rare, low pain outcomes. This is the first time this has happened and the total cost of it is at most a few hours revenue. They’ll likely add safeguards to prevent such a thing now, but if they ran the company preparing for every possible way things could go wrong, they’d get absolutely nothing done.
- tcgv 4y agoGood reasoning. Hindsight bias comes to mind: - https://en.wikipedia.org/wiki/Hindsight_bias https://en.wikipedia.org/wiki/Hindsight_bias
- onion2k 4y agolow pain outcomes All your customers thinking your app isn't secure any more isn't "low pain".
- ivan_gammel 4y ago99% of customers won’t care, because they will only briefly see the news, this hack did not harm them, they don’t care that much about security of an app and they don’t have a good alternative. The impact of such incidents on company reputation and revenue is often exaggerated.
- munk-a 4y agoA few customers will have strong negative opinions "I was waiting at the airport in the rain for four hours!" but most people will indeed shrug this off. It's a much different issue than what happens when payment systems are compromised. A lot more people care if they're informed their credit card was stolen and told to carefully watch statements for the next month - that leverages a real PITA cost on the customer.
- and-not-drew 4y agoThat's got nothing to do with what we're talking about. The first comment didn't say they should have spent more time on security, it said they should have spent time creating a system to detect if too many taxis were in one spot. I think we can all agree that security is valuable and should be prioritized, but spending time worrying about how to stop who is already in your system from sending all the cabs to the wrong place seems like a waste of time. Hell, IF (big if) the worst thing a hacker could do once they had access YandexTaxi's servers is send a bunch of cabs to the wrong place, you could almost spin that in a positive light. "We spent so much time protecting customer data that all they could do is send our divers to the wrong place".
- munk-a 4y agoIn most areas taxi-companies use a zone-based system where cars will flag what zone they're in (rarely automatically using GPS and more often via button presses) this is an effort by the cab company to keep their vacant vehicles well distributed to keep a high response rate and increase customer turnover. It also happens to have the side benefit that an operator watching the flagged zones would be able to see this kind of an issue happening in advance and maybe check into why every cab is suddenly bee-lining it to zone 3.
- wildzzz 4y agoBut there should still be some override that would allow for a bunch of taxis to converge at one spot. Say a sporting event just got out, there's going to be a lot of people looking to catch a ride home. If you don't want all of those customers finding another ride, the system should have no problem dispatching drivers from other zones to pickup. Having a bunch of fares popup at the same location shouldn't be a major concern and it sounds like there were no safeguards preventing every driver from being dispatched. Without just adding a limit, like no more than 50% of taxis can be dispatched to a single zone, I'm not really sure how you could prevent this from happening again. I don't know exactly how the hack happened but if someone was just able to manually spam the dispatch queue directly, the only thing you could do going forward would be to place an automated check on every addition to the queue that it's from a real user with a valid credit card and that no other requests from that user exist in the queue.
- hinkley 4y agoIt’s a distributed system right? How do you prevent saturation of a single service? Backpressure. Add artificial delays to the queuing time, increasing for each taxi.
- lstodd 4y agoMaybe that was the case 10 years ago. Or 15. Nowadays it's all automatic, there are no predefined zones - only past statistics and about zero operators. edit: srsly, that's what Uber is all about. and YTaxi is one of Moscow Ubers.
- renewiltord 4y agoWho is cutting anyone out of the conversation? You sell your product and if I care for 100% uptime, I'll pay for it. I actually don't. I can route through lots of stuff for appropriate savings and most people can. No one wants this single pair of instances in a Tier 4 datacenter that host a single key-pair authenticated process with dual manual approval and an air-gap that dispatches one taxi (and precisely one taxi) every 30 days on a route where it can be guaranteed to hit its time prediction. Any fool can build a bridge that stands. It takes an engineer to build a bridge that barely stands.
- hinkley 4y agoI guarantee you there are two ex employees saying “I told you” right now. Faster faster faster always wins because that’s what the management wants to hear. As long as their options best before the consequences stack up, they have no - and accept no - responsibility for the longevity of the company. If you haven’t worked with any defectors than you’ve managed better than many of us, or you’re very lucky.
- renewiltord 4y agoSure. But it's a market. It's possible for consumers to exercise their preference. And overwhelmingly they don't care about this stuff.
- grishka 4y agoYandex in particular has a system where it would dynamically adjust the price to prevent that sort of thing happening. When many people want to order a taxi to the same place, it gets really expensive, really fast. Uber does that too. This normally works well, but I feel like this hack bypassed the normal ordering system entirely and just sent bogus orders straight to drivers.
- klntsky 4y agoNew customers are allowed to pay in cash to the driver, which makes it effectively anonymous. Nothing was at stake.
- d1sxeyes 4y agoYandex.Taxi, like Uber (in fact, they merged with Uber in Russia), is not really a 'taxi service', they're a marketplace. A real taxi firm would notice and stop taking new calls to the address, but Yandex.Taxi aren't really 'dispatching' taxis, they're just advertising jobs, and letting drivers respond in real time. In fact, I'd imagine that almost none of the orders placed are reviewed in realtime, and the only indicator that anyone would have had for this to begin with would have been a higher than average number on the dashboard for 'trips requested today' - an interesting metric, but not something that I would expect to be monitored closely in real time. I'd imagine there's a 'no show' procedure that doesn't involve human oversight, so the first couple of drivers likely arrived at the address, waited a few minutes, then coded in the no show and moved on to different jobs. This is also likely a metric on a dashboard which would have been the second indicator - booking cancellations/no-shows/driver rejections. But again, it's an analytics metric, rather than realtime actionable business intelligence, so it's the sort of thing that gets put into weekly reports. Maybe someone would have seen it and thought 'huh, that's a bit high', but probably didn't trigger any alarms. Eventually a curious taxi driver would start to question why there are so many taxis outside this address, and would get out of his car and chat to his colleagues. They'd identify that they'd all been asked to the same address, and probably all cancel together and drive off. MAYBE the third indicator here would be a call from one of the drivers to customer support, letting them know about the 'system glitch' that meant multiple taxis were waiting at the same address, but it's equally possible that the drivers just moved onto their next fare without reporting any issue. So potentially, the first time that anyone at YT realised there was a serious issue was already 10-15 minutes after the incident occurred, by which time, it's already late. On top of that, it's unlikely that they have a way to easily and effectively cancel all bookings to a particular address. I don't have any details on the hack itself or YT's infrastructure, so it may have been very difficult to identify and cancel the fraudulent bookings en masse (e.g.: fuzzed addresses, booking times, different users, card details not stored or different card numbers used, etc.). By the time it got escalated to any technical teams, we're already likely 30-40 minutes into the incident itself, at which point they have to analyse what is happening, trace how it happened, and identify a fix. With the immediate nature of taxi booking (I want a taxi NOW, not in 45 minutes), it doesn't surprise me that an incident like this can occur before any technical measures can be put in place to stop or mitigate it.