9 ms·
Uber Eats Canada Ran Production in a MacBook Air
- prepend 3y agoIt’s funny how people assume things. It seems odd that the assumption would be that this laptop runs production. I don’t think anyone who builds things with tech would assume this.
- solardev 3y agoAny respectable company would've used a Mac Mini.
- TylerE 3y agoMy current company, within the past few years, ran the production web server on a couple of mac minis (and like, not exactly new ones either). Seemed to work ok, honestly. Our DB boxes, which do much more heavy lifting are not now (and weren't then) Mini's, or anything close, though, heh.
- dkjaudyeqooe 3y agoWhy? The MacBook has a built-in UPS.
- solardev 3y agoWas just kidding, but in all seriousness, the Mini has slightly better performance and cooling (not by some huge margin), and it's easier to connect accessories. But yeah, having a battery might be good for some use cases.
- meitros 3y agoIt ran something that would cause Uber eats Canada to go down if it got disconnected, but not all of Uber eats production in one laptop (imo)
- Waterluvian 3y agoSo basically: all of the horror with none of the impressiveness.
- paulcole 3y agoI mean if the options you’re facing are Uber Eats Canada running because of something happening on a laptop or Uber Eats Canada not running, it’s not a hard choice to make. Fix the underlying problem later — or don’t.
- inhumantsar 3y agoWould love to hear the story behind this
- ummzokbro 3y agoBasically ran a macro to adjust how far restaurants would deliver by time of day / day of week. The goal was to balance supply and demand of restaurants and orders. Eats product didn't have any demand shifting levers at that time yet restaurants were getting blown up with demand at peak times (ex. top pizza restos at 7pm on a Friday). The product did however allow ops teams to manually set a delivery radius for a given restaurant. Clever ops teams scripted shrinking / expanding this delivery radius to encourage demand at non-peak times and reduce it during period where restaurants were overloaded and likely to ignore / cancel orders. This laptop ran scripts for all cities in Canada to adjust delivery radii for each restaurant until a few different levers were built into product. Ian (who is a _super nice_ guy) owned this process which kept things working smoothly while Eats was experiencing explosive growth. Another fun fact - Eats Marketplace, which is the current Eats you see today, was built and launched in Toronto - SF engineers / product / design supported by Toronto ops teams. Quickly eclipsed the Instant Eats product which was the 3rd (and most successful to date) try by Uber at food delivery.
- ambrood 3y agolmao... figured it was radius reduction or somesuch. i ended up working on Winterfell (the service that managed automated radius reduction) at some point during my time there.
- meepmorp 3y agoNot even an MBP. Professionalism is dead.
- metaphor 3y ago...over wifi??
- givemeethekeys 3y agoMaybe Ian wanted to ensure that no one touched his laptop.
- api 3y agoThis was obviously temporary but is actually cool for showing how powerful modern machines actually are. People routinely underestimate the load that can be handled by one box properly configured and running efficient code. I’ve found that modern developers almost always overestimate the amount of physical hardware needed to run things. I’m not sure why.
- john_max_1 3y agoI was once interviewed by Lyft and I told them that a 64 GB Mac Book Pro is sufficient to run the core business. The interviewer didn't believe me. I did the calculations in front of him. This was the time, I just learned Rust. I did the full analysis and showed him that QPS is great. Of course, my design didn't include 500 microservices, 50 deployments of Hadoop, Kafka, Elastic, Mongo DB, and 500 slightly different dashboards that different teams in Lyft probably use. Most modern tooling in ZIRP phenomenen. Someone's Hadoop experience - https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-...
- meepmorp 3y agoAnd did you get the job?
- john_max_1 3y agoI did. I rejected them as I didn't believe in the future of taxi-app industry.
- eyegor 3y agoHow do you define "core business"? Any laptop os would crumple purely by running out of resource handles on the connections, even without doing work. I've run into file descriptor / handle size limits on small clusters before, I can't imagine trying to deal with 1m+ concurrent tcp connections.
- toast0 3y ago
- hprotagonist 3y agothat’s nothing! how about air traffic control in a text editor? https://www.reddit.com/r/emacs/comments/lly7po/comment/gnvzisy/ https://www.reddit.com/r/emacs/comments/lly7po/comment/gnvzi...
- dmbche 3y agoWhat a gem!!
- groby_b 3y agoOuch. German ATC software in the 90s seems to have been a special place. (My favorite war story is "Hey, we have somebody 10 miles out from FRA airport, low on fuel, ATC software is down, can we escalate" - yes, yes we can.)
- rad_gruchalski 3y agoMayday, mayday! Hello, can you hear us? Can you hear us? Can you ... Over... We are sinking! We are sink... Hello, this is the German Coast Guard. We are sinking! We're sinking! What are you 'sinking' about?
- frankfrankfrank 3y ago[dead]
- Karupan 3y agoFor those who didn’t get this gem - https://youtu.be/gmOTpIVxji8 https://youtu.be/gmOTpIVxji8
- dvrp 3y agowoah
- tiffanyh 3y agoemacs is practically an operating system. So no surprise here really.
- rozal 3y ago[dead]
- throwawaaarrgh 3y agoUber's market cap is 93 billion right? What the fuck are they doing with all that money
- jahewson 3y agoThe shares worth that amount are sat in members of the public’s trading accounts.
- Barrin92 3y agoGiven that Uber Eats is losing money on every order the evaluation might actually be that high because, not despite of the laptop given that it can thankfully only do so many orders at any given time
- deterministic 3y agoUber is not making money. Which makes it not a business. More like a scam to extract $ from investors.
- endisneigh 3y agoLegality aside I’d love how many people you could serve with $2000 worth of hardware (used or new) and a 1gbit up/down home internet connection With max p99 latency of 500ms. It is strange that they didn’t at least use a Mac mini, but I assume they already had the hardware.
- deleted 3y ago[deleted]
- hot_gril 3y agoBuilt-in UPS and peripherals are nice advantages of a laptop as a server.
- sedatk 3y agoThe problem with a laptop is terrible cooling that you can't sustain high performance. They say new Macbooks are better in that aspect but didn't have a chance to try one.
- hot_gril 3y agoYeah. M1/2 ones are a lottt better than Intel in this regard but still slower than the M1/2 MBP or mini.
- euroderf 3y agoGood point.
- thequadehunter 3y agoI wonder if it's because the laptop sorta has its own built in UPS.
- sedatk 3y agoI ran one of the popular web sites in Turkey on a AMD Duron 600Mhz 512MB RAM server until 2003. It used to serve 70 million pages per month on a Windows Server when it was retired. Because the site is a social platform, most of the content had to be dynamically generated too. I believe the PC had cost me around $500 when I bought it in 2000 or so, obviously a lower spec than even a Raspberry Pi4 today.
- 0_____0 3y agoThere's a difference between "hosting all of Eats Canada" and "hosting some component or service until we can fix prod." I think it's more likely similar to the latter
- dvrp 3y agoThat was my thoughts as well. Maybe someone from Uber Eats Canada can clarify.
- whalesalad 3y agowhile true; do ssh intern@eats.uber.ca 'find /var/log -type f -delete'; sleep 300; done
- bpicolo 3y agoThis tracks. At a former employer, one of their acquired startups (that I joined on to help guide technical direction for) ran their entire order flow by calling a url every minute via a cron task from the primary database server. At one point, the newer devops organization migrated to a bigger, beefier MySQL (this is when fast NVMe was first available in specialized EC2 instances). Nobody remembered to migrate the cron script at that moment, so for about an hour I ran all of customer order flow with a bash script curling a public facing (yikes) url to initialize those jobs.
- adrr 3y agoProbably was running a script that ssh'ed into production boxes and restart some service at some specified interval or cleared some log files to stop the drive from filling up.
- water-data-dude 3y ago“until we can fix prod”, but then other stuff comes up, and just look at the backlog! And, I mean, it works pretty well for now…and before you know it, it’s been there forever
- LispSporks22 3y agoI too, have deployed my laptop to production
- PlunderBunny 3y ago(Desktop app, but...) whenever I or another developer would say "it works fine on my computer", my boss would sarcastically reply "so, we should put your computer on the customer's site then?"
- isaacremuant 3y agoAnd that's how docker was built.
- nathants 3y agotoo bad docker for mac isn’t very good. maybe someday.
- pkaye 3y agoSo all the microservices are running on that one system?
- barbazoo 3y agoNot clear but very unlikely.
- netsharc 3y agoSemi off-topic, anyone else get annoyed when people use the word "ping" to mean "contact (me)"? Sadly I've never had the chance to ask "Ah, are you ICMP compliant?" in response to a "ping me!"...
- dsr_ 3y agoI assume someone does, but my friends and family will readily respond with "ack" or "pong".
- eventhorizon77 3y agomy reply is usually something like 64 bytes from eventhorizon77: icmp_seq=1 ttl=64 time=31m
- abdullahkhalids 3y agoI think the origin of that ping is the "ping" sound that various instant messengers made/make on receiving a message, not some nerd reference to internet protocols.
- deleted 3y ago[deleted]
- mcsniff 3y agoPing me when you have a chance to use that comeback, I'm curious to know how it goes. Jokes aside, I use it when I'm chatting (IM) with other net-ops people, it just basically means "get ahold of me however you think is best based on the urgency".
- seized 3y agoBlackberry Messenger had a ping built in. Nothing since has had an equivalent feature. It was absolutely amazing for getting someones attention urgently.... And for annoying friends in their meetings.
- ambrood 3y agoeven for 2015 (when this was likely taken) Uber Eats would most certainly not have run from a single laptop. Core services like dispatch, matching, pricing etc in any case weren't geography specific. If i had to guess, this was some Ops person running a local driver incentive, payout query + email workflow from their laptop.
- kard 3y agoPractical.
- _8j50 3y agoWhen Maersk got ransomwared, they had to use a surface tablet for the domain controller of the company until things went back to normal.
- deterministic 3y agoThat’s really interesting. Any links?
- Altimor 3y agoOP — I’m not that privy to the details of what in the world was running on that MacBook Air, but I’m 100% sure it wasn’t “the production servers for Uber eats Canada.” It was some kind of hack doing some kind of thing on a recurring basis to make sure the service continued. These things were often hacked together by engineers, and sometimes ops people, in the middle of outages, as a stop gap.
- JohnMakin 3y agoyep definitely pulled this move before. Probably on a laptop because it is the only thing that has permissions to do whatever insanely janky thing it’s doing.
- justinclift 3y agoSounds like something probably in the System keychain, that's not getting unlocked when the user is logged out. Incidentally, the command to unlock the system keychain (eg in a cronjob) is: systemkeychain -vt That doesn't seem to be widely known, and is very useful for things like build jobs. ;)
- woleium 3y agoNot that I condone this sort of thing, but folks who do need to do it should know about caffeine [0] which allows a Mac to disable it's lid switch and keep running with the lid closed. [0] https://www.macupdate.com/app/mac/24120/caffeine https://www.macupdate.com/app/mac/24120/caffeine
- Acen 3y agoSide note, Caffeine[1][2] does not disable it's lid switch, but instead disables the automatic sleep behaviour while enabled. [1] Author link (includes link to source): https://www.intelliscapesolutions.com/apps/caffeine https://www.intelliscapesolutions.com/apps/caffeine [2] Original author link (stopped updating before OSX Mojave): https://www.lightheadsw.com/caffeine/ https://www.lightheadsw.com/caffeine/
- mrmincent 3y agoI’ve worked for big companies in the past where mission critical cron jobs were run on a Linux vm running off an old little desktop tucked away on a benchtop in the corner that no one ever noticed. Done to bypass various network controls to allow us to fix problems as they popped up. Absolute cowboy stuff, but effective.
- winrid 3y agoOne place I worked at had some data science stuff that pulled data from prod on a cron and re-uploaded the results every day, on some core2duo in the back of the office. AFAIK it was still being used in 2017 and might still be lol. It was a pretty powerful machine because someone got it to see how much money he could get the company to spend.
- robbywashere_ 3y agoSuch a strange way to run a kubernetes cluster.
- deleted 3y ago[deleted]
- vishnugupta 3y agoI’m not surprised. Amazon payment service’s reconciliation and settlement system was run off of a developer desktop. My memory is hazy as it was 18 years ago but I think it was due to some IP restrictions by banks and Amazon didn’t have tech to migrate IPs across machines.
- upon_drumhead 3y agoI totally believe the job was on a desktop, but I’ll call bullshit on the IP reason. The desktops don’t have direct external IPs allocated, so there’s a NAT with standard DHCP. There was no IP migration issues to deal with on the internal network.
- skykooler 3y agoThat title is quite the garden-path sentence - I got halfway through rationalizing how Uber would eat a Canada-ran production until I figured it out.
- ajani 3y agoIt’s called clickbait.
- rel 3y agoI was just reading in "Losing the Signal"[1] about how Blackberry's relay service of the late 1990s was sometimes run on a sysadmins laptop just because they needed anything to help deal with the traffic they weren't equipped to deal with. [1] https://us.macmillan.com/books/9781250096067/losingthesignal https://us.macmillan.com/books/9781250096067/losingthesignal
- jkestner 3y agoI ran a national self-moving company’s website in the early days off of an original iMac with Apache on Mac OS X Server 1.0 (not a typo). Mostly static pages with a bit of PHP/CGI/SSI back then, but still zippy with a ton of traffic thanks to our obsession with file size. Highly unprofessional!
- xarope 3y agoI'm perfectly fine with using laptops as edge servers in remote areas (Oh did you say Canada?). The thinkpad W series were workhorses, you could get 32GB of RAM and they had built in UPSs. Not that confident about Macbook Airs though, didn't they use to overheat?
- ransackdev 3y agoAs Morty would say, “Sounds like a cron job but with extra steps”
- Cody_C 3y agoI'll be honest, while it probably didn't run the whole production system it probably did something important enough to at least get wired internet.