8 ms·
Errors for the Twilio Rest API impacting multiple services
- IG_Semmelweiss 4y agoFor reference : https://www.cnbc.com/2023/02/13/twilio-layoffs-1500-employees-17percent-of-workforce.html https://www.cnbc.com/2023/02/13/twilio-layoffs-1500-employee...
- jotaen 4y agoDid you come up with the headline “Twilio Flex outage now 6 hours after laying off ~26% staff”? If so, could you provide any factual insights that back the implicit claim how this particular outage is the direct cause of the recent layoffs? Edit/Update: the title of the submission got changed.
- pram 4y agoVery cool. I already started moving all my 2FA stuff out of Authy but guess it needs to be hastened now.
- mfer 4y agoWhat alternative are you using?
- Axsuul 4y agoBitwarden
- grogenaut 4y agoIs there an easy way to import from authy to bw? Or do I have to do one at a time.
- Axsuul 4y agoSorry I'm not sure.
- brewdad 4y agoThe officially supported method is to login to each 2FA account and delete the old 2FA and generate a new one that you make sure to record separately. Authy really locks you in. There are some unsupported methods a Google search away. I have had luck with this one in the past but haven't used in over a year, so can't vouch for it still working. https://gist.github.com/gboudreau/94bb0c11a6209c82418d01a59d958c93 https://gist.github.com/gboudreau/94bb0c11a6209c82418d01a59d...
- mfer 4y agoI don't really like the idea of TOTP and passwords in the same place. If it gets compromised someone has both your password and TOTP code.
- Axsuul 4y agoYep I agree so I use 1Password only for passwords and Bitwarden only for TOTP.
- brewdad 4y agoI use Bitwarden for passwords only. I use Authy for the TOTP, although I should use something else. I backup my TOTP seeds in KeepassXC. I also have an offline backup of my Bitwarden vault that is in a separate KeePassXC vault. I agree that I don't like the idea of one vault holding both bits of info.
- toolz 4y agoI use https://github.com/tadfisher/pass-otp https://github.com/tadfisher/pass-otp with pass, which has a FOSS client for desktop and smartphone (at least for andriod, no idea for ios)
- zikduruqe 4y agoIt's built in to passforios https://github.com/mssun/passforios https://github.com/mssun/passforios
- widerporst 4y agoI migrated to Aegis a few months ago and would recommend it. Exporting the configuration was a bit tricky, but I found a guide on GitHub: https://gist.github.com/gboudreau/94bb0c11a6209c82418d01a59d958c93 https://gist.github.com/gboudreau/94bb0c11a6209c82418d01a59d...
- toomuchtodo 4y agoPasskeys if the site supports them as a secure authenticator for 2FA/MFA (should pop up in your device if you try to setup a secure hardware authenticator on iphones and android) or TOTPs stored securely somewhere that isn't vendor locked.
- stalfosknight 4y agoiCloud Keychain
- bwb 4y agohow does this impact Authy?
- Axsuul 4y agoTwilio owns Authy but regardless it doesn't. Authy is mostly used client-side with a server-side backup option.
- bwb 4y agoah gotcha, i didn't realize that
- ceejayoz 4y agoAuthy is owned by Twilio.
- deleted 4y ago[deleted]
- MBCook 4y agoTwilio owns Authy. I guess they’re assuming future issues there or with syncing.
- AdmiralAsshat 4y agoAuthy appears to be up, but I guess if it were down, the only issue would be trying to setup a new device and have it sync over the 2FA codes? My understanding is after that initial sync, the TOTP it displays should all work offline.
- btown 4y agoIMO it's good practice to not one's vital 2FA codes held hostage on a service where one's accounts or IP address can be flagged for spam or abusive behavior by automated systems. (This also applies to Google Authenticator, for what it's worth!) Especially in a world where customer service teams are being trimmed wherever possible! I use 1Password - its UI leaves some things to be desired, and it's not cheap, but it has zero incentive to cancel the account of any paying customer!
- nanidin 4y agoGoogle Authenticator is completely offline isn't it? How are accounts / 2FA material at risk of lockout by using Google Authenticator?
- jonny_eh 4y agoIt is, and because of that people regularly lose their vault when switching phones and forgetting to transfer the data.
- deleted 4y ago[deleted]
- theshrike79 4y agoI switched out of Google Authenticator when they updated the app and all my 2FA just went away. They did fix it with another update, but that was a seriously un-fun few days. Luckily I was just logged in to my AWS account so I could disable the 2FA.
- wonderwonder 4y agoI had no idea authy was owned by twilio. Well I know what I'm doing this weekend
- shapefrog 4y agoDoes this mean that the staff laid off were doing a good job or a bad job? Building a product that falls over within hours of reducing the head count doesnt suggest they did a very good job, but they were keeping a fragile system up all this time ...
- bbertelsen 4y agoAlternative hypothesis: someone had a backdoor and didn't take their layoff well.
- shapefrog 4y agoAlternative alternative hypothesis: someone got laid off and told to work out the rest of their shift, and used their frontdoor access to take a massive dump in the boses desk drawer.
- Axsuul 4y agoThis is just pure speculation - we don't actually know why it went down.
- BizarreByte 4y agoIt could be literally anything from a failed health check and failed auto mitigation, to a routine failure that no one actually knows who to contact about anymore, to sabotage by a laid off employee. It’s not really worth speculating about fired employees performance, but there’s probably internal chaos after yet another layoff round.
- mfer 4y agoThat depends on the reason. For example, did someone cripple a system on their way out the door to cause the outage? Is this some retaliation? If so it doesn't speak to the quality of what was built. I only use this example to speak to the wide array of reasons that could cause this. Without knowing the reason it's hard to assess anything.
- iforgotpassword 4y agoThey were good at making themselves indispensable, until they were dispensed.
- libpcap 4y agoNo wonder I don't receive any spammy robocalls or text messages today.
- pm90 4y agoFinally something good to come out of the layoffs.
- annoyingnoob 4y ago^ This. Twilio and Sendgrid are outright criminal in the way they handle their own fraudulent customers. Twilio makes money on every single one of those bogus communications and does nothing to proactively prevent that fraud because it would impact the bottom line. When your business is based on mostly fraud you need to find a new business.
- a2tech 4y agoConsidering how many hoops I have to jump through to use their services (under the guise of anti-fraud/spam) I'd love to know how they get away with it
- HappySweeney 4y agoMy guess is extra off-the-books money.
- annoyingnoob 4y agoI think the scammers plan to get kicked off, they get one good campaign out and then get kicked off. Then show up as a new customer with a new name and new payment info, but upload the same list(s) and start spamming again, over and over in perpetuity.
- fragmede 4y agoBuying the Twilio account of a failed startup
- guywithahat 4y ago
- gfd 4y agoReminds me of https://www.reddit.com/r/ProgrammerHumor/comments/10gtbrm/layoff_fiasco https://www.reddit.com/r/ProgrammerHumor/comments/10gtbrm/la...
- jbverschoor 4y agoCoincidinky
- sfc32 4y agoIt says 17% in the article. This is obviously a massive number, but just wondering where the 28% number is coming from? https://www.cnbc.com/2023/02/13/twilio-layoffs-1500-employees-17percent-of-workforce.html https://www.cnbc.com/2023/02/13/twilio-layoffs-1500-employee...
- deleted 4y ago[deleted]
- pawelos 4y ago> The announcement came after the company already laid off around 11% of its workforce as part of a restructuring plan in September.
- jp57 4y agoIf you lay off 11% then lay of 17% of what's left that's only a total layoff of 26% of the original number, but yeah.
- temp_praneshp 4y agoprobably just adding up 11% layoff last september (although the % maybe on different numbers)
- GlobalFrying 4y ago[flagged]
- hakre 4y agoWhat from? Please elaborate.
- bedast 4y agoInflating the layoff percentage doesn't serve anything except to sensationalize an already editorialized headline. The number is 17%, not 28%.
- bink 4y ago11% were laid off in Sept.
- lostlogin 4y agoSo is that total based on staff numbers from last year, or is the new rounds layoff percentage based on current staffing? It seems we are at the stage of layoffs were we can argue about how you measure the percentage sacked..
- bedast 4y agoThe original headline said 6 hours ago. September wasn't 6 hours ago last I checked.
- IG_Semmelweiss 4y agothe article states they laid off 11% , less than 5 months ago. That's 26.1% Anecdata, but in several years of being a Twilio customer, I have never seen such a long outage. EDIT: fixed % of %.
- ghostpepper 4y agoThat's only 28% if the 17% was calculated as a percentage of their pre-first-layoff headcount.
- pkulak 4y agoWell, it's 26%, but close enough.
- deleted 4y ago[deleted]
- p_j_w 4y ago
- marktangotango 4y agoThe old "oops laid of the wrong people" gag. Never gets old! I say that jokingly, but wow does it suck. I've been on both sides. I think it's actually worse to be not laid off and left picking up the pieces for incompetent management. One time I had to get admin rights and log into a "shadow IT" developers machine (who had been laid off) to get his Eclipse workspace directory and get the code and decipher what the state of his 5 apps were. Good times.
- guywithahat 4y agoYou're assuming this wasn't done maliciously
- shapefrog 4y agoThe whole thing might have been held together by chewing gum and they fired the one guy who re-applies it every day.
- jonathankoren 4y agoMost things are.
- deleted 4y ago[deleted]
- beberlei 4y agooccam's razor suggests that
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- MonkeyMalarky 4y agoYeah it's all fun and games until you discover someone had automated a thing using a cron job on their own computer.
- vuln 4y agoEng 1: Yeah we got backups, hit up Bob Eng 2: Bob left in December. Who replaced Bob? Eng 1: I don't know, his boss' email is coming back as invalid in outlook. Eng 2: Fk it I'll just do it myself. 1 hour later Eng 2: So backups stopped when Bob left in December. He was doing them manually due to an error in the automation. I'll try to restore from December. 5 hours later Eng 2: backups are corrupt and won't work. Looks like we've never restore from backup to production -------- Nation State Actor: WTB: Twilio access Recently Laid off Employee: I like money.
- Shorn 4y ago> Looks like we've never restore from backup to production Do you often restore backups to production for testing purposes?
- deleted 4y ago[deleted]
- jay-barronville 4y agoNot to be pedantic, but if they executed a 11% layoff and later executed a 17% layoff of the remaining, that does NOT add up to 28%. That math is incorrect. Please fix the title.
- ezfe 4y ago26.1%
- shapefrog 4y ago26.13%
- lostlogin 4y agoThis is the best I could find: “According to Twilio’s latest earnings release, the company had 8,992 employees as of September 30, 2022 and expected to lay off 816 employees for the 2022 round of layoffs. Based on these figures, around 1,400 people will be impacted by this year’s layoffs.” So that’s 2216 total out of 8992, 24.6% https://techcrunch.com/2023/02/13/twilio-cuts-17-of-its-workforce-just-months-after-previous-round-of-layoffs/amp/?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvLm56Lw&guce_referrer_sig=AQAAAMv-9KB6kjY22wzhJYhmtUtD0EqW7Q6B5owfYLDq4ZJSWZO5d_w1e31i-xTsGQpSJ4-VcaDHz5gwzxywbVWMF5KCms6nHOtShuqROf_TGTPUWkd-3RuHqZiP_2Z9mR_AHsmK8tB271NSwKnDt1ka3FB3IxmWBjGTdh9MhwK003sf https://techcrunch.com/2023/02/13/twilio-cuts-17-of-its-work...
- ryanisnan 4y agoBeing correct with mathematics isn't pedantic, it's just good sense.
- chucksmash 4y agoAnd compulsively annotating where other people are being incorrect with mathematics isn't good sense, it's just being pedantic
- ryanisnan 4y ago
- geogra4 4y agoMy old employer used twilio extensively for allowing the employees of customers to do automated clockouts/clockins. They seemed like a dumpster fire of a company to work with but that they had telephony figured out made them hard to dislodge from the market.
- haswell 4y agoThe title is heavily editorialized and is essentially clickbait. It's fine to speculate about things, but at first glance, the title looks like someone wrote a piece that might try to tie the outage to actual facts about employees who departed and how they might be relevant to the outage. This post is nothing of the sort, seems unlikely to provoke thoughtful discussion. Edit: title has been updated since I wrote this comment.
- choppaface 4y agoThe context about layoffs is important! Users depend on not just reliable services but predictable management. Here’s a related HN post describing a hiring effort during the layoffs https://news.ycombinator.com/item?id=34804077 https://news.ycombinator.com/item?id=34804077 “For example, Twilio 2 days ago announced they were laying off 1000+ people, and here they are opening a staff software engineering role a day later.” That’s just more evidence of significant incompetence at the leadership level that needs to be weighed when considering a service offering.
- debacle 4y agoI worked with a guy once that I wasn't a fan of. Gruff, incompetent, egotistical. Just a general ass. He got laid off two weeks before we were set to do a huge annual update. I stormed into the interim CTO's office (the CTO got laid off too) and explained that laying this guy off was the dumbest idea ever and that it had put the entire project in jeopardy and no one had the knowledge or access to systems that this guy had, and that laying him off was going to cost the company millions of dollars because the release was going to fail. I don't know what wound up happening at that release. Two weeks later I was working elsewhere; a few years later that company completely evaporated.
- rr808 4y agoOften the grumpy people are that way because they're lives are filled with fixing important problems that no one else can be bothered to do.
- maerF0x0 4y agoor they're so undervalued because their systems never hiccup so they never get to be seen as the hero. They get laid off because "They work on this stuff that is not high priority" ...
- deleted 4y ago[deleted]
- debacle 4y agoHe was grumpy because he was vastly overpaid, but snubbed for the CTO role, and knew his job was on the line.
- cortesoft 4y agoIf one guy leaving costs your company millions of dollars, you were screwed to begin with. What if he quit, or even just got sick for a few weeks?
- 4y ago
- jdlyga 4y agoGeneral layoffs are just never a good idea unless as the very last resort. I've personally seen the demoralizing effect it creates, triggering more people to leave the same year. For highly technical teams, you can never really replace the institutional knowledge you're losing.
- pojzon 4y agoI thought the term is “tribal knowledge” and yes it will cost companies ALOT if they fire incorrect ppl. Often times you have 2-3 ppl out of 6-8 that carry the cart while 3-4 are juniors or diversity hires. Firing those 3 will have devastating consequences, firing the other 4 wont make a dent. Firing based on excel is the most stupid thing you can do as a manager.
- sp332 4y agoI think "institutional knowledge" is more common.
- LarsDu88 4y agoThis is way worse than the Twitter layoffs. Twitter engineering built a system that was robust to a 50% layoff. Twilio -- not so much.
- marktangotango 4y ago> robust to a 50% layoff That's an interesting metric worth considering.
- lostlogin 4y agoIt really is, but it’s also a little crazy that a company could lose 50% of its staff and keep working.
- bigmattystyles 4y agoBut companies like twitter, until it was taken private at least, heavily depend on growth and adapting to emerging trends. You might keep the platform 'running' as is, but not more than that. I don't disagree that companies have people not pulling their weight but when there are large layoffs, the future of a company is, as the kids say, 'sus'. (Did I use that right?)
- lostlogin 4y agoDid Twitter heavily depend on growth? They seemed to flat line for an awfully long time.
- bigmattystyles 4y agoI guess not growth, but the potential for growth - as an investor, it's what I look for. If growth is already happening, then it's likely already priced in. If growth is not expected to happen, then unless there's a generous dividend, there is no reason to hold a stock. But if as an investor, you thought Twitter (when public) was primed for growth, it might have made a good bet as an investment. As it stands, I don't see it having any potential for growth, but again, the Twitter example is moot, as it's not public.
- lxm 4y agoIn retrospect, could layoffs be less drastic if the company moved to a lower cost of living area? Vs placing an adamant bet on paying SF rent, SF salaries and urging others to do the same? https://www.sfgate.com/local/article/Twilio-CEO-wants-tech-companies-to-stay-in-sf-15834503.php https://www.sfgate.com/local/article/Twilio-CEO-wants-tech-c...
- maerF0x0 4y agoOne credit due to Twilio is very friendly to remote besides perhaps the pay tiers. But everyone's doing that. They never mandated return to office, so that's in their favor too. iirc the context of the article you're referring to is more of Tesla, Oracle moving out of California
- tschellenbach 4y agoThis will likely cause more layoffs in the future.
- sassycharacter 4y agoClearly they should layoff the people responsible for the previous layoffs. They do not seem to know what they are doing.
- emkemp 4y ago"Those responsible for sacking the people who have just been sacked, have been sacked."
- function_seven 4y agohttps://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffal...
- culi 4y ago(insert spiderman pointing at spiderman meme)
- ohmanjjj 4y agoThey fired the one guy who does nightly reboots nobody knows about but are critical for freeing resources nobody else bothered to figure out how to.
- IanCal 4y agoI used to remember doing a thing where people would be taken from a team and sent to another for a while. During that time no contact with the original team was supposed to happen. Was supposed to help deal with some of the bus factor kinds of problems. Anyone still do that? Harder when so many are let go though.
- ugh123 4y agoSounds like some kind of "forced" rotation program, although the no-contact thing is kind of weird. Rotation programs can be useful for roles that aren't super-specialized, and also motivates people to write better documentation and automation tools.
- IanCal 4y agoThe no contact part was to force the rest of the team to deal with whatever came up, just as they would if the person had left or was on holiday.
- thwayunion 4y agoDuring the pandemic the real economy learned that lean supply chains are more efficient but also present a massive liability. That they needed to accept slightly higher costs in exchange for greater robustness. And then the entire tech world promptly ignored that lesson the very next year.
- lumost 4y agoHonestly not looking forward to a return to understaffed tech teams fighting on-call fires 70% of the time. I'm not entirely certain how much of the recent stability in larger firms was simply a result of having more people.
- nequo 4y agoI also wonder if the service industry labor shortage was somehow related to the tech hiring boom, if indirectly.
- smm11 4y agoInternational company, 500 staff in three countries, $30 million last year. I'm the only IT person, and I have secure servers in my garage.
- paraknight 4y agoI'm glad I migrated our 2FA from Twilio to Vonage some weeks ago after an awful experience with toll fraud (more info here: https://yousefamar.com/memo/log/2023-01-29-00-05-40/ https://yousefamar.com/memo/log/2023-01-29-00-05-40/)