19 ms·

LastPass says DevOps engineer’s hacked computer led to security breach in 2022

376 points by mikece 4y ago

LilBytes 4y ago
Am I assuming in this case the engineer was using his home PC to work? This isn't unheard of in the industry, Engineers using BYOD devices or similar to work from home. But with a company with a risk profile as high as LastPass this seems _incredibly dumb_. You would assume anyone with the keys to the kingdom was working on a company provided device, or any device that fits a compliance framework based on their own risk needs (which should be massive!). E.g., endpoint management, AV/detection, FIDO2 with Yubikeys to access AWS as an admin and otherwise. Just a few off the top of my head.
- SOLAR_FIELDS 4y ago
  I could see this kind of sophisticated attack easily working on some random company with fairly lax BYOD policies. Makes sense, it was a rather sophisticated attack when you look at your typical medium sized company. But if your entire company and organization is built around keeping things secure, THIS is what brought you down? This isn’t getting owned by some unforeseen 0day, it’s just sloppy opsec.
- Johnny555 4y ago
  My company isn't nearly as high profile or security focused and we're not allowed to use our own computers for any work related purposes,and our work laptops run threat detection software and we have a whitelist of software we're allowed to install. I'm surprised that LastPass's policies aren't at least that strict. My company has what I think is a big hole in this policy in that we're allowed to use our own phone for email, a few corporate apps (like Jira) and our corporate password manager (not LastPass), but IT doesn't do any management of phones (other than being able to wipe them remotely if you're connected to the company email server). I suspect that the company doesn't want to spend the money on giving everyone a managed phone.
  - hackernewds 4y ago
    As much as LastPass seems to be trying to pin this on a single engineer, and not a broad vuln, the fact that they have such lax policies around access management, especially for a password management system, tells me enough I need to know never to use them again. Waiting for the rebrand and the incoming lawsuits.
    - victor9000 4y ago
      Yeah, this is the equivalent of blaming an intern for nuking the prod database. Maybe they were careless of maybe that shouldn't be possible to begin with.
    - BLKNSLVR 4y ago
      Yep. The symptom being that a problem of this scale can be caused by a single engineer, which points to the root cause being deeper and potentially systemic. The question for future trust is: what's been / being done to prevent the same thing from happening again due to another single engineer?
      - macrolime 4y ago
        They answer that question directly here https://support.lastpass.com/help/what-have-we-done-to-ensure-lastpass-is-safe-to-use https://support.lastpass.com/help/what-have-we-done-to-ensur... TLDR;not much
      - execveat 4y ago
        Preventing this thing from happening costs a lot of $$$, so pretty much everyone just "accepts the risk" seeing that probability of something like this happening to your company (during your tenure) is still super low. All companies with somewhat robust security posture I know have had a string of incidents in the past, that seems to be the only thing that can motivate to put $ in security.
        Johnny555 4y ago
- metadaemon 4y ago
  I don’t think he’s necessarily working on his pc. He probably just had a shared LastPass account between work and his pc.
  - wolfi1 4y ago
    as a devop a shared account mixing private and business? would be a security breach if you ask me
    - metadaemon 4y ago
      I'm betting work gives him a free family subscription, ideally you'd have them as separate accounts, not sure how the local vault works under the hood in that situation.
  - scarface74 4y ago
    That’s absolutely no better.
- scarface74 4y ago
  This is completely unheard of for any company with any level of security. I’ve worked for 60 person startups that wouldn’t allow this.
- bboygravity 4y ago
  > You would assume anyone with the keys to the kingdom was working on a company provided device, or any device that fits a compliance framework based on their own risk needs (which should be massive!). You mean one of those company devices that is so locked down that they are close to impossible to work on? Like if you need to install a new (part of) a toolchain, you need to go through IT which takes between 3 weeks and 6 months? But your deadline is next week and your manager doesn't care about your IT "excuse", so just use notepad and CLI (or your own PC where you can do in 1 day what would otherwise take you months). Those devices... yeah...
  - marak830 4y ago
    As someone who works in gaming FQA with crazy locked down machines - we get software installed if necessary. Don't paint everyone with that brush. We also have two PC's per desk(plus consoles) so that our clients software isn't exposed. I don't get how a company like LastPass didn't have basic security like this when a FQA with a whole bunch of semi literate (in IT security) has a ssytem setup.
  - rwalle 4y ago
    You are making things up. Such a company probably exists somewhere, but I yet need to hear someone telling me a big-name tech company is doing this.
- jhoelzel 4y ago
  i completly agree. Its not so hard to do MFA all the way down either
cebert 4y ago
> “More specifically, the credentials for the servers were stolen from a DevOps engineer who had access to cloud storage at the company. This made it more difficult for LastPass to detect the suspicious activity.” This comes off as spin to me by LastPass or LogMeIn’s PR department. Even if this was the case, how is it possible for intrusion detection systems to not observe and report abnormally high egress traffic? Downloading every vault should be a rather noticeable event.
- hinkley 4y ago
  If they only took the vaults after a user accessed them, that would be less than double the traffic in the system. It depends how patient they were, or how broken the system was.
- mc32 4y ago
  Maybe it was one of those odd and security-violating cases that give CISOs nightmares where they test by copying PROD to the SANDBOX.
- peanutz454 4y ago
  I would expect that they'd setup something to notify them of - "abnormally high egress traffic downloading every vault", and yet due to alert fatigue, they never noticed. The specific thing with getting too many alerts is that you see them, but they aren't anyone's responsibility in general. The entire team gets them through email or sms or slack, and the new guys looks at them and wonders if we should do something about it like tweaking the criteria, and then gets other stuff to work on and learns to mostly ignore the alerts.
  - BLKNSLVR 4y ago
    That's really just a variation on the same problem though. We installed security product X, job done, walk away happy![0] [0]: https://yewtu.be/watch?v=62NyFTAKgOI https://yewtu.be/watch?v=62NyFTAKgOI
- replaceusb 4y ago
  It did, AWS alerted them on the traffic. It reads like they ignored it and when investigators later started going over that data it jumped up and slapped them.
JadoJodo 4y ago
> This was accomplished by targeting the DevOps engineer’s home computer and exploiting a vulnerable third-party media software package, which enabled remote code execution capability and allowed the threat actor to implant keylogger malware. The threat actor was able to capture the employee’s master password as it was entered, after the employee authenticated with MFA, and gain access to the DevOps engineer’s LastPass corporate vault.
- thehappypm 4y ago
  Bring your own computer for DevOps ?
- deleted 4y ago
  [deleted]
IncRnd 4y ago
I'm not sure the description is what actually happened. It doesn't have the ring of truth to it. That said, LastPass is not deserving of any trust as a password product of any kind. That a password was captured by a keylogger on a Dev Ops home computer shows that they don't understand how to secure remote computers, the meaning of defense in depth, the importance of proper login authentication, or how to secure data at rest. Each of these points are close to the core of their business. I don't wish them ill. I hope they recover from this, but they need to understand security to produce a security product.
- deathanatos 4y ago
  Yeah, because the description is inadequate. Is this BYOD? (… seems like not the employee's fault.) Is this the employee used the same password on the laptop and home, got credential stuffed, and LastPass isn't using MFA¹? (…seems like not the employee's fault.) Was there some jump from compromised home laptop to corp laptop? (The network is never to be trusted. …seems like not the employee's fault.) The buck is supposed to stop at security, not at each employee's personal hygiene … if your game plan depends on the latter, it's game over. There's more here than is being written, and I can only imagine because the truth probably stinks. ¹except TFA mentions MFA … but the mention of it doesn't really make sense.
  - deebosong 4y ago
    I understand what I'm asking for is "educated conjecture" more or less, but what would you surmise might be actual plausible situations rather than what LastPass is putting out as PR? Just asking as a laymen who is curious with no skin in the game.
  - schainks 4y ago
    > The buck is supposed to stop at security, not at each employee's personal hygiene … if your game plan depends on the latter, it's game over. I have to take security trainings twice a year that literally talk about the buck stopping at my digital hygiene and I better not fuck it up for The Company. Companies have to understand breaches will happen, but preparing employees on how to spot attacks or understand when they've been breached is a huge component of their ability to repel or minimize attacks.
ccooffee 4y ago
> Following the incident, LastPass has taken a number of steps to prevent future attacks and investigate what happened. The engineer was assisted in strengthening the security of their personal network [...] I hope this involved something along the lines of: "This zoom meeting won't end until you update your router firmware".
- akerl_ 4y ago
  I'm fascinated that this was part of their remediation. I'd consider "don't trust the employee's local network" to be a pretty basic principle of modern corporate information security. What happens when an employee logs in from hotel wifi? You basically have to treat the network between the user and your environment as hostile, and design for that problem.
  - userbinator 4y ago
    And as an employee, "don't trust the company's local network" with your own devices either.
    - akerl_ 4y ago
      For my personal devices, I trust my company's local network essentially the same as any other network my mobile devices connect to.
mcsniff 4y ago
So, with most password managers, when you authenticate on a new device, you are prompted for MFA. The user had a keylogger installed on their machine, so the attacker could collect the master password, but how did they login to the vault on a new machine without MFA? Did they get the MFA seed and login on a different machine, and nobody received a "You're using LastPass on a new machine, if this wasn't you..." message? Did they exfil all that data to the compromised machine first, then out? Unless I'm missing something, this doesn't seem right.
- lathiat 4y ago
  They had code execution on the persons computer, the encrypted vault is downloaded and stored/cached on the computer - you only need the master password at that point to decrypt it. Or to read the decrypted version out of the process (e.g. your web browsers memory) The 2FA part in the password managers (and least in the major players currently) is to get a copy of the encrypted vault from the server. The user did that part, and the encrypted vault was not easily accessible locally.
  - bobleeswagger 4y ago
    Seems like a broken 2FA implementation.
    - Thorrez 4y ago
      2FA can't protect you from malware on your machine.
      - bobleeswagger 4y ago
        Sure, but I wouldn't expect the vault to be accessible without 2FA each time it is cached. 2FA should be used as part of the decryption process even if the vault is cached.
        Thorrez 4y ago
acdha 4y ago
The Plex angle & note about securing their personal network had me curious about whether this person had a server exposed to the internet or if the attacker was only able to access it because they’d already compromised LastPass’s VPN. Nobody is looking good here but the former case would be especially regrettable.
- metadaemon 4y ago
  You probably at least need remote access within Plex in order to install new plugins, meaning you could probably run the exploit by just having his username and password.
  - eli 4y ago
    Plex itself was hacked around the same time...
electric_mayhem 4y ago
Wait, was he not using a password manager or something? I hear lastpass is pretty good, if he needs recommendations.
- eertami 4y ago
  People are likely to use weak passwords with Plex, just like they do with Netflix, so that they can easily share the details with other users and input it easily from memory on things like TVs and tablets. I think generally people consider it low risk, because "who cares if someone can see what I was watching", but if you can get RCE on a computer running Plex Media Server just by logging in to the Plex account, then of course that's a huge security surface and your Plex login should be a secure randomly generated password and have 2FA enabled. Of course, this can be avoided by paying for the premium Plex subscription which allows you to create accounts for other 'non-admin' users, but for people trying to avoid spending money (a large number of the Plex user base probably), then anyone with the login can change settings/administer the server.
bickfordb 4y ago
What could possibly go wrong with a centralized password database
- NoPicklez 4y ago
  Low value comment.
  - cpncrunch 4y ago
    https://password-managers.bestreviews.net/faq/which-password-managers-have-been-hacked/ https://password-managers.bestreviews.net/faq/which-password...
    - NoPicklez 4y ago
      Interesting that the article you link is from a website that recommends password managers? One key thing however is I'm not seeing Bitwarden in this list, big ups to them.
      - cpncrunch 4y ago
        Well, that's probably because they're making money off adverts for password managers. Just because a service hasn't been hacked yet doesn't mean it is secure. For myself, I just use google's password manager for non-critical passwords, and I use a single password for all my banking. I feel that is safer than using a password manager.
technick 4y ago
Only 4 engineers had this level of privilege at lastpass, how did the attacker identify the target? Linkedin... that's why you should not list where you work until you're no longer working there or list a completely different role than what you're currently in. I'm listed as a janitor of where I work. Only those that know me, know what I really do. I've tried to sell the policy forbidding employees from listing their positions or where they work on linkedin, each time management frowns and says no. One day they'll come around...
- PragmaticPulp 4y ago
  > I've tried to sell the policy forbidding employees from listing their positions or where they work on linkedin, each time management frowns and says no. One day they'll come around... Forbidding employees from listing their role on LinkedIn would put them at a major disadvantage in job searching and recruiting. Forcing employees to hide their role is unreasonable. The company doesn’t own the employee.
  - NoPicklez 4y ago
    I disagree, but disclaimer I work at a company that allows you to. If you work at an organisation like LastPass in a privileged position, then you need to be aware that you are an enormous target. And it's not just your own or the companies security you potentially compromise, but millions of others arguably most sensitive information. In Australia, if you have a security defence clearance, you are not allowed to display that in your social media networks (e.g. Linkedin), despite that potentially being important to your other job prospects in such industry. For those exact reasons. If your LinkedIn said you were a DevOps engineer at LastPass, you know for sure that they're a prime target. I'm not arguing the legality of it, just the problem it poses if you don't. Perhaps the solution is to tighten who can see your position and you diligently only connect with people you absolutely know and not have connections of connections on.
    - adrr 4y ago
      I would assume you're still allowed to post your to social media in Australia. Security clearance is not a role. I doubt people post they have access to all the infrastructure on their linkedin but you can infer it by the role. Linkedin doesn't even matter since you can buy the data any way. Email Signatures are mined to get role and contact info for databrokers. Anytime you email outside of the org, the CRM software could be grabbing that info and feeding back to a databroker. Not to mention people using addons to their mail clients. Why I don't use one at my current company even though it is company policy to have one from our HR department. Thats just email. There's also the credit report data that has your role on credit applications and also when you donate money to politicians/PACs that makes you list your role for compliance reasons.
replaceusb 4y ago
Seems like a lot of their talk of zero knowledge was bs. >In December, we notified a subset of customers whose SCIM, Enterprise API, and SAML keys were stored in unencrypted form. This only affected customers who joined LastPass and used these services in 2019 or before. This part just blew my mind. >Important: Since resetting MFA shared secrets destroys all LastPass sessions and trusted devices for these users, these users will need to log back in, go through location verification, and re-enable their respective MFA apps to continue using the service. I feel sorry for everyones internal helpdesk. This is going to be brutal.
hsbauauvhabzb 4y ago
Ironically, if they’d used a LastPass generated pass on plex, it would have been infeasible to crack the breached plex password. Using LastPass would literally have prevented a breach of LastPass.
- krakenschmidt 4y ago
  No, they breached Plex with a vulnerability. Then they installed a keylogger. Edit: Is there a source about how exactly they got into Plex? I assumed it was via port scanning plus some Plex vulnerability, not that they guess the Plex password.
zx8080 4y ago
Does this breach mean that LastPass effectively "transforms" 2FA into 1FA? Because the attacker did not need a separate __physical__ MFA (2nd F) after collecting the master password (1st F).
- replaceusb 4y ago
  Pretty much, limiting mfa options to otp only. Then the attacker getting access to customer shared secrets means they basically just have to guess the master password. >Backup of LastPass MFA/Federation Database – contained copies of LastPass Authenticator seeds, telephone numbers used for the MFA backup option (if enabled), as well as a split knowledge component (the K2 “key”) used for LastPass federation (if enabled). This database was encrypted, but the separately-stored decryption key was included in the secrets stolen by the threat actor during the second incident. Unless I am misunderstanding this, they mention to business users the need to reset shared secrets from OTP providers. >For users of Duo Security, Symantec VIP, RSA SecurID, or SecureAuth, regenerate the shared secret for each respective MFA solution and paste the new shared secret into the respective MFA app configuration in the Admin Console.
j45 4y ago
We're reminded that cloud services are ultimately someone else's computer. Putting one's secrets someone else's computer under the marketing of convenience is just that. This is not just to do with LastPass. It doesn't make sense why one would put their personal, most valuable passwords in the hand of another party. Of course, it's handy on a team between people. When it comes to our bank accounts, do we trust someone like that with the pin numbers to our credit and bank cards? One positive that comes out of this is that it raises awareness of thinking about the difference between security and convenience to manage one's own passwords. One solution? Locating a zero, or no knowledge file storage system which is encrypted at rest and transit and placing your own files on it is the first start. Spideroak used to fill that slot nicely (not sure if it's available anymore), and others I have heard about are sync.com and syncthing which do this just fine. Are there any other solutions that would be reasonably teachable to the average user?
- oceanplexian 4y ago
  I really don't understand the point of these cloud password managers. Use something like KeepassX. Encrypt it with AES256, upload it to Dropbox, Google Drive, whatever. I personally throw the encrypted p/w file in an encrypted MacOS disk image with a secondary, separate memorized passphrase as well. Literally solves the problem without having to trust or pay some random sketchy service.
  - Fire-Dragon-DoL 4y ago
    Sharing with family sucks with these. No permission control and no simultaneous write from multiple devices make those unusable
    - j45 4y ago
      Wouldn't most of the use cases be reading, instead of writing? I have seen people create multiple files on a share that works fine. I wasn't too keen on it at first but it did seem to work OK.
      - 4y ago
idatum 4y ago
For anyone that had been deep in LastPass with passwords, I recently switched myself and convinced my family also. We switched to another [read recommendations] online centralized password manager. It takes a couple hours of your time-off; totally can stream some whatever show in the meantime to pass the time. One benefit is that it made me audit my online accounts -- I removed many. So: Be prepared to switch if you want this type of service. New normal.
thomastjeffery 4y ago
This article is painfully light on the details. The blog post they linked is much more informative: https://support.lastpass.com/help/incident-2-additional-details-of-the-attack https://support.lastpass.com/help/incident-2-additional-deta... DevOps itself is a huge (and thin) attack surface. This is a feature of software as a centralized business venture. If your company is storing user data, then it must be someone's job to do that. They need administrative server access to do that job. The important takeaway is that the data itself can still be safe. So long as the company did not have a way to decrypt it, that data can rest anywhere. Guaranteeing that reality, however, is very difficult for a business - that is expected to be private about their implementations - to prove to its customers. When this beach happened, LastPass should have focused on telling their users to never reuse the master password that they had set at that time. That's the biggest vulnerability: the content of their vaults (as copied by the attacker) was, and is, still kept behind that password. The need for each user to keep that specific password secret is the main effect of this situation. This is a great reminder that you can't trust anyone to keep your data private. You can only trust math.
donutshop 4y ago
LastPass - The gift that keeps on giving
osigurdson 4y ago
Welp, I guess it is totally not LastPass's fault then. Darn DevOps engineer should have known better.
kc10 4y ago
I went through the pain of resetting hundreds of my passwords since the last breach during which they lost encrypted data. It was brutal, took several weeks and I had to spend late nights and weekends resetting passwords as a hobby project. I am glad I went through all the pain.
- 8organicbits 4y ago
  For anyone who still needs to do this, consider deleting accounts you no longer need. For some sites it's about the same effort to change a password as it is to delete an account (others... a lot more). The next time you need to bulk change passwords, you'll have fewer to change.
mattbillenstein 4y ago
Why anyone ever trusted any of these products is beside me - it's a huge SPOF and you're literally betting your business they don't get compromised...
jason_wo 4y ago
I'am honest, similar things could happen on my laptop for my personal stuff. I have some AWS keys in some files that are used by terraform/packer. A hacker could easily get them. Some other AWS keys are stored in the CI system and provided as env variables. Someone that can merge/push to the specified branches can just change the CI script an exfiltrate them. How can I fix that? I would need some MFA for both cases. I would imagine it would be a good idea that I have to confirm each action on MFA device, which will then generate temporary tokens that are invalid after a few minutes. I locked into some solutions like Hashicorp Vault but I was not able to build something in a short time. New features were always more important. How do you do it?
- j13n 4y ago
  YubiKeys and aws-vault for managing my credentials. Hashicorp Vault and SSM for services.
  - jason_wo 4y ago
    Nice! Do I understand this correctly? You use aws-vault(https://github.com/99designs/aws-vault https://github.com/99designs/aws-vault) and configure it with IAM and MFA with YubiKeys. You configure e.g. the profile jonsmith. When you run aws-vault exec jonsmith -- aws s3 ls it will ask you, e.g. every hour to confirm with YubiKeys and cache the key for one hour. After that the temporary keys expire. Can you also store keys different from AWS?
pabs3 4y ago
Definitely way past time to switch from passwords to mTLS or WebAuthn
deleted 4y ago
[deleted]
theshrike79 4y ago
How many major incidents is this for LastPass already?
mekoka 4y ago
Nevermind the Plex vulnerability. Nevermind that the thieves are called here "hackers" and the theft, an "attack". It all makes it sound like more of a feat than it actually was. The bottom line for me is this: you outsource your security to a company whose sole reason for existing is to be more paranoid about it than you, and they tell you that what should be among their most treasured and garded secrets can casually be found in a coffee shop nearby.
unit_circle 4y ago
First of all: If you have the title of "DevOps" what you're doing is "Operations", you aren't practicing DevOps. Anyway, this company has had incident after incident. This will keep happening every few years for them like it has for the past 10. As will lack of transparency/ outright lying. Some commenters are saying they wish the company the best. I don't. Use something else. LastPass needs to die.
- dividedbyzero 4y ago
  > First of all: If you have the title of "DevOps" what you're doing is "Operations", you aren't practicing DevOps. So what title do you need to have to practice DevOps?
  - 8organicbits 4y ago
    "DevOps enables coordination and collaboration between formerly siloed roles like development, IT operations, quality engineering, and security." - https://learn.microsoft.com/en-us/devops/what-is-devops https://learn.microsoft.com/en-us/devops/what-is-devops It's not a job title, it's an engineering practice. People who participate in DevOps include "software engineer", "network engineer", "IT operations engineer", "platform engineer", "cloud engineer", "quality engineer", "security engineer", "manager", etc. I'm also one of the people who does "operations" but has a DevOps job title. I've grown to accept this as a second definition, it's quite common now.
    - misto 4y ago
      I do not like the title either, but I do understand the motivation of taking all those "non-software engineering", technical roles and putting them under the same umbrella, due to a lack for a better title, because a company might not afford to have separate roles for each of those areas you have listed above. "OPERATIONS ENGINEER" might work, but it raises another set of problems, e.g. does it imply operational responsibility (on-duty) work, which I don't think is a given in DevOps jobs nowadays.
      - 4y ago
revskill 4y ago
It's secure because the Engineering Manager said so.
jhoelzel 4y ago
Im not sure that they are just using this as a scapegoat but if your working from home, as a DevOps/Platform engineer, your very first ticket should be to activate MFA. Kubernetes does MFA, all the Clouds do MFA and the company you work for can afford a "cheap android phone as key". No matter if bare-metal, cloud or managed. If you habe ANY edit rights you need MFA.
- thisgoesnowhere 4y ago
  MFA wouldnt have helped here. The hacker had the encrypted vault all they needed was the password.
  - jhoelzel 4y ago
    which in turn should not be the end of the world, because its MFA all the way down right?
    - Fabricio20 4y ago
      If you have an encrypted vault file and the master password (or decryption key) you don't need 2FA, there is no known encryption algo that uses a rotating key like TOTP, the implementation of 2FA is always software-sided, and in the case of a vault file (like here), you don't need the software.
    - thisgoesnowhere 4y ago
      MFA is not used when you decrypt your vault on any password platform. It's just to receive the encrypted vault
      - jhoelzel 4y ago
        well i was more thinking along the lines of every service in your vault implementing additional 2fa. In the kubernetes world it really is not so difficult
deleted 4y ago
[deleted]
bearmode 4y ago
I can't believe it's taken 6 months for them to divulge this.
jd24 4y ago
imagine trusting a company with multiple incidents with all your passwords
Dorothy014 4y ago
[dead]
londons_explore 4y ago
If lastpass was well designed, the company would store no private user data. Ie. they would be a 'dumb' storage system for the customers encrypted data. The data would be encrypted by the customer before upload. And then decrypted again by the customer after download. Users would be identified by a unique random ID, and users would auth by signing a challenge with a secret key known only to the customer. That way, even if a bad guy worked for lastpass and had full admin access to all servers, they couldn't steal anything. And, in fact, if this system was properly designed, it could all run with opensource server code, and the datastore fully open for anyone to inspect, to prove that the security is down to cryptography rather than trusted yet fallible humans.
- thisgoesnowhere 4y ago
  What part of this is not happening right now with lastpass?
j-bos 4y ago
Am I understanding correctly: A single engineer had both access to the prod database AND the data decryption values?
- secabeen 4y ago
  Depends on what you mean by "the data decryption values". If you mean the encryption keys for the vaults themselves, no. Those are derived from the individual master passwords (in non-SSO setups, with SSO it's more complicated, and I don't fully understand the impact). So the attackers have a bunch of encrypted vaults from a backup. They can now brute-force the vaults, but if the original Master Passwords were secure (16+ characters, all 4 classes), those vaults should remain secure. Of course, many people use bad passwords, and those people are at risk.
  - dvngnt_ 4y ago
    the exposed passwords isn't even the worst part. it's the fact that user vaults sites are unencrypted so now attackers know every website that you thought was important enough to save. they know what bank you use, which adult sites you visit, what medical conditions you might have
    - secabeen 4y ago
      I understand that some people feel that that is sensitive data. It's not a big threat for me. My ISP and Google already has that data in many cases (IP destination data, DNS lookups, etc.) YMMV.
  - j-bos 4y ago
    ah, thanks for the clarification.
lrvick 4y ago
In my experience the majority of hacks are from a compromised laptop of a production engineer. Everyone blindly NPM installs away all their problems and no one checks signatures anymore. Most are using package managers like Brew that don't sign anything to begin with. At Distrust, my security consulting firm, we train all our clients to build production systems that require a minimum of two engineers to mutate and that only pristine operating systems access production that have only signed reproducible used packages that have never been used for anything else. Production environments need to be managed like careful methodical clean-room labs with strict accountability. Instead they are managed like collaborative art projects where everyone is trusted and nothing bad can happen.
- q1w2 4y ago
  In this case, the anonymous source says that Plex server was compromised, so I assume it was a developer's home PC, not a work laptop. The breach was preceded a couple of days by Plex corporate breach which devulged the engineer's credentials and home IP address. This would have allowed the attackers to access Plex sever remotely, after which the source claims they used an RCE to install a keylogger (and probably a back door) on the engineer's PC. The concerning part is that according to Plex devs (in their reddit sub), they have NO KNOWLEDGE of any RCE. They also haven't communicated with Lastpass, and no one reached out to them. So if there's a Plex remote code exploit - it is still unpatched and actively being exploited - 8 months later! Given that there is still no information on this Plex RCE, we should not assume that it requires authentication to function. So if you're using Plex, make sure to turn off public accessibility asap!
  - TheCondor 4y ago
    If you don't need remote access to your plex, you should disable it. The cache features are pretty good, so you could download media while at home and have it on a device in the car or at work pretty easily. I think it's pretty dubious that they could pin it on a plex media server breach though. This is from the same company that sells logmein and goto; I haven't heard definitively that those pieces of software weren't installed on there.. Unless this machine was used for Plex and the occasional log in to work from home type thing, I doubt you could do the forensics on it with much reliability unless you got to it within days of the attack. If this is a normal devops guy? He's got 3 different chat apps, probably Steam, who knows what pirated crap is on there... Plex seems like a very convenient target; the plex corporate attack seems very very plausible as giving the attacker information though.
skarz 4y ago
I think something that should have been in the title is that the breach was facilitated by a vulnerability in an application that many users here might have, Plex. They don’t speak on the nature of the vulnerability in that application. Has it been addressed and fixed through security patches already, or is Plex still potentially dangerous right now?
- q1w2 4y ago
  Plex devs commented in Reddit that this is the first they've heard of it and haven't identified any RCE, let alone patched anything. So if you're running a Plex server, you should disable public access immediately.
skeeter2020 4y ago
There's a weird combination of "what happened" and causation in here: >> the credentials for the servers were stolen from a DevOps engineer who had access to cloud storage at the company. This made it more difficult for LastPass to detect the suspicious activity. How does A => B? >> The threat actor was able to capture the employee’s master password as it was entered, after the employee authenticated with MFA, and gain access to the DevOps engineer’s LastPass corporate vault. The part about authentication and MFA doesn't track with the rest of the sentence. How does a password without also having the MFA channel work? How would this give me access to a LP vault? Why were they on their home machine? I understand it must be hard to try and come clean in a RCA without injecting some excuses and mitigating factors but you can't attempt soften the blow or the entire thing is a big, smelly mess. It should be a bunch of facts with no emotion, THEN the mitigation and lessons learned. LP just issued a bunch of press releases.
- deleted 4y ago
  [deleted]
- secabeen 4y ago
  > The part about authentication and MFA doesn't track with the rest of the sentence. How does a password without also having the MFA channel work? How would this give me access to a LP vault? Why were they on their home machine? I think the idea here is that we want the cryptography operations to happen entirely locally, so that LastPass doesn't have any access to them. However, if you do that, someone with root on that system and the Master Password can replicate the operations the local system does on the vault. I'm not aware of any symmetric-encryption algorithm that includes a time-based un-replayable TOTP or HOTP in the key-generation process.
papito 4y ago
I really wonder how many companies have figured out how to properly secure equipment and access used by a distributed work force.
remram 4y ago
> DevOps engineer So LastPass, a security company, has people who are tasked with both delivering features as fast as possible using a variety of development tools (dev) and administer their production systems where critical data lies (ops). On the same machine, from home. It seems that this breach is not really an accident and more a logical conclusion.
bgro 4y ago
Sometimes I wonder if I shouldn’t google the weather on my work computer because it might have some crazy exploit or cause an alert. DevOps guy here is just running a Plex server and probably pirating porn on his work laptop because why not at that point. Can’t even bother updating this which Plex makes literally about as easy as they possibly could. It’s a one click fully automated process with zero interruption. Was he using internet explorer too? Or was it his work laptop or did they just let him log in to work on any random computer? Did he actually get any work done? I want to hear an interview from this guy. How did he get hired as DevOps while avoiding all the basics of fundamental basic work practices?

376 points by mikece 4y ago

Am I assuming in this case the engineer was using his home PC to work? This isn't unheard of in the industry, Engineers using BYOD devices or similar to work from home. But with a company with a risk profile as high as LastPass this seems _incredibly dumb_. You would assume anyone with the keys to the kingdom was working on a company provided device, or any device that fits a compliance framework based on their own risk needs (which should be massive!). E.g., endpoint management, AV/detection, FIDO2 with Yubikeys to access AWS as an admin and otherwise. Just a few off the top of my head.

SOLAR_FIELDS 4y ago
I could see this kind of sophisticated attack easily working on some random company with fairly lax BYOD policies. Makes sense, it was a rather sophisticated attack when you look at your typical medium sized company. But if your entire company and organization is built around keeping things secure, THIS is what brought you down? This isn’t getting owned by some unforeseen 0day, it’s just sloppy opsec.
Johnny555 4y ago
My company isn't nearly as high profile or security focused and we're not allowed to use our own computers for any work related purposes,and our work laptops run threat detection software and we have a whitelist of software we're allowed to install. I'm surprised that LastPass's policies aren't at least that strict. My company has what I think is a big hole in this policy in that we're allowed to use our own phone for email, a few corporate apps (like Jira) and our corporate password manager (not LastPass), but IT doesn't do any management of phones (other than being able to wipe them remotely if you're connected to the company email server). I suspect that the company doesn't want to spend the money on giving everyone a managed phone.
- hackernewds 4y ago
  As much as LastPass seems to be trying to pin this on a single engineer, and not a broad vuln, the fact that they have such lax policies around access management, especially for a password management system, tells me enough I need to know never to use them again. Waiting for the rebrand and the incoming lawsuits.
  - victor9000 4y ago
    Yeah, this is the equivalent of blaming an intern for nuking the prod database. Maybe they were careless of maybe that shouldn't be possible to begin with.
  - BLKNSLVR 4y ago
    Yep. The symptom being that a problem of this scale can be caused by a single engineer, which points to the root cause being deeper and potentially systemic. The question for future trust is: what's been / being done to prevent the same thing from happening again due to another single engineer?
    - macrolime 4y ago
      They answer that question directly here https://support.lastpass.com/help/what-have-we-done-to-ensure-lastpass-is-safe-to-use https://support.lastpass.com/help/what-have-we-done-to-ensur... TLDR;not much
    - execveat 4y ago
      Preventing this thing from happening costs a lot of $$$, so pretty much everyone just "accepts the risk" seeing that probability of something like this happening to your company (during your tenure) is still super low. All companies with somewhat robust security posture I know have had a string of incidents in the past, that seems to be the only thing that can motivate to put $ in security.
      - Johnny555 4y ago
metadaemon 4y ago
I don’t think he’s necessarily working on his pc. He probably just had a shared LastPass account between work and his pc.
- wolfi1 4y ago
  as a devop a shared account mixing private and business? would be a security breach if you ask me
  - metadaemon 4y ago
    I'm betting work gives him a free family subscription, ideally you'd have them as separate accounts, not sure how the local vault works under the hood in that situation.
- scarface74 4y ago
  That’s absolutely no better.
scarface74 4y ago
This is completely unheard of for any company with any level of security. I’ve worked for 60 person startups that wouldn’t allow this.
bboygravity 4y ago
> You would assume anyone with the keys to the kingdom was working on a company provided device, or any device that fits a compliance framework based on their own risk needs (which should be massive!). You mean one of those company devices that is so locked down that they are close to impossible to work on? Like if you need to install a new (part of) a toolchain, you need to go through IT which takes between 3 weeks and 6 months? But your deadline is next week and your manager doesn't care about your IT "excuse", so just use notepad and CLI (or your own PC where you can do in 1 day what would otherwise take you months). Those devices... yeah...
- marak830 4y ago
  As someone who works in gaming FQA with crazy locked down machines - we get software installed if necessary. Don't paint everyone with that brush. We also have two PC's per desk(plus consoles) so that our clients software isn't exposed. I don't get how a company like LastPass didn't have basic security like this when a FQA with a whole bunch of semi literate (in IT security) has a ssytem setup.
- rwalle 4y ago
  You are making things up. Such a company probably exists somewhere, but I yet need to hear someone telling me a big-name tech company is doing this.
jhoelzel 4y ago
i completly agree. Its not so hard to do MFA all the way down either

cebert 4y ago

> “More specifically, the credentials for the servers were stolen from a DevOps engineer who had access to cloud storage at the company. This made it more difficult for LastPass to detect the suspicious activity.” This comes off as spin to me by LastPass or LogMeIn’s PR department. Even if this was the case, how is it possible for intrusion detection systems to not observe and report abnormally high egress traffic? Downloading every vault should be a rather noticeable event.

hinkley 4y ago
If they only took the vaults after a user accessed them, that would be less than double the traffic in the system. It depends how patient they were, or how broken the system was.
mc32 4y ago
Maybe it was one of those odd and security-violating cases that give CISOs nightmares where they test by copying PROD to the SANDBOX.
peanutz454 4y ago
I would expect that they'd setup something to notify them of - "abnormally high egress traffic downloading every vault", and yet due to alert fatigue, they never noticed. The specific thing with getting too many alerts is that you see them, but they aren't anyone's responsibility in general. The entire team gets them through email or sms or slack, and the new guys looks at them and wonders if we should do something about it like tweaking the criteria, and then gets other stuff to work on and learns to mostly ignore the alerts.
- BLKNSLVR 4y ago
  That's really just a variation on the same problem though. We installed security product X, job done, walk away happy![0] [0]: https://yewtu.be/watch?v=62NyFTAKgOI https://yewtu.be/watch?v=62NyFTAKgOI
replaceusb 4y ago
It did, AWS alerted them on the traffic. It reads like they ignored it and when investigators later started going over that data it jumped up and slapped them.

JadoJodo 4y ago

> This was accomplished by targeting the DevOps engineer’s home computer and exploiting a vulnerable third-party media software package, which enabled remote code execution capability and allowed the threat actor to implant keylogger malware. The threat actor was able to capture the employee’s master password as it was entered, after the employee authenticated with MFA, and gain access to the DevOps engineer’s LastPass corporate vault.

thehappypm 4y ago
Bring your own computer for DevOps ?
deleted 4y ago
[deleted]

IncRnd 4y ago

I'm not sure the description is what actually happened. It doesn't have the ring of truth to it. That said, LastPass is not deserving of any trust as a password product of any kind. That a password was captured by a keylogger on a Dev Ops home computer shows that they don't understand how to secure remote computers, the meaning of defense in depth, the importance of proper login authentication, or how to secure data at rest. Each of these points are close to the core of their business. I don't wish them ill. I hope they recover from this, but they need to understand security to produce a security product.

deathanatos 4y ago
Yeah, because the description is inadequate. Is this BYOD? (… seems like not the employee's fault.) Is this the employee used the same password on the laptop and home, got credential stuffed, and LastPass isn't using MFA¹? (…seems like not the employee's fault.) Was there some jump from compromised home laptop to corp laptop? (The network is never to be trusted. …seems like not the employee's fault.) The buck is supposed to stop at security, not at each employee's personal hygiene … if your game plan depends on the latter, it's game over. There's more here than is being written, and I can only imagine because the truth probably stinks. ¹except TFA mentions MFA … but the mention of it doesn't really make sense.
- deebosong 4y ago
  I understand what I'm asking for is "educated conjecture" more or less, but what would you surmise might be actual plausible situations rather than what LastPass is putting out as PR? Just asking as a laymen who is curious with no skin in the game.
- schainks 4y ago
  > The buck is supposed to stop at security, not at each employee's personal hygiene … if your game plan depends on the latter, it's game over. I have to take security trainings twice a year that literally talk about the buck stopping at my digital hygiene and I better not fuck it up for The Company. Companies have to understand breaches will happen, but preparing employees on how to spot attacks or understand when they've been breached is a huge component of their ability to repel or minimize attacks.

ccooffee 4y ago

> Following the incident, LastPass has taken a number of steps to prevent future attacks and investigate what happened. The engineer was assisted in strengthening the security of their personal network [...] I hope this involved something along the lines of: "This zoom meeting won't end until you update your router firmware".

akerl_ 4y ago
I'm fascinated that this was part of their remediation. I'd consider "don't trust the employee's local network" to be a pretty basic principle of modern corporate information security. What happens when an employee logs in from hotel wifi? You basically have to treat the network between the user and your environment as hostile, and design for that problem.
- userbinator 4y ago
  And as an employee, "don't trust the company's local network" with your own devices either.
  - akerl_ 4y ago
    For my personal devices, I trust my company's local network essentially the same as any other network my mobile devices connect to.

mcsniff 4y ago

So, with most password managers, when you authenticate on a new device, you are prompted for MFA. The user had a keylogger installed on their machine, so the attacker could collect the master password, but how did they login to the vault on a new machine without MFA? Did they get the MFA seed and login on a different machine, and nobody received a "You're using LastPass on a new machine, if this wasn't you..." message? Did they exfil all that data to the compromised machine first, then out? Unless I'm missing something, this doesn't seem right.

lathiat 4y ago
They had code execution on the persons computer, the encrypted vault is downloaded and stored/cached on the computer - you only need the master password at that point to decrypt it. Or to read the decrypted version out of the process (e.g. your web browsers memory) The 2FA part in the password managers (and least in the major players currently) is to get a copy of the encrypted vault from the server. The user did that part, and the encrypted vault was not easily accessible locally.
- bobleeswagger 4y ago
  Seems like a broken 2FA implementation.
  - Thorrez 4y ago
    2FA can't protect you from malware on your machine.
    - bobleeswagger 4y ago
      Sure, but I wouldn't expect the vault to be accessible without 2FA each time it is cached. 2FA should be used as part of the decryption process even if the vault is cached.
      - Thorrez 4y ago

acdha 4y ago

The Plex angle & note about securing their personal network had me curious about whether this person had a server exposed to the internet or if the attacker was only able to access it because they’d already compromised LastPass’s VPN. Nobody is looking good here but the former case would be especially regrettable.

metadaemon 4y ago
You probably at least need remote access within Plex in order to install new plugins, meaning you could probably run the exploit by just having his username and password.
- eli 4y ago
  Plex itself was hacked around the same time...

electric_mayhem 4y ago

Wait, was he not using a password manager or something? I hear lastpass is pretty good, if he needs recommendations.

eertami 4y ago
People are likely to use weak passwords with Plex, just like they do with Netflix, so that they can easily share the details with other users and input it easily from memory on things like TVs and tablets. I think generally people consider it low risk, because "who cares if someone can see what I was watching", but if you can get RCE on a computer running Plex Media Server just by logging in to the Plex account, then of course that's a huge security surface and your Plex login should be a secure randomly generated password and have 2FA enabled. Of course, this can be avoided by paying for the premium Plex subscription which allows you to create accounts for other 'non-admin' users, but for people trying to avoid spending money (a large number of the Plex user base probably), then anyone with the login can change settings/administer the server.

bickfordb 4y ago

What could possibly go wrong with a centralized password database

NoPicklez 4y ago
Low value comment.
- cpncrunch 4y ago
  https://password-managers.bestreviews.net/faq/which-password-managers-have-been-hacked/ https://password-managers.bestreviews.net/faq/which-password...
  - NoPicklez 4y ago
    Interesting that the article you link is from a website that recommends password managers? One key thing however is I'm not seeing Bitwarden in this list, big ups to them.
    - cpncrunch 4y ago
      Well, that's probably because they're making money off adverts for password managers. Just because a service hasn't been hacked yet doesn't mean it is secure. For myself, I just use google's password manager for non-critical passwords, and I use a single password for all my banking. I feel that is safer than using a password manager.

technick 4y ago

Only 4 engineers had this level of privilege at lastpass, how did the attacker identify the target? Linkedin... that's why you should not list where you work until you're no longer working there or list a completely different role than what you're currently in. I'm listed as a janitor of where I work. Only those that know me, know what I really do. I've tried to sell the policy forbidding employees from listing their positions or where they work on linkedin, each time management frowns and says no. One day they'll come around...

PragmaticPulp 4y ago
> I've tried to sell the policy forbidding employees from listing their positions or where they work on linkedin, each time management frowns and says no. One day they'll come around... Forbidding employees from listing their role on LinkedIn would put them at a major disadvantage in job searching and recruiting. Forcing employees to hide their role is unreasonable. The company doesn’t own the employee.
- NoPicklez 4y ago
  I disagree, but disclaimer I work at a company that allows you to. If you work at an organisation like LastPass in a privileged position, then you need to be aware that you are an enormous target. And it's not just your own or the companies security you potentially compromise, but millions of others arguably most sensitive information. In Australia, if you have a security defence clearance, you are not allowed to display that in your social media networks (e.g. Linkedin), despite that potentially being important to your other job prospects in such industry. For those exact reasons. If your LinkedIn said you were a DevOps engineer at LastPass, you know for sure that they're a prime target. I'm not arguing the legality of it, just the problem it poses if you don't. Perhaps the solution is to tighten who can see your position and you diligently only connect with people you absolutely know and not have connections of connections on.
  - adrr 4y ago
    I would assume you're still allowed to post your to social media in Australia. Security clearance is not a role. I doubt people post they have access to all the infrastructure on their linkedin but you can infer it by the role. Linkedin doesn't even matter since you can buy the data any way. Email Signatures are mined to get role and contact info for databrokers. Anytime you email outside of the org, the CRM software could be grabbing that info and feeding back to a databroker. Not to mention people using addons to their mail clients. Why I don't use one at my current company even though it is company policy to have one from our HR department. Thats just email. There's also the credit report data that has your role on credit applications and also when you donate money to politicians/PACs that makes you list your role for compliance reasons.

replaceusb 4y ago

Seems like a lot of their talk of zero knowledge was bs. >In December, we notified a subset of customers whose SCIM, Enterprise API, and SAML keys were stored in unencrypted form. This only affected customers who joined LastPass and used these services in 2019 or before. This part just blew my mind. >Important: Since resetting MFA shared secrets destroys all LastPass sessions and trusted devices for these users, these users will need to log back in, go through location verification, and re-enable their respective MFA apps to continue using the service. I feel sorry for everyones internal helpdesk. This is going to be brutal.

hsbauauvhabzb 4y ago

Ironically, if they’d used a LastPass generated pass on plex, it would have been infeasible to crack the breached plex password. Using LastPass would literally have prevented a breach of LastPass.

krakenschmidt 4y ago
No, they breached Plex with a vulnerability. Then they installed a keylogger. Edit: Is there a source about how exactly they got into Plex? I assumed it was via port scanning plus some Plex vulnerability, not that they guess the Plex password.

zx8080 4y ago

Does this breach mean that LastPass effectively "transforms" 2FA into 1FA? Because the attacker did not need a separate __physical__ MFA (2nd F) after collecting the master password (1st F).

replaceusb 4y ago
Pretty much, limiting mfa options to otp only. Then the attacker getting access to customer shared secrets means they basically just have to guess the master password. >Backup of LastPass MFA/Federation Database – contained copies of LastPass Authenticator seeds, telephone numbers used for the MFA backup option (if enabled), as well as a split knowledge component (the K2 “key”) used for LastPass federation (if enabled). This database was encrypted, but the separately-stored decryption key was included in the secrets stolen by the threat actor during the second incident. Unless I am misunderstanding this, they mention to business users the need to reset shared secrets from OTP providers. >For users of Duo Security, Symantec VIP, RSA SecurID, or SecureAuth, regenerate the shared secret for each respective MFA solution and paste the new shared secret into the respective MFA app configuration in the Admin Console.

j45 4y ago

We're reminded that cloud services are ultimately someone else's computer. Putting one's secrets someone else's computer under the marketing of convenience is just that. This is not just to do with LastPass. It doesn't make sense why one would put their personal, most valuable passwords in the hand of another party. Of course, it's handy on a team between people. When it comes to our bank accounts, do we trust someone like that with the pin numbers to our credit and bank cards? One positive that comes out of this is that it raises awareness of thinking about the difference between security and convenience to manage one's own passwords. One solution? Locating a zero, or no knowledge file storage system which is encrypted at rest and transit and placing your own files on it is the first start. Spideroak used to fill that slot nicely (not sure if it's available anymore), and others I have heard about are sync.com and syncthing which do this just fine. Are there any other solutions that would be reasonably teachable to the average user?

oceanplexian 4y ago
I really don't understand the point of these cloud password managers. Use something like KeepassX. Encrypt it with AES256, upload it to Dropbox, Google Drive, whatever. I personally throw the encrypted p/w file in an encrypted MacOS disk image with a secondary, separate memorized passphrase as well. Literally solves the problem without having to trust or pay some random sketchy service.
- Fire-Dragon-DoL 4y ago
  Sharing with family sucks with these. No permission control and no simultaneous write from multiple devices make those unusable
  - j45 4y ago
    Wouldn't most of the use cases be reading, instead of writing? I have seen people create multiple files on a share that works fine. I wasn't too keen on it at first but it did seem to work OK.
    - 4y ago

idatum 4y ago

For anyone that had been deep in LastPass with passwords, I recently switched myself and convinced my family also. We switched to another [read recommendations] online centralized password manager. It takes a couple hours of your time-off; totally can stream some whatever show in the meantime to pass the time. One benefit is that it made me audit my online accounts -- I removed many. So: Be prepared to switch if you want this type of service. New normal.

thomastjeffery 4y ago

This article is painfully light on the details. The blog post they linked is much more informative: https://support.lastpass.com/help/incident-2-additional-details-of-the-attack https://support.lastpass.com/help/incident-2-additional-deta... DevOps itself is a huge (and thin) attack surface. This is a feature of software as a centralized business venture. If your company is storing user data, then it must be someone's job to do that. They need administrative server access to do that job. The important takeaway is that the data itself can still be safe. So long as the company did not have a way to decrypt it, that data can rest anywhere. Guaranteeing that reality, however, is very difficult for a business - that is expected to be private about their implementations - to prove to its customers. When this beach happened, LastPass should have focused on telling their users to never reuse the master password that they had set at that time. That's the biggest vulnerability: the content of their vaults (as copied by the attacker) was, and is, still kept behind that password. The need for each user to keep that specific password secret is the main effect of this situation. This is a great reminder that you can't trust anyone to keep your data private. You can only trust math.

donutshop 4y ago

LastPass - The gift that keeps on giving

osigurdson 4y ago

Welp, I guess it is totally not LastPass's fault then. Darn DevOps engineer should have known better.

kc10 4y ago

I went through the pain of resetting hundreds of my passwords since the last breach during which they lost encrypted data. It was brutal, took several weeks and I had to spend late nights and weekends resetting passwords as a hobby project. I am glad I went through all the pain.

8organicbits 4y ago
For anyone who still needs to do this, consider deleting accounts you no longer need. For some sites it's about the same effort to change a password as it is to delete an account (others... a lot more). The next time you need to bulk change passwords, you'll have fewer to change.

mattbillenstein 4y ago

Why anyone ever trusted any of these products is beside me - it's a huge SPOF and you're literally betting your business they don't get compromised...

jason_wo 4y ago

I'am honest, similar things could happen on my laptop for my personal stuff. I have some AWS keys in some files that are used by terraform/packer. A hacker could easily get them. Some other AWS keys are stored in the CI system and provided as env variables. Someone that can merge/push to the specified branches can just change the CI script an exfiltrate them. How can I fix that? I would need some MFA for both cases. I would imagine it would be a good idea that I have to confirm each action on MFA device, which will then generate temporary tokens that are invalid after a few minutes. I locked into some solutions like Hashicorp Vault but I was not able to build something in a short time. New features were always more important. How do you do it?

j13n 4y ago
YubiKeys and aws-vault for managing my credentials. Hashicorp Vault and SSM for services.
- jason_wo 4y ago
  Nice! Do I understand this correctly? You use aws-vault(https://github.com/99designs/aws-vault https://github.com/99designs/aws-vault) and configure it with IAM and MFA with YubiKeys. You configure e.g. the profile jonsmith. When you run aws-vault exec jonsmith -- aws s3 ls it will ask you, e.g. every hour to confirm with YubiKeys and cache the key for one hour. After that the temporary keys expire. Can you also store keys different from AWS?

pabs3 4y ago

Definitely way past time to switch from passwords to mTLS or WebAuthn

deleted 4y ago

[deleted]

theshrike79 4y ago

How many major incidents is this for LastPass already?

mekoka 4y ago

Nevermind the Plex vulnerability. Nevermind that the thieves are called here "hackers" and the theft, an "attack". It all makes it sound like more of a feat than it actually was. The bottom line for me is this: you outsource your security to a company whose sole reason for existing is to be more paranoid about it than you, and they tell you that what should be among their most treasured and garded secrets can casually be found in a coffee shop nearby.

unit_circle 4y ago

First of all: If you have the title of "DevOps" what you're doing is "Operations", you aren't practicing DevOps. Anyway, this company has had incident after incident. This will keep happening every few years for them like it has for the past 10. As will lack of transparency/ outright lying. Some commenters are saying they wish the company the best. I don't. Use something else. LastPass needs to die.

dividedbyzero 4y ago
> First of all: If you have the title of "DevOps" what you're doing is "Operations", you aren't practicing DevOps. So what title do you need to have to practice DevOps?
- 8organicbits 4y ago
  "DevOps enables coordination and collaboration between formerly siloed roles like development, IT operations, quality engineering, and security." - https://learn.microsoft.com/en-us/devops/what-is-devops https://learn.microsoft.com/en-us/devops/what-is-devops It's not a job title, it's an engineering practice. People who participate in DevOps include "software engineer", "network engineer", "IT operations engineer", "platform engineer", "cloud engineer", "quality engineer", "security engineer", "manager", etc. I'm also one of the people who does "operations" but has a DevOps job title. I've grown to accept this as a second definition, it's quite common now.
  - misto 4y ago
    I do not like the title either, but I do understand the motivation of taking all those "non-software engineering", technical roles and putting them under the same umbrella, due to a lack for a better title, because a company might not afford to have separate roles for each of those areas you have listed above. "OPERATIONS ENGINEER" might work, but it raises another set of problems, e.g. does it imply operational responsibility (on-duty) work, which I don't think is a given in DevOps jobs nowadays.
    - 4y ago

revskill 4y ago

It's secure because the Engineering Manager said so.

jhoelzel 4y ago

Im not sure that they are just using this as a scapegoat but if your working from home, as a DevOps/Platform engineer, your very first ticket should be to activate MFA. Kubernetes does MFA, all the Clouds do MFA and the company you work for can afford a "cheap android phone as key". No matter if bare-metal, cloud or managed. If you habe ANY edit rights you need MFA.

thisgoesnowhere 4y ago
MFA wouldnt have helped here. The hacker had the encrypted vault all they needed was the password.
- jhoelzel 4y ago
  which in turn should not be the end of the world, because its MFA all the way down right?
  - Fabricio20 4y ago
    If you have an encrypted vault file and the master password (or decryption key) you don't need 2FA, there is no known encryption algo that uses a rotating key like TOTP, the implementation of 2FA is always software-sided, and in the case of a vault file (like here), you don't need the software.
  - thisgoesnowhere 4y ago
    MFA is not used when you decrypt your vault on any password platform. It's just to receive the encrypted vault
    - jhoelzel 4y ago
      well i was more thinking along the lines of every service in your vault implementing additional 2fa. In the kubernetes world it really is not so difficult

deleted 4y ago

[deleted]

bearmode 4y ago

I can't believe it's taken 6 months for them to divulge this.

jd24 4y ago

imagine trusting a company with multiple incidents with all your passwords

Dorothy014 4y ago

[dead]

londons_explore 4y ago

If lastpass was well designed, the company would store no private user data. Ie. they would be a 'dumb' storage system for the customers encrypted data. The data would be encrypted by the customer before upload. And then decrypted again by the customer after download. Users would be identified by a unique random ID, and users would auth by signing a challenge with a secret key known only to the customer. That way, even if a bad guy worked for lastpass and had full admin access to all servers, they couldn't steal anything. And, in fact, if this system was properly designed, it could all run with opensource server code, and the datastore fully open for anyone to inspect, to prove that the security is down to cryptography rather than trusted yet fallible humans.

thisgoesnowhere 4y ago
What part of this is not happening right now with lastpass?

j-bos 4y ago

Am I understanding correctly: A single engineer had both access to the prod database AND the data decryption values?

secabeen 4y ago
Depends on what you mean by "the data decryption values". If you mean the encryption keys for the vaults themselves, no. Those are derived from the individual master passwords (in non-SSO setups, with SSO it's more complicated, and I don't fully understand the impact). So the attackers have a bunch of encrypted vaults from a backup. They can now brute-force the vaults, but if the original Master Passwords were secure (16+ characters, all 4 classes), those vaults should remain secure. Of course, many people use bad passwords, and those people are at risk.
- dvngnt_ 4y ago
  the exposed passwords isn't even the worst part. it's the fact that user vaults sites are unencrypted so now attackers know every website that you thought was important enough to save. they know what bank you use, which adult sites you visit, what medical conditions you might have
  - secabeen 4y ago
    I understand that some people feel that that is sensitive data. It's not a big threat for me. My ISP and Google already has that data in many cases (IP destination data, DNS lookups, etc.) YMMV.
- j-bos 4y ago
  ah, thanks for the clarification.

lrvick 4y ago

In my experience the majority of hacks are from a compromised laptop of a production engineer. Everyone blindly NPM installs away all their problems and no one checks signatures anymore. Most are using package managers like Brew that don't sign anything to begin with. At Distrust, my security consulting firm, we train all our clients to build production systems that require a minimum of two engineers to mutate and that only pristine operating systems access production that have only signed reproducible used packages that have never been used for anything else. Production environments need to be managed like careful methodical clean-room labs with strict accountability. Instead they are managed like collaborative art projects where everyone is trusted and nothing bad can happen.

q1w2 4y ago
In this case, the anonymous source says that Plex server was compromised, so I assume it was a developer's home PC, not a work laptop. The breach was preceded a couple of days by Plex corporate breach which devulged the engineer's credentials and home IP address. This would have allowed the attackers to access Plex sever remotely, after which the source claims they used an RCE to install a keylogger (and probably a back door) on the engineer's PC. The concerning part is that according to Plex devs (in their reddit sub), they have NO KNOWLEDGE of any RCE. They also haven't communicated with Lastpass, and no one reached out to them. So if there's a Plex remote code exploit - it is still unpatched and actively being exploited - 8 months later! Given that there is still no information on this Plex RCE, we should not assume that it requires authentication to function. So if you're using Plex, make sure to turn off public accessibility asap!
- TheCondor 4y ago
  If you don't need remote access to your plex, you should disable it. The cache features are pretty good, so you could download media while at home and have it on a device in the car or at work pretty easily. I think it's pretty dubious that they could pin it on a plex media server breach though. This is from the same company that sells logmein and goto; I haven't heard definitively that those pieces of software weren't installed on there.. Unless this machine was used for Plex and the occasional log in to work from home type thing, I doubt you could do the forensics on it with much reliability unless you got to it within days of the attack. If this is a normal devops guy? He's got 3 different chat apps, probably Steam, who knows what pirated crap is on there... Plex seems like a very convenient target; the plex corporate attack seems very very plausible as giving the attacker information though.

skarz 4y ago

I think something that should have been in the title is that the breach was facilitated by a vulnerability in an application that many users here might have, Plex. They don’t speak on the nature of the vulnerability in that application. Has it been addressed and fixed through security patches already, or is Plex still potentially dangerous right now?

q1w2 4y ago
Plex devs commented in Reddit that this is the first they've heard of it and haven't identified any RCE, let alone patched anything. So if you're running a Plex server, you should disable public access immediately.

skeeter2020 4y ago

There's a weird combination of "what happened" and causation in here: >> the credentials for the servers were stolen from a DevOps engineer who had access to cloud storage at the company. This made it more difficult for LastPass to detect the suspicious activity. How does A => B? >> The threat actor was able to capture the employee’s master password as it was entered, after the employee authenticated with MFA, and gain access to the DevOps engineer’s LastPass corporate vault. The part about authentication and MFA doesn't track with the rest of the sentence. How does a password without also having the MFA channel work? How would this give me access to a LP vault? Why were they on their home machine? I understand it must be hard to try and come clean in a RCA without injecting some excuses and mitigating factors but you can't attempt soften the blow or the entire thing is a big, smelly mess. It should be a bunch of facts with no emotion, THEN the mitigation and lessons learned. LP just issued a bunch of press releases.

deleted 4y ago
[deleted]
secabeen 4y ago
> The part about authentication and MFA doesn't track with the rest of the sentence. How does a password without also having the MFA channel work? How would this give me access to a LP vault? Why were they on their home machine? I think the idea here is that we want the cryptography operations to happen entirely locally, so that LastPass doesn't have any access to them. However, if you do that, someone with root on that system and the Master Password can replicate the operations the local system does on the vault. I'm not aware of any symmetric-encryption algorithm that includes a time-based un-replayable TOTP or HOTP in the key-generation process.

papito 4y ago

I really wonder how many companies have figured out how to properly secure equipment and access used by a distributed work force.

remram 4y ago

> DevOps engineer So LastPass, a security company, has people who are tasked with both delivering features as fast as possible using a variety of development tools (dev) and administer their production systems where critical data lies (ops). On the same machine, from home. It seems that this breach is not really an accident and more a logical conclusion.

bgro 4y ago

Sometimes I wonder if I shouldn’t google the weather on my work computer because it might have some crazy exploit or cause an alert. DevOps guy here is just running a Plex server and probably pirating porn on his work laptop because why not at that point. Can’t even bother updating this which Plex makes literally about as easy as they possibly could. It’s a one click fully automated process with zero interruption. Was he using internet explorer too? Or was it his work laptop or did they just let him log in to work on any random computer? Did he actually get any work done? I want to hear an interview from this guy. How did he get hired as DevOps while avoiding all the basics of fundamental basic work practices?