4 ms·

Are there any reliable ways to secure remote computers from keyloggers _and_ still provide an efficient software development environment for non-trivial project

by theamk 4y ago

Are there any reliable ways to secure remote computers from keyloggers _and_ still provide an efficient software development environment for non-trivial projects? All of the software engineers I have seen have a fairly unrestricted environment -- Linux machines, with sudo access, often with passwordless root access via "docker" group, and with non-intrusive "endpoint protection" system. It would be normal to for someone to run "npm install" on their machine, or check out a random github repo they read about and run code from it. Such machine would be a prime target for malware. And endpoint protection I have seen seems to be really stupid -- basically hooking "exec" calls and checking for exact hash match (!). Any serious malware should be able to bypass it without much effort, and if it only stays on a single computer, the detection chance is pretty low. (I have also seen some poor souls who were stuck on locked-down Windows machines.. but they usually ended up using their machines as remote terminals, doing most their actual work on some remote server. And that server is sudo-capable Linux with light/no protection, and see previous paragraph. I suppose if _that_ is infected, at least Lastpass might not be stolen... unless people start browser on server and log into lastpass there, I've seen this happen)

gavanm 4y ago
You've already included the answer - "using their machines as remote terminals, doing most their actual work on some remote server". The developer uses MFA (TOTP, Push Notification, Yubikey etc) into a virtual desktop inside the organisation (Citrix, VMWare Horizon, etc). From there, the developer can SSH / whatever into their development environment - which is hosted "inside" the corporate network, or their cloud provider, via internal links. All code, and dev boxes live "inside" the corporate network, and only keypresses, mouse movement, and screen diffs are sent back and forth. Most remote access packages can prevent clipboard, USB device, file transfer etc. If you need a password manager for work purposes, then it lives on the corporate managed network - not your remote laptop/desktop - and to be really paranoid - you only ever "copy/paste" those passwords - you don't type them in. If you really want to lock it down further, give the remote workers dedicated corporate equipment that they only use to access the remote desktops, so you can prevent some things like screen capturing, and really lock down the software to prevent things like keylogging software/malware. You also should have the entire development environment segregated from the "business" corporate network as well. It's only really an issue if you want to have offline developers - in which case I don't have any thoughts ready to hand - (but would expect it to be a very locked down machine, possibly with an even more locked down VM inside it). As someone who regularly uses multiple layers of Virtual Desktop -> Virtual Desktop -> Remote Desktop, provided the network can handle it (on both your local network, and the corporate network), it works surprisingly well.
- wepzen 4y ago
  In my experience, I find that working with Virtual Desktops is the most frustrating user experience as a developer you could have. I prefer working in containerized environments which are more efficient and do not require the same amount of configuration processes as a Virtual Desktop. You should check some solutions out.
- faeyanpiraat 4y ago
  > You also should have the entire development environment segregated from the "business" corporate network as well. This is the key point, otherwise you'll also just get a linux machine with developer root access, but EVEN CLOSER to the corporate network.
- realharo 4y ago
  For most development work, this would probably cause a serious productivity drop, but it definitely makes sense for the portion of the work that involves accessing critical production resources. For DevOps roles, that could well be the majority.
- execveat 4y ago
  I'm all in for VM based privilege separation, but that won't protect you from infected endpoint. Assuming this was a targeted attack, folks that achieved RCE on DevOp engineer's machine could have waited for her to authenticate and then inject keystrokes into VM, SSH, VNC, Remote Desktop, Citrix or whatever remote management system they're using. Honestly, this HN thread is full of bad advice and factually incorrect patronizing. Okta-style system asking to accept every single permission would not have protected from an attack, because Okta caches and reuses authentication tokens. Clipboard snooping / keylogger detection wouldn't have worked because none of these solutions are robust against targeted attacks. The only thing I can think of which would have (and should have) helped is alert SOC / incident reponse team. Good luck finding one though.
  - paradox242 4y ago
    Glad to see someone else with the same reaction, because a lot of this advice is... interesting, like people who are worried about keyloggers but think the clipboard is safe.
hnbear 4y ago
Give you a MacBook and a cloud Linux dev machine (or as many as you want). Secure access to dev environment with multiple easy factors - MBP biometrics, yubikey, etc Enforce strong password and FDE on the laptop. Obviously doesn’t need to be a MacBook, but you stated an aversion to overly locked down windows laptops. In general, just enforce strong multifactor creds when accessing internal resources. You can also enforce device very requirements, so a stole device or yubi can’t be used either. Then do what you want on your Linux instances in the cloud. They’re protected with their own controls.
kodah 4y ago
Personally, I'm not a fan of the answers that amount to a cloud-hosted thin client. I use these at work, they're absolute technological marvels, but they suck. The real answer is a zero trust network that implements: - multi factor auth - deployment approval gates - end to end service encryption - ALE for secrets and keys - password managers - WireGuard tunneling or equivalent - read only production environments by default; major levers to pull in order to write - fully partitioned environments, all of which partitioned away from the corporate network of laptops, printers, and security cameras
- eru 4y ago
  > - read only production environments by default; major levers to pull in order to write Yes. In general, it's a good idea to split state management from business logic. In the simplest thing, that means that eg you have a database that's separate from the rest of your site. But the principle applies more generally. Useful for keeping things simple. To go further: if you want to log something, you send it to a log server that is super simple and can only write to one location. So if someone takes over your business logic service, they can't write arbitrarily.
pid-1 4y ago
My company operates in a Windows centric industry and our software team uses it as well. It turns out you don't need administrative privileges for a lot of dev work (installing and running vs code, python, node, many databases, etc...). My experience is that sudo apt-get install is a Linux Distro thing, most programs don't need special permissions as long they are installed in user scope. So, answering your question, our devs are like regular users: when they need to install something that needs privileges they call IT. Surprisingly, that rarely happens.
- execveat 4y ago
  Privilege escalation on Windows is super easy though, every red teamer I know has a bunch of ready to use exploits (most of them public) up their sleeve. And it is virtually impossible to get a good baseline of a developer's machine, so I'm pretty sure every SOC out there is simply allowlisting huge swaths of your software. You can sorta kinda harden these systems, but that would only work against common malware. And you generally can't isolate senior engineers in their own little DMZ, so any RAT on their machines usually leads to catastrophic consequences.
- marcosdumay 4y ago
  Privilege escalation is a red herring. Everything you need to compromise production from a developer PC is either available for a regular user or not available at all.
worksonmine 4y ago
Anyone with this level of access should know not to run random github projects or npm install in the most sensitive context. The choice is between easy or secure. You can't have both and that's a reality one has to accept. It's not that difficult to spin up a VM when you want to fuck around and isolate it from sensitive data. Especially as a "DevOps engineer", gatekeeping and providing least privilege access is in the job description. I understand getting lazy and relaxing the rules in some contexts but not when running a password manager on this scale, unacceptable.
locustous 4y ago
Don't do development on any system that has access to production. Develop on dev lane resources. Production access should only be allowed from a locked down system with no open ports and a very small whitelist set of software. Operations for said system should be simple. Deploy version x with necessary provisioning. Backup system. Restore system. View monitoring and logs. You must minimize the surface area connected to production. On the network side the system with the keys for production is also firewalled off from general Internet access. So potential malware can't phone home.

by theamk 4y ago

gavanm 4y ago

You've already included the answer - "using their machines as remote terminals, doing most their actual work on some remote server". The developer uses MFA (TOTP, Push Notification, Yubikey etc) into a virtual desktop inside the organisation (Citrix, VMWare Horizon, etc). From there, the developer can SSH / whatever into their development environment - which is hosted "inside" the corporate network, or their cloud provider, via internal links. All code, and dev boxes live "inside" the corporate network, and only keypresses, mouse movement, and screen diffs are sent back and forth. Most remote access packages can prevent clipboard, USB device, file transfer etc. If you need a password manager for work purposes, then it lives on the corporate managed network - not your remote laptop/desktop - and to be really paranoid - you only ever "copy/paste" those passwords - you don't type them in. If you really want to lock it down further, give the remote workers dedicated corporate equipment that they only use to access the remote desktops, so you can prevent some things like screen capturing, and really lock down the software to prevent things like keylogging software/malware. You also should have the entire development environment segregated from the "business" corporate network as well. It's only really an issue if you want to have offline developers - in which case I don't have any thoughts ready to hand - (but would expect it to be a very locked down machine, possibly with an even more locked down VM inside it). As someone who regularly uses multiple layers of Virtual Desktop -> Virtual Desktop -> Remote Desktop, provided the network can handle it (on both your local network, and the corporate network), it works surprisingly well.

wepzen 4y ago
In my experience, I find that working with Virtual Desktops is the most frustrating user experience as a developer you could have. I prefer working in containerized environments which are more efficient and do not require the same amount of configuration processes as a Virtual Desktop. You should check some solutions out.
faeyanpiraat 4y ago
> You also should have the entire development environment segregated from the "business" corporate network as well. This is the key point, otherwise you'll also just get a linux machine with developer root access, but EVEN CLOSER to the corporate network.
realharo 4y ago
For most development work, this would probably cause a serious productivity drop, but it definitely makes sense for the portion of the work that involves accessing critical production resources. For DevOps roles, that could well be the majority.
execveat 4y ago
I'm all in for VM based privilege separation, but that won't protect you from infected endpoint. Assuming this was a targeted attack, folks that achieved RCE on DevOp engineer's machine could have waited for her to authenticate and then inject keystrokes into VM, SSH, VNC, Remote Desktop, Citrix or whatever remote management system they're using. Honestly, this HN thread is full of bad advice and factually incorrect patronizing. Okta-style system asking to accept every single permission would not have protected from an attack, because Okta caches and reuses authentication tokens. Clipboard snooping / keylogger detection wouldn't have worked because none of these solutions are robust against targeted attacks. The only thing I can think of which would have (and should have) helped is alert SOC / incident reponse team. Good luck finding one though.
- paradox242 4y ago
  Glad to see someone else with the same reaction, because a lot of this advice is... interesting, like people who are worried about keyloggers but think the clipboard is safe.

hnbear 4y ago

Give you a MacBook and a cloud Linux dev machine (or as many as you want). Secure access to dev environment with multiple easy factors - MBP biometrics, yubikey, etc Enforce strong password and FDE on the laptop. Obviously doesn’t need to be a MacBook, but you stated an aversion to overly locked down windows laptops. In general, just enforce strong multifactor creds when accessing internal resources. You can also enforce device very requirements, so a stole device or yubi can’t be used either. Then do what you want on your Linux instances in the cloud. They’re protected with their own controls.

kodah 4y ago

Personally, I'm not a fan of the answers that amount to a cloud-hosted thin client. I use these at work, they're absolute technological marvels, but they suck. The real answer is a zero trust network that implements: - multi factor auth - deployment approval gates - end to end service encryption - ALE for secrets and keys - password managers - WireGuard tunneling or equivalent - read only production environments by default; major levers to pull in order to write - fully partitioned environments, all of which partitioned away from the corporate network of laptops, printers, and security cameras

eru 4y ago
> - read only production environments by default; major levers to pull in order to write Yes. In general, it's a good idea to split state management from business logic. In the simplest thing, that means that eg you have a database that's separate from the rest of your site. But the principle applies more generally. Useful for keeping things simple. To go further: if you want to log something, you send it to a log server that is super simple and can only write to one location. So if someone takes over your business logic service, they can't write arbitrarily.

pid-1 4y ago

My company operates in a Windows centric industry and our software team uses it as well. It turns out you don't need administrative privileges for a lot of dev work (installing and running vs code, python, node, many databases, etc...). My experience is that sudo apt-get install is a Linux Distro thing, most programs don't need special permissions as long they are installed in user scope. So, answering your question, our devs are like regular users: when they need to install something that needs privileges they call IT. Surprisingly, that rarely happens.

execveat 4y ago
Privilege escalation on Windows is super easy though, every red teamer I know has a bunch of ready to use exploits (most of them public) up their sleeve. And it is virtually impossible to get a good baseline of a developer's machine, so I'm pretty sure every SOC out there is simply allowlisting huge swaths of your software. You can sorta kinda harden these systems, but that would only work against common malware. And you generally can't isolate senior engineers in their own little DMZ, so any RAT on their machines usually leads to catastrophic consequences.
marcosdumay 4y ago
Privilege escalation is a red herring. Everything you need to compromise production from a developer PC is either available for a regular user or not available at all.

worksonmine 4y ago

Anyone with this level of access should know not to run random github projects or npm install in the most sensitive context. The choice is between easy or secure. You can't have both and that's a reality one has to accept. It's not that difficult to spin up a VM when you want to fuck around and isolate it from sensitive data. Especially as a "DevOps engineer", gatekeeping and providing least privilege access is in the job description. I understand getting lazy and relaxing the rules in some contexts but not when running a password manager on this scale, unacceptable.

locustous 4y ago

Don't do development on any system that has access to production. Develop on dev lane resources. Production access should only be allowed from a locked down system with no open ports and a very small whitelist set of software. Operations for said system should be simple. Deploy version x with necessary provisioning. Backup system. Restore system. View monitoring and logs. You must minimize the surface area connected to production. On the network side the system with the keys for production is also firewalled off from general Internet access. So potential malware can't phone home.