4 ms·

I'm the only person in my company with any IT knowledge. I'm a developer, not an IT expert. These stories terrify me. I have no mandate or time to work on our s

by xupybd 5y ago

I'm the only person in my company with any IT knowledge. I'm a developer, not an IT expert. These stories terrify me. I have no mandate or time to work on our security. What would you do to protect your company. I have limited backups but we would be done with systems down for days.

amelius 5y ago
Please note that backups aren't a good measure against ransomware, unless you do them absolutely correctly. The problem is that ransomware will encrypt your files, rendering them useless, but they still end up in encrypted form in your backup.
- WJW 5y ago
  Almost every backup system I've seen will keep multiple versions of the file around with decreasing frequency as time progresses, ie one for every day of last week, every sunday of the last month, the first of every month for the last year, etc. That way if you get hit by ransomware, you can restore to a point in time where you are (fairly) sure no infection was present yet. Nothing is perfect, but this does give a decent amount of protection for "most" important files as they tend not to change that often. For things that do change often like databases, different strategies may be needed.
  - amelius 5y ago
    Yes, but if your system is hacked, then any application (including your backup software) might "see" the file as unencrypted.
    - WJW 5y ago
      Yes? Some of your backups will be the encrypted version of the file. As long as your system remains hacked it is useless to restore anything. You will first need to purge every disk in your organisation and reinstall everything from scratch (depending on the sophistication of the ransomware, maybe just buy new disks altogether), THEN restore from a version that is good.
      - amelius 5y ago
        Yes, but backing up from an infected system is probably not a good idea. Better to mount the drives on a different system, and backup from there.
    - nix23 5y ago
      That's why you use WORM Tapes and versioned files if its really important data.
  - beermonster 5y ago
    So long as the compromised system isn’t able to access and alter any historic backups. Things like zfs snapshots or append-only backups help protect your backups by not permitting this.
dgellow 5y ago
Report this to your manager and make it clear (using reports from cases, such as those ransomware cases) that you need to invest right away in security and a backup process you trust.
- deleted 5y ago
  [deleted]
- nix23 5y ago
  Absolute correct, be honest with downtime include worst and best cases, extend those to non IT-Matters and let your Manager take over the responsibility if nothing changes. And this: https://www.usenix.org/system-administrators-code-ethics https://www.usenix.org/system-administrators-code-ethics
beermonster 5y ago
Identify data assets that are important to you and back them up and work out recovery objectives in advance. This may sound complicated at first but follow this to get you started: https://docs.borgbase.com/strategy/ https://docs.borgbase.com/strategy/ Also a tool like borg [1] will help you ‘do backups right’. If you’re on Windows maybe checkout restic [2]. Next, make sure you patch software as unpatched software may contain vulnerabilities which can be exploited by a threat actor. Quicker is better and automating this allows audit, consistency, and expediency. If you’re on MacOS check out RansomWhere from objective-see : https://objective-see.com/products/ransomwhere.html https://objective-see.com/products/ransomwhere.html Educate users so they are aware of the risks of opening emails from people they do not know and how to identify potential phishing. Various anti malware/anti virus software can be used in conjunction with the above but cannot be relied on by itself - defence in depth. Also you shouldn’t feel this is all your responsibility. Try and raise this business risk with the powers that be. See if you can not only get a company mandate but also maybe obtain someone with more expertise? [1] https://www.borgbackup.org/ https://www.borgbackup.org/ [2] https://restic.net/ https://restic.net/
1337shadow 5y ago
I suppose the best is to outsource, we have such offers in one of my businesses for example: https://yourlabs.org/secops/ https://yourlabs.org/secops/
uniqueid 5y ago
These stories terrify me. If they scare you now, just wait a few months or years until we see some massive breaches of Google, Facebook and Amazon data. It's possible the 'cloud-storage' model is a house of cards, because it may be the case that the whole thing collapses after a sufficiently high percent of the public experiences a financially-, socially- or career-devastating leak.
dspillett 5y ago
> What would you do to protect your company. How big is the company? If the company is big enough then it should really have an infrastructure manager or similar who is directly responsible for this sort of thing, rather than relying on seconding the dev team into managing IT. When we were a team of five plus the owner and I gave up on getting time+budget to properly setup off-site backups, I ended up spending a weekend hacking together something with ssh+rsync to the machine under my desk at home for key data (the source repos, email, etc.) basically replicating what I did for my home data (backing up to an external site). I can't recommend this. It no doubt breaks many data protection rules. But I wasn't comfortable with the idea that my job would be entirely gone if the building burned down overnight and we lost what would be needed to restore operation. I was protecting me, not the company at that point. Luckily when we were in the process of being bought and due diligence audits came around, backup & DR concerns were taken a bit more seriously and I did get the time allocated to do something better. > I have limited backups but we would be done with systems down for days. Make a case to management that these attacks are not targetted at the big companies, they just happen to catch them in the net and we hear about them because they are big companies. The bots out there infecting sites will get into anywhere they can, and the blackmailers are more than happy to have many small marks instead of a few big ones. You are a target as much as Fuji or Garmin are. That case should list how long it would take to get operational again (refreshed infrastructure, restored data) if you paid for decryption and if you didn't (which in your case might be "it'll never happen" currently). Make a recommended plan and list what the restore time is for that if you had to rebuild everything. Break the restore time into essentials (what you need to support current clients) and everything else (what you need to continue new work and chase new clients). Also include in your plan time to regularly test your backups and arrange some automated tests of key parts. That, other than taking matters into your own hands in your own time which is as likely to get you slapped as it is to get you thanked, is all you can do. If they don't take these matters seriously, consider if you can get a job with a company that does (you'll have to if the worst happens anyway so consider planning this to be your personal DR plan even if you don't want to jump ship now).
Woodi 5y ago
If you are only one with IT knowledge there it is already your responsibility - in case of ransomware or some disaster you can be hold responsible. Of course all depends on your contract, potential loses, managers mood and managers need to cover their asses. Good news: looks like your position makes you kind of manager yourself - you, to some degree, can influence and even demand things. And for sure your responsibility is to communicate state of business to the upper chain. Or maybe just make critical systems off the internets :>

by xupybd 5y ago

amelius 5y ago

Please note that backups aren't a good measure against ransomware, unless you do them absolutely correctly. The problem is that ransomware will encrypt your files, rendering them useless, but they still end up in encrypted form in your backup.

WJW 5y ago
Almost every backup system I've seen will keep multiple versions of the file around with decreasing frequency as time progresses, ie one for every day of last week, every sunday of the last month, the first of every month for the last year, etc. That way if you get hit by ransomware, you can restore to a point in time where you are (fairly) sure no infection was present yet. Nothing is perfect, but this does give a decent amount of protection for "most" important files as they tend not to change that often. For things that do change often like databases, different strategies may be needed.
- amelius 5y ago
  Yes, but if your system is hacked, then any application (including your backup software) might "see" the file as unencrypted.
  - WJW 5y ago
    Yes? Some of your backups will be the encrypted version of the file. As long as your system remains hacked it is useless to restore anything. You will first need to purge every disk in your organisation and reinstall everything from scratch (depending on the sophistication of the ransomware, maybe just buy new disks altogether), THEN restore from a version that is good.
    - amelius 5y ago
      Yes, but backing up from an infected system is probably not a good idea. Better to mount the drives on a different system, and backup from there.
  - nix23 5y ago
    That's why you use WORM Tapes and versioned files if its really important data.
- beermonster 5y ago
  So long as the compromised system isn’t able to access and alter any historic backups. Things like zfs snapshots or append-only backups help protect your backups by not permitting this.

dgellow 5y ago

Report this to your manager and make it clear (using reports from cases, such as those ransomware cases) that you need to invest right away in security and a backup process you trust.

deleted 5y ago
[deleted]
nix23 5y ago
Absolute correct, be honest with downtime include worst and best cases, extend those to non IT-Matters and let your Manager take over the responsibility if nothing changes. And this: https://www.usenix.org/system-administrators-code-ethics https://www.usenix.org/system-administrators-code-ethics

beermonster 5y ago

Identify data assets that are important to you and back them up and work out recovery objectives in advance. This may sound complicated at first but follow this to get you started: https://docs.borgbase.com/strategy/ https://docs.borgbase.com/strategy/ Also a tool like borg [1] will help you ‘do backups right’. If you’re on Windows maybe checkout restic [2]. Next, make sure you patch software as unpatched software may contain vulnerabilities which can be exploited by a threat actor. Quicker is better and automating this allows audit, consistency, and expediency. If you’re on MacOS check out RansomWhere from objective-see : https://objective-see.com/products/ransomwhere.html https://objective-see.com/products/ransomwhere.html Educate users so they are aware of the risks of opening emails from people they do not know and how to identify potential phishing. Various anti malware/anti virus software can be used in conjunction with the above but cannot be relied on by itself - defence in depth. Also you shouldn’t feel this is all your responsibility. Try and raise this business risk with the powers that be. See if you can not only get a company mandate but also maybe obtain someone with more expertise? [1] https://www.borgbackup.org/ https://www.borgbackup.org/ [2] https://restic.net/ https://restic.net/

1337shadow 5y ago

I suppose the best is to outsource, we have such offers in one of my businesses for example: https://yourlabs.org/secops/ https://yourlabs.org/secops/

uniqueid 5y ago

These stories terrify me. If they scare you now, just wait a few months or years until we see some massive breaches of Google, Facebook and Amazon data. It's possible the 'cloud-storage' model is a house of cards, because it may be the case that the whole thing collapses after a sufficiently high percent of the public experiences a financially-, socially- or career-devastating leak.

dspillett 5y ago

> What would you do to protect your company. How big is the company? If the company is big enough then it should really have an infrastructure manager or similar who is directly responsible for this sort of thing, rather than relying on seconding the dev team into managing IT. When we were a team of five plus the owner and I gave up on getting time+budget to properly setup off-site backups, I ended up spending a weekend hacking together something with ssh+rsync to the machine under my desk at home for key data (the source repos, email, etc.) basically replicating what I did for my home data (backing up to an external site). I can't recommend this. It no doubt breaks many data protection rules. But I wasn't comfortable with the idea that my job would be entirely gone if the building burned down overnight and we lost what would be needed to restore operation. I was protecting me, not the company at that point. Luckily when we were in the process of being bought and due diligence audits came around, backup & DR concerns were taken a bit more seriously and I did get the time allocated to do something better. > I have limited backups but we would be done with systems down for days. Make a case to management that these attacks are not targetted at the big companies, they just happen to catch them in the net and we hear about them because they are big companies. The bots out there infecting sites will get into anywhere they can, and the blackmailers are more than happy to have many small marks instead of a few big ones. You are a target as much as Fuji or Garmin are. That case should list how long it would take to get operational again (refreshed infrastructure, restored data) if you paid for decryption and if you didn't (which in your case might be "it'll never happen" currently). Make a recommended plan and list what the restore time is for that if you had to rebuild everything. Break the restore time into essentials (what you need to support current clients) and everything else (what you need to continue new work and chase new clients). Also include in your plan time to regularly test your backups and arrange some automated tests of key parts. That, other than taking matters into your own hands in your own time which is as likely to get you slapped as it is to get you thanked, is all you can do. If they don't take these matters seriously, consider if you can get a job with a company that does (you'll have to if the worst happens anyway so consider planning this to be your personal DR plan even if you don't want to jump ship now).

Woodi 5y ago

If you are only one with IT knowledge there it is already your responsibility - in case of ransomware or some disaster you can be hold responsible. Of course all depends on your contract, potential loses, managers mood and managers need to cover their asses. Good news: looks like your position makes you kind of manager yourself - you, to some degree, can influence and even demand things. And for sure your responsibility is to communicate state of business to the upper chain. Or maybe just make critical systems off the internets :>