10 ms·

Think from the beginning what will be the end: "I thought your security policy was too overbearing, so I used tor." IT departments make their choices for reaso

by ISL 5y ago

Think from the beginning what will be the end: "I thought your security policy was too overbearing, so I used tor." IT departments make their choices for reasons. The key is to help them understand your use-case, and they'll probably help you through the problem in a way that might limit collateral damage. Source: have seen firewall bypasses (with a pre-shared key) get leveraged as a way to hack an entire university lab/department.

azalemeth 5y ago
I tried doing that, and largely succeeded, but the specific area of the university in question will not have a bastion SSH host anywhere on their network. They will not allow SSH access in at all. They will however allow SSH access to other parts of the university, with different people in charge, which explicitly do allow an SSH bastion host to exist (and provide several for that purpose). So, the net result is that they've effectively out-sourced the control and responsibility of their environment to someone else. Normally this is fine, but my job involves programming and controlling large, expensive, and strangely fragile lab equipment. There's a resilience problem, and it's got to the point where others have suggested putting a GSM modem on a pci-e card inside some of the boxes in question, as the relevant IT department decides on a whim to block ports with no warning or justification. Some manufacturers of the devices in question do this as standard if you have a support contract. Trying to complain results in responses like "you have been used to doing things one way and this change now prevents you from working as before." I completely accept that this is a political problem and best solved as one, but ultimately SSH is an industry standard for a reason -- it's secure, and it's flexible. The machines in question are valuable, prone to breaking in the middle of the night, and we are an international bunch who cannot always connect from a well-defined ipv4 address, or from the university's VPN. (The latter is blocked by the IT department automatically, as it has too large a pool of potential users). The thing I find most frustrating is that this sort of political decision creates days worth of work instantaneously, for little benefit. All of the actually confidential or sensitive information is held in a completely separate network at any rate...
- unethical_ban 5y ago
  To quote Dr. Manhattan, "Without condemning, or condoning, I understand". I am in network security. I have stopped shadow IT, and been a part of it. Your situation seems so ungodly stupid and anathema to the point of IT, that the remaining courses of action should be the following. Thoroughly document via email your attempts at explaining requirements to Netsec, to document in writing their objections, to do your best with what they provide you... and WHEN things catastrophically break, point the finger at them and thoroughly document how if you had the proper, industry-standard tooling, you could have prevented the loss of research/time/money.
  - zaphar 5y ago
    THIS. Don't paper over the issues with shadow IT. Make them painfully obvious to the point where IT has to do something or answer to it. Otherwise it will not change. I've given teams the option to turn off their pagers when this sort of thing happens with the justification that they can't fix it anyway. And then documented the crap out of why they can't fix it so when someone asks I can point to existing policy. It's very effective if done right.
  - dooglius 5y ago
    "What did you accomplish during your time with X research group?" "Nothing since all our equipment broke, but we documented how it was all IT's fault. You shoulda seen the looks on their faces when we called them out on it to the dean!"
    - Spivak 5y ago
      It seems weird to dump on IT when they’re a department responding to the incentive structure they’re placed under like everyone else. You going to the Dean/someone with actual authority to get top down approval for IT to give you what you want is basically how IT operates in large orgs. I have nigh infinite technical power but in return I am bound politically by polities that I’m explicitly not allowed to have any authority over (i.e. I can’t approve my own policy exceptions). I want to give you literally anything you ask for. As long as my ass is covered it literally doesn’t matter at all to me. When I worked Uni IT if someone wanted something we couldn’t give them because $dumb_reason weren’t in a position to have that fight with the higher-ups on their behalf. It doesn’t mean much coming from us and since it’s not impacting our work it falls on deaf ears. From your tone you make it seem like you were proud to waste everyone’s time and money when one single meeting with the Dean and the CIO/Director of IT when the problem happened would have opened every door for you.
ryneandal 5y ago
> IT departments make their choices for reasons. In a perfect world, yes. But I've worked with/at places where ineptitude is rampant, and any attempts of understanding their reasoning is seen as insubordination.
novok 5y ago
IT departments make choices that benefit their own needs and for their own convience, often forgetting that the entire point of their department is to make the rest of the organization more effective. Sadly, it often goes the other way. Shadow IT is a signal that the IT organization is doing things wrong. People use shadow IT because the IT department is not doing it's job properly, serving it's customer base based on the needs they show via their actions. For example, if you see someone like azalemeth do the things he does, it shows that the IT department needs to become responsive enough and cooperative enough to not push him to do such things in the first place. You notice he's tried to do thing the IT department standard way first, and spent considerable effort before he started his shadow IT method.
- relax88 5y ago
  “Policy made my job slightly harder so because I know better than the netsec team who clearly has or should have unlimited time and resources to help me I will do what I want anyways, and put the organization at risk.” Also known as “how to make the netsec team hate you 101” I agree with you about why shadow IT exists, but most IT departments are spread so thin that expecting them to be super responsive to anything but the most critical business projects is often totally unreasonable. Then they have to waste even more time hunting down idiots setting up Tor nodes on their internal networks.
  - still_grokking 5y ago
    If the IT department can't do its job because of resource constraints likely the whole organization is a failure. If you find something like that, run… If you can't run, do whatever makes your live better. The org is doomed anyway.
    - azalemeth 5y ago
      A recent example from me -- one VPN client of mine suddenly refused to connect one day for no discernible reason when they made a configuration change to their cisco vpn "concentrator" without documenting it or announcing it. Cisco AnyConnect GUI clients were fine and some magic happened behind the scenes to push the configuration change and, in typical Cisco style, avoid saying what exactly it was. I had some esoteric monitoring machine that couldn't run anyconnect (for reasons I forget but almost certainly relating to it not having a linux arm64 client at that time) and naturally couldn't connect randomly one day with openconnect (which previously had worked perfectly). I asked what the configuration change was to prevent me having to reverse-engineer it. The response was "if you want to use unsupported clients we cannot offer any assistance [...] we are currently operating two heads down and we simply do not have the resources [...]." It took me about four or five hours to work out what change they had made, change the (122 line long) configuration file for openconnect, and then, boom, everything good again. A friendly "Hey, sorry about that -- we just $FLICKED_THIS_SWITCH because $REASON" would have been massively helpful and arguably take less words than their original response. (Edit: For context, approximately 10-20k people use that specific VPN. And their team is such that losing two members of staff temporarily is a major inconvenience.) I totally understand it from the other side. IT departments have everything from state-sponsored ransomware attacks to important people loudly going "why doesn't the printer work any more". It's a different set of skills to being a C-junkie, a programming wizard, or, in my case, a young academic with one big grant and three PhD students trying to both do work, publish work, and get money to do more work where "work" is poorly defined and highly flexible. Over time I've noticed universities get far more corporate and many academics absolutely hate this, of which I am one. The "we control the network, bug off" may be technically true but at times it does feel a bit like an imposition of some sort of academic freedom, to be honest. At the very least, it's a nice little "dog egg" to find added to the pile of administrative crap to do for that day.
pope_meat 5y ago
A simulated conversation with IT: "Hey, IT department...I was wondering..." "No."
- eitland 5y ago
  Lucky me. Our IT department goes out the of their way to help us stay sane and productive - they're making sure most of us can continue to use our favourite Linux distro (I think most Debian/Ubuntu, Fedora and Arch is supported) - make sure VPN etc works on Linux even if it is not officially supported - taking time to sit down and debug hard problems (weird issues with WSL2 on one particular Windows laptop) instead of just blaming us engineers
nephanth 5y ago
Especially when, since it's Tor, potential attackers cannot be traced
- Theitheave42 5y ago
  This.
- tga_d 5y ago
  This makes no sense. Onion services don't hide the source, they hide the destination -- a destination that, in this threat model, you run. If the client connecting wants to hide their source, they can use Tor, a VPN, an existing botnet, etc.; whether you're running SSH over an onion service or with vanilla exposed IPs/DNS is immaterial.
belorn 5y ago
The problem in my experience is not that the security policy is too overbearing, but rather that the security policy is too rigid and designed with assumptions that are false. A common policy for example is that port 22 must always be closed. One can use a hardware secured two token authentication over ssh, and still the policy is that the port must be closed and that is that. That the policy allow remote desktop with just a password is completely irrelevant because the policy doesn't forbid that. I have tried so many times to help people understand security and the purpose of a security policy when it is designed correctly, but it doesn't work. The policy exist so people don't need to think, not to make people understand why it exist and what use-cases should be given exceptions.
- gpapilion 5y ago
  Often times these policies are driven by industry compliance. Exceptions have to be documented, and depend on the compliance regime, may carry liability. Lastly when exceptions are made the user often doesn’t know what they signed up for, and it ends up holding the bag for a breech. It’s usually better to not make an exception.

by ISL 5y ago

azalemeth 5y ago

I tried doing that, and largely succeeded, but the specific area of the university in question will not have a bastion SSH host anywhere on their network. They will not allow SSH access in at all. They will however allow SSH access to other parts of the university, with different people in charge, which explicitly do allow an SSH bastion host to exist (and provide several for that purpose). So, the net result is that they've effectively out-sourced the control and responsibility of their environment to someone else. Normally this is fine, but my job involves programming and controlling large, expensive, and strangely fragile lab equipment. There's a resilience problem, and it's got to the point where others have suggested putting a GSM modem on a pci-e card inside some of the boxes in question, as the relevant IT department decides on a whim to block ports with no warning or justification. Some manufacturers of the devices in question do this as standard if you have a support contract. Trying to complain results in responses like "you have been used to doing things one way and this change now prevents you from working as before." I completely accept that this is a political problem and best solved as one, but ultimately SSH is an industry standard for a reason -- it's secure, and it's flexible. The machines in question are valuable, prone to breaking in the middle of the night, and we are an international bunch who cannot always connect from a well-defined ipv4 address, or from the university's VPN. (The latter is blocked by the IT department automatically, as it has too large a pool of potential users). The thing I find most frustrating is that this sort of political decision creates days worth of work instantaneously, for little benefit. All of the actually confidential or sensitive information is held in a completely separate network at any rate...

unethical_ban 5y ago
To quote Dr. Manhattan, "Without condemning, or condoning, I understand". I am in network security. I have stopped shadow IT, and been a part of it. Your situation seems so ungodly stupid and anathema to the point of IT, that the remaining courses of action should be the following. Thoroughly document via email your attempts at explaining requirements to Netsec, to document in writing their objections, to do your best with what they provide you... and WHEN things catastrophically break, point the finger at them and thoroughly document how if you had the proper, industry-standard tooling, you could have prevented the loss of research/time/money.
- zaphar 5y ago
  THIS. Don't paper over the issues with shadow IT. Make them painfully obvious to the point where IT has to do something or answer to it. Otherwise it will not change. I've given teams the option to turn off their pagers when this sort of thing happens with the justification that they can't fix it anyway. And then documented the crap out of why they can't fix it so when someone asks I can point to existing policy. It's very effective if done right.
- dooglius 5y ago
  "What did you accomplish during your time with X research group?" "Nothing since all our equipment broke, but we documented how it was all IT's fault. You shoulda seen the looks on their faces when we called them out on it to the dean!"
  - Spivak 5y ago
    It seems weird to dump on IT when they’re a department responding to the incentive structure they’re placed under like everyone else. You going to the Dean/someone with actual authority to get top down approval for IT to give you what you want is basically how IT operates in large orgs. I have nigh infinite technical power but in return I am bound politically by polities that I’m explicitly not allowed to have any authority over (i.e. I can’t approve my own policy exceptions). I want to give you literally anything you ask for. As long as my ass is covered it literally doesn’t matter at all to me. When I worked Uni IT if someone wanted something we couldn’t give them because $dumb_reason weren’t in a position to have that fight with the higher-ups on their behalf. It doesn’t mean much coming from us and since it’s not impacting our work it falls on deaf ears. From your tone you make it seem like you were proud to waste everyone’s time and money when one single meeting with the Dean and the CIO/Director of IT when the problem happened would have opened every door for you.

ryneandal 5y ago

> IT departments make their choices for reasons. In a perfect world, yes. But I've worked with/at places where ineptitude is rampant, and any attempts of understanding their reasoning is seen as insubordination.

novok 5y ago

IT departments make choices that benefit their own needs and for their own convience, often forgetting that the entire point of their department is to make the rest of the organization more effective. Sadly, it often goes the other way. Shadow IT is a signal that the IT organization is doing things wrong. People use shadow IT because the IT department is not doing it's job properly, serving it's customer base based on the needs they show via their actions. For example, if you see someone like azalemeth do the things he does, it shows that the IT department needs to become responsive enough and cooperative enough to not push him to do such things in the first place. You notice he's tried to do thing the IT department standard way first, and spent considerable effort before he started his shadow IT method.

relax88 5y ago
“Policy made my job slightly harder so because I know better than the netsec team who clearly has or should have unlimited time and resources to help me I will do what I want anyways, and put the organization at risk.” Also known as “how to make the netsec team hate you 101” I agree with you about why shadow IT exists, but most IT departments are spread so thin that expecting them to be super responsive to anything but the most critical business projects is often totally unreasonable. Then they have to waste even more time hunting down idiots setting up Tor nodes on their internal networks.
- still_grokking 5y ago
  If the IT department can't do its job because of resource constraints likely the whole organization is a failure. If you find something like that, run… If you can't run, do whatever makes your live better. The org is doomed anyway.
  - azalemeth 5y ago
    A recent example from me -- one VPN client of mine suddenly refused to connect one day for no discernible reason when they made a configuration change to their cisco vpn "concentrator" without documenting it or announcing it. Cisco AnyConnect GUI clients were fine and some magic happened behind the scenes to push the configuration change and, in typical Cisco style, avoid saying what exactly it was. I had some esoteric monitoring machine that couldn't run anyconnect (for reasons I forget but almost certainly relating to it not having a linux arm64 client at that time) and naturally couldn't connect randomly one day with openconnect (which previously had worked perfectly). I asked what the configuration change was to prevent me having to reverse-engineer it. The response was "if you want to use unsupported clients we cannot offer any assistance [...] we are currently operating two heads down and we simply do not have the resources [...]." It took me about four or five hours to work out what change they had made, change the (122 line long) configuration file for openconnect, and then, boom, everything good again. A friendly "Hey, sorry about that -- we just $FLICKED_THIS_SWITCH because $REASON" would have been massively helpful and arguably take less words than their original response. (Edit: For context, approximately 10-20k people use that specific VPN. And their team is such that losing two members of staff temporarily is a major inconvenience.) I totally understand it from the other side. IT departments have everything from state-sponsored ransomware attacks to important people loudly going "why doesn't the printer work any more". It's a different set of skills to being a C-junkie, a programming wizard, or, in my case, a young academic with one big grant and three PhD students trying to both do work, publish work, and get money to do more work where "work" is poorly defined and highly flexible. Over time I've noticed universities get far more corporate and many academics absolutely hate this, of which I am one. The "we control the network, bug off" may be technically true but at times it does feel a bit like an imposition of some sort of academic freedom, to be honest. At the very least, it's a nice little "dog egg" to find added to the pile of administrative crap to do for that day.

pope_meat 5y ago

A simulated conversation with IT: "Hey, IT department...I was wondering..." "No."

eitland 5y ago
Lucky me. Our IT department goes out the of their way to help us stay sane and productive - they're making sure most of us can continue to use our favourite Linux distro (I think most Debian/Ubuntu, Fedora and Arch is supported) - make sure VPN etc works on Linux even if it is not officially supported - taking time to sit down and debug hard problems (weird issues with WSL2 on one particular Windows laptop) instead of just blaming us engineers

nephanth 5y ago

Especially when, since it's Tor, potential attackers cannot be traced

Theitheave42 5y ago
This.
tga_d 5y ago
This makes no sense. Onion services don't hide the source, they hide the destination -- a destination that, in this threat model, you run. If the client connecting wants to hide their source, they can use Tor, a VPN, an existing botnet, etc.; whether you're running SSH over an onion service or with vanilla exposed IPs/DNS is immaterial.

belorn 5y ago

The problem in my experience is not that the security policy is too overbearing, but rather that the security policy is too rigid and designed with assumptions that are false. A common policy for example is that port 22 must always be closed. One can use a hardware secured two token authentication over ssh, and still the policy is that the port must be closed and that is that. That the policy allow remote desktop with just a password is completely irrelevant because the policy doesn't forbid that. I have tried so many times to help people understand security and the purpose of a security policy when it is designed correctly, but it doesn't work. The policy exist so people don't need to think, not to make people understand why it exist and what use-cases should be given exceptions.

gpapilion 5y ago
Often times these policies are driven by industry compliance. Exceptions have to be documented, and depend on the compliance regime, may carry liability. Lastly when exceptions are made the user often doesn’t know what they signed up for, and it ends up holding the bag for a breech. It’s usually better to not make an exception.