6 ms·

Hadoop (and hence HDFS) is a stack of services designed to work together to serve a file system and manage jobs. The hadoop stack has a pluggable authentication

Hadoop (and hence HDFS) is a stack of services designed to work together to serve a file system and manage jobs. The hadoop stack has a pluggable authentication/authorization by design. And yes, the default is "no security". Given the distributed nature, HDFS runs on multiple machines. In linux distributed service security fits well with kerberos. Normally if you want a "secure" HDFS you must "kerberize" the services such that any hadoop operation requires a valid/authorized TGT. To most people kerberizing a hadoop cluster is a major barrier to getting hadoop running. I dont see this changing but certain vendor hadoop distros break down some of the barriers. Sometimes it is OK if you run a cluster insecure. Please dont do it if youre handling my financial or medical records though. As Mr.T once said 'dont write checks that yo ass cant cash'

jamesblonde 9y ago
It is changing. Disclosure- I work on Hops Hadoop. In Hops Hadoop, we have replaced Kerberos with TLS/SSL certificates everywhere. It massively reduces the operational complexity (Certificates are managed by a new service called Hopsworks that also provides UI to Spark/Tensorflow/Hadoop) with about 7-10% overhead for I/O (maximal b/w when reading/writing a file in HDFS is reduced). We will be launching a full release the end of August on www.hops.io.
- lars_francke 9y ago
  Yes, Kerberos seems complex but in practice it's not the technology being a problem but usually organizational problems. But one big benefit that Kerberos has over certificates is that almost every big company has a central Active Directory and almost every user already has Kerberos tickets. The same is not true for certificates which (in my subjective experience from securing tens of clusters) always cause issues. I wonder what you mean by "operational complexity" considering that once you have the principals created there's basically no overhead.... Or to put it differently: Can you elaborate on which problem you're trying to solve?
  - smegel 9y ago
    > The same is not true for certificates which (in my subjective experience from securing tens of clusters) always cause issues. Yeah, getting a new client certificate issued for a system account is like trying to squeeze blood out of stone.
    - jamesblonde 9y ago
      We ship the certificates out transparently to users using Yarn. Yarn keeps them as private local resources and cleans them up for us.
  - jamesblonde 9y ago
    Actually, we use LDAP. If you have an LDAP account, you're authenticated. The use of the certs is transparent to clients - it's wrapped in library calls. The problem we're addressing is multi-tenancy for users on Hadoop. If you have sensitive data and want to restrict access to it, you need at a minimum dynamic roles or attribute-based access controls. With Kerberos, you have a single ID, a single role. You can't revoke privileges from users, give him/her access to a dataset, then restore privileges. We introduced dynamic roles by introducing a new abstraction - a project. You get a project-specific username - and a new cert for every project-specific username. Unlike Kerberos with the centralized KDC, cert generation can be scaled out with intermediate servers. This way, you can give a project-specific user access to sensitive data in a project - and the user can't export the data, as that user only has privileges for resources within the project. You can still share datasets across projects using either ACLs or group privileges in HDFS.
    - puzzle 9y ago
      Rigidity when it comes to tenancy and lack of groups were two of the reasons why Google went with its own LOAS instead of Kerberos for the production environment. The even bigger one was performance.
      - jamesblonde 9y ago
deleted 9y ago
[deleted]
jpk 9y ago
> Sometimes it is OK if you run a cluster insecure. Sure, but put it behind a VPN or something. This article is literally about clusters accessible via the public Internet. If there's a legitimate, non-trivial use case in which that's ever okay, I'm curious to know them.
- taneq 9y ago
  > If there's a legitimate, non-trivial use case in which that's ever okay, I'm curious to know them. Honeypot? :P
- pweissbrod 9y ago
  I dont think this article is about clusters accessible via public internet. I think this article is about clusters accessible after breaking through an easily defeated ssh barrier and gaining full access.
  - jpk 9y ago
    Linked from the article: https://www.shodan.io/search?query=NODATA4U_SECUREYOURSHIT https://www.shodan.io/search?query=NODATA4U_SECUREYOURSHIT That's a list of hadoop nodes you can access via the public Internet.

jamesblonde 9y ago

It is changing. Disclosure- I work on Hops Hadoop. In Hops Hadoop, we have replaced Kerberos with TLS/SSL certificates everywhere. It massively reduces the operational complexity (Certificates are managed by a new service called Hopsworks that also provides UI to Spark/Tensorflow/Hadoop) with about 7-10% overhead for I/O (maximal b/w when reading/writing a file in HDFS is reduced). We will be launching a full release the end of August on www.hops.io.

lars_francke 9y ago
Yes, Kerberos seems complex but in practice it's not the technology being a problem but usually organizational problems. But one big benefit that Kerberos has over certificates is that almost every big company has a central Active Directory and almost every user already has Kerberos tickets. The same is not true for certificates which (in my subjective experience from securing tens of clusters) always cause issues. I wonder what you mean by "operational complexity" considering that once you have the principals created there's basically no overhead.... Or to put it differently: Can you elaborate on which problem you're trying to solve?
- smegel 9y ago
  > The same is not true for certificates which (in my subjective experience from securing tens of clusters) always cause issues. Yeah, getting a new client certificate issued for a system account is like trying to squeeze blood out of stone.
  - jamesblonde 9y ago
    We ship the certificates out transparently to users using Yarn. Yarn keeps them as private local resources and cleans them up for us.
- jamesblonde 9y ago
  Actually, we use LDAP. If you have an LDAP account, you're authenticated. The use of the certs is transparent to clients - it's wrapped in library calls. The problem we're addressing is multi-tenancy for users on Hadoop. If you have sensitive data and want to restrict access to it, you need at a minimum dynamic roles or attribute-based access controls. With Kerberos, you have a single ID, a single role. You can't revoke privileges from users, give him/her access to a dataset, then restore privileges. We introduced dynamic roles by introducing a new abstraction - a project. You get a project-specific username - and a new cert for every project-specific username. Unlike Kerberos with the centralized KDC, cert generation can be scaled out with intermediate servers. This way, you can give a project-specific user access to sensitive data in a project - and the user can't export the data, as that user only has privileges for resources within the project. You can still share datasets across projects using either ACLs or group privileges in HDFS.
  - puzzle 9y ago
    Rigidity when it comes to tenancy and lack of groups were two of the reasons why Google went with its own LOAS instead of Kerberos for the production environment. The even bigger one was performance.
    - jamesblonde 9y ago

deleted 9y ago

[deleted]

jpk 9y ago

> Sometimes it is OK if you run a cluster insecure. Sure, but put it behind a VPN or something. This article is literally about clusters accessible via the public Internet. If there's a legitimate, non-trivial use case in which that's ever okay, I'm curious to know them.

taneq 9y ago
> If there's a legitimate, non-trivial use case in which that's ever okay, I'm curious to know them. Honeypot? :P
pweissbrod 9y ago
I dont think this article is about clusters accessible via public internet. I think this article is about clusters accessible after breaking through an easily defeated ssh barrier and gaining full access.
- jpk 9y ago
  Linked from the article: https://www.shodan.io/search?query=NODATA4U_SECUREYOURSHIT https://www.shodan.io/search?query=NODATA4U_SECUREYOURSHIT That's a list of hadoop nodes you can access via the public Internet.