6 ms·

We've been using Keylocak in production as a multi-tenant SSO solution for our service delivery. We've been incredibly impressed with the stability and performa

We've been using Keylocak in production as a multi-tenant SSO solution for our service delivery. We've been incredibly impressed with the stability and performance and found it extremely effective. Keycloak is the upstream project of Red Hat SSO (edit: correct name, thanks snuxoll.) Running in Kubernetes with RDS Postgres in AWS.

toomuchtodo 6y ago
Any pitfalls you’ve encountered when implementing?
- snuxoll 6y ago
  It’s highly integrated with Wildfly (or JBoss EAP for the commercial product), so if you’re not deploying it with the Docker images expect to have fun dealing with the special hell that is Java application servers - setting up infispan and configuring the database in JNDI at a minimum will require some moderate reading. If you do use the Docker images it’s pretty straightforward though. Past that, customization could be better - not because it doesn’t support it but because many of the SPI’s are poorly documented at best, or totally undocumented at worst. You’ll need to read the code and understand Java EE to do anything not supported out of the box, which, to be fair is a lot - but I’m having to spend far more time looking through code than I’d like to add a Steam login for PCGamingWiki, as an example. Thankfully I’ve dabbled with Java EE before so it’s no big deal to me, but something to consider if you wanna do something simple like add extra profile fields. EDIT: one major nag I have is that the LDAP integration has an annoying bug related to renaming of users. Keycloak can be configured to use a GUID in an LDAP store as the link between a Keycloak user profile and an LDAP object, but when the username changes it will delete and recreate the account instead of updating the username. This creates a whole new sub identifier in the JWT assertions, which has caused me headaches. I have a bug on file for this which just recently got updated targeting a fix in 10.0, so hopefully this gets fixed soon.
  - bebop 6y ago
    I would agree that a pain point is the lack of documentation, examples, and googleability of the SPI's. I have spent much longer than I would have expected integrating an existing user database.
  - thinkharderdev 6y ago
    Agree on the lack of documentation. But one thing I've found really nice is that the source code is really well structured and readable. Every time the documentation has let me down I've been able to find what I needed by reading the source code. That's one of the great underrated advantages of using open source solutions, you're not completely hamstrung when the docs don't give you what you need.
  - cybrix12 6y ago
    It looks like Quarkus is going to be considered for one of the next major release [1]. That said, do you believe it will still be possible to extend Keycloak using the deployment-scanner? Also, do you happen to have open-source code related to Keycloak and/or custom extensions? Beside the poor doc, finding more open-source code is one of the best way to learn this. [1]: https://issues.redhat.com/browse/KEYCLOAK-13068?jql=project%20%3D%20KEYCLOAK%20AND%20component%20%3D%20%22Distribution%20-%20Quarkus%22 https://issues.redhat.com/browse/KEYCLOAK-13068?jql=project%...
    - tofflos 6y ago
      Some more scraps of information regarding Keycloak on Quarkus is available at https://www.keycloak.org/2019/10/keycloak-x https://www.keycloak.org/2019/10/keycloak-x. I'm hoping this will lead to significantly faster startup times so that my integration tests will run faster.
    - folmar 6y ago
      > do you believe it will still be possible to extend Keycloak using the deployment-scanner? Even if it wouldn't bundling your keycloak-with-amenities is a 10-minutes job (1. make a pom.xml with keycloak dependency and your stuff 2. mvn package 4. there is no step 3)
- realdavidops 6y ago
  The biggest thing we encountered was related actually to our initial deployment with active directory. This made logins slow, but actually found we could remove the requirement for Active Directory. It is super heavily based on Wildfly, and if you're not using a tool like docker, it can be kind-of a burden. It runs decently well in standalone mode, but we ended up using the docker container's clustering with Kubernetes service discovery helping to find the other nodes to achieve a clustered deployment. Outside of that is has been extremely stable, we use Kubernetes deployment mechanism along with a correctly defined readiness check to allow us to seamlessly upgrade, and we've gone from 4.3.0.Final to 7.0.1 in production without any problems. We haven't upgraded to 8 or 9 yet as we're actually working on some new frontend UI changes we wanted to get out the door with the release.
  - thinkharderdev 6y ago
    I'm curious how you manage upgrades. I am in the process of rolling Keycloak out to production now and the only thing I don't quite grok is how to do zero-downtime upgrades. It seems like the upgrade may make backwards-incompatible changes the DB schema. Do you replicate the entire database for the upgraded environment?
    - cybrix12 6y ago
      When using a K8s cluster with the helm chart [1], it's actually the stateful set that takes care to the update. When the first replica restart, Keycloak makes the updates to the database itself. Sometimes rolling back to a previous version can break. They do not hold the reverse of the database version [2]. I believe the reason behind the STS (StatefulSet) is so the cache have the time to spread among the replicas as it get upgraded. [1]: https://github.com/codecentric/helm-charts/tree/master/charts/keycloak https://github.com/codecentric/helm-charts/tree/master/chart... [2]: https://www.keycloak.org/docs/9.0/upgrading/ https://www.keycloak.org/docs/9.0/upgrading/
    - realdavidops 6y ago
      We schedule a downtime window during upgrade, but typically see no frontend impact to the core service, SSO for end users. We snapshot the DB for rollback if needed as the migrations are not reversible. Our actual DB size is pretty small so these are very non-intensive tasks.
snuxoll 6y ago
Upstream of Red Hat SSO, Red Hat IdM is the commercial product based on FreeIPA.
- realdavidops 6y ago
  Oof! Good catch! Yes Red Hat SSO
B3NE 6y ago
Did you build any custom extensions for Keycloak by implementing Keycloak's Service Provider Interfaces? If you are running any custom extensions, what features did you have to add?
closeparen 6y ago
I've only played with it, but was kind of put off by how much of the 2FA credential management is only available to admins. It's not like Duo where you can update your own enrolled phones, U2F devices, and defaults. End users would have to ask admins to do all that for them.
- realdavidops 6y ago
  Hmm. I'm not sure what you mean. Users by default can use the console to update their 2FA credentials. The only time I have to intervene is when they lose their 2FA as it doesn't really do backup codes. We do require 2FA as a part of our login flows so this is something we're using heavily.
  - closeparen 6y ago
    It may be better for TOTP; I was looking at U2F and WebAuthn.
    - tialaramex 6y ago
      What would admin enrollment even look like for WebAuthn? Do I need to FedEx my FIDO security keys to the company IT security department? I can't imagine any scenario in which you have FIDO keys and admin enrollment and security but I'm prepared to be enlightened.
      - closeparen 6y ago
        You can assign the user a temporary password so that they get prompted to enroll their credential on first login. But: a) Because the password is assigned first, it has higher priority, so subsequent logins will prompt for password first until the admin manually changes the user's credential ordering to put the WebAuthn (passwordless) token higher. The user's credential priority overrides the order of challenges in the login flow. b) There is no option to add or replace one of these credentials, or manage credential ordering yourself, in the end-user webapp that does profile editing / password updates. An admin may be able to reset your account so that you get the first-login experience again and can enroll new credentials.

toomuchtodo 6y ago

Any pitfalls you’ve encountered when implementing?

snuxoll 6y ago
It’s highly integrated with Wildfly (or JBoss EAP for the commercial product), so if you’re not deploying it with the Docker images expect to have fun dealing with the special hell that is Java application servers - setting up infispan and configuring the database in JNDI at a minimum will require some moderate reading. If you do use the Docker images it’s pretty straightforward though. Past that, customization could be better - not because it doesn’t support it but because many of the SPI’s are poorly documented at best, or totally undocumented at worst. You’ll need to read the code and understand Java EE to do anything not supported out of the box, which, to be fair is a lot - but I’m having to spend far more time looking through code than I’d like to add a Steam login for PCGamingWiki, as an example. Thankfully I’ve dabbled with Java EE before so it’s no big deal to me, but something to consider if you wanna do something simple like add extra profile fields. EDIT: one major nag I have is that the LDAP integration has an annoying bug related to renaming of users. Keycloak can be configured to use a GUID in an LDAP store as the link between a Keycloak user profile and an LDAP object, but when the username changes it will delete and recreate the account instead of updating the username. This creates a whole new sub identifier in the JWT assertions, which has caused me headaches. I have a bug on file for this which just recently got updated targeting a fix in 10.0, so hopefully this gets fixed soon.
- bebop 6y ago
  I would agree that a pain point is the lack of documentation, examples, and googleability of the SPI's. I have spent much longer than I would have expected integrating an existing user database.
- thinkharderdev 6y ago
  Agree on the lack of documentation. But one thing I've found really nice is that the source code is really well structured and readable. Every time the documentation has let me down I've been able to find what I needed by reading the source code. That's one of the great underrated advantages of using open source solutions, you're not completely hamstrung when the docs don't give you what you need.
- cybrix12 6y ago
  It looks like Quarkus is going to be considered for one of the next major release [1]. That said, do you believe it will still be possible to extend Keycloak using the deployment-scanner? Also, do you happen to have open-source code related to Keycloak and/or custom extensions? Beside the poor doc, finding more open-source code is one of the best way to learn this. [1]: https://issues.redhat.com/browse/KEYCLOAK-13068?jql=project%20%3D%20KEYCLOAK%20AND%20component%20%3D%20%22Distribution%20-%20Quarkus%22 https://issues.redhat.com/browse/KEYCLOAK-13068?jql=project%...
  - tofflos 6y ago
    Some more scraps of information regarding Keycloak on Quarkus is available at https://www.keycloak.org/2019/10/keycloak-x https://www.keycloak.org/2019/10/keycloak-x. I'm hoping this will lead to significantly faster startup times so that my integration tests will run faster.
  - folmar 6y ago
    > do you believe it will still be possible to extend Keycloak using the deployment-scanner? Even if it wouldn't bundling your keycloak-with-amenities is a 10-minutes job (1. make a pom.xml with keycloak dependency and your stuff 2. mvn package 4. there is no step 3)
realdavidops 6y ago
The biggest thing we encountered was related actually to our initial deployment with active directory. This made logins slow, but actually found we could remove the requirement for Active Directory. It is super heavily based on Wildfly, and if you're not using a tool like docker, it can be kind-of a burden. It runs decently well in standalone mode, but we ended up using the docker container's clustering with Kubernetes service discovery helping to find the other nodes to achieve a clustered deployment. Outside of that is has been extremely stable, we use Kubernetes deployment mechanism along with a correctly defined readiness check to allow us to seamlessly upgrade, and we've gone from 4.3.0.Final to 7.0.1 in production without any problems. We haven't upgraded to 8 or 9 yet as we're actually working on some new frontend UI changes we wanted to get out the door with the release.
- thinkharderdev 6y ago
  I'm curious how you manage upgrades. I am in the process of rolling Keycloak out to production now and the only thing I don't quite grok is how to do zero-downtime upgrades. It seems like the upgrade may make backwards-incompatible changes the DB schema. Do you replicate the entire database for the upgraded environment?
  - cybrix12 6y ago
    When using a K8s cluster with the helm chart [1], it's actually the stateful set that takes care to the update. When the first replica restart, Keycloak makes the updates to the database itself. Sometimes rolling back to a previous version can break. They do not hold the reverse of the database version [2]. I believe the reason behind the STS (StatefulSet) is so the cache have the time to spread among the replicas as it get upgraded. [1]: https://github.com/codecentric/helm-charts/tree/master/charts/keycloak https://github.com/codecentric/helm-charts/tree/master/chart... [2]: https://www.keycloak.org/docs/9.0/upgrading/ https://www.keycloak.org/docs/9.0/upgrading/
  - realdavidops 6y ago
    We schedule a downtime window during upgrade, but typically see no frontend impact to the core service, SSO for end users. We snapshot the DB for rollback if needed as the migrations are not reversible. Our actual DB size is pretty small so these are very non-intensive tasks.

snuxoll 6y ago

Upstream of Red Hat SSO, Red Hat IdM is the commercial product based on FreeIPA.

realdavidops 6y ago
Oof! Good catch! Yes Red Hat SSO

B3NE 6y ago

Did you build any custom extensions for Keycloak by implementing Keycloak's Service Provider Interfaces? If you are running any custom extensions, what features did you have to add?

closeparen 6y ago

I've only played with it, but was kind of put off by how much of the 2FA credential management is only available to admins. It's not like Duo where you can update your own enrolled phones, U2F devices, and defaults. End users would have to ask admins to do all that for them.

realdavidops 6y ago
Hmm. I'm not sure what you mean. Users by default can use the console to update their 2FA credentials. The only time I have to intervene is when they lose their 2FA as it doesn't really do backup codes. We do require 2FA as a part of our login flows so this is something we're using heavily.
- closeparen 6y ago
  It may be better for TOTP; I was looking at U2F and WebAuthn.
  - tialaramex 6y ago
    What would admin enrollment even look like for WebAuthn? Do I need to FedEx my FIDO security keys to the company IT security department? I can't imagine any scenario in which you have FIDO keys and admin enrollment and security but I'm prepared to be enlightened.
    - closeparen 6y ago
      You can assign the user a temporary password so that they get prompted to enroll their credential on first login. But: a) Because the password is assigned first, it has higher priority, so subsequent logins will prompt for password first until the admin manually changes the user's credential ordering to put the WebAuthn (passwordless) token higher. The user's credential priority overrides the order of challenges in the login flow. b) There is no option to add or replace one of these credentials, or manage credential ordering yourself, in the end-user webapp that does profile editing / password updates. An admin may be able to reset your account so that you get the first-login experience again and can enroll new credentials.