WM IAM deployment and provisioning

Deploys and provisions the central WM Keycloak IAM to acceptance and production.

Special notes on this open source release

This project is provided as an open source copy of our custom helmfile deployment. It expects some middleware on top of plain kubernetes to work. This is up to the implementer to address.

• We rely on the external secrets operator and we use Hashicorp Vault as an external backing for these secrets. OpenBao may work too, we haven't tested this.
• We use a custom TLS trust bundle so that the internal PKI signed certificates work ok.
• We rely on the 'rendered manifest' pattern. The rendering is done as a pipeline step and normally committed back to this repo in the path rendered/ . It was removed from this repo as it contains the actual configuration and is automaticly generated from the other configuration in this repository. The rendering stage is included here for reference:

.store-rendered-manifests:
  extends: .withToolingImage
  variables:
    HELMFILE: "helmfile.yaml"
    RENDERED_CHART_DIR: "rendered"
    CHART_DIR_TEMPLATE: "{{ .Release.Name }}"
  script:
    - helm registry login -u $HARBOR_CICD_USER -p $HARBOR_CICD_PASSWORD $HARBOR_URL
    - mkdir -p ${CI_PROJECT_DIR}/${RENDERED_CHART_DIR}/ || true
    - find ${CI_PROJECT_DIR}/${RENDERED_CHART_DIR}/ -name "*.yaml" -delete
    - helmfile template -f ${HELMFILE} --output-dir-template "${CI_PROJECT_DIR}/${RENDERED_CHART_DIR}/${CHART_DIR_TEMPLATE}"
    - gitlab-push-mr.sh
  rules:
    - if: $CI_COMMIT_TAG
      when: never
    - if: $CI_COMMIT_BRANCH

Questions regarding this code can be sent to et@rws.nl which we will answer on a best effort basis.

Resources

All off the resources are managed as ArgoCD applications, seperated by environment.

Code for that is in the argocd folder.

High level design (HLD)

HLD including only keycloak

HLD with keycloak and a single external IDP in scope

Proces flow authentication using an external IDP (Waterschapshuis in this example):

Deployment

• Keycloak instances (acceptance and production). Values for the instances charts are in the /values/keycloak folder.
• Deployment is automated by deploying the keycloak operator and providing it a Keycloaks custom resource
• Currently, the acceptance instances are deployed on both wm-tab-dc1 and wm-tab-dc2 clusters. We have external and internal global loadbalancers in front of them. These instances are deployed in an Active-Active configuration.
  • NOTE: Both acceptance instances use a single database, that is deployed in one datacenter. In the future, once we have synchronously replicated Postgres, we will have a fully DC-fault tolerant setup.
  • We only deploy the provisioning resources on AM4 (wm-tab-dc2 cluster).

Provisioning

The provisioning process is done in a GitOps way. This means that if we need to add a client to a realm, we'd add the client representation to the realm values file.

The value files are located in the /values/provisioning folder. There are subfolders for acceptance and production.

The provisioning is done by deploying Crossplane resources.

• Crossplane Provider for Keycloak
• Crossplane ProviderConfig for both acceptance and production instances
• Realm chart that deploys
  • A Crossplane Realm
  • A Crossplane IDP for that realm
  • A number of Crossplance Clients to be added for that realm

Each realm is a separate Helm release.

Usage

Adding a new realm

To add a new realm, create a new values file for that realm in the provisioning/production/realms or the provisioning/acceptance/realms directory.

This file would contain the realm description, the ADFS IDP configuration and the clients to be added to that realm.

Then, add the realm deployment to the Helmfile.

For example, to add the new-realm realm to the acceptance environment, you would add new-realm.yaml to values/provisioning/acceptance/realms/ directory and then add the following to the helmfile.yaml file:

  - inherit: [template: "xkeycloakrealm"]
    name: wm-iam-realm-new-realm-acc-dc2
    namespace: acc-example-org-iam
    values:
      - values/provisioning/acceptance/realms/new-realm.yaml

Adding a client to a realm

Adding a client to an existing realm is done by adding a new client configuration to the clients section of the realm values file.

For example, to add my-new-client the new-realm realm in acceptance environment, you would add the following to the /values/provisioning/acceptance/new-realm.yaml file:

...
clients:
  - name: my-new-client
    adfsRolePrefix: MyNewClient
    config:
      accessType: CONFIDENTIAL
      name: My New Client
      description: My New Client
      implicitFlowEnabled: true
      rootUrl: "https://my-new-client.app.cluster.dc2.internal.example.org/"
      validRedirectUris:
        - "https://my-new-client.app.cluster.dc2.internal.example.org/*"
...

Entry ID IDP setup using SAML

When creating a SAML based setup with Entra ID, some attribute mappings are required. For Waterschapshuis IDP we mapped the following Entra ID attributes to keycloak attributes:

Entra ID	Keycloak attribute
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname	firstName
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname	lastName
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress	email
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name	username

(Re)setting client secrets for clients of the type CONFIDENTIAL

When the realm is recreated, the secrets for confidential clients are regenerated. Updating the applications using these clients is an option. You then need to take the new client secret for the particular client from Keycloak and update the corrosponding secret in Vault. The other option is taking the old secret from Vault and overriding this value for the particular client in keycloak, using the keycloak CLI.

Sample cli usage

Please refer to current keycloak documentation for details:

• get a shell in the running Keycloak container

/opt/keycloak/bin/kcadm.sh config credentials \
  --server http://localhost:8080 \
  --realm master \
  --user admin \
  --password adminpassword
  
/opt/keycloak/bin/kcadm.sh help

/opt/keycloak/bin/kcadm.sh get clients -r [REALM] --fields id,clientId
[ {
  "id" : "[VERY LONG STRING]",
  "clientId" : "example"
}, ...

 /opt/keycloak/bin/kcadm.sh update clients/[CLIENT ID] -r [REALM NAME] -s secret=[SECRET KEY]

list of CONFIDENTIAL clients

The best source of information is the values/provisioning/acceptance/realms/* yaml files under the section clients. Each listed as type accessType: CONFIDENTIAL should be taken into account.

Finding the Vault secret for a particular application might be trivial, otherwise contact the particular TAB sub team for guidance.

Developing new features on a feature branch

With our pipeline code it is possible to deploy values to just acceptance. Please note that updates to chart versions may effect production too, even when running from a feature branch. So take care to keep them backwards compatible or accept out of sync resources.