Deploying Keycloak HA with ArgoCD, SOPS, and PgBouncer

Nov 4, 2025 · 6 min read

This post documents the deployment of a production-ready High Availability Keycloak setup using GitOps principles with ArgoCD, encrypted secrets with SOPS, and connection pooling with PgBouncer. This deployment has been made using Claude under human supervision, as part of my personal training in using AI.

Architecture Overview

The deployment consists of:

PostgreSQL Cluster: 2 replicas managed by Zalando Operator
PgBouncer: 4 instances (2 primary + 2 replica) for connection pooling
Keycloak: 3 replicas with JGroups clustering
SOPS: GPG-encrypted secrets with KSOPS plugin for ArgoCD
ArgoCD: GitOps deployment with automated sync

Namespace Organization

database namespace: PostgreSQL cluster and poolers
keycloak namespace: Keycloak application

AI Agent Prompt for One-Step Deployment

If you want to reproduce this setup using an AI agent, use the following prompt:

I have a Kubernetes cluster with:
- ArgoCD installed with SOPS/KSOPS support (GPG key in sops-gpg secret)
- Zalando PostgreSQL Operator installed
- Cert-manager with a ClusterIssuer named "rancher-subca"
- Nginx Ingress Controller
- Kubeconfig available at ~/.kube/<cluster-name>.yaml

Please do the following:

1. Create a database cluster in the "database" namespace with:
   - 2 PostgreSQL replicas (Zalando operator)
   - PgBouncer connection pooling enabled (2 instances)
   - Session mode for PgBouncer (not transaction mode)
   - Create a database named "keycloak" with user "keycloak"

2. Create a Keycloak installation with:
   - High availability enabled (3 replicas)
   - Deployed in "keycloak" namespace
   - JGroups clustering configured
   - Hostname: keycloak.<domain>.local
   - Connected to database via PgBouncer

3. Use SOPS to manage all database credentials:
   - Encrypt secrets with the existing GPG key
   - Use KSOPS plugin pattern (kustomization.yaml with generators)
   - Store secrets in both database and keycloak namespaces

4. Use ArgoCD for deployment:
   - Create two separate Applications (one per namespace)
   - Repository: https://github.com/<user>/keycloak-argocd.git
   - Enable automated sync with prune and selfHeal
   - Use HTTPS for repository access

5. Create an AI_INSTRUCTIONS.md file documenting:
   - How to use ArgoCD for all deployments
   - SOPS encryption workflow with KSOPS
   - Requirement to always use PgBouncer for database connections

Important: Separate resources by namespace directories. Use kustomization.yaml files with secret generators for SOPS integration.

Key Implementation Details

1. PostgreSQL with PgBouncer

The PostgreSQL cluster configuration includes built-in PgBouncer support:

apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
  name: keycloak-postgres
  namespace: database
spec:
  enableConnectionPooler: true
  enableReplicaConnectionPooler: true
  connectionPooler:
    numberOfInstances: 2
    mode: "session"  # Critical: session mode required
    maxDBConnections: 60

Pitfall #1: PgBouncer Transaction Mode

Initially configured with mode: "transaction", which caused Keycloak to fail with:

ERROR: prepared statement "S_1" already exists

Solution: Change to mode: "session". Keycloak (and Liquibase) use prepared statements extensively, which are not supported in transaction pooling mode. Session mode provides full SQL compatibility while still offering connection pooling benefits.

2. SOPS Integration with ArgoCD

Secrets are encrypted with SOPS and automatically decrypted by ArgoCD using the KSOPS plugin.

Pitfall #2: SOPS Secrets Not Decrypting

Initially, ArgoCD applied encrypted secrets directly to the cluster, causing Keycloak pods to receive values like ENC[AES256_GCM,data:...] instead of actual credentials.

Solution: Use the KSOPS plugin pattern with Kustomize:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml

generators:
  - secret-generator.yaml

# secret-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops      
files:
  - encrypted-secret.yaml

Important: Never list encrypted secrets in resources:. They MUST go through generators: to be decrypted.

3. Keycloak Configuration

Pitfall #3: Missing Hostname Configuration

Keycloak 23+ requires hostname configuration in production mode:

ERROR: Strict hostname resolution configured but no hostname setting provided

Solution: Add these environment variables:

env:
  - name: KC_HOSTNAME
    value: keycloak.<domain>.local
  - name: KC_HOSTNAME_STRICT
    value: "false"
  - name: KC_HTTP_ENABLED
    value: "true"

4. Namespace Separation

Resources are organized by namespace with separate ArgoCD Applications:

# Application for database namespace
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: keycloak-database
spec:
  source:
    path: database
  destination:
    namespace: database

---
# Application for keycloak namespace
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: keycloak
spec:
  source:
    path: keycloak
  destination:
    namespace: keycloak

This allows independent management and easier RBAC configuration.

5. Database Credentials Across Namespaces

Since Keycloak (in keycloak namespace) needs to access PostgreSQL (in database namespace), credentials are stored in both namespaces:

database/keycloak-db-secret.yaml - Original secret
keycloak/keycloak-db-secret.yaml - Copy for app access

Both are SOPS-encrypted and use the PgBouncer service endpoint:

stringData:
  DB_ADDR: keycloak-postgres-pooler.database.svc.cluster.local
  JDBC_URL: jdbc:postgresql://keycloak-postgres-pooler.database.svc.cluster.local:5432/keycloak

Repository Structure

.
├── .sops.yaml                    # SOPS configuration
├── database/
│   ├── kustomization.yaml
│   ├── secret-generator.yaml
│   ├── postgres-cluster.yaml
│   └── keycloak-db-secret.yaml   # SOPS-encrypted
├── keycloak/
│   ├── kustomization.yaml
│   ├── secret-generator.yaml
│   ├── keycloak-db-secret.yaml   # SOPS-encrypted
│   ├── keycloak-deployment.yaml
│   └── keycloak-ingress.yaml
├── argocd-application.yaml
├── AI_INSTRUCTIONS.md
└── README.md

Verification

After deployment, verify everything is working:

# Check ArgoCD applications
kubectl get application -n argocd
# Should show both apps as Synced and Healthy

# Check PostgreSQL
kubectl get postgresql -n database
kubectl get pods -n database -l application=spilo

# Check PgBouncer
kubectl get pods -n database -l connection-pooler=keycloak-postgres-pooler
# Should show 4 pods total (2 primary + 2 replica)

# Check Keycloak
kubectl get pods -n keycloak
# Should show 3 ready pods

# Verify secrets are decrypted
kubectl get secret keycloak-db-credentials -n keycloak \
  -o jsonpath='{.data.DB_VENDOR}' | base64 -d
# Should output: postgres (not ENC[...])

Key Lessons Learned

PgBouncer Mode Matters: Session mode is required for applications using prepared statements. Transaction mode is more efficient but has compatibility limitations.
KSOPS Pattern: Encrypted secrets must use the generators: section in kustomization.yaml, not resources:. This was not immediately obvious from documentation.
Keycloak Hostname: Modern Keycloak versions enforce hostname configuration in production mode. This can be relaxed with KC_HOSTNAME_STRICT: false for internal deployments.
Cross-Namespace Access: Services can be accessed across namespaces using FQDN: <service>.<namespace>.svc.cluster.local. Credentials need to be present in the consuming namespace.
GitOps Workflow: Never apply manifests directly. Always commit to Git and let ArgoCD sync. This maintains audit trail and enables easy rollbacks.

Benefits of This Architecture

High Availability: All components have multiple replicas with automatic failover
Connection Pooling: PgBouncer reduces database connections and improves performance
Security: Secrets encrypted at rest in Git repository
GitOps: Declarative configuration with automated deployment
Scalability: Easy to scale Keycloak replicas independently
Maintainability: Clear namespace separation and well-documented patterns

Conclusion

This setup provides a production-ready Keycloak deployment following GitOps best practices. The combination of ArgoCD, SOPS, and PgBouncer creates a secure, scalable, and maintainable infrastructure.

The AI_INSTRUCTIONS.md file in the repository ensures consistency for future modifications and helps other agents understand the architectural decisions and required patterns.

As for me, I’m still surprised about how helpful can be AI assistants on my daily work, not only by saving lots of hours of googling, but also for allowing me to understand the steps needed to perform each task. In this case I think this deployment took me less than a quarter of the time that it would had taken without using AI.

Repository

Full configuration available at: https://github.com/juanjo-vlc/keycloak-argocd