ArgoCD with SOPS Support for Secret Management

· 8 min read

This post explains how to deploy ArgoCD with SOPS support using KSOPS plugin, enabling GitOps workflows with encrypted secrets managed via GPG keys. This has been done with extensive support from Claude, also this post was written by it in an attempt to incorporate AI use to my skills.

Motivation

When implementing GitOps practices, one of the biggest challenges is managing secrets. You want to store everything in Git, including secrets, but you can’t commit them in plain text. SOPS (Secrets OPerationS) solves this by encrypting secrets using GPG or age keys, while keeping the file structure readable.

ArgoCD is a declarative GitOps continuous delivery tool for Kubernetes. Combining ArgoCD with SOPS allows you to:

  • Store encrypted secrets safely in Git repositories
  • Maintain full GitOps workflow without exposing sensitive data
  • Enable team collaboration with per-user GPG key access
  • Audit secret changes through Git history
  • Automatically decrypt and deploy secrets in the cluster

This integration is particularly useful for on-premises deployments where you need to manage secrets across multiple environments while maintaining security and auditability.

Prerequisites

Before starting, you’ll need:

  • A running Kubernetes cluster (K3s, RKE2, or any other distribution)
  • Helm 3 installed
  • kubectl configured to access your cluster
  • GPG key pair for encrypting/decrypting secrets
  • SOPS CLI installed locally for testing

This is intended as a continuation from my previous post Rancher on K3s so the prerequisites are covered by it.

Architecture Overview

The solution consists of:

  1. ArgoCD - GitOps continuous delivery tool
  2. SOPS - Encryption tool for secrets
  3. KSOPS - Kustomize plugin that decrypts SOPS-encrypted files
  4. GPG - Encryption key management

When ArgoCD syncs an application:

  1. ArgoCD’s repo-server clones the Git repository
  2. Kustomize builds the manifests
  3. KSOPS plugin detects SOPS-encrypted files
  4. SOPS decrypts the files using the GPG private key
  5. Decrypted manifests are applied to the cluster

Generating GPG Keys

First, generate a GPG key pair dedicated to Kubernetes secret encryption:

# Generate a new GPG key
gpg --full-generate-key

# Choose:
# - RSA and RSA (default)
# - 3072 bits
# - Key expiration (e.g., 3 years)
# - Real name: anthrax.garmo.local (k8s)
# - Email: anthrax@garmo.local

# List your keys to get the fingerprint
gpg --list-secret-keys --keyid-format LONG

# Export the private key (you'll need this for ArgoCD)
gpg --export-secret-keys --armor YOUR_KEY_ID > gpg-private-key.asc

# Export the public key (share with team members)
gpg --export --armor YOUR_KEY_ID > gpg-public-key.asc

Installing ArgoCD with SOPS Support

I’ve created an Ansible role that automates the ArgoCD installation with SOPS support. The role:

  1. Creates the ArgoCD namespace
  2. Installs ArgoCD using Helm
  3. Configures an init container to install KSOPS and SOPS
  4. Sets up the KSOPS plugin in the correct directory structure
  5. Imports the GPG private key
  6. Configures environment variables for GPG home directory

Create GPG Secret in ArgoCD Namespace

Before deploying ArgoCD, create a secret with your GPG private key:

kubectl create namespace argocd

kubectl create secret generic sops-gpg \
  --from-file=sops.asc=gpg-private-key.asc \
  -n argocd

Ansible Role for ArgoCD Installation

The key parts of the Ansible role configuration:

- name: Install/Upgrade ArgoCD
  kubernetes.core.helm:
    kubeconfig: "{{ kubeconfig }}"
    name: argocd
    chart_ref: argo/argo-cd
    chart_version: "{{ argocd_chart_version }}"
    namespace: argocd
    values:
      global:
        domain: "{{ argocd_host }}"
      configs:
        cm:
          kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"
        params:
          server.insecure: "true"
      server:
        ingress:
          enabled: true
          annotations:
            kubernetes.io/ingress.class: nginx
            cert-manager.io/cluster-issuer: "{{ certmanager_cluster_issuer_name }}"
            nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
            nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
          tls: true
      repoServer:
        env:
          - name: GNUPGHOME
            value: /sops-gpg/.gnupg
          - name: XDG_CONFIG_HOME
            value: /kustomize-plugin-home
        volumes:
          - name: custom-tools
            emptyDir: {}
          - name: sops-gpg
            secret:
              secretName: sops-gpg
              defaultMode: 0400
        initContainers:
        - name: install-ksops
          image: your-registry/install-ksops:4.4.0
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -euo pipefail
              # Install KSOPS and SOPS binaries
              tar -C /custom-tools -xzf /tools/ksops_4.4.0_Linux_x86_64.tar.gz ksops
              chmod +x /custom-tools/ksops
              cp /tools/sops-v3.11.0.linux.amd64 /custom-tools/sops
              chmod +x /custom-tools/sops

              # Setup KSOPS as a kustomize exec plugin
              mkdir -p /custom-tools/kustomize/plugin/viaduct.ai/v1/ksops
              cp /custom-tools/ksops /custom-tools/kustomize/plugin/viaduct.ai/v1/ksops/ksops

              # Import GPG key
              mkdir -p /custom-tools/sops-gpg/.gnupg
              chmod 700 /custom-tools/sops-gpg/.gnupg
              gpg --homedir /custom-tools/sops-gpg/.gnupg --import /sops-gpg-keys/sops.asc

              # Set ultimate trust for the imported key
              KEY_FP=$(gpg --homedir /custom-tools/sops-gpg/.gnupg --list-keys --with-colons | awk -F: '/^fpr:/ {print $10; exit}')
              echo "${KEY_FP}:6:" | gpg --homedir /custom-tools/sops-gpg/.gnupg --import-ownertrust

              # Fix permissions
              chmod -R 600 /custom-tools/sops-gpg/.gnupg/*
              chmod 700 /custom-tools/sops-gpg/.gnupg
              chown -R 999:999 /custom-tools/sops-gpg/
              chown -R 999:999 /custom-tools/kustomize/              
          volumeMounts:
            - mountPath: /custom-tools
              name: custom-tools
            - mountPath: /sops-gpg-keys
              name: sops-gpg
        volumeMounts:
          - mountPath: /usr/local/bin/ksops
            name: custom-tools
            subPath: ksops
          - mountPath: /usr/local/bin/sops
            name: custom-tools
            subPath: sops
          - mountPath: /sops-gpg
            name: custom-tools
            subPath: sops-gpg
          - mountPath: /kustomize-plugin-home/kustomize/plugin
            name: custom-tools
            subPath: kustomize/plugin

Key configuration points:

  • kustomize.buildOptions: Enables alpha plugins and exec mode for KSOPS
  • server.insecure: Runs ArgoCD server in HTTP mode (nginx handles TLS)
  • GNUPGHOME: Points to GPG keyring location
  • XDG_CONFIG_HOME: Tells Kustomize where to find plugins
  • Init container: Installs KSOPS/SOPS and sets up the GPG key

Building the KSOPS Init Container Image

You’ll need to create a container image with KSOPS and SOPS binaries:

FROM alpine:latest

RUN apk add --no-cache curl tar gnupg

WORKDIR /tools

# Download KSOPS
RUN curl -L -o ksops_4.4.0_Linux_x86_64.tar.gz \
    https://github.com/viaduct-ai/kustomize-sops/releases/download/v4.4.0/ksops_4.4.0_Linux_x86_64.tar.gz

# Download SOPS
RUN curl -L -o sops-v3.11.0.linux.amd64 \
    https://github.com/getsops/sops/releases/download/v3.11.0/sops-v3.11.0.linux.amd64

CMD ["sleep", "infinity"]

Build and push to your registry:

docker build -t your-registry/install-ksops:4.4.0 .
docker push your-registry/install-ksops:4.4.0

Creating SOPS Configuration

Create a .sops.yaml file in your Git repository to configure encryption rules:

creation_rules:
  - path_regex: \.yaml$
    encrypted_regex: ^(data|stringData)$
    pgp: YOUR_GPG_KEY_FINGERPRINT

This configuration:

  • Applies to all .yaml files
  • Only encrypts the data and stringData fields
  • Uses your GPG key for encryption

Encrypting Secrets

Create a secret file (e.g., secret.yaml):

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  namespace: default
type: Opaque
stringData:
  username: admin
  password: supersecret123
  api-key: sk-1234567890abcdef

Encrypt it with SOPS:

sops --encrypt secret.yaml > secret.enc.yaml

The encrypted file keeps the structure readable but encrypts the values:

apiVersion: v1
kind: Secret
metadata:
    name: my-secret
    namespace: default
type: Opaque
stringData:
    username: ENC[AES256_GCM,data:JTJrv9c=,iv:DzK0YF...]
    password: ENC[AES256_GCM,data:AaZWC5H/0HO/...]
    api-key: ENC[AES256_GCM,data:xj6Qqj0m8FaN...]
sops:
    pgp:
        - created_at: "2025-11-02T08:14:56Z"
          fp: YOUR_GPG_KEY_FINGERPRINT

Setting Up Kustomize with KSOPS

Create a KSOPS generator file (secret-generator.yaml):

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: sops-secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops      
files:
  - secret.enc.yaml

Create a kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - configmap.yaml
  - deployment.yaml

generators:
  - secret-generator.yaml

Directory structure:

my-app/
├── .sops.yaml
├── kustomization.yaml
├── secret-generator.yaml
├── secret.enc.yaml        # Encrypted
├── configmap.yaml
└── deployment.yaml

Deploying with ArgoCD

Create an ArgoCD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/your-repo.git
    targetRevision: main
    path: my-app
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Apply it:

kubectl apply -f application.yaml

ArgoCD will:

  1. Clone the repository
  2. Run Kustomize build
  3. KSOPS plugin decrypts the SOPS-encrypted files
  4. Deploy the decrypted manifests to the cluster

Managing Secrets

Editing Encrypted Secrets

Use SOPS built-in editor:

sops secret.enc.yaml

SOPS will decrypt, open your editor, and re-encrypt on save.

Adding New Values

# Open in editor
sops secret.enc.yaml

# Add new key under stringData:
# new-credential: my-new-value

# Save and exit - automatically re-encrypted

Team Member Access

Add a team member’s GPG key:

# Update .sops.yaml with their key
# Then rotate the encrypted file
sops updatekeys secret.enc.yaml

# Commit and push
git add .sops.yaml secret.enc.yaml
git commit -m "Add team member GPG key"
git push

SOPS uses envelope encryption - the actual data is encrypted with a data key, which is then encrypted with each GPG key. This allows adding/removing keys without re-encrypting the entire file.

Verification

Verify the setup works:

# Check ArgoCD can list keys
kubectl exec -n argocd deploy/argocd-repo-server -- \
  gpg --list-secret-keys

# Check SOPS is installed
kubectl exec -n argocd deploy/argocd-repo-server -- \
  sops --version

# Check KSOPS plugin location
kubectl exec -n argocd deploy/argocd-repo-server -- \
  ls -la $XDG_CONFIG_HOME/kustomize/plugin/viaduct.ai/v1/ksops/

# Verify secret is decrypted in cluster
kubectl get secret my-secret -o jsonpath='{.data.username}' | base64 -d
# Should output: admin (decrypted!)

Troubleshooting

ArgoCD Shows “MAC Mismatch” Error

This means the encrypted file is corrupted. Re-encrypt:

sops --encrypt secret.yaml > secret.enc.yaml
git add secret.enc.yaml
git commit -m "Re-encrypt secret"
git push

Secret Still Encrypted in Cluster

Check if KSOPS is properly configured:

# Verify plugin directory exists
kubectl exec -n argocd deploy/argocd-repo-server -- \
  ls -la /kustomize-plugin-home/kustomize/plugin/viaduct.ai/v1/ksops/

# Check repo-server logs
kubectl logs -n argocd deploy/argocd-repo-server | grep -i sops

Kustomize Build Fails

Ensure kustomize build options are set:

kubectl get cm argocd-cm -n argocd -o yaml | grep buildOptions
# Should show: --enable-alpha-plugins --enable-exec

Security Best Practices

  1. Never commit unencrypted secrets - Always use .gitignore for plain secret files
  2. Backup GPG keys - Store private keys securely (password manager, vault)
  3. Rotate keys regularly - Use sops rotate to change encryption keys periodically
  4. Audit access - Git history shows who modified secrets and when
  5. Separate keys per environment - Use different GPG keys for dev/staging/production
  6. Limit key access - Only share GPG keys with team members who need access

Conclusion

This setup provides a complete GitOps workflow with encrypted secrets:

  • ✅ All configuration in Git (including secrets)
  • ✅ Secrets encrypted at rest in repository
  • ✅ Team collaboration with individual GPG keys
  • ✅ Automatic decryption and deployment via ArgoCD
  • ✅ Full audit trail through Git history
  • ✅ No plain-text secrets in the cluster or repository

The combination of ArgoCD and SOPS enables true GitOps for applications with sensitive data, maintaining security without sacrificing the benefits of declarative infrastructure.

References

  1. SOPS Documentation
  2. ArgoCD SOPS Integration
  3. KSOPS Plugin
  4. KodeKloud SOPS Notes
  5. Files in my homelab repo https://github.com/juanjo-vlc/homelab/tree/main/k8s-rancher-additional