Key Vault and secret management

Omada Identity Cloud Private uses Azure Key Vault as the single source of truth for all secrets, certificates, and sensitive configuration values. This page explains how secrets are stored, how they flow into running pods, and how to rotate them when needed.

Key Vault architecture

The deployment uses three Azure Key Vaults:

Key Vault	Purpose
Workspace Key Vault (<project_name>-<workspace_prefix>-kv)	Application secrets: SQL credentials, Application Insights keys, Entra ID service principal credentials, machine key, WinRM password.
CAG Key Vault (<project_name>-<workspace_prefix>-cagkv)	Private keys for the Cloud Application Gateway (CAG) instance group. The CAG workers read and write these keys at runtime.
Shared Key Vault (<project_name>-shared-kv)	Shared certificates: PFX certificate and password, used across workspaces.

Secrets from the workspace and shared Key Vaults are mounted into pods using the Azure Key Vault Provider for Secrets Store CSI Driver. This replaces manual secret injection at deployment time – secret values live only in Key Vault and are never handled by the deployment pipeline. The CAG Key Vault is the exception: the CAG workers access it directly using Azure Workload Identity.

How secrets reach pods

The CSI driver mounts Key Vault secrets directly into running pods and optionally syncs them to Kubernetes Secrets. The flow is:

Azure Key Vault
    │
    ▼
CSI Secrets Store Driver (polls every 2 minutes)
    │
    ├─► Mounted volume files inside pods
    │     Updated automatically by the driver
    │
    └─► Synced Kubernetes Secrets
          Updated automatically when mounted files change
          │
          └─► Pod environment variables
                Set once at pod startup – not refreshed automatically

warning

Environment variables injected from Kubernetes Secrets are set once at pod startup. Updating the secret in Key Vault does not automatically update running pods. A pod restart is required for environment variable changes to take effect.

Access control

Access to the workspace and shared Key Vaults is granted through access policies. Access to the CAG Key Vault is granted through Azure RBAC role assignments. The following identities are granted access during deployment:

Identity	Key Vault	Permissions
AKS cluster identity	Workspace Key Vault	Get, List (secrets and keys)
CSI Secrets Store addon identity – Workspace KV	Workspace Key Vault	Get, List (secrets)
CSI Secrets Store addon identity – Shared KV	Shared Key Vault	Get, List (secrets)
Application managed identities (<project_name>-<workspace_prefix>-*-mi, except the CAG workers identity)	Workspace Key Vault	Get, List (secrets)
COPS API web app identity	Workspace Key Vault	Get, List, Set (secrets)
CAG workers managed identity (<project_name>-<workspace_prefix>-cag-workers-mi)	CAG Key Vault	Key Vault Secrets Officer (RBAC role: read and write secrets)
Azure App Service (Microsoft.Web service principal)	Shared Key Vault	Get, List (secrets) – used to import the TLS certificate for custom domain bindings

All access policies and role assignments are configured automatically during deployment and follow the principle of least privilege – each identity is granted only the permissions required for its function.

Certificates

Required certificates

Before deployment, the following certificates must be prepared and uploaded to Azure Key Vault. The installation script validates their presence and will stop if any are missing.

All certificates must be issued for the same domain used in your Cloud Private deployment (matching your dns_zone_name).

Primary DNS zone certificate

Property	Details
Purpose	SSL/TLS for custom domain bindings on App Services; enables HTTPS for all external-facing endpoints
Format	Certificate imported directly into Key Vault (not a secret)
Validation	Must be in ready state with a valid secret ID

External certificate pair

Key Vault secret	Description
aks-extCertBase64	Base64-encoded certificate (PEM format)
aks-extKeyBase64	Base64-encoded private key (PEM format)

Purpose: Secures external HTTPS endpoints. The domain must match your external DNS name.

PFX certificate bundle

Key Vault secret	Description
aks-pfxCertificate	Base64-encoded PFX/PKCS#12 bundle
aks-pfxPassword	Password protecting the PFX file (plaintext)

Purpose: Used by Windows container instances running Enterprise Server for IIS SSL bindings.

Obtaining certificates

Obtain valid TLS certificates for your domain from your organization's certificate authority or preferred certificate provider. All certificates must be issued for the same domain used in your Cloud Private deployment. Upload them to Azure Key Vault under the secrets defined above.

Encoding certificates for Key Vault secrets

The aks-extCertBase64, aks-extKeyBase64, and aks-pfxCertificate secrets must be Base64-encoded before uploading. The aks-pfxPassword is stored as plaintext.

The encoded values are provided in deployment-config.json under the secrets section:

Format: JSON

{
  "secrets": {
    "aks-extCertBase64": "<base64-encoded-external-cert>",
    "aks-extKeyBase64": "<base64-encoded-external-key>",
    "aks-pfxCertificate": "<base64-encoded-pfx-bundle>",
    "aks-pfxPassword": "<pfx-password>"
  }
}

Automatically generated secrets

The following secrets are created and populated automatically during deployment. You do not need to provide these manually.

Secret	Purpose
aks-machineKey	Shared cryptographic keys for ASP.NET encryption across all Enterprise Server pod replicas
aks-containerRegistryDockerConfigJson	Credentials for AKS to pull container images from the container registry

Pre-deployment checklist

Before running the installation script, verify:

• Primary DNS zone certificate exists in the shared Key Vault and is in ready state.
• External certificate and key are Base64-encoded, valid, and paired.
• PFX bundle is Base64-encoded and password-protected.
• PFX password is correct and stored in Key Vault.
• All certificate secrets are populated in deployment-config.json.
• No certificates have expired.

Secret rotation

How rotation works

The CSI driver has a built-in rotation reconciler that polls Azure Key Vault every 2 minutes. When a secret value changes in Key Vault:

1. The driver detects the change within 2 minutes.
2. Mounted volume files inside running pods are updated automatically.
3. Synced Kubernetes Secrets are updated immediately after the volume files update.
4. Pod environment variables are not updated – a pod restart is required.

How to rotate a secret

1. Update the secret value in Azure Key Vault using the Azure Portal or Azure CLI.

2. Wait up to 2 minutes for the CSI driver to detect the change and update the mounted files and Kubernetes Secrets.

3. Restart the affected pods to pick up the new value in environment variables:

   Tool: kubectl

   kubectl rollout restart deployment/<deployment-name> -n <namespace>

tip

If your application reads secrets directly from the mounted file path rather than environment variables, a pod restart is not required – the updated value is available immediately after the CSI driver polls.

Prerequisites for rotation

For automatic rotation to work, the following must be in place:

Requirement	Details
CSI Secrets Store addon enabled on AKS	Configured during deployment
Secret rotation enabled	Set to true with a 2-minute polling interval
CSI identity has Key Vault access	Get and List permissions on the workspace and shared Key Vaults
At least one pod mounting the CSI volume is running	The driver only polls Key Vault while a pod with the volume is active

Known limitations

• No automatic pod restart. The CSI driver updates files and Kubernetes Secrets but never restarts pods. Environment variables remain stale until a manual restart.
• Rotation only works while pods are running. If all pods referencing a secret are scaled to zero, the driver stops polling. Rotation resumes when a pod mounts the volume again.
• One Key Vault per secret class. The CSI driver does not support multiple Key Vaults in a single secret class configuration. This is why the deployment uses two separate secret classes for the workspace and shared Key Vaults.

Bring your own key (BYOK) and key material ownership

Omada Identity Cloud Private does not use a centralized, Omada-operated key management service. All cryptographic key material used by a Cloud Private deployment resides in the customer's own Azure subscription, in Key Vault instances deployed there as part of that deployment.

Where key material resides

A Cloud Private deployment uses three Key Vault instances, all deployed into the customer's own Azure subscription:

Key Vault	Scope	Purpose
Workspace Key Vault	Per workspace	SQL credentials, Entra ID service principal secrets, machine key, and other application secrets for that workspace.
Shared Key Vault	Shared across workspaces in a subscription	The TLS certificate and its private key, together with other secrets common to all workspaces sharing the same global infrastructure.
Cloud Application Gateway (CAG) Key Vault	Per workspace, dedicated	Holds only the Linux CAG worker group's private key material. Kept separate from the Workspace Key Vault so that the CAG workers' read and write grant reaches nothing else – they cannot read SQL credentials, the machine key, or service principal secrets.

note

For reference, also see Key Vault and secret management.

Customer-supplied vs. generated-during-deployment key material

There are two categories of key material used by Cloud Private:

Key material	Origin
TLS certificate and private key	Supplied by the customer, imported into the Shared Key Vault before or during deployment. One certificate serves every workspace that shares the same global infrastructure.
SQL credentials	Generated during deployment and stored directly in the Workspace Key Vault.
Entra ID service principal client secrets	Generated during deployment and stored directly in the Workspace Key Vault.
Machine key (ASP.NET machine key used by Enterprise Server pods)	Generated during deployment and stored in the Workspace Key Vault.

important

None of this key material is transmitted to, stored by, or escrowed in any system operated by Omada.

Cloud Application Gateway private keys

The two CAG types manage their private keys differently:

• Linux CAG workers, which run inside the Cloud Private AKS cluster, read and write their group's private key at runtime via Azure Workload Identity, scoped to the dedicated CAG Key Vault described above.
• A customer-hosted Windows CAG, which the customer deploys outside the Cloud Private deployment for connectivity that has no cross-platform package, holds its private key locally on that host.

For full CAG security documentation, see Cloud Application Gateway security.

COPS API and public key registration

The COPS API, with which the CAG's public key is registered, runs as a service inside the customer's own Cloud Private deployment. It is deployed per workspace, alongside the rest of the Cloud Private application tier; it is not a shared or Omada-hosted service.

Who owns key and secret management

Consistent with the shared responsibility model, the customer owns all key and secret management for their Cloud Private deployment. This includes the rotation of:

• TLS certificates.
• SQL credentials.
• Entra ID service principal secrets.
• The machine key.

important

Omada does not manage, rotate, or have access to this key material as part of operating Cloud Private.

Out of scope

This section covers key and secret management for the Cloud Private application tier. Encryption at rest for underlying Azure platform resources and customer-managed keys (CMK) for that encryption are a separate topic.