Version: Omada Identity Cloud Private

Post-deployment health check

The health check script (health_check.sh) validates that a deployed Omada Identity Cloud Private environment is running correctly. It checks endpoint availability, pod readiness, component versions, error logs, and Service Bus dead letters, and produces a summary report suitable for pre-handover validation or routine operational checks.

Prerequisites

Before running the script, ensure the following:

Azure CLI is installed and authenticated (az login) against the correct subscription.
jq and curl are available on the machine running the script.
The executing identity has the following RBAC permissions on the environment resource group:

Permission	Purpose
`Microsoft.ContainerService/managedClusters/read`	AKS – read cluster
`Microsoft.ContainerService/managedClusters/listClusterAdminCredential/action`	AKS – list admin credentials
`Microsoft.Web/sites/read`	App Service – read site
`Microsoft.Web/sites/config/read`	App Service – read config
`Microsoft.Web/sites/config/list/action`	App Service – list app settings
`Microsoft.OperationalInsights/workspaces/read`	Log Analytics – read workspace
`Microsoft.OperationalInsights/workspaces/query/*/read`	Log Analytics – query
`Microsoft.ServiceBus/namespaces/read`	Service Bus – list namespaces
`Microsoft.ServiceBus/namespaces/topics/read`	Service Bus – list topics
`Microsoft.ServiceBus/namespaces/topics/subscriptions/read`	Service Bus – list subscriptions

note

Some of these permissions are not included in the built-in Reader role. Assign a custom role covering all 10 actions, or use Contributor / Owner. RBAC changes can take 5–10 minutes to propagate after assignment.

Running the script

From the repository root, run:

Tool: Bash shell

./scripts/health_check.sh --global-prefix <global-prefix> [--log-window-minutes <n>]

Flag	Required	Description
`--global-prefix`	Yes	Environment global prefix (for example, `oisaas-ab001`).
`--log-window-minutes`	No	Time window for error log check in minutes (default: `30`).
`--tfvars-path`	No	Path to `variables.auto.tfvars.json` (default: `iac/infrastructure/iac/variables.auto.tfvars.json`).
`--apps-config`	No	Path to `health_check_apps.json` (default: `scripts/health_check_apps.json`).
`--dns-zone`	No	DNS zone for the Enterprise Server (AKS) endpoint. Auto-detected from `config.artifact_source.dns_zone_name` in `--tfvars-path` if not provided. Pass explicitly to override.
`--retries`	No	Extra retry attempts for transient failures (default: `2`).
`--retry-delay`	No	Seconds between retry attempts (default: `2`).
`--manifest`	No	Path to a manifest JSON to use as the expected-version source (enables manifest mode).
`--use-manifest`	No	Manifest mode: use a local `manifest.json`, or fetch `<global-prefix>.json` from blob storage if no local file exists.

Exit codes:

0 – all checks passed, or only warnings.
1 – one or more FAIL results.

What the script checks

The script runs five checks in sequence.

Check 1: Endpoint ping

Verifies that all services respond on their expected endpoints. All requests run in parallel – the total time equals the slowest single endpoint (15-second timeout).

Result	HTTP response
PASS	`200`, `301`, `302`, `401`
WARN	`404` (endpoint not found), `503` (cold start)
FAIL	`000` (no connection / timeout), `5xx`

Web apps are checked at /swagger/index.html. Function apps are checked at /api/Heartbeat – a 404 response is treated as WARN rather than FAIL, since not all function apps expose an HTTP trigger.

Check 2: AKS pod readiness

Checks that each AKS deployment has at least one ready pod. The following deployments are checked:

Deployment	Service
`<alias>-es`	Enterprise Server
`<alias>-rope`	Role and Policy Engine
`<alias>-timer`	Timer
`<alias>-cagops`	CAG Operations
`<alias>-cag-import-default`	CAG Import (`Default` instance group)
`<alias>-cag-import-omadahosted`	CAG Import (`OmadaHosted` instance group)
`<alias>-cag-provisioning-default`	CAG Provisioning (`Default` instance group)
`<alias>-cag-provisioning-omadahosted`	CAG Provisioning (`OmadaHosted` instance group)

If kubectl is unavailable or the AKS cluster is unreachable, this check is skipped with WARN.

Check 3: Component version verification

Confirms that all running components match the versions declared in variables.auto.tfvars.json. Checks both App Service / Function App blob URLs (WEBSITE_RUN_FROM_PACKAGE) and AKS container image tags, including the four CAG worker deployments listed in Check 2. All app settings calls run in parallel.

By default, expected versions are read from variables.auto.tfvars.json (config mode); with --manifest or --use-manifest, they are read from a release manifest instead.

Result	Condition
PASS	Running version matches the expected version.
WARN	The expected version key is not found in the version source, or the running version could not be parsed from the blob URL.
FAIL	The running version differs from the expected version, or `WEBSITE_RUN_FROM_PACKAGE` is not set.

Check 4: Error log report

Queries the Log Analytics workspace (<global-prefix>-law) for errors across three sources within the configured time window:

Source	What is checked
OIS (`OIS_CL`)	Custom log table entries at level 4 (Error) or 5 (Fatal).
LRDO	Container logs from pods matching `lrdo` (long-running data operations), filtered to `critical`/`error` log levels or messages containing fail, error, or exception*.
CAG	Container logs from the `cag-import-` and `cag-provisioning-` containers – both instance groups – filtered the same way as LRDO.

Each source is reported independently as PASS or WARN – not FAIL, since isolated log errors do not necessarily block handover. For each source with errors, the report includes a per-component/pod/container breakdown, a deep link to the relevant Log Analytics logs blade, and the underlying KQL query (including a ±5 minute context query for investigation).

note

If Log Analytics is not enabled for the environment (for example, in some development workspaces), all three checks are skipped and reported as N/A.

Check 5: Service Bus dead letters

Checks all Service Bus namespaces in the resource group for dead-lettered messages. Dead letters indicate processing failures that were not retried successfully.

Dead letter count = 0 → PASS
Dead letter count > 0 → WARN with per-topic/subscription breakdown

Namespace and topic checks run in parallel at two levels – the total time equals the slowest single namespace.

Interpreting results

Each check produces a line per resource with a PASS, WARN, FAIL, or N/A label, followed by a summary; for example:

======================================================
 Omada Private Cloud – Environment Health Check
 Workspace : oisaas-ab001
 Date/Time : 2026-05-26 14:32 UTC
 Version   : 1.0.13
======================================================

[1/5] ENDPOINT PING
  PASS  oisaas-ab001-wa-copsapi   (200)
  FAIL  oisaas-ab001-wa-ods       (000 - no connection / timeout)
  ...
  N/A   oisaas-ab001-fa-is (Heartbeat)   Heartbeat check skipped - no /api/Heartbeat endpoint (404) - not implemented
  elapsed: 18s

[2/5] AKS POD READINESS
  PASS  oisaas-es                            1 pod(s) ready
  FAIL  oisaas-rope                          no ready pods
  PASS  oisaas-cag-import-default            1 pod(s) ready
  PASS  oisaas-cag-import-omadahosted        1 pod(s) ready
  PASS  oisaas-cag-provisioning-default      1 pod(s) ready
  PASS  oisaas-cag-provisioning-omadahosted  1 pod(s) ready
  ...

[3/5] BLOB VERSION CHECK
  PASS  oisaas-ab001-wa-copsapi   418163_06fa5d…
  FAIL  oisaas-ab001-wa-ods       running: 414000_abcd…  expected: 415909_6fcbb8…
  WARN  oisaas-ab001-fa-ci-rp     version key not found: coreingestion_rp_funapp
  ...

[4/5] ERROR LOGS (last 30m)
  WARN  OIS_CL – 2 component(s) with errors:
          Component=ImportJob  Level_d=4  Count=3
  PASS  LRDO – no errors/criticals in last 30m
  PASS  CAG – no errors/criticals in last 30m
  ...

[5/5] SERVICE BUS DEAD LETTERS
  PASS  oisaas-ab001-en-bus  0 dead letters
  WARN  oisaas-ab001-ci-bus  7 dead letters
          topic-a/sub-1       7 dead letters
  ...

======================================================
 SUMMARY: 28 passed  |  1 failed  |  2 warned
 RESULT : FAIL – environment not ready for handover
======================================================

The final result is:

PASS – all checks passed; environment is ready for handover.
WARN – one or more checks produced a warning, but nothing failed; review before handover.
FAIL – one or more checks failed; investigate before handover.

Known limitations

Permission pre-check is informational only. WARN results in the permissions pre-check do not abort the run – they indicate which downstream checks are likely to fail and why. notActions (deny overrides) are not evaluated.
AKS checks require kubectl. If kubectl is absent or the cluster is unreachable, check 2 is skipped with WARN, and check 3's AKS container-tag check is skipped with WARN as well (its blob-version checks still run).
Log Analytics must be enabled. Check 4 is skipped with N/A in environments where Log Analytics is not configured.
Timeout flakiness. Occasional FAIL results on timeout-sensitive checks may resolve on a second run.
Naming convention dependency. The script derives resource names from the global prefix. Customized resource names require updating scripts/health_check_apps.json.

tip

If the health check reports failures or warnings and you need to troubleshoot further, see Collecting debug logs to gather diagnostic data for analysis or to send to Omada Support.

Prerequisites​

Running the script​

What the script checks​

Check 1: Endpoint ping​

Check 2: AKS pod readiness​

Check 3: Component version verification​

Check 4: Error log report​

Check 5: Service Bus dead letters​

Interpreting results​

Known limitations​