Upgrade guide

This page describes how to upgrade your Omada Identity Cloud Private environment to a new version. It covers the pre-upgrade checklist, the upgrade process itself, and what to do if the installer stops.

note

There is no separate upgrade tool for Omada Identity Cloud Private. You upgrade an environment by running the new release's installer against the workspace you already have. The installer detects the existing environment and updates it in place: Terraform reconciles the infrastructure, Ansible re-applies the configuration, and the new container images and application packages are rolled out.

Before you start upgrading

This page assumes that you already have an Omada Identity Cloud Private environment and that you are familiar with Installation Script mode. An upgrade uses the same script and the same deployment-config.json, with one extra flag.

tip

Throughout this document, <version> stands for the release you are upgrading to – for example 16.51.90. You do not type or choose this value separately: it is the version that appears in the /releases/<version>/ path segment of the URL Omada provides, and the installer reads it from there. If that segment is missing from the URL, the installer stops before its first phase.

See the following sections for the prerequisites, limitations of the upgrade process, and expected impact of the upgrade.

Prerequisites

The prerequisites for an upgrade are the same as for a fresh installation. See the following pages for details:

• General prerequisites
• System and networking prerequisites
• Installation Script mode for the exact steps of the installer, for reference.

Limitations and expected impact of the upgrade process

Supported upgrade path

Upgrading directly from the July 2026 release to the September 2026 release with the Installation Script is currently not supported.

For details and a recommended path, see the related known issue.

The upgrade procedure does not cover:

• Moving an environment to a different Azure region. A resource group's region cannot be changed.
• Rolling back to an earlier release. Downgrades are blocked by design (see If the installer stops), and there is no supported down-migration for database schema changes. Recovery from a bad release is a restore from backup – take one before you start.
• Changing the workspace or project name of an existing environment.

impact

You should expect the following impact during an upgrade:

Application packages are deployed by updating the WEBSITE_RUN_FROM_PACKAGE app setting on each App Service to point at the new package. There are no deployment slots, so each App Service has a short outage while it restarts.

Azure Kubernetes Service (AKS) workloads roll out in the order of the Ansible sequence:

1. Database baseline.
2. AKS infrastructure, Enterprise Server, and the Cloud Application Gateway (CAG) workers.
3. Full Enterprise Server configuration.
4. Role and Policy Engine (RoPE), CAG operations, and Omada Provisioning Service (OPS), in a second AKS pass once the first is healthy.

As a result, components run mixed versions for a while, so plan the upgrade inside a maintenance window.

An upgrade also cannot start while an import is running – see step 3 of the pre-upgrade checklist.

Updating your configuration

The new installer package also ships its own deployment-config.json. For a new release, you can replace it with your existing deployment-config.json (for details, see Configure deployment-config.json again) and update the release section to point at the new package:

• release.package_url to the new oicp_<version>.zip.

The installer downloads and extracts the package during its release phase. Everything else in the file stays as it is: the same subscription, workspace, project, region, backend, certificate, and Entra groups remain as in your existing installation.

Pre-upgrade checklist

1. Record the version you are on.

   The installer reads this from a tag on your workspace resource group:

   Tool: Bash shell

   az group show --name <global-prefix>-rg --query tags.ProductVersion -o tsv

   • A version number, for example 16.51.88 – normal. Note it down.
   • Empty – also normal if you have never upgraded since installing a release older than the version guard. Your first upgrade requires --upgrade and reports itself as unverified; see the first row in If the installer stops. This is expected and self-correcting.

2. Take a database backup.

   Automated backups are enabled in Azure SQL, but take an explicit point-in-time reference before a release that includes schema migrations, and confirm you know how to restore it. There is no supported down-migration.

   The installer enforces this. Before it captures the pre-upgrade snapshot, it checks that point-in-time restore is available on every product database that already exists, and refuses to continue if it is not:

   Output: installer log

   Point-in-time restore is not available on existing database '<db>' – refusing to upgrade without a restore anchor. Investigate before retrying with --upgrade.

3. Confirm no import is running.

   The installer checks this for you and refuses to proceed (see If the installer stops), but finding out now is much cheaper than finding out mid-window. Run this read-only query against your Enterprise Server database (<global-prefix>-ois-db):

   Tool: SQL (OIS database)

   SELECT Id, ImportType, State, StartTime
   FROM [isp].[Import]
   WHERE ImportType NOT IN (2, 5)          -- exclude Preview / ExportPreview
     AND State IN ('Queued', 'Started', 'StagingCompleted',
                   'Dequeued', 'ReadyToStart', 'WaitingForCoreHorizons')
   ORDER BY Id;

   Any row returned is an import that has not finished. Let it complete, or cancel it, before you start. A Queued row whose StartTime is more than two days old is treated as stale and does not block the upgrade – but it is worth investigating separately.

   The installer's own check queries Import Service rather than the table directly, so treat the query above as an equivalent read-only check you can run ahead of time, not as the same call.

   Once past the gate, the installer pauses new imports for the duration of the run (phase pause_imports, right after the import gate) and releases the pause when the run ends, whether it succeeds or fails. This happens on any run against an existing environment, not only an --upgrade run. The installer then waits for any import that started between the gate and the pause to finish. You can control this wait with the following options:

   • --drain-timeout <minutes> sets how long the installer waits (default 30).
   • --skip-import-drain pauses imports without waiting.
   • --drain-timeout 0 fails immediately instead of waiting.

4. Confirm your Azure sign-in and permissions.

   Tool: Bash shell

   az account show --query "{subscription:name, id:id, user:user.name}" -o table
   az account get-access-token --query expiresOn -o tsv

   Use the same identity and subscription as the original installation. The installer runs its own permission and quota preflight checks and tells you what is missing.

5. Check quota headroom.

   If the release adds infrastructure, the upgrade needs room for it. The installer checks this as part of its preflight checks; you can check ahead of time with az vm list-usage for your region.

Run the upgrade

Unpack the new installer package.

warning

Do not use the installer you used for the original deployment. The installer and the safety checks described on this page ship with the release, so an older installer cannot protect an upgrade to a newer version.

Run the installer package with the --upgrade flag:

Tool: Bash shell

unzip oicp_install_<version>.zip
cd scripts/install

./install.sh --config ./deployment-config.json --upgrade --verbose

note

In interactive mode, expect a few yes/no confirmation prompts in phase 7 – before each terraform apply, and before each of the two artifact-publish steps. Each defaults to yes. If a required certificate is missing or invalid, secrets_validate presents a menu rather than a prompt. For unattended runs, use --non-interactive (as in Long runs and interruptions) to accept the default answer for every prompt.

The rollback prompt described in Installation Script mode never appears on an upgrade run – --upgrade disables rollback outright.

If ./install.sh fails with Permission denied, restore execute permission on the scripts. See Extract the installation package.

The --upgrade flag ensures that the installer treats the operation as an intentional forward version change. It has three effects:

1. It lets the version guard allow the upgrade for a forward version change, and for the one-time case where your environment has no version tag yet.

2. It disables rollback. The installer forces ROLLBACK_ON_FAILURE=false and prints:

   Output: installer log

   --upgrade active: forcing ROLLBACK_ON_FAILURE=false (was 'true') – a failed upgrade must never reach terraform destroy.

   If a later phase fails, the installer stops and leaves everything in place for investigation. It does not tear down your environment. If any part of the installer still tries to roll back, the installer refuses and logs:

   Output: installer log

   Rollback refused: --upgrade is active. A failed upgrade must never run terraform destroy against workspace '<workspace_prefix>'. Resources are left as-is for manual investigation.

3. It is recorded in the run's summary.json as "run_mode": "upgrade".

note

You cannot use the --upgrade flag to install an older release. Downgrades are always blocked.

What the installer does

The upgrade runs the same phases documented in Stage dependency overview, plus the following additions that run immediately after the release package is extracted, before anything is changed:

 1  preflight            OS, disk, network connectivity
 2  config               load and validate deployment-config.json
 3  tools                install/verify terraform, ansible, az, kubectl, kubelogin,...
 4  azure                sign-in, permission and quota checks
 5  terraform_backend    locate the existing Terraform state backend
 6  release              download and extract the release package
 6b version_guard        ◀ compares the release version to your environment's tag
 6c import_gate          ◀ refuses to continue while an import is active
 6c.5 upgrade_snapshot   ◀ upgrade mode only – captures pre-upgrade state as the restore anchor
─── nothing above this line changes your environment ──────────────────────────
 6d pause_imports        ◀ pauses new imports for the run, then drains any already in flight
 7  terraform_iac_shared → artifact publish → tfvars_sync → secrets_validate → terraform_iac
 8  ansible: database → AKS phase 1 → Enterprise Server → AKS phase 2
 9  health_check
 9b environment_snapshot_post  captures post-deployment state (also attempted if a phase fails)
 9c upgrade_verify       ◀ upgrade mode only – verifies the upgrade took effect (checks A1–A5), then marks it complete

If version_guard or import_gate blocks the run, your environment is untouched. upgrade_snapshot writes only to the artifacts storage account, so your environment is still untouched at that point. pause_imports is the first phase that changes your environment. It runs on any run against an existing environment, not only in upgrade mode.

Long runs and interruptions

An upgrade takes a long time. On a remote host, run it detached so a dropped SSH connection does not stop it:

Tool: Bash shell

nohup ./install.sh \
  --config ./deployment-config.json \
  --upgrade --non-interactive --verbose \
  > install-upgrade.log 2>&1 < /dev/null & disown

Monitor with tail -f install-upgrade.log, or follow the installer's own log under ../../Deployment_OICP_<version>/run-<project_name>-<workspace_prefix>/logs/install-<timestamp>.log as described in Run the installation script.

If a phase fails for a transient reason, first read the error message and check If the installer stops. Then re-run the same command with --resume appended to continue from the last completed phase rather than starting over.

note

--resume skips phases that already completed, which can include the version guard. The active-import check, the pre-upgrade snapshot, and the import pause always run on every invocation – the installer never reuses a checkpoint from an earlier attempt for these three phases. A re-run adopts the snapshot the earlier attempt captured rather than taking a new one, so the restore anchor stays fixed at the first attempt.

Upgrade verification

In upgrade mode, the installer's final phase (upgrade_verify) checks that the upgrade actually took effect. It compares the pre-upgrade and post-upgrade state, and checks both against the release manifest and the Terraform plan in the install log:

Check	What it verifies
A1	Every component recorded the new release's version – nothing silently stayed behind.
A2	The running applications and AKS workloads report the versions in the release manifest. Note: A2 cannot version-check a few components: connectivity packages, long-running data operations (LRDO) CronJobs, and the Windows import and provisioning workers on the CAG VM. These appear as SKIP with the reason, and are expected.
A3	The environment passes the health check.
A4	Customer settings, master settings, and authentication configuration survived the upgrade. Note: A4 masks values whose names look like secrets (containing secret, password, key, token, or connectionstring). A credential stored under another name can appear, abbreviated, in the results file and the install log. Treat both as sensitive when you share them.
A5	Terraform did not plan to replace or destroy any resource.

Each check reports PASS, FAIL, WARN (a change worth reviewing, never a failure on its own), or SKIP (could not be checked – the reason is shown).

The run ends with one of these results:

Result	What it means
PASS – upgrade verified	All checks that ran passed.
WARN – review warnings	Nothing failed, but review the warnings before closing the maintenance window.
INCONCLUSIVE – nothing was verified	No check could produce a result. The run does not fail, but the upgrade is not proven. See Upgrade verification failed or was inconclusive.
FAIL – upgrade did not verify	At least one check failed. The installer run fails and, because --upgrade disables rollback, leaves the environment as is.

Results are written to the run's logs/ directory (../../Deployment_OICP_<version>/run-<project_name>-<workspace_prefix>/logs/):

• assert-upgrade-results.xml – every check with its status and detail (JUnit format).
• assert-upgrade-health-check.log – the health check report used for A2 and A3.

After the upgrade

1. Confirm the version advanced.

   Tool: Bash shell

   az group show --name <global-prefix>-rg --query tags.ProductVersion -o tsv

   This must now report <version>. If your environment previously had no tag, it has one now, and every future upgrade is fully verified.

2. Read the run summary.

   Tool: Bash shell

   cat ../../Deployment_OICP_<version>/run-<project_name>-<workspace_prefix>/logs/summary.json

   Confirm "status": "success", "run_mode": "upgrade", and "rollback_on_failure": false. The same file also records "drain_timeout_seconds" and "skip_import_drain" for the run, and "restore_anchor_utc". This is the UTC time when the installer took the pre-upgrade snapshot, and the point in time to restore to if you need to recover.

3. Review the health check, which the installer runs near the end of the upgrade. It reports any issues that need attention before you close the maintenance window.

   tip

   See Post-deployment health check for how to read the report and re-run it on demand.

4. Review the upgrade verification result.

   Tool: Bash shell

   grep -E "RESULT :|Upgrade verification" ../../Deployment_OICP_<version>/run-<project_name>-<workspace_prefix>/logs/install-<timestamp>.log

   Expect PASS – upgrade verified. On WARN, read each warning in logs/assert-upgrade-results.xml – typically settings that the upgrade added, changed, or retired. On INCONCLUSIVE or FAIL, see Upgrade verification failed or was inconclusive.

5. Re-enable anything you paused: scheduled imports, monitoring silences, maintenance-mode banners.

6. Run a functional verification: sign in to Enterprise Server, confirm the version shown, and run one small import end to end before closing the maintenance window.

Troubleshooting

The following sections describe the most common issues that can occur during an upgrade and how to resolve them.

If the installer stops

Every message below is a deliberate refusal before any change was made. Your environment is in the state it was in before you started.

Message	What it means	What to do
Resource group '<rg>' has no 'ProductVersion' tag – its prior version cannot be verified. Pass --upgrade to proceed anyway.	Your environment was installed from a release older than the version guard, so there is no recorded version to compare against.	Add --upgrade as described in Run the upgrade. The installer proceeds and warns that the starting point is unverified. This run writes the tag, so every future upgrade is fully verified. You only need to do this once per environment.
Target version <new> is higher than the deployed version <old> on '<rg>'. Pass --upgrade to acknowledge this upgrade.	A normal forward upgrade, without the acknowledgement flag.	Add --upgrade.
Target version <old> is lower than the deployed version <new> on '<rg>'. This is a downgrade and is always blocked – no flag can override this.	The package you are installing is older than what is deployed.	Check whether you unpacked the right release. Important: Rolling an older release over a newer database has no clean recovery path. If this is a deliberate rollback, it must be planned with Omada support; the resource group tag has to be corrected out of band first.
Resource group '<rg>' is already at <version> – repair/retry, proceeding.	Not an error. You are re-running the same version, for example to finish an interrupted install.	Nothing – the run continues. No flag needed.
--upgrade was given but resource group '<rg>' does not exist – there is no existing environment to upgrade.	There is nothing at that name to upgrade.	The cause is almost always a wrong --workspace or deployment.workspace_prefix. Check both values. If this really is a first install, remove --upgrade and follow the Installation Script mode documentation.
Aborting before any change is made: N active import(s) detected – Id <n>, State='Started', StartTime=...	An import is in flight. Restarting components now risks corrupting it.	Wait for it to finish, or cancel it, then re-run. See step 3 of the pre-upgrade checklist for the query. If a hotfix genuinely cannot wait and you accept the risk, see Bypassing the active import check.
Discounted: Id <n>, State='Queued', ... Queued row stale (StartTime more than 2 days ago)	Not an error. A queued import older than two days is treated as abandoned and does not block.	Nothing – the run continues. Worth investigating the stale row separately.
<db> on <server> is not reachable yet, or isp.Import does not exist yet (during a fresh install)	Informational. Normal before the database baseline has run.	Nothing – the run continues.
<db> on <server> is not reachable yet, or isp.Import does not exist yet (during an upgrade)	There should already be a reachable database with this table at this point, so your database connectivity or sign-in is wrong.	Check step 4 of the pre-upgrade checklist.
Product version guard blocked this deployment (resource group: <rg>, target version: <version>).	The wrapper line that follows any of the version guard messages above.	Act on the specific message printed just above it.

Upgrade verification failed or was inconclusive

Unlike the messages above, these appear after the upgrade has been applied. Open logs/assert-upgrade-results.xml to see which check failed – each result is named, for example A1-markers/container_es@... or A5-terraform/<resource address>.

Failing check	What it means	What to do
A1	A component still records its old version.	Re-run the same command without --resume. A component that still records its old version is redeployed on the next full run. If it persists, raise a support case.
A2	An application or AKS workload is running a version other than the one in the release. The message shows running and expected versions.	Re-run the same command without --resume. If it persists, raise a support case.
A3	The environment failed its health check.	See Post-deployment health check and logs/assert-upgrade-health-check.log.
A4	An authentication entry (customerAuth) is missing after the upgrade. Removed customer or master settings are only warnings, since upgrades retire deprecated settings.	Do not continue using the environment. Raise a support case.
A5	Terraform replaced or destroyed a resource (named in the result). The change has already been applied – A5 reports it, it does not prevent it.	Check that every resource named is one you meant to change, for example through a SKU change you made in deployment-config.json, and that A1–A4 passed. If so, the upgrade has taken effect and there is nothing to roll back – confirm the replaced resource is healthy, and record the result with the maintenance window. If any resource named was not expected, do not continue using the environment. Raise a support case.
INCONCLUSIVE	No check could produce a result, for example because the pre- or post-upgrade capture was unavailable.	Do the manual checks in After the upgrade and raise a support case so the cause can be found.

note

Do not use --resume to recover from a failed upgrade verification. After the upgrade has been applied, --resume skips every deployment phase and re-checks the state it already recorded, so it cannot change the result.

tip

If you plan a SKU change, run the upgrade interactively, without --non-interactive. The installer then shows the Terraform plan and asks for confirmation before applying it. Look for must be replaced or will be destroyed in the plan. In non-interactive mode, the plan is applied without confirmation.

New settings appearing as warnings

A new release can introduce new customer settings or master settings, and can change the value of existing ones. Check A4 compares the settings before and after the upgrade and reports each difference as a warning. For example:

Output: installer log

A4 — Configuration survived the upgrade
○ WARN  A4-config/customerSettings@ProvisioningTaskHistoryTrackingBatchSize — added between pre and post capture — not present before the upgrade
✓ PASS  A4-config/customerAuth — 5 entries compared, 0 changed, 0 added, 0 removed

The warnings are expected and do not mean the upgrade went wrong. Each settings group still reports PASS in its summary line, and the run ends with WARN – review warnings rather than FAIL.

tip

Review each new setting carefully before you close the maintenance window: a new setting comes with a default value, and that value can change how features behave in your environment. Check the customer settings and master settings documentation to learn for what each setting does, and adjust its value if the default does not suit your environment.

The same applies to settings whose value changed, or that the upgrade removed.

Bypassing the active import check

There might be a case where you need to upgrade while an import is active, and you understand the risk. The installer refuses to continue in that case, but you can bypass the check with an environment variable.

If you need a forced hotfix, run:

Tool: Bash shell

SKIP_IMPORT_CHECK=true ./install.sh \
  --config ./deployment-config.json --upgrade --verbose --skip-import-drain

SKIP_IMPORT_CHECK bypasses the import_gate phase only. The pause_imports phase that follows still waits for the running import to finish, for up to --drain-timeout minutes (default: 30), so a forced hotfix needs --skip-import-drain as well. Otherwise, the run stalls for half an hour at the point you were trying to skip.

The gate bypass is logged where the check would have run:

Output: installer log

Active-import safety check SKIPPED (SKIP_IMPORT_CHECK=true) – forced override, proceeding without verification.

warning

The check exists because restarting Enterprise Server, RoPE, or CAG/OPS while an import is mid-flight can leave that import's data in an inconsistent state. Bypassing it means you may have to re-run the affected import afterwards, and in the worst case, clean up partially written data.

Checking the version guard on its own

You can run the version guard by hand at any time, without starting an upgrade – it is the same check the installer performs:

Tool: Bash shell

../check-product-version.sh \
  --resource-group <global-prefix>-rg \
  --target-version <version>

The exit codes are:

Exit code	Meaning
0	Safe to proceed.
1	Blocked.
2	Usage or environment error.

Add --upgrade to see whether the flag would unblock it. This is a read-only check – it changes nothing.

note

If the upgrade verification results include WARN version-guard/bypassed, the SKIP_UPGRADE_PATH_CHECK environment variable overrode the version guard's check that the release supports upgrading from your deployed version. A run that would otherwise pass then reports WARN – review warnings. Releases skipped this way may carry one-time migrations that never ran, and nothing detects that afterwards. Use this override only when instructed by Omada Support.

Getting support

If you need to raise a support case, collect diagnostics with the script shipped in the release, as described in Collecting debug logs. Attach the resulting archive, together with:

• The full installer log.
• logs/summary.json from the run directory.
• logs/health-check.log.
• logs/assert-upgrade-results.xml (upgrade verification results).
• logs/assert-upgrade-health-check.log.
• The version you upgraded from (step 1 of the pre-upgrade checklist) and to.

important

If an upgrade fails partway, leave the environment as it is. --upgrade deliberately prevents the installer from tearing anything down so that the failed state is available for diagnosis.