Upgrade guide

This page describes how to upgrade your Omada Identity Cloud Private environment to a new version. It covers the pre-upgrade checklist, the upgrade process itself, and what to do if the installer stops.

note

There is no separate upgrade tool for Omada Identity Cloud Private. You upgrade an environment by running the new release's installer against the workspace you already have. The installer detects the existing environment and updates it in place: Terraform reconciles the infrastructure, Ansible re-applies the configuration, and the new container images and application packages are rolled out.

Before you start upgrading

The assumption of the following sections is that you have an existing Omada Identity Cloud Private environment, and you want to upgrade it to a new release. Therefore, you are familiar with the Installation Script mode, because an upgrade uses the same script and the same deployment-config.json, with one additional flag.

tip

Throughout this document, <version> stands for the release you are upgrading to – for example 16.51.90. You do not type or choose this value separately: it is the version that appears in the /releases/<version>/ path segment of the URL Omada provides, and the installer reads it from there. If that segment is missing from the URL, the installer stops before its first phase.

See the following sections for the prerequisites, limitations of the upgrade process, and expected impact of the upgrade.

Prerequisites

The prerequisites for an upgrade are the same as for a fresh installation. See the following pages for details:

• General prerequisites
• System and networking prerequisites
• Installation Script mode for the exact steps of the installer, for reference.

Limitations and expected impact of the upgrade process

This procedure does not cover:

• Moving an environment to a different Azure region. A resource group's region cannot be changed.
• Rolling back to an earlier release. Downgrades are blocked by design (see If the installer stops), and there is no supported down-migration for database schema changes. Recovery from a bad release is a restore from backup – take one before you start.
• Changing the workspace or project name of an existing environment.

impact

You should expect the following impact during an upgrade:

Application packages are deployed by updating the WEBSITE_RUN_FROM_PACKAGE app setting on each App Service to point at the new package. There are no deployment slots, so each App Service has a short outage while it restarts.

AKS workloads roll out across the Ansible sequence: database baseline first, then AKS infrastructure, Enterprise Server, and the Cloud Application Gateway workers, then full Enterprise Server configuration, and finally RoPE, CAG operations, and OPS in a second AKS pass once the first is healthy. Therefore, there is a window during which components are running mixed versions. An upgrade also cannot start while an import is running – see step 3 of the pre-upgrade checklist.

Therefore, you should plan the upgrade inside a maintenance window.

Updating your configuration

The new installer package also ships its own deployment-config.json. For a new release, you can replace it with your existing deployment-config.json (for details, see Configure deployment-config.json again) and update the release section to point at the new package:

• release.package_url to the new oicp_<version>.zip.

The installer downloads and extracts the package during its release phase. Everything else in the file stays as it is: the same subscription, workspace, project, region, backend, certificate, and Entra groups remain as in your existing installation.

Pre-upgrade checklist

While you should go through the whole list of pre-upgrade steps, steps 3 and 4 are the ones that most often turn a routine upgrade into an incident.

1. Record the version you're on.

   The installer reads this from a tag on your workspace resource group:

   Tool: Bash shell

   az group show --name <global-prefix>-rg --query tags.ProductVersion -o tsv

   • A version number, for example 16.51.88 – normal. Note it down.
   • Empty – also normal if you've never upgraded since installing a release older than the version guard. Your first upgrade requires --upgrade and reports itself as unverified; see the first row in If the installer stops. This is expected and self-correcting.

2. Take a database backup.

   Automated backups are enabled in Azure SQL, but take an explicit point-in-time reference before a release that includes schema migrations, and confirm you know how to restore it. There is no supported down-migration.

   The installer enforces this. Before it captures the pre-upgrade snapshot, it checks that point-in-time restore is available on every product database that already exists, and refuses to continue if it is not:

    Point-in-time restore is not available on existing database '<db>' — refusing to upgrade without a restore anchor. Investigate before retrying with --upgrade.

3. Confirm no import is running.

   The installer checks this for you and will refuse to proceed (see If the installer stops), but finding out now is much cheaper than finding out mid-window. Run this read-only query against your Enterprise Server database (<global-prefix>-ois-db):

   SELECT Id, ImportType, State, StartTime
   FROM [isp].[Import]
   WHERE ImportType NOT IN (2, 5)          -- exclude Preview / ExportPreview
     AND State IN ('Queued', 'Started', 'StagingCompleted',
                   'Dequeued', 'ReadyToStart', 'WaitingForCoreHorizons')
   ORDER BY Id;

   Any row returned is an import that has not finished. Let it complete, or cancel it, before you start. A Queued row whose StartTime is more than two days old is treated as stale and will not block the upgrade – but it is worth investigating separately.

   The installer's own check queries Import Service rather than the table directly, so treat the query above as an equivalent read-only check you can run ahead of time, not as the same call.

   Once past the gate, the installer pauses new imports for the duration of the run (phase pause_imports, right after the import gate) and releases the pause when the run ends, however it ends. This happens on any run against an existing environment, not only an --upgrade run. It then waits for any import that started between the gate and the pause to finish, for up to --drain-timeout minutes (default 30); --skip-import-drain pauses without waiting, and --drain-timeout 0 fails immediately instead of waiting.

4. Confirm your Azure sign-in and permissions.

   Tool: Bash shell

   az account show --query "{subscription:name, id:id, user:user.name}" -o table
   az account get-access-token --query expiresOn -o tsv

   Use the same identity and subscription as the original installation. The installer runs its own permission and quota pre-flight checks and will tell you what's missing.

5. Check quota headroom.

   If the release adds infrastructure, the upgrade needs room for it. The installer checks this as part of its pre-flight checks; you can check ahead of time with az vm list-usage for your region.

Run the upgrade

Unpack the new installer package.

warning

Do not use the installer you used for the original deployment. The installer and the safety checks described on this page ship with the release, so an older installer cannot protect an upgrade to a newer version.

Run the installer package with the --upgrade flag:

Tool: Bash shell

unzip oicp_install_<version>.zip
cd scripts/install

./install.sh --config ./deployment-config.json --upgrade --verbose

note

In the interactive mode, expect a few yes/no confirmation prompts in phase 7 – before each terraform apply, and before each of the two artifact-publish steps. Each defaults to yes. If a required certificate is missing or invalid, secrets_validate presents a menu rather than a prompt. Use --non-interactive (as in Long runs and interruptions) to resolve each prompt to its default for unattended runs.

The rollback prompt described in Installation Script mode never appears on an upgrade run – --upgrade disables rollback outright.

The --upgrade flag ensures that the installer treats the operation as an intentional forward version change. It has three effects:

1. It satisfies the version guard for a forward version change, and for the one-time case where your environment carries no version tag yet.

2. It disables rollback. The installer forces ROLLBACK_ON_FAILURE=false and prints:

   --upgrade active: forcing ROLLBACK_ON_FAILURE=false (was 'true') – a failed upgrade must never reach terraform destroy.

   If a later phase fails, the installer stops and leaves everything in place for investigation. It will not tear your environment down. If something internally still attempts a rollback, it is refused explicitly:

   Rollback refused: --upgrade is active. A failed upgrade must never run terraform destroy against workspace '<workspace_prefix>'. Resources are left as-is for manual investigation.

3. It is recorded in the run's summary.json as "run_mode": "upgrade".

note

The --upgrade flag does not override the downgrade block.

What the installer does

The upgrade runs the same phases documented in Stage dependency overview, plus the following additions that run immediately after the release package is extracted, before anything is changed:

 1  preflight            OS, disk, network connectivity
 2  config               load and validate deployment-config.json
 3  tools                install/verify terraform, ansible, az, kubectl, kubelogin,...
 4  azure                sign-in, permission and quota checks
 5  terraform_backend    locate the existing Terraform state backend
 6  release              download and extract the release package
 6b version_guard        ◀ compares the release version to your environment's tag
 6c import_gate          ◀ refuses to continue while an import is active
 6c.5 upgrade_snapshot   ◀ upgrade mode only – captures pre-upgrade state as the restore anchor
─── nothing above this line changes your environment ──────────────────────────
 6d pause_imports        ◀ pauses new imports for the run, then drains any already in flight
 7  terraform_iac_shared → artifact publish → tfvars_sync → secrets_validate → terraform_iac
 8  ansible: database → AKS phase 1 → Enterprise Server → AKS phase 2
 9  health_check
 9b environment_snapshot_post  captures post-deployment state (also attempted if a phase fails)
 9c upgrade_verify       ◀ upgrade mode only – marks the upgrade complete

The version_guard and import_gate are true guards – if either blocks, your environment is untouched. upgrade_snapshot writes only to the artifacts storage account, so your environment is still untouched at that point. pause_imports runs on any run against an existing environment, not only in upgrade mode, and is where the first change is made.

Long runs and interruptions

An upgrade takes a long time. On a remote host, run it detached so a dropped connection cannot hinder it:

Tool: Bash shell

nohup ./install.sh \
  --config ./deployment-config.json \
  --upgrade --non-interactive --verbose \
  > install-upgrade.log 2>&1 < /dev/null & disown

Monitor with tail -f install-upgrade.log, or follow the installer's own log under Deployment_OICP_<version>/run-<project_name>-<workspace_prefix>/logs/install-<timestamp>.log as described in Run the installation script.

If a phase fails for a transient reason, re-run the same command with --resume appended to continue from the last completed phase rather than starting over. Read If the installer stops and the error message first.

note

--resume skips phases that already completed, which can include the version guard. The active-import check, the pre-upgrade snapshot, and the import pause always run on every invocation – a stale checkpoint from an earlier attempt is never trusted for any of the three. A re-run adopts the snapshot the earlier attempt captured rather than taking a new one, so the restore anchor stays fixed at the first attempt.

After the upgrade

1. Confirm the version advanced.

   Tool: Bash shell

   az group show --name <global-prefix>-rg --query tags.ProductVersion -o tsv

   This must now report <version>. If your environment previously had no tag, it has one now, and your next upgrade will be fully verified.

2. Read the run summary.

   Tool: Bash shell

   cat Deployment_OICP_<version>/run-<project_name>-<workspace_prefix>/logs/summary.json

   Confirm "status": "success", "run_mode": "upgrade", and "rollback_on_failure": false. The same file records "restore_anchor_utc" – the UTC instant the pre-upgrade snapshot was taken, and therefore the point in time to restore to if you need to roll back – along with "drain_timeout_seconds" and "skip_import_drain" for the run.

3. Review the health check which is run by the installer as its final phase. It reports any issues that need attention before you close the maintenance window.

   tip

   See Post-deployment health check for how to read the report and re-run it on demand.

4. Re-enable anything you paused: scheduled imports, monitoring silences, maintenance-mode banners.

5. Run a functional verification: sign in to Enterprise Server, confirm the version shown, and run one small import end to end before closing the maintenance window.

Troubleshooting

The following sections describe the most common issues that can occur during an upgrade and how to resolve them.

If the installer stops

Every message below is a deliberate refusal before any change was made. Your environment is in the state it was in before you started.

Message	What it means	What to do
Resource group '<rg>' has no 'ProductVersion' tag – its prior version cannot be verified. Pass --upgrade to proceed anyway.	Your environment was installed from a release older than the version guard, so there is no recorded version to compare against. This leniency is specific to the current release line. From a future major version onward, a tag-absent resource group will fail unconditionally, with no --upgrade override – the same as a downgrade. If your environment has never completed a guarded upgrade, do so well before that boundary.	Add --upgrade as in Run the upgrade section. The installer will proceed and warn that the starting point is unverified. This run writes the tag, so every future upgrade is fully verified. One-time, per environment.
Target version <new> is higher than the deployed version <old> on '<rg>'. Pass --upgrade to acknowledge this upgrade.	A normal forward upgrade, without the acknowledgement flag.	Add --upgrade.
Target version <old> is lower than the deployed version <new> on '<rg>'. This is a downgrade and is always blocked – no flag can override this.	The package you are installing is older than what is deployed.	Check whether you unpacked the right release. Important: Rolling an older release over a newer database has no clean recovery path. If this is a deliberate rollback, it must be planned with Omada support; the resource group tag has to be corrected out-of-band first.
Resource group '<rg>' is already at <version> – repair/retry, proceeding.	Not an error. You are re-running the same version, for example to finish an interrupted install.	Nothing – the run continues. No flag needed.
--upgrade was given but resource group '<rg>' does not exist – there is no existing environment to upgrade.	There is nothing at that name to upgrade.	Almost always a wrong --workspace or deployment.workspace_prefix. Check it. If this really is a first install, drop --upgrade and follow the Installation Script mode documentation.
Aborting before any change is made: N active import(s) detected – Id <n>, State='Started', StartTime=...	An import is in flight. Restarting components now risks corrupting it.	Wait for it to finish, or cancel it, then re-run. See step 3 of the pre-upgrade checklist for the query. If a hotfix genuinely cannot wait and you accept the risk, see Bypassing the active import check.
Discounted: Id <n>, State='Queued', ... Queued row stale (StartTime more than 2 days ago)	Not an error. A queued import older than two days is treated as abandoned and does not block.	Nothing – the run continues. Worth investigating the stale row separately.
<db> on <server> is not reachable yet, or isp.Import does not exist yet (during a fresh install)	Informational. Normal before the database baseline has run.	Nothing – the run continues.
<db> on <server> is not reachable yet, or isp.Import does not exist yet (during an upgrade)	There should already be a reachable database with this table at this point, so your database connectivity or sign-in is wrong.	Check step 4 of the pre-upgrade checklist.
Product version guard blocked this deployment (resource group: <rg>, target version: <version>).	The wrapper line that follows any of the version guard messages above.	Act on the specific message printed just above it.

Bypassing the active import check

There might be a case where you need to upgrade while an import is active, and you understand the risk. The installer refuses to continue in that case, but you can bypass the check with an environment variable.

If you need a forced hotfix, run:

Tool: Bash shell

SKIP_IMPORT_CHECK=true ./install.sh \
  --config ./deployment-config.json --upgrade --verbose --skip-import-drain

SKIP_IMPORT_CHECK bypasses the import_gate phase only. The pause_imports phase that follows still waits for the running import to finish, for up to --drain-timeout minutes (default: 30), so a forced hotfix needs --skip-import-drain as well. Otherwise, the run stalls for half an hour at the point you were trying to skip.

The gate bypass is logged where the check would have run:

Active-import safety check SKIPPED (SKIP_IMPORT_CHECK=true) – forced override, proceeding without verification.

warning

The check exists because restarting Enterprise Server, RoPE, or CAG/OPS while an import is mid-flight can leave that import's data in an inconsistent state. Bypassing it means you may have to re-run the affected import afterwards, and in the worst case, clean up partially-written data.

Checking the version guard on its own

You can run the version guard by hand at any time, without starting an upgrade – it's the same check the installer performs:

Tool: Bash shell

./scripts/check-product-version.sh \
  --resource-group <global-prefix>-rg \
  --target-version <version>

The exit codes are:

Exit code	Meaning
0	Safe to proceed.
1	Blocked.
2	Usage or environment error.

Add --upgrade to see whether the flag would unblock it. This is a read-only check – it changes nothing.

Getting support

If you need to raise a support case, collect diagnostics with the script shipped in the release, as described in Collecting debug logs. Attach the resulting archive, together with:

• The full installer log.
• logs/summary.json from the run directory.
• logs/health-check.log.
• The version you upgraded from (step 1 of the pre-upgrade checklist) and to.

important

If an upgrade fails part-way, leave the environment as it is. --upgrade deliberately prevents the installer from tearing anything down so that the failed state is available for diagnosis.