Monitoring system configuration

This page explains how to find logs, search them, check whether imports succeeded, and identify errors in an Omada Identity Cloud Private environment running on Azure (AKS, App Services, SQL, Service Bus). It also describes what to collect before raising a support ticket with Omada.

Tools overview

There are several ways to examine the same environment. The right tool depends on your access level and the question you are asking.

Tool	Best for	Access required
Health check script (scripts/health_check.sh)	Checking whether the whole environment is healthy and whether there are errors in the last 30 minutes	Shell with az CLI login and RBAC reader role
Azure Log Analytics (Portal > Logs / KQL)	Searching application and container logs across all services by time, component, user, or correlation ID	Azure Portal read access to the Log Analytics workspace
Application Insights (Portal)	End-to-end request traces, exceptions, and failures for App Services and Function Apps	Azure Portal read access to Application Insights
OIS web portal (Event Log, Mail Log, Import status)	Functional and business-level events visible to Omada admins; import and email status	Omada admin or operator login with read permissions
Azure Service Bus (Portal)	Stuck or failed import messages (dead-letter queues)	Azure Portal read access to the Service Bus namespaces
State SQL DB ([state].*)	Definitive per-import outcome and which component failed	SQL read access (advanced; usually arranged with Omada support)

For most day-to-day questions, the health check combined with Log Analytics is sufficient. Imports have their own section because they involve the most moving parts.

Finding your environment

Everything in your environment is named from a single workspace prefix.

• global_prefix = oisaas-<workspace> – for example, oisaas-prod, oisaas-uat, oisaas-kl0001.
• Resource group = <global_prefix>-rg – for example, oisaas-prod-rg.

Throughout this page, replace <gp> with your own global_prefix. If you do not know it, it is the prefix on every resource in your resource group in the Azure Portal.

Azure Portal resource locations

Resource	Name	Portal location
Log Analytics Workspace	<gp>-law	Monitor > Log Analytics workspaces
Application Insights	<gp>-appins	Monitor > Application Insights
AKS cluster (Enterprise Server, RoPE, Timer, CAGOps)	<gp>-aks	Kubernetes services
Web Apps	<gp>-wa-<svc> (e.g. -wa-copsapi, -wa-is, -wa-ci)	App Services
Function Apps	<gp>-fa-<svc>	Function App
Service Bus namespaces	<gp>-en-bus, <gp>-is-bus, <gp>-ps-bus, <gp>-ci-bus	Service Bus
Action group (alert recipients)	<gp>-lrdo-alerts	Monitor > Alerts > Action groups

Log routing

Logs from different components flow to different destinations:

• Enterprise Server (ES) application logs flow to the custom table OIS_CL in the Log Analytics workspace, via the ES Log Ingestion API – not via container logs.
• Container and pod logs flow to ContainerLogV2. In this deployment, Container Insights runs the Linux Azure Monitor Agent, and the only application workload on Linux nodes is LRDO (its own dedicated, tainted workload=lrdo node pool). The Windows workloads – ES, RoPE, Timer, CAGOps – run on the Windows node pool and do not flow into ContainerLogV2. ES uses OIS_CL instead. For RoPE, Timer, and CAGOps, use kubectl logs directly (see AKS container logs).
• App Services and Function Apps (COPS API, Import Service, Provisioning, Core Ingestion, History Tracking, ODS) send telemetry to Application Insights (<gp>-appins), which is backed by the same workspace.

Log retention: OIS_CL is interactive (queryable) for 60 days and archived to 180 days; container and platform logs follow the workspace retention setting (typically 30 days in dev, 90 days in production). These are recommended defaults and can be adjusted to meet your organization's requirements.

Quick triage – health check

The repository ships a script that checks the whole environment and prints a PASS/WARN/FAIL report. Run this first whenever something appears to be wrong.

Prerequisites

• az CLI installed and logged in (az login), pointed at the correct subscription (az account set --subscription <sub>).
• jq, curl, and kubectl available.
• An identity with reader-level RBAC on the resource group. The script checks the exact permissions it needs and warns if any are missing – see docs/HEALTH_CHECK_PLAN.md for the minimum role.

Running the script

Tool: Bash shell

./scripts/health_check.sh --global-prefix <gp>
# To look further back than the default 30-minute error window:
./scripts/health_check.sh --global-prefix <gp> --log-window-minutes 120

Exit code 0 means all checks passed or only warnings were raised. Exit code 1 means at least one check failed.

What the checks mean

Check	What it tells you	If it warns or fails, go to…
1. Endpoint ping	Each Web App, Function App, and ES instance responds	App was down or cold – re-run; if persistent, see Application Insights
2. AKS pod readiness	ES, RoPE, Timer, and CAGOps deployments have ready pods	Use kubectl logs for Windows pods; for ES errors use OIS_CL queries
3. Version check	Running build matches the expected deployed version	Deployment or release issue – escalate
4. Error log report	Count of Error/Fatal in OIS_CL and error/critical in LRDO logs over the window	See Searching logs in Azure Log Analytics
5. Service Bus dead-letter	Messages stuck in dead-letter queues	See Service Bus dead-letter queues

Check 4 prints a per-component breakdown for OIS_CL and a per-pod breakdown for LRDO. Use those component and pod names to narrow the KQL queries in the next section.

Searching logs in Azure Log Analytics

Azure Log Analytics is the primary way to search application and infrastructure logs across the environment using KQL (Kusto Query Language).

To access it, go to the Azure Portal > Monitor > Logs, or open the <gp>-law workspace > Logs. Paste a query, set the time range in the top right, and select Run.

note

The time picker in the top right usually overrides any ago(...) you write in the query. Either set the picker, or use an explicit where TimeGenerated > ago(...) and set the picker to Set in query.

Enterprise Server logs – the OIS_CL table

OIS_CL holds Enterprise Server application logs. Key columns:

Column	Meaning
TimeGenerated	When the event was logged
Level_d	Severity: 1 = Trace, 2 = Debug, 3 = Info, 4 = Error, 5 = Fatal
Component_s	Which ES component produced it
Message	The log message
Exception_s	Exception text and stack trace (when present)
CorrelationId	Ties related entries together across a request
Username_s / UserId_s	The user in context (if any)

All errors and fatals in the last hour:

Format: KQL

OIS_CL
| where TimeGenerated > ago(1h)
| where Level_d >= 4
| project TimeGenerated, Level_d, Component_s, Message, Exception_s, CorrelationId
| order by TimeGenerated desc

Error counts by component:

Format: KQL

OIS_CL
| where TimeGenerated > ago(30m) and Level_d >= 4
| summarize count() by Component_s, Level_d
| order by count_ desc

All entries for a single correlation ID:

Format: KQL

OIS_CL
| where CorrelationId == "<paste-correlation-id>"
| project TimeGenerated, Level_d, Component_s, Message, Exception_s
| order by TimeGenerated asc

Full-text search across messages and exceptions:

Format: KQL

OIS_CL
| where TimeGenerated > ago(24h)
| where Message has "timeout" or Exception_s has "timeout"
| project TimeGenerated, Level_d, Component_s, Message, Exception_s
| order by TimeGenerated desc

Activity for a specific user:

Format: KQL

OIS_CL
| where TimeGenerated > ago(24h)
| where Username_s =~ "<username>"
| project TimeGenerated, Level_d, Component_s, Message
| order by TimeGenerated desc

AKS container logs – ContainerLogV2

note

In this deployment, ContainerLogV2 contains LRDO (Linux) pod logs and Kubernetes system pods only. Container Insights runs the Linux monitor agent, and LRDO is the only application workload on the Linux node pool. The Windows workloads – ES, RoPE, Timer, CAGOps – are not in this table. ES logs go to OIS_CL; for RoPE, Timer, and CAGOps, use kubectl logs directly (see below).

Useful columns: TimeGenerated, PodName, ContainerName, LogLevel, LogMessage, PodNamespace.

LRDO errors:

Format: KQL

ContainerLogV2
| where TimeGenerated > ago(1h)
| where PodName startswith "lrdo"
| where LogLevel in~ ("critical", "error")
      or LogMessage has_any ("fail", "error", "exception")
| project TimeGenerated, PodName, LogLevel, LogMessage
| order by TimeGenerated desc

Which pods exist and recently logged:

Format: KQL

ContainerLogV2
| where TimeGenerated > ago(1h)
| summarize lines = count(), last = max(TimeGenerated) by PodName, PodNamespace
| order by last desc

For Windows workloads (ES, RoPE, Timer, CAGOps), read logs live with kubectl (requires AKS access):

Tool: Bash shell

kubectl logs -n <namespace> deploy/<gp>-es --tail=200 -f
# Also: -rope, -timer, -cagops

For ES application-level logs (errors, exceptions), prefer the OIS_CL queries above.

Application Insights

App Services and Function Apps (COPS API, Import Service, Provisioning Service, Core Ingestion, History Tracking, ODS, and their Function App variants) send request, dependency, and exception telemetry to <gp>-appins.

To access it, go to the Azure Portal > Application Insights > <gp>-appins.

Useful blades, no KQL required:

• Failures – failed requests and exceptions, grouped by operation, with drill-down to individual occurrences.
• Performance – slow operations and dependencies, such as slow SQL or Service Bus calls.
• Transaction search – free-text search across recent telemetry; click any item to see the end-to-end transaction with every dependency and trace for that operation.

Recent failed requests:

Format: KQL

requests
| where timestamp > ago(1h) and success == false
| project timestamp, name, resultCode, duration, operation_Id, cloud_RoleName
| order by timestamp desc

Recent exceptions:

Format: KQL

exceptions
| where timestamp > ago(1h)
| project timestamp, cloud_RoleName, type, outerMessage, operation_Id
| order by timestamp desc

Follow one operation end-to-end:

Format: KQL

union requests, traces, exceptions, dependencies
| where operation_Id == "<operation-id>"
| project timestamp, itemType, message = coalesce(message, name, outerMessage), severityLevel
| order by timestamp asc

tip

Application Insights is best for App Service and Function App behavior (HTTP requests, dependencies, exceptions). For Enterprise Server application logs, use OIS_CL. For raw pod output, use ContainerLogV2.

Checking imports (Horizon ingestion)

Imports are the most common thing an operations team needs to verify. In Cloud Private, imports run through the Horizon ingestion pipeline – an event-driven set of services connected by Azure Service Bus and coordinated by a StateService that records the outcome of every import.

Each import has a lifecycle the StateService tracks:

Stage	Description
Initializing	Import is starting up.
Running	Import is actively processing data.
AllDataReceived	All source data has been received.
AllDataProjected	All data has been projected into the target system.
Finishing	Import is completing final steps.
Finished	Import has completed.

The import ends in one of three results: Success, Failure, or Stopped.

note

If no activity occurs while the import is Running, it times out and is marked Stale, then Failed.

Identifiers

Identifier	What it is	Where it appears
ImportId	Sequential integer for one import run	State DB, logs, App Insights (IngestionImportId)
ImportServiceImportUid	The import's GUID	State DB, request tracking
TenantId	Customer or environment ID	Everywhere (App Insights dimension Tenant)
CorrelationId	Trace ID across components	Logs, App Insights
SystemId	Source system ID	Per-system message counts

There are four ways to check an import, from easiest to most detailed. Start with Path A.

Path A – OIS web portal

The Import status / Data source synchronization view in the Horizons UI shows recent import runs and a component heartbeat (Healthy / Unhealthy / Recovered / Unknown). The Data Exchange Log (DataExchangeLogDlg.aspx) shows results of data exchange runs: configuration name, timestamp, duration, success or failure, and a result message.

Path B – Log Analytics and Application Insights

Find everything logged for a specific import. In Application Insights > Logs:

Format: KQL

traces
| where timestamp > ago(24h)
| where customDimensions.IngestionImportId == "<ImportId>"
| project timestamp, severityLevel, message, customDimensions
| order by timestamp asc

Format: KQL

exceptions
| where timestamp > ago(24h)
| where customDimensions.IngestionImportId == "<ImportId>"
| project timestamp, cloud_RoleName, type, outerMessage
| order by timestamp desc

Severity levels: 0 = Verbose, 1 = Info, 2 = Warning, 3 = Error, 4 = Critical.

Path C – State SQL DB (advanced)

The StateService database (schema [state]) is the source of truth for whether an import succeeded and which component failed. SQL access is typically arranged with Omada support.

Latest state and failure reason for one import:

Format: SQL

SELECT TOP 1
    ImportId, ImportServiceImportUid, State, Result, Description, Started, LastChange
FROM   [state].[ImportStates]
WHERE  ImportId = @ImportId
ORDER  BY Version DESC;

Which components had message failures:

Format: SQL

SELECT Component, SystemId, SentCount, ReceivedSuccessCount, ReceivedFailureCount
FROM   [state].[ImportMessageCounts]
WHERE  ImportId = @ImportId
  AND  ReceivedFailureCount > 0;

Per-component completion progress:

Format: SQL

SELECT Component, SystemId, DataObjectType, CompleteTimestamp
FROM   [state].[ImportComponentProcessings]
WHERE  ImportId = @ImportId
ORDER  BY Component;

Path D – Service Bus dead-letter queues

When a message fails repeatedly (exceeding the maximum delivery count), it lands in a dead-letter queue (DLQ). A non-zero DLQ count is a strong signal that an import is stuck or failing.

The health check (check 5) already reports DLQ counts per subscription across all namespaces. To inspect in the Portal: Service Bus > <gp>-en-bus (and -is-bus, -ps-bus, -ci-bus) > Topics > a topic > Subscriptions > Dead-letter message count > use Service Bus Explorer to peek at the dead-lettered messages. Each message carries a DeadLetterReason and DeadLetterErrorDescription explaining why it failed.

OIS web portal log viewers

The Omada web portal provides in-product log viewers showing functional and business-level events. These require an Omada login with the relevant read permission. Paths are relative to your portal base URL.

Viewer	Page	Purpose	Access
Event Log	EventLogEntryLst.aspx	Main system events and errors – filter by Level, Component, Category, and date range; click an entry for full exception detail (maximum 10,000 rows)	EventLog read
Mail Log	MailLogLst.aspx	Whether notification emails were sent; filter by user, object, or event definition	EmailLog read
Code Method Log	CodeMethodLogLst.aspx	Custom code-method executions and their success or failure	CodeMethodLog read
Data Exchange Log	DataExchangeLogDlg.aspx	Results of data exchange (export/import) runs	–
Configuration Change Import	ConfigurationChangeImport.aspx	Importing configuration change-sets; per-row status, test mode, and live progress	Administrator
Integrity Checks	IntegrityCheckLst.aspx	Database consistency and integrity diagnostics	Administrator
Build Info / About	BuildInfo.aspx, About.aspx	Exact product version and assembly information (useful when raising a support ticket)	–

Configured alerts

The environment ships with one automated alert:

• <gp>-lrdo-cronjob-failure – a scheduled Log Analytics alert that fires when an LRDO pod logs a critical or error event (or a message containing fail, error, or exception) in AKS. It evaluates every 15 minutes. Notifications go to the <gp>-lrdo-alerts action group (by default, the Omada admin address). To have your own operations distribution list notified, contact Omada to add a receiver to that action group.

To view alert history: Azure Portal > Monitor > Alerts, filtered to your resource group.

note

Only the LRDO alert is provisioned by default. If you want additional alerts – for example, on OIS_CL Error/Fatal counts or on Service Bus dead-letter counts – contact Omada. The same KQL queries used in Searching logs in Azure Log Analytics can back a new alert rule.

Appendix

Severity levels

Source	Field	Values
OIS_CL (Log Analytics)	Level_d	1 Trace · 2 Debug · 3 Info · 4 Error · 5 Fatal
ContainerLogV2	LogLevel	info, warning, error, critical
Application Insights	severityLevel	0 Verbose · 1 Info · 2 Warning · 3 Error · 4 Critical
Import outcome ([state].[ImportStates])	Result	None · Stopped · Success · Failure
Import lifecycle	State	Initializing > Running > AllDataReceived > AllDataProjected > Finishing > Finished (or Stale)

KQL quick reference

Format: KQL

// Last hour of ES errors
OIS_CL | where TimeGenerated > ago(1h) and Level_d >= 4

// LRDO container errors (ContainerLogV2 = Linux/LRDO pods only)
ContainerLogV2 | where PodName startswith "lrdo" and LogLevel in~ ("error","critical")

// Failed app requests
requests | where success == false and timestamp > ago(1h)

// Everything for one import (App Insights)
traces | where customDimensions.IngestionImportId == "<ImportId>"