Threshold
This section provides information on the use and configuration of the import and provisioning thresholds.
Using thresholds
The Configure threshold tasks located on the individual system’s page are used in the system onboarding process to allow you to set threshold values for the data import process from Omada Identity Data Warehouse and for the provisioning of rights from Omada Provisioning Service.
You can set up threshold values from the two tasks named Configure thresholds, located under the Data import and Provisioning headers respectively.
Since the Omada Identity Cloud Version 14 update 7, Cloud update 3v14.0.8 (Update 8) release, the Threshold functionality is available for .NET-based collectors configured with content as Identity data, and applied to such data objects like Identity, IdentityOwner, Context, ContextOwner, ContextAssignment.
For .NET-based collectors with content configured as Both, it will only apply to the access data objects, such as, Account, Resource, ResourceAssignment.
In addition, the Configure thresholds menu item is hidden for systems where the feature is not available.
Even though the two tasks share the same name, they are not connected, so you can set up values for either of the tasks without using the other task. The Configure thresholds task for Provisioning only appears if you use Omada Provisioning Service as the provisioning method.
When you change the threshold settings for either of the two Configure thresholds tasks, the changes take effect immediately. Note, however, that you must also click Commit settings for the Configure thresholds task for Provisioning.
Data import thresholds
In this section you can find detailed information on the use and configuration of the data import thresholds.
Configuring thresholds for data import
The data import threshold feature is intended to prevent unintentional or unlucky incidents in Omada Identity in case of errors with a system’s imported data. Examples of such errors could be an incomplete CSV file or accidental changes in connection details.
The Data import threshold feature works by calculating the percentile rate of created, changed, or deleted objects since the last import. It suspends the import process if the rate is equal or higher than the configured maximum value. The system owner is notified about the suspension and can decide to either resume or discard the import.
The threshold calculation applies to accounts, resources and resource assignments.
The systems that you register to Omada Identity may include different number of objects and different number of changes. For this reason, you must set the threshold values for each system individually.
By default, all threshold values are set to zero, which means that thresholds are disabled.
You can set different values (in percentage) for New objects, Modified objects, and Deleted objects. The value must be a whole number (integer).
An object is considered as changed/modified even if you have changed only the master data, for example its compliance state. When you specify the threshold for changed objects, you must consider changes to master data when you set the value.
Applying thresholds for data import
The threshold settings are applied during the import process.
If the threshold value is zero or the number of existing objects in Omada Identity Data Warehouse is zero, for example during the initial load, then the threshold calculation is skipped.
The percentage of created, changed, and deleted accounts, resources, and resource assignments are calculated. If any of these elements exceed the configured percentage, the import of the system is suspended. Several things happen when a system is suspended.
A set of tables is created in the staging database to hold the suspended data.
The tables are created in a schema named by the SystemID with (suspended) added at the end. Access data for the suspended system is moved to those tables, where you can inspect it if you need to.
Omada Identity sends an email to the system owner if you have assigned an owner to the system and specified an email address for that system owner.
The import proceeds differently depending on the content of the system. If the system only contains access data, the import proceeds with the other systems in the system category. If the system contains identity data, the import skips the other systems in the system category, and proceeds with the next system category.
A threshold error is reported on the individual system page. The run details contain detailed error messages about the objects and actual percentages of the exceeded thresholds.
A suspended system is excluded from imports, until you have decided to resume or discard the import. If a suspended system contains identity data, all systems in the system category are excluded from imports.
Evaluating exceeded thresholds for data import
When a threshold value is exceeded, the dialog box named A configured threshold was exceeded appears. System owners must evaluate if the exceeded threshold is intended or not. You can inspect the suspended data if you need to, by browsing the data in Omada Identity Data Warehouse.
Options:
-
Accept data and commit data on next import: Resume the import of the suspended data. If you choose to resume the import, the suspended data is moved to the staging tables and imported on the next import.
-
Reject data and stage new data on next import: reject the import of the suspended data. If you choose to reject the suspended data, the same data is extracted from the target system on the next import.
When you have made your decision, the system is no longer suspended. This means that the system is no longer excluded from imports.
If the system contains identity data, none of the systems in the system category are excluded from imports any longer.
Evaluating exceeded thresholds for provisioning
When a threshold is exceeded, the overall system status shows Error in the Status column. All provisioning is suspended, including operations that are not exceeded, except jobs of high priority, as mentioned in the previous section.
Note that all provisioning is suspended regardless of which threshold was exceeded. In other words, updates and deletes are also suspended even if it was only the create threshold that was exceeded.
The status of Enable provisioning on the individual system page now displays Error. A new link is presented in the Details column: A configured threshold was exceeded.
When you click this link, you are presented with the following options:
Options:
- Resume processing till thresholds are exceeded again: when you select this option, provisioning is resumed, and the calculation of thresholds statistics is started.
- Allow processing current pending jobs and resume processing: when you select this option, provisioning is also resumed, but the calculation of threshold statistics is not started until all currently pending jobs have run.
If new jobs are received during processing the already pending jobs, these are included in the threshold statistics.
- Suspend threshold settings for a specified number of hours: Use this option if you know that there are many tasks to complete within a given timeframe. The option allows you to suspend the checking of thresholds for a specified number of hours.
When you suspend thresholds, the system goes into a warning state and shows the following on the individual system page.
To resume the checking of thresholds earlier, you can select to clear threshold statistics and resume processing immediately. To do so, open the dialog from the Ellipsis menu (…) in the Provisioning jobs dialog box, and select Clear threshold statistics. You can also select one of the other two options, change the number of hours to suspend threshold calculation.
Provisioning thresholds
In this section you can find detailed information on the use and configuration of the data provisioning thresholds.
Configure thresholds for provisioning
The OPS Thresholds feature is a functionality designed to prevent that a high number of unwanted provisioning tasks are run. The feature works as an emergency brake where provisioning to a specific target system is suspended, when the number of performed operations exceeds a defined number within a defined time interval.
It is important to note that the operations that you have started before the threshold is reached are still processed. You cannot set up the system to roll back the operations.
As mentioned previously, you configure thresholds for each system individually and can do this from the individual system page by opening the Configure thresholds task in the Provisioning section:
You do not need to configure or activate the task, so you must set the status of the Configure threshold task manually from the Edit status link to track the progress of this task.
If you click the Configure thresholds task, the following dialog box opens:
The dialog box shows the defined threshold values and the defined interval. You can define a threshold for Create, Update, and Delete. Create includes the actions Create and CreateOrUpdate. Delete includes the actions Delete and DeleteIfExists.
In the Value, enter the number of tasks of the selected operation that is allowed before the threshold is exceeded. Set the value to 0 (zero) to disable the threshold for the selected operation.
In Interval, enter the interval in the following format hh:mm:ss. Here, hh is the number of hours, mm is the number of minutes, and ss is the number of seconds.
For example, if you set Value to 50 and Interval to 00:10:00, provisioning is suspended if 50 tasks of the selected operation happens within 10 minutes.
The Threshold value is not an exact number. Because of the multithreaded nature and the options for multiple instances, there is no way of guaranteeing that the system is stopped at the exact number when the threshold is exceeded.
Threshold statistics
OPS calculates the threshold statistics in memory to ensure a minimal load on the OPS database.
The threshold statistics refreshes at a configured interval to ensure that the number is as accurate as possible.
Configure the refresh interval in the omada.ops.service.exe.config configuration using the ThresholdStatisticsValidity application settings:
<add key="ThresholdStatisticsValidity" value="60"></add>
You must type in the value in minutes. Set the value to 0 (zero) to refresh statistics before you run each task. If you run multiple OPS instances, they do not share the threshold statistics. This means that the validity should be set to a low number.
Working with high priority jobs
All provisioning jobs have a defined priority. The component that sends the provisioning job to OPS, for example RoPE, decides what that priority is.
RoPE regularly sends jobs that were caused by changed assignment policies validity changes with a low priority.
When RoPE calculates new assignments because of an access request, or because of emergency lockout operations, the jobs are sent with high priority.
OPS does not include jobs with high priority in its threshold calculations, and the jobs are performed even if the system is suspended because of an exceeded threshold.
Threshold notifications
As mentioned previously, email notifications are sent to system owners when a defined import threshold or provisioning threshold is exceeded.
The emails are sent via two standard event definitions and email templates which ensures these notifications takes place. They are named OPS threshold violation and ODW threshold violation respectively.
You can change the event definitions to perform other actions if you need to.
Threshold in relayed provisioning
If the selected connector is a relay connector and thus the Relayed provisioning is used, the provisioning thresholds are not supported. For such a connector, the Provisioning thresholds task on the system’s page will not be available.
This constraint will be removed in a future version of the Omada Identity.
This restriction does not influence the Data import thresholds.