Step 3 – Writing the collector

A Query collector allows you to write a fully customized collector without the need to dive to deep into technical aspects of SSIS or deploy custom packages.

Query collector data flow

The Query collector uses the following data flow:

Create a Query collector

A Query collector is a standard .NET assembly. This assembly must have a class that implements both the IQuery and ICollector interfaces.

The .NET assembly must target .NET framework 4.0 for SQL Server 2016, as well as, .NET Framework 4.5 for newer SQL versions.

ICollector

The ICollector interface has two methods: Initialize and Finalize.

• Initialize -- this method is called when the connector is initialized.

  • Variables -- every variable defined in the system onboarding file is available in the Initialize method.
  • Water marks
  note

  Omada Identity uses different set of water marks for inter-component synchronization.

  The dictionary Key is packagePath.

• Finalize -- this method is called after all queries have been run or if any Query processing throws an exception.

See the following code sample:

public interface ICollector
{
bool Initialize(Dictionary<string, string> Variables, string Watermarks);

Query method

The Query method has the following parameters:

Parameter	Description
QueryAndMapping	XML string that contains a single query and a corresponding mapping of attributes.
Token	GUID that indicates a query that is being processed. The GUID is the same for calls of the same query.
Done	An ‘out’ property that indicates whether the query processing is finished. A query can be called multiple times if, for example, the result data set is large.

The Query method is called on each query defined in the system onboarding file. The method will be kept executing, using the same token until the Done variable is set to true upon returning from the method.

See the following code example:

public interface IQuery
{
List<IDictionary<string, List<object>>> Query(string QueryAndMapping, Guid Token,
out bool Done);
}

Tutorial project

The end-to-end tutorial contains a sample project which acts as a tutorial to write a .NET Query collector. This sample project reads data from the XML file introduced in 1.1 Target system section.

The default import for this project sample is set to Full in the OnboardingConfiguration.xml file. This can also be found in the Advanced settings menu in the system onboarding page.

note

The data supplied by the .NET assembly during import must adhere exactly to the type of settings for each data type object.

If the load method is set to Full, the Action attribute must be empty (NULL).

Initialization

During initialization, the sample Collector takes the path to the XML file, which in turn contains identity data.

If the path is empty, null, or whitespace, the Collector indicates that initialization failed. Otherwise the Collector proceeds further and control is given back to SSIS.

Flow

The collector parses Query information using the Helper method (provided by the Omada.ODW.Interfaces assembly). The node name indicates that the entity is accessible through the indexer with the attribute name as a key.

The collector then checks all the nodes in the XML file and returns data to SSIS. On each of the entities, an internal loop is created that checks all the Query rules.

The Query collector returns source attributes, not destination attributes. For more details, see the source code of the Test project.

On any exception, the Done flag is set to true, and an exception is thrown again.