Table of Contents

Welcome to Semarchy xDM.
This guide contains information about administering and monitoring Semarchy xDM.

Preface

Overview

Using this guide, you will:

  • Understand the Semarchy xDM architecture and components.
  • Learn how to manage the various component of the architecture.
  • Learn how to manage run-time and troubleshoot errors.
  • Learn how to enable and manage a secure environment for Semarchy xDM.

Audience

This document is intended for administrators managing and configuring Semarchy xDM in a data management initiative.

To discover Semarchy xDM, you can watch our tutorials.
The Semarchy xDM Documentation Library, including the development, administration and installation guides is available online.

Document Conventions

This document uses the following formatting conventions:

ConventionMeaning

boldface

Boldface type indicates graphical user interface elements associated with an action, or a product specific term or concept.

italic

Italic type indicates special emphasis or placeholder variable that you need to provide.

monospace

Monospace type indicates code example, text or commands that you enter.

Other Semarchy Resources

In addition to the product manuals, Semarchy provides other resources available on its web site: https://www.semarchy.com.

Obtaining Help

There are many ways to access the Semarchy Technical Support. You can call or email our global Technical Support Center (support@semarchy.com). For more information, see https://www.semarchy.com.

Feedback

We welcome your comments and suggestions on the quality and usefulness of this documentation.
If you find any error or have any suggestion for improvement, please mail support@semarchy.com and indicate the title of the documentation along with the chapter, section, and page number, if available. Please let us know if you want a reply.

Introduction to Semarchy xDM

Semarchy xDM is the Intelligent Data Hub platform for Master Data Management (MDM), Reference Data Management (RDM), Application Data Management (ADM), Data Quality, and Data Governance.
It provides all the features for data quality, data validation, data matching, de-duplication, data authoring, workflows, and more.

Semarchy xDM brings extreme agility for defining and implementing data management applications and releasing them to production. The platform can be used as the target deployment point for all the data in the enterprise or in conjunction with existing data hubs to contribute to data transparency and quality.
Its powerful and intuitive environment covers all use cases for setting up a successful data governance strategy.

Architecture Overview

The Semarchy xDM architecture includes:

  • The Semarchy xDM Server, a Java EE application deployed and running in an application server. This application serves:
    • The Application Builder, Dashboard Builder, xDM Discovery, Setup and Configuration user interfaces: These web application are used by designers and administrators to create, manage and administer the models and applications designed in Semarchy xDM.
    • Data Management Applications and Dashboard Applications: These web applications are used by business users to browse and manage data and visualize metrics dashboards.
    • A Rest API to perform programmatically data integration, management and administrative operations.
  • The Repository that stores all the metadata used by Semarchy xDM. This includes the data models as well as the definition of the data management and dashboard applications, the Discovery datasources and profiles. The repository is hosted in a database schema. A given Semarchy xDM server is always attached to a single repository.
  • The Data Locations that contain data for the data models. This data include the golden, master and source data, with all the lineage and history. A data location is hosted in a database schema. Multiple data locations can be attached to a single repository.

The following sections detail the component of this architecture.

Semarchy xDM Server

The Semarchy xDM server is a Java EE application deployed and running in a supported application server. It stores all its information in the repository. One application is always attached to a single repository, and connects this repository using a JDBC datasource named SEMARCHY_REPOSITORY. This datasource is configured in the application server.

The Semarchy xDM application is used at design-time to:

  • design Data models and applications, using the Application Builder,
  • design Dashboard applications, using the Dashboard Builder.
  • define and profile datasources, using xDM Discovery.

At run-time, the application:

  • serves the Data Management Application,
  • serves the Dashboard Applications,
  • serves the REST API to manage data and platform operations,
  • runs the processes to profile data.
  • runs the jobs that certify golden data.

Certification Process

Semarchy xDM creates or modifies golden data using data feeds from:

  • Data feeds from source applications (the Publishers). Such data is published programmatically using the SQL or REST APIs.
  • Data creation or changes performed by users in the Data Management applications.

The certification process is triggered as explained below:

  1. The Publishers or the Data Management application submit to the data hub a batch of source data. Such data is submitted as a Load that contains multiple datasets to process together.
  2. The Integration Batch Poller polls integration batches at regular intervals. When a new batch is detected, the batch poller requests the Execution Engine to start the Integration Job associated with this batch.
  3. The Integration Job is created from a Job Definition and executed by the Execution Engine. Data is passed through the different steps of the certification process, and golden data is certified from the source data.
The various steps of the certification process are detailed in the Certification Process Design chapter of the Semarchy xDM Developer’s Guide.

Repository

The repository contains the design-time and run-time information for a given Semarchy xDM Server instance.

Repository Contents

The repository stores the following information:

  • For Data Management Models
    • The entities, attributes, etc.
    • The model versions: branches, editions, etc.
    • The platform configuration & security information: roles, privileges, notification servers, preferences, etc.
    • The data locations information: deployed model, job definitions, notifications, etc.
    • Run-time information: Engine queues, logs, etc.
  • For Dashboard Applications
    • The dashboard application metadata.
  • For xDM Discovery datasources:
    • The datasource configuration and profiled statistics.

A repository is stored in a database schema accessed from the application using a JDBC datasource named SEMARCHY_REPOSITORY.

The repository should never be accessed directly via SQL queries. Access to the Semarchy xDM information must be performed through the Semarchy user interfaces.

Repository Types

There are two types of repositories:

  • Design: All design-time and run-time operations are possible in this type of repository.
  • Deployment: With this repository type, you can only import closed model editions and cannot edit them.

The deployment repositories are suitable for production sites. Model transfer from design to deployment repositories is handled via incremental export/import of closed model editions. Refer to the Planning the Installation chapter in the Semarchy xDM Installation Guide for examples of repository deployment patterns.

The repository type is selected at creation time and cannot be modified afterwards.
Both type of repositories can be used indifferently for Semarchy xDM Dashboards applications.

Data Locations

When a data management hub must be available for run-time (for testing or production purposes), it is generated from a data model defined in the repository, and deployed in a data location. Data Locations contain the deployed data hubs. Each hub contains the golden data, the source data pushed by publishers in the hub, and the various stages of this data in the certification process.

The data location is hosted in a database schema and accessed via a JDBC datasource defined in the application server. A data location refers to the datasource via its JNDI URL.

Data locations are only used for Semarchy models designed in the Application Builder. Dashboard Applications created in Semarchy xDM Dashboards do not require a data location.

Data Locations, Repositories and Models

A data location is attached to a repository: You can declare as many data locations as you want in a repository, but a data location is always attached to a single repository. It is not possible to have a data location attached to two repositories at the same time.

You may deploy several model editions successively in a data location, but only one model edition is deployed and is active in the data location at a certain point in time.

Data Location Contents

A Data Location contains the hub data, stored in the schema accessed using the data location’s datasource. This schema contains database tables and other objects generated from the model edition.

The data location also refers three type of jobs (stored in the repository):

  • Installation Jobs: The jobs for creating or modifying in a non-destructive way the data structures in the schema.
  • Integration Jobs: The jobs for certifying data in these data structures, according to the model job definitions.
  • Purge Jobs: The jobs for purging the logs, data lineage and history according to the retention policies.

Data Location Types

There are two types of data locations:

  • Development Data Locations: A data location of this type supports deploying open or closed model editions. This type of data location is suitable for testing models in development and quality assurance environments.
  • Production Data Location: A data location of this type supports deploying only closed model editions. This type of data location is suitable for deploying MDM hubs in production environments.
The type is selected when the data location is created and cannot be changed afterwards.

Refer to the Planning the Installation chapter in the Semarchy xDM Installation Guide for examples of data location deployment patterns.

Data Structures and Certification Process

A deployed model edition is made of a Data Structure and a set of Integration Jobs.

  • The Data Structure of the MDM Hub is a set of tables stored in the database schema of the data location. This structure contains the landing tables for the loads pushed in the hub by the publishers, the golden records tables and the tables handled by the certification process to create golden records from the source records.
  • The Integration Jobs are sequences of tasks stored in the repository, which perform the certification for data entering the hub. They are specific the deployed model edition. When a model edition is deployed in a data location, the integration job definition for this model edition replace previous job definitions.
Data Structure and Integration Jobs work together and are deployed simultaneously by the model edition deployment process.
Data Structure Details

The data structure is implemented to support all the successive model editions deployed in the data location. The data structure is created when the first model edition is deployed, and is altered for new model editions. Changes to the structure are only incremental and non-destructive.

For example, the GD_CUSTOMER table holds the data for all successive design states of the Customer entity. If a new FaxNumber attribute is added to the entity and deployed with model edition 1.1, the FAX_NUMBER column is created to contain the fax number data. If this attribute is removed in subsequent model editions (1.2 and so forth), the column for this attribute remains in the data structures, when these model editions are deployed.

Platform Components

The Semarchy xDM platform contains several components described in the sections below.

Integration Batch Poller

The Integration Batch Poller polls the integration batches submitted to the platform on a defined schedule, and starts the Integration Jobs on the Execution Engine.

Execution Engine

The Execution Engine processes the Integration Jobs submitted by the Integration Batch Poller. It orchestrates the certification process to generate the golden data. This engine sequences the jobs in Clusters and Queues.

The execution engine logs the activity of the platform and manages notification policies.

The execution engine and integration batch poller management and configuration are covered in the Managing Executions chapter in the Semarchy xDM Application Management Guide.

Security

The application uses role-based security for accessing Semarchy xDM features. The users and roles used to connect to the application must be defined in the security realm as part of the application server configuration and then declared in Semarchy xDM.

Role base security is used in Semarchy xDM to define the access privileges to the features of the platform (Platform-Level Security), as well as the privileges to access and modify data in the data locations (Model-Level Security).

Introduction to Semarchy Configuration

Platform-level Administrative tasks are performed in a dedicated user interface called Semarchy Configuration. This section explains how to work with it.

Connecting

To access Semarchy Configuration, you need an URL, a user name and password that have been provided by your Semarchy xDM administrator.

To log in to Semarchy Configuration:

  1. Open your web browser and connect to the URL provided to you by your administrator. For example http://<host>:<port>/semarchy/ where <host> and <port> represent the name and port of host running the Semarchy xDM application. The Login Form is displayed.
  2. Enter your user name and password.
  3. Click Log In. The Semarchy xDM Welcome page opens.
  4. Click the Configuration button. Semarchy xDM Configuration opens.

To log out of Semarchy Configuration:

  1. In the Semarchy Configuration User Menu, select Log Out.

Semarchy Configuration

Semarchy Configuration allows to view and administer the following components:

  • Applications Configuration: Global parameters for all applications.
  • Image Libraries: Manage image libraries and images in these libraries.
  • Custom Translations: Override user interfaces built-in strings or translate them into new languages.
  • Logging Configuration: Configure the platform logging (trace) for debugging purposes.
  • Notification Servers: Add, remove and configure servers used to send job notifications and application emails.
  • Plug-ins: View, add or update user-created Java plug-ins.
  • REST Clients: Add, remove or modify REST clients.
  • Roles: Declare in Semarchy xDM the application server roles, and grant them with platform-level privileges.
  • API Keys: Keys used by applications to authenticate and interact with the REST API.
  • Variable Value Providers: Configure the system queried by Semarchy xDM to retrieve values for model variables.

Managing the Repository

The repository contains the design-time and run-time information for the Semarchy xDM server.

Understanding Repositories

The repository is created when Semarchy xDM is installed. The type of the repository (Design Repository or Deployment Repository) is set also at creation time, and the application always connects to a single repository.

The repository creation process is detailed in the Semarchy xDM Installation Guide.

The type of a repository defines the capabilities of this repository:

  • A Design Repository allows you to perform all design-time and run-time operations.
  • A Deployment Repository only allows run-time operations. You can import closed model editions in such repository but cannot edit them.
Typical Patterns for repository deployment are detailed in the Planning the Installation chapter of the Semarchy xDM Installation Guide. Simple and advanced deployment tasks are explained in the Deployment chapter of the Semarchy xDM Developer’s Guide.

Repository and System Information

Semarchy xDM exposes the repository and system details in the About dialog.

To view the repository and system details:

  1. In the Semarchy Configuration User menu, select About.
  2. In the About dialog:
    • The Manage License link displays the current license information and allow for Managing the License.
    • The Repository Information link displays the repository details (including name and version).
    • The System Information link displays the platform system details and may be used for support purposes.

Managing the License

Semarchy xDM stores in the repository the information about your license as well as the license key.

Licenses in Semarchy xDM work as described below:

  • A newly installed xDM instance runs for a limited "grace period" with no license key.
  • Before this period is over, you should activate the instance using that license key. This license key may extend your evaluation period of Semarchy xDM.
  • If the license key expires or is modified (for example, when switching from an evaluation license to a full customer license), you must update the license.
    Alternately, you may synchronize the license key automatically or manually to keep your license up-to-date.
Each license key is unique to one installed xDM instance, and should not be used on another instance.
You must be connected with a user having the semarchyAdmin role to perform any operation on the license.

Activating an Instance

A newly installed Semarchy xDM instance runs for a limited time - the grace period - without a license. Within that grace period, you must activate your instance.

When the grace period is over, the instance will stop working: the applications, as well as the data stored in Semarchy xDM, will no longer be available, and incoming data will no longer be processed. The data stored in the hub is preserved as-is.
You will need internet access and a valid corporate email address to activate your instance and receive the license key.

To activate an instance:

  1. In the Semarchy Configuration User menu, select About.
  2. In the About dialog select the Manage License link.
  3. In the Manage License page, click the Activate button
  4. Follow the instructions on the Activate xDM Instance page to collect your license key.
  5. Paste your license key and then click Activate

The instance is activated, and the license information appears in the Manage License page.

Updating the License Key

When the license key expires or is modified, you must update that license key.

The license key update is a manual process. To simplify license management, you may choose to synchronize the license key.

To update the license key:

  1. In the Semarchy Configuration User menu, select About.
  2. In the About dialog select the Manage License link.
  3. In the Manage License page, click the Update button.
  4. Follow the instructions on the Update License page to collect your updated license key.
  5. Paste your license key and then click Update

The license key is updated, and the updated license information appears in the Manage License page.

Synchronizing the License Key

Synchronizing the license key automatically updates the license key. For example, if your subscription is renewed, the synchronization extends this subscription without having to update the license key manually.

You can perform the synchronization with a single-click operation, or run this synchronization on an automated schedule using Auto-Sync.

Synchronizing the license does not require any external access to your instance. The instance will send to Semarchy the information listed using an HTTP connection. See License Key Synchronization Details for more information.

To synchronize the license key:

  1. In the Semarchy Configuration User menu, select About.
  2. In the About dialog select the Manage License link.
  3. In the Manage License page, click the Sync Now button, and then confirm the synchronization if prompted.

The synchronization starts. You can review the Last Sync field that shows the current synchronization status.

Auto-sync performs the same operations as the manual synchronization on a daily schedule. It is strongly recommended to enable auto-sync on instances that require regular license updates (for example those using a subscription license).

To enable auto-sync:

  1. In the Semarchy Configuration User menu, select About.
  2. In the About dialog select the Manage License link.
  3. In the Manage License page, select the Auto-sync checkbox.
  4. Review the terms and confirm that you want to Enable auto-sync.

Auto-sync is enabled. You can review later the synchronization status in the Manage License page, in the Last Sync field that shows the latest synchronization timestamp.

License Key Synchronization Details

This section provides details about the license key synchronization process.

Network Requirements

The license key synchronization process does not require additional configuration on the server hosting the instance, and does not require incoming network access to the instance. However, you should make sure that this server has access to the internet to send and receive information during the synchronization process.

During the synchronization process, the instance sends information to the Semarchy license server and retrieves the updated license key. This is performed using REST API calls (HTTP/S POST and GET requests) from the instance to the license-api.semarchy.com server.

Synchronized Information

During the synchronization process, the instance connects to the License Server and sends a report composed of the information detailed below. The updated license key is returned to the instance automatically.

If you do not want the report to be sent, do not use the Sync Now button, and do not select the Auto-sync checkbox. Use instead the Update button to manually update your license.

The report contains the following instance information:

  • Base URL and version number (e.g.: 5.1.0) of the application.
  • Name, type (Design or Deployment), version number, creation and latest upgrade date of the repository
  • Repository database information: JDBC URL (as declared in the application server), database server name, technology, version and schema name

The report contains, for each data location:

  • Name, label, data source name, deployment date, type (Development or Production), internal version number and status of the data location
  • Data Location database information: JDBC URL (as declared in the application server), database server name, technology, version and schema name

The report contains, for each entity(*) deployed in a data location:

  • The physical table name (e.g. PARTY), the entity name (e.g. Party) and the type of entity ( BASIC, FUZZY_MATCHED or ID_MATCHED)
  • The total number of attributes and the number of attributes of type BLOB or BLOB in the entity.
  • The count of golden records, authored golden records (fuzzy-matched entities only), historized golden records, master records (Fuzzy or ID-matched entities only), historized master records (Fuzzy or ID-matched entities only), source data records (Fuzzy or ID-matched entities only) and source authoring records.

(*) For a group of entities in an inheritance relationship, only the parent entity from which all child entities inherit appear in the report, with all the attributes of its child entities.

Important Notes

The reported information:

  • DOES NOT contain any personal information, users, passwords or connection credential information.
  • DOES NOT contain any data from your MDM hub tables.
  • Is intended for statistical and technical support purposes.
  • Is transmitted using secured (HTTPS) connections.
  • Is not visible by anyone outside Semarchy.
  • Is stored in secured servers located in Ireland.

Configuring the Platform

The platform contains of several components that can be managed from Semarchy Configuration.

Configuring Notifications Servers

Notifications tell users or applications when a job completes or when operations are performed into workflows, for example, when task is assigned to a role.

There are two types of notifications:

  • Job Notifications issued under certain conditions when an integration job completes. These notifications are used for administration, monitoring, or integration automation. These notifications are configured with Notification Policies in the data locations.
    For more information about notification policies, see the Semarchy xDM Application Management Guide.
  • Workflow Notifications are emails sent to users when operations are performed in a workflow. They are configured in workflow transitions and tasks.
    For more information about workflow notifications, see the Semarchy xDM Developer’s Guide.

Both families of Notifications are issued via Notification Servers.

Notifications Servers Types

Notifications recipients may be users or systems. The type of notification sent as well as the recipient depends on the type of notification server configured.

Each notification server uses a Notification Plug-in that:

  • defines the configuration parameters for the notification server,
  • defines the configuration and form of the notification,
  • sends the notifications via the notification servers.

Semarchy xDM is provided with several built-in notification plug-ins:

  • JavaMail: The notification is sent in the form of an email via a Mail Session server configured in the application server, and referenced in the notification server. For more information about configuring Mail Session, see the Semarchy xDM Installation Guide.
  • SMTP: The notification is sent in the form of an email via a SMTP server entirely configured in the notification server.
  • File: The notification is issued as text in a file stored in a local directory or in a FTP/SFTP file server.
  • HTTP: The notification is issued as a GET or POST request sent to a remote HTTP server. Use this server type to call a web service with the notification information.
  • JMS: The notification is issued as a JMS message in a message queue.
It is possible to develop additional plug-ins to issue other type of notifications. See the Semarchy xDM Plug-in Development Guide for more information about plug-in development.

A single notification server having either the JavaMail or SMTP type can be used to send Workflow Notifications. This server is flagged as the Workflow Notification Server

Any servers can be used to send Job Notifications. Each Job Notification Policy specifies the notification server it uses.

Creating a Notification Server

To create a notification server:

  1. In Semarchy Configuration, select the Overview editor, and then click the Notification Servers link. The Notification Servers editor opens.
  2. Select the Notification Servers list, right click and select image New Notification Server. The Create New Notification Server wizard opens.
  3. Enter the following workflow parameters:
    • Name: Internal name of the notification server.
    • Label: User-friendly label for the server.
    • Plug-in ID: Select one of the available notification server plug-in.
    • Workflow Notification Server: Select this option to use this notification server by default in the workflows. This options can be selected only if the Plug-in ID is JavaMail or SMTP.
  4. Click Next.
  5. In the second wizard page, enter the configuration information for your type of server:
    • JavaMail:
      • JNDI URL: JNDI URL of the Java Mail Session service available in the application server. This URL is typically java:comp/env/mail/Session if the Mail Session service is declared as mail/Session in the application server.
      • From User: Email address of the sender of the notifications from this server. This address is also used in the reply-to address for notification emails.
      • Password: If this server requires specific authentication, enter a password for this server. Note that the authentication configured in the Java Mail resource has precedence over this password.
    • SMTP:
      • SMTP Host Name and SMTP Port: Name or address, and port of the SMTP host.
      • From User: Email address of the sender of the notifications from this server. This address is also used in the reply-to address for notification emails.
      • Authentication Required: If this server requires specific authentication, select the Authentication Required option and enter a User Name and Password for this server, and indicate whether it uses TLS or SSL.
      • Additional SMTP Properties Enter additional properties as property=value pairs.
    • File:
      • File System: Select the file system of the file server. FILE for a local server, FTP or SFTP for a remote server.
      • Host, Port, Login, Password are required to connect an FTP or SFTP server.
      • Root Path: Provide the root path for storing the notification file. For example c:\work\notifications for Windows or /work/notifications for UNIX/Linux
    • HTTP:
      • Scheme: Specify whether the HTTP request should be done using HTTP or HTTPS
      • Host, Port and optionally Login, Password are used to connect the HTTP server.
      • Base Path: Root path appended added after the host and port in the URL. For example: / or /rest/api/.
      • Use System Properties: Check this option to use the system-defined properties to configure the HTTP connection. This option allows using a proxy configuration defined in the Java parameters for the application server.
      • Proxy Host, Proxy Port, Proxy Login and Proxy Password are used to configure the connection through an HTTP proxy and possibly override the proxy configured via the system properties.
      • Headers: Enter additional HTTP headers as property=value pairs.
    • JMS:
      • Connection Factory URL: JNDI URL of the factory used to create a connection to the JMS destination. The URL is typically java:comp/env/jms/ConnectionFactory if the connection factory is declared as jms/ConnectionFactory in the application server.
      • Login and Password used when initiating the JMS connection.
  6. Press CTRL-S to save the configuration.

Testing a Notification Server

After configuring the notification server, it is recommended to run a test email on this server.

To test a notification server:

  1. In the Notification Servers editor, select the notification server you want to test, right-click and select Test Configuration.
  2. The next steps depend on the type of notification servers:
    • File, HTTP, JMS: No further operation is needed. A connection attempt is made on the notification server.
    • JavaMail and SMTP: Provide a comma-separated list of email addresses and then click OK. An email is sent via the notification server to these recipients.

Configuring Variable Value Providers

Semarchy xDM uses variables defined in models to enforce certain data governance policies for a user’s session.
For more information about model variables, see the Semarchy xDM Developer’s Guide.

A Variable Value Provider is a system that can be queried by Semarchy xDM to retrieve the values for these variables. Typically, this system is a server containing information about the user connected to Semarchy xDM.

Two type of variable value providers are supported out-of-the-box:

  • Datasource Variable Provider: This variable value provider is a relational database that is accessed through a JDBC datasource. Semarchy xDM can issue SQL statements against this database to retrieve variable values. For example, an employee database that can be queried to retrieve the country of the connected user.
  • LDAP Variable Provider: This variable value provider is a directory server that is accessed using the LDAP protocol. Semarchy xDM can issue queries against this directory server to retrieve variable values. For example, an LDAP directory that can be used to retrieve the organizational unit of the connected user.

Variable value providers are configured in the repository, and can be used by any model in this repository.

When working with a deployment repository, make sure to configure the variable value providers used in the models before importing or deploying them in this repository.

Creating a Variable Value Provider

To create a variable value provider:

  1. In Semarchy Configuration, select the Overview editor, and then click the Variable Value Providers link. The Variable Value Providers editor opens.
  2. Select the Variable Value Providers list, right-click and then select image New Variable Value Provider. The Install Variable Value Provider wizard opens.
  3. Enter the following information:
    • Name: Internal name of the variable value provider.
    • Label: User-friendly label for the variable value provider. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.
  4. Select the Plug-in ID corresponding to the variable value provider type: LDAP Variable Provider or Datasource Variable Provider.
  5. Click Next.
  6. Click the image Edit Expression button.
  7. In the the Variable Value Configuration dialog, enter the configuration information.
    This information differs depending on the selected Plug-In.
    • For a Datasource Variable Value Provider, enter the following information:
      • JNDI Data Source Name: Select a JDBC datasource in the list.
        This datasource must be defined in the application server. It is used to connect the database acting as the variable value provider.
    • For an LDAP Variable Value Provider, enter the following information:
      • LDAP Host: Name or IP address of the LDAP server.
      • LDAP Port: Listening port of the LDAP server. The port is typically 389 for non-SSL connections, and 636 for SSL connections.
      • Use SSL: Check this options to use SSL to connect to the LDAP server.
      • User: Name of the user used to retrieve data from the LDAP Server. Note that this user should have read privileges to the LDAP structure.
      • Password: This user’s password.
  8. Click OK to close the Variable Value Configuration dialog.
  9. Click Finish.

The variable value provider is added to the list.

Testing the Variable Value Provider Configuration

After configuring a new variable value provider, it is recommended to test its configuration.

to test a variable value provider configuration:

  1. In the Variable Value Providers editor, select of the variable value provider in the list.
  2. Right-click and select Test Configuration.

A message indicates whether the connection test was successful or not.

The configuration test only tests the connection information, but does not check the privileges granted to the user to retrieve the values from the provider.

Managing Image Libraries

Image libraries store images and icons used by applications. Libraries are stored in the repository, are global to the platform and can be used by all applications. Designers use, in applications, images and icons from the libraries by reference, using image library URLs.

Default Image Libraries

When the repository is created, built-in image libraries are seeded with default Semarchy xDM images and icons:

  • mdi is the library for all the Material Design icons and assets. This library cannot be modified.
  • default contains Semarchy branding images as well as the default images used by applications, such as the default images for the actions. Icons provided in addition to the mdi library are in this library. This library cannot be modified.
  • demo<application_name>_ contains the images/icons used in the <application_name> demonstration application. Such a library is seeded when a demo application is installed.
The default image libraries content changes with new versions of Semarchy xDM. When an image is removed or changed in the default libraries, it is indicated as deprecated in the validation report for models using it. Model designers should update their models to use newer versions of such images.

Browsing Image Libraries

The Image Libraries editor, available in Semarchy Configuration, allows administrators to view and manage the image libraries.

To browse the image libraries:

  • Open Semarchy Configuration.
  • In the Overview, click the Image Libraries link.
    The Image Libraries editor opens.

Use the filter box in the image libraries editor to reduce the number of images displayed. You can use the '%' sign to replace any number of character.

Managing Image Libraries

Image libraries are created and modified using file import. You can import two types of files:

  • Image files, for which you must select a target library.
  • Zip files, into which images are organized into folders. When a Zip file is imported, one library is created for each folder at the root of the Zip file, and images in this folder are imported into the library. Exporting images creates a file that follows the same format.

To import images:

  • Open the image libraries editor.
  • Click the Import Image Library button. The import dialog opens.
  • Drop either a Zip file or multiple image files into the dialog, or click Add to select a Zip file or multiple image files.
  • If you import image files, select the Target Library for these images. Select Create a New Library to create a new library for these images.
  • Click OK to import the images.

To delete images:

  • Open the image libraries editor.
  • Click an image to select it. Repeat this operation for all images you want to delete. You can also click the Select All option in the toolbar to select all the filtered images.
  • Click the Delete Selected Images button, and then confirm the deletion.
If you delete all the images from a library, the library disappears.

To export images:

  • Open the image libraries editor.
  • Click the image to select it. Repeat this operation for all images you want to delete. You can also click the Select All option in the toolbar to select all the filtered images.
  • Click the Export Selected Images button.

A zip file is downloaded, containing all the selected images stored into folders named after their parent library.

Using this export mechanism, you can move an image library to a remote repository, or reorganize the library before reimporting it.
If an image is removed from a library while being used in a model, it will raise a warning in the model validation report. Designers should fix such stale image references and update their models to point to valid images.

Managing Plug-ins

Semarchy xDM allows extending its capabilities using Java code. Using the Open Plug-in Architecture, existing services or information systems can contribute to the master data processing and enrichment. You can extend the Enrichment and Validation capabilities in Semarchy xDM through user-defined plug-ins.

For detailed information about plug-in development and packaging, see the Semarchy xDM Plug-in Development Guide.

A Plug-in is delivered as a jar file bundle that must be deployed in each Semarchy xDM server instance running integration jobs that use the plug-in. You do not need to restart the server to take new or updated bundles into account.

These bundles are tagged with a version number. As a consequence, updating an existing plug-in with a newer version of this plug-in will automatically make the platform work with the newer plug-in version. The deployment process installs a new plug-in or replaces an existing plug-in version with a new one.

To deploy a plug-in:

  1. In Semarchy Configuration, select the Overview editor, and then click the Plug-ins link.
  2. Click the image Install or Update Plug-in button in the upper right corner of the Plug-ins editor. The Install/Update Plug-ins dialog opens.
  3. Click the Add button and select the plug-in binary file. For example: com.acme.phoneStandardizer_1.0.0.jar.
  4. Click OK. A Status window shows the number of plug-ins installed or updated.
  5. Your session is closed to take this new plug-in into account. Restart the session.
  6. In Semarchy Configuration, select the Overview editor, and then click the Plug-ins link.

The plug-in now appears in the list, and can be used in the models and the integration jobs.

Make sure to install the plug-ins required by the jobs of a model before deploying the model. If a job starts before its required plug-ins are installed, then it will fail. The plug-in can be installed and the job resumed after the installation.

To uninstall a plug-in:

  1. In Semarchy Configuration, select the Overview editor, and then click the Plug-ins link.
  2. Select the plug-in in the list.
  3. Click the image Uninstall Selected Plug-ins button in the editor’s toolbar.

Managing REST Clients

REST Clients extend the capabilities of Semarchy xDM to process data by calling external services via their REST API.

You can define a REST Client for the entire Semarchy xDM platform in Semarchy Configuration, and then use this client in all the models in the repository.

REST Clients Overview

When Semarchy calls a REST CLient to enrich a record, the client sends a request sent to the external service.
This request is defined by:

  • An HTTP URL, accessed by an HTTP Method (GET, POST, etc).
  • Optional HTTP Headers, for example, to pass authentication information.
  • An optional Body, typically in JSON format.

The elements of the request can be parameterized by Request Inputs, passed to the REST Client. These inputs are typically data from the record being enriched.

The external service returns a response which contains:

  • An HTTP status code (e.g.: 200: success, 404: not found)
  • A response body, typically in JSON format from which Response Outputs will be extracted. These outputs are mapped to attributes in the records to enrich.

Declaring a REST Client

To create a REST Client:

  1. In Semarchy Configuration, select the Overview editor and then click the REST Clients link.
  2. In the REST Clients editor toolbar, select Add New image > New REST Client image. The Create New REST Client dialog opens.
  3. Enter the following information:
    • Name: Name of the REST Client. When a model uses a REST client, it refers to this client by this name.
    • Label: User-friendly label for the REST Client. This label appears in the Workbench.
    • Description: Provide a meaningful description for the model designers who will use this client.
  4. Click Next.
  5. Select the HTTP Method for the REST endpoint.
  6. Enter the URL of the REST endpoint.
    You can make parts of this URL variable by replacing constant values with request inputs, using the ${request_input_name} syntax. For example, if the service URL contains a variable part to pass an email address, you can use the syntax below to define this address as an input of the REST client:
    https://information.acme.com/information?email_adr=${email}
  7. Enter the HTTP Headers possibly required by the endpoint. Similarly to the URL, you can use variable inputs in the headers.
  8. Select Body Format for the Request: Use NONE if the endpoint does not require a body, JSON for a JSON payload and BINARY for a binary payload.
  9. In the Body field, compose the JSON or binary payload required by the endpoint. Similarly to the URL, you can use variable inputs in the body.
  10. Click Finish. The REST Client editor opens.
  11. Review and update the Name, Label and Description of the client, as well as the HTTP Request parameters.
  12. If you have used inputs using the ${request_input_name} in the wizard, they are automatically added to the Request Inputs list.
    You can also manage these inputs manually:
    • To detect new inputs added to the URL, headers or body, click Detect Inputs image in the Request Inputs table toolbar.
    • To create a new input, click Add Input image in the Request Inputs table toolbar
    • To delete unused request inputs, select them and then click Delete image in the Request Inputs table toolbar
  13. In the Request Inputs table, provide for each input:
    • A User-friendly Label and a meaningful Description for the model designer who will use the REST Client.
    • A Data Type for the input. Note that this is the expected type of the input value. See Request Inputs Datatype and Formatting below for more information.
  14. In the Response Output table, click Add Output image.
    The new output appears in the list.
    • Define a Name, Label and Description for this output. The label and description appear in the Workbench for model designers. The Name is used by an enricher to refer to the output.
    • Set the Value Selector. This selector is a JSON Path, starting from the root of the response payload and pointing to a single value in the payload.
    • Select the Data Type of the output.
    • The Behavior on Absent Value defines whether the enricher should fail or return the Default Value if the JSON Path expression returns no value for the payload.
    • The Behavior on Data Type Mismatch defines whether the enricher should fail or return the Default Value if the JSON Path expression returns a value that is not compatible with the output data type.
    • Set a Default Value or leave it empty to have a null default value.
  15. Repeat the previous step to each output returned by the REST Client.
    For more information about outputs, see the Response Processing section.
  16. In the Advanced Configuration section of the editor set the following parameters:
    • Timeout for the HTTP request. After this timeout, the call is considered failed.
    • Max Retries: Number of times the request should be attempted. The retries occur only if the service reports a code corresponding to a temporary failure condition. These include HTTP 404, 410, 429 and 503)
    • Success Status Codes: A comma-separated list of HTTP response status (e.g.: 200, 201, 202) and response status patterns (e.g. 2xx) considered successful. The default is 2xx.
    • Error Selector: JSON Path expression returning a possible error to search in the response payload. Note that this path is evaluated even for a successful response. If it returns a value in that case, the error is raised.
  17. Press CTRL-S to save the REST Client.
Before proceeding, make sure to test the REST Client.
Request Inputs Datatype and Formatting

When using a request input in the REST Client definition, the datatype defines how values for this input are processed and inserted in the request:

Automatic formatting:

  • Binary inputs are automatically encoded in BASE64 when used in the URL, the headers or in a JSON Body. In a Binary body, a binary input not formatted.
  • Date and Timestamp inputs are formatted as strings in the ISO_8601 format (YYYY-MM-DD for dates, YYYY-MM-DDTHH:mm:ss.SSSZ for timestamps).
  • Number inputs are formatted as strings with a dot as the decimal separator and no thousand separators.
  • Boolean inputs are not formatted and will appear as true or false.

The inputs are then substituted by their formatted values in the request.

During that replacement:

  • Encoding and escaping are automatically handled in the URLs and JSON.
  • Quotes are added in the JSON Body around strings, binary, dates and timestamps.
  • Input with null values will be replaced by null in the JSON payload.
Example 1. Formatting in a JSON Body.

In the example below:

  • fullName and comment are string. They are automatically escaped and surrounded by quotes.
  • height in a numeric input. It is formatted with no quotes.
  • Picture is a binary. It is encoded in Base64 and surrounded by quotes.
{
    "Name"      : ${fullName},
    "Comment "  : ${comment},
    "Height"    : ${height},
    "Photo"     : ${picture}
}

A possible result will be:

{
    "Name"      : "John \"Joe\" Doe",
    "Comment "  : null,
    "Height"    : 180.2,
    "Photo"     : "data:image/png;base64,iVBOR...."
}
Example 2. Example of URL Encoding

If you pass the value "Mountain View California" for the address input in the following URL:

https://information.acme.com/address?addr=${address}

The address input will be automatically encoded in the URL:

https://information.acme.com/address?addr=Mountain%20View%20California
Response Processing

The REST Client extracts from the response payload:

  • The output values, using Value Selectors
  • A possible error message, using the Error Selector

Both these selectors use JSON Path, a JSON query language.

The full JSON Path syntax supported by Semarchy xDM is detailed at https://github.com/json-path/JsonPath.
JSON Path supports retrieving arrays of data. For the Value Selectors, if an array is returned by the JSON Path, the first element of the array is extracted in the output.
Example 3. Simple JSON Path Examples:

With the following payload

{
    "customers":  [
        {
        "first name": "Joe",
        "last name": "Celko",
        "age": 58,
        },
        {
        "first name": "Roger",
        "last name": "Moore",
        "age": 69,
        }
    ],
    "customers_count": 2
}

The following JSON Path expressions return the following values:

  • $.customers_count returns '2'
  • $.customers[0].age returns '58'
  • $['customers'][0]['age'] returns '58' (different syntax)
  • $['customers'][1]['last name'] returns 'Moore'

Testing the REST Client

To test a REST Client:

  1. In the REST Clients editor toolbar, select Manage image > Test.
    The REST Client test dialog opens.
  2. Enter values for the request inputs, and then click Execute.

The REST Client calls the endpoints, processes the outputs and shows the resulting values.

Importing and Exporting REST Clients

Rest Clients are stored in the repository. When moving a model to a different repository (for example, from a development to a test environment), you must also move the REST Clients used by this model.

  • In the source environment, click Export image in the REST Client editor to export the client definition to a file.
  • In the target environement, in the REST Clients editor toolbar, select Add New image > Import image to import the exported client definition.
You can also use the Platform REST API to automate the export/import of the REST Clients across environments.

Managing API Keys

Semarchy xDM supports authentication to the REST API using API Keys. Such an authentication method is used by applications to interact with the API.

API Keys are defined at platform-level, and have the following characteristics:

  • They have a set of Roles whose privileges apply in API calls.
  • They have a Default User Name used when the API requires one.
  • They may be de-activated temporarily or permanently.
  • They have an optional Expiry Date.
  • Calls made with a key may be restricted to originate from specific IP addresses or IP address ranges.

Make sure to select for an API key the right roles to interact with the data, since data model privilege grants apply:

  • The API key should have a role attached to a model privilege grant with the Grant access to the Integration API option selected in order to interact with data stored in the given model.
  • In order to set a different user when interacting with data, the API key should have a role attached to a model privilege grant with the Allow Publishing as user in API option selected.

Creating an API Key

To create an API key:

  1. In Semarchy Configuration, select the Overview editor, and then click the API Keys link.
  2. In the API Keys editor, right-click in the API Keys table and select image New API Key. The Install API Key wizard opens.
  3. Enter the following information:
    • Label: User-friendly label for the API Key. Modifying this label is optional.
    • Default User Name: Default user name used for operations made with this API Key.
    • Roles: Click the image Edit button to select the roles to grant to this API key.
    • Expiry Date: Optionally select an expiry date for the key. By default, a key has no expiry date and never expires.
    • Origin IP Addresses or Ranges: Enter a comma-separated list IP addresses (e.g.: 192.168.0.1) or ranges (e.g.: 172.16.0.0/12). Only addresses listed are allowed to connect using the key. Leave this field empty to allow all addresses.
  4. Click FINISH. The Save API Key dialog opens and displays the generated API Key.
    Copy this key to a safe place, or click the link to save a file containing the key value.
  5. Make sure that you have saved the key and then click OK.

The key is created. You can use it to connect the REST API.

After creating an API Key, from the API Keys editor, you can:

  • Edit a key, for example, to change its roles or set its expiry date.
  • Delete a key.
  • Expire a key (Right-click then Expire Now), which sets the expiry date to the current date and disables the key.
  • Re-enable a key by editing it, then selecting an expiry date in the future, or using the Clear button in the Expiry Date selection dialog to make the key active with no expiry date.
  • Review a key last usage in the Last Interaction section of the key editor.
    To trace the detailed usage of API keys, use the com.semarchy.xdm.rest.accesslog logger.

Using an API Key

To use an API key:

To use this API key in a REST API call, set an HTTP header called API-Key, containing the API key previously created. For example:

API-Key:rXXTAYe.X2jMCtbNrC72xu7x91kB7cSYJKCt3SFKohj

Managing Applications Configuration

The global behavior of the data management applications is configured from Semarchy Configuration.

Changes performed in this configuration apply to all the applications started in the instance of Semarchy xDM.
The semarchyAdmin role is required to configure the global application parameters.

To set the Applications Configuration:

  1. In Semarchy Configuration, select the Overview editor, and then click the Applications Configuration link. the Applications Configuration editor opens.

This editor displays the global application parameters:

  • Configure the Server Base URL of your Semarchy xDM installation. This URL is seeded when creating the repository and used in workflow notifications to provide links to the server.
  • Set the CSV Export Limit and Excel Export Limit to define the maximum export size allowed in each format. Note that generating export files is resource consuming on the server. It is recommended to test the scalability of the new export limits.
  • Set the Autocomplete Max Results value to define the maximum number of results that should be fetched by the autocomplete components in the applications. These dynamic drop drown lists fetch records as you type a search strings.
  • Change the Header Logo that appears for all applications and in the welcome page by uploading a picture. Note that this picture is scaled to a height of 96 pixels.
    You can also change the Favicon used for this page.
  • The Enable First Login Wizard enables of disables the wizard that appears the first time a user connect to an application, in order to configure his profile.

Managing Custom Translations

Semarchy xDM allows translating the strings you define in applications (for example, the labels of the business views or fields) as part of the model and application localization effort.

In addition, strings built in Semarchy, visible in all data management and dashboard applications, such as 'Inbox', 'Filter' or 'Global Search', can also be modified in the platform configuration with Custom Translations.

With custom translations, you can:

  • Override existing built-in strings. For example, replace 'Inbox' by 'My Tasks' in the Inbox editor title.
  • Create translations for new languages for the built-in strings.
Custom translations are configured at platform-level and apply to all the applications served by the platform.
For more information about model translation, see the Model Localization section in the Semarchy xDM Application Management Guide.

The translation process consists of:

  1. Exporting existing languages' localization file, or blank localization files for new languages. These files are properties files, which can be modified using a text editor.
  2. Editing these files using a text editor.
  3. Import the modified files.

Exporting Custom Translations

To export custom translations:

  1. In Semarchy Configuration, select the Overview editor and then click on the image Custom Translations link.
    The Custom Translations editor opens and lists the custom translations that you have configured.
  2. If you want to modify existing custom translations, select these translations in the list.
  3. Click on the image Export Translations…​ button in the editor toolbar.
  4. In the Export Translation Bundles dialog, select the translation bundles that you want to export. The Hide Languages with no Custom Translations option masks the languages for which you have no custom translation.
  5. Click OK
    The selected translation bundles are exported in a zip file.

Custom Translations Export File Format

The translations are exported as a compressed file that contains properties files (containing key=value pairs).

These files are organized into folders:

  • The /commons folder contains property files for all the languages with strings common to the dashboard and data management applications.
  • The /dashboards folder contains property files for all the languages with strings specific to the Semarchy xDM Dashboards component.
  • The /discovery folder contains property files for all the languages with strings specific to the xDM Discovery component.
  • The /applications folder contains the property files for all the languages with strings specific to the data management applications.
After editing the custom translation properties files, make sure to compress these files in a zip file that follows the exact same structure, as it is expected by the import process.
Languages and Variants

The property files have a suffix corresponding to the language code and an optional variant code.

Example 4. Example of language and variant

en is the suffix for the English language, and en_US is the suffix for United States variant of this language. When a key needs to be translated:

  • Users having their browser configured in the English (United States) variant will use the en_US translation if it is available, or else use the English en translation.
  • Users having their browser in English will directly use the English en translation.

In addition, specific properties files with the en_default and en_default suffixes represent the default translations built-in Semarchy xDM for English and French. These are here for informational purposes and are ignored during the import process.

Language translation fallback

You do not need to define all translations in the custom translations since the string localization process has a fallback process.

In this process, translations are searched in the following order:

  1. In the language variant custom translation (for example fr_FR for the French (France) language variant)
  2. In the language custom translation (for example fr for the French language)
  3. In the language built-in translation (for example fr_default), if any exist for the user language.
  4. In the custom English translation (en).
  5. In the built-in English translation (en_default).

Semarchy xDM includes built-in translations for English and French.
Using the fallback mechanism, you can create custom translations for these two languages to override the built-in translations.

The translations that you provide for these base languages automatically override, in the fallback mechanism, the built-in ones.

To override a single string in English, you need to provide a custom translation for the English language that contains only this string.

Importing Custom Translations

To import custom translations:

  1. In Semarchy Configuration, select the Overview editor and then click on the image Custom Translations link.
    The Custom Translations editor opens.
  2. Click on the image Import Translations…​ button in the editor toolbar.
  3. Click the Browse button and select a zip file containing custom translations.
    The editor shows in the Languages to import the custom translations available in the file.
  4. Select the property files that you want to import, and then click Finish
    The selected translation bundles are imported.

When importing a custom translation for a language that is not translated yet, a new translation is created. If a custom translation already exists for a given language, it is removed and replaced entirely by the new one.

Configuring the Logging

Logging is used to troubleshoot the Semarchy xDM platform issues and monitor the platform activity.

The default logging configuration set up in Semarchy works in most cases. Most of the times, you do not need to change this configuration.
You can reset the logging configuration to the default values by clicking the Restore Logging Configuration to Default button in the Logging Configuration editor toolbar.

By default, the important log events appear in the Semarchy Configuration Error view. To access this error view, in the Semarchy Configuration user menu, choose the Window > Show View…​ > Others and then select General > Error Log.

Semarchy xDM uses Apache Log4J to log its activity. This section provide key concepts and configuration samples to work with Log4J. Always refer to the Log4J Manual for reference information.

Log4J works with three key concepts: Appenders, Loggers and Log Levels.

  • The Appenders deliver log events to a destination. For example they write events to a file on the application server file system, to a message queue or to the Semarchy Configuration Errors view.
  • The Loggers capture log events emitted by software certain components in Semarchy xDM.
  • The Log Levels define at the same time the granularity and importance of log events.

Each log events is emitted with a certain log level, is captured by a logger that sends it to an appender for delivery.

Setting the Logging Configuration

Appenders, loggers and log levels are configured through the Logging Configuration editor.

To set the Logging Configuration:

  1. In Semarchy Configuration, select the Overview editor, and then click the Logging Configuration link. The Logging Configuration editor opens.
  2. Change the configuration in the editor.
  3. Press CTRL-S to save this editor.
    The new logging configuration is immediately taken into account.

Understanding Log Levels

The Log Level define the granularity and importance of a log event. The log level is one of the following: TRACE, DEBUG, INFO, WARN, ERROR, FATAL.

These values are ordered from highest to lowest granularity, but also in order of importance: The DEBUG level is very granular and most messages are not relevant in the normal course of operations, whereas the ERROR and FATAL messages will appear less frequently but are more important.

When you capture events, via a logger, at a certain level, all events of higher importance are also captured. For example, when setting a logger to INFO, then the WARN, ERROR, and FATAL events are also captured.

When parameterizing a logger, it is possible to set the level to ALL to capture all log events, and to OFF to disable log capture.
INFO is typically a good level to start with, as it captures information messages as well as possible warnings and errors.

Defining Appenders

Appenders deliver log events to a destination. Appenders are defined with a name and an appender class (that is, a Java class the writes log events to a file, to a JMS message queue, to an email, etc).
Appender classes may have one or more properties to configure their behavior. They also support Layout properties to define the format of the logged event.

In the logging configuration, the appender is defined with the following syntax:

log4J.appender.<appender_name> = <appender_class>

Each property for this appender is defined with the following syntax:

log4J.appender.<appender_name>.<property_name> = <property_value>
Sample Appenders Configuration

Sample appender configurations are provided in this section.

Configuring an appender called FILE to write to a daily rolling log file.
# Appender Class
log4j.appender.FILE=org.apache.log4j.DailyRollingFileAppender 	(1)

# Appender Properties
log4j.appender.FILE.DatePattern='.'yyyy-MM-dd'.log'		(2)
log4j.appender.FILE.file=${user.home}/.semarchy/logfile.log 	(2)

log4j.appender.FILE.append=true 				(3)
log4j.appender.FILE.encoding=UTF-8 				(3)

# Appender Layout Configuration
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout 	(4)
log4j.appender.FILE.layout.ConversionPattern=%d{yyyy-MM-dd HH\:mm s S}-%X{authenticatedUser}-%t-%x-%-5p-%-10c\:%m%n (4)
1The DailyRollingFileAppender appender class creates a daily logging file. You can also use the FileAppender to write to a simple file.
2The daily log file is located in the .semarchy/ sub-directory of the user home directory. Its name is logfile.log to which the date, formatted with the pattern, is concatenated.
3These properties configure the appender behavior: Add log events at the end of the daily file, and use UTF-8 encoding.
4Logged event are formatted using a PatternLayout pattern string, which supports the date, priority, etc, of the event to diagnose. This pattern also supports a specific %X{authenticatedUser} context key corresponding to the authenticated user performing the operation in Semarchy xDM, if any.
Configuring an appender called PDE to log to the Semarchy Configuration Error view
# Appender writing to the Configuration Error view
log4j.appender.PDE=com.semarchy.commons.log4j.appender.pde.PDEAppender 	(1)
log4j.appender.PDE.Threshold=INFO 					(2)
1This appender class logs events into the Semarchy Configuration Error view.
2Only log events more important that this threshold will be considered by the appender. Make sure to align this value with the loggers that send events to this appender.
Configuring an appender called SMTP to send log events by email
# Appender sending emails via an SMTP host (usually for important error messages)
log4j.appender.SMTP = org.apache.log4j.net.SMTPAppender 		(1)
log4j.appender.SMTP.SMTPHost = <smtp_host_name> 			(2)
log4j.appender.SMTP.SMTPPort = <smtp_host_port> 			(2)
log4j.appender.SMTP.From = email@my-domain.com  			(2)
log4j.appender.SMTP.To = admin1@my-domain.com,admin2@my-domain.com 	(3)
log4j.appender.SMTP.Subject = Log Event 				(3)
log4j.appender.SMTP.layout =  org.apache.log4j.TTCCLayout 		(4)
1This appender class sends log events by email.
2Mail Server configuration. Note that additional properties may be set for the SMTP user, password, protocol, etc.
3Email recipients and subject.
4Log events are formatted using the TTCC Layout, a detailed format containing the time, thread, category and nested diagnostic context.

Defining Loggers

Loggers are named and organized as a hierarchy of classes emitting log events.
For example, com.semarchy is the parent of com.semarchy.mdm. The highest logger in the hierarchy is the rootLogger.

You configure loggers with a log level, and attach them to one or more appenders. Any event emitted by the class that is more important than (or as important as) the given log level is captured by the logger and sent to the appender.

Use the following syntax to attach a logger to appenders with a given log level:

log4j.logger.<loggerName> = <level>, <appender_name> [,<appender_name>, …]

For example, to send DEBUG, INFO, WARN, ERROR and FATAL events emitted by the com.semarchy.platform.setup.IPlatformManager class to the appender named PDE:

log4j.logger.com.semarchy.platform.setup.IPlatformManager=DEBUG, PDE

Logger inherit their appenders additively from their parents in the hierarchy. They can override the level provided by their parent.

In the following configuration, the com.semarchy.platform.setup.IPlatformManager logger inherits from the PDE appender of the com.semarchy logger, and sends ERROR events to the PDE appender in addition to the SMTP appender.

log4j.logger.com.semarchy=INFO, PDE
log4j.logger.com.semarchy.platform.setup.IPlatformManager=ERROR, SMTP

To prevent such inheritance, use the following syntax:

log4j.additivity.<logger_name> = false

For example, in the the following configuration, com.semarchy.platform.setup.IPlatformManager will no longer log into the PDE appender in addition to SMTP.

log4j.logger.com.semarchy=INFO, PDE
log4j.logger.com.semarchy.platform.setup.IPlatformManager=ERROR, SMTP
log4j.additivity.com.semarchy.platform.setup.IPlatformManager=false

The following configuration is the default one for Semarchy xDM.

Sample logger configuration for Semarchy xDM.
log4j.logger.com.semarchy=INFO
log4j.logger.com.semarchy.commons.sql=INFO
log4j.logger.com.semarchy.mdm.datahub.services.query.datamgr.IDataManager=INFO
log4j.logger.com.semarchy.platform.engine=INFO
log4j.logger.com.semarchy.platform.engine.PluginExecution=INFO
log4j.logger.com.semarchy.platform.engine.core.impl.product.SL4JExecutionMonitor=INFO
log4j.logger.com.semarchy.platform.product.notification.JobNotificationHandler=ERROR
log4j.logger.com.semarchy.platform.setup.IPlatformManager=INFO
log4j.logger.org=INFO
log4j.logger.org.apache.aries.blueprint=WARN
log4j.logger.org.apache.commons.beanutils.converters=WARN
log4j.logger.org.apache.cxf=WARN
log4j.logger.org.apache.cxf.services = WARN
log4j.logger.org.apache.directory.shared.asn1.ber.Asn1Decoder=ERROR
log4j.logger.org.ops4j.pax.logging=WARN
log4j.logger.org.quartz=WARN
log4j.logger.org.springframework.jdbc.core.JdbcTemplate=INFO
log4j.rootLogger=INFO, PDE

The following table lists the various loggers available in Semarchy xDM:

Logger NameDescription

com.semarchy

Root logger for the Semarchy platform.

com.semarchy.commons.sql

TRACE logs SQL queries made by the Semarchy Application Builder and Configuration user interfaces.

com.semarchy.mdm.datahub.services.query.datamgr.IDataManager

Logs database operations performed by the platform for the MDM applications while browsing data. DEBUG logs all queries and execution times.

com.semarchy.platform.engine.core.impl.DefaultStandaloneEngineImpl

Logs execution engine events. DEBUG logs all submitted jobs, and ERROR can be used to troubleshoot engine issues.

com.semarchy.platform.engine.core.impl.product.SL4JExecutionMonitor

Logs execution engine job processing. Use DEBUG to see execution engine details.

com.semarchy.platform.engine.PluginExecution

Logs enricher and validation plugins execution. DEBUG logs plugins feedback, TRACE logs the processing of every row.

com.semarchy.platform.integration.polling.IntegrationLoadDequeuer

Logs the Batch Poller activity. DEBUG logs every polled interval.

com.semarchy.platform.product.notification.JobNotificationHandler

Logs job notifications. ERROR traces notification failures.

com.semarchy.platform.setup.IPlatformManager

Logs platform status changes. INFO traces normal status changes. ERROR traces abnormal statuses.

org.apache.cxf.services

INFO traces REST requests, responses, and faults between the MDM applications and the server. This includes also REST requests performed by plugins on external services (Melissa Data, for example).

 org.springframework.jdbc.core.JdbcTemplate

DEBUG traces the SQL queries made by the MDM applications and REST API calls as well as Dashboards-related SQL queries.

 com.semarchy.xdm.rest.accesslog

DEBUG traces REST API calls. This log is in JSON format and contains: username, API key name (or null for basic authentication), origin IP address, date and time, endpoint (relative to base server URL), response code and execution duration.

com.semarchy.mdm.discovery.core.impl.DefaultProfilingService

Logs the profiling activity for xDM Discovery. DEBUG traces the details of the profiling processes and queues.

Managing the Security

The application uses role-based security and privilege grants for accessing the Semarchy xDM features as well as the data contained in the data location.

Understanding the Security Model

Platform-Level and Model-Level Security

There are two levels of security in Semarchy xDM:

  • Platform-Level Security defines access to the features of the platform. For example, access to the administrative features, or access to the design-time capabilities. Platform-level security sets platform users' privileges (who can design models, monitor executions, manage security, etc.).
  • Model-Level Security defines security privileges to access and modify data in the data locations. Defining these privileges is a data governance decision and should be defined as part of the data governance initiative. Defining Model Security is covered in the Securing Data chapter of the Semarchy xDM Developer’s Guide.

Role-Based Security

Both levels of security are role-based:

  • The Privileges (platform level/model level) are granted to Roles in Semarchy xDM.
  • These Roles are declared in Semarchy xDM. The roles declared in Semarchy xDM must map roles that pre-exist in the application server. These application server roles are created and granted to application server users as part of the application server configuration
  • Users logging in to Semarchy xDM use their application server credentials.

Depending on the application server hosting the Semarchy xDM server, the roles/user association may be made through a concept of group: A user belongs to a group and the role is granted to the group.
Depending on its configuration, the application server may delegate user authentication and management in general to a security provider (SSO, LDAP, etc…).

Note that roles are not only used for security purposes. They are also used as email aliases for email notifications.

Security Context

When you log in to Semarchy:

  1. You enter the user and password in the Semarchy xDM login window.
  2. This information is passed to the application server.
  3. The application server itself or its security provider (SSO, LDAP, etc.) authenticates the user, gets the list of roles associated to this user (possibly via groups) and returns this list of roles in the session’s Security Context.
  4. Semarchy xDM starts a session with this security context, allowing:
    • Certain platform features depending on the Platform-Level Privileges granted to the roles.
    • Certain data access/modification capabilities depending on the Model-Level Privileges granted to the roles.
Role Names are Case-Sensitive
The role names defined in Semarchy must exactly match the role names returned by the security provider.
For example, for a Tomcat installation, you define in the tomcat-user.xml file a role named FINANCEADMIN, and in Semarchy, you declare a FinanceAdmin role. These two roles will not match. Tomcat users with the FINANCEADMIN role will not be granted the privileges defined in Semarchy for the FinanceAdmin role.
Semarchy xDM enforces security at several layers in the application. Insufficient privileges for a user will reflect in the user interface as missing elements or disabled menus. In the REST API, insufficient privileges to perform an operation will cause an error with the operation.
Semarchy xDM does not store the users, password and and user/roles associations. All this critical information remains in the application sever or in the enterprise security provider.

Privilege Precedence

Privileges apply in order of precedence: Read/Write then Read then None. As a consequence, a user always has the best privileges associated to all his roles.

For example: The user John has two user-defined roles granted to him:

  • ModelDesigner role has Read privileges for Application Management and Read/Write for Application Design.
  • ProductionManager has Read/Write privileges for Application Management and None on for Application Design

The resulting privileges for John are Read/Write for both Application Management and Application Design.

Privileges Description

The following table describes the platform privileges you can grant to a role:

Platform PrivilegeDescription

Platform Administration

Grants access to the Semarchy Configuration interface to view or configure the Repository, Notification Servers, Variable Value Providers, Image Libraries, Plug-ins, Logging Configuration and Applications Configuration.

This privilege does not give access to the Roles configuration. Only a user with the SemarchyAdmin role can create and modify roles.

Application Design

Grants access to all the components of the Model Design perspective in the Application Builder to view or design models.
Grants also access to the model creation/export, as well as model editions management (close a model editions, create a branch, manage the translations), and the Image Library.

Application Management

Grants access to all components for model and application management (the Management view in the Application Builder), including deploying model editions, creating and configuring the data locations (notification policies, continuous loads, etc), the batch poller, the execution engine, the job logs and the purge schedules. This privilege also allows upgrading the data locations, and configuring Variable Value Providers.

Dashboard Management

Grants access to the Dashboard Builder to create applications in Semarchy xDM Dashboards.

Discovery Management

Grants access to xDM Discovery to define, profile and browse the profiling statistics of datasources.

Built-in Roles

The following roles are built in the platform:

  • semarchyConnect: This role must be granted for a user to log in. It should be granted by default to all users connecting to Semarchy xDM.
  • semarchyAdmin : This role has full access to all the features of the platform. semarchyAdmin is the only role that gives you access to the Roles in the Semarchy Configuration user interface.
When a creating a new model, a model-level privilege grant is automatically created for the semarchyAdmin role, giving this role full access to the data. By modifying this privilege grant, the model designer can reduce the privileges of the semarchyAdmin role on the data.
Be cautious when granting the semarchyAdmin role. This role defines a super user who can create roles, grant privileges and update the license information.

Managing Roles and Privileges

Creating the Roles and Users in the Application Server Security Realm

Before declaring a new role in Semarchy xDM, make sure that this role is defined in the application server and that a user is granted with this role and the semarchyConnect role to log in to Semarchy xDM.

The role/user creation procedure depends on the application server hosting Semarchy xDM. Please refer to your application server documentation for more information.

An example is given below for creating a role and a user for Apache Tomcat.

To configure a new role and user for Semarchy xDM:

  1. Stop the Apache Tomcat Server using the stop the Apache Tomcat server using <tomcat>/bin/shutdown.bat (Windows) or <tomcat>/bin/shutdown.sh (UNIX/Linux), where <tomcat> is the Apache Tomcat installation folder.
  2. Edit the <tomcat>/conf/tomcat-users.xml file.
  3. In the <tomcat-users> section, add the following lines (<password> is the password for this user):
     <role rolename="MDMDev">
     <user username="john" password="<password>" roles="semarchyConnect,MDMDev"/>
  4. Save the file.
  5. Restart the Apache Tomcat server using <tomcat>/bin/startup.bat (Windows) or <tomcat>/bin/startup.sh (UNIX/Linux).

A new role MDMDev is created. The user john is also created with the semarchyConnect built-in role and the MDMDev role.

Declaring the Roles in Semarchy xDM

To create a new role:

  1. In Semarchy Configuration, select the Overview editor, and then click the Roles link.
  2. In the Roles editor, right-click in the Roles table and select image New Role. The Install Role wizard opens.
  3. Enter the following information:
    • Name: Role name. This role name must exactly match the role name in the application server security configuration. For example: MDMDev.
    • Label: User-friendly label for the role. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.
    • Email(s): Enter a comma-separated list of email addresses of recipients for notifications sent to this role.
  4. Click Next.
  5. Select the privileges to grant to this role. For example: Model Design: Read/Write, Job and Job Log Administration: Read.
  6. Click Finish.
    The role is created. You can connect a user with this role to test the set of privileges.
Make sure to use a role name that matches exactly (with the same case) a role name defined in the application server configuration.

Sample Roles

You can use the following role examples in a typical Semarchy xDM configuration:

Platform PrivilegeDeveloperProduction UserAdministrator

Platform administration

Read

Read

Read/Write

Application Design

Read/Write

Read

Read

Application Management

Read

Read/Write

Read/Write

Dashboard Management

Read/Write

None

None

Discovery Management

Read/Write

None

None

These roles are given as examples and should be adapted to your environment’s requirements.

Scripting Administration

All platform administration tasks are exposed by the REST API to enable automated administration and environment setup.

Model and application management operations such as creating data locations or managing model edition are also available in the REST API. See the Scripting Deployment and Management chapter in the Semarchy xDM Application Management Guide for more information.

Scripted Administration Overview

The REST API exposes endpoints for the following administrative operations:

  • Repository: Create and upgrade repositories, get the platform status and repository information.
  • License: Read current license information and upload a new license file.
  • Roles: Export and import (replace all) roles in the platform.
  • Image Libraries: List available libraries, export and import image libraries content in a zip format.
  • Plugins Bundles: List, delete, import and export bundles containing enricher and validation plugins.
  • Notification Servers and Variable Value Providers: List, get, set or delete the configuration of all or some of these servers.
  • Users: List, create, update or delete all or specific user profiles with their preferences.
  • API Keys: Create, delete or update REST API Keys.
  • Configuration: configure the platform and the logging configuration.
  • Custom Translations: Export and import custom translations.
  • REST Clients: Export and import REST client definitions.
For details about using the endpoints, see the built-in REST API documentation.

REST API Documentation

The REST API exposes its built-in documentation as a link in the Welcome page. In addition, this documentation is available for tools as an OpenAPI specification.

To Access the REST API Documentation:

  1. Log in to Semarchy xDM as a user with one of the Application Design, Application Management and Platform Administration platform privileges.
  2. Click the REST API link on the Welcome page.
  3. In the upper-right menu, select Platform Administration.

The documentation exposes the endpoint description and provides request and response samples.