Connector Configuration

GitLab Configuration

GitLab Instance Configuration

These configuration options determines the GitLab instance to traverse, the user for that, and the HTTP settings to use.

Name Description

Base URL

Base URL of the GitLab instance to be traversed and indexed. Opening this URL in a browser should yield the start page of GitLab (after authentication), adding the REST API suffix (/api/v4) is not supported.

Auth Token

Personal Access Token created on the GitLab instance for/with the technical user to be used to traverse GitLab.

Max. Requests per Second

The maximum number of HTTP requests to send to the GitLab instance per second. By default, 100 requests per second are allowed.

This is only an upper limit, the connector may or may not actually reach it at runtime.

HTTP Timeout

The maximum time to wait for response to HTTP requests. By default, the connector waits for 30 seconds.

GitLab Content Processor Configuration (Optional)

This configuration option determines which types of objects in GitLab shall be indexed. Traversals are not otherwise influenced.

Name	Description
Choose object types to be indexed	The object types to be processed and sent to the target system to be indexed. Excluding any of the following object types will not influence the traversal of GitLab or the indexing of other object types. Projects adds GitLab projects themselves as documents to the index, including the projects' description, the URL to their avatars, their topics, and their readme files (if available). Branches adds all repository branches as documents to the index. Tags adds all repository tags as documents to the index. Commits adds all repository commits as documents to the index, including their message, timestamps, and users involved. Files at the head of branches adds all files included in the commits that are the head of any included branch to the index. Files in tagged commits adds all files included in any commits that are tagged to the index. All file versions adds all versions of all files referenced by any commit to the index. By default, only projects, commits, files at the head of branches, and files in tagged commits are included.

Name

Description

Choose object types to be indexed

The object types to be processed and sent to the target system to be indexed. Excluding any of the following object types will not influence the traversal of GitLab or the indexing of other object types.

Projects: adds GitLab projects themselves as documents to the index, including the projects' description, the URL to their avatars, their topics, and their readme files (if available).
Branches: adds all repository branches as documents to the index.
Tags: adds all repository tags as documents to the index.
Commits: adds all repository commits as documents to the index, including their message, timestamps, and users involved.
Files at the head of branches: adds all files included in the commits that are the head of any included branch to the index.
Files in tagged commits: adds all files included in any commits that are tagged to the index.
All file versions: adds all versions of all files referenced by any commit to the index.

By default, only projects, commits, files at the head of branches, and files in tagged commits are included.

GitLab Projects Filter (Optional)

These configuration options determine which projects are traversed and their contents indexed. Projects and everything they contain can be filtered by excluding or including project paths. By default, all non-archived projects are traversed and indexed (if not excluded through the GitLab Content Processor Configuration).

This filter also affects the traversal and indexing of all branches, tags, commits, and files: those in an excluded project are not traversed and not indexed, either.

Both the include list and the exclude list use path prefixes: each entry names either one project or one namespace. Projects are used as is, namespaces include all subgroups of that namespace and all projects in that namespace and its subgroups.

As in GitLab, all entries in the include and exclude lists are case-insensitive. The entries should not end with a slash or contain multiple consecutive slashes.

The project filter does not use longest-prefix matching: for a project to be traversed and indexed, its path must match

at least one entry in the include list if that is not empty, and
none of the entries in the exclude list.

So, if the exclude list contains for example an entry group/subgroup, adding an entry group/subgroup/project-a to the include list will not make the connector traverse project-a.

Name	Description
Included Projects	The path prefixes of projects to be included in traversals. If this list is empty (the default), all projects that are not excluded will be included automatically.
Excluded Projects	The path prefixes of projects to be excluded from traversals. By default, no projects are excluded.
Synchronize archived projects	Whether archived projects should be included in synchronization. By default, archived projects are ignored.

Name

Description

Included Projects

The path prefixes of projects to be included in traversals. If this list is empty (the default), all projects that are not excluded will be included automatically.

Excluded Projects

The path prefixes of projects to be excluded from traversals. By default, no projects are excluded.

Synchronize archived projects

Whether archived projects should be included in synchronization. By default, archived projects are ignored.

GitLab Branches/Tags Filter (Optional)

These configuration options determine which branches and which tags are traversed and their contents indexed. By default, all branches and all tags of traversed projects are traversed and indexed (if not excluded through the GitLab Content Processor Configuration).

Branches and tags each have their own filter; they are only described as one to reduce the duplication here.

This filter also affects the traversal and indexing of all commits and files: those in an excluded branch/tag are not traversed and not indexed, either.

For added flexibility, all entries here are interpreted as regular expressions as they are defined by the class java.util.regex.Pattern in Java 8, using the find() method of the corresponding Matcher class to define whether an entry matches the name of the branch/tag (without the refs/heads/ or refs/tags/ prefix). That is, a match can start anywhere in the name and does not need to match everything up to the end of the name.

The branches filter and the tags filter do not use longest-prefix matching: for a branch/tag to be traversed and indexed, its name must match

at least one entry in the include list if that is not empty, and
none of the entries in the exclude list.

Name	Description
Included Branch/Tag Regular Expressions	The regular expressions of branches/tags to be included in traversals. If this list is empty (the default), all branches/tags that are not excluded will be included automatically.
Excluded Branch/Tag Regular Expressions	The regular expressions of branches/tags to be excluded from traversals. By default, no branches/tags are excluded.

Name

Description

Included Branch/Tag Regular Expressions

The regular expressions of branches/tags to be included in traversals. If this list is empty (the default), all branches/tags that are not excluded will be included automatically.

Excluded Branch/Tag Regular Expressions

The regular expressions of branches/tags to be excluded from traversals. By default, no branches/tags are excluded.

GitLab File Filter (Optional)

These configuration options determine which files are indexed. By default, all files of traversed branches, tags, and commits are indexed (if not excluded through the GitLab Content Processor Configuration).

For added flexibility, all entries here are interpreted as regular expressions as they are defined by the class java.util.regex.Pattern in Java 8, using the find() method of the corresponding Matcher class to define whether an entry matches the full path of the file (without any / prefix). That is, a match can start anywhere in the name and does not need to match everything up to the end of the name.

The file filter does not use longest-prefix matching: for a file to be indexed, its name must match

at least one entry in the include list if that is not empty, and
none of the entries in the exclude list.

Name Description

Name	Description
Included File Regular Expressions	The regular expressions of files to be indexed. If this list is empty (the default), all files that are not excluded will be included automatically.
Excluded File Regular Expressions	The regular expressions of files to be excluded from the index. By default, no files are excluded.
Maximum file size	For files with content larger than this size, only the metadata will be indexed. If the size of the content is known to be bigger than this size in advance, the connector will not fetch it from GitLab. Setting this to zero (0) bytes is supported and will result in: no files having any content, and the connector not accessing the API endpoints to fetch such content. In those cases where the content of projects is fetched from a `README` file, this property is taken into account when determining eligible `README` files, too.
Fetch file creator and contributors	Whether to fetch the information which committer created a file and which committers all modified that file. This requires one potentially expensive API call per indexed file and is therefore disabled by default.

Included File Regular Expressions

The regular expressions of files to be indexed. If this list is empty (the default), all files that are not excluded will be included automatically.

Excluded File Regular Expressions

The regular expressions of files to be excluded from the index. By default, no files are excluded.

Maximum file size

For files with content larger than this size, only the metadata will be indexed. If the size of the content is known to be bigger than this size in advance, the connector will not fetch it from GitLab. Setting this to zero (0) bytes is supported and will result in:

no files having any content, and
the connector not accessing the API endpoints to fetch such content.

In those cases where the content of projects is fetched from a README file, this property is taken into account when determining eligible README files, too.

Fetch file creator and contributors

Whether to fetch the information which committer created a file and which committers all modified that file. This requires one potentially expensive API call per indexed file and is therefore disabled by default.

CSM Configuration

Required Configuration Properties

CSM Connection Settings

Configuration options for the connection to the target CSM instance.

Name	Description
CSM endpoint	URL of the CSM instance to connect to.

Name

Description

CSM endpoint

URL of the CSM instance to connect to.

CSM Authentication Settings

Configuration Options for the authentication against the target CSM instance.

Name	Description
Username	Username of the technical user.
Password	Password of the technical user.

Name

Description

Username

Username of the technical user.

Password

Password of the technical user.

Optional Configuration Properties

CSM Connection Settings

Configuration options for fine-tuning the Http connection parameters.

Name	Description
Concurrent Connections	Maximum number of concurrent open connections.
Requests Rate	Maximum number of requests per second.
Connect Timeout in Milliseconds	Timeout of the connect request.
Socket Timeout in Milliseconds	Timeout of the socket connected to CSM.
Request Timeout in Milliseconds	Timeout of a request to CSM.

Name

Description

Concurrent Connections

Maximum number of concurrent open connections.

Requests Rate

Maximum number of requests per second.

Connect Timeout in Milliseconds

Timeout of the connect request.

Socket Timeout in Milliseconds

Timeout of the socket connected to CSM.

Request Timeout in Milliseconds

Timeout of a request to CSM.

CSM Ingestion Settings

Configuration options to specify how principals are ingested in the CSM.

Name	Description
Domain	Namespace under which to ingest principals.

Name

Description

Domain

Namespace under which to ingest principals.

Apache Solr Configuration

Instance Configuration

Configuration options related to specifying the target Solr Instance and Collection including authentication/authorization settings.

Setting Description

Setting	Description
Deployment	Deployment of the target Solr instance. Use `Collection Server` to feed against a standalone collection server. If your target instance is a Cloud Deployment, select the option `Zookeeper`.
Collection Server URL	For Standalone Collection Server, the URL to the collection server including the data path, e.g. `http://localhost:8983/solr`.
Zookeeper Instances	For Solr Cloud target, list of zookeeper instances including host and port.
Zookeeper Chroot	Znode Chroot of the cluster.
Collection ID	ID of the target collection.
Use Authentication	Enable this option to use Basic Authentication to authenticate against your Solr instance.
Username	Basic Authentication Username, if authentication is enabled.
Password	Basic Authentication Password, if authentication is enabled.
Use Proxy	If enabled, the connection to the Solr instance will be established through HTTP/HTTPS proxy.
Proxy Endpoint	Target proxy URL including protocol, host and port.
Proxy Authentication	If enabled, the connector uses the specified credentials to authenticate towards proxy.
Proxy Username	Proxy authentication username.
Proxy Password	Proxy authentication password. The value will be stored encrypted by the connector.
ACL Prefix	A string prepended to ACE entries in ACLs.

Deployment

Deployment of the target Solr instance. Use Collection Server to feed against a standalone collection server. If your target instance is a Cloud Deployment, select the option Zookeeper.

Collection Server URL

For Standalone Collection Server, the URL to the collection server including the data path, e.g. http://localhost:8983/solr.

Zookeeper Instances

For Solr Cloud target, list of zookeeper instances including host and port.

Zookeeper Chroot

Znode Chroot of the cluster.

Collection ID

ID of the target collection.

Use Authentication

Enable this option to use Basic Authentication to authenticate against your Solr instance.

Username

Basic Authentication Username, if authentication is enabled.

Password

Basic Authentication Password, if authentication is enabled.

Use Proxy

If enabled, the connection to the Solr instance will be established through HTTP/HTTPS proxy.

Proxy Endpoint

Target proxy URL including protocol, host and port.

Proxy Authentication

If enabled, the connector uses the specified credentials to authenticate towards proxy.

Proxy Username

Proxy authentication username.

Proxy Password

Proxy authentication password. The value will be stored encrypted by the connector.

ACL Prefix

A string prepended to ACE entries in ACLs.

Advanced HTTP Configuration (Optional)

Configuration options for fine-tuning the Http connection parameters.

Setting	Description
Socket Timeout	Timeout value for receiving data from server.
Connection Timeout	Timeout value for establishing a connection to server.
Connection Request Timeout	Timeout value for requesting a connection from connection manager.
Max. Number of Connections	Max. number of connections maintained by the connection manager.
Max. Number Requests per Second	Max. number of requests send to the server per second.

Setting

Description

Socket Timeout

Timeout value for receiving data from server.

Connection Timeout

Timeout value for establishing a connection to server.

Connection Request Timeout

Timeout value for requesting a connection from connection manager.

Max. Number of Connections

Max. number of connections maintained by the connection manager.

Max. Number Requests per Second

Max. number of requests send to the server per second.

Cultural/Language Suffix Configuration (Optional)

Specify the fields which should be appended with the cultural suffix retrieved from the language information attached to an item.

Setting Description

Setting	Description
Standard Fields	Select any arbitrary fields to append the cultural suffix from a list of standardized fields provided by the connector. The fields are: `Content`, `Source`, `Title`, `Item Type`, `Keywords`, `Author`, and `Contributors`.
Include Additional Fields	Enable this option to include also additional fields not listed in the standard field list, e.g. fields generated by the pipeline.
Additional Fields	List of field names to apply the suffix to.
Fallback Language	Fallback language code in ISO 639 format to apply for items with missing language information.

Standard Fields

Select any arbitrary fields to append the cultural suffix from a list of standardized fields provided by the connector. The fields are: Content, Source, Title, Item Type, Keywords, Author, and Contributors.

Include Additional Fields

Enable this option to include also additional fields not listed in the standard field list, e.g. fields generated by the pipeline.

Additional Fields

List of field names to apply the suffix to.

Fallback Language

Fallback language code in ISO 639 format to apply for items with missing language information.

Metadata Mapping Configuration (Optional)

Defines Solr Field Names for standardized metadata provided by the connector.

Setting Description

Setting	Description
Mapping Entries	Mapping entries which will be applied to determine the field name in the Solr schema. Metadata with missing entry in this list will be processed with its default field name. The available fields are: `Content`, `Allow Access Control List`, `Deny Access Control List`, `Source Name`, `Click URL`, `Title`, `Item Type`, `Mime Type`, `File Extension`, `Preview URL`, `Keywords`, `Languages`, `Author`, `Contributors`, `Created Date`, `Last Modified Date`, `Breadcrumbs`, `Breadcrumb URLS`, or a custom field.

Mapping Entries

Mapping entries which will be applied to determine the field name in the Solr schema. Metadata with missing entry in this list will be processed with its default field name. The available fields are: Content, Allow Access Control List, Deny Access Control List, Source Name, Click URL, Title, Item Type, Mime Type, File Extension, Preview URL, Keywords, Languages, Author, Contributors, Created Date, Last Modified Date, Breadcrumbs, Breadcrumb URLS, or a custom field.

General Configuration

Database Configuration

Name Property Key Description

Name	Property Key	Description
URL	`spring.datasource.url`	JDBC URL for the target database. Out of the box, the connector will use H2 file database. For productive usage, use PostgreSQL specifying the URL in format: `jdbc:postgresql:<host>:<port>/<database>`
Username	`spring.datasource.username`	Database Username to read and write to database.
Password	`spring.datasource.password`	Database Password for the specified user

URL

spring.datasource.url

JDBC URL for the target database. Out of the box, the connector will use H2 file database. For productive usage, use PostgreSQL specifying the URL in format: jdbc:postgresql:<host>:<port>/<database>

Username

spring.datasource.username

Database Username to read and write to database.

Password

spring.datasource.password

Database Password for the specified user

Traversal Configuration

Name Property Key Description

Name	Property Key	Description
Traversal History Length	`raytion.connector.agent.traversal .store.historyLength`	Max. number of traversals to store in the history. Once the limit is exceeded, the connector will automatically remove oldest entries in the history. (default: 100)
Number of Traversal Workers	`raytion.connector.agent.traversal .workers.worker`	Number of workers to execute the traversal in parallel. Increasing this value might improve the performance, but will footprint higher memory consumption. It is recommended to keep the default value. (default: 10)
Traversal Job Poll Interval	`raytion.connector.agent.traversal .workers.jobPollInterval`	Interval between the workers to be triggered to fetch and process the next tasks. (default: 10ms)
Completion Timeout	`raytion.connector.agent.traversal .workers.completionTimeout`	If the search engine indexes the items asynchronously, there might be some processing still in-flight during the completion process of a traversal. This value specifies the timeout value until all asynchronous callbacks are expected to return before completing the traversal. (default: 10m)

Traversal History Length

raytion.connector.agent.traversal .store.historyLength

Max. number of traversals to store in the history. Once the limit is exceeded, the connector will automatically remove oldest entries in the history. (default: 100)

Number of Traversal Workers

raytion.connector.agent.traversal .workers.worker

Number of workers to execute the traversal in parallel. Increasing this value might improve the performance, but will footprint higher memory consumption. It is recommended to keep the default value. (default: 10)

Traversal Job Poll Interval

raytion.connector.agent.traversal .workers.jobPollInterval

Interval between the workers to be triggered to fetch and process the next tasks. (default: 10ms)

Completion Timeout

raytion.connector.agent.traversal .workers.completionTimeout

If the search engine indexes the items asynchronously, there might be some processing still in-flight during the completion process of a traversal. This value specifies the timeout value until all asynchronous callbacks are expected to return before completing the traversal. (default: 10m)

Principal Aliaser Configuration

Principal Aliasing is applied on user information as part of Content ACL processing during Content Synchronization and Principal processing during Principal Synchronization. It’s purpose is to map external source system user to the corresponding user in search engines domain. You can configure a list of aliasers in the connector which will be applied in sequence and in order on user ACEs and user principals. The Connector supports following custom aliasing mechanism.

Custom Aliaser Disabled

If the Custom Aliaser checkbox is not selected, the connector will process user information on ACE and user principals unchanged to Search Engine. If all relevant users in the source system can be found with the same identifier in the search engine, this setup is sufficient to reflect the same secure search experience in the search engine as defined by the policy in the source system. The connector uses this option as default to process user information.

Custom Aliaser Enabled

If custom aliasing is enable then there are four types of aliaser avaialble:

Simple XML Table Aliaser

Static mapping table which can be uploaded as XML file. The connector uses the uploaded file as lookup table to map a user in the source system to a user in the search engine. Users missing a record in the file will be dropped from the ACE and during Principal Synchronization. This option is only recommended for environment with a manageable amount of users as for each user the corresponding mapping entry needs to be specified in the file.

Name	Description
XML Mapping File	Browse and upload or drag and drop.

Name

Description

XML Mapping File

Browse and upload or drag and drop.

Sample XML mapping file:

<?xml version="1.0" encoding="UTF-8"?>
<storeddata>
    <entry keyValue="user1">user1@raytion.com</entry>
    <entry keyValue="user2">user2@raytion.com</entry>
    <entry keyValue="user3">user3@raytion.com</entry>
</storeddata>

Regex Replacer Aliaser

Regex Replacer Aliaser computes aliases based on a regular expression. Principals that match the regular expression are replaced by the Substitution String.

Name Property Key Description

Name	Property Key	Description
Pattern	`raytion.connector.aliaser.aliasers[*] .replacer.pattern`	The regular expression to match, this is the part that will be replaced. If braces (…) are used in the pattern then the matched value can be retrieved using $1
Substitute String	`raytion.connector.aliaser.aliasers[*] .replacer.substituteString`	String to replace the matching part of the find string. Matched value is accessed by employing $1

Pattern

raytion.connector.aliaser.aliasers[*] .replacer.pattern

The regular expression to match, this is the part that will be replaced. If braces (…) are used in the pattern then the matched value can be retrieved using $1

Substitute String

raytion.connector.aliaser.aliasers[*] .replacer.substituteString

String to replace the matching part of the find string. Matched value is accessed by employing $1

Regex Extractor Aliaser

Regex Extractor Aliaser computes aliases based on a regular expression. Principals that match the regular expression are inserted into the Insert-Into String.

Name PropertyKey Description

Name	PropertyKey	Description
Pattern	`raytion.connector.aliaser.aliasers[*] .extractor.pattern`	The regular expression to match, this is the part that will be inserted into the new value. If braces (…) are used in the pattern then the matched value can be retrieved using $$
Insert-Into String	`raytion.connector.aliaser.aliasers[*] .extractor.insertIntoString`	String to replace the matching part of the pattern. Matched value is accessed by employing $$

Pattern

raytion.connector.aliaser.aliasers[*] .extractor.pattern

The regular expression to match, this is the part that will be inserted into the new value. If braces (…) are used in the pattern then the matched value can be retrieved using $$

Insert-Into String

raytion.connector.aliaser.aliasers[*] .extractor.insertIntoString

String to replace the matching part of the pattern. Matched value is accessed by employing $$

LDAP Aliaser

Ldap Aliaser searches for an LDAP entry with the requested name in the input value and returns the specified output attribute.

Name Property Key Description

Name	Property Key	Description
Host	`raytion.connector.aliaser.aliasers[*] .ldap.host`	Fully Qualified Domain Name of an LDAP server
Port	`raytion.connector.aliaser.aliasers[*] .ldap.port`	Port to use for LDAP connection, defaults are 389/636 or (recommended) 3268/3269 for simple/SSL
AccountDN	`raytion.connector.aliaser.aliasers[*] .ldap.bindAccountDN`	AccountDN for bind to LDAP
Password	`raytion.connector.aliaser.aliasers[*] .ldap.password`	Password part of credentials
Input Field	`raytion.connector.aliaser.aliasers[*] .ldap.inputField`	The Active Directory attribute name for this equality filter
Search Root DN	`raytion.connector.aliaser.aliasers[*] .ldap.baseDN`	Distinguished Name of the subtree which is searched. The smaller the subtree the better the performance but the higher the chance of encountering principals which are not part of this subtree
Output Field	`raytion.connector.aliaser.aliasers[*] .ldap.outputField`	Attribute that should be returned in result entries

Host

raytion.connector.aliaser.aliasers[*] .ldap.host

Fully Qualified Domain Name of an LDAP server

Port

raytion.connector.aliaser.aliasers[*] .ldap.port

Port to use for LDAP connection, defaults are 389/636 or (recommended) 3268/3269 for simple/SSL

AccountDN

raytion.connector.aliaser.aliasers[*] .ldap.bindAccountDN

AccountDN for bind to LDAP

Password

raytion.connector.aliaser.aliasers[*] .ldap.password

Password part of credentials

Input Field

raytion.connector.aliaser.aliasers[*] .ldap.inputField

The Active Directory attribute name for this equality filter

Search Root DN

raytion.connector.aliaser.aliasers[*] .ldap.baseDN

Distinguished Name of the subtree which is searched. The smaller the subtree the better the performance but the higher the chance of encountering principals which are not part of this subtree

Output Field

raytion.connector.aliaser.aliasers[*] .ldap.outputField

Attribute that should be returned in result entries