Connector Configuration
Configuration for Microsoft SharePoint Online
Please note that certain settings affect the performance of content or principal traversals, see Performance section of the FAQ for details.
Mandatory Settings
In order for the connector to connect to SharePoint Online, some actions are required on the source system as well as on the connector site. The following sections assume that the SharePoint Online related preparations described in SharePoint Online Setup have been done.
Authentication Settings
To establish a successful connection to Microsoft SharePoint Online an Azure application with certificate authentication has to be configured.
| Name | Key | Description |
|---|---|---|
Directory (Tenant) ID |
|
The ID of the Microsoft tenant. It can be found in the 'Overview' tab of the corresponding Azure application registration in the Azure Portal. |
Application (Client) ID |
|
The ID of the application registered in the Azure Portal. It can be found in the 'Overview' tab of the corresponding Azure application registration in the Azure Portal. |
Certificate File |
|
Upload for a certificate file (extension: pfx). See Certificate File Generation for a guide on how to generate a certificate. |
Certificate Password |
|
The password of the certificate file. See Certificate File Generation for a guide on how to generate a certificate. |
Optional Settings
The following sections describe which optional configuration adjustments can be done.
Principal Synchronization Settings
In various scenarios, varying strategies for principal resolution may be of interest. The connector offers support for three distinct modes:
-
Full SharePoint Online and all Azure AD groups and users:-
Fetches and resolves all SharePoint Online site collection specific groups.
-
Fetches and resolves all Entra ID users and groups.
-
-
Full SharePoint Online and referenced Azure AD groups and users-
Fetches and resolves all SharePoint Online site collection specific groups.
-
Resolves only Entra ID groups which were found as members of site collection groups.
-
-
Only SharePoint Online groups-
Fetches and resolves all SharePoint Online site collection specific groups.
-
Entra ID users and groups will not be fetched nor resolved.
-
| Resolution of SharePoint Online as well as Entra ID principals is required for secure search. This is the case, because Microsoft added Entra ID support on top of an already existing permission structure in SharePoint Online. |
| Name | Key | Description |
|---|---|---|
Principal Crawl Algorithm |
|
Defines how and in which scope the principal synchronization should be done.
Possible values are:
We recommend |
Site Discovery Settings
SharePoint Online’s top level content structure are site collections. Each site collection contains sites and lists. In order to define which site collection to traverse, the connector provides two strategies. Either a list of site collections is provided or a request against the Microsoft API will be done, fetching all site collections present in the given tenant. The result of the automatic site discovery can be narrowed down via filtering.
| Name | Key | Description |
|---|---|---|
Auto Site Discovery Enabled |
|
Defines if auto site discovery will be used. To manually state the site
collections to be traversed, unset this flag. Note: In order to use the
automatic site discovery, the Azure application requires the Microsoft Graph
permission |
Site Collections |
|
Upload for a file which contains site collections delimited by line. Only the site collections found in this file will be considered in the traversals. Needs to be configured if auto site discovery is disabled. |
General Filter Settings
The following general filter settings can be configured to restrict the traversals.
| Name | Key | Description |
|---|---|---|
Max File Size |
|
Maximal allowed size for content of documents. If the size is exceeding this value, the content of the document is discarded and ignored. |
Ignore ACL |
|
If this flag is set, the documents will be public. For public search scenarios this makes sense, because fewer requests will be executed. Furthermore, fewer permissions are required for the content traversal. |
Site Collection Filter Settings
Configuration options to restrict the traversal of site collections.
| Name | Key | Description |
|---|---|---|
Enable Site Collection URL Filtering |
|
Enables site collection filtering based on URL. |
Site Collection URL Matching Method |
|
Defines how the include / exclude entries should be matched against the site collection URL. Current supported methods are: EXACT: The site collection is filtered if its URL matches exactly with the entry. PREFIX: The site collection is filtered if its URL starts with the entry. SUFFIX: The site collection is filtered if its URL ends with the entry. SUBSTRING: The site collection is filtered if its URL contains the entry. REGEX: The site collection is filtered if its URL matches the RegEx pattern defined in the entry. |
Site Collection URL Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
Site Collection URL Include List |
|
A list of URL-encoded site collection URLs. The include list entries are matched against site collection URLs. On match, the corresponding site will be traversed. |
Site Collection URL Exclude List |
|
A list of URL-encoded site collection URLs. The exclude list entries are matched against site collection URLs. On match, the corresponding site will not be traversed. |
Enable Geolocation Code Filtering |
|
Enables site collection filtering based on the geolocation code. |
Geolocation Code Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
Geolocation Code Include List |
|
An include list of geolocation codes, against which all discovered site collections are evaluated via exact matching. If no include geolocation codes are configured, all site collections found will be traversed. For a list of available geolocation codes see the Official Microsoft Documentation. This list can not be configured simultaneously with the exclude list. |
Geolocation Code Exclude List |
|
An exclude list of geolocation codes, against which all discovered site collections are evaluated via exact matching. If no exclude geolocation codes are configured, all site collections found will be traversed. For a list of available geolocation codes see the Official Microsoft Documentation. This list can not be configured simultaneously with the include list. |
Site Filter Settings
Configuration options to restrict the traversed content based on site templates.
| Name | Key | Description |
|---|---|---|
Enable Site Template Filtering |
|
Enables site filtering based on templates. |
Site Template Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
Included Site Templates |
|
An include list of site templates, against which all discovered site collections are matched via prefix matching.
If no include site templates are configured, all site collections found will be traversed.
The following site templates may be of interest: |
Excluded Site Templates |
|
An exclude list of site templates, against which all discovered site collections are matched via prefix matching. If no exclude site templates are configured, all site collections found will be traversed. Can not be configured simultaneously with include site templates. See Site Templates for a short guide on how to list all site templates present in your tenant. |
Enable Site URL Filtering |
|
Enables site collection filtering based on URL. |
Site URL Matching Method |
|
Defines how the include / exclude entries should be matched against the site URL. Current supported methods are: EXACT: The site is filtered if its URL matches exactly with the entry. PREFIX: The site is filtered if its URL starts with the entry. SUFFIX: The site is filtered if its URL ends with the entry. SUBSTRING: The site filtered if its URL contains the entry. REGEX: The site is filtered if its URL matches the RegEx pattern defined in the entry. |
Site URL Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
Site URL Include List |
|
A list of URL-encoded site URLs. The include list entries are matched against site URLs. On match, the corresponding site will be traversed. |
Site URL Exclude List |
|
A list of URL-encoded site URLs. The exclude list entries are matched against site URLs. On match, the corresponding site will not be traversed. |
List Filter Settings
Configuration options to restrict the traversed lists.
| Name | Key | Description |
|---|---|---|
Traverse Generic Lists |
|
When disabled, generic lists will not be traversed by the connector. |
Traverse Document Libraries |
|
When disabled, document libraries will not be traversed by the connector. |
Enable List Template ID Filtering |
|
Enables list filtering based on template IDs. For a list of standard list templates see the Microsoft Documentation. |
List Template ID Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
List Template ID Include List |
|
Defines list templates which should be traversed. This list can not be used together with the template ID exclude list. For a list of standard list templates see the Microsoft Documentation. |
List Template ID Exclude List |
|
Defines list templates which should not be traversed. This list can not be used together with the template ID include list. For a list of standard list templates see the Microsoft Documentation. |
Enable List Title Filtering |
|
Enables list filtering based on titles. |
List Title Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
List Title Matching Method |
|
Defines how the include / exclude entries should be matched against the list title. Current supported methods are: EXACT: The list is filtered if its title matches exactly with the entry. PREFIX: The list is filtered if its title starts with the entry. SUFFIX: The list is filtered if its title ends with the entry. SUBSTRING: The list is filtered if its title contains the entry. REGEX: The list is filtered if its title matches the RegEx pattern defined in the entry. |
List Title Include List |
|
Its entries are matched against list titles. On match, the corresponding list will be traversed. |
List Title Exclude List |
|
Its entries are matched against list titles. On match, the corresponding list will not be traversed. |
File Extension Filter Settings
Configuration options to restrict the traversed files based on their extension. Files which are effected by this filter are document library items and generic list item attachments.
| Name | Key | Description |
|---|---|---|
Enable File Extension Filtering |
|
Enables filtering based on file extensions. |
File Extension Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
File Extension Include List |
|
The include list entries are matched against file names. Only if the file name ends with an extension mentioned in this list, the corresponding file will be traversed. |
File Extension Exclude List |
|
The exclude list entries are matched against file names. If the file name ends with an extension mentioned in this list, the corresponding file will not be traversed. |
Transcription Settings
Microsoft SharePoint Online supports transcriptions for audio and video files. By enabling this option the connector will fetch the default transcription for such files and add its metadata and content to the document library item.
(File extensions currently supported by the connector: mp4, avi, mp4v, wmv, mov)
| Name | Key | Description |
|---|---|---|
Enable Transcription Fetching |
|
If enabled the connector will embed the default transcription into document library items. |
Transcription Placement |
|
The transcription content is either placed into the item’s metadata or it will replace the binary content of the item. (Transcription metadata is always added to the library item’s metadata.) |
Metadata Settings
Configuration options to customize the metadata provided by the connector.
| Name | Key | Description |
|---|---|---|
Enable Manual Item Type Definition |
|
Enables manual definition of item types. |
Override Site Item Type |
|
Override the item type for site objects. Default: |
Override List Item Type |
|
Override the item type for list objects. Default: |
Override Document Library File Item Type |
|
Override the item type for document library files. Default: |
Override Document Library Folder Item Type |
|
Override the item type for document library folders. Default: |
Override Generic List Item Item Type |
|
Override the item type for generic list items. Default: |
Override Generic List Folder Item Type |
|
Override the item type for generic list folders. Default: |
Override Attachment Item Type |
|
Override the item type for attachments. Default: |
Connection Settings
In order to adjust the connector on HTTP client level, the following parameters can be adjusted:
| Name | Key | Description |
|---|---|---|
Socket Timeout |
|
Duration after which inactive connections will be terminated. |
Connection Timeout |
|
Maximal duration a client should wait for an initial response from a server. |
Retry Count |
|
Maximal number of times a request should be retried on failure. |
Retry Delay |
|
Fix delay between request retries. |
Max Total Connections |
|
Maximum number of allowed connections overall. |
Max Connections Per Route |
|
Number of concurrent connections per route. |
Connection Time To Live (TTL) |
|
Duration after which unused connections will be discarded. |
Requests Per Second |
|
Maximum number of requests which are active in parallel. |
Rate Limit Requests Per Second |
|
Maximum number of requests which are active in parallel while the API rate is
near its limit. |
User Agent HTTP Header |
|
HTTP header which is sent with every request. It identifies the agent responsible for the request. |
Cache Settings
Configuration options determining values for the connector’s caches.
| Name | Key | Description |
|---|---|---|
Enable Cache Stat Logging |
|
When enabled, internal cache stats (cache hit / miss rate) are logged in a given interval. |
Stat Logging Interval |
|
The interval in which the cache stats should be logged. |
Site Templates
To restrict the number of site collections fetched by auto site discovery,
the connector provides the functionality to define a set of site templates
that should be considered or not. To obtain a list of site templates used in your
SharePoint Online tenant, you can run a PowerShell script provided by Raytion.
You can find it in the scripts folder of the connector.
-
Install the SharePoint Online PowerShell module. See the official documentation from Microsoft.
-
Open a PowerShell console as an administrator and navigate to the script’s folder location.
-
Run the PowerShell script:
.\SiteCollectionsToFile.ps1
-
If the Execution Policy denies execution, it can be bypassed by using the -ExecutionPolicy parameter with the Bypass option:
powershell -ExecutionPolicy Bypass -File .\SiteCollectionsToFile.ps1
-
Enter the URL to the Microsoft SharePoint Online admin center. Format:
https://<domain>-admin.sharepoint.com -
In the appearing prompt login with a tenant administrator account.
-
Two files will be generated: one containing all the site collection URLs, and another listing all the site templates found in the tenant.
CSM Configuration
Required Configuration Properties
Optional Configuration Properties
CSM Connection Settings
Configuration options for fine-tuning the Http connection parameters.
| Name | Description |
|---|---|
Concurrent Connections |
Maximum number of concurrent open connections. |
Requests Rate |
Maximum number of requests per second. |
Connect Timeout in Milliseconds |
Timeout of the connect request. |
Socket Timeout in Milliseconds |
Timeout of the socket connected to CSM. |
Request Timeout in Milliseconds |
Timeout of a request to CSM. |
Apache Solr Configuration
Instance Configuration
Configuration options related to specifying the target Solr Instance and Collection including authentication/authorization settings.
| Setting | Description |
|---|---|
Deployment |
Deployment of the target Solr instance. Use |
Collection Server URL |
For Standalone Collection Server, the URL to the collection server including the data path, e.g. |
Zookeeper Instances |
For Solr Cloud target, list of zookeeper instances including host and port. |
Zookeeper Chroot |
Znode Chroot of the cluster. |
Collection ID |
ID of the target collection. |
Use Authentication |
Enable this option to use Basic Authentication to authenticate against your Solr instance. |
Username |
Basic Authentication Username, if authentication is enabled. |
Password |
Basic Authentication Password, if authentication is enabled. |
Use Proxy |
If enabled, the connection to the Solr instance will be established through HTTP/HTTPS proxy. |
Proxy Endpoint |
Target proxy URL including protocol, host and port. |
Proxy Authentication |
If enabled, the connector uses the specified credentials to authenticate towards proxy. |
Proxy Username |
Proxy authentication username. |
Proxy Password |
Proxy authentication password. The value will be stored encrypted by the connector. |
ACL Settings
Configuration options influencing the document ACL creation.
| Setting | Description |
|---|---|
Domain Prefix |
Prefix applied to access control entries of document ACLs. In case a domain considering CSM (Query type: 'sharepoint') is used with the connector, this prefix needs to match the CSM domain concatenated with the CSM domain separator (':'). Otherwise, leave the prefix blank. |
Language Settings
Settings that define which metadata fields require a language specific processing. These fields will be suffixed with the language code, belonging to the document (example: 'content_en'). The language specific processing can be defined in the Apache Solr schema for fields with a dedicated language suffix.
| Setting | Description |
|---|---|
Standard Fields |
Select any arbitrary fields to append the language suffix.
The list consists of standardized fields provided by the connector.
The fields are: |
Include Additional Fields |
Enable this option to include also additional fields not listed in the standard field list, e.g. fields generated by the pipeline. |
Additional Fields |
List of field names to apply the suffix to. |
Fallback Language |
Fallback language code in ISO 639 format to apply for items with missing language information. |
| Adjustments to these configuration options, may require appropriate changes to the Apache Solr schema and a full content synchronization to index the altered metadata fields. |
Advanced ACL Settings (Optional)
Advanced options for handling document ACL.
| Setting | Description |
|---|---|
Everyone ACE |
Identifier of the access control entry which marks a document as public. |
Nobody ACE |
Identifier of the access control entry which indicate that a document is not accessible by anyone. |
Metadata Field Settings (Optional)
Define the names of Apache Solr specific fields.
| Setting | Description |
|---|---|
Allow Document ACL Field |
Metadata field name for the allow access control list. |
Deny Document ACL Field |
Metadata field name for the deny access control list. |
Content Field |
Metadata field name for the document’s content. |
Advanced HTTP Settings (Optional)
Configuration options for fine-tuning the Http connection parameters.
| Setting | Description |
|---|---|
Socket Timeout |
Timeout value for receiving data from server. |
Connection Timeout |
Timeout value for establishing a connection to server. |
Connection Request Timeout |
Timeout value for requesting a connection from connection manager. |
Max. Number of Connections |
Max. number of connections maintained by the connection manager. |
Max. Number Requests per Second |
Max. number of requests send to the server per second. |
Max. Number of Retries |
The maximum amount of times failed request shall be retried. Infinite retries are not supported. The delay between retries follows the pattern: [1s, 5s, 30s, 30s, …]. |
General Configuration
Database Configuration
| Name | Property Key | Description |
|---|---|---|
Configuration Type |
|
Supported are PostgreSQL, MS SQL Server, and JDBC URL configuration. |
PostgreSQL
| Name | Property Key | Description |
|---|---|---|
Host |
|
Domain name or IP address of the database server. |
Port |
|
Specifies the port number PostgreSQL is listening on, default is 5432. |
Database Name |
|
Name of the database. |
Username |
|
Username to authenticate with. The regarding user has to have read and write permissions to the database. |
Password |
|
Password of the configured database user. |
Add Custom Parameter |
|
Enables the configuration of additional parameters. |
MS SQL Server
| Name | Property Key | Description |
|---|---|---|
Host |
|
Domain name or IP address of the database server. Instance to connect to on server can be specified by '‹server_name>|<instance_name>'. |
Port |
|
Specifies the port number MS SQL Server is listening on, default is 1433. |
Database Name |
|
Name of the database. |
Username |
|
Username to authenticate with. The regarding user has to have read and write permissions to the database. |
Password |
|
Password of the configured database user. |
Add Custom Parameter |
|
Enables the configuration of additional parameters. |
JDBC URL
| Name | Property Key | Description |
|---|---|---|
URL |
|
JDBC URL for the target database. Out of the box, the connector will use H2 file database. For productive usage, use PostgreSQL specifying the URL in format: |
Username |
|
Database Username to read and write to database. |
Password |
|
Database Password for the specified user |
Traversal Configuration
| Name | Property Key | Description |
|---|---|---|
Traversal History Length |
|
Max. number of traversals to store in the history. Once the limit is exceeded, the connector will automatically remove oldest entries in the history. (default: 100) |
Include Checksum |
|
If enabled, any changes made to the pipeline e.g. configuration, the subsequent incremental run triggers a refeed of all items. |
Change Processing Interval |
|
Interval between change processing traversals. |
Resume on Start |
|
If enabled, any traversals in paused state are automatically resumed after the connector restart. Otherwise, the traversal remains in paused state. |
Number of Traversal Workers |
|
Number of workers to execute the traversal in parallel. Increasing this value might improve the performance, but will footprint higher memory consumption. It is recommended to keep the default value. (default: 10) |
Traversal Job Poll Interval |
|
Interval between the workers to be triggered to fetch and process the next tasks. (default: 10ms) |
Completion Timeout |
|
If the search engine indexes the items asynchronously, there might be some processing still in-flight during the completion process of a traversal. This value specifies the timeout value until all asynchronous callbacks are expected to return before completing the traversal. (default: 10m) |
Executor Size |
|
The executor size restricts the max. number of concurrent running traversals. |
Queue Size |
|
The queue size restricts the max. number of queued traversals. If the value is exceeded, the connector rejects further traversal requests until the queue size is below the configured size. |
Traversal Jobs
| Name | Property Key | Description |
|---|---|---|
Job Timeout Check Frequency |
|
Configures how often the connector checks for timed out jobs. |
Job Timeout |
|
The duration for which a job can stay idle before it is timed out. |
Job Cache Size |
|
Max. cache size of Jobs waiting for processing in memory. When cache is empty, next batch is fetched. |
Security Configuration
Request Restriction Settings
| Name | Property Key | Description |
|---|---|---|
Accepted Host Domains |
|
A list of domains (+ port) that are allowed as host names in the headers of HTTP requests
to the connector. This means that you can access the connector only via a URL that
employs one of the configured domains. Each entry must have the format
If no domains are configured (the default), then you can use any domain via which the connector host is reachable. |
Principal Aliaser Configuration
Principal Aliasing is applied on user information as part of Content ACL processing during Content Synchronization and Principal processing during Principal Synchronization. It’s purpose is to map external source system user to the corresponding user in search engines domain. You can configure a list of aliasers in the connector which will be applied in sequence and in order on user ACEs and user principals. The Connector supports following custom aliasing mechanism.
Custom Aliaser Disabled
If the Custom Aliaser checkbox is not selected, the connector will process user information on ACE and user principals unchanged to Search Engine. If all relevant users in the source system can be found with the same identifier in the search engine, this setup is sufficient to reflect the same secure search experience in the search engine as defined by the policy in the source system. The connector uses this option as default to process user information.
Custom Aliaser Enabled
If custom aliasing is enable then there are four types of aliaser avaialble:
Simple XML Table Aliaser
Static mapping table which can be uploaded as XML file. The connector uses the uploaded file as lookup table to map a user in the source system to a user in the search engine. Users missing a record in the file will be dropped from the ACE and during Principal Synchronization. This option is only recommended for environment with a manageable amount of users as for each user the corresponding mapping entry needs to be specified in the file.
| Name | Description |
|---|---|
XML Mapping File |
Browse and upload or drag and drop. |
Sample XML mapping file:
<?xml version="1.0" encoding="UTF-8"?>
<storeddata>
<entry keyValue="user1">user1@raytion.com</entry>
<entry keyValue="user2">user2@raytion.com</entry>
<entry keyValue="user3">user3@raytion.com</entry>
</storeddata>
Regex Replacer Aliaser
Regex Replacer Aliaser computes aliases based on a regular expression. Principals that match the regular expression are replaced by the Substitution String.
| Name | Property Key | Description |
|---|---|---|
Pattern |
|
The regular expression to match, this is the part that will be replaced. If braces (…) are used in the pattern then the matched value can be retrieved using $1 |
Substitute String |
|
String to replace the matching part of the find string. Matched value is accessed by employing $1 |
Regex Extractor Aliaser
Regex Extractor Aliaser computes aliases based on a regular expression. Principals that match the regular expression are inserted into the Insert-Into String.
| Name | PropertyKey | Description |
|---|---|---|
Pattern |
|
The regular expression to match, this is the part that will be inserted into the new value. If braces (…) are used in the pattern then the matched value can be retrieved using $$ |
Insert-Into String |
|
String to replace the matching part of the pattern. Matched value is accessed by employing $$ |
LDAP Aliaser
Ldap Aliaser searches for an LDAP entry with the requested name in the input value and returns the specified output attribute.
| Name | Property Key | Description |
|---|---|---|
Host |
|
Fully Qualified Domain Name of an LDAP server |
Port |
|
Port to use for LDAP connection, defaults are 389/636 or (recommended) 3268/3269 for simple/SSL |
AccountDN |
|
AccountDN for bind to LDAP |
Password |
|
Password part of credentials |
Input Field |
|
The Active Directory attribute name for this equality filter |
Search Root DN |
|
Distinguished Name of the subtree which is searched. The smaller the subtree the better the performance but the higher the chance of encountering principals which are not part of this subtree |
Output Field |
|
Attribute that should be returned in result entries |