Connector Configuration
|
In some scenarios, it can be useful to provide a system property or an environment
variable within a configuration value. For example, you might want to distribute multiple
connector instances over different containers and have certain configured parameters
adapted according to specific system properties or environment variables, respectively. You
can do this by providing a placeholder of the form However, for security reasons this replacement is disabled by default. A malicious user could misuse this feature to obtain sensitive information about the connector environment such as the host’s operating system, the user under which the connector is running etc. You can mitigate this issue e.g. by restricting the access to the connector UI. If you want to enable the resolution of system properties and environment variables,
set the system property |
SMB File Share Configuration
SMB File Share Connection Settings
Configuration Options related to establish connection to the target SMB File Share.
Share Connection
The connection details for a share. Multiple Shares can be configured.
| Name | Property Key | Description |
|---|---|---|
User Domain |
|
The domain of the user used to access the file share. |
Username |
|
The username used to access the file share. |
User Password |
|
The password of the user used to access the file share. |
Host Name |
|
The host that provides the file share. |
Share Name |
|
The name of the file share. |
Start Folders |
|
A list of folders to crawl. |
Enable Filters |
|
Enable Filter options for the SMB file share. |
Share Filter Settings
Multiple filters can be configured for a Share.
Folder Filter Settings
| Name | Property Key | Description |
|---|---|---|
Regular Expression |
|
The regular expression the path will be matched against. |
File Filter Settings
| Name | Property Key | Description |
|---|---|---|
Action |
|
When the Filter Rule matches, this action is performed. |
Regular Expression |
|
The regular expression the path will be matched against |
File Size Filter Settings
| Name | Property Key | Description |
|---|---|---|
Action |
|
When the Filter Rule matches, this action is performed. |
Rule |
|
The applied rule. |
File Size |
|
The applied File Size. |
Path Length Filter Settings
| Name | Property Key | Description |
|---|---|---|
Maximum Path Length |
|
Maximum path length allowed. |
Date Index Filter Settings
| Name | Property Key | Description |
|---|---|---|
Date Field |
|
Date field of the Item/Folder. |
Mode |
|
Choose if the filter will be applied on a period or on a specific date. |
Unit |
|
To calculate the relative date take the current date and go back N time. |
Quantity |
|
The amount of units which shall be used to calculate the cut-off date. |
Format |
|
A date format string.E.g.’yyyy-MM-dd’for year-month-day. |
Date |
|
A fixed date of the specified format. |
SMB File Share Global Index Settings
Index Options related to all configured SMB File Shares.
| Name | Property Key | Description |
|---|---|---|
Index Folders |
|
If Setting is enabled then the information of folders will be indexed too. |
Max Content Size |
|
Global Max Content Size Filter which will be applied across all Shares. If a Document exceeds this size, then only the Metadata will be indexed. A Greater-Than File Size Filter configured on a Share, will override this global content filter. |
SMB File Share Global Connection Settings
Configuration Options related to all configured SMB File Shares.
| Name | Property Key | Description |
|---|---|---|
Disable Security |
|
If you experience low download speed, changing the download buffer size may help (default: 1MB). |
Download Buffer Size |
|
If you have to decrease load on the file server you can configure a crawl throttle. Each crawled item will get delayed for the configured amount of milliseconds (default: 0 = unthrottled). |
Throttle |
|
If you have to decrease load on the file server you can configure a crawl throttle. Each crawled item will get delayed for the configured amount of milliseconds (default: 0 = unthrottled). |
Retry Pause |
|
The pause between retrying to connect to a file share after a failed connection. |
Enable DFS |
|
If set to true, the connector connects to both, DFS and non-DFS SMB fileshares. If you know that you don’t have DFS set up then setting this option to false generates less network traffic, but the connector won’t connect to a DFS file share. (default: true). |
Kerberos Realm |
|
To authenticate via Kerberos, the Kerberos realm needs to be configured. If you authenticate via NTLM you can leave this field blank. |
Kerberos KDC |
|
Please enter the KDC (Key Distribution Center) hostname or the Active Domain were the KDC can be looked up from. If you authenticate via NTLM you can leave this field blank. |
Auto close idle period |
|
The number of seconds a connection can be idle before being automatically closed. |
Auto close check period |
|
The number of seconds between checking for idle connections to closed. |
SMB File Share ACL Cache Settings
Configuration Options related the ACL cache.
| Name | Property Key | Description |
|---|---|---|
Maximum ACL Cache Size |
|
Maximum number of ACL entries that can be cached before the cache attempts to remove entries due to the cache size. |
Expiration Period |
|
The cache entry expiration period in seconds, the expiration period is calculated from the last time the cache entry was accessed. |
LDAP SID Well Known Groups Settings
Configuration options related to bypass LDAP lookup for groups that are not part of LDAP but are known to be groups.
SIDs which are known to be groups
Some SIDs that are used on the fileshares are not part of LDAP, e.g. local groups. These can be added here to prevent warnings during the traversal.
| Name | Property Key | Description |
|---|---|---|
sid |
|
A well known SID |
label |
|
A human readable debug label for the SID |
LDAP SID resolution
| Name | Property Key | Description |
|---|---|---|
Use LDAP for SID Resolution |
|
If enabled, the configured LDAP is used to resolve SID names. |
LDAP SID Resolution Connection Settings
| Name | Property Key | Description |
|---|---|---|
Address |
|
Fully Qualified Domain Name of an LDAP server |
Port |
|
Port to use for LDAP connection, defaults are 389/636 or (recommended) 3268/3269 for simple/SSL |
Username |
|
Username for SIMPLE bind to LDAP |
Password |
|
Password part of credentials |
Search Root DN |
|
Distinguished Name of the subtree which is searched. The smaller the subtree the better the performance but the higher the chance of encountering principals which are not part of this subtree |
LDAP SID Resolution Caching Settings
| Name | Property Key | Description |
|---|---|---|
Maximum number of cached results |
|
To prevent memory overflow the maximum size of the SID resolution cache is capped. |
Maximum age of a SID resolution result |
|
SIDs will be re-resolved after the configuration time has passed to capture changes in LDAP |
Cache unresolvable SIDs |
|
If true, unresolvable SIDs will be cached. |
LDAP SID Resolution Query Settings
| Name | Property Key | Description |
|---|---|---|
SID field name |
|
Name of the field which is searched for SIDs |
Type field name |
|
Name of the field which is used to determine the SID type |
User type value |
|
This type indicates a user SID |
User name field |
|
Name of the field which contains the output for user SIDs |
Field for group labels |
|
Name of the field which contains a label for group SIDs |
LDAP SID Resolution Retry Settings
| Name | Property Key | Description |
|---|---|---|
Interval before retry |
|
This determines how long the connector will wait before retrying after an error occurred. |
Fail on error |
|
If true, errors are propagated. If false, errors are converted to 'not found' |
Infinite retry |
|
If true, the last retry interval is reused until abortion or success. This can lead to connector starvation. |
Apache Solr Configuration
Instance Configuration
Configuration options related to specifying the target Solr Instance and Collection including authentication/authorization settings.
| Setting | Description |
|---|---|
Deployment |
Deployment of the target Solr instance. Use |
Collection Server URL |
For Standalone Collection Server, the URL to the collection server including the data path, e.g. |
Zookeeper Instances |
For Solr Cloud target, list of zookeeper instances including host and port. |
Zookeeper Chroot |
Znode Chroot of the cluster. |
Collection ID |
ID of the target collection. |
Use Authentication |
Enable this option to use Basic Authentication to authenticate against your Solr instance. |
Username |
Basic Authentication Username, if authentication is enabled. |
Password |
Basic Authentication Password, if authentication is enabled. |
Use Proxy |
If enabled, the connection to the Solr instance will be established through HTTP/HTTPS proxy. |
Proxy Endpoint |
Target proxy URL including protocol, host and port. |
Proxy Authentication |
If enabled, the connector uses the specified credentials to authenticate towards proxy. |
Proxy Username |
Proxy authentication username. |
Proxy Password |
Proxy authentication password. The value will be stored encrypted by the connector. |
ACL Settings
Configuration options influencing the document ACL creation.
| Setting | Description |
|---|---|
Domain Prefix |
Prefix applied to access control entries of document ACLs. In case a domain considering CSM (Query type: 'sharepoint') is used with the connector, this prefix needs to match the CSM domain concatenated with the CSM domain separator (':'). Otherwise, leave the prefix blank. |
Language Settings
Settings that define which metadata fields require a language specific processing. These fields will be suffixed with the language code, belonging to the document (example: 'content_en'). The language specific processing can be defined in the Apache Solr schema for fields with a dedicated language suffix.
| Setting | Description |
|---|---|
Standard Fields |
Select any arbitrary fields to append the language suffix.
The list consists of standardized fields provided by the connector.
The fields are: |
Include Additional Fields |
Enable this option to include also additional fields not listed in the standard field list, e.g. fields generated by the pipeline. |
Additional Fields |
List of field names to apply the suffix to. |
Fallback Language |
Fallback language code in ISO 639 format to apply for items with missing language information. |
| Adjustments to these configuration options, may require appropriate changes to the Apache Solr schema and a full content synchronization to index the altered metadata fields. |
Advanced ACL Settings (Optional)
Advanced options for handling document ACL.
| Setting | Description |
|---|---|
Everyone ACE |
Identifier of the access control entry which marks a document as public. |
Nobody ACE |
Identifier of the access control entry which indicate that a document is not accessible by anyone. |
Metadata Field Settings (Optional)
Define the names of Apache Solr specific fields.
| Setting | Description |
|---|---|
Allow Document ACL Field |
Metadata field name for the allow access control list. |
Deny Document ACL Field |
Metadata field name for the deny access control list. |
Content Field |
Metadata field name for the document’s content. |
Advanced HTTP Settings (Optional)
Configuration options for fine-tuning the Http connection parameters.
| Setting | Description |
|---|---|
Socket Timeout |
Timeout value for receiving data from server. |
Connection Timeout |
Timeout value for establishing a connection to server. |
Connection Request Timeout |
Timeout value for requesting a connection from connection manager. |
Max. Number of Connections |
Max. number of connections maintained by the connection manager. |
Max. Number Requests per Second |
Max. number of requests send to the server per second. |
Max. Number of Retries |
The maximum amount of times failed request shall be retried. Infinite retries are not supported. The delay between retries follows the pattern: [1s, 5s, 30s, 30s, …]. |
General Configuration
Database Configuration
| Name | Property Key | Description |
|---|---|---|
Configuration Type |
|
Supported are PostgreSQL, MS SQL Server, and JDBC URL configuration. |
PostgreSQL
| Name | Property Key | Description |
|---|---|---|
Host |
|
Domain name or IP address of the database server. |
Port |
|
Specifies the port number PostgreSQL is listening on, default is 5432. |
Database Name |
|
Name of the database. |
Username |
|
Username to authenticate with. The regarding user has to have read and write permissions to the database. |
Password |
|
Password of the configured database user. |
Add Custom Parameter |
|
Enables the configuration of additional parameters. |
MS SQL Server
| Name | Property Key | Description |
|---|---|---|
Host |
|
Domain name or IP address of the database server. Instance to connect to on server can be specified by '‹server_name>|<instance_name>'. |
Port |
|
Specifies the port number MS SQL Server is listening on, default is 1433. |
Database Name |
|
Name of the database. |
Username |
|
Username to authenticate with. The regarding user has to have read and write permissions to the database. |
Password |
|
Password of the configured database user. |
Add Custom Parameter |
|
Enables the configuration of additional parameters. |
JDBC URL
| Name | Property Key | Description |
|---|---|---|
URL |
|
JDBC URL for the target database. Out of the box, the connector will use H2 file database. For productive usage, use PostgreSQL specifying the URL in format: |
Username |
|
Database Username to read and write to database. |
Password |
|
Database Password for the specified user |
Traversal Configuration
| Name | Property Key | Description |
|---|---|---|
Traversal History Length |
|
Max. number of traversals to store in the history. Once the limit is exceeded, the connector will automatically remove oldest entries in the history. (default: 100) |
Include Checksum |
|
If enabled, any changes made to the pipeline e.g. configuration, the subsequent incremental run triggers a refeed of all items. |
Change Processing Interval |
|
Interval between change processing traversals. |
Resume on Start |
|
If enabled, any traversals in paused state are automatically resumed after the connector restart. Otherwise, the traversal remains in paused state. |
Number of Traversal Workers |
|
Number of workers to execute the traversal in parallel. Increasing this value might improve the performance, but will footprint higher memory consumption. It is recommended to keep the default value. (default: 10) |
Traversal Job Poll Interval |
|
Interval between the workers to be triggered to fetch and process the next tasks. (default: 10ms) |
Completion Timeout |
|
If the search engine indexes the items asynchronously, there might be some processing still in-flight during the completion process of a traversal. This value specifies the timeout value until all asynchronous callbacks are expected to return before completing the traversal. (default: 10m) |
Executor Size |
|
The executor size restricts the max. number of concurrent running traversals. |
Queue Size |
|
The queue size restricts the max. number of queued traversals. If the value is exceeded, the connector rejects further traversal requests until the queue size is below the configured size. |
Traversal Jobs
| Name | Property Key | Description |
|---|---|---|
Job Timeout Check Frequency |
|
Configures how often the connector checks for timed out jobs. |
Job Timeout |
|
The duration for which a job can stay idle before it is timed out. |
Job Cache Size |
|
Max. cache size of Jobs waiting for processing in memory. When cache is empty, next batch is fetched. |
Security Configuration
Request Restriction Settings
| Name | Property Key | Description |
|---|---|---|
Accepted Host Names |
|
A list of domains (+ port) that are allowed as host names in the headers of HTTP requests
to the connector. This means that you can access the connector only via a URL that
employs one of the configured domains. Each entry must have the format
If no domains are configured (the default), then you can use any domain via which the connector host is reachable. |
Principal Aliaser Configuration
Principal Aliasing is applied on user information as part of Content ACL processing during Content Synchronization and Principal processing during Principal Synchronization. It’s purpose is to map external source system user to the corresponding user in search engines domain. You can configure a list of aliasers in the connector which will be applied in sequence and in order on user ACEs and user principals. The Connector supports following custom aliasing mechanism.
Custom Aliaser Disabled
If the Custom Aliaser checkbox is not selected, the connector will process user information on ACE and user principals unchanged to Search Engine. If all relevant users in the source system can be found with the same identifier in the search engine, this setup is sufficient to reflect the same secure search experience in the search engine as defined by the policy in the source system. The connector uses this option as default to process user information.
Custom Aliaser Enabled
If custom aliasing is enable then there are four types of aliaser avaialble:
Simple XML Table Aliaser
Static mapping table which can be uploaded as XML file. The connector uses the uploaded file as lookup table to map a user in the source system to a user in the search engine. Users missing a record in the file will be dropped from the ACE and during Principal Synchronization. This option is only recommended for environment with a manageable amount of users as for each user the corresponding mapping entry needs to be specified in the file.
| Name | Description |
|---|---|
XML Mapping File |
Browse and upload or drag and drop. |
Sample XML mapping file:
<?xml version="1.0" encoding="UTF-8"?>
<storeddata>
<entry keyValue="user1">user1@raytion.com</entry>
<entry keyValue="user2">user2@raytion.com</entry>
<entry keyValue="user3">user3@raytion.com</entry>
</storeddata>
Regex Replacer Aliaser
Regex Replacer Aliaser computes aliases based on a regular expression. Principals that match the regular expression are replaced by the Substitution String.
| Name | Property Key | Description |
|---|---|---|
Pattern |
|
The regular expression to match, this is the part that will be replaced. If braces (…) are used in the pattern then the matched value can be retrieved using $1 |
Substitute String |
|
String to replace the matching part of the find string. Matched value is accessed by employing $1 |
Regex Extractor Aliaser
Regex Extractor Aliaser computes aliases based on a regular expression. Principals that match the regular expression are inserted into the Insert-Into String.
| Name | PropertyKey | Description |
|---|---|---|
Pattern |
|
The regular expression to match, this is the part that will be inserted into the new value. If braces (…) are used in the pattern then the matched value can be retrieved using $$ |
Insert-Into String |
|
String to replace the matching part of the pattern. Matched value is accessed by employing $$ |
LDAP Aliaser
Ldap Aliaser searches for an LDAP entry with the requested name in the input value and returns the specified output attribute.
| Name | Property Key | Description |
|---|---|---|
Host |
|
Fully Qualified Domain Name of an LDAP server |
Port |
|
Port to use for LDAP connection, defaults are 389/636 or (recommended) 3268/3269 for simple/SSL |
AccountDN |
|
AccountDN for bind to LDAP |
Password |
|
Password part of credentials |
Input Field |
|
The Active Directory attribute name for this equality filter |
Search Root DN |
|
Distinguished Name of the subtree which is searched. The smaller the subtree the better the performance but the higher the chance of encountering principals which are not part of this subtree |
Output Field |
|
Attribute that should be returned in result entries |