Logging and Auditing

Convier produces two different types of logs: Service logs and Audit logs.

Service logs

The service logs contains a mix of access log and other internal service related log entries. It is produced using log4j. An excerpt from an access log:

2024-12-18T07:29:11: [INFO]  INFO - 123.123.123.123 - [email protected] [18/Dec/2024:07:29:11 +0000] "PUT /convier/api/read/aml_platform_develop/main/count HTTP/1.1" 200 53 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0)"
2024-12-18T07:29:11: [INFO]  INFO - Post-filtering 157 found links
2024-12-18T07:29:11: [INFO]  INFO - 123.123.123.123 - [email protected] [18/Dec/2024:07:29:11 +0000] "POST /convier/api/async-read/aml_platform_develop/main/request HTTP/1.1" 200 18406 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0)"

The access log format may be modified using the LOG_FORMAT environment setting. See the Undertow documentation for more information.

Audit logs

The audit log is written to Azure Log Service or a SQL database of your choice, but must be enabled. To remove the possibility of removing traces of misuse, it is recommended that users of Convier do not have access to this database themselves.

Write to SQL database

Use the configuration settings:

INTERNAL_AUDIT_LOG_JDBC_CONNECTION_STRING (mandatory, convier will not start if not set)
INTERNAL_AUDIT_LOG_JDBC_USERNAME
INTERNAL_AUDIT_LOG_JDBC_PASSWORD

Tables and fields in the audit logging database is automatically created.

Write to Azure Log Service

With Log Ingestion API

Available from 0.2511.35
From Azure portal, go to Monitor
Settings -> Data Collection Endpoint -> Create
Copy Logs Ingestion URL (https://$endpointName.$region.ingest.monitor.azure.com)
Create an Azure Log Service workspace
Tables -> Create -> “New custom log (Direct Ingest)”
- Table name: convier_audit_log
- Create data collection rule
Sample schema { "oidc_client_id":"guid", "BRANCH":"main", "USER_ID":"[email protected]", "PAYLOAD":"{}", "TOPIC":"SEARCH", "RESPONSE":"{\\"status\\":\\"OK\\"}", "PROJECT_ID":"prod" }
Under the data collection rule
- Copy “Immutable ID” (dcr-...)
- Under “data sources”, copy data source name (tableName) (Custon-$name_CL)
Full URL: https://$endpointName.$region.ingest.monitor.azure.com/dataCollectionRules/$dcrImmutableId/streams/$tableName?api-version=2023-01-01
Set Convier environement variable AZURE_AUDIT_LOG_WORKSPACE_URL to the full URL
Enable System Assigned Managed Identity on the App Service, and assign role Monitoring Metrics Publisher on the Data Collection Rule

Shared key (From 0.2507.1)

Deprecated: Data Collector API is being retired by Microsoft Q3 2026 (https://learn.microsoft.com/en-us/azure/azure-monitor/logs/custom-logs-migrate)
Create an Azure Log Service workspace
Assign role Log Analytics Reader to users who should be able to query to audit log
Copy Workspace ID and set as environment variable AZURE_AUDIT_LOG_WORKSPACE_ID in Convier
If necessary, set AZURE_AUDIT_LOG_WORKSPACE_URL. It defaults to https://{workspaceId}.ods.opinsights.azure.com/api/logs?api-version=2016-04-01
Back in the Azure landing page of the created service, go to Agents, then Linuxand copy the Primary Key value. Set as environment variable AZURE_AUDIT_LOG_SHARED_KEY in Convier
By default, a table named CONVIER_AUDIT_LOG_CL is created automatically. The name can be overridden by setting environment variables AZURE_AUDIT_LOG_TABLE_NAME (_CL is added automatically). Make sure to adjust retention settings per your needs (Default is 30 days)
Note that it may take a few minutes for a log entry to appear

Google log sink

Give identity access to write logs in project
Set environment variables GOOGLE_PROJECT_ID and GOOGLE_LOG_SINK_ID and restart service
- Set environment variable GOOGLE_LOG_SINK_ID_PROJECT_ID if a separate project is used for logging

Audit log contents

An audit log entry contains the following fields:

UTC Timestamp
OIDC client ID (Azure Log Service only)
User ID
Project ID
Project Branch Name
Topic
Request payload (JSON string)
Response status (JSON string)

We currently do not include raw response playloads in the audit log, as they may be quite large. Audit log entries are immediately written to the database, there are not written in batches.

These are the topics being logged

connect_to_source
search
browse_data
load_object
get_linked_objects
count
compute_analytics
load_report_data
report_save
report_other_action

An example log entry on the search topic:

field	value
project_id	`aml`
branch	`main`
timestamp	`2024-12-10T10:00:00Z`
user_id	`[email protected]`
topic	`search`
payload	`{"query":"test","fields":["CUSTOMER_NAME"],"sourceIds":["dwh"]}`
response	`{"status": "OK", "primary_related_object_ids": []}`

The field primary_related_object_ids contains references to loaded objects if any. It is as of version 0.2601.7 populated for topics

load_object: loaded objects
load_report_data: the report object and all “connected objects” (e.g. customer associated with a task)

It is not populated for topics search and compute_analytics as they are covered by previous or subsequent load_object audit entries.

Interpreting the logs

To find user actions in the logs that correspond to opening a data view on a particular object:

| --- | --- | --- |