Updated for version: 0.2601.1

Logging and Auditing

Convier produces two different types of logs: Service logs and Audit logs.

Service logs

The service logs contains a mix of access log and other internal service related log entries. It is produced using log4j. An excerpt from an access log:

2024-12-18T07:29:11: [INFO]  INFO - 123.123.123.123 - user@acme.com [18/Dec/2024:07:29:11 +0000] "PUT /convier/api/read/aml_platform_develop/main/count HTTP/1.1" 200 53 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0)"
2024-12-18T07:29:11: [INFO]  INFO - Post-filtering 157 found links
2024-12-18T07:29:11: [INFO]  INFO - 123.123.123.123 - user@acme.com [18/Dec/2024:07:29:11 +0000] "POST /convier/api/async-read/aml_platform_develop/main/request HTTP/1.1" 200 18406 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0)"

The access log format may be modified using the LOG_FORMAT environment setting. See the Undertow documentation for more information.

Audit logs

The audit log is written to Azure Log Service or a SQL database of your choice, but must be enabled. To remove the possibility of removing traces of misuse, it is recommended that users of Convier do not have access to this database themselves.

Write to SQL database

Use the configuration settings:

  • INTERNAL_AUDIT_LOG_JDBC_CONNECTION_STRING (mandatory, convier will not start if not set)
  • INTERNAL_AUDIT_LOG_JDBC_USERNAME
  • INTERNAL_AUDIT_LOG_JDBC_PASSWORD

Tables and fields in the audit logging database is automatically created.

Write to Azure Log Service

With Data Collection Rule (From 0.2601.1)

  • From Azure portal, go to Monitor
  • Settings -> Data Collection Endpoint -> Create
  • Copy Logs Ingestion URL (https://$endpointName.$region.ingest.monitor.azure.com)
  • Create an Azure Log Service workspace
  • Tables -> Create -> “New custom log (Direct Ingest)”
    • Table name: convier_audit_log
    • Create data collection rule
      • Sample schema { "oidc_client_id":"guid", "BRANCH":"main", "USER_ID":"user@example.com", "PAYLOAD":"{}", "TOPIC":"SEARCH", "RESPONSE":"{\"status\":\"OK\"}", "PROJECT_ID":"prod" }
  • Under the data collection rule
    • Copy “Immutable ID” (dcr-...)
    • Under “data sources”, copy data source name (tableName) (Custon-$name_CL)
  • Full URL: https://$endpointName.$region.ingest.monitor.azure.com/dataCollectionRules/$dcrImmutableId/streams/$tableName?api-version=2023-01-01
  • Set Convier environement variable AZURE_AUDIT_LOG_WORKSPACE_URL to the full URL
  • Enable System Assigned Managed Identity on the App Service, and assign role Monitoring Metrics Publisher on the Data Collection Rule

Shared key (From 0.2507.1)

  • Deprecated: Data Collector API is being retired by Microsoft Q3 2026 (https://learn.microsoft.com/en-us/azure/azure-monitor/logs/custom-logs-migrate)

  • Create an Azure Log Service workspace
  • Assign role Log Analytics Reader to users who should be able to query to audit log
  • Copy Workspace ID and set as environment variable AZURE_AUDIT_LOG_WORKSPACE_ID in Convier
  • If necessary, set AZURE_AUDIT_LOG_WORKSPACE_URL. It defaults to https://{workspaceId}.ods.opinsights.azure.com/api/logs?api-version=2016-04-01
  • Back in the Azure landing page of the created service, go to Agents, then Linuxand copy the Primary Key value. Set as environment variable AZURE_AUDIT_LOG_SHARED_KEY in Convier
  • By default, a table named CONVIER_AUDIT_LOG_CL is created automatically. The name can be overridden by setting environment variables AZURE_AUDIT_LOG_TABLE_NAME (_CL is added automatically). Make sure to adjust retention settings per your needs (Default is 30 days)
  • Note that it may take a few minutes for a log entry to appear

Audit log contents

An audit log entry contains the following fields:

  • UTC Timestamp
  • OIDC client ID (Azure Log Service only)
  • User ID
  • Project ID
  • Project Branch Name
  • Topic
  • Request payload (JSON string)
  • Response status (JSON string)

We currently do not include raw response playloads in the audit log, as they may be quite large. Audit log entries are immediately written to the database, there are not written in batches.

These are the topics being logged

  • connect_to_source
  • search
  • browse_data
  • load_object
  • get_linked_objects
  • count
  • compute_analytics
  • load_report_data
  • report_save
  • report_other_action

An example log entry on the search topic:

field value
project_id aml
branch main
timestamp 2024-12-10T10:00:00Z
user_id user@acme.com
topic search
payload {"query":"test","fields":["CUSTOMER_NAME"],"sourceIds":["dwh"]}
response {"status": "OK"}