The WorksAudit Book
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode
  1. Welcome to The WorksAudit Book
  2. /
  3. Architecture and Design
  4. /
  5. Producer
  6. /
  7. Hue Log Synchronization

Hue Log Synchronization

Hue Log Synchronization

Architecture

Currently SCM (HUE) is sending logs to WorksAudit using two methods running in parallel:

  1. HUE native raw log is copied by a lambda from HUE log bucket to WorksAudit central bucket.
  2. Data operations performed through RDBClient (UPDATE and DELETE) are intercepted by audit logger and sent to WorksAudit central bucket as Protobuf through Kinesis Firehose.

These two methods are shown in the image below:

HUE Producer

This diagram shows:

  1. HUE native raw log flow:
    1. HUE native is using the original (old) BT’s Audit Log Framework, where the logs are forwarded by fluentd to HUE log bucket.
    2. There is a lambda that is triggered by a new file being created in HUE log bucket (hue-worksaudit-lambda-log-sync). This lambda copies the log to WorksAudit central bucket (under /hue subfolder).
    3. Once the HUE native log is copied to WorksAudit central bucket, there is another lambda triggered (hue-worksaudit-lambda-log-gzcheck) to check whether the file is not broken. If this lambda detect that the file is broken, it will send a notification to an SNS topic.
  2. RDBClient Protobuf flow:
    1. All JDBC operations goes through RDBClient will be intercepted, all INSERT, UPDATE, and DELETE will be automatically logged using WorksAudit Protobuf logger for Java.
    2. Protobuf data is sent through Kinesis Firehose in HUE environment. A lambda (hue-worksaudit-lambda-firehose-batch-filter) is attached to the Firehose to filter data. The reason why this filter is necessary is that, originally almost 90% data coming in through the Firehose is produced by batches that are not the target of audit, and the RDBClient is logging SQLs indiscriminately. Currently the filtering process is controlled by a set of rules. This rules covers about 90% of cases of logs that should have been dropped in the first place.
    3. The filtered logs are sent to WorksAudit central bucket under /protobuf subfolder.
  3. The third case is an ideal case that is currently not implemented anywhere. Ideally all activities in HUE are logged using Protobuf based producer. This producer is integrated in HUE Audit Log Framework from BT 20.04.

Deployment

The deployment of all infrastructures on HUE side (including the lambda, Firehose, etc.) is included in environment-setup-script, specifically in this file and this file.

For the details on how to setup for the deployment, please see the HUE integration documentation.