Firehose Setup

Follow these steps to setup connection to Audit from your environment using AWS CLI:

1. Export necessary environment variables AWS_ACCOUNT_ID (AWS 12 digits account ID) and AWS_REGION_ID (region code), e.g.:

   $ export AWS_ACCOUNT_ID=111122223333
   $ export AWS_REGION_ID=ap-northeast-1
   

2. Create IAM role for the Kinesis Firehose to be able to assume roles:

   $ aws iam create-role --role-name wap-audit-pipeline-firehose-role --assume-role-policy-document file://wap-audit-pipeline-firehose-assume-role-policy.json
   

   You will need a role policy file that should look like the following:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Service": "firehose.amazonaws.com"
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }
   

3. Use following JSON and save it to a file:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "WapAuditLogCentralDev",
               "Effect": "Allow",
               "Action": [
                   "s3:PutObject",
                   "s3:GetObject",
                   "s3:ListBucketMultipartUploads",
                   "s3:AbortMultipartUpload",
                   "s3:PutBucketLogging",
                   "s3:PutObjectVersionAcl",
                   "s3:PutBucketAcl",
                   "s3:PutBucketPolicy",
                   "s3:ListBucket",
                   "s3:GetBucketLocation",
                   "s3:PutObjectAcl"
               ],
               "Resource": [
                   "arn:aws:s3:::wap-audit-central-dev",
                   "arn:aws:s3:::wap-audit-central-dev/*"
               ]
           }
       ]
   }
   

   You will need to edit following lines:

   "arn:aws:s3:::wap-audit-central-dev",
   "arn:aws:s3:::wap-audit-central-dev/*"
   

   Keep the line as is if you want to use dev environment. Change the -dev part to one of the following values if you want to use other environments:

   1. -eva for evaluation environment.
   2. -stg for staging environment.
   3. -pro for production (commercial) environment.

4. Add policy for the Kinesis Firehose so that it is allowed to access Audit’s S3 bucket. Please use the policy file edited in the previous step:

   $ aws iam put-role-policy --role-name wap-audit-pipeline-firehose-role --policy-name s3-access-policy --policy-document file://wap-audit-pipeline-firehose-s3-access-policy.json
   

5. Create the Kinesis Data Firehose resource itself. Please replace all -dev suffix in the wap-audit-pipeline-dev and arn:aws:s3:::wap-audit-central-dev part with any appropriate environment suffix that you wish to use:

   $ aws firehose create-delivery-stream --delivery-stream-name wap-audit-pipeline-dev --delivery-stream-type DirectPut --extended-s3-destination-configuration "{\"RoleARN\": \"arn:aws:iam::${AWS_ACCOUNT_ID}:role/wap-audit-pipeline-firehose-role\", \"BucketARN\": \"arn:aws:s3:::wap-audit-central-dev\", \"Prefix\": \"protobuf/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/${AWS_ACCOUNT_ID}-${AWS_REGION_ID}-\", \"ErrorOutputPrefix\": \"${AWS_ACCOUNT_ID}-${AWS_REGION_ID}-unknown/\", \"CompressionFormat\": \"GZIP\"}" --region ${AWS_REGION_ID}
   

   wap-audit-pipeline-dev in above example is just a name. It doesn’t have to be that particular name, but it is a name that will be used when you’re initializing the SDK.

Follow these steps to setup connection to Audit from your environment using CloudFormation:

1. Save following CloudFormation template to a file:

   ---
   AWSTemplateFormatVersion: "2010-09-09"
   Description: A sample template
   Parameters:
     Stage:
       Type: String
       Default: dev
   Resources:
     wapAuditPipelineFirehose:
       DependsOn: wapAuditPipelineFirehoseRole
       Type: AWS::KinesisFirehose::DeliveryStream
       Properties:
         DeliveryStreamName: !Sub "wap-audit-pipeline-${Stage}"
         ExtendedS3DestinationConfiguration:
           BucketARN: !Sub "arn:aws:s3:::wap-audit-central-${Stage}"
           RoleARN:
             Fn::GetAtt: [wapAuditPipelineFirehoseRole, Arn]
           Prefix:
             Fn::Join:
               - ""
               - - "protobuf/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/"
                 - Ref: AWS::AccountId
                 - "-"
                 - Ref: AWS::Region
                 - "-"
           ErrorOutputPrefix:
             Fn::Join:
               - ""
               - - Ref: AWS::AccountId
                 - "-"
                 - Ref: AWS::Region
                 - "-unknown/"
           CompressionFormat: GZIP
           BufferingHints:
             IntervalInSeconds: 300
             SizeInMBs: 5
   
           # You can add following configurations as you want:
           # CloudWatchLoggingOptions
     wapAuditPipelineFirehoseRole:
       Type: AWS::IAM::Role
       Properties:
         RoleName: !Sub "wap-audit-pipeline-firehose-role-${Stage}"
         AssumeRolePolicyDocument:
           Version: "2012-10-17"
           Statement:
             Effect: Allow
             Principal:
               Service:
                 - firehose.amazonaws.com
             Action: sts:AssumeRole
         Policies:
           - PolicyName: s3-access-inline-policy
             PolicyDocument:
               Version: "2012-10-17"
               Statement:
                 - Effect: Allow
                   Action:
                     - s3:PutObject
                     - s3:GetObject
                     - s3:ListBucketMultipartUploads
                     - s3:AbortMultipartUpload
                     - s3:PutbucketLogging
                     - s3:PutObjectVersionAcl
                     - s3:PutBucketAcl
                     - s3:PutBucketPolicy
                     - s3:ListBucket
                     - s3:GetBucketLocation
                     - s3:PutObjectAcl
                   Resource:
                     - !Sub "arn:aws:s3:::wap-audit-central-${Stage}"
                     - !Sub "arn:aws:s3:::wap-audit-central-${Stage}/*"
   
   

2. Please specify AUDIT_ENV_STAGE as one of the following values in next step’s command:

   1. dev for development environment (by default).
   2. eva for evaluation environment.
   3. stg for staging environment.
   4. pro for production (commercial) environment.

3. Execute following command:

   $ STACK_NAME=*your-favorite-name*
   $ REGION=*stack deploy region(e.g. ap-northeast-1)*
   $ AUDIT_ENV_STAGE=*audit env stage*
   $ aws cloudformation deploy --template-file ./cloudformation.yml --stack-name "${STACK_NAME}" --capabilities CAPABILITY_NAMED_IAM --parameter-overrides Stage="${AUDIT_ENV_STAGE}" --region "${REGION}"