Ingest Audit Logs from AWS Cloud Trail

Take advantage of Cortex XDR investigation capabilities and set up audit log ingestion for your AWS CloudTrail logs.
Ingesting logs and data requires a Cortex XDR Pro per TB license.
You can forward audit logs for the relative service to Cortex XDR from AWS CloudTrail.
To receive audit logs from Amazon Simple Storage Service (Amazon S3) via AWS CloudTrail, you must first configure data collection from Amazon S3. You can then configure the Collection Integrations settings in Cortex XDR for Amazon S3. After you set up collection integration, Cortex XDR begins receiving new logs and data from the source.
For more information on configuring data collection from Amazon S3 using AWS CloudTrail, see the AWS CloudTrail Documentation.
As soon as Cortex XDR begins receiving logs, the app automatically creates an Amazon S3 XQL dataset (
aws_s3_raw
). This enables you to search the logs with XQL Search using the dataset. For example queries, refer to the in-app XQL Library. You can also configure Cortex XDR to stitch Amazon S3 audit logs with other Cortex XDR authentication stories across all cloud providers using the same format, which you can query with XQL Search using the
cloud_audit_logs
dataset. Cortex XDR can also raise Cortex XDR alerts (IOC, BIOC, and Correlation Rule only) when relevant from Amazon S3 logs.
Be sure you do the following tasks before you begin configuring data collection from Amazon S3 via AWS CloudTrail.
  • Ensure that you have the proper permissions to access AWS CloudTrail and have the necessary permissions to create audit logs. You need at a minimum the following permissions in AWS for an Amazon S3 bucket and Amazon Simple Queue Service (SQS):
    • Amazon S3 bucket
      GetObject
    • SQS
      ChangeMessageVisibility
      ,
      ReceiveMessage
      , and
      DeleteMessage
      .
  • Determine how you want to provide access to Cortex XDR to your logs and to perform API operations. You have the following options:
Configure Cortex XDR to receive audit logs from Amazon S3 via AWS Cloudtrail.
  1. Log in to the AWS Management Console.
  2. From the menu bar, ensure that you have selected the correct region for your configuration.
  3. Configure an AWS CloudTrail trail with audit logs.
    For more information on creating an AWS CloudTrail trail, see Create a trail.
    If you already have an Amazon S3 bucket configured with AWS CloudTrail audit logs, skip this step and go to Configure an Amazon Simple Queue Service (SQS).
    1. Open the CloudTrail Console, and click
      Create trail
      .
    2. Configure the following settings for your CloudTrail trail, where the default settings should be configured unless otherwise indicated.
      • Trail name
        —Specify a descriptive name for your CloudT rail trail.
      • Storage location
        —Select
        Create new S3 bucket
        to configure a new Amazon S3 bucket, and specify a unique name in the
        Trail log bucket and folder
        field, or select
        Use existing S3 bucket
        and
        Browse
        to the S3 bucket you already created. If you select an existing Amazon S3 bucket, the bucket policy must grant CloudTrail permission to write to it. For information about manually editing the bucket policy, see Amazon S3 Bucket Policy for CloudTrail.
        It is the customer’s responsibility to define a retention policy for your Amazon S3 bucket by creating a
        Lifecycle rule
        in the
        Management
        tab. We recommend setting the retention policy to at least 7 days to ensure that the data is retrieved under all circumstances.
      • Customer managed AWS KMS key
        —You can either select a
        New
        key and specify the
        AWS KMS alias
        , or select an
        Existing
        key, and select the
        AWS KMS alias
        . The KMS key and S3 bucket must be in the same region.
      • SNS notification delivery
        —(
        Optional
        ) If you want to be notified whenever CloudTrail publishes a new log to your Amazon S3 bucket, click
        Enabled
        . Amazon Simple Notification Service (Amazon SNS) manages these notifications, which are sent for every log file delivery to your S3 bucket, as opposed to every event. When you enable this option, you can either
        Create a new SNS topic
        by selecting
        New
        and the
        SNS topic
        is displayed in the field, or use an
        Existing
        topic and select the
        SNS topic
        . For more information, see Configure SNS Notifications for CloudTrail.
      The
      CloudWatch Logs - optional
      settings are not supported and should be left disabled.
    3. Click
      Next
      , and configure the following
      Choose log events
      settings.
      • Event type
        —Leave the default
        Management events
        checkbox selected to capture audit logs. Depending on your system requirements, you can also select
        Data events
        to log the resource operations performed on or within a resource, or
        Insights events
        to identify unusual activity, errors, or user behavior in your account. Based on your selection, additional fields are displayed on the screen to configure under section headings with the same name as the event type.
      • Management events
        section—Configure the following settings:
        -
        API activity
        —For
        Management events
        , select the API activities you want to log. By default, the
        Read
        and
        Write
        activities are logged.
        -
        Exclude AWS KMS events
        —(
        Optional
        ) If you want to filter AWS Key Management Service (AWS KMS) events out of your trail, select the checkbox. By default, all AWS KMS events are included.
      • Data events
        section—(
        Optional
        ) This section is displayed when you configure the
        Event type
        to include
        Data events
        , which relate to resource operations performed on or within a resource, such as reading and writing to a S3 bucket.. For more information on configuring these optional settings in AWS CloudTrail, see Creating a trail.
      • Insights events
        section—(
        Optional
        ) This section is displayed when you configure the
        Event type
        to include
        Insight events
        , which relate to unusual activities, errors, or user behavior on your account. For more information on configuring these optional settings in AWS CloudTrail, see Creating a trail.
    4. Click
      Next
      .
    5. In the
      Review and create
      page, look over the trail configurations settings that you have configured and if they are correct, click
      Create trail
      . If you need to make a change, click
      Edit
      beside the particular step that you want to update.
      The new trail is listed in the
      Trails
      page, which lists the trails in your account from all Regions. It can take up to 15 minutes for CloudTrail to begin publishing log files. You can see the log files in the S3 bucket that you specified. For more information, see Creating a trail.
  4. Configure an Amazon Simple Queue Service (SQS).
    Ensure that you create your Amazon S3 bucket and Amazon SQS queue in the same region.
    1. In the Amazon SQS Console, click
      Create Queue
      .
    2. Configure the following settings, where the default settings should be configured unless otherwise indicated.
      • Type
        —Select
        Standard
        queue (default).
      • Name
        —Specify a descriptive name for your SQS queue.
      • Configuration
        section—Leave the default settings for the various fields.
      • Access policy
        Choose method
        —Select
        Advanced
        and update the Access policy code in the editor window to enable your Amazon S3 bucket to publish event notification messages to your SQS queue. Use this sample code as a guide for defining the
        “Statement”
        with the following definitions:
        -
        “Resource”
        —Leave the automatically generated ARN for the SQS queue that is set in the code, which uses the format
        “arn:sns:Region:account-id:topic-name”
        .
        -
        “Resource”
        —Leave the automatically generated ARN for the SQS queue that is set in the code, which uses the format
        “arn:sns:Region:account-id:topic-name”
        .
        You can retrieve your bucket’s ARN by opening the Amazon S3 Console in a browser window. In the
        Buckets
        section, select the bucket that you created for collecting the Amazon S3 flow logs, click
        Copy ARN
        , and paste the ARN in the field.
        For more information on granting permissions to publish messages to an SQS queue, see Granting permissions to publish event notification messages to a destination.
        { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "SQS:SendMessage", "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]", "Condition": { "ArnLike": { "aws:SourceArn": "[ARN of your Amazon S3 bucket]" } } }, ] }
      • Dead-letter queue
        section—We recommend that you configure a queue for sending undeliverable messages by selecting
        Enabled
        , and then in the
        Choose queue
        field selecting the queue to send the messages. You may need to create a new queue for this, if you do not already have one set up. For more information, see Amazon SQS dead-letter queues.
    3. Click
      Create queue
      .
      Once the SQS is created, a message indicating that the queue was successfully configured is displayed at the top of the page.
  5. Configure an event notification to your Amazon SQS whenever a file is written to your Amazon S3 bucket.
    1. Open the Amazon S3 Console and in the
      Properties
      tab of your Amazon S3 bucket, scroll down to the
      Event notifications
      section, and click
      Create event notification
      .
    2. Configure the following settings:
      • Event name
        —Specify a descriptive name for your event notification containing up to 255 characters.
      • Prefix
        —Do not set a prefix as the Amazon S3 bucket is meant to be a dedicated bucket for collecting only network flow logs.
      • Event types
        —Select
        All object create events
        for the type of event notifications that you want to receive.
      • Destination
        —Select
        SQS queue
        to send notifications to an SQS queue to be read by a server.
      • Specify SQS queue
        —You can either select
        Choose from your SQS queues
        and then select the
        SQS queue
        , or select
        Enter SQS queue ARN
        and specify the ARN in the
        SQS queue
        field.
        You can retrieve your SQS queue ARN by opening another instance of the AWS Management Console in a browser window, and opening the Amazon SQS Console, and selecting the Amazon SQS that you created. In the
        Details
        section, under
        ARN
        , click the copy icon ( )), and paste the ARN in the field.
    3. Click
      Save changes
      .
      Once the event notification is created, a message indicating that the event notification was successfully created is displayed at the top of the page.
      If your receive an error when trying to save your changes, you should ensure that the permissions are set up correctly.
  6. Configure access keys for the AWS IAM user that Cortex XDR uses for API operations.
    • It is the responsibility of the customer’s organization to ensure that the user who performs this task of creating the access key is designated with the relevant permissions. Otherwise, this can cause the process to fail with errors.
    • Skip this step if you are using an
      Assumed Role
      for Cortex XDR.
    1. Open the AWS IAM Console, and in the navigation pane, select
      Access management
      Users
      .
    2. Select the
      User name
      of the AWS IAM user.
    3. Select the
      Security credentials
      tab, and scroll down to the
      Access keys
      section, and click
      Create access key
      .
    4. Click the copy icon next to the
      Access key ID
      and
      Secret access key
      keys, where you must click
      Show secret access key
      to see the secret key, and record them somewhere safe before closing the window. You will need to provide these keys when you edit the Access policy of the SQS queue and when setting the
      AWS Client ID
      and
      AWS Client Secret
      in Cortex XDR. If you forget to record the keys and close the window, you will need to generate new keys and repeat this process.
      For more information, see Managing access keys for IAM users.
  7. Update the Access policy of your Amazon SQS queue.
    Skip this step if you are using an
    Assumed Role
    for Cortex XDR.
    1. In the Amazon SQS Console, select the SQS queue that you created in Configure an Amazon Simple Queue Service (SQS).
    2. Select the
      Access policy
      tab, and
      Edit
      the Access policy code in the editor window to enable the IAM user to perform operations on the Amazon SQS with permissions to
      SQS:ChangeMessageVisibility
      ,
      SQS:DeleteMessage
      , and
      SQS:ReceiveMessage
      . Use this sample code as a guide for defining the
      “Sid”: “__receiver_statement”
      with the following definitions.
      • “aws:SourceArn”
        —Specify the ARN of the AWS IAM user. You can retrieve the
        User ARN
        from the
        Security credentials
        tab, which you accessed when configuring access keys for the AWS API user.
      • “Resource”
        —Leave the automatically generated ARN for the SQS queue that is set in the code, which uses the format
        “arn:sns:Region:account-id:topic-name”
        .
        For more information on granting permissions to publish messages to an SQS queue, see Granting permissions to publish event notification messages to a destination.
        { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "SQS:SendMessage", "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]", "Condition": { "ArnLike": { "aws:SourceArn": "[ARN of your Amazon S3 bucket]" } } }, { "Sid": "__receiver_statement", "Effect": "Allow", "Principal": { "AWS": "[Add the ARN for the AWS IAM user]" }, "Action": [ "SQS:ChangeMessageVisibility", "SQS:DeleteMessage", "SQS:ReceiveMessage" ], "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]" } ] }
  8. Configure the Amazon S3 collection in Cortex XDR.
    1. Select
      Settings ( )
      Configurations
      Data Collection
      Collection Integrations
      .
    2. In the
      Amazon S3
      configuration, click the
      here
      link to begin a new configuration.
    3. Set these parameters, where the parameters change depending on whether you configured an
      Access Key
      or
      Assumed Role
      .
      • To provide access to Cortex XDR to your logs and perform API operations using a designated AWS IAM user, leave the
        Access Key
        option selected. Otherwise, select
        Assumed Role
        , and ensure that you Create an Assumed Role for Cortex XDR before continuing with these instructions. In addition, when you create an Assumed Role for Cortex XDR, ensure that youedit the policy that defines the permissions for the Cortex XDR role with the Amazon S3 Bucket ARN and SQS ARN.
      • SQS URL
        —Specify the
        SQS URL
        , which is the ARN of the Amazon SQS that you configured in the AWS Management Console. For more information on how to retrieve your Amazon SQS ARN, see Specify SQS queue.
      • Name
        —Specify a descriptive name for your log collection configuration.
      • When setting an
        Access Key
        , set these parameters.
      • When setting an
        Assumed Role
        , set these parameters.
      • Log Type
        —Select
        Audit Logs
        to configure your log collection to receive audit logs from Amazon S3 via AWS CloudTrail. When configuring audit log collection, the following additional field is displayed for the
        Configuration
        .
        You can
        Normalize and enrich audit logs
        by selecting the checkbox. If selected, Cortex XDR stitches Amazon S3 audit logs with other Cortex XDR authentication stories across all cloud providers using the same format, which you can query with XQL Search using the
        cloud_audit_logs
        dataset.
    4. Click
      Test
      to validate access, and then click
      Enable
      .
      Once events start to come in, a green check mark appears underneath the
      Amazon S3
      configuration with the number of logs received.

Recommended For You