Ingest Network Flow Logs from Amazon S3

Take advantage of Cortex XDR investigation capabilities and set up network flow log ingestion for your Amazon S3 logs using an AWS CloudFormation Script.
Ingesting logs and data requires a Cortex XDR Pro per TB license.
You can forward network flow logs for the relative service to Cortex XDR from Amazon Simple Storage Service (Amazon S3).
To receive network flow logs from Amazon S3, you must first configure data collection from Amazon S3. You can then configure the Collection Integrations settings in Cortex XDR for Amazon S3. After you set up collection integration, Cortex XDR begins receiving new logs and data from the source.
You can either configure Amazon S3 with SQS notification manually on your own or use the AWS CloudFormation Script that we have created for you to make the process easier. The instructions below explain how to configure Cortex XDR to receive network flow logs from Amazon S3 using SQS. To perform these steps manually, see Configure Data Collection from Amazon S3 Manually.
For more information on configuring data collection from Amazon S3, see the Amazon S3 Documentation.
As soon as Cortex XDR begins receiving logs, the app automatically creates an Amazon S3 XQL dataset (
aws_s3_raw
). This enables you to search the logs with XQL Search using the dataset. For example queries, refer to the in-app XQL Library. You can also configure Cortex XDR to ingest network flow logs as XDR network connection stories, which you can query with XQL Search using the
xdr_dataset
dataset with the preset called
network_story
. Cortex XDR can also raise Cortex XDR alerts (Analytics, Correlation Rules, IOC, and BIOC only) when relevant from Amazon S3 logs. Analytics alerts are only raised on normalized logs.
Be sure you do the following tasks before you begin configuring data collection from Amazon S3 using the AWS CloudFormation Script.
  • Ensure that you have the proper permissions to run AWS CloudFormation with the script provided in Cortex XDR. You need at a minimum the following permissions in AWS for an Amazon S3 bucket and Amazon Simple Queue Service (SQS):
    • Amazon S3 bucket
      GetObject
    • SQS
      ChangeMessageVisibility
      ,
      ReceiveMessage
      , and
      DeleteMessage
      .
  • Ensure that you can access your Amazon Virtual Private Cloud (VPC) and have the necessary permissions to create flow logs.
  • Determine how you want to provide access to Cortex XDR to your logs and to perform API operations. You have the following options:
Configure Cortex XDR to receive network flow logs from Amazon S3 using the CloudFormation Script.
  1. Download the CloudFormation Script in Cortex XDR.
    1. Select
      Settings ( )
      Configurations
      Data Collection
      Collection Integrations
      .
    2. In the
      Amazon S3
      configuration, click the
      here
      link to begin a new configuration.
    3. To provide access to Cortex XDR to your logs and to perform API operations using a designated AWS IAM user, leave the
      Access Key
      option selected. Otherwise, select
      Assumed Role
      , and ensure that you Create an Assumed Role for Cortex XDR before continuing with these instructions.
    4. For the
      Log Type
      , select
      Flow Logs
      to configure your log collection to receive network flow logs from Amazon S3, and the following text is displayed under the field
      Download CloudFormation Script. See instructions here.
    5. Click the
      Download CloudFormation Script.
      link to download the script to your computer.
  2. Create a new Stack in the CloudFormation Console with the script you downloaded from Cortex XDR.
    For more information on creating a Stack, see Creating a stack on the AWS CloudFormation console.
    1. Log in to the CloudFormation Console.
    2. From the
      CloudFormation
      Stacks
      page, ensure that you have selected the correct region for your configuration.
    3. Select
      Create Slack
      With new resources (standard)
      .
    4. Specify the template that you want AWS CloudFormation to use to create your stack. This template is the script that you downloaded from Cortex XDR, which will create an Amazon S3 bucket, Amazon Simple Queue Service (SQS) queue, and Queue Policy. Configure the following settings in the
      Specify template
      page.
      • Prerequisite - Prepare template
        Prepare template
        —Select
        Template is ready
        .
      • Specify Template
        • Template source
          —Select
          Upload a template file
          .
        • Upload a template file
          Choose file
          , and select the
          cortex-xdr-create-s3-with-sqs-flow-logs.json
          file that you downloaded from Cortex XDR.
    5. Click
      Next
      .
    6. In the
      Specify stack details
      page, configure the following stack details.
      • Stack name
        —Specify a descriptive name for your stack.
      • Parameters
        Cortex XDR Flow Logs Integration
        • Bucket Name
          —Specify the name of the S3 bucket to create, where you can leave the default populated name as
          xdr-flow-logs
          or create a new one. The name must be unique.
        • Publisher Account ID
          —Specify the AWS IAM user account ID with whom you are sharing access.
        • Queue Name
          —Specify the name for your Amazon SQS queue to create, where you can leave the default populated name as
          xdr-flow
          or create a new one. The name must be unique.
    7. Click
      Next
      .
    8. In the
      Configure stack options
      page, there is nothing to configure, so click
      Next
      .
    9. In the
      Review
      page, look over the stack configurations settings that you have configured and if they are correct, click
      Create stack
      . If you need to make a change, click
      Edit
      beside the particular step that you want to update.
      The stack is created and is opened with the
      Events
      tab displayed. It can take a few minutes for the new Amazon S3 bucket, SQS queue, and Queue Policy to be created. Click
      Refresh
      to get updates. Once everything is created, leave the stack opened in the current browser as you will need to access information in the stack for other steps detailed below.
      For the Amazon S3 bucket created using CloudFormation, it is the customer’s responsibility to define a retention policy by creating a
      Lifecycle rule
      in the
      Management
      tab. We recommend setting the retention policy to at least 7 days to ensure that the data is retrieved under all circumstances.
  3. Configure your Amazon Virtual Private Cloud (VPC) with flow logs:
    1. Open the Amazon VPC Console, and in the
      Resources by Region
      listed, select
      VPCs
      to view the VPCs configured for the current region selected. To select another VPC from another region, select
      See all regions
      , and select one of them.
      To create a new VPC, click
      Launch VPC Wizard
      . For more information, see AWS VPC Flow Logs.
    2. From the list of
      Your VPCs
      , select the checkbox beside the VPC that you want to configure to create flow logs, and then select
      Actions
      Create flow log
      .
    3. Configure the following
      Flow log settings
      :
      • Name - optional
        —(
        Optional
        ) Specify a descriptive name for your VPC flow log.
      • Filter
        —Select
        All
        types of traffic to capture.
      • Maximum aggregation interval
        —If you anticipate a heavy flow of traffic, select
        1 minute
        . Otherwise, leave the default setting as
        10 minutes
        .
      • Destination
        —Select
        Send to an Amazon S3 bucket
        as the destination to publish the flow log data.
      • S3 bucket ARN
        —Specify the Amazon Resource Name (ARN) for your Amazon S3 bucket.
        You can retrieve your bucket’s ARN by opening another instance of the AWS Management Console in a browser window, and opening the Amazon S3 console. In the
        Buckets
        section, select the bucket that you created for collecting the Amazon S3 flow logs when you created your stack, click
        Copy ARN
        , and paste the ARN in this field.
      • Log record format
        —Specify the fields to include in the flow log record, where we recommend leaving the default
        AWS default format
        selected.
    4. Click
      Create flow log
      .
      Once the flow log is created, a message indicating that the flow log was successfully created is displayed at the top of the
      Your VPCs
      page.
      In addition, if you open your Amazon S3 bucket configurations, by selecting the bucket from the Amazon S3 console, the
      Objects
      tab contains a folder called
      AWSLogs/
      to collect the flow logs.
  4. Configure access keys for the AWS IAM user that Cortex XDR uses for API operations.
    • It is the responsibility of the customer’s organization to ensure that the user who performs this task of creating the access key is designated with the relevant permissions. Otherwise, this can cause the process to fail with errors.
    • Skip this step if you are using an
      Assumed Role
      for Cortex XDR.
    1. Open the AWS IAM Console, and in the navigation pane, select
      Access management
      Users
      .
    2. Select the
      User name
      of the AWS IAM user.
    3. Select the
      Security credentials
      tab, and scroll down to the
      Access keys
      section, and click
      Create access key
      .
    4. Click the copy icon next to the
      Access key ID
      and
      Secret access key
      keys, where you must click
      Show secret access key
      to see the secret key, and record them somewhere safe before closing the window. You will need to provide these keys when you edit the Access policy of the SQS queue and when setting the
      AWS Client ID
      and
      AWS Client Secret
      in Cortex XDR. If you forget to record the keys and close the window, you will need to generate new keys and repeat this process.
    For more information, see Managing access keys for IAM users.
  5. Skip this step if you are using an
    Access Key
    to provide access to Cortex XDR.
  6. Configure the Amazon S3 collection in Cortex XDR:
    1. Select
      Settings ( )
      Configurations
      Data Collection
      Collection Integrations
      .
    2. In the
      Amazon S3
      configuration, click the
      here
      link to begin a new configuration.
    3. Set these parameters, where the parameters change depending on whether you configured an
      Access Key
      or
      Assumed Role
      .
      • SQS URL
        —Specify the
        SQS URL
        , which is taken from the Stack you created. In the browser you left open after creating the stack, open the
        Outputs
        tab, and copy the
        Value
        of the
        QueueURL
        and paste it in this field.
      • Name
        —Specify a descriptive name for your log collection configuration.
      • When setting an
        Access Key
        , set these parameters.
      • When setting an
        Assumed Role
        , set these parameters.
      • Log Type
        —Select
        Flow Logs
        to configure your log collection to receive network flow logs from Amazon S3. When configuring network flow log collection, the following additional field is displayed for the
        Configuration
        .
        You can
        Normalize and enrich flow logs
        by selecting the checkbox. If selected, Cortex XDR ingests the network flow logs as XDR network connection stories, which you can query using XQL Search from the
        xdr_dataset
        dataset using the preset called
        network_story
        .
    4. Click
      Test
      to validate access, and then click
      Enable
      .
      Once events start to come in, a green check mark appears underneath the
      Amazon S3
      configuration with the number of logs received.

Recommended For You