Configure Data Collection from Amazon S3 Manually

Set up network flow log ingestion for your Amazon S3 logs manually (without a script).
Ingesting logs and data requires a Cortex XDR Pro per TB license.
There are various reasons why you may need to configure data collection from Amazon S3 manually, as opposed to using the CloudFormation Script provided in Cortex XDR. For example, if your organization does not use CloudFormation scripts, you will need to follow the instructions below, which explain at a high-level how to perform these steps manually with a link to the relevant topic in the Amazon S3 documentation with the detailed steps to follow.
As soon as Cortex XDR begins receiving logs, the app automatically creates an Amazon S3 XQL dataset (
aws_s3_raw
). This enables you to search the logs with XQL Search using the dataset. For example queries, refer to the in-app XQL Library. You can also configure Cortex XDR to ingest network flow logs as XDR network connection stories, which you can query with XQL Search using the
xdr_dataset
dataset with the preset called
network_story
. Cortex XDR can also raise Cortex XDR alerts (Correlations, IOC, and BIOC only) when relevant from Amazon S3 logs.
Be sure you do the following tasks before you begin configuring data collection manually from Amazon CloudWatch to Amazon S3.
If you already have an Amazon S3 bucket configured with VPC flow logs that you want to use for this configuration, you do not need to perform the prerequisite steps detailed in the first two bullets.
  • Ensure that you have at a minimum the following permissions in AWS for an Amazon S3 bucket and Amazon Simple Queue Service (SQS):
    • Amazon S3 bucket
      GetObject
    • SQS
      ChangeMessageVisibility
      ,
      ReceiveMessage
      , and
      DeleteMessage
      .
  • Create a dedicated Amazon S3 bucket for collecting network flow logs with the default settings. For more information, see Creating a bucket using the Amazon S3 Console.
    It is the customer’s responsibility to define a retention policy for your Amazon S3 bucket by creating a
    Lifecycle rule
    in the
    Management
    tab. We recommend setting the retention policy to at least 7 days to ensure that the data is retrieved under all circumstances.
  • Ensure that you can access your Amazon Virtual Private Cloud (VPC) and have the necessary permissions to create flow logs.
  • Determine how you want to provide access to Cortex XDR to your logs and to perform API operations. You have the following options:
Configure Cortex XDR to receive network flow logs from Amazon S3 manually.
  1. Log in to the AWS Management Console.
  2. From the menu bar, ensure that you have selected the correct region for your configuration.
  3. Configure your Amazon Virtual Private Cloud (VPC) with flow logs. For more information, see AWS VPC Flow Logs.
    If you already have an Amazon S3 bucket configured with VPC flow logs, skip this step and go to Configure an Amazon Simple Queue Service (SQS).
  4. Configure an Amazon Simple Queue Service (SQS). For more information, see Configuring Amazon SQS queues (console).
    Ensure that you create your Amazon S3 bucket and Amazon SQS queue in the same region.
  5. Configure an event notification to your Amazon SQS whenever a file is written to your Amazon S3 bucket. For more information, see Amazon S3 Event Notifications.
  6. Configure access keys for the AWS IAM user that Cortex XDR uses for API operations. For more information, see Managing access keys for IAM users.
    • It is the responsibility of the customer’s organization to ensure that the user who performs this task of creating the access key is designated with the relevant permissions. Otherwise, this can cause the process to fail with errors.
    • Skip this step if you are using an
      Assumed Role
      for Cortex XDR.
  7. Update the Access Policy of your SQS queue and grant the required permissions mentioned above to the relevant IAM user. For more information, see Granting permissions to publish event notification messages to a destination.
    Skip this step if you are using an
    Assumed Role
    for Cortex XDR.
  8. Configure the Amazon S3 collection in Cortex XDR:
    1. Select
      Settings ( )
      Configurations
      Data Collection
      Collection Integrations
      .
    2. In the
      Amazon S3
      configuration, click the
      here
      link to begin a new configuration.
    3. Set these parameters, where the parameters change depending on whether you configured an
      Access Key
      or
      Assumed Role
      .
    4. Click
      Test
      to validate access, and then click
      Enable
      .
      Once events start to come in, a green check mark appears underneath the
      Amazon S3
      configuration with the number of logs received.

Recommended For You