Ingest Generic Logs from Amazon S3

Take advantage of Cortex XDR investigation capabilities and set up generic log ingestion for your Amazon S3 logs.
Ingesting logs and data requires a Cortex XDR Pro per TB license.
You can forward generic logs for the relative service to Cortex XDR from Amazon S3.
To receive generic data from Amazon Simple Storage Service (Amazon S3), you must first configure data collection from Amazon S3. You can then configure the Collection Integrations settings in Cortex XDR for Amazon S3. After you set up collection integration, Cortex XDR begins receiving new logs and data from the source.
For more information on configuring data collection from Amazon S3, see the Amazon S3 Documentation.
As soon as Cortex XDR begins receiving logs, the app automatically creates an Amazon S3 XQL dataset (
<Vendor>_<Product>_raw
). This enables you to search the logs using XQL Search with the dataset. For example queries, refer to the in-app XQL Library. Cortex XDR can also raise Cortex XDR alerts (IOC, BIOC, and Correlation Rule only) when relevant from Amazon S3 logs.
You need to set up an Amazon S3 data collector to receive generic logs when collecting logs from BeyondTrust Privilege Management Cloud. For more information, see Ingest Logs from BeyondTrust Privilege Management Cloud.
Be sure you do the following tasks before you begin configuring data collection from Amazon S3.
  • Create a dedicated Amazon S3 bucket, which collects the generic logs that you want captured. For more information, see Creating a bucket using the Amazon S3 Console.
    It is the customer’s responsibility to define a retention policy for your Amazon S3 bucket by creating a
    Lifecycle rule
    in the
    Management
    tab. We recommend setting the retention policy to at least 7 days to ensure that the data is retrieved under all circumstances.
  • The logs collected by your dedicated Amazon S3 bucket must adhere to the following guideline:
    • Each log file must use the 1 log per line format as multi-line format is not supported.
    • The log format must be compressed as gzip or uncompressed.
    • For best performance, we recommend limiting each file size to up to 50 MB (compressed).
  • Ensure that you have at a minimum the following permissions in AWS for an Amazon S3 bucket and Amazon Simple Queue Service (SQS):
    • Amazon S3 bucket
      GetObject
    • SQS
      ChangeMessageVisibility
      ,
      ReceiveMessage
      , and
      DeleteMessage
      .
  • Determine how you want to provide access to Cortex XDR to your logs and to perform API operations. You have the following options:
Configure Cortex XDR to receive generic logs from Amazon S3.
  1. Log in to the AWS Management Console.
  2. From the menu bar, ensure that you have selected the correct region for your configuration.
  3. Configure an Amazon Simple Queue Service (SQS).
    Ensure that you create your Amazon S3 bucket and Amazon SQS queue in the same region.
    1. In the Amazon SQS Console, click
      Create Queue
      .
    2. Configure the following settings, where the default settings should be configured unless otherwise indicated.
      • Type
        —Select
        Standard
        queue (default).
      • Name
        —Specify a descriptive name for your SQS queue.
      • Configuration
        section—Leave the default settings for the various fields.
      • Access policy
        Choose method
        —Select
        Advanced
        and update the Access policy code in the editor window to enable your Amazon S3 bucket to publish event notification messages to your SQS queue. Use this sample code as a guide for defining the
        “Statement”
        with the following definitions:
        -
        “Resource”
        —Leave the automatically generated ARN for the SQS queue that is set in the code, which uses the format
        “arn:sns:Region:account-id:topic-name”
        .
        -
        “Resource”
        —Leave the automatically generated ARN for the SQS queue that is set in the code, which uses the format
        “arn:sns:Region:account-id:topic-name”
        .
        You can retrieve your bucket’s ARN by opening the Amazon S3 Console in a browser window. In the
        Buckets
        section, select the bucket that you created for collecting the Amazon S3 flow logs, click
        Copy ARN
        , and paste the ARN in the field.
        For more information on granting permissions to publish messages to an SQS queue, see Granting permissions to publish event notification messages to a destination.
        { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "SQS:SendMessage", "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]", "Condition": { "ArnLike": { "aws:SourceArn": "[ARN of your Amazon S3 bucket]" } } }, ] }
      • Dead-letter queue
        section—We recommend that you configure a queue for sending undeliverable messages by selecting
        Enabled
        , and then in the
        Choose queue
        field selecting the queue to send the messages. You may need to create a new queue for this, if you do not already have one set up. For more information, see Amazon SQS dead-letter queues.
    3. Click
      Create queue
      .
      Once the SQS is created, a message indicating that the queue was successfully configured is displayed at the top of the page.
  4. Configure an event notification to your Amazon SQS whenever a file is written to your Amazon S3 bucket.
    1. Open the Amazon S3 Console and in the
      Properties
      tab of your Amazon S3 bucket, scroll down to the
      Event notifications
      section, and click
      Create event notification
      .
    2. Configure the following settings:
      • Event name
        —Specify a descriptive name for your event notification containing up to 255 characters.
      • Prefix
        —Do not set a prefix as the Amazon S3 bucket is meant to be a dedicated bucket for collecting only network flow logs.
      • Event types
        —Select
        All object create events
        for the type of event notifications that you want to receive.
      • Destination
        —Select
        SQS queue
        to send notifications to an SQS queue to be read by a server.
      • Specify SQS queue
        —You can either select
        Choose from your SQS queues
        and then select the
        SQS queue
        , or select
        Enter SQS queue ARN
        and specify the ARN in the
        SQS queue
        field.
        You can retrieve your SQS queue ARN by opening another instance of the AWS Management Console in a browser window, and opening the Amazon SQS Console, and selecting the Amazon SQS that you created. In the
        Details
        section, under
        ARN
        , click the copy icon ( )), and paste the ARN in the field.
    3. Click
      Save changes
      .
      Once the event notification is created, a message indicating that the event notification was successfully created is displayed at the top of the page.
      If your receive an error when trying to save your changes, you should ensure that the permissions are set up correctly.
  5. Configure access keys for the AWS IAM user.
    • It is the responsibility of the customer’s organization to ensure that the user who performs this task of creating the access key is designated with the relevant permissions. Otherwise, this can cause the process to fail with errors.
    • Skip this step if you are using an
      Assumed Role
      for Cortex XDR.
    1. Open the AWS IAM Console, and in the navigation pane, select
      Access management
      Users
      .
    2. Select the
      User name
      of the AWS IAM user.
    3. Select the
      Security credentials
      tab, and scroll down to the
      Access keys
      section, and click
      Create access key
      .
    4. Click the copy icon () next to the
      Access key ID
      and
      Secret access key
      keys, where you must click
      Show secret access key
      to see the secret key, and record them somewhere safe before closing the window. You will need to provide these keys when you edit the Access policy of the SQS queue and when setting the
      AWS Client ID
      and
      AWS Client Secret
      in Cortex XDR. If you forget to record the keys and close the window, you will need to generate new keys and repeat this process.
      For more information, see Managing access keys for IAM users.
  6. Update the Access policy of your Amazon SQS queue.
    Skip this step if you are using an
    Assumed Role
    for Cortex XDR.
    1. In the Amazon SQS Console, select the SQS queue that you created in Configure an Amazon Simple Queue Service (SQS).
    2. Select the
      Access policy
      tab, and
      Edit
      the Access policy code in the editor window to enable the IAM user to perform operations on the Amazon SQS with permissions to
      SQS:ChangeMessageVisibility
      ,
      SQS:DeleteMessage
      , and
      SQS:ReceiveMessage
      . Use this sample code as a guide for defining the
      “Sid”: “__receiver_statement”
      with the following definitions.
      • “aws:SourceArn”
        —Specify the ARN of the AWS IAM user. You can retrieve the
        User ARN
        from the
        Security credentials
        tab, which you accessed when configuring access keyps for the AWS API user.
      • “Resource”
        —Leave the automatically generated ARN for the SQS queue that is set in the code, which uses the format
        “arn:sns:Region:account-id:topic-name”
        .
        For more information on granting permissions to publish messages to an SQS queue, see Granting permissions to publish event notification messages to a destination.
        { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "SQS:SendMessage", "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]", "Condition": { "ArnLike": { "aws:SourceArn": "[ARN of your Amazon S3 bucket]" } } }, { "Sid": "__receiver_statement", "Effect": "Allow", "Principal": { "AWS": "[Add the ARN for the AWS IAM user]" }, "Action": [ "SQS:ChangeMessageVisibility", "SQS:DeleteMessage", "SQS:ReceiveMessage" ], "Resource": "[Leave automatically generated ARN for the SQS queue defined by AWS]" } ] }
  7. Configure the Amazon S3 collection in Cortex XDR:
    1. Select
      Settings ( )
      Configurations
      Data Collection
      Collection Integrations
      .
    2. In the
      Amazon S3
      configuration, click the
      here
      link to begin a new configuration.
    3. Set these parameters, where the parameters change depending on whether you configured an
      Access Key
      or
      Assumed Role
      .
      • To provide access to Cortex XDR to your logs and perform API operations using a designated AWS IAM user, leave the
        Access Key
        option selected. Otherwise, select
        Assumed Role
        , and ensure that you Create an Assumed Role for Cortex XDR before continuing with these instructions. In addition, when you create an Assumed Role for Cortex XDR, ensure that youedit the policy that defines the permissions for the Cortex XDR role with the Amazon S3 Bucket ARN and SQS ARN.
      • SQS URL
        —Specify the
        SQS URL
        , which is the ARN of the Amazon SQS that you configured in the AWS Management Console. For more information on how to retrieve your Amazon SQS ARN, see Specify SQS queue.
      • Name
        —Specify a descriptive name for your log collection configuration.
      • When setting an
        Access Key
        , set these parameters.
      • When setting an
        Assumed Role
        , set these parameters.
      • Log Type
        —Select
        Generic
        to configure your log collection to receive generic logs from Amazon S3, which can include different types of data, such as file and metadata. When selecting this option, the following additional fields are displayed.
        • Log Format
          —Select the log format type as
          Raw
          ,
          JSON
          ,
          CEF
          ,
          LEEF
          ,
          Cisco
          ,
          Corelight
          , or
          Beyondtrust Cloud ECS
          .
          -The
          Vendor
          and
          Product
          defaults to
          Auto-Detect
          when the
          Log Format
          is set to
          CEF
          or
          LEEF
          .
          -For a
          Log Format
          set to
          CEF
          ,
          LEEF
          ,
          Cisco
          , or
          Corelight
          , Cortex XDR reads events row by row to look for the
          Vendor
          and
          Product
          configured in the logs. When the values are populated in the event log row, Cortex XDR uses these values even if you specified a value in the
          Vendor
          and
          Product
          fields in the Amazon S3 data collector settings. Yet, when the values are blank in the event log row, Cortex XDR uses the
          Vendor
          and
          Product
          that you specified in the
          Vendor
          and
          Product
          fields in the Amazon S3 data collector settings. If you did not specify a
          Vendor
          or
          Product
          in the Amazon S3 data collector settings, and the values are blank in the event log row, the values for both fields are set to
          unknown
          .
          -For a
          Log Format
          set to
          Beyondtrust Cloud ECS
          , the following fields are automatically set and not configurable.
          -
          Vendor
          Beyondtrust
          -
          Product
          Privilege Management
          -
          Compression
          Uncompressed
        • Vendor
          —(
          Optional
          ) Specify a particular vendor name for the Amazon S3 generic data collection, which is used in the Amazon S3 XQL dataset
          <Vendor>_<Product>_raw
          that Cortex XDR creates as soon as it begins receiving logs.
        • Product
          —(
          Optional
          ) Specify a particular product name for the Amazon S3 generic data collection, which is used in the Amazon S3 XQL dataset name
          <Vendor>_<Product>_raw
          that Cortex XDR creates as soon as it begins receiving logs.
        • Compression
          —Select whether the logs are compressed into a
          gzip
          file or are
          uncompressed
          .
    4. Click
      Test
      to validate access, and then click
      Enable
      .
      Once events start to come in, a green check mark appears underneath the
      Amazon S3
      configuration with the number of logs received.

Recommended For You