Ingest Logs and Data from a GCP Pub/Sub

If you use the Pub/Sub messaging service from Global Cloud Platform (GCP), you can send logs and data from GCP to Cortex XDR.
Ingesting logs and data requires a Cortex XDR Pro per TB license.
If you use the Pub/Sub messaging service from Global Cloud Platform (GCP), you can send logs and data from your GCP instance to Cortex XDR. Data from GCP is then searchable in Cortex XDR to provide additional information and context to your investigations using the GCP XQL dataset (
google_cloud_logging_raw
). For example queries, refer to the in-app XQL Library.
You can also configure Cortex XDR to normalize GCP audit logs, which you can query with XQL Search using the
cloud_audit_logs
dataset. In addition, you can configure Cortex XDR to ingest network flow logs as XDR network connection stories, which you can query with XQL Search using the
xdr_dataset
dataset with the preset called
network_story
. Cortex XDR can also raise Cortex XDR alerts (Analytics, IOC, BIOC, and Correlation Rule only) when relevant from GCP logs. Analytics alerts are only raised on normalized logs.
When collecting flow logs, we recommend that you include GKE annotations in your logs, which enable you to view the names of the containers that communicated with each other. GKE annotations are only included in logs if appended manually using the custom metadata configuration in GCP. For more information, see VPC Flow Logs Overview. In addition, to customize metadata fields, you must use the gcloud command-line interface or the API. For more information, see Using VPC Flow Logs.
To receive logs and data from GCP, you must first set up log forwarding using a Pub/Sub topic in GCP. You can configure GCP settings using either the GCP web interface or a GCP cloud shell terminal. After you set up your service account in GCP, you configure the Data Collection settings in Cortex XDR. The setup process requires the subscription name and authentication key from your GCP instance.
After you set up log collection, Cortex XDR immediately begins receiving new logs and data from GCP.

Set up Log Forwarding Using the GCP Web Interface

  1. Log in to your GCP account.
  2. Set up log forwarding from GCP to Cortex XDR:
    1. Select
      Logging
      Logs Router
      .
    2. Select
      Create Sink
      Cloud Pub/Sub topic
      and then click
      Next
      .
    3. To filter only specific types of data, select the filter or desired resource.
    4. In the
      Edit Sink
      configuration, define a descriptive
      Sink Name
      .
    5. Select
      Sink Destination
      Create new Cloud Pub/Sub topic
      .
    6. Enter a descriptive
      Name
      that identifies the sink purpose for Cortex XDR, and then
      Create
      .
    7. Create Sink
      and then
      Close
      when finished.
  3. Create a subscription for your Pub/Sub topic.
    1. Select the hamburger menu in G Cloud and then select
      Pub/Sub
      Topics
      .
    2. Select the name of the topic you created in the previous steps. Use the filters if necessary.
    3. Create Subscription
      Create subscription
      .
    4. Enter a unique Subscription ID.
    5. Choose
      Pull
      as the
      Delivery Type
      .
    6. Create
      the subscription.
      After the subscription is set up, G Cloud displays statistics and settings for the service.
    7. In the subscription details, identify and note your
      Subscription Name
      .
      Optionally, use the copy button to copy the name to the clipboard. You will need the name when you configure Collection in Cortex XDR.
  4. Create a service account and authentication key.
    You will use the key to enable Cortex XDR to authenticate with the subscription service.
    1. Select the hamburger menu and then select
      IAM & Admin
      Service Accounts
      .
    2. Create Service Account
      .
    3. Enter a
      Service account name
      and then
      Create
      .
    4. Select a role for the account:
      Pub/Sub
      Pub/Sub Subscriber
      .
    5. Click
      Continue
      Done
      .
    6. Locate the service account by name, using the filters to refine the results, if needed.
    7. Click the
      Actions
      menu identified by the three dots in the row for the service account and then
      Create Key
      .
    8. Select JSON as the key type, and then
      Create
      .
      After you create the service account key, G Cloud automatically downloads it.
  5. In Cortex XDR, set up Data Collection.
    1. Select
      Settings ( )
      Configurations
      Data Collection
      Collection Integrations
      .
    2. In the Google Cloud Platform configuration, click the
      here
      link.
    3. Specify the
      Subscription Name
      that you previously noted or copied.
    4. Browse to the JSON file containing your authentication key for the service account.
    5. (
      Optional
      ) You can
      Normalize and enrich flow and audit logs
      by selecting the checkbox. If selected, Cortex XDR ingests the network flow logs as XDR network connection stories, which you can query using XQL Search from the
      xdr_dataset
      dataset with the preset called
      network_story
      . In addition, you can configure Cortex XDR to normalize GCP audit logs, which you can query with XQL Search using the
      cloud_audit_logs
      dataset.
    6. Test
      the provided settings and, if successful, proceed to
      Enable
      log collection.

Set up Log Forwarding Using the GCP Cloud Shell Terminal

  1. Launch the GCP cloud shell terminal or use your preferred shell with gcloud installed.
  2. Define your project ID.
    gcloud config set project
    <PROJECT_ID>
  3. Create a Pub/Sub topic.
    gcloud pubsub topics create
    <TOPIC_NAME>
  4. Create a subscription for this topic.
    gcloud pubsub subscriptions create
    <SUBSCRIPTION_NAME>
    --topic=
    <TOPIC_NAME>
    Note the subscription name you define in this step as you will need it to set up log ingestion from Cortex XDR.
  5. Create a logging sink.
    During the logging sink creation, you can also define additional log filters to exclude specific logs. To filter logs, supply the optional parameter
    --log-filter=
    <LOG_FILTER>
    gcloud logging sinks create
    <SINK_NAME>
    pubsub.googleapis.com/projects/
    <PROJECT_ID>
    /topics/
    <TOPIC_NAME>
    --log-filter=
    <LOG_FILTER>
    If setup is successful, the console displays a summary of your log sink settings:
    Created [https://logging.googleapis.com/v2/projects/PROJECT_ID/sinks/SINK_NAME]. Please remember to grant `serviceAccount:LOGS_SINK_SERVICE_ACCOUNT` \ the Pub/Sub Publisher role on the topic. More information about sinks can be found at /logging/docs/export/configure_export
  6. Grant log sink service account to publish to the new topic
    Note the
    serviceAccount
    name from the previous step and use it to define the service for which you want to grant publish access.
    gcloud pubsub topics add-iam-policy-binding
    <TOPIC_NAME>
    --member serviceAccount:
    <LOGS_SINK_SERVICE_ACCOUNT>
    --role=roles/pubsub.publisher
  7. Create a service account.
    For example, use cortex-xdr-sa as the service account name and Cortex XDR Service Account as the display name.
    gcloud iam service-accounts create
    <SERVICE_ACCOUNT>
    --description="
    <DESCRIPTION>
    " --display-name="
    <DISPLAY_NAME>
    "
  8. Grant the IAM role to the service account.
    gcloud pubsub subscriptions add-iam-policy-binding
    <SUBSCRIPTION_NAME>
    --member serviceAccount:
    <SERVICE_ACCOUNT>
    @
    <PROJECT_ID>
    .iam.gserviceaccount.com --role=roles/pubsub.subscriber
  9. Create a JSON key for the service account.
    You will need the JSON file to enable Cortex XDR to authenticate with the GCP service. Specify the file destination and filename using a .json extension.
    gcloud iam service-accounts keys create
    <OUTPUT_FILE>
    --iam-account
    <SERVICE_ACCOUNT>
    @
    <PROJECT_ID>
    .iam.gserviceaccount.com
  10. In Cortex XDR, set up Data Collection.
    1. Select
      Settings ( )
      Configurations
      Data Collection
      Collection Integrations
      .
    2. In the Google Cloud Platform configuration, click the
      here
      link.
    3. Specify the
      Subscription Name
      that you previously noted or copied.
    4. Browse to the JSON file containing your authentication key for the service account.
    5. (
      Optional
      ) You can
      Normalize and enrich flow and audit logs
      by selecting the checkbox. If selected, Cortex XDR ingests the network flow logs as XDR network connection stories, which you can query using XQL Search from the
      xdr_dataset
      dataset with the preset called
      network_story
      . In addition, you can configure Cortex XDR to normalize GCP audit logs, which you can query with XQL Search using the
      cloud_audit_logs
      dataset.
    6. Test
      the provided settings and, if successful, proceed to
      Enable
      log collection.
  11. After Cortex XDR begins receiving information from the GCP Pub/Sub service, you can use the XQL Query language to search for specific data.

Recommended For You