Ingest Incidents from a SIEM Using Splunk - Tutorials - 6.x - Cortex XSOAR - Cortex - Security Operations

Cortex XSOAR Tutorials 6.x

Product
Cortex XSOAR
Version
6.x
Creation date
2022-09-02
Last date published
2023-04-16
Category
Tutorials

This tutorial shows you how to design the incident lifecycle using Splunk, from ingesting to closing security events.

The incident lifecycle is designed for Cortex XSOAR (SOC) engineers or architects. The aim is to automate as much of the process as possible and leave the analyst to make accurate and confident decisions when needed.

By the end of this tutorial, you should have configured your Splunk integration, set up a basic flow, started ingesting incidents from the Splunk to Cortex XSOAR, which enables the analyst to start investigating an incident.

This tutorial includes the following stages.

Analyst Flow

Depending on how you design the incident lifecycle, many actions are performed automatically even before the analyst investigates an incident. For example, incidents may be closed as duplicates, closed as false-positives, data is enriched, etc.

Usually, the analyst does the following:

  • Selects an open incident.

  • Analyzes enough data to make decisions based on the steps that have already been taken (in the Incident Summary, and Work Plan, etc.).

  • Collects additional data by running commands on third-party integrations, such as VirusTotal, WildFire, etc.

  • Collaborates with SOC team members in the War Room (ask questions, post comments, etc.).

  • Decides how to handle the incident. Usually the architect designs the Incident page to include a set of buttons that the analyst can select from (Escalate, Close, etc.)

  • Activates one of several possible responses either within Cortex XSOAR or externally (for example, open a ticket for IT).

  • Closes the incident and specify a closed reason.

Architect Flow

To explain the architect flow we need to define and design the incident lifecycle using Splunk.

Define the incident Type

The first step when planning your incident identification and response is to create a basic outline of the flows that your SOC handles. Consider the same information in the Architect Flow in using QRadar.

Plan the workflow for each incident type

This section describes the aspects of the workflow that occur before the analyst is assigned.

  • Assign incidents to the flow

    In Cortex XSOAR, incident classification is the process by which an incoming security event/alert is assigned to a Cortex XSOAR incident type. When you fetch security events from Splunk, the event data is ingested in JSON format. You classify the security event based on the fields (keys) from the event JSON.

    Install the Splunk Content Pack, which includes Splunk Classifiers, although you do not need to define a classifier, as all Splunk incidents are ingested as the Splunk Notable Generic incident type (when you assign incident type to the integration in Cortex XSOAR).Classification and Mapping

  • Pre-process incoming incidents

    Usually you want to eliminate false-positives and deduplicate incidents, which enables you to maintain a clean set of incidents for analysts.

    Pre-processing rules enable you to perform certain actions on security events before they are ingested as incidents in Cortex XSOAR. You perform actions when the condition of a rule is met. For example, link the incoming incident to an existing incident, or under configured conditions, drop the incoming incident altogether. You can also run a script on the incident when the conditions are met. For an example, see the Phishing Pre-Process Tutorial.

  • Run a playbook on the incident

    After the security event is ingested, classified to an Cortex XSOAR incident type, and filtered through the pre-processing rules, the default playbook for the incident type is run on the incident. Usually, the default playbook extracts one or a combination of the following:

    • Extracts and enriches indicators

    • Determines the incident severity

    • Assigns the incident to an analyst for manual review

    Depending on how you design the workflow for the incident type, when the playbook completes its run, it is the first time that the incident data is presented to analysts in the layout. Although Splunk comes out of the box with the SplunkPy playbook, consider the following when customizing or creating your own playbook.

    1. Plan the incident process per incident type (fully automated, manual, or hybrid).

    2. Define how incidents are classified (assigned to an incident type) in Cortex XSOAR.Classification and Mapping

    3. Pre-processing: apply automations to incidents before they are ingested to eliminate false-positives and duplicate incidents.Create Pre-Process Rules for IncidentsIncident De-Duplication

    4. Steps an incident goes through.

    5. Analyst intervention in the incident (communication tasks, ad-hoc tasks, and manual tasks).

    6. How data is displayed in Cortex XSOAR.

    7. What happens when an investigation is remediated, including post-processing.Post Processing for Incidents

Incident Lifecycle using Splunk

Using Cortex XSOAR, you can define integrations with your security products, your security services and any other component that makes up your security operations environment. You can then ingest events from these integrations and transform them into incidents in Cortex XSOAR.

After the incidents are created, you can run playbooks on these incidents to enrich them with information from other products and services, which helps you complete the picture. Usually, you can use automation to determine if an incident requires further investigation or can be closed based on the findings. This enables your analysts to focus on the minority of incidents that require further investigation.

  • Analyze SOC’s current workflows and categorize the incident types you expect to handle.

  • Plan what the process should be per each type of incident (fully automated or manual or hybrid).

  • Put your plan in action, by defining the following per each incident type:

    • What incidents are going to be included in this flow (classification).

    • In many cases you might want to eliminate false positives and duplicate incidents early (pre-processing).

    • What steps each incident should go through (playbook).

    • When analyst intervention is required, if at all (also in the playbook, and dashboards).

    • How the incident data is going to be presented (mapping).

    • How the incident details would be presented to the analyst (layouts, dashboards).

    • What happens when the incident is over (for example, close incident and force or encourage analyst to specify close reason).

Prerequisites

This tutorial assumes you already have components installed and configured in your Cortex XSOAR instance, as workflows require their functionality. For example, a playbook task might send an email notification to users, but if you do not have an outgoing email integration configured, you cannot complete the playbook workflow.

  1. Install the Cortex XSOAR server and verify that it is running.

  2. Configure the following integrations by installing the required Content Packs in the Marketplace:

    To install and configure the Splunk Content Pack, see Set up your Splunk integration instance.

    Integration

    Name

    Messaging integration

    Such as Gmail or Microsoft Graph Mail

    Outgoing email integration

    Such as Mail Sender

    Provision Cortex XSOAR users

    Such as Active Directory or Okta

    Data enrichment and threat intelligence

    Such as AutoFocus or Wildfire

Set up your Splunk integration instance

In Cortex XSOAR you can ingest Splunk notable events, queries and alerts through one of the following:

  • Demisto add-on for Splunk from the Splunkbase.

    The Demisto add-on for Splunk works inside the Splunk platform. It is installed on Splunk and it is used to establish a connection between Splunk and Cortex XSOAR. You create an alert or event by selecting Create Demisto Event. When sending a single alert or event, it needs to be configured every time in Cortex XSOAR.

  • SplunkPy

  • Enables you to run queries on Splunk, edit notable events, fetch results from Splunk, parse raw events, etc. This is setup on Splunk’s API port, so you have control over which events can be ingested into Cortex XSOAR. You do not need to go to Splunk each time you ingest. Everything is configured when you map the fields in Cortex XSOAR. This tutorial uses the SplunkPy integration.

To set up your Splunk integration you need to install the Splunk Content Pack from the Marketplace. The integration enables you to do the following:

  • Splunk Notable Queries

    In Cortex XSOAR, the query fetches notable events from Splunk Enterprise Security (ES). The integration uses the Splunk ES Notable macro, which leverages built-in Splunk ES capabilities that provide additional data with every notable fetched event.

  • Native enrichment

    You can enable relevant enrichment notable events, when defining the integration instance. The fetched notable events can contain the results of drilldown searches. Splunk searches are configured by the user within the Splunk alerts. Users can run additional searches, so those search results are returned in Cortex XSOAR incidents.

    When fetching incidents, data may be returned from the look-up table, which contains data about the assets and identities that are detected within the notable event. This means you do not need to run additional playbooks or view additional commands. This is all part of the initial fetch.

  • Mirroring Capabilities

    By enabling the mirroring option (can be inbound, outbound, or both), selected fields can be mirrored from Splunk ES to Cortex XSOAR and vice versa. Some specific fields are supported from Splunk to Cortex XSOAR, such as owner. Urgency and status can be mirrored in in both directions. If a Splunk user closes Notable Events this effectively closes the incident in Cortex XSOAR. You can also do this on Cortex XSOAR, which closes the Splunk side.

    Outbound mirroring is recommended for Cortex version 6.2 and above.

    You can use the following Cortex XSOAR command to update the status of an incident in Splunk:

    !splunk-notable-event-edit status=

    Possible values are: 0 - Unassigned, 1 - Assigned, 2 - In Progress, 3 - Pending, 4 - Resolved, 5 - Closed.

  • Onboarding

    The Content Pack includes mappers, so that you can see all relevant fields that exist on the notable events, assets, identities, and on the drill down events. All of this data is easily mapped into incident fields in Cortex XSOAR.

    In addition the Content Pack contains a Splunk playbook, which enables you to manage cases, assist with adjusting your alerts.

  • Searching

    You can search for things in Splunk using the following Cortex XSOAR command and by creating the query using the Splunk Search Processing Language (SPL):

    !splunk-search query

Before you begin, ensure that you have your Splunk credentials.

  1. Install the Splunk Content Pack.

    1. Go to the Marketplace.

    2. Search for Splunk.

    3. Click the Content Pack and click Install.

    4. Click Install again to confirm the installation.

  2. Configure the SplunkPy instance.

    Before you configure the instance:

    • Ensure you have your Splunk authentication details.

    • Define your email sender integration and the SIEM admin email address.

    1. Select SettingsINTEGRATIONSInstancesSplunkPy.

    2. Click Add Instance.

    3. Select Fetches incidents.

    4. Under Classifier, select N/A.

    5. Under Incident Type, select Splunk Notable Generic.

      You do not need to specify the classifier as all Splunk incidents are ingested as Splunk Notable Generic. As you become more familiar with Cortex XSOAR, you can create custom incident types as needed instead of using the Splunk Notable Generic incident type.

    6. Under Mapper (incoming), select Splunk - Notable Generic Incoming Mapper.

    7. Under Mapper (outgoing), select Splunk - Notable Generic Outgoing Mapper.

      splunk-integration.png
    8. Type the Host -ip, username, password, and Port.

    9. Keep the Get notable events EQ query as is, as we use the Notable macro when ingesting events. You can create a more granular search by specifying specific conditions such as specific security domain, event ID, etc.

    10. Keep the defaults for fetch limit, first fetch timestamp, earliest time to fetch and latest time to fetch.

    11. To add mirroring in both environments, in the Incident Mirroring Direction field, select Incoming and Outgoing.

      Outgoing mirroring is recommended for Cortex XSOAR version 6.2 and above. If you enable mirroring, you need to add the time zone of the Splunk server (in minutes). For example, if using GMT and the time zone is GMT +3 hours, set the time zone to +180. For UTC, set the time zone to 0. Set this only if the Splunk server is different than the Cortex XSOAR server. This is relevant only for fetching notable events.

    12. Select Close Mirrored XSOAR Incident and Close Mirrored Splunk Notable Event, so when closing in one environment, it closes in the other.

    13. In the Enrichment Types field, select Asset, Drilldown and Identity.

      This enrichment provides additional information about Assets, drilldown, and identities that are related to the notable events you ingest.

    14. Click Test and then Save & exit.

      splunk-int-success.png
  3. Go to the Incidents page and verify that incidents are being ingested.

    splunk-incidents.png

Run a Playbook

After you define the instance and start ingesting incidents, the Spunk Generic default playbook assigned to the Splunk incident type automatically processes the incoming incidents. No additional steps are required from the analyst to initiate the playbook run/incident-handling process.

You can change the default playbook by selecting SettingsOBJECTS SETUP IncidentsTypesSplunk Notable GenericEdit. You need to either duplicate or detach the incident type to edit.

Playbook Inputs

Every playbook has inputs that help determine how the playbook flows. The inputs (and outputs) are configured at the beginning of the playbook, under Playbook Triggered.

For a full description of each of the inputs for this playbook, refer to the playbook documentation. For purposes of this tutorial, we focus on the following Splunk Generic playbook inputs:

  • Enrich: Determines whether you want the playbook to enrich all of the indicators in the incident (default is true). As enrichment can be a very resource intensive operation, you may want to change this setting to false and enrich only specific indicators and only using certain integrations.

    If you do want to enrich indicators, you should ensure that you have enabled at least some of the out-of-the-box enrichment integrations, such as AutoFocus, VirusTotal, etc.

  • UseCalculateSeverity: Determines if the severity of the incident is set based on the Calculate Severity playbook, or if you want to set the severity based on the Splunk severity value. By default, this input is set to true.

Enrich Indicators

In the playbook inputs, if you select to enrich all of the indicators from the incident, the playbook extracts the indicators and uses whichever integrations you have enabled for the enrichment. Enriched indicators provide the analyst with more information about each indicator.

The Entity Enrichment - Generic v2 playbook checks for information about all kinds of indicators, including malicious URLs, domains, or IP addresses. It can also check against a list of VIP assets in your organization that might have been targeted, or detonate a file that was attached to the incident.

While these capabilities are available in the playbook, they are only available if you have enabled or installed integrations for these checks, such as VirusTotal, Threat Crowd, etc. For information about enabling additional integrations, refer to the integration documentation.

If you also configured the playbook to use the Calculate Severity playbook, the playbook takes the information from the enriched indicators and provides a severity based on the DBot score.

Manual Investigation

After the playbook gathers all of the necessary an SLA timer is triggered and the incident is assigned to an analyst for manual investigation. The manual investigation task requires that the analyst intervene to determine if the incident is a true positive or false positive.

True Positive

If the incident is a true positive, an incident report is generated and the necessary remediation steps are taken.

False Positive

If the incident is a false positive, the analyst can use the Provide data for rule adjustment task (Questions tab) to provide information that can help fine tune the system for future cases.

tutorial-siem-which-rules.png

The feedback from the analyst is sent to the Splunk administrator so they can review the suggested changes and make any necessary adjustments.

Analyze Data

After events are fetched into Cortex XSOAR, the notable events appear as incidents in the Incidents page. All the information that is displayed in the respective tabs is determined by the layout you designed for the Splunk Notable Generic incident type.

The layout appearance, including which tabs, their order, names, etc., is customizable.Customize Incident Layouts

The analyst can click on the ID of any incident to view additional incident data that you configured for the incident type.

Incident Info View

The Incident Info tab provides information about the incident itself. For example, when was the incident opened and updated, what’s the severity of the incident and from where did it originate. It gives you easy access to all of the indicators extracted from the incident, what is the current state of the incident, etc.

For example, in the Work Plan section, you can see that the incident is currently on hold waiting for the analyst to manually investigate the incident.

splunk-incident.png

Notable Summary

The Notable Summary tab fetches the data from Splunk and provides it to you in one tab within Cortex XSOAR. For example, you can see how many of the events associated with the event were fetched (default is 50), the Splunk severity, status, urgency, etc.

splunk-summary.png

The view includes all of the network data that is available from Splunk.

Drilldown

The Drilldown tab, shows detailed drilldown data that was enriched from the Splunk Server, such as Error code, event code, event ID, etc.

splunk-drilldown.png

Assets

The Assets tab presents information about any critical assets that were involved in the event, whether they were targeted or tangentially related.

splunk-assets.png

Identities

The Identities tab shows enriched identity information that was involved in the event, such as the source email address, identity tag, etc.

splunk-ids.png