Ingest Incidents from a SIEM Using Splunk

Step-by-step tutorial for ingesting and handling incidents and events from Splunk.
This tutorial shows you how to design the incident lifecycle using Splunk, from ingesting to closing security events.
The incident lifecycle is designed for Cortex XSOAR (SOC) engineers or architects. The aim is to automate as much of the process as possible and leave the analyst to make accurate and confident decisions when needed.
By the end of this tutorial, you should have configured your Splunk integration, set up a basic flow, started ingesting incidents from the Splunk to Cortex XSOAR, which enables the analyst to start investigating an incident.
This tutorial includes the following stages.

Analyst Flow

Depending on how you design the incident lifecycle, many actions are performed automatically even before the analyst investigates an incident. For example, incidents may be closed as duplicates, closed as false-positives, data is enriched, etc.
Usually, the analyst does the following:
  • Selects an open incident.
  • Analyzes enough data to make decisions based on the steps that have already been taken (in the Incident Summary, and Work Plan, etc).
  • Collects additional data by running commands on third-party integrations, such as VirusTotal, WildFire, etc.
  • Collaborates with SOC team members in the War Room (ask questions, post comments, etc).
  • Decides how to handle the incident. Usually the architect designs the Incident page to include a set of buttons that the analyst can select from (Escalate, Close, etc.)
  • Activates one of several possible responses either within Cortex XSOAR or externally (for example, open a ticket for IT).
  • Closes the incident and specify a closed reason.

Architect Flow

To explain the architect flow we need to define and design the incident lifecycle using Splunk.
Define the incident Type
The first step when planning your incident identification and response is to create a basic outline of the flows that your SOC handles. Consider the same information in the Architect Flow in using QRadar.
Plan the workflow for each incident type
This section describes the aspects of the workflow that occur before the analyst is assigned.
  • Assign incidents to the flow
    In Cortex XSOAR, incident classification is the process by which an incoming security event/alert is assigned to a Cortex XSOAR incident type. When you fetch security events from Splunk, the event data is ingested in JSON format. You classify the security event based on the fields (keys) from the event JSON.
    Install the
    Splunk
    Content Pack, which includes Splunk Classifiers, although you do not need to define a classifier, as all Splunk incidents are ingested as the
    Splunk Notable Generic
    incident type (when you assign incident type to the integration in Cortex XSOAR).
  • Pre-process incoming incidents
    Usually you want to eliminate false-positives and deduplicate incidents, which enables you to maintain a clean set of incidents for analysts.
    Pre-processing rules enable you to perform certain actions on security events before they are ingested as incidents in Cortex XSOAR. You perform actions when the condition of a rule is met. For example, link the incoming incident to an existing incident, or under configured conditions, drop the incoming incident altogether. You can also run a script on the incident when the conditions are met. For an example, see the Phishing Pre-Process Tutorial.
  • Run a playbook on the incident
    After the security event is ingested, classified to an Cortex XSOAR incident type, and filtered through the pre-processing rules, the default playbook for the incident type is run on the incident. Usually, the default playbook extracts one or a combination of the following:
    • Extracts and enriches indicators
    • Determines the incident severity
    • Assigns the incident to an analyst for manual review
    Depending on how you design the workflow for the incident type, when the playbook completes its run, it is the first time that the incident data is presented to analysts in the layout. Although Splunk comes out of the box with the
    SplunkPy
    playbook, consider the following when customizing or creating your own playbook.
    1. Plan the incident process per incident type (full-automated, manual, or hybrid).
    2. Define how incidents are classified (assigned to an incident type) in Cortex XSOAR.
    3. Pre-processing: apply automations to incidents before they are ingested to eliminate false-positives and duplicate incidents.
    4. Steps an incident goes through.
    5. Analyst intervention in the incident (communication tasks, ad-hoc tasks, and manual tasks).
    6. How data is displayed in Cortex XSOAR.
    7. What happens when an investigation is remediated, including post-processing.

Incident Lifecycle using Splunk

Using Cortex XSOAR, you can define integrations with your security products, your security services and any other component that makes up your security operations environment. You can then ingest events from these integrations and transform them into incidents in Cortex XSOAR.
After the incidents are created, you can run playbooks on these incidents to enrich them with information from other products and services, which helps you complete the picture. Usually, you can use automation to determine if an incident requires further investigation or can be closed based on the findings. This enables your analysts to focus on the minority of incidents that require further investigation.
  • Analyze SOC’s current workflows and categorize the incident types you expect to handle.
  • Plan what the process should be per each type of incident (fully automated or manual or hybrid).
  • Put your plan in action, by defining the following per each incident type:
    • What incidents are going to be included in this flow (classification).
    • In many cases you might want to eliminate false positives and duplicate incidents early (preprocessing).
    • What steps each incident should go through (playbook).
    • When analyst intervention is required, if at all (also in the playbook, and dashboards).
    • How the incident data is going to be presented (mapping).
    • How the incident details would be presented to the analyst (layouts, dashboards).
    • What happens when the incident is over (for example, close incident and force or encourage analyst to specify close reason).

Prerequisites

This tutorial assumes you already have components installed and configured in your Cortex XSOAR instance, as workflows require their functionality. For example, a playbook task might send an email notification to users, but if you do not have an outgoing email integration configured, you cannot complete the playbook workflow.
  1. Install the Cortex XSOAR server and verify that it is running.
  2. Configure the following integrations by installing the required Content Packs in the Marketplace:
    To install and configure the Splunk Content Pack, see Set up your Splunk integration instance.
    Integration
    Name
    Messaging integration
    Outgoing email integration
    Such as Mail Sender
    Provision Cortex XSOAR users
    Data enrichment and threat intelligence
    Such as AutoFocus or Wildfire

Set up your Splunk integration instance

In Cortex XSOAR you can ingest Splunk notable events, queries and alerts through one of the following:
  • Demisto Add on for Splunk from the Splunkbase.
    The Demisto Add on for Splunk works inside the Splunk platform. It is installed on Splunk and it is used to establish a connection between Splunk and Cortex XSOAR. You create an alert or event by selecting a
    Create Demisto Event
    . When sending a single alert or event, it needs to be configured every time in Cortex XSOAR.
  • SplunkPy
  • Enables you to run queries on Splunk, edit notable events, fetch results from Splunk, parse raw events, etc. This is setup on Splunk’s API port, so you have control over which events can be ingested into Cortex XSOAR. You do not need to go to Splunk each time you ingest. Everything is configured when you map the fields in Cortex XSOAR. This tutorial uses the SplunkPy integration.
To set up your Splunk integration you need to install the Splunk Content Pack from the Marketplace. The integration enables you to do the following:
  • Splunk Notable Queries
    In Cortex XSOAR, the query fetches notable events from Splunk Enterprise Security (ES). The integration uses the Splunk ES Notable macro, which leverages built-in Splunk ES capabilities that provide additional data with every notable fetched event.
  • Native enrichment
    You can enable relevant enrichment notable events, when defining the integration instance. The fetched notable events can contain the results of
    drilldown
    searches. Splunk searches are configured by the user within the Splunk alerts. Users can run additional searches, so those search results are returned in Cortex XSOAR incidents.
    When fetching incidents, data may be returned from the look-up table, which contains data about the assets and identities that are detected within the notable event. This means you do not need to run additional playbooks or view additional commands. This is all part of the initial fetch.
  • Mirroring Capabilities
    By enabling the mirroring option (can be inbound, outbound, or both), selected fields can be mirrored from Spluk ES to Cortex XSOAR and vice versa. Some specific fields are supported from Splunk to Cortex xSOAR, such as owner. Urgency and status can be mirrored in both directions. If a Splunk user closes Notable Events this effectively closes the incident in Cortex XSOAR. You can also do this on Cortex XSOAR, which closes the Splunk side.
    Outbound mirroring is recommended for Cortex version 6.2 and above.
  • Onboarding
    The Content Pack includes mappers, so that you can see all relevant fields that exist on the notable events, assets, identities, and on the drill down events. All of this data is easily mapped into incident fields in Cortex XSOAR.
    In addition the Content Pack contains a Splunk playbook, which enables you to manage cases, assist with adjusting your alerts.
Before you begin, ensure that you have your Splunk credentials.
  1. Install the Splunk Content Pack.
    1. Go to the Marketplace.
    2. Search for Splunk.
    3. Click on the Content Pack and click
      Install
      .
    4. Click
      Install
      again to confirm the installation.
  2. Configure the SplunkPy instance.
    Before you configure the instance:
    • Ensure you have your Splunk authentication details.
    • Define your email sender integration and the SIEM admin email address.
    1. Select
      Settings
      Integrations
      Servers & Services
      SplunkPy
      .
    2. Click
      Add Instance
      .
    3. Select
      Fetches incidents
      .
    4. Under
      Classifier
      , select
      N/A
      .
    5. Under
      Incident Type
      , select
      Splunk Notable Generic
      .
      You do not need to specify the classifier as all Splunk incidents are ingested as Splunk Notable Generic. As you become more familiar with Cortex XSOAR, you can create custom incident types as needed instead of using the Splunk Notable Generic incident type.
    6. Under
      Mapper (incoming)
      , select
      Splunk - Notable Generic Incoming Mapper
      .
    7. Under
      Mapper (outgoing)
      , select
      Splunk - Notable Generic Outgoing Mapper
      .
    8. Type the Host -ip, username, password, and Port.
    9. Keep the
      Get notable events EQ query
      as is, as we use the Notable macro when ingesting events. You can create a more granular search by specifying specific conditions such as specific security domain, event ID, etc.
    10. Keep the defaults for fetch limit, first fetch timestamp, earliest time to fetch and latest time to fetch.
    11. To add mirroring in both environments, in the
      Incident Mirroring Direction
      field, select
      Incoming and Outgoing
      .
      Outgoing miroriring is recommended for Cortex XSOAR version 6.2 and above. If you enable mirroring, you need to add the timezone of the splunk server (in minutes). For example, if using GMT and the timezone is GMT +3 hours, set the timezone to +180. For UTC, set the timezone to 0. Set this only if the Splunk server is different than the Cortex XSOAR server. This is relevant only for fetching notable events.
    12. Select
      Close Mirrored XSOAR Incident
      and
      Close Mirrored Splunk Notable Event
      , so when closing in one environment, it closes in the other.
    13. In the
      Enrichment Types
      field, select
      Asset
      ,
      Drilldown
      and
      Identity
      .
      This enrichment provides additional information about Assets, drilldown, and identities that are related to the notable events you ingest.
    14. Click
      Test
      and then
      Save & exit
      .
  3. Go to the
    Incidents
    page and verify that incidents are being ingested.

Run a Playbook

After you define the instance and start ingesting incidents, the
Spunk Generic
default playbook assigned to the Splunk incident type automatically processes the incoming incidents. No additional steps are required from the analyst to initiate the playbook run/incident-handling process.
You can change the default playbook by selecting
Settings
Incident Types
Splunk Notable Generic
Edit
. You need to either duplicate or detach the incident type to edit.
Playbook Inputs
Every playbook has inputs that help determine how the playbook flows. The inputs (and outputs) are configured at the beginning of the playbook, under
Playbook Triggered
.
For a full description of each of the inputs for this playbook, refer to the playbook documentation. For purposes of this tutorial, we focus on the following Splunk Generic playbook inputs:
  • Enrich
    : Determines whether you want the playbook to enrich all of the indicators in the incident (default is true). As enrichment can be a very resource intensive operation, you may want to change this setting to false and enrich only specific indicators and only using certain integrations.
    If you do want to enrich indicators, you should ensure that you have enabled at least some of the out-of-the-box enrichment integrations, such as AutoFocus, VirusTotal, etc.
  • UseCalculateSeverity
    : Determines if the severity of the incident is set based on the Calculate Severity playbook, or if you want to set the severity based on the Splunk severity value. By default, this input is set to true.
Enrich Indicators
In the playbook inputs, if you select to enrich all of the indicators from the incident, the playbook extracts the indicators and uses whichever integrations you have enabled for the enrichment. Enriched indicators provide the analyst with more information about each indicator.
The
Entity Enrichment - Generic v2
playbook checks for information about all kinds of indicators, including malicious URLs, domains, or IP addresses. It can also check against a list of VIP assets in your organization that might have been targeted, or detonate a file that was attached to the incident.
While these capabilities are available in the playbook, they are only available if you have enabled or installed integrations for these checks, such as VirusTotal, Threat Crowd, etc. For information about enabling additional integrations, refer to the integration documentation.
If you also configured the playbook to use the Calculate Severity playbook, the playbook takes the information from the enriched indicators and provides a severity based on the DBot score.
Manual Investigation
After the playbook gathers all of the necessary an SLA timer is triggered and the incident is assigned to an analyst for manual investigation. The manual investigation task requires that the analyst intervene to determine if the incident is a true positive or false positive.
True Positive
If the incident is a true positive, an incident report is generated and the necessary remediation steps are taken.
False Positive
If the incident is a false positive, the analyst can use the
Provide data for rule adjustment
task (
Questions
tab) to provide information that can help fine-tune the system for future cases.
The feedback from the analyst is sent to the Splunk administrator so they can review the suggested changes and make any necessary adjustments.

Analyze Data

After events are fetched into Cortex XSOAR, the notable events appear as incidents in the Incidents page. All the information that is displayed in the respective tabs is determined by the layout you designed for the Splunk Notable Generic incident type.
The layout appearance, including which tabs, their order, names, etc., is customizable.
The analyst can click on the ID of any incident to view additional incident data that you configured for the incident type.
Incident Info View
The
Incident Info
tab provides information about the incident itself. For example, when was the incident opened and updated, what’s the severity of the incident and from where did it originate. It gives you easy access to all of the indicators extracted from the incident, what is the current state of the incident, etc.
For example, in the
Work Plan
section, you can see that the incident is currently on hold waiting for the analyst to manually investigate the incident.
Notable Summary
The Notable Summary tab fetches the data from Splunk and provides it to you in one tab within Cortex XSOAR. For example, you can see how many of the events associated with the event were fetched (default is 50), the Splunk severity, status, urgency, etc.
The view includes all of the network data that is available from Splunk.
Drilldown
The
Drilldown
tab, shows detailed drilldown data that was enriched from the Splunk Server, such as Error code, event code, event ID, etc.
Assets
The
Assets
tab presents information about any critical assets that were involved in the event, whether they were targeted or tangentially related.
Identities
The
Identities
tab shows enriched identity information that was involved in the event, such as the source email address, identity tag, etc.

Recommended For You