INGEST

Understanding how to write a [INGEST] section in a Parsing Rules file and the syntax to use.
An
INGEST
section is used to define the resulting Parsing Rule. The
CONST
and
RULE
sections are only add-ons, used to help organize the
INGEST
sections, and are optional to configure. Yet, a Parsing Rules file that contains no
INGEST
sections, generates no Parsing Rules, and is mandatory to configure.
INGEST
syntax is derived from XQL with a few modifications as explained in the Parsing Rules syntax. In addition,
INGEST
sections contain the following syntax add-ons.
  • INGEST
    sections can have more than one XQLp statement, separated by a semicolon (
    ;
    ). Each statement creates a different Parsing Rule.
  • Another new stage is available called
    drop
    .
    • drop
      takes a condition similar to the XQL
      filter
      stage (same syntax), but drops every log entry that passes that condition. One can think of it as a negative filter, so
      drop <condition>
      is not equivalent to
      filter not <condition>
      .
    • drop
      can only appear last in a statement. No other XQLp rules can follow.
  • INGEST
    sections take parameters, and not names as
    RULE
    sections use, where some are mandatory and others optional.
    [ingest:vendor=<vendor>, product=<product>, dataset=<dataset>, no_hit=<keep\drop>] filter raw_log not contains "alert";
    Code copied to clipboard
    Unable to copy due to lack of browser support.
The parameter descriptions are explained in the following table.
Parameter
Description
vendor
The vendor that the specified Parsing Rules apply to (mandatory).
product
The product that the specified Parsing Rules apply to (mandatory).
dataset
The name of the dataset to insert every row with the results after applying any of the specified Parsing Rules (mandatory).
no_hit
No-match strategy to use for the entire specified group of rules (optional). The default is
keep
.
  • If
    no_hit = drop
    , then in a scenario where none of the rules in the group generates output for a given log record, that record is discarded.
  • If
    no_hit = keep
    , then in a scenario where none of the rules in the group generates output for a given log record, that record is kept in the
    _raw_log
    field. This record is inserted into the group's dataset once, but every column holds
    NULL
    except for
    _raw_log
    which holds the original JSON log record.
Each statement represents a different Parsing Rule in the same group as depicted in the following example.
[CONST] DEVICE_NAME = "ngfw"; [rule:use_two_rules] filter severity = "medium" | call basic_rule | call use_xql_and_another_rule; [rule:basic_rule] fields log_type, severity | filter log_type="eal" and severity="HIGH" and type="something"; [rule:use_xql_and_another_rule]call multiline_statement | filter severity = "medium"; [rule:multiline_statement] alter url = json_extract(_raw_log, "$.url") | join type = inner conflict_strategy = both (dataset=my_lookup) as inn url=inn.url |filter severity = "medium"; [ingest:vendor=panw, product=ngfw, dataset=panw_ngfw_ds, no_hit=drop] filter log_type="traffic" | alter url = json_extract(_raw_log, "$.url"); call use_two_rules | join type = inner conflict_strategy = both (dataset=my_lookup) as inn severity=inn.severity | fields severity, log_type | drop device_name = $DEVICE_NAME;
Code copied to clipboard
Unable to copy due to lack of browser support.
This generates 1 group of 2 Parsing Rules for panw/ngfw, where all the ingested data into
panw_ngfw_ds
dataset.
The following represents the syntax for the rules.
Rule #1: filter log_type="traffic" | alter url = json_extract(_raw_log, "$.url"); Rule #2: filter severity = "medium" | fields log_type, severity | filter log_type="eal" and severity="HIGH" and type="something" | alter url = json_extract(_raw_log, "$.url") | join type = inner conflict_strategy = both (dataset=my_lookup) as inn url=inn.url | filter severity = "medium" | filter severity = "medium" | join type = inner conflict_strategy = both (dataset=my_lookup) as inn severity=inn.severity | fields severity, log_type | drop device_name = $DEVICE_NAME
Code copied to clipboard
Unable to copy due to lack of browser support.
A few more points to keep in mind when writing
INGEST
sections.
  • INGEST
    parameter names are not case sensitive. Therefore,
    vendor=PANW
    and
    vendor=panw
    are the same.
  • Since section order is unimportant, you do not have to declare a
    RULE
    or a
    CONST
    before using it in an
    INGEST
    section.
  • You can have multiple
    INGEST
    sections with the same
    vendor
    ,
    product
    ,
    dataset
    and
    no_hit
    values. Yet, this can lead to unexpected results. Consider the following example:
    [ingest:vendor=panw, product=ngfw, dataset=panw_ngfw_ds, no_hit=keep] filter raw_log not contains "alert"; [ingest:vendor=panw, product=ngfw, dataset=panw_ngfw_ds, no_hit=keep] filter device_type not contains "agent";
    Code copied to clipboard
    Unable to copy due to lack of browser support.
    Let
    lw
    be a log row. If
    lw.raw_log
    contains an
    alert
    and
    lw.device_type
    contains an
    agent
    , then
    lw
    is inserted twice into the
    pan_ngfw_ds
    dataset as every section is standalone.
    • To eliminate these kind of errors and misunderstandings, it is highly advised to group all rules having the same
      vendor
      ,
      product
      ,
      dataset
      and
      no_hit
      values in a single
      INGEST
      section.
    • Logs that were discarded by a
      drop
      stage are considered ingested with a no-match policy. This means they are not kept even if
      no_hit = keep
      .
    • Keep in mind that all rules inside a group get evaluated independently. This is in contrast to firewall-like rules, which stop evaluating at the first rule that is able to make a decision. Therefore, without proper filtering, it is possible to ingest the same log more than once.
  • You can override the default raw dataset in
    INGEST
    sections. For more information, see Parsing Rules Raw Dataset.
  • Cortex XDR supports configuring case sensitivity in Parsing Rules only within the INGEST section using the following configuration stage:
    config case_sensitive = true | false
    Code copied to clipboard
    Unable to copy due to lack of browser support.
  • You can add a single tag
    or list of tags
    to the ingested data as part of the ingestion flow that you can easily query in XQL Search. You can add tags as part of the
    INGEST
    section or using both the
    INGEST
    and
    RULE
    sections. The following are examples of each.
    • INGEST
      section.
      Adding a single tag.
      [INGEST:vendor="MSFT", product="Azure AD Audit", target_dataset="msft_ad_audit_tagging", no_hit=drop, ingestnull = false ] tag add "New Event"
      Code copied to clipboard
      Unable to copy due to lack of browser support.
      Adding a list of tags.
      [INGEST:vendor="MSFT", product="Azure AD Audit", target_dataset="msft_ad_audit_tagging", no_hit=drop, ingestnull = false ] tag add "New Event1", "New Event2", "New Event3"
      Code copied to clipboard
      Unable to copy due to lack of browser support.
    • INGEST
      and
      RULE
      sections.
      Adding a single tag.
      [INGEST:vendor="Check Point", product="Anti Malware", target_dataset="malware_test", no_hit= drop , ingestnull = true ] alter xx = call new_tag_rule;
      Code copied to clipboard
      Unable to copy due to lack of browser support.
      [RULE:new_tag_rule] tag add "test";
      Code copied to clipboard
      Unable to copy due to lack of browser support.
      Adding a list of tags.
      [INGEST:vendor="Check Point", product="Anti Malware", target_dataset="malware_test", no_hit= drop , ingestnull = true ] alter xx = call new_tag_rule;
      Code copied to clipboard
      Unable to copy due to lack of browser support.
      [RULE:new_tag_rule] tag add "test1", "test2", "test3";
      Code copied to clipboard
      Unable to copy due to lack of browser support.

Recommended For You