Activate the FTP Collector

Activating a broker VM with a FTP Collector applet enables you to monitor and collect logs from files and folders via FTP, FTPS, and SFTP to your log repository.
Ingesting logs and data from external sources requires a Cortex® XDR™ Pro per TB license.
The broker VM provides a FTP Collector applet that enables you to monitor and collect logs from files and folders via FTP, FTPS, and SFTP directly to your log repository for query and visualization purposes. A maximum file size of 500 MB is supported. After you activate the FTP Collector applet on a broker VM in your network, you can collect files as datasets (
<Vendor>_<Product>_raw
) by defining the following.
  • FTP, FTPS, or SFTP (default) connection details with the path to the folder containing the files that you want to monitor and upload to Cortex XDR.
  • Settings related to the list of files to monitor and upload to Cortex XDR, where the log format is either JSON, CSV, or Raw. Once the files are uploaded to Cortex XDR, you can define whether in the source directory the files are renamed or deleted.
Complete the following tasks before you begin setting up the FTP Collector applet.
  • Ensure that the user permissions for the FTP, SFTP, or FTPS include the ability to rename and delete files in the folder that you want to configure collection.
  • When setting up an FTPS Collector with a server using a Self-signed certificate, you must upload the certificate first to the broker VM as a Trusted CA certificate.
Activate the FTP Collector.
  1. In Cortex XDR, select
    Settings ( )
    Configurations
    Broker VM
    and locate your broker VM.
  2. Right-click the broker VM and select
    FTP Collector
    Activate
    .
  3. Configure the
    FTP Connection
    settings.
    1. Configure the
      FTP Connection
      settings.
      • Type
        —Select the type of FTP connection as
        FTP
        ,
        SFTP
        , or
        FTPS
        .
      • Host
        —Specify the hostname, IP address, or FQDN of the FTP server. When configuring a
        FTPS
        Collector, you must specify the FQDN.
      • Port
        —Specify the FTP port number.
      • Username
        —Specify the username to login to the FTP server.
      • Password
        —Specify the password to login to the FTP server.
      • SSH Key-Based Authentication
        —This checkbox is only displayed when setting a
        SFTP
        Collector, which works with both
        Username
        and
        Password
        authentication or
        SSH Key-Based Authentication
        . You can either leave this checkbox clear and set a
        Username
        and
        Password
        (default) or select
        SSH Key-Based Authentication
        to
        Browse
        to a
        Private Key
        . When this connection is established with a server using a Self-signed certificate, you must upload it first to the broker VM as a Trusted CA Certificate.
        When configuring an SFTP connection, Cortex XDR expects the private key to be in the RSA format that is included in the
        -----BEGIN RSA PRIVATE KEY-----
        tag. Cortex XDR does not support providing the private key in the OpenSSH format from the
        -----BEGIN OPENSSH PRIVATE KEY-----
        tag.
        When using
        ssh-keygen
        using a Mac, you get the OpenSSH format by default. The command for getting the RSA format is:
        ssh-keygen -t rsa -b 4096 -C <email address> -m PEM
      • Folder Path
        —Specify the path to the folder on the FTP site where the files are located that you want to collect.
      • Recursive
        —Select this checkbox to configure the FTP Collector applet to recursively examine any subfolders for new files as long as the folders are readable. This is not configured by default.
      • Test Connection
        —Select to validate the FTP connection.
    2. Configure the
      FTP Settings
      .
      • Collect Every
        —Specify the execution frequency of collection by designating a number and then selecting the unit as either
        Minutes
        ,
        Hours
        , or
        Days
        .
      • After Files Uploaded
        —Select what to do with the files after they are uploaded to the Cortex XDR server. You can either select
        Rename files with a suffix
        (default) and then you must specify the
        Suffix
        or
        Delete files
        . When adding a suffix, the suffix is added at the end of the original file name using the format
        <file name>.<suffix>
        , which becomes the new name of the file.
      • Include
        —Specify the files and folders that must match to be monitored by Cortex XDR. Multiple values are allowed with commas separating the values.
        Allowed wildcard:
        • '?' matches a single alphabet character in a specific position.
        • '*' matches any character or set of characters, including no character.
        Example:
        log*.json
        includes any JSON file starting with 'log'.
      • Exclude
        —(
        Optional
        ) Specify the files and folders that must match to not be monitored by Cortex XDR. Multiple values are allowed with commas separating the values.
        Allowed wildcard:
        • '?' matches a single alphabet character in a specific position.
        • '*' matches any character or set of characters, including no character.
        Example:
        *.backup
        excludes any file ending with '.backup'.
      • Log Format
        —Select the
        Log Format
        from the list as either
        Raw
        (default),
        JSON
        ,
        CSV
        ,
        TSV
        , or
        PSV
        , which indicates to Cortex XDR how to parse the data in the file. This setting defines the parser used to pars all the processed files as defined in the
        Include
        and
        Exclude
        fields, regardless of the file names and extension. For example, if the
        Include
        field is set
        *
        and the
        Log Format
        is
        JSON
        , all files (even those named
        file.log
        ) in the specified folder are processed by the FTP Collector as JSON, and any entry that does not comply with the JSON format are dropped.
        When uploading
        JSON
        files, Cortex XDR only parses the first level of nesting and only supports single line JSON format, such that every new line means a separate entry.
      • # of Lines to Skip
        —(
        Optional
        ) Specify the number of lines to skip at the beginning of the file. This is set to 0 by default.
    3. Configure the
      Data Source Mapping
      .
      Vendor
      and
      Product
      —Specify the
      Vendor
      and
      Product
      for the type of data being collected. The vendor and product are used to define the name of your XQL dataset (
      <Vendor>_<Product>_raw
      ).
    4. Generate Preview
      .
      Select
      Generate Preview
      to display up to 10 rows from the first file and
      Preview
      the results. The
      Preview
      works based on the FTP Collector settings, which means that if all the files that were configured to be monitored were already processed, then the
      Preview
      returns no records.
  4. (
    Optional
    )
    Add Connection
    to define another FTP connection for collecting logs from files and folders via FTP, FTPS, or SFTP.
  5. (
    Optional
    ) Other available options.
    As needed, you can return to your FTP Collector settings to manage your connections. Here are the actions available to you.
    • Edit the connection name by hovering over the default
      Collection
      name, and selecting the edit icon to edit the text.
    • Disable
      /
      Enable
      a connection by hovering over the top area of the connection section, on the opposite side of the connection name, and selecting the applicable button.
    • Delete a connection by hovering over the top area of the connection section, on the opposite side of the connection name, and selecting the delete icon. You can only delete a connection when you have more than one connection configured. Otherwise, this icon is not displayed.
  6. Activate
    the FTP Collector applet.
    After a successful activation, the
    Apps
    field displays
    FTP Collector - Active
    .
  7. (
    Optional
    ) To view metrics about the FTP Collector, hover over the
    FTP Collector
    link in the Apps field.
    Cortex XDR displays
    Resources
    , including the amount of
    CPU
    ,
    Memory
    , and
    Disk
    space the applet is using.
  8. Manage the FTP Collector.
    After you activate the FTP Collector, you can make additional changes as needed. To modify a configuration, right-click your broker VM and select:
    • FTP Collector
      Configure
      to redefine the FTP Collector configurations.
    • FTP Collector
      Deactivate
      to disable the FTP Collector.

Recommended For You