>
    URL Category Exceptions
    Focus
    Download PDF
    Focus
    1. Home
    2. PAN-OS
    3. URL Filtering
    4. URL Category Exceptions
    Download PDF

    URL Category Exceptions

    Table of Contents
    End-of-Life (EoL)

    URL Category Exceptions

    Guidelines for adding entries to a custom URL list or external dynamic list you want to use in a URL Filtering profile or policy.
    You can exclude specific websites from URL category enforcement, ensuring that these websites are blocked or allowed regardless of the policy action associated with its URL categories. For example, you might block the social-networking URL category but allow access to LinkedIn. To create exceptions to URL category policy enforcement:
    • Add the IP addresses or URLs of sites you want to block or allow to a custom URL category of type URL List (ObjectsCustom ObjectsURL Category). Then, define site access for the custom URL category in a URL Filtering profile. Finally, attach the profile to a Security policy rule.
      You can also use a custom URL category as match criteria in a Security policy rule (PoliciesSecurity, and select Service/URL Category). The exception rule must be placed above any rules that block or allow the categories to which the URL exceptions belong.
    • Add the URLs of sites you want to block or allow to an external dynamic list of type URL List (ObjectsExternal Dynamic Lists). Then, use the external dynamic list in a URL Filtering profile or as match criteria in a Security policy rule. The benefit to using an external dynamic list is that you can update the list without performing a configuration change or commit on the firewall.
    External dynamic lists of type URL List should not be confused with external dynamic lists of type Domain List or IP Address. While external dynamic lists of URLs permit domains and IP addresses, the reverse is not true and result in invalid entries.
    The following guidelines describe how to populate URL category exception lists—custom URL categories or external dynamic lists of URLs:

    Basic Guidelines For URL Category Exception Lists

    Consider the potential matches an entry might have before adding it to a URL category exception list. The following guidelines specify how to create an entry that blocks or allows the websites and pages you intend. Create and evaluate existing entries against these guidelines to ensure you do not block or allow access to more URLs than anticipated.
    The firewall assumes an implicit asterisk at the end of domain entries that do not end in a trailing slash (/) or asterisk (*). For example, if you add example.com to a URL list of allowed websites, the firewall interprets that entry as example.com.*. As a result, the firewall allows access to sites such as example.com.domain.xyz.
    To prevent the firewall from assuming the implicit asterisk, you can append a trailing slash to domain entries that do not end in a / or *. The addition of the trailing slash changes the URLs that the firewall considers a match and for which it enforces policy. In non-wildcard domain entries, the trailing slash limits matches to the given domain and its subdirectories. For example, example.com (example.com/ after processing) matches itself and example.com/search.
    In wildcard domain entries (entries with asterisks or carets), the trailing slash limits matches to URLs that conform to the specified pattern. For example, to match the entry *.example.com, a URL must include at least one subdomain and end with the root domain, example.com. The pattern is: <subdomain>.example.com; news.example.com is a match, but example.com is not because it lacks a subdomain.
    You can enable the firewall to automatically append a trailing slash to applicable entries using the following command line interface (CLI) commands:
    admin@PA-850> debug device-server append-end-token on
    admin@PA-850> configure
    admin@PA-850# commit
    We recommend manually adding trailing slashes to clarify the intended matching behavior of an entry for anyone who inspects it. The trailing slash is invisible if added by the firewall. To disable this feature:
    admin@PA-850> debug device-server append-end-token off
    admin@PA-850> configure
    admin@PA-850# commit
    Panorama™ management servers running PAN-OS® 10.2 cannot enable this feature for firewalls running PAN-OS 10.1 or earlier. You have to enable this feature on each firewall running PAN-OS 10.1 or earlier.
    • List entries are case-insensitive.
    • Omit http and https from URL entries.
    • Each URL entry can be up to 255 characters in length.
    • Every domain that does not end in a / or an * has an implicit asterisk to its end. The firewall processes the entry as if you entered: <domain>.*.
    • Enter an exact match to the website you want to block or allow or use wildcards to create a pattern match.
      Different entries result in different exact matches. If you enter the URL for a specific web page (example.com/contact), the firewall limits matches to that page alone. Exact matching for a domain with a trailing slash restricts matches to the domain itself and its subdirectories. If you enter a domain (without a trailing slash), the firewall matches additional URLs due to the implicit asterisk.
    • Consider adding the URLs most commonly used to access a website or page to your exception list (for example, blog.paloaltonetworks.com and paloaltonetworks.com/blog) if the original entry is accessible from more than URL.
    • The entry example.com is distinct from www.example.com. The domain name is the same, but the second entry contains the www subdomain.
    Palo Alto Networks does not support regular expression use in custom URL category or external dynamic list entries. You must know the specific URLs or be able to construct the URL patterns you want to match using wildcards and the following characters: . / ? & = ; +.

    Wildcard Guidelines for URL Category Exception Lists

    You can use asterisks (*) and carets (^) in URL category exception lists to configure a single entry to match multiple subdomains, domains, top-level domains (TLD), or pages without specifying exact URLs.
    Add a trailing slash (/) to domain entries to ensure the firewall ignores matches to the right of the domain. For the trailing slash to have an impact, an entry must not end in a *.

    How to Use Asterisk (*) and Caret (^) Wildcards

    The following characters are token separators: . / ? & = ; +. Every string separated by one or two of these characters is a token. Use wildcard characters as token placeholders to indicate that a specific token can contain any value. In the entry docs.paloaltonetworks.com, the tokens are “docs”, “paloaltonetworks”, and “com”.
    The following table describes how asterisks and carets work and provides examples.
    *^
    Indicates one or more variable subdomains, domains, TLDs, or subdirectories.
    Can use asterisk after trailing slash, for example, example.com/*.
    Ex: *.domain.com matches docs.domain.com and abc.xyz.domain.com.
    Indicates one variable subdomain, root domain, or TLD.
    Cannot use caret after . The following entry is invalid: example.com/^.
    Ex: ^.domain.com matches docs.domain.com and blog.domain.com.
    Key Point: Asterisks match a greater range of URLs than carets. An asterisk corresponds to any number of consecutive tokens, while a caret corresponds to exactly one token.
    An entry like xyz.*.com matches more sites than xyz.^.^.com; xyz.*.com matches sites with any number of tokens between the strings, and xyz.^.^.com matches sites with exactly two tokens.
    The firewall interprets entries that do not end in a / or * with an implicit asterisk to their end, which further increases the potential matches.
    • A wildcard must be the only character within a token. For example, example*.com is an invalid entry because example and * are in the same token. An entry can contain wildcards in more than one token, however.
    • You can use asterisks and carets in the same entry (for example, *.example.^).
    Do not create an entry with consecutive asterisks (*) or more than nine consecutive carets (^)—entries like these can affect firewall performance.
    For example, do not add an entry like mail.*.*.com. Instead, depending on the range of websites you want to control access to, enter mail.*.com or mail.^.^.com.

    URL Category Exception List—Examples

    The following table displays example URL list entries, matching sites, and explanations for the matching behavior. URL Category Exceptions—Examples (starting at PAN-OS 10.2) shows matching behavior when the firewall appends a trailing slash to domain entries without a trailing slash or terminating asterisk by default.
    URL Exception List EntryMatching SitesExplanation and Notes
    Example Set 1
    paloaltonetworks.com
    paloaltonetworks.com
    paloaltonetworks.com.au
    paloaltonetworks.com.random.org
    paloaltonetworks.com/your-page
    An implicit asterisk is assumed at the end of the domain because a trailing slash is not present. Therefore, matches also include all URLs that begin with the domain.
    Add a trailing slash to your entry to limit matches to the exact domain and its subdirectories.
    paloaltonetworks.com/example
    paloaltonetworks.com/example
    The domain is followed by the subdirectory example. When you enter the URL for a specific web page, the firewall applies the exception action to the specified web page.
    Example Set 2—Asterisks
    *.example.com
    www.example.com
    docs.example.com
    support.tools.example.com.uk
    blog.example.com/your-page
    The asterisk expands matches to all example.com subdomains.
    An implicit asterisk is assumed at the end of the domain because a is not present. Therefore, matches also include URLs that begin with the pattern <subdomain>.example.com.
    To ensure your entry matches only subdomains of the domain, include a .
    mail.example.*
    mail.example.com
    mail.example.co.uk
    mail.example.com/#inbox
    The asterisk expands matches to any URL following the mail.example.<TLD> pattern.
    An asterisk is not implied at the end because the entry already ends in one.
    example.*.com
    example.yoursite.com
    example.es.domain.com
    example.a.b.com.info.us
    example.company.com/1234
    The asterisk expands matches to URLs where example is the left-most subdomain and the last token represented by the asterisk is followed by .com (example.<domain>.com).
    An implicit asterisk is assumed at the end of the domain because a is not present.
    example.com/*
    example.com/photos
    example.com/blog/latest
    any example.com subdirectory
    The domain is followed by a / and an asterisk, which indicates that a subdirectory must be present. The asterisk serves as a token placeholder for any example.com subdirectory.
    An asterisk is not implied at the end because the entry ends in one.
    Example Set 3—Carets
    google.^
    Patterns such as example.co.^ are typically used to match country-specific domains such as example.co.jp. However, generic top-level domains (gTLDs) result in patterns such as example.co.^ matching example.co.info or example.co.amzn, which may not belong to the same organization.
    google.com
    google.co.uk
    google.com/search?q=paloaltonetworks
    An implicit asterisk is assumed to the right of the caret because a trailing slash is not present. Therefore, matches also include URLs with more than one token after google.
    Add a trailing slash to limit matches to sites with one token after the domain.
    ^.google.com
    www.google.com
    news.google.com.test.info
    docs.google.com/document
    The caret expands matches to single-level subdomains of google.com.
    An implicit asterisk is assumed at the end of the domain because a trailing slash is not present. Therefore, matches also include URLs that begin with the <subdomain>.google.com pattern.
    ^.^.google.com
    www.maps.google.com
    support.tools.google.com.abc.xyz
    www.tools.google.com/example-page
    The two carets expand matches to URLs with two consecutive subdomains before google.com.
    An implicit asterisk is assumed at the end of the domain because a trailing slash is not present. Therefore, matches also include URLs that begin with the <subdomain>.<subdomain>.google.com pattern.
    google.^.com
    google.example.com
    google.company.com.it
    google.info.com/example
    The caret expands matches to URLs where google is the left-most subdomain, followed by one token and .com.
    An implicit asterisk is assumed at the end of the domain because a trailing slash is not present. Therefore, matches also include URLs that begin with the google.<domain>.com pattern.

    © 2024 Palo Alto Networks, Inc. All rights reserved.