URL Category Exceptions
Focus
Focus

URL Category Exceptions

Table of Contents
End-of-Life (EoL)

URL Category Exceptions

Guidelines for adding entries to a custom URL list or external dynamic list you want to use in a URL Filtering profile or policy.
You can exclude specific websites from URL category enforcement, ensuring that these websites are blocked or allowed regardless of the policy action associated with its URL categories. For example, you might block the social-networking URL category but allow access to LinkedIn. To create exceptions to URL category policy enforcement:
  • Add the IP addresses or URLs of sites you want to block or allow to a custom URL category of type
    URL List
    (
    Objects
    Custom Objects
    URL Category
    ). Then, define site access for the custom URL category in a URL Filtering profile. Finally, attach the profile to a Security policy rule.
    You can also use a custom URL category as match criteria in a Security policy rule (
    Policies
    Security
    , and select
    Service/URL Category
    ). The exception rule must be placed above any rules that block or allow the categories to which the URL exceptions belong.
  • Add the URLs of sites you want to block or allow to an external dynamic list of type
    URL List
    (
    Objects
    External Dynamic Lists
    ). Then, use the external dynamic list in a URL Filtering profile or as match criteria in a Security policy rule. The benefit to using an external dynamic list is that you can update the list without performing a configuration change or commit on the firewall.
External dynamic lists of type
URL List
should not be confused with external dynamic lists of type Domain List or IP Address. While external dynamic lists of URLs permit domains and IP addresses, the reverse is not true and result in invalid entries.
The following guidelines describe how to populate URL category exception lists—custom URL categories or external dynamic lists of URLs:

Basic Guidelines For URL Category Exception Lists

Consider the potential matches an entry might have before adding it to a URL category exception list. The following guidelines specify how to create an entry that blocks or allows the websites and pages you intend. Create and evaluate existing entries against these guidelines to ensure you do not block or allow access to more URLs than anticipated.
The firewall assumes an
implicit asterisk
at the end of domain entries that do not end in a trailing slash (/) or asterisk (*). For example, if you add
example.com
to a URL list of allowed websites, the firewall interprets that entry as
example.com.*
. As a result, the firewall allows access to sites such as
example.com.domain.xyz
.
To prevent the firewall from assuming the implicit asterisk, you can append a trailing slash to domain entries that do not end in a
/
or
*
. The addition of the trailing slash changes the URLs that the firewall considers a match and for which it enforces policy. In non-wildcard domain entries, the trailing slash limits matches to the given domain and its subdirectories. For example,
example.com
(
example.com/
after processing) matches itself and
example.com/search
.
In wildcard domain entries (entries with asterisks or carets), the trailing slash limits matches to URLs that conform to the specified pattern. For example, to match the entry
*.example.com
, a URL must include at least one subdomain and end with the root domain,
example.com
. The pattern is:
<subdomain>
.example.com
;
news.example.com
is a match, but
example.com
is not because it lacks a subdomain.
You can enable the firewall to automatically append a trailing slash to applicable entries using the following command line interface (CLI) commands:
admin@PA-850>
debug device-server append-end-token on
admin@PA-850>
configure
admin@PA-850#
commit
We recommend manually adding trailing slashes to clarify the intended matching behavior of an entry for anyone who inspects it. The trailing slash is invisible if added by the firewall. To disable this feature:
admin@PA-850>
debug device-server append-end-token off
admin@PA-850>
configure
admin@PA-850#
commit
Panorama™ management servers running PAN-OS
®
10.2 cannot enable this feature for firewalls running PAN-OS 10.1 or earlier. You have to enable this feature on each firewall running PAN-OS 10.1 or earlier.
  • List entries are case-insensitive.
  • Omit
    http
    and
    https
    from URL entries.
  • Each URL entry can be up to 255 characters in length.
  • Every domain that does not end in a
    /
    or an
    *
    has an implicit asterisk to its end. The firewall processes the entry as if you entered:
    <domain>
    .*
    .
  • Enter an
    exact match
    to the website you want to block or allow or use wildcards to create a
    pattern match
    .
    Different entries result in different exact matches. If you enter the URL for a specific web page (
    example.com/contact
    ), the firewall limits matches to that page alone. Exact matching for a domain with a trailing slash restricts matches to the domain itself and its subdirectories. If you enter a domain (without a trailing slash), the firewall matches additional URLs due to the implicit asterisk.
  • Consider adding the URLs most commonly used to access a website or page to your exception list (for example,
    blog.paloaltonetworks.com
    and
    paloaltonetworks.com/blog
    ) if the original entry is accessible from more than URL.
  • The entry
    example.com
    is distinct from
    www.example.com
    . The domain name is the same, but the second entry contains the
    www
    subdomain.
Palo Alto Networks does not support regular expression use in custom URL category or external dynamic list entries. You must know the specific URLs or be able to construct the URL patterns you want to match using wildcards and the following characters:
. / ? & = ; +
.

Wildcard Guidelines for URL Category Exception Lists

You can use asterisks (*) and carets (^) in URL category exception lists to configure a single entry to match multiple subdomains, domains, top-level domains (TLD), or pages without specifying exact URLs.
Add a trailing slash (/) to domain entries to ensure the firewall ignores matches to the right of the domain. For the trailing slash to have an impact, an entry must not end in a
*
.

How to Use Asterisk (*) and Caret (^) Wildcards

The following characters are token separators:
. / ? & = ; +
. Every string separated by one or two of these characters is a token. Use wildcard characters as token placeholders to indicate that a specific token can contain any value. In the entry
docs.paloaltonetworks.com
, the tokens are “docs”, “paloaltonetworks”, and “com”.
The following table describes how asterisks and carets work and provides examples.
*
^
Indicates one or more variable subdomains, domains, TLDs, or subdirectories.
Can use asterisk after trailing slash, for example,
example.com/*
.
Ex:
*.domain.com
matches
docs.domain.com
and
abc.xyz.domain.com
.
Indicates one variable subdomain, root domain, or TLD.
Cannot use caret after . The following entry is invalid:
example.com/^
.
Ex:
^.domain.com
matches
docs.domain.com
and
blog.domain.com
.
Key Point:
Asterisks match a greater range of URLs than carets. An asterisk corresponds to any number of consecutive tokens, while a caret corresponds to exactly one token.
An entry like
xyz.*.com
matches more sites than
xyz.^.^.com
;
xyz.*.com
matches sites with any number of tokens between the strings, and
xyz.^.^.com
matches sites with exactly two tokens.
The firewall interprets entries that do not end in a
/
or
*
with an implicit asterisk to their end, which further increases the potential matches.
  • A wildcard must be the
    only
    character within a token. For example,
    example*.com
    is an invalid entry because
    example
    and
    *
    are in the same token. An entry can contain wildcards in more than one token, however.
  • You can use asterisks and carets in the same entry (for example,
    *.example.^
    ).
Do not create an entry with consecutive asterisks (*) or more than nine consecutive carets (^)—entries like these can affect firewall performance.
For example, do not add an entry like
mail.*.*.com
. Instead, depending on the range of websites you want to control access to, enter
mail.*.com
or
mail.^.^.com
.

URL Category Exception List—Examples

The following table displays example URL list entries, matching sites, and explanations for the matching behavior. URL Category Exceptions—Examples (starting at PAN-OS 10.2) shows matching behavior when the firewall appends a trailing slash to domain entries without a trailing slash or terminating asterisk by default.
URL Exception List Entry
Matching Sites
Explanation and Notes
Example Set 1
paloaltonetworks.com
paloaltonetworks.com
paloaltonetworks.com.au
paloaltonetworks.com.random.org
paloaltonetworks.com/your-page
An implicit asterisk is assumed at the end of the domain because a trailing slash is not present. Therefore, matches also include all URLs that
begin
with the domain.
Add a trailing slash to your entry to limit matches to the exact domain and its subdirectories.
paloaltonetworks.com/example
paloaltonetworks.com/example
The domain is followed by the subdirectory
example
. When you enter the URL for a specific web page, the firewall applies the exception action to the specified web page.
Example Set 2—Asterisks
*.example.com
www.example.com
docs.example.com
support.tools.example.com.uk
blog.example.com/your-page
The asterisk expands matches to all
example.com
subdomains.
An implicit asterisk is assumed at the end of the domain because a is not present. Therefore, matches also include URLs that
begin
with the pattern
<subdomain>
.example.com
.
To ensure your entry matches only subdomains of the domain, include a .
mail.example.*
mail.example.com
mail.example.co.uk
mail.example.com/#inbox
The asterisk expands matches to any URL following the
mail.example.
<TLD>
pattern.
An asterisk is not implied at the end because the entry already ends in one.
example.*.com
example.yoursite.com
example.es.domain.com
example.a.b.com.info.us
example.company.com/1234
The asterisk expands matches to URLs where
example
is the left-most subdomain and the last token represented by the asterisk is followed by
.com
(
example.
<domain>
.com
).
An implicit asterisk is assumed at the end of the domain because a is not present.
example.com/*
example.com/photos
example.com/blog/latest
any example.com subdirectory
The domain is followed by a
/
and an asterisk, which indicates that a subdirectory must be present. The asterisk serves as a token placeholder for any
example.com
subdirectory.
An asterisk is not implied at the end because the entry ends in one.
Example Set 3—Carets
google.^
Patterns such as example.co.^ are typically used to match country-specific domains such as
example.co.jp
. However, generic top-level domains (gTLDs) result in patterns such as example.co.^ matching example.co.info or example.co.amzn, which may not belong to the same organization.
google.com
google.co.uk
google.com/search?q=paloaltonetworks
An implicit asterisk is assumed to the right of the caret because a trailing slash is not present. Therefore, matches also include URLs with more than one token after
google
.
Add a trailing slash to limit matches to sites with one token after the domain.
^.google.com
www.google.com
news.google.com.test.info
docs.google.com/document
The caret expands matches to single-level subdomains of
google.com
.
An implicit asterisk is assumed at the end of the domain because a trailing slash is not present. Therefore, matches also include URLs that
begin
with the
<subdomain>
.google.com
pattern.
^.^.google.com
www.maps.google.com
support.tools.google.com.abc.xyz
www.tools.google.com/example-page
The two carets expand matches to URLs with two consecutive subdomains before
google.com
.
An implicit asterisk is assumed at the end of the domain because a trailing slash is not present. Therefore, matches also include URLs that
begin
with the
<subdomain>
.
<subdomain>
.google.com
pattern.
google.^.com
google.example.com
google.company.com.it
google.info.com/example
The caret expands matches to URLs where
google
is the left-most subdomain, followed by one token and
.com
.
An implicit asterisk is assumed at the end of the domain because a trailing slash is not present. Therefore, matches also include URLs that
begin
with the
google.<domain>.com
pattern.

Recommended For You