This guide is a little bit different to my other Policy Based Forwarding blog post because it uses different virtual routers for both ISP connections. This is quite common to have a distinct default route for both providers. So, in order to route certain traffic, e.g., http/https, to another ISP connection, policy based forwarding is used.

There are two documents from Palo Alto that give advises how to configure PBF.

I am using a PA-200 with PAN-OS 7.0.1. My lab is the following:

(Note that, unlike Juniper ScreenOS, a zone is not tied to a virtual router. You actually can merge interfaces on different vrouters into the same zone. However, I prefer to configure an extra zone for each ISP to keep my security policies clearly separated.)

These are the configuration steps. See the descriptions under the screenshots for details:

Done.

Similar to my test lab for OSPFv2, I am testing OSPFv3 for IPv6 with the following devices: Cisco ASA, Cisco Router, Fortinet FortiGate, Juniper SSG, and Palo Alto. I am showing my lab network diagram and the configuration commands/screenshots for all devices. Furthermore, I am listing some basic troubleshooting commands. In the last section, I provide a Tcpdump/Wireshark capture of an initial OSPFv3 run.

I am not going into deep details of OSPFv3 at all. But this lab should give basic hints for configuring OSPFv3 for all of the listed devices.

Lab

This is my test lab. All devices are directly connected via a layer 2 switch:

General Information

• Everything takes place in area 0.0.0.0 (backbone area)
• Juniper SSG should be the DR: interface priority set to 100.
• Palo Alto should be the BDR: interface priority set to 50.
• Router-ID is always set manually according to my IPv4 sheme: 172.16.1.x, where x = the interface-ID from the IPv6 addresses (from ::1 to ::6).
• Cost for the interfaces as seen in the figure.
• Passive-interface on all user/access interfaces.
• Redistribution of the remote access VPN clients on the Cisco ASA (AnyConnect).
• No authentication is used .

The following devices are in alphabetic order. Beneath each screenshot is a detailed description of the the configuration that is shown.

During the tests, a single Cisco AnyConnect client was connected and therefore redistributed with a /128 IPv6 address prefix.

Cisco ASA

The Cisco ASA 5505 is running version 9.2(4). Following are the configuration and monitoring screenshots:

This are the relevant CLI commands for the OSPFv3 config:

interface Vlan130
 ipv6 address 2003:51:6012:130::1/64
 ipv6 address autoconfig
 ipv6 enable
 ipv6 ospf cost 100
 ipv6 ospf 1 area 0
 ipv6 ospf encryption null
!
ipv6 router ospf 1
 router-id 172.16.1.3
 passive-interface insideASA130
 passive-interface insideASA131
 log-adjacency-changes
 redistribute static metric 1000
!

While this CLI commands can be used to show the OPSFv3 runtime values:

fd-wv-fw03# show ipv6 ospf

 Routing Process "ospfv3 1" with ID 172.16.1.3
 Event-log enabled, Maximum number of events: 1000, Mode: cyclic
 It is an autonomous system boundary router
 Redistributing External Routes from,
    static with metric 1000
 Initial SPF schedule delay 5000 msecs
 Minimum hold time between two consecutive SPFs 10000 msecs
 Maximum wait time between two consecutive SPFs 10000 msecs
 Minimum LSA interval 5 secs
 Minimum LSA arrival 1000 msecs
 LSA group pacing timer 240 secs
 Interface flood pacing timer 33 msecs
 Retransmission pacing timer 66 msecs
 Number of external LSA 1. Checksum Sum 0x4dac
 Number of areas in this router is 1. 1 normal 0 stub 0 nssa
 Graceful restart helper support disabled
 Reference bandwidth unit is 100 mbps
    Area BACKBONE(0)
        Number of interfaces in this area is 2
        SPF algorithm executed 11 times
        Number of LSA 19. Checksum Sum 0xa3f76
        Number of DCbitless LSA 6
        Number of indication LSA 0
        Number of DoNotAge LSA 0
        Flood list length 0

fd-wv-fw03#
fd-wv-fw03#
fd-wv-fw03# show ipv6 ospf neighbor


Neighbor ID     Pri   State           Dead Time   Interface ID    Interface
     172.16.1.1 100   2WAY/DROTHER    0:00:36                880 outside
     172.16.1.2 50    FULL/DR         0:00:34                 16 outside
     172.16.1.5 1     FULL/BDR        0:00:30                  3 outside
     172.16.1.6 1     2WAY/DROTHER    0:00:31                  6 outside
fd-wv-fw03#
fd-wv-fw03#
fd-wv-fw03# show ipv6 ospf database


            OSPFv3 Router with ID (172.16.1.3) (Process ID 1)

                Router Link States (Area 0)

ADV Router       Age         Seq#        Fragment ID  Link count  Bits
      172.16.1.1   1608      0x80000122             1           1 None
      172.16.1.2    636      0x80000124             0           1 E
      172.16.1.3   1461      0x80000102             0           1 E
      172.16.1.5     74      0x80000102             0           1 None
      172.16.1.6   1371      0x80000122             0           1 None

                Net Link States (Area 0)

ADV Router       Age         Seq#        Link ID    Rtr count
      172.16.1.2    634      0x80000122          16 5

                Link (Type-8) Link States (Area 0)

ADV Router       Age         Seq#        Link ID    Interface
      172.16.1.3    430      0x80000008          15 insideASA130
      172.16.1.1   1653      0x8000011d         880 outside
      172.16.1.2   1310      0x8000011e          16 outside
      172.16.1.3    945      0x80000101          14 outside
      172.16.1.5     74      0x80000101           3 outside
      172.16.1.6   1441      0x8000011d           6 outside

                Intra Area Prefix Link States (Area 0)

ADV Router       Age         Seq#        Link ID    Ref-lstype  Ref-LSID
      172.16.1.1   1648      0x80000242           1 0x2001      0
      172.16.1.2    637      0x80000124           1 0x2001      0
      172.16.1.2    629      0x80000129      458752 0x2002      16
      172.16.1.2    637      0x8000011f      589824 0x2002      257
      172.16.1.3    946      0x80000101           0 0x2001      0
      172.16.1.5   1327      0x80000006           0 0x2001      0
      172.16.1.6   1370      0x80000120           2 0x2001      0

                Type-5 AS External Link States

ADV Router       Age         Seq#       Prefix
      172.16.1.3    606      0x80000001  2003:51:6012:133:feed:cafe:0:10/128
fd-wv-fw03#
fd-wv-fw03#
fd-wv-fw03# show ipv6 ospf database self-originate


            OSPFv3 Router with ID (172.16.1.3) (Process ID 1)

                Router Link States (Area 0)

ADV Router       Age         Seq#        Fragment ID  Link count  Bits
      172.16.1.3   1495      0x80000102             0           1 E

                Link (Type-8) Link States (Area 0)

ADV Router       Age         Seq#        Link ID    Interface
      172.16.1.3    464      0x80000008          15 insideASA130
      172.16.1.3    979      0x80000101          14 outside

                Intra Area Prefix Link States (Area 0)

ADV Router       Age         Seq#        Link ID    Ref-lstype  Ref-LSID
      172.16.1.3    979      0x80000101           0 0x2001      0

                Type-5 AS External Link States

ADV Router       Age         Seq#       Prefix
      172.16.1.3    639      0x80000001  2003:51:6012:133:feed:cafe:0:10/128
fd-wv-fw03#
fd-wv-fw03#

Cisco Router

I am running a Cisco 2811 router with version 15.1(4)M9. The configuration commands are the following: (Just for fun I set the OSPF process to “17”.)

interface FastEthernet0/0
 ipv6 address 2003:51:6012:101::5/64
 ipv6 enable
 ipv6 nd ra suppress
 ipv6 ospf 17 area 0.0.0.0
!
interface FastEthernet0/1
 ipv6 address 2003:61:6012:102::1/64
 ipv6 enable
 ipv6 ospf 17 area 0.0.0.0
!
ipv6 router ospf 17
 router-id 172.16.1.5
 auto-cost reference-bandwidth 10000
 passive-interface default
 no passive-interface FastEthernet0/0

And the show commands:

fd-wv-ro03#show ipv6 ospf
 Routing Process "ospfv3 17" with ID 172.16.1.5
 Event-log enabled, Maximum number of events: 1000, Mode: cyclic
 Initial SPF schedule delay 5000 msecs
 Minimum hold time between two consecutive SPFs 10000 msecs
 Maximum wait time between two consecutive SPFs 10000 msecs
 Minimum LSA interval 5 secs
 Minimum LSA arrival 1000 msecs
 LSA group pacing timer 240 secs
 Interface flood pacing timer 33 msecs
 Retransmission pacing timer 66 msecs
 Number of external LSA 1. Checksum Sum 0x004DAC
 Number of areas in this router is 1. 1 normal 0 stub 0 nssa
 Graceful restart helper support enabled
 Reference bandwidth unit is 10000 mbps
    Area BACKBONE(0.0.0.0)
        Number of interfaces in this area is 2
        SPF algorithm executed 23 times
        Number of LSA 19. Checksum Sum 0x098B75
        Number of DCbitless LSA 6
        Number of indication LSA 0
        Number of DoNotAge LSA 0
        Flood list length 0

fd-wv-ro03#
fd-wv-ro03#
fd-wv-ro03#show ipv6 ospf neighbor

Neighbor ID     Pri   State           Dead Time   Interface ID    Interface
172.16.1.1      100   FULL/DROTHER    00:00:35    880             FastEthernet0/0
172.16.1.2       50   FULL/DR         00:00:32    16              FastEthernet0/0
172.16.1.3        1   FULL/DROTHER    00:00:38    14              FastEthernet0/0
172.16.1.6        1   FULL/DROTHER    00:00:30    6               FastEthernet0/0
fd-wv-ro03#
fd-wv-ro03#
fd-wv-ro03#show ipv6 ospf database

            OSPFv3 Router with ID (172.16.1.5) (Process ID 17)

                Router Link States (Area 0.0.0.0)

ADV Router       Age         Seq#        Fragment ID  Link count  Bits
 172.16.1.1      622         0x80000123  1            1           None
 172.16.1.2      1455        0x80000124  0            1           E
 172.16.1.3      243         0x80000103  0            1           E
 172.16.1.5      892         0x80000102  0            1           None
 172.16.1.6      389         0x80000123  0            1           None

                Net Link States (Area 0.0.0.0)

ADV Router       Age         Seq#        Link ID    Rtr count
 172.16.1.2      1453        0x80000122  16         5

                Link (Type-8) Link States (Area 0.0.0.0)

ADV Router       Age         Seq#        Link ID    Interface
 172.16.1.5      131         0x80000007  4          Fa0/1
 172.16.1.1      667         0x8000011E  880        Fa0/0
 172.16.1.2      330         0x8000011F  16         Fa0/0
 172.16.1.3      1766        0x80000101  14         Fa0/0
 172.16.1.5      892         0x80000101  3          Fa0/0
 172.16.1.6      459         0x8000011E  6          Fa0/0

                Intra Area Prefix Link States (Area 0.0.0.0)

ADV Router       Age         Seq#        Link ID    Ref-lstype  Ref-LSID
 172.16.1.1      662         0x80000244  1          0x2001      0
 172.16.1.2      1455        0x80000124  1          0x2001      0
 172.16.1.2      1448        0x80000129  458752     0x2002      16
 172.16.1.2      1455        0x8000011F  589824     0x2002      257
 172.16.1.3      1766        0x80000101  0          0x2001      0
 172.16.1.5      131         0x80000007  0          0x2001      0
 172.16.1.6      388         0x80000121  2          0x2001      0

                Type-5 AS External Link States

ADV Router       Age         Seq#       Prefix
 172.16.1.3      1426        0x80000001  2003:51:6012:133:FEED:CAFE:0:10/128
fd-wv-ro03#
fd-wv-ro03#
fd-wv-ro03#show ipv6 ospf database self-originate

            OSPFv3 Router with ID (172.16.1.5) (Process ID 17)

                Router Link States (Area 0.0.0.0)

ADV Router       Age         Seq#        Fragment ID  Link count  Bits
 172.16.1.5      898         0x80000102  0            1           None

                Link (Type-8) Link States (Area 0.0.0.0)

ADV Router       Age         Seq#        Link ID    Interface
 172.16.1.5      137         0x80000007  4          Fa0/1
 172.16.1.5      898         0x80000101  3          Fa0/0

                Intra Area Prefix Link States (Area 0.0.0.0)

ADV Router       Age         Seq#        Link ID    Ref-lstype  Ref-LSID
 172.16.1.5      137         0x80000007  0          0x2001      0
fd-wv-ro03#
fd-wv-ro03#
fd-wv-ro03#show ipv6 route
IPv6 Routing Table - default - 15 entries
Codes: C - Connected, L - Local, S - Static, U - Per-user Static route
       B - BGP, HA - Home Agent, MR - Mobile Router, R - RIP
       I1 - ISIS L1, I2 - ISIS L2, IA - ISIS interarea, IS - ISIS summary
       D - EIGRP, EX - EIGRP external, NM - NEMO, ND - Neighbor Discovery
       l - LISP
       O - OSPF Intra, OI - OSPF Inter, OE1 - OSPF ext 1, OE2 - OSPF ext 2
       ON1 - OSPF NSSA ext 1, ON2 - OSPF NSSA ext 2
S   ::/0 [1/0]
     via 2003:51:6012:101::1
C   2003:51:6012:101::/64 [0/0]
     via FastEthernet0/0, directly connected
L   2003:51:6012:101::5/128 [0/0]
     via FastEthernet0/0, receive
O   2003:51:6012:110::/64 [110/200]
     via FE80::219:E2FF:FEA1:F98A, FastEthernet0/0
O   2003:51:6012:120::/64 [110/110]
     via FE80::B60C:25FF:FE05:8E10, FastEthernet0/0
O   2003:51:6012:121::/64 [110/110]
     via FE80::B60C:25FF:FE05:8E10, FastEthernet0/0
O   2003:51:6012:123::/64 [110/110]
     via FE80::B60C:25FF:FE05:8E10, FastEthernet0/0
O   2003:51:6012:124::/64 [110/110]
     via FE80::B60C:25FF:FE05:8E10, FastEthernet0/0
O   2003:51:6012:125::/64 [110/110]
     via FE80::B60C:25FF:FE05:8E10, FastEthernet0/0
O   2003:51:6012:130::/64 [110/200]
     via FE80::2A94:FFF:FEA8:772D, FastEthernet0/0
OE2 2003:51:6012:133:FEED:CAFE:0:10/128 [110/1000]
     via FE80::2A94:FFF:FEA8:772D, FastEthernet0/0
O   2003:51:6012:160::/64 [110/200]
     via FE80::A5B:EFF:FE3C:115D, FastEthernet0/0
C   2003:61:6012:102::/64 [0/0]
     via FastEthernet0/1, directly connected
L   2003:61:6012:102::1/128 [0/0]
     via FastEthernet0/1, receive
L   FF00::/8 [0/0]
     via Null0, receive
fd-wv-ro03#
fd-wv-ro03#

Fortinet FortiGate

Unfortunately the FortiGate has no possibility to configure anything of OSPFv3 via the GUI. Everything must be done via the CLI. (And this is called a “Next-Generation Firewall”???)

These are the configuration commands for my lab:

config router ospf6
    set auto-cost-ref-bandwidth 10000
    set router-id 172.16.1.6
        config area
            edit 0.0.0.0
            next
        end
        config ospf6-interface
            edit "wan1"
                set interface "wan1"
            next
            edit "fg-trust"
                set interface "fg-trust"
            next
        end
    set passive-interface "fg-trust"

And the following shows the get commands:

fd-wv-fw04 # get router info6 ospf status
 Routing Process "OSPFv3 (*null*)" with ID 172.16.1.6
 Process uptime is 50 days 22 hours 5 minutes
 SPF schedule delay 5 secs, Hold time between SPFs 10 secs
 Minimum LSA interval 5 secs, Minimum LSA arrival 1 secs
 Number of incomming current DD exchange neighbors 0/5
 Number of outgoing current DD exchange neighbors 0/5
 Number of external LSA 1. Checksum Sum 0x4BAD
 Number of AS-Scoped Unknown LSA 0
 Number of LSA originated 23
 Number of LSA received 37398
 Number of areas in this router is 2
    Area BACKBONE(0)
        Number of interfaces in this area is 2(2)
        SPF algorithm executed 15 times
        Number of LSA 13.  Checksum Sum 0x5C289
        Number of Unknown LSA 0
    Area 0.0.0.51 (Inactive)
        Number of interfaces in this area is 0(0)
        SPF algorithm executed 33 times
        Number of LSA 0.  Checksum Sum 0x0000
        Number of Unknown LSA 0



fd-wv-fw04 #
fd-wv-fw04 #
fd-wv-fw04 # get router info6 ospf neighbor
OSPFv3 Process (*null*)
Neighbor ID     Pri   State           Dead Time   Interface  Instance ID
172.16.1.1      100   2-Way/DROther   00:00:36    wan1       0
172.16.1.2       50   Full/DR         00:00:31    wan1       0
172.16.1.3        1   2-Way/DROther   00:00:32    wan1       0
172.16.1.5        1   Full/Backup     00:00:37    wan1       0


fd-wv-fw04 #
fd-wv-fw04 #
fd-wv-fw04 # get router info6 ospf database

            OSPFv3 Router with ID (172.16.1.6) (Process *null*)

                Link-LSA (Interface wan1)

Link State ID   ADV Router      Age  Seq#       CkSum  Prefix
0.0.3.112       172.16.1.1      1496 0x8000011e 0x6247      1
0.0.0.16        172.16.1.2      1158 0x8000011f 0x4293      1
0.0.0.14        172.16.1.3       578 0x80000102 0xf084      1
0.0.0.3         172.16.1.5      1722 0x80000101 0xf2b9      1
0.0.0.6         172.16.1.6      1287 0x8000011e 0xf486      1

                Link-LSA (Interface fg-trust)

Link State ID   ADV Router      Age  Seq#       CkSum  Prefix
0.0.0.63        172.16.1.6      1261 0x8000011e 0xca19      1

                Router-LSA (Area 0.0.0.0)

Link State ID   ADV Router      Age  Seq#       CkSum    Link
0.0.0.1         172.16.1.1      1451 0x80000123 0x197c      1
0.0.0.0         172.16.1.2       484 0x80000125 0x2b24      1
0.0.0.0         172.16.1.3      1073 0x80000103 0x9562      1
0.0.0.0         172.16.1.5      1722 0x80000102 0xea19      1
0.0.0.0         172.16.1.6      1217 0x80000123 0x84d4      1

                Network-LSA (Area 0.0.0.0)

Link State ID   ADV Router      Age  Seq#       CkSum
0.0.0.16        172.16.1.2       482 0x80000123 0xb390

                Intra-Area-Prefix-LSA (Area 0.0.0.0)

Link State ID   ADV Router      Age  Seq#       CkSum  Prefix  Reference
0.0.0.1         172.16.1.1      1491 0x80000244 0x6d9e      2  Router-LSA
0.0.0.1         172.16.1.2       484 0x80000125 0x265e      5  Router-LSA
0.7.0.0         172.16.1.2       477 0x8000012a 0xb764      1  Network-LSA
0.9.0.0         172.16.1.2       484 0x80000120 0x4fc3      1  Network-LSA
0.0.0.0         172.16.1.3       578 0x80000102 0x972f      1  Router-LSA
0.0.0.0         172.16.1.5       961 0x80000007 0x518b      1  Router-LSA
0.0.0.2         172.16.1.6      1216 0x80000121 0x422d      1  Router-LSA

                AS-external-LSA

Link State ID   ADV Router      Age  Seq#       CkSum
0.0.0.0         172.16.1.3       321 0x80000002 0x4bad E2


fd-wv-fw04 #
fd-wv-fw04 #
fd-wv-fw04 # get router info6 ospf route
OSPFv3 Process (*null*)
Codes: C - connected, D - Discard, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2

   Destination                                   Metric
     Next-hop
C  2003:51:6012:101::/64                             10
     directly connected, wan1, Area 0.0.0.0
O  2003:51:6012:110::/64                            110
     via fe80::219:e2ff:fea1:f98a, wan1, Area 0.0.0.0
O  2003:51:6012:120::/64                             20
     via fe80::b60c:25ff:fe05:8e10, wan1, Area 0.0.0.0
O  2003:51:6012:121::/64                             20
     via fe80::b60c:25ff:fe05:8e10, wan1, Area 0.0.0.0
O  2003:51:6012:123::/64                             20
     via fe80::b60c:25ff:fe05:8e10, wan1, Area 0.0.0.0
O  2003:51:6012:124::/64                             20
     via fe80::b60c:25ff:fe05:8e10, wan1, Area 0.0.0.0
O  2003:51:6012:125::/64                             20
     via fe80::b60c:25ff:fe05:8e10, wan1, Area 0.0.0.0
O  2003:51:6012:130::/64                            110
     via fe80::2a94:fff:fea8:772d, wan1, Area 0.0.0.0
E2 2003:51:6012:133:feed:cafe:0:10/128          10/1000
     via fe80::2a94:fff:fea8:772d, wan1
C  2003:51:6012:160::/64                            100
     directly connected, fg-trust, Area 0.0.0.0
O  2003:61:6012:102::/64                            110
     via fe80::21a:6cff:fea1:2b98, wan1, Area 0.0.0.0

fd-wv-fw04 #
fd-wv-fw04 #

Furthermore, the GUI can at least show the routing table:

Juniper ScreenOS

My SSG 5 runs at version 6.3.0r19. Unlike OSPF for IPv4, in which the “enable” checkmark for each interface is inside the interface configuration section, OSPFv3 is completely configured inside the virtual routers menu:

The config commands via the CLI are the following:

set vrouter trust-vr protocol ospfv3 enable
set vrouter trust-vr protocol ospfv3 area 0.0.0.0
set interface ethernet0/5.10 protocol ospfv3 area 0.0.0.0
set interface ethernet0/5.10 protocol ospfv3 passive
set interface ethernet0/5.10 protocol ospfv3 enable
set interface ethernet0/5.10 protocol ospfv3 cost 100
set interface ethernet0/6 protocol ospfv3 area 0.0.0.0
set interface ethernet0/6 protocol ospfv3 enable
set interface ethernet0/6 protocol ospfv3 priority 100
set interface ethernet0/6 protocol ospfv3 cost 100

And the get commands for displaying the runtime values are this:

fd-wv-fw01-> get vrouter trust-vr protocol ospfv3
VR: trust-vr RouterId: 172.16.1.1
----------------------------------
Status:                                 enabled
State:                                  internal router
Number of areas:                        1
Number of LSA(s):                       20
Number of AS-flooding-scope LSA(s):     1
Area 0.0.0.0
        Total number of interfaces is 2, Active number of interfaces is 2
        Intra-SPF algorithm executed 25 times
        Last Intra-SPF executed before 03:30:25
        Number of LSA(s) is 19

Inter-SPF algorithm executed: 27 times
Last Inter-SPF executed before 01:01:30
Extern-SPF algorithm executed: 28 times
Last Extern-SPF executed before 01:01:30
fd-wv-fw01->
fd-wv-fw01->
fd-wv-fw01-> get vrouter trust-vr protocol ospfv3 neighbor
VR: trust-vr RouterId: 172.16.1.1
----------------------------------
                Neighbor(s) on interface ethernet0/5.10 (Area 0.0.0.0)

                Neighbor(s) on interface ethernet0/6 (Area 0.0.0.0)
RouterId        Nbr-saw-DR      Nbr-saw-BDR     Nbr-If-Id  Opt      Pri State   (Down, Up)
------------------------------------------------------------------------------
172.16.1.3      172.16.1.2      172.16.1.5      0x0000000e --V6|E|R 1   2WAY    (+2    -0)
172.16.1.6      172.16.1.2      172.16.1.5      0x00000006 --V6|E|R 1   2WAY    (+2    -0)
172.16.1.2      172.16.1.2      172.16.1.5      0x00000010 --V6|E|R 50  FULL    (+6    -0)
172.16.1.5      172.16.1.2      172.16.1.5      0x00000003 --V6|E|R 1   FULL    (+6    -0)

fd-wv-fw01->
fd-wv-fw01->
fd-wv-fw01-> get vrouter trust-vr protocol ospfv3 database
VR: trust-vr RouterId: 172.16.1.1
----------------------------------


As-External-LSA
--------------------------------------------------------------------------------
Link-State-Id     Adv-Router-Id      Age      Sequence#    CheckSum
--------------------------------------------------------------------------------
0x00000000        172.16.1.3         1786     0x80000002   0x4bad


Router-LSA for area 0.0.0.0
--------------------------------------------------------------------------------
Link-State-Id     Adv-Router-Id      Age      Sequence#    CheckSum
--------------------------------------------------------------------------------
0x00000000        172.16.1.5         1169     0x80000103   0xe81a
0x00000000        172.16.1.6         884      0x80000124   0x82d5
0x00000001        172.16.1.1         1111     0x80000124   0x177d
0x00000000        172.16.1.3         516      0x80000104   0x9363
0x00000000        172.16.1.2         149      0x80000126   0x2925


Network-LSA for area 0.0.0.0
--------------------------------------------------------------------------------
Link-State-Id     Adv-Router-Id      Age      Sequence#    CheckSum
--------------------------------------------------------------------------------
0x00000010        172.16.1.2         147      0x80000124   0xb191


Intra-Area-Prefix-LSA for area 0.0.0.0
--------------------------------------------------------------------------------
Link-State-Id     Adv-Router-Id      Age      Sequence#    CheckSum
--------------------------------------------------------------------------------
0x00000000        172.16.1.5         417      0x80000008   0x4f8c
0x00000002        172.16.1.6         884      0x80000122   0x402e
0x00000001        172.16.1.1         1152     0x80000246   0x69a0
0x00000000        172.16.1.3         13       0x80000103   0x9530
0x00000001        172.16.1.2         150      0x80000126   0x245f
0x00070000        172.16.1.2         143      0x8000012b   0xb565
0x00090000        172.16.1.2         150      0x80000121   0x4dc4


Link-LSA for link ethernet0/5.10, area 0.0.0.0
--------------------------------------------------------------------------------
Link-State-Id     Adv-Router-Id      Age      Sequence#    CheckSum
--------------------------------------------------------------------------------
0x00000368        172.16.1.1         1157     0x8000011f   0xac59


Link-LSA for link ethernet0/6, area 0.0.0.0
--------------------------------------------------------------------------------
Link-State-Id     Adv-Router-Id      Age      Sequence#    CheckSum
--------------------------------------------------------------------------------
0x00000003        172.16.1.5         1171     0x80000102   0xf0ba
0x00000006        172.16.1.6         956      0x8000011f   0xf287
0x00000370        172.16.1.1         1158     0x8000011f   0x6048
0x0000000e        172.16.1.3         14       0x80000103   0xee85
0x00000010        172.16.1.2         826      0x80000120   0x4094

-----------------------
 printed 20 LSA(s).
fd-wv-fw01->
fd-wv-fw01->
fd-wv-fw01-> get vrouter trust-vr protocol ospfv3 database self-originate
VR: trust-vr RouterId: 172.16.1.1
----------------------------------


Router-LSA for area 0.0.0.0
--------------------------------------------------------------------------------
Link-State-Id     Adv-Router-Id      Age      Sequence#    CheckSum
--------------------------------------------------------------------------------
0x00000001        172.16.1.1         1129     0x80000124   0x177d


Intra-Area-Prefix-LSA for area 0.0.0.0
--------------------------------------------------------------------------------
Link-State-Id     Adv-Router-Id      Age      Sequence#    CheckSum
--------------------------------------------------------------------------------
0x00000001        172.16.1.1         1169     0x80000246   0x69a0


Link-LSA for link ethernet0/5.10, area 0.0.0.0
--------------------------------------------------------------------------------
Link-State-Id     Adv-Router-Id      Age      Sequence#    CheckSum
--------------------------------------------------------------------------------
0x00000368        172.16.1.1         1174     0x8000011f   0xac59


Link-LSA for link ethernet0/6, area 0.0.0.0
--------------------------------------------------------------------------------
Link-State-Id     Adv-Router-Id      Age      Sequence#    CheckSum
--------------------------------------------------------------------------------
0x00000370        172.16.1.1         1175     0x8000011f   0x6048

-----------------------
 printed 4 LSA(s).
fd-wv-fw01->
fd-wv-fw01->
fd-wv-fw01-> get vrouter trust-vr route protocol ospfv3
H: Host C: Connected S: Static A: Auto-Exported
I: Imported R: RIP/RIPng P: Permanent D: Auto-Discovered
N: NHRP
iB: IBGP eB: EBGP O: OSPF/OSPFv3 E1: OSPF external type 1
E2: OSPF/OSPFv3 external type 2 trailing B: backup route

Total 19/max entries

         ID                                   IP-Prefix       Interface
                                                Gateway   P Pref    Mtr     Vsys
--------------------------------------------------------------------------------------
         56                       2003:51:6012:101::/64          eth0/6
                                                     ::   O   60    100     Root
*        67         2003:51:6012:133:feed:cafe:0:10/128          eth0/6
                               fe80::2a94:fff:fea8:772d  E2  200   1000     Root
         54                       2003:51:6012:110::/64       eth0/5.10
                                                     ::   O   60    100     Root
*        57                       2003:51:6012:121::/64          eth0/6
                              fe80::b60c:25ff:fe05:8e10   O   60    110     Root
*        58                       2003:51:6012:120::/64          eth0/6
                              fe80::b60c:25ff:fe05:8e10   O   60    110     Root
*        59                       2003:51:6012:123::/64          eth0/6
                              fe80::b60c:25ff:fe05:8e10   O   60    110     Root
*        60                       2003:51:6012:125::/64          eth0/6
                              fe80::b60c:25ff:fe05:8e10   O   60    110     Root
*        61                       2003:51:6012:124::/64          eth0/6
                              fe80::b60c:25ff:fe05:8e10   O   60    110     Root
*        64                       2003:51:6012:130::/64          eth0/6
                               fe80::2a94:fff:fea8:772d   O   60    200     Root
*        66                       2003:61:6012:102::/64          eth0/6
                               fe80::21a:6cff:fea1:2b98   O   60    200     Root
*        63                       2003:51:6012:160::/64          eth0/6
                                fe80::a5b:eff:fe3c:115d   O   60    200     Root

Total number of ospfv3 routes: 11
fd-wv-fw01->
fd-wv-fw01->

Palo Alto

Finally, this is the Palo Alto guide. I am using a PA-200 with version 7.0.2. To my mind, this is the best OSPFv3 GUI from all firewalls in my lab. Here we go:

To show some runtime stats on the CLI, use this show commands:

weberjoh@fd-wv-fw02> show routing protocol ospfv3 summary

Router ID 172.16.1.2, instance 0 in virtual router default
  OSPFv3 is up, oper status active
  ABR: no, ASBR: yes, Allow transit traffic: yes
  reject-default-route: yes , redist-default-route: n/a
  originated LSA count: 3497, received LSA count: 6676
  num AS-scoped LSA: 0, AS-external LSA count: 1
  num update pending: 0, num update merged: 1
  SPF calc delay: 5.00, min lsa interval : 5.00
  external refresh interval: 1800

weberjoh@fd-wv-fw02>
weberjoh@fd-wv-fw02>
weberjoh@fd-wv-fw02> show routing protocol ospfv3 neighbor

Neighbor ID 172.16.1.1, in virtual router default
  Neighbor Link-local addr fe80:0:0:0:219:e2ff:fea1:f98a,Neighbor If ID 880
  Through local Interface ethernet1/1, local IF ID 16
  Area 0.0.0.0, instance ID 0, status up
  priority 100, state full, event count 10
  Options 0x13, V6(1),E(1),MC(0),N(0),R(1),DC(0)
  Retransmission queue length 0, Waiting on 0 LSA request
  Dead time is 38 sec
  Graceful restart helper status: not helping, time remaining: 0
  Graceful restart helper exit reason: none
Neighbor ID 172.16.1.3, in virtual router default
  Neighbor Link-local addr fe80:0:0:0:2a94:fff:fea8:772d,Neighbor If ID 14
  Through local Interface ethernet1/1, local IF ID 16
  Area 0.0.0.0, instance ID 0, status up
  priority 1, state full, event count 6
  Options 0x13, V6(1),E(1),MC(0),N(0),R(1),DC(0)
  Retransmission queue length 0, Waiting on 0 LSA request
  Dead time is 31 sec
  Graceful restart helper status: not helping, time remaining: 0
  Graceful restart helper exit reason: none
Neighbor ID 172.16.1.5, in virtual router default
  Neighbor Link-local addr fe80:0:0:0:21a:6cff:fea1:2b98,Neighbor If ID 3
  Through local Interface ethernet1/1, local IF ID 16
  Area 0.0.0.0, instance ID 0, status up
  priority 1, state full, event count 6
  Options 0x13, V6(1),E(1),MC(0),N(0),R(1),DC(0)
  Retransmission queue length 0, Waiting on 0 LSA request
  Dead time is 37 sec
  Graceful restart helper status: not helping, time remaining: 0
  Graceful restart helper exit reason: none
Neighbor ID 172.16.1.6, in virtual router default
  Neighbor Link-local addr fe80:0:0:0:a5b:eff:fe3c:115d,Neighbor If ID 6
  Through local Interface ethernet1/1, local IF ID 16
  Area 0.0.0.0, instance ID 0, status up
  priority 1, state full, event count 6
  Options 0x13, V6(1),E(1),MC(0),N(0),R(1),DC(0)
  Retransmission queue length 0, Waiting on 0 LSA request
  Dead time is 29 sec
  Graceful restart helper status: not helping, time remaining: 0
  Graceful restart helper exit reason: none

weberjoh@fd-wv-fw02>
weberjoh@fd-wv-fw02>
weberjoh@fd-wv-fw02> show routing protocol ospfv3 dumplsdb

** OSPF AS-Scope link state database
 VIRTUAL ROUTER: default (id 1)
 VR Type       Adv Router ID   LS id           Seq ID     Cksum  Age   Size
  1 External   172.16.1.3      0.0.0.1         0x80000003 0x3FB7 638   44
     Flags [External Type 2], metric 1000
        2003:51:6012:133:feed:cafe:0:10/128
** OSPF Area Scope link state database
 VIRTUAL ROUTER: default (id 1)
 VR Type       Adv Router ID   LS id           Seq ID     Cksum  Age   Size
  1 Router     172.16.1.1      0.0.0.1         0x8000017B 0x68D4 1698  40
      Options [V6, External, Router], RLA-Flags [none]
      Neighbor Network-ID 172.16.1.2
      Neighbor Interface-ID 0.0.0.16, Interface ID 0.0.3.112
      type 2, metric 100
  1 Router     172.16.1.2      0.0.0.0         0x8000017D 0x7A7C 1131  40
      Options [V6, External, Router], RLA-Flags [External]
      Neighbor Network-ID 172.16.1.2
      Neighbor Interface-ID 0.0.0.16, Interface ID 0.0.0.16
      type 2, metric 10
  1 Router     172.16.1.3      0.0.0.0         0x80000152 0xF6B1 884   40
      Options [V6, External, Router, Demand Circuit], RLA-Flags [External]
      Neighbor Network-ID 172.16.1.2
      Neighbor Interface-ID 0.0.0.16, Interface ID 0.0.0.14
      type 2, metric 100
  1 Router     172.16.1.5      0.0.0.0         0x80000152 0x4A69 296   40
      Options [V6, External, Router, Demand Circuit], RLA-Flags [none]
      Neighbor Network-ID 172.16.1.2
      Neighbor Interface-ID 0.0.0.16, Interface ID 0.0.0.3
      type 2, metric 100
  1 Router     172.16.1.6      0.0.0.0         0x8000017C 0xD12E 68    40
      Options [V6, External, Router], RLA-Flags [none]
      Neighbor Network-ID 172.16.1.2
      Neighbor Interface-ID 0.0.0.16, Interface ID 0.0.0.6
      type 2, metric 10
  1 Network    172.16.1.2      0.0.0.16        0x8000017B 0x3E8  1129  44
      Options [V6, External, Router, Demand Circuit]
      Connected Routers:
        172.16.1.1
        172.16.1.3
        172.16.1.5
        172.16.1.6
        172.16.1.2
  1 IntraArPfx 172.16.1.1      0.0.0.1         0x800002F4 0xC4F  1737  56
      Prefixes 2:
        2003:51:6012:110:0:0:0:0/64, metric 100
        2003:51:6012:101:0:0:0:0/64, metric 100
  1 IntraArPfx 172.16.1.2      0.0.0.1         0x8000017D 0x75B6 1131  92
      Prefixes 5:
        2003:51:6012:123:0:0:0:0/64, metric 10
        2003:51:6012:120:0:0:0:0/64, metric 10
        2003:51:6012:125:0:0:0:0/64, metric 10
        2003:51:6012:121:0:0:0:0/64, metric 10
        2003:51:6012:124:0:0:0:0/64, metric 10
  1 IntraArPfx 172.16.1.2      0.7.0.0         0x80000182 0x7BC  1124  44
      Prefixes 1:
        2003:51:6012:101:0:0:0:0/64, metric 0
  1 IntraArPfx 172.16.1.2      0.9.0.0         0x80000178 0x9E1C 1131  44
      Prefixes 1:
        2003:51:6012:120:0:0:0:0/64, metric 0
  1 IntraArPfx 172.16.1.3      0.0.0.0         0x80000151 0xF87E 884   44
      Prefixes 1:
        2003:51:6012:130:0:0:0:0/64, metric 100
  1 IntraArPfx 172.16.1.5      0.0.0.0         0x80000056 0xB2DA 1272  44
      Prefixes 1:
        2003:61:6012:102:0:0:0:0/64, metric 100
  1 IntraArPfx 172.16.1.6      0.0.0.2         0x8000017A 0x8F86 67    44
      Prefixes 1:
        2003:51:6012:160:0:0:0:0/64, metric 100
** OSPF Link Scope link state database
 VIRTUAL ROUTER: default (id 1)
 VR Type       Adv Router ID   LS id           Seq ID     Cksum  Age   Size
  1 Link       172.16.1.1      0.0.3.112       0x80000176 0xB19F 1742  56
      Options [V6, External, Router]
      Priority 100, Link-local address fe80:0:0:0:219:e2ff:fea1:f98a,
      Prefixes 1:
        2003:51:6012:101:0:0:0:0/64
  1 Link       172.16.1.2      0.0.0.16        0x80000178 0x8FEC 5     56
      Options [V6, External, Router]
      Priority 50, Link-local address fe80:0:0:0:b60c:25ff:fe05:8e10,
      Prefixes 1:
        2003:51:6012:101:0:0:0:0/64
  1 Link       172.16.1.3      0.0.0.14        0x80000151 0x52D3 884   56
      Options [V6, External, Router, Demand Circuit]
      Priority 1, Link-local address fe80:0:0:0:2a94:fff:fea8:772d,
      Prefixes 1:
        2003:51:6012:101:0:0:0:0/64
  1 Link       172.16.1.5      0.0.0.3         0x80000151 0x520A 296   56
      Options [V6, External, Router, Demand Circuit]
      Priority 1, Link-local address fe80:0:0:0:21a:6cff:fea1:2b98,
      Prefixes 1:
        2003:51:6012:101:0:0:0:0/64
  1 Link       172.16.1.6      0.0.0.6         0x80000177 0x42DF 137   56
      Options [V6, External, Router]
      Priority 1, Link-local address fe80:0:0:0:a5b:eff:fe3c:115d,
      Prefixes 1:
        2003:51:6012:101:0:0:0:0/64
  1 Link       172.16.1.2      0.0.1.1         0x80000178 0x92A3 5     56
      Options [V6, External, Router]
      Priority 100, Link-local address fe80:0:0:0:b60c:25ff:fe05:8e13,
      Prefixes 1:
        2003:51:6012:120:0:0:0:0/64
weberjoh@fd-wv-fw02>
weberjoh@fd-wv-fw02>
weberjoh@fd-wv-fw02> show routing route type ospf

flags: A:active, ?:loose, C:connect, H:host, S:static, ~:internal, R:rip, O:ospf, B:bgp,
       Oi:ospf intra-area, Oo:ospf inter-area, O1:ospf ext-type-1, O2:ospf ext-type-2, E:ecmp


VIRTUAL ROUTER: default (id 1)
  ==========
destination                                 nexthop                                 metric flags      age   interface          next-AS
[IPv4 routes omitted]
2003:51:6012:101::/64                       ::                                      10       Oi       675410 ethernet1/1
2003:51:6012:110::/64                       fe80::219:e2ff:fea1:f98a                110    A Oi       674960 ethernet1/1
2003:51:6012:120::/64                       ::                                      10       Oi       945349 ethernet1/4.120
2003:51:6012:121::/64                       ::                                      10       Oi       945349 ethernet1/4.121
2003:51:6012:123::/64                       ::                                      10       Oi       945349 ethernet1/3
2003:51:6012:124::/64                       ::                                      10       Oi       945349 ethernet1/4.124
2003:51:6012:125::/64                       ::                                      10       Oi       945349 ethernet1/4.125
2003:51:6012:130::/64                       fe80::2a94:fff:fea8:772d                110    A Oi       672653 ethernet1/1
2003:51:6012:133:feed:cafe:0:10/128         fe80::2a94:fff:fea8:772d                1000   A O2       4598  ethernet1/1
2003:51:6012:160::/64                       fe80::a5b:eff:fe3c:115d                 110    A Oi       673436 ethernet1/1
2003:61:6012:102::/64                       fe80::21a:6cff:fea1:2b98                110    A Oi       172024 ethernet1/1
total routes shown: 38

weberjoh@fd-wv-fw02>

Wireshark Dump

I captured all OSPF packets while I restarted (reload) the Cisco router. The pcapng therefore contains all five types of OSPFv3 packets (Hello, DBD, LSR, LSU, LSAack). Here it is for download:

OSPFv3 Capture 41.66 KB

Download Wireshark Capture

As an example, these are the messages after the Cisco router has booted (red marked area). After some database description packets (DBD), the router requested (LSR) many details. After that, the designated router (DR) sent many link-state updates (LSU) which contain the link-state advertisements (LSA). The yellow highlighted section shows a LSA for one of the intra-area-prefix LSAs:

For a basic remote access VPN connection to a Palo Alto Networks firewall (called “GlobalProtect”), the built-in VPN feature from Android can be used instead of the GlobalProtect app from Palo Alto itself. If the additional features such as HIP profiling are not needed, this variant fits perfectly.

I am showing a few screenshots and logs from the Android smartphone as well as from the Palo Alto to show the differences.

This post is very similar to the post about the iPhone. I am running a PA-200 with PAN-OS version 7.0.3. The phone is a Samsung Galaxy S4 Mini with Android version 4.4.2.

The GlobalProtect app from Palo Alto works without any problems if a correct Portal and Gateway are already configured. In order to use the native “IPSec Xauth PSK” on Android, the “X-Auth Support” must be enabled on the GlobalProtect Gateway, such as shown here in my post about the Linux vpnc client.

GlobalProtect App vs. Native VPN

The following Android screenshots show the configuration steps for the native IPsec VPN tunnel. The “IPSec Xauth PSK” type must be chosen:

Just for a comparison: The GlobalProtect app looks like that:

Palo Alto Logs

It is interesting to see the differences in the Palo Alto logs, i.e., the GlobalProtect Previous User, System Log and Traffic Log. Here are the differences:

That’s it.

Since a few weeks I am using Tufin SecureTrack in my lab. A product which analyzes firewall policies about their usage and their changes by administrators (and much more). Therefore, the first step is to connect the firewalls to SecureTrack in two directions: SSH from SecureTrack to the device to analyze the configuration, as well as Syslog from the device to SecureTrack to real-time monitor the policy usage.

This blog post shows the adding of the following firewalls into Tufin: Cisco ASA, Fortinet FortiGate, Juniper ScreenOS, and Palo Alto PA.

I am running TufinOS 2.10 on a virtual machine. The Tufin Orchestration Suite (SecureTrack, etc.) is version R15-3.

Pre Note: No IPv6

Though the Tufin appliance can be configured with an IPv6 address, it is not able to communicate with firewalls via IPv6. All connections must traverse via the legacy Internet Protocol. I asked the Tufin support about that, which replied with: “It is not part of the current IPv6 plans, nor any road-map we are aware of.” Oh oh. At least IPv6 network objects can be analyzed, which is the main part of using SecureTrack. For the other features, I mailed a few feature requests to Tufin.

Start monitoring a new device

The configuration steps to add a new device are always the same. Under Settings -> Monitoring -> Manage Devices, select the device type under “Start monitoring a new device” and continue. Give the device a name and set the IP address to which Tufin should connect to. Since I am running OSPF as well as OSPFv3 between all of my firewalls, I am always enabling the “Collect dynamic topology information” feature. Finally, enter the login credentials for connecting via ssh to the firewall. I am always creating a new read-only user for Tufin. The “Monitoring Settings” configuration can be left as default.

The second step is to send syslog messages from each device to Tufin. This is solely done at the firewalls. Of course, all intermediate routers/firewalls must allow the traffic for ssh and syslog between Tufin and the monitored devices.

Cisco ASA

These are the steps for connecting to a Cisco ASA firewall via ssh and syslog. (ASA 5505, 9.2(4)).

Fortinet FortiGate

The ssh connection for a FortiGate is configured through the GUI. (FortiWiFi 90D, v5.2.4, build688).

Since I am already using a syslog-ng server, and since only one syslog server is configurable through the FortiGate GUI (oh Fortinet, why aren’t you improving your GUI?), this must be done via the CLI:

config log syslogd2 setting
    set status enable
    set server "192.168.120.19"
end

Juniper ScreenOS

The SSG firewalls are listed as “Juniper NetScreen” within Tufin. These are the steps. (SSG 5, 6.3.0r20.0).

Palo Alto PA

Finally, the Palo Alto. Note that every security policy rule needs the log forwarding profile attached. Furthermore, the “Config” log messages can be sent to Tufin, too. (PA-200, PAN-OS 7.0.3).

Verifying Syslog

If you want to have the rule and object usage analysis, it is crucial that Tufin receives syslog messages. But after adding a new monitored device, the appropriate icon turns green even though no syslog messages are received yet. Only after some time it will get yellow to warn that “Usage data is not being saved”, if there is no receiving of syslog messages.

If you want to verify that syslog messages are received by Tufin, use tcpdump from the CLI:

[root@jw-tufin01 ~]# tcpdump -i eth0 -vv -w /tmp/syslog.log  -s 1500 src 192.168.86.1 and udp dst port 514
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 1500 bytes
Got 662

Note that it is not relevant that the syslog messages come in from the same source IP address as the device is connected. Under certain circumstances, this can be the case. (E.g., I am connecting to my Juniper firewall via a different vrouter interface than the syslog messages are generated.) Tufin matches the received syslog messages to the correct device.

After some hours/days/weeks of information beeing processed by Tufin SecureTrack, you can analyse the configuration or run certain rule and object policy usage reports.

When using a multilayer firewall design it is not directly clear on which of these firewalls remote site-to-site VPNs should terminate. What must be considered in such scenarios? Differentiate between partners and own remote offices? Or between static and dynamic peer IPs? What about the default routes on the remote sites?

Following is a discussion about different approaches and some best practices. Since not all concepts work with all firewall vendors, the following strategies are separated by common firewalls, i.e., Cisco ASA, Fortinet FortiGate, Juniper ScreenOS, Palo Alto.

Of course, if there is only a single firewall in place, this discussion is not necessary at all. All VPN tunnels must solely terminate on this single firewall. You’re done. But most customers have at least a two-firewall strategy which is not a bad idea at all. While the first firewall is merely for stopping all unused IP connections and for allowing connections to the DMZ, the second firewall has next-gen features such as APT scanning, URL filtering, user recognition, etc. Normally, both firewalls have a default route directed to the Internet (if no proxies are used).

Who has a Static S2S Tunnel?

• Remote Offices: Locations that are owned by the same company. Mostly coming from static IP addresses. If the remote office has no local Internet breakout, it has a default route back to the headquarter. That is: all Internet traffic must be processed by the second, next-generation firewall (or web-proxy). The network behind this remote office can be considered as internal, but on another location. It should be treated in the same way os other internal traffic.
  
• Home Offices: Users that are working from home. Normally coming from dynamic IP addresses. No local Internet breakout -> all traffic must traverse through the second NGFW, too. (I.e., same as “remote office”, but from a dynamic IP address.)
• Partners: Other companies that must access certain services in the DMZ (such as servers or proxies). Almost coming from static IP addresses. No routing/ACLs for accessing the internal networks are required. That is: This traffic must not go through the second firewall.

The Problems

1. The first firewall has a default route to the Internet. Otherwise, no connections could be made from/to the firewall at all.
2. But for the remote- and home-offices, the traffic coming out of the VPN-interface need a default route to the second firewall (and not directly to the Internet). If all VPNs are terminated on the first firewall, this is not the case. -> The best approach is to have an own routing instance for the tunnel-interfaces with a default route to the second firewall.
3. Only a few firewall appliances implement the concept of “virtual routers” (Juniper ScreenOS, Palo Alto). For the FortiGate, policy-based forwarding can be used. For the Cisco ASA, none of these concepts work. Only a workaround can be used there (if it is not an option to buy a better firewall).

(Note that for some remote access VPN solutions, it is default to have two virtual routers / routing tables in the box. This is true, for example, for the Pulse Secure, formerly Juniper SA/MAG. With these devices, traffic coming out from the VPN has another default route than Internet traffic with its default route to the Internet.)

All following concepts describe the case in which the first firewall is from vendor X. It is not related to the second firewall.

Concept 1: Two Virtual Routers on the first firewall (Juniper, Palo Alto)

Both firewall vendors (Juniper ScreenOS and Palo Alto) have virtual routers implemented. That means, the “tunnel-interface” for the VPN can be on another virtual router with another default route. While the default virtual router can point to the Internet (for all outgoing connections and for terminating the VPN), the second virtual router (with the tunnel-interface in it) can point to the second firewall.

Concept 2: Policy-Based Routing/Forwarding (FortiGate)

Unluckily, the FortiGate has not virtual routers, but only virtual domains (vdoms). This is great for separating firewall instances, but does not solve the VPN problem here. However, it is possible to configure a policy route: All traffic coming out of the tunnel-interface has a “new” default route to the second firewall.

Concept 3: Partners on First, Remote Office on Second Firewall (Cisco ASA)

Cisco ASA has neither virtual routers nor policy-based routing. The only “concept” is to terminate the partners on the Cisco ASA and to allow the remote offices to access the second firewall for VPN termination. That is, the static IP addresses from the remote offices must be allowed in an ACL on the Cisco ASA to reach the second firewall directly. Note that it is NOT recommended to allow “from any” to the second firewall, which means that VPN connections from home offices (coming from dynamic IP addresses) will not work in this scenario. That’s definitely a major drawback.

Any Comments?

Please write a comment if I missed something or if you disagree with one of these concepts. Thanks a lot.

The most common transition method for IPv6 (that is: how to enable IPv6 on a network that does not have a native IPv6 connection to the Internet) is a “6in4” tunnel. Other tunneling methods such as Teredo or SixXS are found on different literatures as well. However, another method that is not often explained is to tunnel the IPv6 packets through a normal VPN connection. For example, if the main office has a native IPv6 connection to the Internet as well as VPN connections to its remote offices, it is easy to bring IPv6 subnets to these stations. Here comes an example with two Palo Alto firewalls.

(This post is very similar to this one in which I covered Juniper ScreenOS firewalls.)

I assume that there is already a VPN connection between two Palo Alto firewalls in place. If so, the steps to tunnel IPv6 through this VPN tunnel are quite easy. (Note that this is NOT a 6in4 tunnel! It is simply a forwarding of IPv6 packets through a common IPsec site-to-site VPN.)

Tunneling

I am using two PA-200 firewalls with PAN-OS 7.1.1. One firewall has a static IPv4 address, the other one is dynamic. I am routing a /60 through the tunnel to have a maximum of 16 subnets on the remote site. This is how the lab looks like:

The basic step is to enable IPv6 on the tunnel interfaces and to route the IPv6 traffic appropriately. Of course, all other interface configuration steps such as router advertisements as well as security policies must be configured, too.

Main Site

These are the configuration steps on the main site in order to route the IPv6 subnet through the tunnel. See the screenshot descriptions for more details:

Remote Site

Very similar: the remote site. Note the default IPv6 route through the tunnel:

Latency & Hop Count

After committing everything the traffic log shows some IPv6 connections, either from trust to vpn-s2s (on the remote site), or from vpn-s2s to untrust (on the main site):

It is interesting to see the IPv6 ping times from the remote site compared to the IPv4 ping times. Since the IPv6 connections must travel through the IPv4 VPN tunnel via the Internet before reaching the real destination, the RTT should be much higher than the IPv4 times. However, everything is relative. The ping times (IPv6 and IPv4) on my main site are both around 5-8 ms in the following example:

C:\Users\weberj>ping -4 www.insinuator.net

Ping wird ausgeführt für www.insinuator.net [62.159.96.203] mit 32 Bytes Daten:
Antwort von 62.159.96.203: Bytes=32 Zeit=8ms TTL=57
Antwort von 62.159.96.203: Bytes=32 Zeit=6ms TTL=57
Antwort von 62.159.96.203: Bytes=32 Zeit=8ms TTL=57
Antwort von 62.159.96.203: Bytes=32 Zeit=5ms TTL=57

Ping-Statistik für 62.159.96.203:
    Pakete: Gesendet = 4, Empfangen = 4, Verloren = 0
    (0% Verlust),
Ca. Zeitangaben in Millisek.:
    Minimum = 5ms, Maximum = 8ms, Mittelwert = 6ms

C:\Users\weberj>
C:\Users\weberj>
C:\Users\weberj>ping -6 www.insinuator.net

Ping wird ausgeführt für www.insinuator.net [2003:60:4010:11b0::12] mit 32 Bytes Daten:
Antwort von 2003:60:4010:11b0::12: Zeit=5ms
Antwort von 2003:60:4010:11b0::12: Zeit=5ms
Antwort von 2003:60:4010:11b0::12: Zeit=5ms
Antwort von 2003:60:4010:11b0::12: Zeit=5ms

Ping-Statistik für 2003:60:4010:11b0::12:
    Pakete: Gesendet = 4, Empfangen = 4, Verloren = 0
    (0% Verlust),
Ca. Zeitangaben in Millisek.:
    Minimum = 5ms, Maximum = 5ms, Mittelwert = 5ms

But since my remote site is a DSL connection with a slower latency at all, the IPv6 burden is not that high, as seen here. Both connections are around 30 ms:

C:\Users\Johannes Weber>ping -4 www.insinuator.net

Ping wird ausgeführt für www.insinuator.net [62.159.96.203] mit 32 Bytes Daten:
Antwort von 62.159.96.203: Bytes=32 Zeit=32ms TTL=54
Antwort von 62.159.96.203: Bytes=32 Zeit=31ms TTL=54
Antwort von 62.159.96.203: Bytes=32 Zeit=31ms TTL=54
Antwort von 62.159.96.203: Bytes=32 Zeit=31ms TTL=54

Ping-Statistik für 62.159.96.203:
    Pakete: Gesendet = 4, Empfangen = 4, Verloren = 0
    (0% Verlust),
Ca. Zeitangaben in Millisek.:
    Minimum = 31ms, Maximum = 32ms, Mittelwert = 31ms

C:\Users\Johannes Weber>
C:\Users\Johannes Weber>
C:\Users\Johannes Weber>ping -6 www.insinuator.net

Ping wird ausgeführt für www.insinuator.net [2003:60:4010:11b0::12] mit 32 Bytes Daten:
Antwort von 2003:60:4010:11b0::12: Zeit=35ms
Antwort von 2003:60:4010:11b0::12: Zeit=29ms
Antwort von 2003:60:4010:11b0::12: Zeit=30ms
Antwort von 2003:60:4010:11b0::12: Zeit=29ms

Ping-Statistik für 2003:60:4010:11b0::12:
    Pakete: Gesendet = 4, Empfangen = 4, Verloren = 0
    (0% Verlust),
Ca. Zeitangaben in Millisek.:
    Minimum = 29ms, Maximum = 35ms, Mittelwert = 30ms

That’s it.

I had an error on my PA-200 with PAN-OS 7.0.5 while trying to download a new firmware version. “Error: There is not enough free disk space to complete the desired operation. […]”. Even the tips to delete older software, dynamic updates, etc., and to use the “set max-num-images count” command did not lead to a successful download.

Finally, the TAC support could solve the problem via root access to the Palo Alto firewall and by manually moving data files…

This was the disk space on the firewall before the TAC support corrected it (note the 5th line with 92 % usage):

weberjoh@fd-wv-fw02> show system disk-space

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             1.9G  1.5G  393M  79% /
/dev/sda5             6.6G  5.8G  541M  92% /opt/pancfg
/dev/sda6             1.9G  685M  1.2G  38% /opt/panrepo
tmpfs                 1.3G  116M  1.2G  10% /dev/shm
/dev/sda8             2.4G  1.9G  372M  84% /opt/panlogs

And on my monitoring server I could see that the management config partition increased its usage a few weeks ago:

The supporter now used the 

debug tac-login challenge

debug tac-login response

1. He navigated to the folder in which the AV and content packages reside.
2. He moved the “newav” folder to another partition. Now there was enough disk space on the current partition.
3. Via the GUI, he upgraded the content packages. This worked and PAN-OS correctly deleted the temporary content files.
4. He then moved back the AV files (newav) to the original partition
5. and upgraded the AV packages appropriately. This worked, too, and PAN-OS correctly deleted the temporary AV files, too.

Now the usage of /dev/sda5, mounted on /opt/pancfg, was about 73 %:

weberjoh@fd-wv-fw02> exitshow system disk-spacejobs allsystem disk-space

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             1.9G  1.5G  391M  79% /
/dev/sda5             6.6G  4.6G  1.8G  73% /opt/pancfg
/dev/sda6             1.9G  685M  1.2G  38% /opt/panrepo
tmpfs                 1.3G  116M  1.2G  10% /dev/shm
/dev/sda8             2.4G  2.0G  330M  86% /opt/panlogs

This is the daily graph of the management config partition which shows the decrease of the space usage during the support call:

After this procedure, I was able to update man PAN-OS to a newer version. Thank you. And hopefully this bug will be fixed in an upcoming version! (Bug number 94106.)

Since almost two years I am running a RIPE Atlas Probe in my server room. It resides in an own security zone on a Palo Alto firewall (which also powers the probe via its USB port :)). With this post I publish a few traffic statistics about the RIPE Atlas Probe.

Back in August 2014 I started the RIPE Atlas probe in my lab, of course dual-stacked (IPv6 and legacy IP) and with a 24/7 uptime. (If you are interested in how the RIPE Atlas tool can be used to test servers on the Internet, have a look at this blog post.) I am hosting Probe #19341. Here is a picture of the hardware (Twitter Tweet @2014-08-28):

Just started my @RIPE_Atlas probe, powered by an @PaloAltoNtwks firewall pic.twitter.com/19BkUySAKc

— Johannes Weber (@webernetz) August 28, 2014

Applications and Destinations

And, as always, I am interested in some statistics of it. Since the Palo Alto firewall offers a great “Application Visibility & Control” feature, it was easy to get some basic statistics about the bandwidth and application usage. The following graphs are generated over a 2-week range about all sessions that were generated by the probe. See the descriptions below the graphs:

Some facts (absolute values over two weeks):

• 175 MB SSH traffic, 3 sessions (-> this is a firmware version upgrade from the probe since the destination IPv6 address has a PTR of “ctr-ams06.atlas.ripe.net”)
• 142 MB traceroute, 307k session
• 112 MB DNS, 260k sessions
• 102 MB pingv6, 717k sessions
• 50 MB pingv4, 360k sessions

Generated Traffic

My monitoring system (MRTG with Routers2) also reveals some information about the bandwidth and absolute traffic values. Here are two graphs, the monthly and the yearly one. The monthly shows a straightforward bandwidth usage of 2 kbit/s in both directions, while the yearly shows a few peaks:

The absolute values, measured with MRTG/Routers2 again, are the following (as seen from the firewall interface, i.e., “In” is coming from the probe, while “Out” is traffic to the probe):

Total over rolling last month:          In:  673.44 Mbytes   Out:  705.04 Mbytes	
95th Percentile for rolling last month: In:  2.22 kbps       Out:  2.36 kbps
Total over rolling last year:           In:  8.05 Gbytes     Out:  8.28 Gbytes	
95th Percentile for rolling last year:  In:  2.17 kbps       Out:  2.29 kbps

That is: Over one year, the probe sent 8,05 GB and received 8,28 GB to/from the Internet. In sum, it’s 16,33 GB/y = 45,81 MB/d = 1,91 MB/h = 0,54 KB/s. This is 4,34 kbit/s. Due to the RIPE Atlas FAQs (How much bandwidth will the probe consume?), “an IPv4+IPv6 probe uses approximately 6 kb/s with some user-defined measurements assigned to it.” This seems to be correct.

After I have done some speedtests on the FortiGate firewall I was interested in doing the same tests on a Palo Alto. That is: What are the throughput differences of IPv4 vs. IPv6, measured with and without security profiles, i.e., with and without threat prevention.

It turned out that the throughput is much higher than the official information from Palo Alto. Furthermore, I was not able to test the threat prevention at all, because non of my traffic (Iperf and mere HTTP) went through the antivirus engines. I have to test this again. However, here are the measured values so far:

Laboratory & Scenarios

I had a single PA-200 with PAN-OS 7.1.1 installed. The two test notebooks ran Knoppix 7.6.1 and Iperf version 2.0.5. This time I only used the TCP test. For the HTTP tests I used an Apache2 webserver on the one side and wget on the other side.

I tested the following scenarios, all of them routed through the Palo Alto:

1. Iperf, single policy rule, NO security profiles, both directions. This was recognized as “unknown-tcp” though the application “iperf” exists. (Refer to Applipedia.)
2. Iperf, with security profiles, still recognized as unknown-tcp, both directions.
3. Iperf, with security profiles and app-override, now displayed as “iperf”, both directions.
4. HTTP, with security profiles, recognized as “web-browsing”, only server to client direction, but with enabled server response inspection.

Here are a few screenshots of the policies and the traffic log:

And a listing from one wget HTTP test:

root@Microknoppix:~# wget http://[2003:51:6012:171:221:70ff:fee9:bb47]/1G-file
--2016-05-06 11:14:49--  http://[2003:51:6012:171:221:70ff:fee9:bb47]/1G-file
Verbindungsaufbau zu [2003:51:6012:171:221:70ff:fee9:bb47]:80 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 1048576069 (1000M)
Wird in »»1G-file.1«« gespeichert.

1G-file.1                     100%[=================================================>]   1000M  88,1MB/s    in 11s

2016-05-06 11:15:01 (87,7 MB/s) - »»1G-file.1«« gespeichert [1048576069/1048576069]

Results

These are the test results:

All measured values are almost around 930 Mbps, except the Apache/wget/HTTP traffic. I suppose that this comes from other parameters such as the Apache server, the local file reading/writing, or wget.

For the sake of completeness, these are the raw values, each with Tx/Rx in Mbps:

• Iperf without security profiles, unknown-tcp: IPv4 = 940/934, IPv6 = 926/921
• Iperf with security profiles, unknown-tcp: IPv4 = 939/934, IPv6 = 926/921
• Iperf with security profiles and app override -> iperf: IPv4 = 942/936, IPv6 = 929/923
• HTTP with security profiles, web-browsing, single direction: IPv4 = 774, IPv6 = 704

Conclusion

Ok, this is really strange! Palo Alto lists a firewall throughput for the PA-200 of 100 Mbps and a threat prevention throughput of 50 Mbps. My tested results are way beyond that.

Much better compared to the results from the FortiGate are the “v4 vs. v6” differences. Both protocols are almost at the same speed.

Concerning the security profiles: As long as the application is not recognized correctly (and only shown as unknown-tcp), the threat prevention will not work. That’s ok. Similar to the app override, refer to this PAN documentation: “An application override with a custom application will prevent the session from being processed by the App-ID engine, which is a Layer-7 inspection.” But why is the threat prevention not working for the web-browsing traffic? The throughput should dramatically decrease when the whole traffic is scanned for viruses, etc., shouldn’t it?

Once more some throughput tests, this time the Palo Alto Networks firewalls site-to-site IPsec VPN. Similar to my VPN speedtests for the FortiGate firewall, I set up a small lab with two PA-200 firewalls and tested the bandwidth of different IPsec phase 2 algorithms. Compared to the official data sheet information from Palo Alto that state an IPsec VPN throughput of 50 Mbps, the results are really astonishing.

Lab

My lab consists of two PA-200 firewalls with PAN-OS 7.1.1 installed. They were plugged into a simple layer 2 switch. The two notebooks were booted with Knoppix 7.6.1 and used Iperf version 2.0.5.

I first tested the throughput with only routing and then built the VPN. After every test I changed the phase 2 parameters. The Iperf tests ran in both directions. Here are some configuration screenshots:

Of course I verified the correct IPsec algorithms after each change, such as here:

weberjoh@fd-wv-fw02> show vpn ipsec-sa tunnel VPN-Test

GwID/client IP  TnID   Peer-Address           Tunnel(Gateway)                                Algorithm          SPI(in)  SPI(out) life(Sec/KB)
--------------  ----   ------------           ---------------                                ---------          -------  -------- ------------
20              24     80.154.108.226         VPN-Test(VPN-Test)                             ESP/3DES/SHA1      9AA65C85 D49DF3F6 3481/0

Show IPSec SA: Total 8 tunnels found. 1 ipsec sa found.

Test Results

Here are the results, each Tx/Rx in Mbps:

And the raw values:

• Only routing: 937/934
• esp-3des-sha1-group2-1h: 198/228
• esp-aes128-sha1-group5-1h: 215/271
• esp-aes256-sha256-group14-1h: 205/254
• esp-aes256-sha512-group20-1h: 212/260

That is: All tests are around 200 Mbps. The Tx direction is always a bit slower, which might be a test failure. The AES algorithms are faster than the old 3DES cipher. This might be related to the fact that AES is made to be fast in software and in hardware.

Conclusion

Wow, these are really high values. The data sheet talks about 50 Mbps, even for the bigger PA-500 firewall. I don’t know why, but my test results are four times greater than the official notes. Ok, I can live with that.

This blog post is about using NetFlow for sending network traffic statistics to an nProbe collector which forwards the flows to the network analyzer ntopng. It refers to my blog post about installing ntopng on a Linux machine. I am sending the NetFlow packets from a Palo Alto Networks firewall.

My current ntopng installation uses a dedicated monitoring ethernet port (mirror port) in order to “see” everything that happens in that net. This has the major disadvantage that it only gets packets from directly connected layer 2 networks and vlans. NetFlow on the other hand can be used to send traffic statistics from different locations to a NetFlow flow collector, in this case to the tool nProbe. This single flow collector can receive flows from different subnets and routers/firewalls and even VPN tunnel interfaces, etc. However, it turned out that the “real-time” functionalities of NetFlow are limited since it only refreshes flows every few seconds/bytes, but does not give a real-time look at the network. It should be used only for statistics but not for real-time troubleshooting.

Some Pre Notes

I am using a Ubuntu 14.04.5 LTS (GNU/Linux 3.16.0-77-generic x86_64) server. At the time of writing, nProbe had version v.7.4.160802 while ntopng was in version v.2.4.160802. Furthermore note that nProbe requires a license.

For general information about NetFlow use Wikipedia or Cisco or RFC 3954. For the other tools, use the official web sites: nProbe and ntopng. The nProbe site offers a detailed documentation PDF. A similar tutorial for installing nProbe is this one.

Installation of nProbe

(Since I already showed how to install ntopng, I will only show how to use nProbe here.) The stable builds for nProbe and ntopng are listed here. That is, to install nProbe, I used the following commands:

wget http://apt-stable.ntop.org/14.04/all/apt-ntop-stable.deb
sudo dpkg -i apt-ntop-stable.deb
sudo apt-get update

sudo apt-get install nprobe

Since I want to receive NetFlow packets and forward them to ntopng, nProbe must run in Collector Mode. That is, I am using the following configuration file:

sudo nano /etc/nprobe/nprobe-none.conf

with these entries:

--zmq="tcp://*:5556"
--collector-port=2055
-n=none
-i=none

Note the naming of the config file: “nprobe-none.conf“. This is mandatory due to the documentation of nProbe: “When nProbe is used in probe mode it is not bound to any interface as its job is to collect NetFlow from some other device. In this case the configuration file to be created is: nprobe-none.conf.” (To my mind, this is a spelling mistake because it should read “When nProbe is NOT used in probe mode…”. However, it is working.)

Furthermore, an empty “start” file is needed to tell the init process to use this configuration file:

sudo touch /etc/nprobe/nprobe-none.start

After a start of the service with

sudo service nprobe start

sudo nano /etc/ntopng/ntopng.conf

and add the following interface (= localhost):

--interface="tcp://127.0.0.1:5556"

Finally, restart the ntopng process:

sudo service ntopng restart

A netstat view should indicate the listening 2055 UDP port for nProbe, the 5556 TCP port for the connection between nProbe and ntopng, as well as the common 3000 TCP port from the ntopng WebGUI:

weberjoh@jw-nb10-syslog-mirror:~$ sudo netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      107        12714       1184/redis-server 1
tcp        0      0 0.0.0.0:5556            0.0.0.0:*               LISTEN      0          15260       1641/nprobe
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          12157       1017/sshd
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      65534      14983       1676/ntopng
tcp6       0      0 :::22                   :::*                    LISTEN      0          12159       1017/sshd
udp        0      0 0.0.0.0:2055            0.0.0.0:*                           0          15261       1641/nprobe
udp        0      0 192.168.120.10:123      0.0.0.0:*                           0          14413       1526/ntpd
udp        0      0 127.0.0.1:123           0.0.0.0:*                           0          14412       1526/ntpd
udp        0      0 0.0.0.0:123             0.0.0.0:*                           0          14405       1526/ntpd
udp        0      0 0.0.0.0:161             0.0.0.0:*                           0          12958       1224/snmpd
udp        0      0 0.0.0.0:514             0.0.0.0:*                           0          12684       1157/syslog-ng
udp        0      0 0.0.0.0:55059           0.0.0.0:*                           0          12943       1224/snmpd
udp6       0      0 :::2055                 :::*                                0          15262       1641/nprobe
udp6       0      0 2003:51:6012:120::1:123 :::*                                0          14416       1526/ntpd
udp6       0      0 fe80::21d:92ff:fe53:123 :::*                                0          14415       1526/ntpd
udp6       0      0 ::1:123                 :::*                                0          14414       1526/ntpd
udp6       0      0 :::123                  :::*                                0          14406       1526/ntpd
udp6       0      0 ::1:161                 :::*                                0          12959       1224/snmpd

Since all services are now configured within configuration files that are referenced in the init scripts, they are started automatically after a system reboot. Great.

Palo Alto NetFlow

I am using a Palo Alto Networks firewall (version 7.1.3) to send NetFlow statistics to the nProbe collector. (More information about NetFlow on Palo.) This is configured in the following way: Adding of a NetFlow Server Profile and referencing this profile on all needed Network Interfaces, such as:

I am using quite fast values for the Template Refresh Rate as well as the Active Timeout. On interfaces with huge amount of traffic other values are probably better.

A small tcpdump capture shows some samples of the NetFlow packets sent by the Palo Alto. The following Wireshark screenshots show a NetFlow template as well as a sample flow:

ntopng Usage

Now here is the usage within ntopng. Simply choose the tcp://127.0.0.1:5556 interface at the upper right side. All features of ntopng remain the same, such as using the Dashboard, the Flows or the Hosts pages. (Refer to my post to see some features.)

However, here comes the problem with NetFlow: It is NOT a real-time application that lets ntopng show every single flow and its bandwidth correctly. It can be used to see a rough view of all flows during the past few seconds, but not its actual throughput at the moment.

Refer to the following two dashboard screenshots from ntopng. The first shows the Realtime Top Application Traffic from the NetFlow probe, while the second one shows the same from the mirror port eth1. The 54 MBit/s peak in the first screenshot is not true at all. In fact, it was a constant download over a few minutes. Whereas the second screenshot from eth1 shows the correct real-time bandwidth usage.

Conclusion

nProbe for ntopng can be used quite easily. It is possible to receive flows from different locations which can be displayed in a single instance of ntopng. However, if the primary goal is to have a real-time look at the network, e.g., which hosts or flows are consuming bandwidth, this approach does not fit. NetFlow data must be used with statistical applications that can report traffic stats, but not with real-time analyzers such as ntopng.

While I tested the FQDN objects with a Palo Alto Networks firewall, I ran into some strange behaviours which I could not reproduce, but have documented them. I furthermore tested the usage of FQDN objects with more than 32 IP addresses, which are the maximum that are supported due to the official Palo Alto documentation. Here we go:

Some Notes

I am using a Palo Alto PA-200 with PAN-OS 7.1.4-h2. A description of how to use the FQDN objects by Palo Alto Networks is this “How to Configure and Test FQDN Objects” article. To show and refresh them via the CLI, these commands can be used (refer to my list of CLI troubleshooting commands):

request system fqdn show
request system fqdn refresh

Note that at least one policy must use an FQDN object to be queried by the firewall. Otherwise, it won’t be resolved at all.

The release notes from PAN-OS 7.1 state: “Issue ID 98576: In PAN-OS 7.1 and later releases, the maximum number of address objects you can resolve for an FQDN is increased from 10 of each address type (IPv4 and IPv6) to a maximum of 32 each. However, the combination of IPv4 and IPv6 addresses cannot exceed 512B; if it does, addresses that are not included in the first 512B are dropped and not resolved.”

In oder to test different scenarios, I generated the following FQDNs on my DNS server. The names are almost self-explanatory. I am using the documentation prefixes for both, IPv4 (RFC 5737) and IPv6 (RFC 3849):

• 16a.weberdns.de
• 16aaaa.weberdns.de
• 16dual.weberdns.de
• 32a.weberdns.de
• 32aaaa.weberdns.de
• 32dual.weberdns.de
• 32dual-long.weberdns.de <- with full random IP addresses that are long, i.e., no :: abbreviations for IPv6
• 64a.weberdns.de
• 64aaaa.weberdns.de
• 64dual.weberdns.de

Online tools such as DNSWatch can be used to query the DNS names.

Tests

I added a few security policies in order to use the FQDN objects.

16 – no problem

The objects with the 16x address were no problem. Neither the A, AAAA, nor the dual ones:

weberjoh@pa> request system fqdn show

FQDN Table : Last Request time Fri Aug 26 11:07:15 2016
--------------------------------------------------------------------------------
                      IP Address     Remaining TTL     Secs Since Refreshed
--------------------------------------------------------------------------------
VSYS  : vsys1 (using mgmt-obj dnsproxy object)

16a.weberdns.de  (Objectname h_fqdn_16a.weberdns.de):

                       192.0.2.1              2048                     1552
                      192.0.2.10              2048                     1552
                      192.0.2.11              2048                     1552
                      192.0.2.12              2048                     1552
                      192.0.2.13              2048                     1552
                      192.0.2.14              2048                     1552
                      192.0.2.15              2048                     1552
                      192.0.2.16              2048                     1552
                       192.0.2.2              2048                     1552
                       192.0.2.3              2048                     1552
                       192.0.2.4              2048                     1552
                       192.0.2.5              2048                     1552
                       192.0.2.6              2048                     1552
                       192.0.2.7              2048                     1552
                       192.0.2.8              2048                     1552
                       192.0.2.9              2048                     1552

16aaaa.weberdns.de  (Objectname h_fqdn_16aaaa.weberdns.de):

            2001:db8:0:0:0:0:0:1              2048                     1552
           2001:db8:0:0:0:0:0:10              2048                     1552
           2001:db8:0:0:0:0:0:11              2048                     1552
           2001:db8:0:0:0:0:0:12              2048                     1552
           2001:db8:0:0:0:0:0:13              2048                     1552
           2001:db8:0:0:0:0:0:14              2048                     1552
           2001:db8:0:0:0:0:0:15              2048                     1552
           2001:db8:0:0:0:0:0:16              2048                     1552
            2001:db8:0:0:0:0:0:2              2048                     1552
            2001:db8:0:0:0:0:0:3              2048                     1552
            2001:db8:0:0:0:0:0:4              2048                     1552
            2001:db8:0:0:0:0:0:5              2048                     1552
            2001:db8:0:0:0:0:0:6              2048                     1552
            2001:db8:0:0:0:0:0:7              2048                     1552
            2001:db8:0:0:0:0:0:8              2048                     1552
            2001:db8:0:0:0:0:0:9              2048                     1552

16dual.weberdns.de  (Objectname h_fqdn_16dual.weberdns.de):

                       192.0.2.1              2048                     1552
                      192.0.2.10              2048                     1552
                      192.0.2.11              2048                     1552
                      192.0.2.12              2048                     1552
                      192.0.2.13              2048                     1552
                      192.0.2.14              2048                     1552
                      192.0.2.15              2048                     1552
                      192.0.2.16              2048                     1552
                       192.0.2.2              2048                     1552
                       192.0.2.3              2048                     1552
                       192.0.2.4              2048                     1552
                       192.0.2.5              2048                     1552
                       192.0.2.6              2048                     1552
                       192.0.2.7              2048                     1552
                       192.0.2.8              2048                     1552
                       192.0.2.9              2048                     1552
            2001:db8:0:0:0:0:0:1              2048                     1552
           2001:db8:0:0:0:0:0:10              2048                     1552
           2001:db8:0:0:0:0:0:11              2048                     1552
           2001:db8:0:0:0:0:0:12              2048                     1552
           2001:db8:0:0:0:0:0:13              2048                     1552
           2001:db8:0:0:0:0:0:14              2048                     1552
           2001:db8:0:0:0:0:0:15              2048                     1552
           2001:db8:0:0:0:0:0:16              2048                     1552
            2001:db8:0:0:0:0:0:2              2048                     1552
            2001:db8:0:0:0:0:0:3              2048                     1552
            2001:db8:0:0:0:0:0:4              2048                     1552
            2001:db8:0:0:0:0:0:5              2048                     1552
            2001:db8:0:0:0:0:0:6              2048                     1552
            2001:db8:0:0:0:0:0:7              2048                     1552
            2001:db8:0:0:0:0:0:8              2048                     1552
            2001:db8:0:0:0:0:0:9              2048                     1552

32 – the problems began

As expected, some problems arose when I used the 32x FQDN objects. But not only within the objects, but with the whole Palo Alto. After the commit, the GUI displayed a “not ready” and logged me out after a few seconds,

while the CLI session ended, too. After a re-login, it displayed a “system initializing” message:

Using username "weberjoh".
Last login: Wed Aug 24 11:32:17 2016 from p20030050aa0082001494745f15bf1c6c.dip0.t-ipconnect.de
System initializing; please wait... (CTRL-C to bypass)

Hm. A bit strange. After a re-login I listed the FQDN objects via the CLI. While the 32a object was ok, the 32aaaa was missing some entries (probably due to the longer as 512 byte DNS answer), while the 32dual and 32dual-long were displayed as “Not used” while they were definitely used. That is, I suppose that there is still a bug within the handling of multiple records from a single FQDN name.

weberjoh@pa> request system fqdn show

FQDN Table : Last Request time Fri Aug 26 11:38:37 2016
--------------------------------------------------------------------------------
                      IP Address     Remaining TTL     Secs Since Refreshed
--------------------------------------------------------------------------------
VSYS  : vsys1 (using mgmt-obj dnsproxy object)

32a.weberdns.de  (Objectname h_fqdn_32a.weberdns.de):

                       192.0.2.1              2025                      807
                      192.0.2.10              2025                      807
                      192.0.2.11              2025                      807
                      192.0.2.13              2025                      807
                      192.0.2.14              2025                      807
                      192.0.2.15              2025                      807
                      192.0.2.16              2025                      807
                      192.0.2.17              2025                      807
                      192.0.2.18              2025                      807
                      192.0.2.19              2025                      807
                      192.0.2.20              2025                      807
                      192.0.2.21              2025                      807
                      192.0.2.22              2025                      807
                      192.0.2.23              2025                      807
                      192.0.2.24              2025                      807
                      192.0.2.25              2025                      807
                      192.0.2.27              2025                      807
                      192.0.2.28              2025                      807
                      192.0.2.29              2025                      807
                       192.0.2.3              2025                      807
                      192.0.2.30              2025                      807
                      192.0.2.31              2025                      807
                      192.0.2.32              2025                      807
                       192.0.2.4              2025                      807
                       192.0.2.5              2025                      807
                       192.0.2.6              2025                      807
                       192.0.2.7              2025                      807
                       192.0.2.8              2025                      807
                       192.0.2.9              2025                      807

32aaaa.weberdns.de  (Objectname h_fqdn_32aaaa.weberdns.de):

           2001:db8:0:0:0:0:0:12              2025                      807
           2001:db8:0:0:0:0:0:13              2025                      807
           2001:db8:0:0:0:0:0:14              2025                      807
           2001:db8:0:0:0:0:0:15              2025                      807
           2001:db8:0:0:0:0:0:17              2025                      807
           2001:db8:0:0:0:0:0:18              2025                      807
           2001:db8:0:0:0:0:0:19              2025                      807
           2001:db8:0:0:0:0:0:20              2025                      807
           2001:db8:0:0:0:0:0:24              2025                      807
           2001:db8:0:0:0:0:0:26              2025                      807
           2001:db8:0:0:0:0:0:28              2025                      807
           2001:db8:0:0:0:0:0:29              2025                      807
            2001:db8:0:0:0:0:0:3              2025                      807
           2001:db8:0:0:0:0:0:30              2025                      807
            2001:db8:0:0:0:0:0:4              2025                      807
            2001:db8:0:0:0:0:0:7              2025                      807
            2001:db8:0:0:0:0:0:9              2025                      807

32dual-long.weberdns.de  (Objectname h_fqdn_32dual-long.weberdns.de):

                        Not used

32dual.weberdns.de  (Objectname h_fqdn_32dual.weberdns.de):

                        Not used

I made a second try with only the 32dual-long object (while I disabled all other rules containing my test FQDN objects). The management plane restarted again and logged me out, too. Furthermore, the object was still listed as “Not used” while the other (disabled!) objects were still listed, even after a manual refresh!

weberjoh@pa> request system fqdn show

FQDN Table : Last Request time Fri Aug 26 12:10:15 2016
--------------------------------------------------------------------------------
                      IP Address     Remaining TTL     Secs Since Refreshed
--------------------------------------------------------------------------------
VSYS  : vsys1 (using mgmt-obj dnsproxy object)

32dual-long.weberdns.de  (Objectname h_fqdn_32dual-long.weberdns.de):

                        Not used

After these results I decided to not even test the 64x objects.

What Was That?

Another strange behaviour was this test case: Before I used the 16x and 32x objects, I simply created a domain name called “many.weberdns.de” with 17x A records. But after a first usage, the following records were listed by the CLI. The strange parts are the lines 9, 10, and 20, because these are my DNS servers IP addresses that are definitely not part of the “many.weberdns.de” domain name!

many.weberdns.de  (Objectname h_fqdn_many.weberdns.de):

                     11.31.62.97              3180                       51
                      11.7.40.94              3180                       51
                     116.4.29.46              3180                       51
                     124.9.42.64              3180                       51
                    132.17.21.71              3180                       51
                       17.4.2.11              3180                       51
     2003:51:6012:110:0:0:a07:53              3180                       51
                   213.61.29.182              3180                       51
                      25.65.7.44              3180                       51
                     28.97.1.119              3180                       51
                     38.170.12.0              3180                       51
                     4.73.254.24              3180                       51
                     52.29.39.21              3180                       51
                     56.18.17.64              3180                       51
                    56.43.229.35              3180                       51
                     62.33.29.91              3180                       51
                    67.225.70.20              3180                       51
                  80.154.108.230              3180                       51
                    82.53.61.170              3180                       51
                     95.44.87.54              3180                       51

After some more commits this FQDN object was listed correctly and I was not able to reproduce this behaviour. However, it looked not that reliable. Here it is without any wrong entries:

many.weberdns.de  (Objectname h_fqdn_many.weberdns.de):

                     11.31.62.97              3392                      208
                      11.7.40.94              3392                      208
                     116.4.29.46              3392                      208
                     124.9.42.64              3392                      208
                    132.17.21.71              3392                      208
                       17.4.2.11              3392                      208
                      25.65.7.44              3392                      208
                     28.97.1.119              3392                      208
                     38.170.12.0              3392                      208
                     4.73.254.24              3392                      208
                     52.29.39.21              3392                      208
                     56.18.17.64              3392                      208
                    56.43.229.35              3392                      208
                     62.33.29.91              3392                      208
                    67.225.70.21              3392                      208
                    82.53.61.170              3392                      208
                     95.44.87.54              3392                      208

Conclusion

Let’s first notice that the limitation of 10 IP addresses is not present anymore, which is good. At least 16 IP address objects worked without any problems. Even the 32 A records worked. But all other objects, e.g., 32 AAAA records or even more, are not just “not working” but forced a restart (?) of the management plane. Uh. I am not fully sure whether this is related to my small hardware (PA-200), or to some other side effects such as DNS proxies configured on the Palo, etc. Maybe someone has similar experiences?

I am using the DNS Proxy on a Palo Alto Networks firewall for some user subnets. Beside the default/primary DNS server it can be configured with proxy rules (sometimes called conditional forwarding) which I am using for reverse DNS lookups, i.e., PTR records, that are answered by a BIND DNS server. While it is easy and well-known to configure the legacy IP (IPv4) reverse records, the IPv6 ones are slightly more difficult. Fortunately there are some good tools on the Internet to help reversing IPv6 addresses.

I am using a PA-200 with PAN-OS 7.1.2. The BIND server runs on a Ubuntu 12.04.5 LTS with BIND version 9.8.1-P1. For some general information about DNS reverse lookups, use this Wikipedia article. Similar for the notation of IPv6 address in the DNS.

DNS Proxy Rule

This is the configuration of my DNS Proxy with one proxy rule for the reverse lookups. Note that the connections from the Palo Alto to the DNS servers are established via IPv6 though the bulk of DNS lookups is still IPv4 (A records).

These are the four “domain names” I configured. The first three are the well-known legacy IP reverse zones (RFC1918) while the last one is my /48 global unicast IPv6 subnet.

*.168.192.in-addr.arpa
*.16.172.in-addr.arpa
*.10.in-addr.arpa
*.2.1.0.6.1.5.0.0.3.0.0.2.ip6.arpa

Now all DNS queries are primarily sent to DNS server at 2003:51:6012:120::a08:53, while the reverse DNS (rDNS) lookups are sent to 2003:51:6012:120::11.

BIND Zone

I am using a BIND server for my reverse zones. I am using this tool to generate the IPv6 zone file as well as this for further IPv6 PTR records. These are two of the four zones configured within the “named.conf.local” configuration file:

zone "168.192.in-addr.arpa" {
  type master;
  file "/etc/bind/db.168.192";
};

zone "2.1.0.6.1.5.0.0.3.0.0.2.ip6.arpa" {
  type master;
  file "/etc/bind/db.2.1.0.6.1.5.0.0.3.0.0.2";
};

And this is part of my /48 zone for IPv6 PTR records, generated with the tool just mentioned:

weberjoh@jw-vm10:/etc/bind$ cat db.2.1.0.6.1.5.0.0.3.0.0.2
;
; 2003:51:6012::/48
;
; Zone file built with the IPv6 Reverse DNS zone builder
; http://rdns6.com/
;
$TTL 1h ; Default TTL
@       IN      SOA     jw-vm10.webernetz.net.  webmaster.webernetz.net. (
        2016061401      ; serial
        1h              ; slave refresh interval
        15m             ; slave retry interval
        1w              ; slave copy expire time
        1h              ; NXDOMAIN cache time
        )

;
; domain name servers
;
@       IN      NS      jw-vm10.webernetz.net.


; IPv6 PTR entries
; 110
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.1.0.2.1.0.6.1.5.0.0.3.0.0.2.ip6.arpa. IN PTR fd-wv-fw01-dmz.webernetz.net.
9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.1.0.2.1.0.6.1.5.0.0.3.0.0.2.ip6.arpa. IN PTR jw-nb12-lx.webernetz.net.
2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.1.0.2.1.0.6.1.5.0.0.3.0.0.2.ip6.arpa. IN PTR jw-vm06-planes.webernetz.net.
5.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.1.0.2.1.0.6.1.5.0.0.3.0.0.2.ip6.arpa. IN PTR jw-esa.webernetz.net.
7.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.1.0.2.1.0.6.1.5.0.0.3.0.0.2.ip6.arpa. IN PTR f5-external.webernetz.net.
0.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.1.0.2.1.0.6.1.5.0.0.3.0.0.2.ip6.arpa. IN PTR jw-ts01-perle.webernetz.net.
2.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.1.0.2.1.0.6.1.5.0.0.3.0.0.2.ip6.arpa. IN PTR jw-vm16-ns1.weberdns.de.
2.d.3.5.3.0.e.f.f.f.7.7.3.1.2.0.0.1.1.0.2.1.0.6.1.5.0.0.3.0.0.2.ip6.arpa. IN PTR jw-nb04-ftp-adsb.webernetz.net.

Test ‘n Wireshark

This is a basic test from a Windows 7 client behind one of the user subnets on the Palo Alto. It uses the IPv4 address of the Palo Alto layer 3 interface (192.168.125.1) for DNS queries. I tested a normal DNS name

blog.webernetz.net

192.168.100.0

C:\Users\weberjoh>nslookup
Standardserver:  pa-user.webernetz.net
Address:  192.168.125.1

> blog.webernetz.net
Server:  pa-user.webernetz.net
Address:  192.168.125.1

Nicht autorisierende Antwort:
Name:    blog.webernetz.net
Addresses:  2a01:488:42:1000:50ed:8588:8a:c570
          80.237.133.136

>
>
> 192.168.100.0
Server:  pa-user.webernetz.net
Address:  192.168.125.1

Name:    nc-client-0.webernetz.net
Address:  192.168.100.0

>
>

Captured on the Palo Alto (Monitor -> Packet Capture), these are two screenshots from Wireshark that show the connections to the different DNS servers for the different use cases. In any case, the queries from the Palo Alto are made from the appropriate layer 3 interfaces with the corresponding IPv6 addresses, in my case 2003:51:6012:125::1, etc.:

Some Notes

For more information about the DNS proxy use this Palo Alto Networks article: How to Configure DNS Proxy on a Palo Alto Networks Firewall.

Also note that during some configuration changes (commits) on the Palo Alto, the DNS proxy was not working anymore at all! The only way to bring it back to life was to restart the process from the CLI on the Palo:

debug software restart process dnsproxy

Cheers.

Another great tool from Babak Farrokhi is dnstraceroute. It is part of the DNSDiag toolkit from which I already showed the dnsping feature. With dnstraceroute you can verify whether a DNS request is indeed answered by the correct DNS server destination or whether a man-in-the-middle has spoofed/hijacked the DNS reply. It works by using the traceroute trick by incrementing the TTL value within the IP header from 1 to 30.

Beside detecting malicious DNS spoofing attacks, it can also be used to verify security features such as DNS sinkholing. I am showing the usage as well as a test case for verifying a sinkhole feature.

./dnstraceroute

To use dnstraceroute, the GitHub repository must be cloned, as already shown on other sites or on the official documentation:

git clone https://github.com/farrokhi/dnsdiag.git
cd dnsdiag
git submodule update --init

Since dnstraceroute needs root privileges, it must be run with sudo (or the like). It basically needs the hostname to be tested against the first DNS resolver of the system. With other options the DNS server can be set manually

-s <server>

-x

Here is a typical execution of dnstraceroute. I tested the domain “blog.webernetz.net” to the Google public DNS server 8.8.8.8:

weberjoh@jw-nb12-lx:~/dnsdiag$ sudo ./dnstraceroute.py -x -s 8.8.8.8 blog.webernetz.net
dnstraceroute.py DNS: 8.8.8.8:53, hostname: blog.webernetz.net, rdatatype: A
1       pa-dmz.webernetz.net (192.168.110.1) 1 ms
2       80.154.108.225 (80.154.108.225) 3 ms
3       f-eb12.F.DE.NET.DTAG.DE (62.154.10.9) 3 ms
4       217.239.49.206 (217.239.49.206) 4 ms
5       72.14.196.17 (72.14.196.17) 3 ms
6       216.239.56.110 (216.239.56.110) 3 ms
7       216.239.57.188 (216.239.57.188) 4 ms
8       209.85.251.178 (209.85.251.178) 6 ms
9       209.85.255.39 (209.85.255.39) 18 ms
10      108.170.234.47 (108.170.234.47) 13 ms
11       *
12      google-public-dns-a.google.com (8.8.8.8) 12 ms

=== Expert Hints ===
 [*] public DNS server is next to an invisible hop (probably a firewall)

Captured with Wireshark, it looks like this: Beginning with a TTL of 1, the value is incremented until the reply came. The ICMP time-to-live exceeded messages are displayed as the hops by dnstraceroute:

DNS Sinkholing

I am happy that my ISPs here in Germany are not spoofing my DNS queries. That is, I could not test such a scenario. (Refer to “Is Your ISP Hijacking Your DNS Traffic?” at RIPE Labs for such examples.) BUT, with dnstraceroute some “good” spoofing examples can be tested, too, such as DNS sinkholing. With this technique, next-generation firewalls such as the Palo Alto Networks firewall reply to DNS queries for malicious sites with a sinkhole IP address in order to protect the client before he even TCP-SYNed to the destination.

I tested a domain name that was listed as malicious with dnstraceroute. Obviously, the real 8.8.8.8 is not directly connected to my network ;), but listed as the first and only hop by dnstraceroute:

weberjoh@jw-nb12-lx:~/dnsdiag$ sudo ./dnstraceroute.py -x -s 8.8.8.8 3qtep.69khz.com
dnstraceroute.py DNS: 8.8.8.8:53, hostname: 3qtep.69khz.com, rdatatype: A
1       google-public-dns-a.google.com (8.8.8.8) 3 ms

=== Expert Hints ===
 [*] path too short (possible DNS hijacking, unless it is a local DNS resolver)

A look at the Wireshark capture shows the DNS query with a TTL with 1, but instead of an ICMP TTL exceeded reply, the Palo Alto firewall directly replied with the DNS sinkhole address:

Yeah, the DNS sinkholing works and dnstraceroute was able to verify it.

What it does not detect

Note that this tool does not detect false DNS answers from a correct DNS server. For example, though my German ISP “Deutsche Telekom” is not hijacking DNS queries, their DNS resolvers are answering with falsified A records for unregistered domains. Of course, this is a spoofed reply, but not from a man-in-the-middle but from the real DNS resolver.

In the following example I queried my home router (FRITZ!Box) which uses the DNS resolvers from Deutsche Telekom about “weberfoobar.de”, which does not exists. The DNS proxy is two hops away, so nothing seems to be wrong:

weberjoh@jw-nb12-lx:~/dnsdiag$ sudo ./dnstraceroute.py -s 192.168.7.1 weberfoobar.de
dnstraceroute.py DNS: 192.168.7.1:53, hostname: weberfoobar.de, rdatatype: A
1       pa-dmz.webernetz.net (192.168.110.1) 1 ms
2       fritzbox-fdorf.webernetz.net (192.168.7.1) 24 ms

But when testing the same non-existent domain name to the Google server, the correct answer (after the correct traceroute path) is “non exist”:

weberjoh@jw-nb12-lx:~/dnsdiag$ sudo ./dnstraceroute.py -s 8.8.8.8 weberfoobar.de
dnstraceroute.py DNS: 8.8.8.8:53, hostname: weberfoobar.de, rdatatype: A
1       pa-dmz.webernetz.net (192.168.110.1) 1 ms
2       80.154.108.225 (80.154.108.225) 2 ms
3       f-eb12.F.DE.NET.DTAG.DE (62.154.10.9) 10 ms
4       217.239.49.206 (217.239.49.206) 14 ms
5       72.14.196.17 (72.14.196.17) 3 ms
6       216.239.56.26 (216.239.56.26) 3 ms
7       216.239.57.125 (216.239.57.125) 22 ms
8       209.85.143.26 (209.85.143.26) 7 ms
9       209.85.255.39 (209.85.255.39) 17 ms
10      209.85.243.65 (209.85.243.65) 18 ms
11       *
Invalid hostname: None of DNS query names exist: weberfoobar.de., weberfoobar.de.webernetz.net.

That is, such types of DNS spoofing must be detected with manual queries, e.g., with dig. Note that the Telekom DNS resolver answers with two A records, while the Google DNS resolver (correctly) has no answer:

weberjoh@jw-nb12-lx:~/dnsdiag$ dig @192.168.7.1 weberfoobar.de

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @192.168.7.1 weberfoobar.de
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46490
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;weberfoobar.de.                        IN      A

;; ANSWER SECTION:
weberfoobar.de.         3563    IN      A       62.157.140.133
weberfoobar.de.         3563    IN      A       80.156.86.78

;; Query time: 4 msec
;; SERVER: 192.168.7.1#53(192.168.7.1)
;; WHEN: Wed Aug 31 15:24:27 CEST 2016
;; MSG SIZE  rcvd: 64

weberjoh@jw-nb12-lx:~/dnsdiag$ dig @8.8.8.8 weberfoobar.de

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @8.8.8.8 weberfoobar.de
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 53504
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;weberfoobar.de.                        IN      A

;; AUTHORITY SECTION:
de.                     1799    IN      SOA     f.nic.de. its.denic.de. 2016083157 7200 7200 3600000 7200

;; Query time: 31 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Aug 31 15:24:36 CEST 2016
;; MSG SIZE  rcvd: 95

Cheers!

Featured image: “decisions” by Martin Fisch is licensed under CC BY-SA 2.0.

I wanted to configure a weekly email report on a Palo Alto Networks firewall. “Yes, no problem”, I thought. Well, it was absolutely not that easy. ;(

While the PAN firewalls have a great GUI and a good design at all they lack an easy-to-use email reporting function, especially when compared to the FortiGate firewalls which have a great local report feature. –> If you want some stats on a weekly basis you must configure it completely from scratch. Unluckily this is not that easy since you must pass several steps for that. Therefore, I drew an outline of the Palo Alto reporting stages to have an overview of them.

I am currently running a PA-200 with PAN-OS 7.1.6.

Key Facts

• No predefined weekly report (only daily)
• For weekly reports, you must configure them manually
• Emails for a weekly report won’t have any graphs but only tables
• No possibility to generate weekly reports out of the ACC <- which is really bad, because the ACC draws very nice graphs

Palo Alto Reporting Outline

This is my sketch of the reporting menus. Hopefully it helps. The comments in red are disadvantages from my point of view. The comments in green are advantages:

Steps for a weekly Reporting Mail

In order to have a weekly reporting mail with a PDF that has some interesting stuff, proceed as follows:

1. Monitor -> Manage Custom Reports -> Add -> Load Template: For each template you are interested in, save it with a name (e.g., the same as the template to avoid confusion), check the scheduling, and set the time frame to “Last Calendar Week”. [Optional: Increase the rows/sort by value from top 10 to top 25, …]
2. Monitor -> PDF Reports -> Report Groups -> Add: Select all “Custom Reports” you just created.
3. Monitor -> PDF Reports -> Email Scheduler -> Add: Select the report group just created, an email profile and a recurrence of “Every Monday”.

Here are a few screenshots that might help:

Now, at the beginning of the week you’ll have an email with a PDF that includes all the reports over the last week.

Links

At least there are some docs from Palo Alto that describe some of the points I mentioned above:

• Whitepaper: Actionable Threat Intelligence
• Manage PDF Summary Reports
• Custom Reports

Featured image: “That was supposed to be going up, wasn’t it?” by Rafael Matsunaga is licensed under CC BY 2.0.

This is a cool and easy to use (security) feature from Palo Alto Networks firewalls: The External Dynamic Lists which can be used with some (free) 3rd party IP lists to block malicious incoming IP connections. In my case I am using two free IP lists to deny any connection from these sources coming into my network/DMZ. I am showing the configuration of such lists on the Palo Alto as well as some stats about it.

Lists

What is an external dynamic list? It is a list of known malicious sources maintained by some providers/persons on the Internet. These IP lists can be used to blacklist/block/deny connections from those sources. A good overview of such lists is “Blocklists of Suspected Malicious IPs and URLs” from Lenny Zeltser. I am currently using the following two well-known lists:

• OpenBL.org project: http://www.openbl.org/
• All Cybercrime IP Feeds by FireHOL: http://iplists.firehol.org/

While the first one is simply a list of “malicious” IPv4 addresses, the second one is a combination of other source that also include fullbogons and other entries. FireHOL shows many graphs and stats about the distribution from their listed IP addresses. Follow the link above and have a look!

What about IPv6? Well, it seems that only legacy IP is widely supported. Bad. While FireHOL does not list any statement about that, the OpenBL FAQ says: “We are fully IPv6 enabled but the lists and the reporting currently only handle IPv4 since most of our hosts do not have a IPv6 address and also because there basically are no attacks against IPv6 targets worth mentioning, at least not yet.” I am not happy with this statement at all, but I also know that it is not easy to maintain IPv6 lists due to the large address space. (Should an IPv6 blacklist block a /128, /64 or even a /48 in case of abuse?) At least the Spamhaus Project has an IPv6 list, called DROPv6.

Some further notes:

1. You should always check the quality of the list before using it. To my mind the two mentioned lists are quite “good”, however, note that they could be abused, too. Do some research about the trustworthiness before using it in your policies.
2. Both lists are only IP address lists, that is, they are useful for blocking incoming connections. For outgoing (user initiated) connections you can use URL lists rather than IP lists. Lenny mentioned a few of them in his blog post. And the Palo Alto firewall is also able to use domain and even URL lists for security policies, etc.

Usage within Palo Alto

I am currently using a PA-200 with PAN-OS 7.1.7. The blacklists are configured under Objects -> External Dynamic Lists. They are from type “IP List”. Those dynamic objects can then be used within a security policy. In my case I have added two deny policies at the very beginning of my whole ruleset. Immediately after committing the traffic log shows denied connection from various IPv4 addresses:

Some Stats

At first I was interested whether the whole blacklists are used correctly by the firewall. There are some CLI commands to see/refresh the lists such as:

weberjoh@pa> request system external-list ?
> refresh    refresh external-lists
> show       Print IPs/Domains/URLs in an external list
> url-test   test accessibility for url

I captured this screenshot from the FireHOL page that shows 17.299 entries on January 30th, 2017. In fact, exactly the same valid entries were listed in the Palo Alto dynamic list at the same time, as the following listing shows. (Note that there are not only /32 host IPv4 addresses but also bigger [bigly?] networks such as /20):

weberjoh@pa> request system external-list show type ip name dyn_firehol

vsys1/dyn_firehol:
        Next update at        : Mon Jan 30 21:00:21 2017
        Source                : https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset
        Referenced            : Yes
        Valid                 : Yes

        Total valid entries   : 17299
        Total invalid entries : 34
        Valid ips:
                0.0.0.0/8
                1.10.16.0/20
                1.32.128.0/18
                1.93.0.224
                1.116.0.0/14
                1.178.179.217
                1.179.170.7
[...]

Of course it looks quite the same with IPv6, here with the Spamhaus DROPv6 list:

weberjoh@pa> request system external-list show type ip name dyn_Spamhaus-DROPv6


vsys1/dyn_Spamhaus-DROPv6:
        Next update at        : Wed Feb 15 05:13:38 2017
        Source                : https://www.spamhaus.org/drop/dropv6.txt
        Referenced            : Yes
        Valid                 : Yes

        Total valid entries   : 19
        Total invalid entries : 4
        Valid ips:
                2a07:5807::/32
                2a06:2a00::/29
                2a06:e480::/29
                2a06:4740::/29
                2a06:df80::/29
                2402:6680::/32
                2a0a:c00::/29
                2a06:f680::/29
                2404:e180::/32
                2a07:5780::/29
                2a06:d240::/29
                2a00:dfe0::/29
                2a07:5786::/32
                2607:f2d0::/32
                2803:5380:ffff::/48

Now here is a custom report that shows all denied connections during the last calendar week, sorted by count (top 5), grouped by port. Many well-known ports such as SSH, telnet, SMTP, HTTP, NTP, SNMP, etc. are scanned from different IPv4 addresses all over the world:

I really like this feature, at least for my lab where not everything is business critical. To my mind blocking some “false positives” is still better than allowing some malicious connections (false negative).

More Links

• Palo Alto: External Dynamic List
• Palo Alto: Formatting Guidelines for an External Dynamic List
• Palo Alto: Enforce Policy on Entries in an External Dynamic List
• Palo Alto: View the List of Entries in an External Dynamic List
• Palo Alto: Retrieve an External Dynamic List from the Web Server

Featured image: “Lists” by Steven Lilley is licensed under CC BY-SA 2.0.

I migrated an old Juniper SSG ScreenOS firewall to a Palo Alto Networks firewall. While almost everything worked great with the Palo (of course with much more functionalities) I came across one case in which a connection did NOT work due to a bug on the Palo side. I investigated this bug with the support team from Palo Alto Networks and it turned out that it “works as designed”. Hm, I was not happy with this since I still don’t understand the design principle behind it.

However, it was a specific and not business critical case: One Palo Alto firewall with two ISP connections using a destination network address translation (DNAT, an old IPv4 problem) and policy based forwarding (PBF) with the same destination ports. Following are some more details:

(This problem appeared in April 2016. I was running a PA-200 with PAN-OS 7.1.0. After the support team told me that it works as designed, I built a workaround and did not follow this case anymore.)

The Problem

Refer to the following figure with the two different connections (red and green):

The following NAT policy forwards incoming IPv4 connections (from ISP1 on eth1/1 for some destination ports, in my case 33004) to the private DMZ IPv4 address:

And the following PBF policy forwards outgoing IPv4 connections (from the DMZ) with the same range of destination ports (in my case again 33004) to ISP2 on eth1/2:

While outgoing connections through the PBF to eth1/2 worked anytime, incoming connections through the DNAT did NOT work while the PBF rule was enabled, see the “aged-out” connections:

The core problem was the returning path for incoming connections (red, eth1/1) which was hit by the PBF policy though it had NOT a destination port of 33004 but some other higher ports. This should not be the case. The rule was hit since the returning traffic had (of course) a source port of 33004. That is, the returning packets which should flow out of eth1/1 back to ISP1 were re-routed by the PBF rule to eth1/2. In fact, neither the NAT nor the PBF policy clearly specify which type of port (source or destination) it is using. There is only one single “service” that must be chosen, which should be always the destination port to my mind. (In case of the Juniper ScreenOS firewall there was a distinction between source and destination port.) [UPDATE] As Dmitry noted in the comment section (thanks!), the service objects are indeed configured with source and/or destination ports. However, my service objects were only configured with destination ports. That is, the hit by the PBF rule was still not correct. [/UPDATE]

The PAN support asked their engineering about this behaviour and gave me the following reply (which I don’t fully understand): “The reason we do not switch port when we do s2c direction pbf lookup is because we have some use cases that need pbf in s2c direction only. In these cases, we use either service type or destination port of c2s flow to do s2c lookup. That’s why the s2c flow hit the pbf rule.” This was captured under bug number 95540 but not fixed since it already works as designed.

Conclusion

Yes, almost everything that worked on a Juniper ScreenOS firewall (or any other firewall) also works on a Palo Alto firewall. But only “almost everything” since I had no problems with this same situation on the old ScreenOS firewall but only on the Palo.

Featured image: “Down Low” by André Hofmeister is licensed under CC BY-NC 2.0.