Problem
Last week I came across a DHCP problem. Devices from certain VLAN were not getting DHCP assigned IP.
Even though firewall policy should have accepted those packets, tcpdump showed Checkpoint gateway was not forwarding them to the destination.
To capture DHCP traffic:
# tcpdump -n -i <interface> port 67 or port 68 -eIncoming interface DHCP tcpdump:
# tcpdump -n -i bond3.45 port 67 or port 68 -e
09:21:26.963685 ee:f7:24:33:9b:b6 > 00:11:7f:66:8b:39, ethertype IPv4 (0x0800), length 349: 10.100.0.254.bootps > 192.168.100.5.bootps: BOOTP/DHCP, Request from b4:18:d1:eb:c0:27, length: 307
09:21:26.963962 00:11:7f:66:8b:39 > 00:55:56:88:6c:66, ethertype IPv4 (0x0800), length 349: 10.100.0.254.bootps > 192.168.100.6.bootps: BOOTP/DHCP, Request from b4:18:d1:eb:c0:27, length: 307
09:21:26.986389 ee:f7:24:33:9b:b6 > 00:11:7f:66:8b:39, ethertype IPv4 (0x0800), length 349: 10.100.0.254.bootps > 192.168.100.5.bootps: BOOTP/DHCP, Request from 20:82:c0:20:e4:f4, length: 307
09:21:26.986808 00:11:7f:66:8b:39 > 00:55:56:88:6c:66, ethertype IPv4 (0x0800), length 349: 10.100.0.254.bootps > 192.168.100.6.bootps: BOOTP/DHCP, Request from 20:82:c0:20:e4:f4, length: 307
09:21:26.987345 ee:f7:24:33:9b:b6 > 00:11:7f:66:8b:39, ethertype IPv4 (0x0800), length 372: 10.100.0.254.bootps > 192.168.100.5.bootps: BOOTP/DHCP, Request from 4c:74:03:70:aa:0f, length: 330
...
Outgoing interface DHCP tcpdump:...
# tcpdump -n -i bond3.48 port 67 or port 68 -e
09:21:29.708143 00:11:7f:66:8b:39 > 00:55:56:88:6c:66, ethertype IPv4 (0x0800), length 347: 10.100.0.254.bootps > 192.168.100.6.bootps: BOOTP/DHCP, Request from dc:86:d8:05:88:75, length: 305
09:21:29.708653 00:11:7f:66:8b:39 > 00:55:56:88:6c:66, ethertype IPv4 (0x0800), length 345: 10.100.0.254.bootps > 192.168.100.6.bootps: BOOTP/DHCP, Request from 18:65:90:7a:ad:1a, length: 303
09:21:29.709181 00:11:7f:66:8b:39 > 00:55:56:88:6c:66, ethertype IPv4 (0x0800), length 345: 10.100.0.254.bootps > 192.168.100.6.bootps: BOOTP/DHCP, Request from b0:e2:35:d0:09:f6, length: 303
09:21:29.710100 00:11:7f:66:8b:39 > 00:55:56:88:6c:66, ethertype IPv4 (0x0800), length 366: 10.100.0.254.bootps > 192.168.100.6.bootps: BOOTP/DHCP, Request from ec:1f:72:2d:f9:09, length: 324
09:21:29.710541 00:11:7f:66:8b:39 > 00:55:56:88:6c:66, ethertype IPv4 (0x0800), length 342: 10.100.0.254.bootps > 192.168.100.6.bootps: BOOTP/DHCP, Request from 64:9a:be:28:f7:eb, length: 300
...So, this is the network scheme of the problem:
Why are packets sent to destination “192.168.100.5” (DHCP Server 1) dropped??
The answer can be found using zdebug:
# fw ctl zdebug drop | grep 10.100.0.254
;[vs_2];[tid_3];[fw4_3];fw_log_drop_ex: Packet proto=17 10.100.0.254:67 -> 192.168.100.5:67 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
;[vs_2];[tid_3];[fw4_3];fw_log_drop_ex: Packet proto=17 10.100.0.254:67 -> 192.168.100.5:67 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
;[vs_2];[tid_3];[fw4_3];fw_log_drop_ex: Packet proto=17 10.100.0.254:67 -> 192.168.100.5:67 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
;[vs_2];[tid_3];[fw4_3];fw_log_drop_ex: Packet proto=17 10.100.0.254:67 -> 192.168.100.5:67 dropped by fw_handle_old_conn_recovery Reason: UDP packet that belongs to an old session;
Solution
I applied SK103598 scenario 2 to try to fix the issue:
Follow these steps in SmartDashboard:
- Open the object of the involved Security Gateway – go to ‘Advanced’ / ‘Other’ pane – go to ‘Connection Persistence’ – select ‘Keep all connections’ – click on ‘OK’.
- Install policy on the involved Security Gateway.
After confirming this didnt solve the issue, I tried to clear the connection table by executing the command:
# fw tab -t connections -x
(Deletes all connections from all cluster members simultaneously)
Finally, after running it, the Checkpoint gateway stopped dropping the packets and clients started to get assigned IP addresses.
