Upgrading to FortiGate 70G: Resolving IPsec VPN Sync Issues

I recently pulled the trigger on upgrading our branch firewalls from FortiGate 60E models to the new 70G series. The performance leap with Fortinet's new SP5 processor is incredible, but as with any major hardware architecture shift, the upgrade threw me a bit of a curveball.

Right after the cutover, a critical sync service running across our IPsec VPN tunnel back to HQ started failing. Packets were clearly getting dropped in transit.

I’d actually seen this exact issue before on the older 60-series boxes. In the past, I resolved it quickly by overriding the MTU (Maximum Transmission Unit) on the VPN interface to account for the tunnel overhead. My go-to fix looked like this:

config system interface
    edit VPN-NAME
        set mtu-override enable
        set mtu 1366
    next
end

I confidently dropped this same configuration onto the new 70G, expecting the sync to immediately turn green. Spoiler alert: it didn't. The packets were still dropping, and the sync kept failing.

After stepping back and looking at how the new hardware handles transit traffic and hardware offloading, I realized I needed a completely different approach. Instead of forcing an MTU limit, I needed to actively clamp the TCP payload size.

Here is the exact command block that finally resolved the issue on the 70G:

config system interface
    edit VPN-NAME
        set tcp-mss 1326
    next
end

Once I committed that change, the sync fired right back up without any failures. But it got me thinking—why did tcp-mss 1326 work instantly when mtu 1366 completely failed?

As network engineers, we tend to toss both of these terms around when troubleshooting tunnel overhead, but they operate at different layers and handle traffic in very different ways. Here is what I learned from this deployment.

Maximum Transmission Unit (set mtu 1366)

MTU operates at Layer 3 of the OSI model and dictates the absolute largest size an entire IP packet can be (headers + payload) when leaving an interface.

When you apply set mtu 1366, you are essentially putting a strict bouncer at the door of your VPN tunnel. You're telling the firewall, "Do not let a single packet larger than 1366 bytes through." It's a reactive limit. If a massive packet arrives from your sync server, the firewall has to react by either fragmenting it (chopping it into pieces, which is terrible for IPsec performance) or dropping it. Because many modern sync applications hardcode a "Do Not Fragment" (DF) bit onto their packets, the firewall's only choice is to drop the packet entirely.

To make matters trickier, on modern architectures like the 70G, an interface-level MTU override often only applies cleanly to local-out traffic (traffic generated by the firewall itself, like a ping). For transit traffic passing through the box, the hardware offloading can be unforgiving with oversized packets.

Maximum Segment Size (set tcp-mss 1326)

TCP-MSS (Maximum Segment Size) operates at Layer 4 and is beautifully proactive. Rather than defining the size of the whole packet, the MSS defines the maximum size of the TCP Payload only—ignoring the IP and TCP headers.

Unlike MTU, which waits for a packet to be too big and then punishes it, TCP-MSS clamping fixes the problem before the data is even sent.

When you set tcp-mss 1326, you are telling the firewall to actively intercept the initial TCP three-way handshake between your local server and the HQ server. The firewall rewrites the negotiated payload size on the fly. You are forcing both endpoints to agree to pack their data into smaller boxes from the very beginning. Because the servers are natively sending smaller payloads, nothing needs to be fragmented, and no packets get dropped at the tunnel entrance.

The Math Behind 1326

You might be wondering why I used 1326 for the MSS when my target MTU was 1366. It comes down to standard header sizes:

• Standard IPv4 Header: 20 bytes
• Standard TCP Header: 20 bytes
• Total Overhead: 40 bytes

If I know my maximum safe packet size (MTU) to get through the IPsec tunnel without fragmentation is 1366 bytes, I simply subtract the 40 bytes of headers.

1366 (Target MTU) - 40 (Headers) = 1326 (TCP-MSS)

The Takeaway

• MTU is a passive limit. It acts as a strict doorway; if the packet is too big, it gets chopped up or denied entry.
• TCP-MSS is proactive negotiation. It tells the sender to pack smaller boxes from the very beginning, ensuring they easily fit through the door.

Upgrading to newer firewall hardware sometimes means unlearning old habits. While forcing the MTU works in a pinch on older gear, it’s a passive and reactive fix. For modern firewalls heavily utilizing hardware offloading, proactively clamping the TCP-MSS is the cleanest, most efficient way to ensure your transit traffic flows smoothly over a VPN.