Problem Definition
Several users have a network design requirement to run standard IPSec-based VPN technology in concurrence with SSL-based VPN technology on a single client workstation.
The following case study explores the possibility of running concurrent instances of the Citrix Access Gateway Client (SSL VPN) and the Cisco VPN Client (IPSec VPN) and the problems that were discovered.
Environment
• Citrix Access Gateway 4.1.2 and 4.2 clients
• Cisco IPSec VPN client 4.6.00

Technical Information and Concepts
Citrix SSL VPN overview:
The Citrix Access Gateway client running on a client PC uses a network shim to intercept packets destined for the Internet at a low layer in the TCP/IP stack to capture outbound network traffic, encapsulate the data stream with secure socket layer (SSL) encryption, and resend it to the destination.

Access Gateway accomplishes this task using a technique similar to SSH port forwarding technology. Access Gateway creates a local-listen port to intercept, encrypt, and forward them.

Cisco IPSec overview:
The Cisco IPSec VPN client employs a BITS approach to intercept network traffic, encrypt it, and forward it to the remote IPSec gateway device for processing.
Unlike the Access Gateway, the Cisco IPSec VPN client does not create local listening ports for processing. It simply grabs data destined for the Internet and encrypts the payload of the packet, and where network address translation is in use, rewrites the source and destination headers appropriately. One key element in this technology is that it is designed to ignore loop back traffic that has a destination on the local machine.
Troubleshooting Methodology
Because both of the VPN clients intercept outbound packets, we noticed several conflicts depending on which client actually processed the traffic.
These results were not consistent and further investigation showed that the CPU timing was the main factor in determining which client intercepted the outbound packet.
Process Explorer was used to see which process (Access Gateway or IPSec VPN client) would intercept the traffic during each test.

If the IPSec VPN client is the first to process the packet and resubmits it to the stack, the Access Gateway sees this as a new connection creates a local listen port to intercept and process it and tries to resubmit it to the stack.
Since the IPSec VPN client is designed to ignore local traffic, it does not see the newly processed packet and ignores it.
In a TCP environment this creates a problem because the receiving end can not send an acknowledgement packet (ACK) back to the IPSec VPN client. In the absence of an ACK packet, the IPSec VPN client assumes a network failure has occurred and resends the packet which causes the process to repeat endlessly.

[Tue Oct 04 13:39:29] Processing a request for a tunnel from 192.168.0.197:8213 to 192.168.0.101:80
[Tue Oct 04 13:39:29] Making client connection....
[Tue Oct 04 13:39:29] Negotiating SSL session on connected socket 788
[Tue Oct 04 13:39:29] 50 bytes of handshake data sent
[Tue Oct 04 13:39:29] PerformClientHandshake: initialized security context, now entering client handshake loop
[Tue Oct 04 13:39:29] ClientHandshakeLoop: 2085 bytes of handshake data received
[Tue Oct 04 13:39:29] ClientHandshakeLoop: InitializeSecurityContext failed with error SEC_E_UNTRUSTED_ROOT: 'The certificate chain was issued by an authority that is not trusted' (2)
[Tue Oct 04 13:39:29] User authorized connection to server although its certificate is untrusted, retrying
[Tue Oct 04 13:39:29] 50 bytes of handshake data sent
[Tue Oct 04 13:39:29] PerformClientHandshake: initialized security context, now entering client handshake loop
[Tue Oct 04 13:39:30] ClientHandshakeLoop: 2085 bytes of handshake data received
[Tue Oct 04 13:39:30] ClientHandshakeLoop: 126 bytes of handshake data sent
[Tue Oct 04 13:39:30] ClientHandshakeLoop: 51 bytes of handshake data received
[Tue Oct 04 13:39:30] ClientHandshakeLoop: Handshake was successful
[Tue Oct 04 13:39:30] Creating SSL socket using SSPI/Schannel succeeded
[Tue Oct 04 13:39:30] TCP connection opened from 192.168.0.197:8213 to 192.168.0.101:80 [You have 1 outstanding connections]
[Tue Oct 04 13:40:08] TCP port deleted from 192.168.0.197:8213 to 192.168.0.101:80
If the Access Gateway client intercepts traffic, first and it will resubmit it to the stack. At this point the IPSec VPN client sees this as a new outbound network connection and tries to encrypt the payload. Then the packet is resubmitted to the stack where the Access Gateway sees it as a new outbound connection and reprocesses it. This scenario creates an endless loop where the packet never leaves the network.

Resolution
Because both clients use a bump in the stack approach this causes a conflict in which client should be responsible for intercepting the traffic. This conflict effectively breaks both clients and makes it impossible to run the Access Gateway and the IPSec VPN client concurrently.
Citrix is investigating ways to cooperate better with various IPSec clients, but due to the large diversity of IPSec solutions available on the market this is not a valid solution.
The best workaround would be to reevaluate the customer network needs and redesign the network implementation to segregate the use of each client independently of each other.