The backend server in this case is an Apache server. When using DH server params on the server with a 4096 bit DH key, the SSL handshake from NetScaler fails.
The following error is also noticed:
Counter : ssl_err_Backend_ssl3_server_keysize_gt_2048 increments.
Complete the following steps to resolve this issue:
Remove the DH parameters from Apache server SSL configuration. For more information refer to - https://www3.trustwave.com/support/kb/KnowledgebaseArticle14785.aspx
DH encryption was configured on the backend Apache server. This is not a problem usually, except that the Apache server was using a 4096 bit DH key to perform a key exchange. This is currently not supported on NetScaler.