SSL connections were dropped due to ssl_err_coleto_card_threshold, so users couldn't access with HTTPS. We have reached the threshold for ssl_cur_sslInfo_nsCardInQCount and it caused the high ssl_err_coleto_card_threshold in logs.

Please check below counters if you see the similar issue. 

nsconmsg111 -K newnslog.143 -g ssl_cur_sslInfo_nsCardInQCount -s disptime=1 -d current
Displaying performance information
NetScaler V20 Performance Data
NetScaler NS11.1: Build 56.19.nc, Date: Dec 10 2017, 03:28:34


reltime:mili second between two records Thu Apr 12 12:09:42 2018
  Index   rtime totalcount-val      delta rate/sec symbol-name&device-no&time
      0 3345997           3403       3403      486 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:09:42 2018
      1    7000           7315       3912      558 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:09:49 2018
      2    7000          11221       3906      558 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:09:56 2018
      3    7000          14871       3650      521 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:10:03 2018
      4    7000          18386       3515      502 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:10:10 2018
      5    7000          22682       4296      613 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:10:17 2018
      6    7000          26808       4126      589 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:10:24 2018
      7    7000          30764       3956      565 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:10:31 2018
      8    7000          34602       3838      548 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:10:38 2018
      9    7000          38633       4031      575 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:10:45 2018
     10    7000          42482       3849      549 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:10:52 2018
     11    7000          46257       3775      539 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:10:59 2018
     12    7000          49906       3649      521 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:11:06 2018
     13    7000          53725       3819      545 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:11:13 2018
     14    7000          57855       4130      590 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:11:20 2018
     15    7000          61353       3498      499 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:11:27 2018
     16    7000          65018       3665      523 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:11:34 2018
     17    7000          68528       3510      501 ssl_cur_sslInfo_nsCardInQCount  Thu Apr 12 12:11:41 2018
     :
    127 1351000         327847          1        0 ssl_cur_sslInfo_nsCardInQCount  Fri Apr 13 02:44:55 2018
    128   84000         327848          1        0 ssl_cur_sslInfo_nsCardInQCount  Fri Apr 13 02:46:19 2018
    129   28000         327849          1        0 ssl_cur_sslInfo_nsCardInQCount  Fri Apr 13 02:46:47 2018


 nsconmsg111 -K newnslog.143 -g ssl_err_coleto_card_threshold -s disptime=1 -d current

Displaying performance information
NetScaler V20 Performance Data
NetScaler NS11.1: Build 56.19.nc, Date: Dec 10 2017, 03:28:34


reltime:mili second between two records Thu Apr 12 12:19:02 2018
  Index   rtime totalcount-val      delta rate/sec symbol-name&device-no&time
      0  300997           1397       1397      199 ssl_err_coleto_card_threshold  Thu Apr 12 12:19:02 2018
      1    7000           5455       4058      579 ssl_err_coleto_card_threshold  Thu Apr 12 12:19:09 2018
      2    7000           9987       4532      647 ssl_err_coleto_card_threshold  Thu Apr 12 12:19:16 2018
      3    7000          19321       9334     1333 ssl_err_coleto_card_threshold  Thu Apr 12 12:19:23 2018
      4    7000          30051      10730     1532 ssl_err_coleto_card_threshold  Thu Apr 12 12:19:30 2018
      5    7000          40972      10921     1560 ssl_err_coleto_card_threshold  Thu Apr 12 12:19:37 2018
      6    7000          52056      11084     1583 ssl_err_coleto_card_threshold  Thu Apr 12 12:19:44 2018
      7    7000          67964      15908     2272 ssl_err_coleto_card_threshold  Thu Apr 12 12:19:51 2018
      :
  10021    7000      125620005      26946     3849 ssl_err_coleto_card_threshold  Fri Apr 13 07:50:20 2018
  10022    7000      125645724      25719     3674 ssl_err_coleto_card_threshold  Fri Apr 13 07:50:27 2018
  10023    7000      125671142      25418     3631 ssl_err_coleto_card_threshold  Fri Apr 13 07:50:34 2018

nsconmsg111 -K newnslog -d finalstatswt0 | grep ssl_err
768709       0                82 ssl_err_alert_level_bad
768711       0                20 ssl_err_atk_ssl3_get_rcd_len_underflow
768713       0              2811 ssl_err_Backend_sessionReuse_attempt_ctx_cfgChangeFlush
768715       0               203 ssl_err_card_process_fail_rst
768717       0              1537 ssl_err_close_notify_before_estb
768719       0               384 ssl_err_coleto_decmsgdp_submit
768721       0                39 ssl_err_coleto_dec_msg
768723       0               267 ssl_err_coleto_encmsgdp_submit
768725       0               131 ssl_err_coleto_expected_finmismatch
768727       0              1627 ssl_err_coleto_keyblock_submit
768729       0            386392 ssl_err_coleto_masterkey_submit
768731       0                23 ssl_err_coleto_paderror
768733       0                36 ssl_err_coleto_pkcspadding
768735       0               856 ssl_err_dht_clone_sessionReuse_ctx_cfgChangeFlush
768737       0            351652 ssl_err_dht_serialise_called_on_not_resumable_sess
768739       0              2442 ssl_err_incomp_rec_fin
768741       0               652 ssl_err_parse_tlsext
768743       0                 4 ssl_err_rec_type_unexpected
768745       0              2169 ssl_err_send_alert_no_enc_started_rst
768747       0               335 ssl_err_sessionReuse_attempt_ctx_cfgChangeFlush
768749       0            352508 ssl_err_session_serial_invalid_len
768751       0            352508 ssl_err_session_util_failure
768753       0            501245 ssl_err_sess_cloning_no_such_core
768755       0             61412 ssl_err_ssl23_sent_rst
768757       0                19 ssl_err_ssl23_unknown_protocol
768759       0              5483 ssl_err_ssl2_protocol_disabled
768761       0             49032 ssl_err_ssl3_check_client_hello_B_OR_get_client_keyEx
768763       0                30 ssl_err_ssl3_get_change_cipher_spec
768765       0             25074 ssl_err_ssl3_get_client_hello
768767       0                 2 ssl_err_ssl3_get_client_key_exchange
768769       0                62 ssl_err_ssl3_get_msg_ivld_msg_size
768771       0               163 ssl_err_ssl3_get_rcd_wng_version
768773       0             22786 ssl_err_ssl3_protocol_disabled
768775       0               126 ssl_err_ssl_version_inappropriate_fallback
768777       0                17 ssl_err_tls12_decrypt_padding
768779       0                18 ssl_err_tls1_protocol_disabled
768781       0               120 ssl_err_tlsext_dup_scsvcipher
768783       0               432 ssl_err_tls_unexpected_ccs

Engineering has made code fix for the issue in 12.0 58.9+, 11.1 59.0004+ builds. Please upgrade to 12.0 58.15 or 11.1 59.X(releasing in Aug 2018) build to have fix for the issue.

---

Problem Cause

Known Fixed issues # 708375, 709406, 708978, 708923, 711264, 711404 The symmetric operations fail because the SSL card becomes unresponsive.This was caused because coleto card(SSL card) went in bad state due to corruption caused by some ssl request and was not able to recover before the reboot.

All SSL-Websites or SSL-Servicegroups stopped working. At the same time connecting via SSH, Remote Console or plain HTTP did work. Only SSL-websites were affected. The command "show ha node" showed "SSL Card Status: UP" as well.