Customer may report that their:

• SVM and VPX eventually become unresponsive
• GUI access is unavailable for SVM and VPX
• SSH access only
• SVM reporting both the instances as Yellow
• Instance state -Yellow (Out of service) message

This issue can be validated by reviewing the following counters getting incremented.

1. ssl_err_coleto_card_threshold
2. ssl_err_card_process_fail_rst
3. ssl_err_coleto_rsadec_pvtkey_submit
4. ssl_err_coleto_*_submit, where * is any text in that counter name, such as masterkey

This problem  started when client connections became high.
SSL connections were dropped due to ssl_err_coleto_card_threshold, so users couldn't access with HTTPS. We have reached the threshold for ssl_cur_sslInfo_nsCardInQCount and it caused the ssl_err_coleto_card_threshold. The CardInQCount needed to be decreased after numbers of client connections were decreased. However, CardInQCount was never decreased and users couldn't access with HTTPS.

• You can confirm this via the CLI using the following command
• Confirm if counter are increasing and if it’s a match to TSK0708375

nsconmsg120 -K newnslog -g ssl_err_card_process_fail_rst  -s disptime=1 -d current | more
nsconmsg -K newnslog -g ssl_err_coleto -s disptime=1 -d current | egrep --line-buffered '_submit'| more
nsconmsg111 -K newnslog.143 -g ssl_cur_sslInfo_nsCardInQCount -s disptime=1 -d current
 
 127 1351000         327847          1        0 ssl_cur_sslInfo_nsCardInQCount  Fri Apr 13 02:44:55 2018
 128   84000         327848          1        0 ssl_cur_sslInfo_nsCardInQCount  Fri Apr 13 02:46:19 2018
 129   28000         327849          1        0 ssl_cur_sslInfo_nsCardInQCount  Fri Apr 13 02:46:47 2018

The Coleto card is becoming unresponsive because of malformed/bad/unexpected packet submitted to card.. As it has single card, no further handshakes are successful. All connections are blocked in card initially and later in few minutes’ software queue buildup happened.

Upgrade to the latest VPX/MPX firmware:  
 12.0.58.9_nc
 12.1+.Build
 11.1  build 59 +
NOTE: On SDX, the VPX Instance, not the SDX, must be upgraded.
NOTE: Not all SDX/MPX hardware uses the Coleto SSL card. Many use a Cavium SSL card and thus do not hit this bug. See here for which hardware uses the Coleto SSL Card.

Workaround:
Rebooting the SDX/MPX appliance will bring the SVM and VPX back online

---

Problem Cause

Coleto card is not responsive because of malformed/bad/unexpected packet submitted to card. As it has single card, no further handshakes are successful. All connections are blocked in card initially and later in few minutes software queue buildup happened.

1. Unable to access SVM via GUI
2. Unable to access VPX Instance Via GUI or SSH
3. Rebooting the SVM / VPX will NOT resolve the issue, however rebooting the SDX or MPX will resolve the issue.

VPX ON SDX/MPX SSL card failure, SVM / VPX in yellow state no GUI access or connections

Check newnslogs

collector_abbr_S_10.151.88.17_29Oct2018_11_35/var/nslog]$ nsconmsg120 -K newnslog -g ssl_err_card_  -d stats
Displaying current counter value information
NetScaler V20 Performance Data
NetScaler NS12.0: Build 57.24.nc, Date: Apr 13 2018, 12:06:28
reltime:mili second between two records Mon Oct 29 05:57:03 2018
Index reltime     counter-value symbol-name&device-no
    1       0               410 ssl_err_card_process_fail_rst
    3       0                 0 ssl_err_card_process_resp_fail_rst

reltime:mili second between two records Mon Oct 29 05:57:03 2018
Index reltime     counter-value symbol-name&device-no
  595       0                 0 ssl_err_coleto_ecdsa_verify_pub_coordinates
  597       0                 0 ssl_err_coleto_ecdsa_verify_submit
  599       0                 0 ssl_err_coleto_encfin
  601       0                 2 ssl_err_coleto_encmsgdp_submit
  603       0                 0 ssl_err_coleto_enc_msg
  605       0               312 ssl_err_coleto_expected_finmismatch
  607       0                 0 ssl_err_coleto_findecdp_submit
  609       0                 0 ssl_err_coleto_finencdp_submit
  611       0                 0 ssl_err_coleto_force_mon_requests
  613       0                12 ssl_err_coleto_keyblock_submit
  615       0              1674 ssl_err_col

---------

/upload/ftp/78466979/SDX/collector_abbr_S_10.151.88.15_29Oct2018_03_33/var/nslog]$ nsconmsg -K newnslog -g ssl_err_coleto -s disptime=1 -d current | egrep --line-buffered '_submit'| more 
 
      2  552995              1          1        0 ssl_err_coleto_masterkey_submit  Mon Oct 29 05:08:01 2018
      3   63000              2          1        0 ssl_err_coleto_masterkey_submit  Mon Oct 29 05:09:04 2018
      4   21000              3          1        0 ssl_err_coleto_masterkey_submit  Mon Oct 29 05:09:25 2018
      5   14000              6          3        0 ssl_err_coleto_masterkey_submit  Mon Oct 29 05:09:39 2018
      6   35000              7          1        0 ssl_err_coleto_masterkey_submit  Mon Oct 29 05:10:14 2018
    399   42000            469          1        0 ssl_err_coleto_masterkey_submit  Mon Oct 29 08:10:35 2018
    400   21000            470          1        0 ssl_err_coleto_masterkey_submit  Mon Oct 29 08:10:56 2018
    401   14000            472          2        0 ssl_err_coleto_masterkey_submit  Mon Oct 29 08:11:10 2018

• Check SVM logs 

Method: GET, URL: https://10.151.88.17/nitro/v1/stat/ns?format=json
Sunday, 28 Oct 18 20:29:11.828 -0700 [Error] [Stat[#2]] https://10.151.88.17/nitro/v1/stat/ns?format=json, Reason: SSL Exception: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
Sunday, 28 Oct 18 20:29:11.846 -0700 [Debug] [Stat[#2]] Sending Message to SYSOP /tmp/mps/ipc_sockets/mps_sysop_sock:{ "errorcode": 0, "message": "Done", "is_user_part_of_default_group": true, "skip_auth_scope": true, "message_id": "", "resrc_driven": true