This error occurs when the key is encrypted or wrapped.

Verify the key encryption or wrapping, go to FreeBSD shell

cd /nsconfig/ssl/

grep BEGIN key.pem

-----BEGIN ENCRYPTED PRIVATE KEY-----PKCS#8 (encrypted)
-----BEGIN PRIVATE KEY-----PKCS#8 (unencrypted, but wrapped)
-----BEGIN RSA PRIVATE KEY-----PKCS#1 (unencrypted, unwrapped in traditional PKCS#1 format) <-- this is how it should be for successful import into HSM

You can also validate if the key is encrypted or wrapped using openssl reading in asn1 format (you may need to use the option "-inform" depending on key format DER, RSA, etc.)

Here are some examples of encrypted and/or wrapped key

openssl asn1parse -in key.pem
    0:d=0  hl=4 l=1212 cons: SEQUENCE
    4:d=1  hl=2 l=   1 prim:  INTEGER           :00
    7:d=1  hl=2 l=  13 cons:  SEQUENCE
    9:d=2  hl=2 l=   9 prim:   OBJECT            :rsaEncryption <-- this shows the key is wrapped

 openssl asn1parse -in key.pem 
    0:d=0  hl=4 l=1305 cons: SEQUENCE
    4:d=1  hl=2 l=  75 cons:  SEQUENCE
    6:d=2  hl=2 l=   9 prim:   OBJECT            :PBES2
   17:d=2  hl=2 l=  62 cons:   SEQUENCE
   19:d=3  hl=2 l=  41 cons:    SEQUENCE
   21:d=4  hl=2 l=   9 prim:     OBJECT            :PBKDF2
   32:d=4  hl=2 l=  28 cons:     SEQUENCE
   34:d=5  hl=2 l=   8 prim:      OCTET STRING      [HEX DUMP]:CC21D6AE9F892015
   44:d=5  hl=2 l=   2 prim:      INTEGER           :0800
   48:d=5  hl=2 l=  12 cons:      SEQUENCE
   50:d=6  hl=2 l=   8 prim:       OBJECT            :hmacWithSHA256 <-- this shows password encrypted

This software application is provided to you as is with no representations, warranties or conditions of any kind. You may use and distribute it at your own risk. CITRIX DISCLAIMS ALL WARRANTIES WHATSOEVER, EXPRESS, IMPLIED, WRITTEN, ORAL OR STATUTORY, INCLUDING WITHOUT LIMITATION WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NONINFRINGEMENT. Without limiting the generality of the foregoing, you acknowledge and agree that: (a) the software application may exhibit errors, design flaws or other problems, possibly resulting in loss of data or damage to property; (b) it may not be possible to make the software application fully functional; and (c) Citrix may, without notice or liability to you, cease to make available the current version and/or any future versions of the software application. In no event should the software application be used to support ultra-hazardous activities, including but not limited to life support or blasting activities. NEITHER CITRIX NOR ITS AFFILIATES OR AGENTS WILL BE LIABLE, UNDER BREACH OF CONTRACT OR ANY OTHER THEORY OF LIABILITY, FOR ANY DAMAGES WHATSOEVER ARISING FROM USE OF THE SOFTWARE APPLICATION, INCLUDING WITHOUT LIMITATION DIRECT, SPECIAL, INCIDENTAL, PUNITIVE, CONSEQUENTIAL OR OTHER DAMAGES, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. You agree to indemnify and defend Citrix against any and all claims arising from your use, modification or distribution of the software application.

Encrypted and/or wrapped key when trying to import it into the HSM NetScaler FIPS

Decrypt and unwrap the key to plain text before trying to import, using openssl tool

openssl rsa -in keyfile.key -out key_plain.pem --> this will decrypt and unwrap the key and output in text decoded format in his original format
openssl rsa -in key_plain.pem -check -noout

Note: the option "traditional" is required in some NetScalers, to decrypt and save in pkcs#1 format

openssl rsa -in keyfile.key -traditional -out key_plain_pkcs1.pem
openssl rsa -in key_plain_pkcs1.pem -check -noout

To validate the key is in plain text, check the key in ASN1 format, asn1parse output should look like this:

openssl asn1parse -in key_plain.pem
    0:d=0  hl=4 l=1188 cons: SEQUENCE
    4:d=1  hl=2 l=   1 prim: INTEGER           :00
    7:d=1  hl=4 l= 257 prim: INTEGER           :BD335CA0483D2C5C52D2BFD43CA1E1DA2EA5E65B6BE30.....

Re-try import the key into the HSM chip

> import ssl fipskey <key_name> -key <key_file_name> -inform PEM

What Key prams are supported?

==========================

Only 2048-bit and 3072-bit keys are supported.
SSL renegotiation using the SSLv3 protocol is not supported on the back end of an MPX FIPS appliance.
1024-bit and 4096-bit keys and an exponent value of 3 are not supported.
4096-bit server certificate isn’t supported.
4096-bit client certificate isn’t supported (if client authentication is enabled on the back-end server).
On the MPX 14000 FIPS appliances, the -exponent parameter in the import ssl fipskey command is not required while importing an external key. Exponent is detected automatically when imported, the value of the -exponent is ignored.
The NetScaler FIPS appliance does not support external keys with a public exponent other than 3 or F4.
The key can't be encrypted or wrap before importing on the MPX 14000 FIPS appliances.
You cannot import an external, encrypted key directly to an MPX 14000 FIPS appliance HSM