Large File Uploads Fails on NetScaler with Content Length 0 POST Requests

Large File Uploads Fails on NetScaler with Content Length 0 POST Requests

book

Article ID: CTX225681

calendar_today

Updated On:

Description

If any large file upload is being failed in 11.1 54+  for HTTP POST streams and if there are any requests with content length "0" prior to the actual POST request sent by NetScaler before sending actual large POST request,  the back-end server may reject the request as the server might not be configured to respond to HTTP requests with content length "0".

Analysis
POST /traveler?s=USERNAME&action=sync&orig=dm&deviceId=Android_12345678gh7 HTTP/1.1
Accept-Charset: UTF-8
Content-Type: application/vnd.syncml+wbxml
Cache-Control: private
Connection: Keep-Alive
User-Agent: Lotus Traveler Android 9.0
Content-Encoding: deflate
ntCoent-Length: 90977
Accept-Language: de-AT, en-US
Accept-Language: de-AT, en-US
Host: abcd.xyz.com
Cookie: DomAuthSessId=73BDE6B8B8AECA6ECB3D649F951AA4BD; SessionID=D4EE8D6F33BC734B08D6887956530A8A5196909D; NSC_wt_mc00_usbwfmfs=ffffffffc3a0b60745525d5f4f58455e445a4a423660
Accept-Encoding: identity
Content-Length: 0
 
HTTP/1.1 400 Bad Request
Server: Application-Server-Name
Date: Sun, 02 Jul 2017 07:23:26 GMT
Connection: close
Pragma: no-cache
Cache-Control: no-store
X-Application-Server-HTTP-TIME: 16
X-Application-Serve-ERROR-MESSAGE: Action sync or Content-Encoding deflate requires data, and there is no data.
X-Application-Serve-HOST: 5050
Content-Type: text/plain;charset=UTF-8
Content-Length: 41
Error: "Unnecessary requirement for Application-Serve-Name."

Note 1: The error codes and details mentioned above may change based on the environment/configuration, as these are dependent on the Application server. However, the Application behavior and the symptoms may be the same as mentioned in this article.

NOTE 2

If POST request contains a Content-Length header, then NetScaler will garble it to 'ntCoent-length"

Also, sometimes the original POST does not have Content-Length defined but Transfer-Encoding: Chunked.  In that case, we will proxy the POST using Content-Length: 0, but we will garble the Transfer-Encoding as either Tnsrafer-Encoding or anTrsfer-Encoding

Resolution

We have nsapimgr knob to handle this issue.

The following knob will disable the new functionality to handle large POST request.
nsapimgr_wr.sh -ys arg1=0 -ys arg2=1 -ys arg3=16 -ys call="set_sso_post_data_handler"

Rollback command for the same is given below:
nsapimgr_wr.sh -ys arg1=1 -ys arg2=1 -ys arg3=16 -ys call="set_sso_post_data_handler"

Note: 1 The command should be executed from NetScaler's shell prompt. Also, add this command to the rc.netscaler file to make it persistent even after reboot of NetScaler. The below article outlines the process on how to make the change in rc.netscaler file: https://support.citrix.com/article/CTX122271

Problem Cause

Why 0 length request are being forwarded by the NetScaler?

Special handling is done to overcome the memory build-up issue when client tries to upload large POST data to backend server.

NetScaler 11.1 54+ handle special case for POST request from client in SSO module. where Netscaler sends additional POST request with content length 0 to backend server in case of large file uploading to avoid failure during large file size document upload.

In many cases where NetScaler is sending 0 length packets but backend server is not expecting content-length 0 POST requests, it may cause an issue returning the error from Backend server and terminating the session.

Issue/Introduction

Sometime Large file upload >1 MB are failing as Backend server responds with BAD request response, If the issue is first occurred with a LENGTH 0 PACKET created by Netscaler and sent to backend and Backend responds with an error Message it may cause the request to be failed.As some servers don't respond to a ZERO length packet.

Additional Information

Using Netscaler DEBUG level SYSLOG messages, we can see the following message to indicate the behavior:
"SSO: Special Post request SSO handling initiated for session-id:55555 content-length 40KB"
Session ID and Content-Length will vary.

If POST request contains Content-Length header, then NetScaler will garble it to 'ntCoent-length"

Also, sometimes the original POST does not have Content-Length defined but Transfer-Encoding: Chunked.  In that case, we will proxy the POST using Content-Length: 0, but we will garble the Transfer-Encoding as either Tnsrafer-Encoding or anTrsfer-Encoding