Problem Definition

In a NetScaler deployment involving Load Balancing and Application Firewall, users are unable to upload files to the backend application servers via HTTP.

Environment

• NetScaler with Application Firewall configured
• Custom-built client

Troubleshooting Methodology

By looking at the network traces from this particular scenario, it became apparent that NetScaler was terminating sessions (sending a TCP reset to the client) each time the client attempted to POST with multipart MIME data. Using a standard web browser, this issue was not reproducible. Only after manually POSTing the same data that was in the trace to the NetScaler, it was possible to recreate this issue.

Resolution

NetScaler was forcefully closing (resetting) the session due to a malformed POST from the custom-built client. The POST from the client did not follow RFC standards, and thus, NetScaler reset the connection.

The malformed portion of the POST from the client is the following line:
Content-Type: multipart/form-data, boundary=|---------------------------4345ec6a1a84232, charset=utf-8

The line should contain ‘;’ characters as field delimiters instead of ‘,’ characters as follows:
Content-Type: multipart/form-data; boundary=|---------------------------4345ec6a1a84232; charset=utf-8

In testing, when the field delimiters are corrected, NetScaler passes the POST data to the backend as expected.

A workaround for this issue is to put all the URLs that are affected by this into a specific Application Firewall policy. 

1. The following options will need to be turned off for the profile that is used for this policy.

   1. URL closure – This is an option under startURL. You will need to define startURL rules to allow all the URLs that are using this profile.
   2. Cookie Consistency – All options should be unchecked (block, learn, log, stats).
   3. Field consistency – All options should be unchecked.

2. Run the following command on the CLI:
   set appfw profile <profile name> -enableFormTagging OFF

Additional Information

• In RFC 2616, section 3.7, the syntax specification for the Content-Type header field is specified. 
  media-type = type "/" subtype *( ";" parameter )type = tokensub
  type = token
  This syntax is in “an augmented Backus-Naur Form (BNF) similar to that used by RFC 822.” (This is described in section 2.1 of RFC 2616.) By definition, the semicolon surrounded by double-quotes in the media-type name above is literal and required before each parameter.

• There is not a specific policy that blocks this. This is one of a few RFC violations that will cause a session to be reset for security reasons.

• It is not currently possibly to log these session resets. However, there is currently a feature request filed to do so.

This article is a case study which explains Application Firewall Resetting Connections for POST Requests