The Field Formats check verifies the data that users send to your web sites in web forms. It examines both the length and type of data to ensure that it is appropriate for the form field in which it appears. If the application firewall detects inappropriate web form data in a user request, it blocks the request.

By preventing an attacker from sending inappropriate web form data to your web site, the Field Formats check prevents certain types of attacks on your web site and database servers. For example, if a particular field expects the user to enter a phone number, the Field Formats check examines the user-submitted input to ensure that the data matches the format of a phone number. If a particular field expects a first name, the Field Formats check ensures that the data in that field is of a type and length appropriate for a first name. It does the same thing for each form field that you configure it to protect.

This check applies to HTML requests only. It does not apply to XML requests. You can configure Field Format checks in HTML profiles or Web 2.0 profiles to inspect HTML payload for protecting your applications. The application firewall also supports Field Format Check protection for Google Web Toolkit (GWT) applications.

The Field Formats check requires that you must enable one or more actions. The application firewall examines the submitted inputs and applies the specified actions.

You have the option to set the default field formats to specify Field Type, and minimum and maximum length of data expected in each form field on each web form that you want to protect. You can deploy relaxation rules to configure a Field Format for an individual field of a specific form. Multiple rules can be added to specify the field name, the action URL, and Field Formats to accept different types of inputs in different form fields. Learning allows you to get recommendations for the relaxation rules.

Note: The application firewall’s learning engine can distinguish only the first 128 bytes of the name for learning. If a form has multiple fields with names that match for the first 128 bytes, the learning engine might not be able to distinguish between them. Similarly, the deployed relaxation rule might inadvertently relax all such fields.

Default Field Format - In addition to configuring the actions, you can configure the default Field Format to specify the type of data expected in all the form fields for your application. A Field Type can be selected as the Field Format type. Minimum length and Maximum length parameters can be used to specify the length of the allowed inputs.

• Field Type - Field Types are named expression to which you assign assigned priority values. Field Type expressions specify the allowed inputs and are matched against the submitted data to determine whether the received values are consistent with the allowed values. The Field Types are checked in the order of their priority numbers. A lower number indicates a higher priority. The application firewall gives you the option to add your own Field Types and assign them the priorities you want. The priority value can range from 0 through 64000. The following built-in Field Types are provided to help simplify the configuration process:
  > sh appfw fieldtype

  1)      Name:  integer           Regex:  "^[+-]?[0-9]+$"
                                Priority:  30            Comment:  Integer
                                Builtin:  IMMUTABLE
  2)      Name:  alpha             Regex:  "^[a-zA-Z]+$"
                                Priority:  40            Comment:  "Alpha characters"
                                 Builtin:  IMMUTABLE
  3)      Name:  alphanum          Regex:  "^[a-zA-Z0-9]+$"
                                Priority:  50            Comment:  "Alpha-numeric characters"
                                Builtin:  IMMUTABLE
  4)      Name:  nohtml            Regex:  "^[^&<>]*$"
                                Priority:  60            Comment:  "Not HTML"
                                Builtin:  IMMUTABLE
  5)      Name:  any               Regex:  "^.*$"
                                 Priority:  70            Comment:  Anything
                                Builtin:  IMMUTABLE
                 Done
  > 

  Note: The built-in Field Types are IMMUTABLE and cannot be modified or removed. Any Field Types that you add are MODIFIABLE. You can edit them or remove them. 

  Configuring a Field Type as a default Field Format might be useful when you have a PCRE expression that can identify the valid inputs in all or most of the form fields for your application and exclude the invalid inputs. For example, if all the inputs in your application forms are expected to contain only numbers and letters, you might want to use the built-in Field Type alphanum as the default Field Type. Any non-alphanumeric character such as a backslash (\) or semicolon (;) in the input will trigger a violation. You can also add your own customized Field Types and use them to configure default Field Formats. For example, if you want to allow only the lowercase "x", "y", and "z" as the allowed alpha characters, you can configure a customized Field Type with regular expression "^[x-z]+$". You could assign it a higher priority (lower priority number) then the built-in Field Types and use it as the default Field Type.

• Minimum Length - The default minimum data length assigned to form fields in web forms that do not have an explicit setting. This parameter is set to 0 by default, which allows the user to leave the field blank. Any higher setting forces users to fill in the field.
  Caution! If the minimum length value is 0 but the Field Type is integer, alpha, or alphanum, a request is blocked if any input field is left empty, despite the minimum length setting. That is because the regEx for these Field Types contains a + character which means one or more characters. Distinguishing an integer from an alpha character requires at least one character.

• Maximum Length - The default maximum data length assigned to form fields in web forms that do not have an explicit setting. This parameter is set to 65535 by default.
  Note: Characters vs bytes - The minimum and the maximum lengths for the field formats represent the number of bytes, not the number of characters. Languages that have greater than one-byte character representation can cause the limit to be exceeded with fewer characters than the number configured for the maximum value. For example, with double-byte character representation, the maximum value of 9 allows no more than 4 characters.
  Tip: The configuration utility allows you to cut and paste UTF-8 characters directly into the GUI without having to convert them to hex. 

• Character-Maps: In addition to recommending the Field Types, application firewall learning engine offers you an additional option, Use Character Maps, to deploy the Format Check rules. A Character Map is a set of all the characters allowed in a particular form field. You can fine tune the Field format specification to allow or disallow specific characters by using Character Maps. A separate Character Map is generated for each form field. The alpha and numeric characters are treated differently in Character Maps. If any alpha character is seen, all alpha characters [A-Za-z] will be allowed by the recommended PCRE expression. Similarly, if any digit is seen, all digits [0-9] will be allowed. Non-printable characters are specified by using the \x construct. Only single Byte characters with values between 0-255 are considered for Character Map recommendations.

  A Character Map can be more specific than the corresponding Field Type recommendation. In some situations, Character Maps might be a better option, because they give you tighter control over the set of characters allowed as inputs. The deployed character maps are displayed as a string that starts with prefix "CM" followed by digits. The priority for the Character Maps starts at 10000. As with user-added Field Types, you can add, edit or remove a Character Map. Character Maps that are currently used in deployed rules cannot be modified or removed.
  Note: Character Maps are not supported in cluster deployments.

Using the Learn Feature with the Field Formats Check

When the learn action is enabled, the application firewall learning engine monitors the traffic and learns the triggered violations. You can periodically inspect these learned rules. After due consideration, you can deploy the learned rule as a Field Format relaxation rule.

Field Formats Learning enhancement: The application firewall learning enhancement was introduced in 11.0. In the previous releases, once the learned field format recommendation is deployed, the application firewall learning engine stops monitoring the valid requests for the purpose of recommending new rules on the basis of the new data points. This limits the configured security protection, because the learning database does not include any representations of the new data seen in the valid requests processed by the security check.

Violations are no longer coupled with learning. The learning engine learns and makes recommendations for the field formats regardless of the violations. In addition to checking the blocked requests to determine whether the current field format is too restrictive and needs to be relaxed, the learning engine also monitors the allowed requests to determine whether the current field format is too permissive, and allows elevating the security by deploying a more restrictive rule.

The following is the summary of the Field Formats learning behavior:

No Field format is bound: The behavior remains unchanged in this scenario. All learn data is sent to the aslearn engine. The learning engine suggests a field format rule based on the data set.

Field format is bound: In the previous releases, observed data is sent to the aslearn engine only in the case of a violation. The learning engine suggests a field format rule based on the data set. In the 11.0 release, all data is sent to aslearn engine even if no violation is triggered. The learning engine suggests a field format rule based on the entire data set of all received inputs.

Use Case for Learning Enhancement

If the initial field format learned rules are based on a small sample of data, a few non typical values might result in a recommendation that is too lenient for the target field. The ongoing learning allows the application firewall to observe data points from every request to collect a representative sample for the learned recommendations. This is helpful in further tightening the security to deploy the optimal input format with an adequate range value.

The field format learning makes use of the priority of the Field Types as well as the configured settings of the following learning thresholds:

• FieldFormatMinThreshold - Minimum number of times a specific form field must be observed before a learned relaxation is generated. Default: 1.

• FieldFormatPercentThreshold - Percentage of times a form field matched a particular Field Type, before a learned relaxation is generated. Default: 0.

The field format rule recommendations are based on the following criteria:

• Field Type recommendations - The Field Type recommendations are determined by the assigned priorities of the existing Field Types and the specified Field Format thresholds. The priorities determine the order in which the Field Types are matched against the inputs. A lower number specifies a higher priority. For example, Field Type integer has the higher priority (30) and is therefore evaluated before Field Type alphanum (50). The thresholds determine the number of inputs evaluated to collect a representative sample for the data point. Assigning the right priority to the configured Field Types, and configuring an appropriate learningsetting value for the fieldFormatPercentThreshold and fieldFormatMinThreshold parameters, is essential for getting the correct Field Format recommendation. The Field Type with the highest priority, based on the configured thresholds, is matched first against the inputs. If there is a match, this Field Type is suggested without considering the other Field Types. For example, three default Field Types, integer, alphanum, and any will match if all the inputs contain only numbers. However, integer will be recommended since it has the highest priority.

• Minimum and Maximum length recommendations - Calculations for the minimum and maximum lengths for the Field Format is done independently of the determination for the Field Type. The field format length calculations are based on the average length of all the observed inputs. Half of this calculated average is suggested as the min value, and twice the value of this average is suggested as the max value. The range for the Minimum Length is 0-65535 and the range for the Maximum Length is 1-65535. The configured value for the minimum length cannot exceed the maximum length.

• Handling of space character - The Field Format check counts every space character when checking for the Field Formats length. Leading or trailing spaces are not stripped, and multiple consecutive spaces in the middle of the input string are no longer consolidated to a single space during input processing.

The following is an example to illustrate the Field Format recommendations:

Total requests: 100
Number of Req with Field Type:
Int : 22                                (22 int values) – 22%
Alpha : 44                           (44 alpha values) – 44%
Alphanum: 14                    (14 + 44 + 22 = 80 alphanum values) = 80%
noHTML: 10                       (80 + 10 = 90 noHTML values) = 90%
any : 10                               (90 + 10 = 100 any values) = 100%
 
% threshold                              Suggested Field Type
0-22                                                    int
23-44                                                  alpha
45-80                                                  alphanum
81-90                                                  noHTML
91-100                                               any

Deployment Tip

• Enable Field format actions log, learn and stats.

• After running a representative sample of the traffic to your application, review the learned recommendations.

• If a Field Type is recommended by most of the learned rules, configure that Field Type as the Default Field Type. For minimum and maximum lengths, use the widest range suggested by these rules. 

• Deploy rules for other fields for which different Field Types or different minimum/maximum lengths are better suited.

• Enable blocking and disable learning.

• Monitor stats and logs. If a significant number of violations are still being triggered, you might want to review the log messages to confirm that the violations represent malicious requests that should have been blocked. If valid requests are being flagged as violations, you can either edit the configured Field Format rule to further relax it or enable learning again to get recommendations based on the new data points.
  Note: You can fine tune your configuration by getting new learning recommendations.

Highlights

Note the following points about the Field Format security check:

• Protection - By configuring optimal field format rules, you can protect against many attacks. For example, if you specify that a field can only have integers, hackers will not be able to launch SQL Injection or XSS attacks by using this field, because the inputs required to launch such attacks will not meet the configured field format requirement.

• Performance - You can limit the minimum and the maximum allowed length for the inputs in the field format rules. This can prevent a malicious user from entering excessively large input strings in an attempt to add processing overhead to the server, or worse, cause the server to dump core because of stack overflow. By limiting the input size, you can shorten the time required for processing legitimate requests.

• Configuring Field Formats - You must enable one of the actions (block, log, stats, learn) to engage the field Format protection. You can also specify the Field format rules to identify the allowed inputs in your form fields.

• Selecting Character Maps vs. Field Types - Both Character Maps and Field Types use regular expressions. However, a Character Map provides a more specific expression by narrowing down the list of allowed characters. For example, for an input such as janedoe@citrix.com, the learning engine might recommend the Field Type nohtml but the Character Map [.@-Za-z] might be more specific, because it narrows down the allowed set of non-alpha characters. The Character Map option allows, in addition to alpha characters, only two non-alpha characters: period (.) and at (@).

• Ongoing Learning - The application firewall monitors and takes into account all the incoming data (violations as well as allowed inputs) to build a learning table for recommending rules. The rules are revised and updated as new incoming data arrives. New field format rules are suggested for a field even if it already has a bound field format rule. If the configured Field Formats are too restrictive and are blocking the valid requests, you can deploy a more relaxed Field Format. Similarly, if the current Field Formats are too generic, you can further refine and tighten the security by deploying a more restrictive Field Format.

• Overwriting Rules - If a rule has already been deployed for a field/URL combination, the GUI allows the user to update the field format. A dialog box asks for confirmation to replace the existing rule. If you are using the command line interface, you have to explicitly unbind the previous binding and then bind the new rule.

• Multiple match - If multiple field formats match a given field name and its action URL, the application firewall arbitrarily selects one of them to apply.

• Buffer boundary - If a field value extends across multiple streaming buffers, and the format for these two parts of the field value is different, a field format corresponding to "any" is sent to the learn database.

• Field Format vs. Field Consistency Check - Both the Field Format check and the Field Consistency check are form-based protection checks. The Field Formats check provides a different type of protection than does the Form Field Consistency check. The Form Field Consistency check verifies that the structure of the web forms returned by users is intact, that data format restrictions configured in the HTML are respected, and that data in hidden fields has not been modified. It can do this without any specific knowledge about your web forms other than what it derives from the web form itself. The Field Formats check verifies that the data in each form field matches the specific formatting restrictions that you configured manually, or that the learning feature generated and you approved. In other words, the Form Field Consistency check enforces general web form security, while the Field Formats check enforces the specific rules for the allowed inputs for your web forms.