Virtual patching (VP) has been one of the most popular trends in application protection in recent years. Implemented at the level of a web application firewall, VP allows protecting web applications against exploitation of previously defined vulnerabilities. (For our purposes, a web application firewall, or WAF, will refer to a dedicated solution operating on a separate node between an external gateway and web server.)
In short, VP works by taking the results of static application security testing (SAST) and using them to create rules for filtering HTTP requests on the WAF. The problem, though, is that SAST and WAFs rely on different application presentation models and different decision-making methods. As a result, none of the currently available solutions do an adequate job of integrating SAST with WAFs. SAST is based on the white-box model, which applies formal approaches to detect vulnerabilities in code. Meanwhile, a WAF perceives an application as a black box, so it uses heuristics for attack detection. This state of affairs makes VP sub-optimal for preventing attacks when the exploitation conditions for a vulnerability go beyond the trivial http_parameter=plain_text_attack_vector.
But what if we could make SAST and a WAF "play nice" with each other? Perhaps we could obtain information about an application's internal structure via SAST but then make this information available to the WAF. That way we could detect attacks on vulnerabilities in a provable way, instead of by mere guessing.
Splendors and miseries of traditional VP
The traditional approach to automated virtual patching for web applications involves providing the WAF with information about each vulnerability that has been detected with SAST. This information includes:
- Vulnerability class
- Vulnerable entry point to the web application (full or partial URL)
- Values of additional HTTP request parameters necessary for the attack
- Values of the vulnerable parameter constituting the attack vector
- Set of characters or words (tokens) whose presence in a vulnerable parameter will lead to exploitation of the vulnerability.
The set of HTTP request parameters and dangerous elements of a vulnerable parameter can be defined both by bruteforcing and by using a generic function (typically based on regular expressions). Let us look at a fragment of code from an ASP.NET page that is vulnerable to XSS attacks:
By analyzing this attack vector code, we can generate a symbolic formula for the set of attack vector values:
{condition = "secret" ⇒ param ∈ { XSShtml-text }}, where XSShtml-textis the set of possible vectors of an XSS attack in the context of TEXT, as described in the HTML grammar.
This formula may yield both an exploit and a virtual patch. The descriptor of the WAF virtual patch can be used to generate filtering rules to block all HTTP requests capable of exploiting the relevant vulnerability.
Although this approach surely heads off certain attacks, it has some substantial drawbacks:
- To demonstrate any given vulnerability, SAST needs to discover just one of the possible attack vectors. But to ensure true elimination of a vulnerability, it is necessary to address all possible attack vectors. Passing such information to the WAF is difficult, because the set of vectors is not only infinite but cannot even be expressed in regular expressions due to the irregularity of attack vector grammars.
- The same is true for values of all additional request parameters that are necessary for vulnerability exploitation.
- Information regarding dangerous elements of a vulnerable parameter becomes useless if an attack vector, between the entry point and vulnerable execution point, undergoes intermediate transformations that change the context of its grammar or even its entire grammar (such as with Base64, URL, or HTML encoding, or string transformations).
Due to these flaws, VP technology—which is designed for piecemeal protection—is incapable of offering protection against all possible attacks on SAST-detected vulnerabilities. Attempts to create such "all-encompassing" traffic filtering rules often lead to blocking of legitimate HTTP requests and disrupt operation of the web application. Let us slightly modify the vulnerable code:
The only difference from the previous example is that both request parameters now undergo a transformation and the condition for the "secret" parameter is weakened until the sub-string is included back in. The attack vector formula, based on analysis of this new code, is as follows:
(CustomDecode condition) ⊃ "secret" ⇒ param ∈ (CustomDecode { XSShtml-text }).
The analyzer will derive a formula at the relevant computation flow graph (CompFG) node for the CustomDecode function to describe the Base64—URL—Base64 transformation chain:
(FromBase64Str (UrlDecodeStr (FromBase64Str argument))).
It is still possible to build an exploit on the basis of such formulas (we have considered this issue in a previous article), but the classical approach to generating virtual patches cannot be applied here for the following reasons:
- The vulnerability may be exploited only if the decoded "condition" parameter of the request contains the "secret" substring (String 12). However, this parameter's set of values is quite large and expressing this set via regular expressions is infeasible due to the irregularity of decoding functions.
- A request parameter that is, in fact, an attack vector also is decoded (String 14). Therefore, SAST cannot describe that set of dangerous elements to the WAF.
- SAST provides the WAF with full information about all transformations to which a vulnerable parameter and variables of attack conditions are subjected, from entry point to vulnerable execution point. This enables the WAF to compute argument values based on the values of the parameters of a given HTTP request.
- For attack detection, heuristics are replaced with formal methods that are based on rigorous proof of all statements and describe the exploitation conditions for any particular vulnerability in the most general case, instead of haphazardly describing a limited number of cases.
- Thus was born runtime virtual patching.
Runtime virtual patching
Runtime virtual patching (RVP) is based on the computation flow graph model used in PT Application Inspector (PT AI). The model is built using abstract interpretation of an application's code, expressed in semantics similar to conventional symbolic computations. Nodes of this graph contain generating formulas in the target language. The formulas yield the set of all allowable values associated with all data flows at the relevant execution points:
These flows are called execution point arguments. CompFG is evaluable, and thus able to compute sets of specific values for all arguments at any execution point, based on the values that have been set for input parameters.
RVP occurs in two stages, which correspond to the application lifecycle: Deployment (D) and Run (R).
Deployment stage
Before a new version of an application is deployed, the application is analyzed by PT AI. Three formulas are computed for each CompFG node that describes a vulnerable execution point:
- Conditions for reaching the vulnerable execution point
- Conditions for reaching values of all its arguments
- Sets of values of all arguments and corresponding grammars
All formula sets are grouped by the application entry point to whose control flow the vulnerability relates. The very notion of entry point is specific to each web framework supported by PT AI and is defined in the analyzer's database.
Then a report containing the list of vulnerabilities and related formulas is extracted in the form of code written in a special language based on S-expression syntax. This language describes CompFG formulas in a form that does not depend on the target language. For instance, the formula describing the value of an argument for the vulnerable point in the above code sample is as follows:
(+ (+ "Parameter value is `" (FromBase64Str (UrlDecodeStr (FromBase64Str (GetParameterData (param)))))) "`")
The formula for reaching the vulnerable point is:
(Contains (FromBase64Str (UrlDecodeStr (FromBase64Str (GetParameterData (condition))))) "secret").
The report is then uploaded to PT Application Firewall (PT AF). On the basis of the report, a binary module is generated, which can compute all the formulas contained in the report. For example, the decompiled code for computing the condition for reaching the above-mentioned vulnerable point is as follows:
To make formula computation possible, PT AF must have one of the following:
- Pre-compute database of all functions that may occur in the report
- An isolated sandbox with runtime environment for the language or platform on which the web application runs (such as CLR, JVM, or PHP, Python, or Ruby interpreter), and libraries used in the application
The first method ensures maximum speed but requires a huge volume of manual work by the WAF developers to describe the pre-compute database even if the developers restrict the scope to standard library functions). The second method allows computing all the functions that may occur in the report, but increases the time needed to process each HTTP request, because the WAF needs to access the runtime environment to compute each function. The most appropriate solution here would be to use the first approach for the most common functions while using the second approach for the rest.
It is quite possible for a formula to contain a function that the analyzer cannot process (for instance, calling a method that involves a missing project dependency or native code) and/or a function that PT AF is unable to compute (for instance, a function for reading data from external sources or the server environment). Such functions are flagged "unknown" in formulas and processed in a special way as described below.
Run stage
At the run stage, the WAF delegates processing of each HTTP request to the binary module. The module analyzes a request and detects the relevant entry point in the web application. For this point, formulas of all detected vulnerabilities are selected and then computed in a specific way.
First, formulas are computed for both conditions: 1) reaching the vulnerable point and 2) reaching values of all its arguments. In each formula, variables are substituted with values of the relevant request parameters, after which the formula value is computed. If a formula contains expressions that are flagged "unknown", it is processed as follows:
- Each "unknown" flag spreads bottom-up through the formula expression tree until a Boolean expression is found.
- In the formula, such expressions ("unknown" regions) are substituted with Boolean variables, so the Boolean satisfiability problem is solved.
- The assumption formula generates n conditions by substituting possible values of unknown regions from all the solutions found in the previous step.
- The value of each formula is computed. If at least one formula is satisfiable, the assumption is deemed satisfiable as well.
If computations show that the assumption is false, then the HTTP request in question cannot lead the application to a vulnerable point even with dangerous values of all request arguments. In this case, RVP simply returns request processing to the WAF's core module.
If attack conditions are satisfiable, the value of the argument of the vulnerable point is then computed. Algorithms used depend on the vulnerability class to which the analyzed point belongs. Their only similarity is the logic used to process formulas that contain unknown nodes: unlike assumption formulas, argument formulas cannot possibly be computed, which is immediately communicated to the WAF. Then the next vulnerable point is computed. To better flesh this out, we shall now review the most complicated algorithm, which is used for detecting injection attacks.
Detecting injections
Injections include any attacks that target the integrity of text written in a formal language (including HTML, XML, JavaScript, SQL, URLs, and file paths) on the basis of data controlled by the attacker. The attack is carried out by passing specifically formed input data to the application. When this data is "plugged in" to the target text, the boundaries of the token are exceeded and the text now includes syntactic constructions not intended by the application logic.
If a vulnerable point belongs to this attack class, its argument value is determined using incremental computation with abstract interpretation using taint analysis semantics. The idea behind this method is that each expression is computed separately, from bottom to top, while the computation results obtained at each step are additionally marked with taint intervals, given the semantics of each function and rules of traditional taint checking. This makes it possible to pinpoint all fragments that are the result of transformation of input data (tainted fragments).
For instance, for the code above and the following HTTP request parameter ?condition=YzJWamNtVjA%3d¶m=UEhOamNtbHdkRDVoYkdWeWRDZ3hLVHd2YzJOeWFYQjBQZyUzRCUzRA%3d%3d, the result of applying this algorithm to the formula of a vulnerable point argument is as follows (tainted arguments are marked in red):
The value is then tokenized in accordance with the grammar of the vulnerable point argument. If any tainted fragment matches more than one token, this is a formal sign of an injection attack (based on the definition of injection given at the beginning of this section).
Once formulas have been computed for all vulnerabilities pertaining to the current entry point, request processing is passed on to the WAF's core module together with detection results.
RVP advantages and specific features
This approach to application protection based on code analysis has a range of substantial advantages as compared to traditional VP:
- The shortcomings of traditional VP are addressed, thanks to the formal approach described above and the ability to take into account any and all intermediate transformations.
- The formal approach also completely rules out the possibility of false positives, so long as the formulas do not contain unknown nodes.
- There is no adverse impact on web application functionality, because protection is built on the functions of the application, as opposed to simply trying to work around them.
For testing the technology and confirming its effectiveness, we have developed a prototype of an integration module for PT Application Inspector and PT Application Firewall, in the form of a .NET HTTP module for IIS web server. A video of the prototype handling the code example above is on YouTube. Performance tests on around fifteen open-source content management systems (CMSs) have shown great results: the time required for processing HTTP requests with RVP is comparable to the time that it takes to process such requests with traditional (heuristic) WAF methods. The average performance hit for web applications was as follows:
- 0% for requests that do not lead to a vulnerable point
- 6–10% for requests that lead to a vulnerable point and are not an attack (depending on complexity of the grammar of the vulnerable point)
- 4–7% for requests that lead to a vulnerable point and are an attack
Despite obvious advantages over traditional VP, RVP still has several conceptual shortcomings:
- It is not possible to compute formulas that contain data from external sources absent on the WAF (including file resources, databases, and server environment).
- The quality of formulas directly depends on the quality of approximation of code fragments during analysis (including loops, recursion, and calls to external library methods).
- To describe semantics of transformation functions for the pre-compute database, some engineering work from the developers is required. The description process is difficult to automate and is prone to human error.
However, we have managed to mitigate these weaknesses by offloading some RVP functions to the application and by applying the technologies that underlie runtime application self-protection (RASP).
To be continued in Part 2.
Author: Vladimir Kochetkov
Article Link: http://blog.ptsecurity.com/2017/10/do-wafs-dream-of-static-analyzers.html