A Wild Port Scan Appears. What now?

[MalBot]

Introduction

During the RSA 2018 conference, Lastline launched Breach Defender, a new solution to facilitate the analysis of suspicious anomalies in monitored networks. As part of our internal product QA leading up to any release, we often coordinate with our partners to carry out tests on real data. During our most recent iteration, we happened to detect a port scan within the network of one of our customers (you can see a screenshot of the UI in Figure 1; the orange node represents the event). Normally we tend to gloss over port scans, although we still generate an informational event, as they are often used as part of network security policy to identify hosts running unexpected services. Overall, they are often part of the background noise, and most commonly they are just used to decorate some network activity maps.

Not Your Typical Port Scan

What was unusual in this instance was some additional suspicious activity related to rogue and malformed FTP connections (see the “Suspicious Network Interaction/FTP Based Covert Data Channel” node in Figure 1, click to enlarge). Although quite an old protocol, FTP is still frequently used to exfiltrate data (see the HawkEye keylogger for example). However, a malformed FTP connection can simply be caused by a poorly implemented client. We quickly ruled out this possibility as soon as we noticed how the events were clearly overlapping and involving the very same internal host that had launched the port scan. As visible from the graph, the very same external hosts were also the target/destination of both the port scan and the malformed FTP connection.

It definitely looked like a local host was actively looking for a way to exfiltrate data.

Figure 1: A port scan initiated by one of the local hosts (highlighted) together with some additional suspicious network interaction exploiting an FTP-based covert data channel.

Analyzing the Traffic

It was definitely time to analyze the traffic in a bit more detail. When we started to dig more in depth into the information at our disposal, more and more suspicious inconsistencies surfaced.

First, as displayed in Figure 2 (click to enlarge), our heuristics flagged the hosts as running multiple operating systems. The heuristics build upon network indicators such as user agents or remote endpoints to infer information on the software configuration of each host. The fact that the very same host appears as running two different mobile operating systems (iOS and Android) is unusual and suggests that at least some of the network activities are spoofed. For instance, an iOS application may be hardcoding an Android user agent in its HTTP requests.

Second, the FTP control connection was attempting to store and retrieve the very same file (/home/ftp/db.txt). Note username and password are blank in Figure 2: looking at the raw data, random binary characters appear in those fields, and the characters have been sanitized by our UI. Why would a malicious client want to store and retrieve the same file? Also, the two commands for uploading and downloading are being issued approximately at the same time.

Overall, it felt like something was trying hard to make it look like a legitimate FTP interaction, so we started to suspect we were dealing with something very different. Maybe a clumsy attempt to update a shared resource thereby registering a new infected machine?

Figure 2: Mobile user agents (most likely spoofed) and an anomalous FTP connection that stores and retrieves the very same file.

FTP Traffic

To collect further details related to the FTP connections, we queried our backend and sought to select all connections on port 21 outgoing from the internal host that was under investigation. We found 129 connection attempts (to 129 distinct IPs). Of these, only 13 were successful. Every successful connection translated to similar FTP transactions simultaneously attempting to upload and download a resource with the same name.

A quick check on some of the server IPs revealed that they were still responsive. However, attempting to use a normal FTP client to connect led to strange results: the server responses did not match the commands issued by the client. So rather than using a standard client, we switched to a transport level client (the Linux utility netcat) and attempted to deliver manual commands to the server. We managed to replicate the interaction we saw in Figure 2 using netcat. However, when we tried to introduce some variations, it became obvious that the FTP server dialogue, apparently legitimate, was completely scripted: no matter what input the client provided, the server responses were deterministic and “staged.” Figure 3 shows where we “netcat” into the server and type a bunch of random strings, after which the server replies as if the commands were valid.

Apparently, the client and server somehow “emulated” an FTP control channel to establish a seemingly legitimate bidirectional connection over the data channel. Once again, this behavior seemed to be indicative of an infected host trying to reach out to a C&C server using a stealthy connection.

Figure 3: Regardless of our input, the FTP interaction was always leading to the same result, a bidirectional communication channel opened on a port decided by the server (in this instance 42630 as specified by the “Entering Passive Mode” message, where 166 * 256 + 134 = 42630).

From the perspective of C&C activity, the attempt to store and retrieve the same file via the STOR and RETR commands suddenly opens a potentially reasonable explanation. Passive mode FTP transfers dynamically open data channels on separate network flows, where the server port is dynamically decided by the server. If a stateful firewall is present in the network, it will need to support this by reacting to the control channel interactions and open the associated ports accordingly to allow the transfer. A store and retrieve on the same passive channel can then become an attempt to fool a stateful firewall into allowing bidirectional communication on the port opened by the passive mode.

HTTP Traffic

The FTP traffic was not the only anomaly. Throughout the Breach Defender user interface in Figure 1 we could pivot to the web requests established during the same time-frame by the same host (see Figure 4, click to enlarge). We further correlated the extracted web requests with those available in our backend, giving us a total of 293 connection attempts (towards 293 distinct IPs), of which only 15 were successful.

Figure 4: Web requests were sent by the same host that executed the port scan and established the weird FTP control connections.

As shown in Figure 4, the requests were limited to three different hostnames: 8v9m[.]com, www.bing[.]com, and www.intercom[.]com. All web requests were POST, and besides those directed to 8v9m[.]com (which were using a constant and specific path and user agent), each connection was accessing a different resource, each time spoofing the user agent. Not a single DNS resolution was performed for the last two domains. Indeed, despite the HTTP headers indicating connections towards these hosts, the endpoints involved in the interaction were not associated in any way to the hosted infrastructure of these domains.

• 8v9m.com
  • Path: /ClientApi
  • User-Agent: Go-http-client/1.1
  • Response code: 200
• www.bing.com
  • Path: 6-char strings (e.g., /r7y9sp, /uhmq3a, /tm5qwn)
  • User-Agent:
    • Mozilla/5.0 (iPhone; CPU iPhone OS 10_2_1 like Mac OS X)
    • Mozilla/5.0 (Linux; Android 7.0; SM-G9550 Build/NRD90M)
  • Response code: mostly 4xx
• www.intercom.com
  • Path: 6-char strings (e.g., /ye4zkv, /8yakfu, /qzgp6c)
  • User-Agent:
    • Mozilla/5.0 (iPhone; CPU iPhone OS 10_2_1 like Mac OS X)
    • Mozilla/5.0 (Linux; Android 7.0; SM-G9550 Build/NRD90M)
    • Response code: mostly 4xx

Solving the Mystery

Summarizing the evidence collected so far, we seem to be dealing with something emulating FTP passive transfers and uploading and downloading data across the generated FTP channels, and generating very suspicious HTTP POST requests. This behavior seems clearly deceptive, and the use of these mechanisms for C&C data exfiltration seems a logical conclusion. But how to move the investigation forward?

We proceeded with the investigation by gauging the extent of this behavior and started searching for other endpoints connecting to the same hosts (see Figure 5, click to enlarge). It turned out that our original local host was not an isolated case: many other local hosts were exhibiting the very same traffic dynamics, collectively contacting several thousand external IPs, often belonging to the same CIDR blocks.

Figure 5: Graph of web requests towards intercom[.]com and 8v9m[.]com showing how many different hosts were generating the very same requests (and that sometimes more than one host was accessing the same randomly generated path).

This is when we started considering whether the actual culprit was instead a legitimate application; we searched for the domain names extracted when sifting through all the web requests and, as detailed in a public forum here, we were indeed on the right path. The network footprint matches the behavior of a known VPN client (X-VPN) famous for punching holes through corporate firewalls to evade restrictive local network policies.

The first thing such a client does is connect to a set of IPs on ports assigned to common protocols. This is done to find online and reachable servers (which eventually triggered our port scan alert). The reason why the client abuses the FTP protocol by establishing connections resembling C&C channels is twofold: first, even corporate firewalls often allow connections to the FTP control port 21 (most likely for legacy reasons); and second, unlike normal file transfers, the resulting data channels can be established in either direction, allowing bidirectional dialogue-like interactions.

If FTP connections are filtered or dropped, then the client tries several other protocols, including HTTP, fully explaining the web requests directed to the very same hosts. To further evade advanced policy filtering (for example denying specific operating systems and devices) the client goes even further and spoofs the “Host” and “User-Agent” header fields, a fact we saw in Figure 2.

Conclusions

We were definitely amazed by the rather creative way with which modern VPN clients attempt to punch holes through corporate firewalls and attempt to establish a connection regardless of corporate policy. The high volume of data points generated by these connection attempts clearly shows why tracing network events and producing insights from a corporate network can be quite a challenge for a trained network engineer even when the network is bereft of malicious activity.

On the other hand, with the right tools in hand, we have also demonstrated that it is indeed possible to easily pivot across multiple information domains, and use that information to differentiate security incidents from mere network anomalies. As we showed in this blog post, having an increased visibility over network events can often reveal organizational policy violations like the presence of unexpected or unwanted tools, a common effect of BYOD policies which are only partially enforced.

The post A Wild Port Scan Appears. What now? appeared first on Lastline.

Article Link: https://www.lastline.com/labsblog/a-wild-port-scan-appears-what-now/