Users of Ivanti Connected Secure appliances are currently affected by several security flaws, several of which have been subject to in the wild exploitation in the past weeks.
Ivanti applied several security measures to harden their devices which affect the way forensic evidence can be collected. This article describes several methods discovered by security analysts that can be used to gather forensic evidence needed for a proper investigation.
Photo by George Prentzas on UnsplashBlog post authored by Dominic Helfer
There are two different types of Ivanti Connected Secure appliances. The hardware appliances operate on special hardware provided by Ivanti. The virtual appliance can be operated on various hypervisors. Both have in common that Ivanti utilizes full-disk encryption which makes it difficult to obtain an image for forensic analysis. Snapshot creation using the Integrity Checker Tool (short ICT) is supported by both Ivanti appliance types, but these snapshots are also encrypted.
In the following we list multiple methods on how to gather the forensic evidence from Ivanti appliances.
1. Appliances using LILO Bootloader
Northwave described in their blog post how to dump the encrypted filesystem of appliances using the LILO bootloader. The method described by them allows to either dump a live system or to decrypt an imaged disk by using qemu.
1.1 Preparation
When a live system is dumped, the network and the serial console need to be connected. The serial console can be accessed using a rollover RJ45 connector. If the internal hard drive was already imaged it can be virtualized using the following qemu command.
$ qemu-system-x86_64 -serial stdio \
-device ich9-ahci,id=ahci \
-drive file=ivanti-backup.img,id=disk,if=none \
-device ide-hd,drive=disk,bus=ahci.0
The network configuration needs to be added if necessary.
1.2 Dumping the live or emulated system
The following steps need to be performed according to Northwave to dump the live or emulated system:
- Input the following at the LILO prompt so that the system boots with a shell as init:
current init=//bin/sh debug verbose=5
Note: The double slash at the beginning is required to bypass a hard coded block list implemented in the bootloader. - To substitute the appropriate kernel version and NIC driver enter the following command:
/sbin/insmod/$kernel-version/modules/igb.ko - To substitute the appropriate NIC and IP range enter the following command:
/sbin/ifconfig eth1 192.168.0.2 netmask 255.255.255.0
Note: eth1 seems to correspond to interface labelled int. - On a separate host with the IP 192.168.0.1 a netcat listener needs to be started which pipes the output to a file:
nc -l 1337 > loop6.img - On the Ivanti appliance the /proc filesystem needs to be mounted:
mount -t proc none /proc - The output of the following command should output a single loop device (in this example it is loop6):
grep loop /proc/partitions - The determined loop device is dumped via the network:
dd if=/dev/loop6 bs=1M > /dev/tcp/192.168.0.1/1337
To dump other partitions of the Ivanti appliance losetup /dev/loopXX /dev/sdaXX can be used. Northwave describes that the partitions sda6, sda7and sda12 belong to the current kernel and sda8, sda9 and sda13 belong to the rollback kernel.
1.3 Decrypting the dumped filesystem
The obtained image of the partitions can be decrypted using the following algorithm which was described in the Northwave blog post:
def xor(s1, s2):
return bytes([c1 ^ c2 for c1, c2 in zip(s1, s2)])
def ivanti_cbc(key, sector, data):
cipher = AES.new(key=key, mode=AES.MODE_ECB)
def x(iv):
pre_iv = cipher.decrypt(iv)
for i in range(0, len(data), 16):
ciphertext = xor(data[i:i+16], pre_iv)
plaintext = cipher.decrypt(ciphertext)
yield xor(plaintext, iv)
iv = ciphertext
iv = sector.to_bytes(length=16, byteorder='little')
return b''.join(x(iv)
The following keys were identified by Northwave:
| Kernel | Key |
|----------------------------|-------------------------------------------------|
| 2.6.32-0008-g2f6ef39-dirty | D4 A5 7D 2A 87 D3 D2 CE A2 79 0E E3 CF 4E 7D B3 |
| 2.6.32-0032-g2005e8d-dirty | 29 09 40 A1 74 F1 8E 2B E9 80 F9 FA 51 72 A7 4B |
Northwave also provides a decryption tool called lilo-pulse-secure-decrypt on GitHub for that purpose.
2. Appliances using GRUB bootloader
WatchTower described a method to get the LUKS encryption keys of Ivanti appliances which use the GRUB bootloader.
In GRUB, a recovery shell can be started by entering init=//bin/sh. After that, the full-disk encryption keys can be extracted from the file /etc/lvmkey. Using the same steps as already described in 1.2, the filesystem can be dumped over the network. Alternatively, the disk can be removed and imaged using specialized tools. Native Linux tooling can be used to decrypt the LUKS encrypted disk with the extracted keys.
3. Ivanti Integrity Checker Tool (ICT) Snapshots
The external Integrity Checker Tool (ICT) provided by Ivanti performs an integrity check on the file system of the appliances. If a change was detected, a snapshot is created which can be used for further analysis. The snapshot itself is encrypted and only contains the files where a change was detected by ICT. To analyze this snapshot, it needs to be downloaded from the Ivanti appliance and be decrypted either by the Ivanti support or by using the decryption tool Ivanti-ICT-Snapshot-decryption.
4. Memory Acquisition
The security analysts of Volexity demonstrated in their blog post how memory analysis helped to reconstruct attacks on Ivanti appliances. For memory acquisition they used their own tool called Volexity Surge Collect Pro. The acquired memory image was analyzed using another Volexity tool called Volcano. Both tools are not publicly available.
The following describes a method that could help collect the memory. It is important to note that the method has not yet been tested. The goal is to secure the memory of the Ivanti appliance over the network using dd and netcat as it has been done for the partition in 1.2. Therefore, the same steps can be used to prepare the system and only steps 4 and 7 are modified:
- Modified step 4: On a separate host with the IP 192.168.0.1, a netcat listener needed to be started which pipes the output to a file:
nc -l 1337 > memory.img - Modified step 7: The memory device is dumped via the network:
dd if=/dev/mem bs=1M > /dev/tcp/192.168.0.1/1337
In recent versions of Linux, the device /dev/mem doesn’t provide access to the complete physical memory but only to a restricted range of addresses. Therefore, the possibility exists that the memory acquisition using the described method fails. The use of the kernel module fmem can be considered because this creates a device called /dev/fmem providing access to the physical memory.
5. Enable the Logging of Unauthenticated Requests
The current attacks observed towards Ivanti appliances leverage multiple vulnerabilities which allow to use unauthenticated requests to perform remote code execution. In the default configuration the unauthenticated requests are not logged by the appliances. To detect and investigate attacks logging of unauthenticated requests should be activated.
6. Conclusion
There are multiple ways to collect forensic evidence from hardware and virtual Ivanti appliances, allowing a forensic analysis of these systems. The method using the snapshots produced by the Ivanti Integrity Checker Tool (ICT) offers a fast way to detect and collect modified files. The disadvantage of this method is that it does not produce a full disk image which is needed for an in-depth forensic analysis. The two other methods discovered by security researchers of Northwave and WatchTower allow the creation and decryption of a full disk image, which enables incident responders to investigate attacks on Ivanti appliances.
Article Link: Overview: Evidence Collection of Ivanti Connected Secure Appliances | by DCSO CyTec Blog | Feb, 2024 | Medium