In March of 2022, Mandiant released new research detailing APT41’s persistent campaign leveraging novel exploits, malware, and techniques to compromise U.S. State Government networks. APT41 continued to demonstrate their tempo by exploiting a zero-day in an animal health management application before quickly shifting to operationalize the then fresh Log4j vulnerability.
At the time, APT41’s goals were unclear. The “Double Dragon’s” name is derived from APT41’s well documented dual espionage and cybercrime operation. Were they hitting U.S. State Governments to support greater intelligence collection initiatives, or for financial gain?
Mandiant researchers Van Ta and Rufus Brown take us on a journey of discovery into the mysteries of a long tail, persistent compromise of U.S. Government networks and offer a unique insight into one of the world’s most sophisticated threat actors.
Does This Look Infected 2: Audio automatically transcribed by Sonix
Does This Look Infected 2: this mp4 audio file was automatically transcribed by Sonix with the best speech-to-text algorithms. This transcript may contain errors.
Van Ta:
All right. Thank you, everyone. Thank you for attending. We also wanted to extend a thank you to the lab’s organizers for a great inaugural event so far. So let’s give them a round of applause before we get started. So my name is Van Ta. This is my colleague Rufus Brown, and we’re both part of Mandiant’s Advanced Practices Team. We’re really excited to be here today to expand on a story that we began telling in March of this year. And so, without further ado, this is Does This Look Infected? First. I must disclaim you.
Van Ta:
All right. So in March of this year, we published research on a persistent, months long APT41 campaign to gain access to state government networks. Between May 2021 to February of 2022, APT41 compromised at least six state government victims, primarily through exploitation of deserialization vulnerabilities in Internet facing web applications.
Van Ta:
Now, throughout the roughly ten month time frame, APT41 used two different zero days. The first was in an animal health management application known as USA Herds, which at the time of our analysis was used by 18 different states. Now the nature of this vulnerability was in a static machine key that was present in all default installations of the USA Herds application. And so APT41 in possession of this key could then compromise any server on the internet running this specific application. Now, in December of 2021, APT41 quickly shifted gears to operationalize the then fresh zero day in log4j. Now in the months prior APT41 and our research revealed a number of net new malware variants and that remained the same with log4j.
Van Ta:
What we were observing was apt 41 was exploiting victims with log4j to then deploy the Linux variant of a backdoor that we call KEYPLUG. Now this is notable for a number of reasons. Number one, this was the first time we had observed a Linux port of this backdoor for a piece of malware that’s been around since at least 2019. And number two, the Windows version of this backdoor was heavily used during the government intrusions in the months prior. So not only are they able to shift gears, switch up and operationalize a new zero day, but they’re able to deploy a new malware capability while still simultaneously operating at state government networks. So a lot of tenacity there.
Van Ta:
Now, throughout all this, it was pretty clear that APT41 put the P in APT. Right. They it was frequent that we would begin response at one state government agency only to find APT41 was active in a separate unrelated agency in the same state. And not only that but upon eradication APT41 would quickly recompromise their targets. And that’s something that we observed five different times.
Van Ta:
And so with this research, we were able to unveil quite a bit. But one burning question that we still had that we couldn’t really answer was “Why?”. And that will be the focus of our conversation today.
Van Ta:
So at the time, there were a couple of safe conclusions that we could make. These are state governments. There are treasures within these networks that would be valuable to any adversary. And the evidence of a deliberate, adamant campaign, based on the evidence that I talked about in the previous slide, supported some level of a targeted collection mission. But even then, although we had evidence to support these things, we still don’t really have an answer to why.
Van Ta:
Now, at the time we had a couple of hunches, but nothing really conclusive. But let’s take a look at what that really looked like. So at one state, victim, 41 had deployed the passive version of a backdoor that we call LOWKEY on a server responsible for the state’s financial benefits application. Now being a passive backdoor, it was configured to listen to traffic, to specific URLs, and in this case it was configured to listen or I’m sorry to listen for traffic to a URL in which one of the strings matched that specific benefits server application.
Van Ta:
Now APT41 matching their configurations to kind of blend in with the environment, blend in traffic with these different applications. That’s not something that’s net new. But it did show that APT41 wanted to maintain access to this server and this part of the network. Now, upon seeing something like this, one of the first questions that we would ask is, okay, how many states use this particular application? Do you guys like my breadsticks? Right there. Okay. And so to get a quick and dirty answer, we turn to scan data looking specifically for servers that would elicit a similar response to this particular benefits application. Now, while Rufus was poking around, one server stood out one because it was the only server not in the United States, and two, it was located in China. And so being nosy like we are, we wanted to inspect it a little bit further. So let’s see what we found.
Van Ta:
So. So we found a what appeared to be some sort of custom web app running on an ephemeral port that was leaking PII data for US citizens belonging to one particular state. And digging a little bit further, we found something else that was pretty interesting. We found what appeared to be a custom Baidu map with custom pins located somewhere in China. And so again, being very nosy, we zoomed in a little bit further and we could see that all of the pins are located in the Chengdu province of Chengdu and in particular were four kindergartens in that area. Do you all remember Chengdu 404? That was the front company that was detailed in the September 2020 indictments of APT41 members.
Van Ta:
Now, at this point, we have some loose ties to operations at state government victims. But because we did not directly observe this server in relation to that particular operation, we couldn’t attribute this to APT41 And so at that time, although we had some hunches, we were still back at square one, not really knowing the answer to why. It wasn’t until we completed investigations at two additional victims that we were able to collect the evidence to get us closer to that answer.
Rufus Brown:
All right. Thank you, Van. So for the rest of the presentation, I want to try and focus on these. Two new state government victims. So specifically, new data we haven’t talked about and specifically came from these two new state government victims. So starting out around last summer of June 2021, this is where we saw APT41 first gain initial access at State D, So this was through a proprietary Internet facing web application, which no other state had. Shortly after in August, this is where we saw APT41 gain initial access at the second state. Similar thing, proprietary web application, but this time it was a ASP.NET.
Rufus Brown:
Starting out around August. This is where we first saw the group conduct lateral movement and reconnaissance activities for around 4 to 5 months. So this is a really long time for a technically capable actor such as APT41 to remain active in environment and also really gain a better understanding of the network architecture as well as gain a stronger foothold on many systems across the network.
Rufus Brown:
At the beginning of the year. This is where we saw them first, laterally moved to the state benefits such as state benefit servers and also really conduct some hands on activity. So they started modifying with different software on the server. It really showed that they wanted to stay on these servers. So after an eradication event, about one month after we saw them re compromise via a similar technique, Internet facing web application exploitation, they quickly escalated privileges and got a foothold on over 50 systems in a very short amount of time. So really emphasizing that this group is very technically capable. They’re going to find web applications on your DMZ or Internet facing that are vulnerable.
Rufus Brown:
They have the capability to do that. So the last time we saw any sort of interaction or our last observance of the US state government campaign was around March and then one month after in April is when we saw them turn their focus to other geographic regions and organization verticals.
Rufus Brown:
So what helped us put the pieces of the puzzle together and really what were our big finds? So around out of three dozen systems in a 3 to 4 month time frame, 47% of those systems which were DEADEYE infected endpoints were associated to the state benefits architecture. Right. That’s a pretty large significant number for really showing what APT41 was interested in while in the environment.
Rufus Brown:
Secondly, while we started to investigate the state benefit system servers, we noticed that there was a peculiar malware that was running in memory on the server. This is what we track as FASTPACE, and one of the main capabilities of FASTPACE is to allow for unauthorized potential database modification.
Rufus Brown:
So if you’re not too familiar with fast pace, fast pace, which is aka Skip 2.0, was initially discovered and reported by ESET in late 2019. So pretty much this back door targets only MySQL servers for in-memory database manipulation. The particular backdoor that they discovered and reported on in the initial blog targeted SQL Server versions 11 and 12. While the backdoor malware we identified in the state government victim targeted version 13. This really indicates that APT41 is likely continuing to use FASTPACE in their toolkit and are continuing to update it for different iterations of SQL Server as they come out.
Speaker2:
So the way it works, pretty much this backdoor gets injected into SQL Server process and then it looks for specific byte pattern sequences. So these byte pattern sequences are associated with code functions in like native SQL modules such as SQL Lyngdal and SQL DQ. Basically, these targeted functions are related to credential validation, user authentication, event logging, SQL modification logs, things like that. So basically this pretty much covers up any sort of trace or track of what APT41 was doing on these database servers. So really, really difficult to keep track.
Rufus Brown:
I think it’s important to note, too, that out of all Maneant investigated EPP 41 intrusions. This was the first time we saw fast pace in use by APT41. So they had been active since. I think 2014 is like 78 and this is the first time we’ve seen this malware. And it was particularly at a state government victim, which is pretty interesting.
Rufus Brown:
So lastly, for State D after the eradication event, they went straight back to targeting state benefit servers, really just showing and indicating that they wanted to continue their mission, gather whatever data that they are apparently going after.
Rufus Brown:
Again, similar to state D, but for state E. They both targeted state benefit servers very heavily in both of these environments. So if we recall back to what Van mentioned in one of the beginning slides, so the log4j exploitation event, this is where we first saw the first iteration of the Linux backdoor for KEYPLUG.
Speaker2:
So about one month after we saw that backdoor dropped, we saw the passive version of this backdoor dropped at the state government victim. I think it’s important to note as well that this KEYPLUG passive version was only dropped on state benefit servers. Nowhere else in the environment.
Rufus Brown:
Lastly, so as we continue to investigate these servers in this environment, we saw them begin to tamper with the DNS configuration on the host. So this was a very pivotal point in our investigation and really helped us understand what types of data they were going after.
Rufus Brown:
So initially they began targeting these servers, laterally moved and gained access. Secondly, they deployed malware on these servers. It was just to KEYPLUG Linux passive version that I mentioned. The way it works is basically once it gets injected in the memory, it listens on an interface and looks for a packet that contains another magic byte sequence. This magic byte sequence is generated based on the infected host name of the server.
Rufus Brown:
Pretty similarly to how they target Windows operating systems during this campaign. They attempt to masquerade their files as legitimate binaries such as Microsoft, Fortinet and I believe, VMware. So as we can see here, one was deployed as a shared object file and the other one as a executable, particularly masquerading as VMware Tools.
Rufus Brown:
So after they did that, they immediately went to target the DNS configuration on the host. So specifically the host file. So we acquired this file, took a look at it, And of all the entries in this file, there was only one IP address that was a remote IP address.
Rufus Brown:
So we took a look at this and this remote IP address was mapped to a domain. Particularly this API domain was for a independent user verification service that was related to the state benefit system. So now potentially APT41 is allowing for this user verification traffic to get redirected to their C2.
Rufus Brown:
So potentially what could happen, let’s say, for this like a user logs into the state benefits application, right? They’re going to enter their username password, maybe MFA. Once they do that, likely this back end application is going to generate an API request to this remote domain likely containing a user verification info. So now all that data, all that user verification info is likely getting redirected to APT41 C2 server.
Rufus Brown:
So we took a look at the server, we started profiling it, taking a look at it, and we noticed that on one of the ports there was a Self-signed X509 certificate, particularly the Self-signed X59 certificate masqueraded as the Verification Services company’s country state locality, organization name, as well as the domain and common name. So really just showing that they wanted to blend in with this traffic and really try to masquerade in order to evade detection.
Rufus Brown:
So unfortunately, this is where our investigation ended. Just our scope didn’t include any more of investigating the database servers or web application logs. So this is where it stopped.
Van Ta:
And so we started our story today with a couple of hunches. And with that, we added evidence collected from victims that now in totality paint a convincing argument that what Apt41 was after was specifically our states’ financial benefits data.
Van Ta:
And although we’ve progressed significantly from where we were before, I think still, ultimately after all of this, we really still just want to know why. Now, although this although what we don’t know has been the focus of our presentation today, as we wrap up, I want to talk about the things that we do know.
Van Ta:
So, number one, based on apt41 operations on the state benefits server, based on our understanding of the data that would be exposed to them, it’s very possible that Apt41 has the ingredients to take this in a financial gain direction. And similarly, we know that historically Apt41 has the capability to run both financial gain and espionage operations concurrently.
Van Ta:
But even with that, the data exposed is highly sensitive and still could support some sort of collection mission.
Van Ta:
Now, number two, based on APT 41, just being everywhere as we’re responding to this over a ten month time frame, their willingness to exploit anything available to immediately get back in and retarget these servers, we are confident that the real answer to why does exist out there.
Van Ta:
And lastly, and arguably most importantly, the one thing that we know about this is that APT 41 continues to remain undeterred after their September 2020 indictments.
Van Ta:
And so with that, I hope you all enjoyed this story of essentially Rufus telling me I told you so and thank you all. I will now open it up for questions.
Van Ta:
Yes. Yes. Great question. So for a lot of the exploitation, before log4j, they were crafting a majority of so serial payloads to exploit deserialization vulnerabilities against a diverse set of applications at these different governments. I don’t know if you want to add anything else. Yeah.
Speaker2:
Why? Yet like the net. They’ve been using that for a while.
Van Ta:
Yes, sir. Gentleman in the back.
Speaker3:
He.
Van Ta:
That’s a great suggestion. Thank you for that. Like we we coordinated so closely with law enforcement during this, but didn’t specifically go down that direction. But this is kind of why we like this format as well of a talk with a lot of researchers in the crowd. So we can discuss this in almost in a way crowdsource potentially that answer to why. We’re able to get almost there, but not necessarily across the finish line. But I appreciate that. Yes.
Speaker3:
Are.
Van Ta:
We. We tried to. We tried to. I’ll say that. Yeah. Anything you want. Anything else you want to add?
Rufus Brown:
No, it was just that particular map was just on like another running port on that server. And we still have questions on like what exactly that server is. And it looked like almost maybe it’s something that’s compromised compromise infrastructure. But yeah, don’t know, 100%.
Van Ta:
And I still think that there is a potential that we did stumble upon some sort of operator box. And based on the information that we have here, we have tried to work with partners that would have a deeper, deeper level of visibility into that server itself. Because again, we’re we’re mainly dealing with scan data to identify and further investigate something like that. So yeah. Any other questions? All right. Thank you all so much.
Sonix is the world’s most advanced automated transcription, translation, and subtitling platform. Fast, accurate, and affordable.
Automatically convert your mp4 files to text (txt file), Microsoft Word (docx file), and SubRip Subtitle (srt file) in minutes.
Sonix has many features that you’d love including automatic transcription software, enterprise-grade admin tools, collaboration tools, upload many different filetypes, and easily transcribe your Zoom meetings. Try Sonix for free today.
About the Presenters
Van Ta is a Principal Threat Analyst on Mandiant’s Advanced Practices Team, where he leads historical research into the most impactful adversaries facing Mandiant’s customers. His research on various named threat actors FIN11, FIN12, FIN13, and APT41, has been referenced by both private and public organizations.
Rufus Brown is a Senior Threat Analyst on Mandiant’s Advanced Practices Team specializing in attribution and malware tradecraft. His joint research into APT41 was covered by national media outlets.
About LABScon
This presentation was featured live at LABScon 2022, an immersive 3-day conference bringing together the world’s top cybersecurity minds, hosted by SentinelOne’s research arm, SentinelLabs.
Want to join us for LABScon 2023? The Call for Papers is now open!
Article Link: LABScon Replay | Does This Look Infected 2 (APT41) - SentinelOne