Dart’s custom calling convention

Context

Dart is an object-oriented programming language with a C-style syntax, and a few features like sound null safety. The language is used Flutter, a cross-platform framework to build apps with the same codebase on various platforms such as Android, iOS, Windows, Linux… Dart can be compiled in various formats with more or less portability or performance. For example, Flutter uses Dart kernel snapshots for its debug builds and Dart Ahead of Time (AOT) snapshots for release builds.

“Dart Custom ABI prevents identification of function parameters”

In this article, Boris Batteux remarks that “Dart code uses a non-standard ARM64 ABI where most of the parameters of function calls are pushed to the Dart VM stack. Thus, IDA Pro isn’t able to identify function parameters and it considers that functions have no parameters.”

Normally, on ARM64, the first 8 parameters of functions are expected to be passed to a function in registers X0 to X7. Only subsequent parameters are pushed on the stack. With Dart, he notices that all parameters are pushed on the stack. Because this is unusual, IDA Pro fails to consider the first 8 parameters.

In addition, he remarks “because of the Dart custom stack, the real parameters are not pushed on the system stack, and thus they are not detected as function parameters by reverse engineering tools.”

In this blog post, we are going to explain this custom stack, and we will also discover that custom ABI is also used for other platforms like x86_64 and ARM32.

Custom calling convention for x86_64 too!

We write a small Dart program where 2 strings are concatenated:

void example() {
String header = 'ph0wn{';
print('stage 2: ' + header);
}

The x86_64 assembly of string concatenation is the following:

mov r11, qword [r15 + 0x1d3f]
push r11
mov r11, qword [r15 + 0x1d47]
push r11
call fcn.string_concat

To understand the first line, we must introduce the concept of Object Pool in Dart. In Dart, all frequently used objects, immediates and constants are not accessed directly but through a table-like structure which stores and references their use. So, actually, we do not ask direct access to string “stage2:” but we ask the object pool “please give me access to index XYZ”. Then, the object pool retrieves the corresponding address and returns the string.

The access to Dart’s Object Pool is done through a custom register: R15 in x86_64, R5 for ARM32 and X27 for ARM64.

As for the pool index, it is actually the address // 8. So, the first line requests pool index 935 (0x1d3f // 8), which actually turns out to be “stage2:”.

The string is written to R11, and notice in the second line R11 is pushed on the stack (push r11). This is unexpected: on x86_64, the first 6 arguments are usually provided through rdi, rsi etc, and only subsequent arguments are pushed on the stack. So, Dart’s x86_64 assembly uses the same custom calling convention as ARM64.

To summarize the 5 assembly lines:

  1. Get object pool index 935 “stage2:”
  2. Push it on the stack
  3. Get object pool index 936 “ph0wn{“
  4. Push it on the stack
  5. Call string concatenation

Custom calling convention for ARM32

We do the same exercise and compile for ARM32 platform.

Sidenote. For people who wonder how to compile Dart for ARM32, note there are currently no cross-compilers (except if you use Flutter, then your app is automatically compiled for the platforms you wish). So, to compile for ARM32, you need to be on a ARM32 platform. Personally, I used a Raspberry Pi 3 running a 32-bit OS, installed Dart on it, and compiled.

The produced assembly for string concatenation is similar:

ldr lr, [r5, 0xe9f] ; "stage2: "
ldr sb, [r5, 0xea3] ; "ph0wn{"
; push them on the stack
stm sp, {sb, lr}
; concatenate strings
bl fcn.concat

The register to access the Object Pool is R5 (note the pool indexes are platform-dependant). The mechanism is the same, using ARM32 instructions: get the strings, push them on the stack (this is done in one step on line 4) and finally call the concatenation function.

Same: arguments are pushed on the stack.

Stack register on ARM64

On ARM64, the assembly for string concatenation is:

LDR X16, [X27, #1C90h] ; "stage2: "
LDR X30, [X27, #1C98h] ; "ph0wn{"
STP X30, X16, [X15] ; push on the stack
BL _StringBase.+ ; concatenate

On ARM64, the register for Object Pool is X27. The assembly gets the 2 strings, and then pushes them “on the stack”. Note they are pushed on register X15, which is a custom, temporary register.

I wondered: “why isn’t it using a stack register like SP on other platforms?!”. The answer is simple: stack pointer (SP) and program counter (PC) aren’t indexed registers on AAarch64! So, Dart has no other way than using its own custom stack register for that.

— Cryptax

Article Link: Dart’s custom calling convention. Context | by @cryptax | Jun, 2023 | Medium