How do I analyze and debug an Android debuggerd tombstone crash log caused by an MTE tag mismatch?

When an ARMv9 Memory Tagging Extension (MTE) tag mismatch occurs on Android, the bionic libc signal handler catches the SIGSEGV signal and requests a crash dump from debuggerd. The resulting tombstone file located in /data/tombstones/ contains diagnostic metadata generated by the hardware and the Scudo memory allocator.

1. Locating and Pulling the Tombstone

Retrieve the tombstone from the device using adb:

Bash

# Find the latest tombstone path from logcat
adb logcat | grep -i "tombstone"

# Pull the specific tombstone file (requires root or debuggable build)
adb pull /data/tombstones/tombstone_00 .

# Alternatively, capture via bugreport on non-rooted production devices
adb bugreport bugreport.zip

2. Key Sections of an MTE Tombstone

An MTE crash log is divided into five distinct sections:

A. Signal & Fault Address Header

Look at the signal code and the fault address near the top of the file:

Plaintext

Cmd line: com.example.myapp
pid: 12345, tid: 12345, name: myapp  >>> com.example.myapp <<<
signal 11 (SIGSEGV), code 9 (SEGV_MTESERR), fault addr 0x0b000073c323d595

• SEGV_MTESERR (Code 9): Synchronous MTE fault. The CPU aborted execution on the exact instruction that attempted the invalid memory access.

• SEGV_MTEAERR (Code 8): Asynchronous MTE fault. The hardware detected a mismatch asynchronously; the fault address and register state are not precise.

• Fault Address Pointer Tag (0x0b...): The top byte (0x0b) represents the Pointer Tag ($4$ bits) attached to the memory address being dereferenced.

B. Crash Cause Diagnostic

Android’s Scudo allocator automatically cross-references the fault address against its allocation history to state the cause:

Plaintext

Cause: [MTE]: Use After Free
# OR
Cause: [MTE]: Buffer Overflow (0x5 bytes right of 16-byte allocation)

C. Crash Backtrace

Shows the immediate call stack of the thread where the invalid memory access occurred:

Plaintext

backtrace:
      #00 pc 000000000001a4bc  /data/app/lib/arm64/libnative-lib.so (process_data+24)
      #01 pc 000000000001a510  /data/app/lib/arm64/libnative-lib.so (Java_com_example_MainActivity_runTest+32)

D. Memory Allocation & Deallocation Stack Traces

For Use After Free or Buffer Overflow errors, Scudo records where the affected block was allocated (and freed):

Plaintext

deallocated by thread 12345:
      #00 pc 000000000004bc20  /system/lib64/libc.so (free+28)
      #01 pc 000000000001a480  /data/app/lib/arm64/libnative-lib.so (cleanup_buffer+16)

allocated by thread 12345:
      #00 pc 000000000004b710  /system/lib64/libc.so (malloc+32)
      #01 pc 000000000001a440  /data/app/lib/arm64/libnative-lib.so (init_buffer+12)

E. Memory Tags Around Fault Address

Shows the actual $4$-bit Memory Tags assigned to the $16$-byte physical RAM granules surrounding the fault address:

Plaintext

Memory tags around fault address (0x0b000073c323d590):
    0x73c323d580: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   >0x73c323d590: a a 0 0 0 0 0 0 0 0 0 0 0 0 0 0

• The pointer tag was b (0x0b...).

• The memory granule tag at 0x73c323d590 is a.

• Because pointer tag b $\neq$ memory tag a, the hardware trapped the access.

3. Step-by-Step Debugging Workflow

+-------------------------------------------------------------------------+
| Step 1: Check Signal Code                                               |
| • SEGV_MTESERR (SYNC)  --> Proceed to Step 2                           |
| • SEGV_MTEAERR (ASYNC) --> Re-run in SYNC mode first                    |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| Step 2: Symbolicate Backtraces                                          |
| Run: ndk-stack -sym $UNSTRIPPED_SO_DIR -dump tombstone_00               |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| Step 3: Analyze MTE Tag Mismatch & Allocation Logs                      |
| • Compare Pointer Tag vs Memory Granule Tag                             |
| • Cross-reference Deallocated/Allocated stack traces with fault trace   |
+-------------------------------------------------------------------------+

Step 1: Ensure You Are in Synchronous (SYNC) Mode

If the signal is SEGV_MTEAERR (Asynchronous), switch the application to Synchronous mode to get an exact instruction pointer and stack trace:

Bash

# Force SYNC mode globally for the app via adb
adb shell setprop arm64.memtag.app.com.example.myapp sync
adb shell am force-stop com.example.myapp

Step 2: Symbolicate the Tombstone

Convert raw memory addresses (#00 pc 000000000001a4bc) into source code file names and line numbers using ndk-stack from the Android NDK:

Bash

$NDK_HOME/ndk-stack -sym /path/to/your/app/build/intermediates/merged_native_libs/debug/out/lib/arm64-v8a -dump tombstone_00

Step 3: Identify the Root Cause

1. For Use-After-Free (Cause: [MTE]: Use After Free):

   • Compare the backtrace frame against the deallocated by thread stack trace.

   • Look for a pointer that was saved locally or shared across threads, freed in cleanup_buffer(), and subsequently dereferenced in process_data().

2. For Buffer Overflow (Cause: [MTE]: Buffer Overflow):

   • Look at the offset specified in the cause line (e.g., 0x5 bytes right of 16-byte allocation).

   • Inspect loops or array indexing operations near the faulting line in backtrace #00. The pointer tag matched the target allocation, but arithmetic pushed the pointer into an adjacent granule with a different tag.

4. Helpful Utility Commands

Bash

# Dynamically change MTE mode without rebuilding the APK
adb shell setprop arm64.memtag.app.<package_name> [sync|async|off]

# Check MTE status of running processes on device
adb shell ps -EF | grep -i memtag

# Symbolicate inline with logcat live output
adb logcat | $NDK_HOME/ndk-stack -sym /path/to/unstripped/symbols