How can I tune Scudo allocator flags and Primary/Secondary allocator parameters to minimize memory overhead?

To minimize memory overhead (Resident Set Size / RSS and virtual memory footprint) in the Scudo Hardened Allocator, you need to tune four key areas: quarantine sizing, page release frequency, thread cache (TSD) pooling, and secondary allocator caching.

When ARMv9 MTE is enabled, you can safely apply aggressive memory-saving configurations—such as disabling quarantine—because hardware tag re-tagging already handles Use-After-Free (UAF) protection.

1. Runtime Tuning via SCUDO_OPTIONS / __scudo_default_options()

Scudo can be configured at runtime via the SCUDO_OPTIONS environment variable or programmatically by exposing the __scudo_default_options() function in your C/C++ binary.

C++

extern "C" const char* __scudo_default_options() {
  // Aggressive memory-saving defaults
  return "quarantine_size_kb=0:"
         "thread_local_quarantine_size_kb=0:"
         "release_to_os_interval_ms=0:"
         "secondary_cache_max_entries=0";
}

High-Impact Runtime Options

Option
Default / Typical
Minimal Memory Setting
Impact on Memory Overhead

quarantine_size_kb
$256\text{--}1024\text{ KB}$
0
High. Completely disables the heap quarantine array. Releases freed chunks back to the allocator immediately without holding them in RSS.

thread_local_quarantine_size_kb
$64\text{ KB}$
0
High. Prevents per-thread quarantine accumulation.

release_to_os_interval_ms
$1000\text{--}5000\text{ ms}$
0 (or $\le 100$)
High. Controls how aggressively Scudo calls madvise(MADV_DONTNEED) to return unused dirty physical pages back to the kernel OS pool. 0 purges unused pages almost immediately.

secondary_cache_max_entries
$32\text{--}64$
0
Medium. Disables the Secondary Allocator's mmap cache. Freed large allocations are immediately unmapped (munmap) rather than held in cache.

secondary_cache_max_capacity_kb
$2048\text{ KB}$
0
Medium. Sets the maximum total bytes allowed in the secondary allocator cache to 0.

2. Primary Allocator Tuning

The Primary Allocator handles small-to-medium allocations using pre-reserved virtual memory regions partitioned into Size Classes.

A. Switch to a Shared TSD (Thread Specific Data) Model

Scudo supports two thread-cache architectures:

• ExclusiveTSD: Assigns a dedicated local cache to every thread. In high-thread-count applications (e.g., $50+$ threads), this results in severe RSS bloat as unused chunks sit idle in thread-local caches.

• SharedTSD (Recommended for low RAM): Uses a small, bounded pool of shared caches (e.g., $2\text{--}8$ total caches) shared across all threads via atomic locks/try-locks.

C++

// In custom Scudo C++ Config: Use Shared TSD to cap cache memory
using TSDConfig = scudo::SharedTSD<scudo::SharedTSDParameters<
    /*MaxTSDCount=*/4,             // Hard cap of 4 total caches across all threads
    /*ScanOnlyOnce=*/true>>;

B. Size Class Map Selection

Internal fragmentation occurs when a requested size (e.g., $33$ bytes) is rounded up to the nearest size class (e.g., $64$ bytes).

• Use DefaultSizeClassMap or a custom dense size-class map rather than coarse maps. A higher number of granular size classes decreases internal waste per allocation.

• Lower the TransferBatch size in the size class map so threads pull fewer free chunks at a time from the central page map into local caches.

3. Secondary Allocator Tuning

The Secondary Allocator handles large allocations exceeding the primary size class limit (typically $> 64\text{ KB}$ or $> 256\text{ KB}$) directly via mmap.

C++

// Scudo Custom Secondary Allocator Config for minimal footprint
struct MinimalSecondaryConfig {
  // Disable the secondary cache entirely
  static const scudo::u32 EntriesArraySize = 0;
  static const scudo::u32 DefaultMaxEntriesCount = 0;
  static const scudo::uptr DefaultMaxEntrySize = 0;
};

• Eliminate Cache Footprint: By setting EntriesArraySize = 0, large buffers (e.g., images, large arrays) are returned to the operating system immediately upon free(), preventing sudden RSS spikes.

• Guard Pages: Keep UseBufferBoundsChecking = true and guard pages enabled. Guard pages consume Virtual Address Space (VMA), not physical RAM (RSS), so they provide safety with zero physical memory cost.

4. Complete C++ Minimal Memory Config Example

If compiling a custom Scudo build or configuring compiler-rt for an embedded/constrained environment, define a custom scudo::Config:

C++

#include "scudo/standalone/allocator_config.h"

namespace scudo {

struct LowMemoryConfig {
  // 1. Primary Allocator Configuration
  using SizeClassMap = scudo::DefaultSizeClassMap; // Granular size buckets
  static const uptr PrimaryRegionSizeLog = 28;     // 256MB regions
  static const s32 PrimaryReleaseToOsIntervalMs = 0; // Immediate madvise

  typedef scudo::SizeClassAllocator64<LowMemoryConfig> Primary;

  // 2. Secondary Allocator Configuration (No Caching)
  struct Secondary {
    template <typename Config> using Cache = scudo::MapAllocatorNoCache<Config>;
  };

  // 3. Shared TSD (Thread Specific Data) Pool
  template <typename Allocator>
  using TSDRegistry = scudo::TSDRegistrySharedT<Allocator, 8, 1>; // Max 8 shared TSDs

  // 4. Disable Quarantine Completely
  // (Safe and recommended when ARMv9 MTE is active)
  struct Quarantine {
    static const uptr Bytes = 0;
    static const uptr ThreadLocalQuarantineBytes = 0;
  };
};

} // namespace scudo

Summary Checklist for Minimal Memory Footprint

1. Set quarantine_size_kb=0: Instant win; stops Scudo from hoarding freed blocks.

2. Set release_to_os_interval_ms=0: Forces immediate madvise(MADV_DONTNEED) to relinquish physical pages to the kernel.

3. Set secondary_cache_max_entries=0: Unmaps large allocations (>64KB) immediately upon free().

4. Use Shared TSDs: Cap the number of thread caches in multi-threaded applications.