FerrousPath

Level 4 · Expert· Lesson 5 of 9 · 40 min

Atomics and memory ordering

The memory model behind Relaxed, Acquire, Release, AcqRel and SeqCst — happens-before, compare-and-swap, and how to choose an ordering.

Atomics are the foundation under every mutex, channel and Arc. Used directly they give you lock-free code — and the most subtle bugs in systems programming. This lesson gives you the model needed to reason about them rather than guess.

Atomic types

std::sync::atomic provides AtomicBool, AtomicUsize, AtomicU64, AtomicPtr<T> and friends. They are interior-mutable (&self methods) and Sync. Each operation is indivisible:

Operation Meaning
load, store read, write
swap write and return the old value
fetch_add, fetch_sub, fetch_or, fetch_and, fetch_max… read-modify-write, returns the old value
compare_exchange(current, new, success, failure) if value == current, set to new; returns Ok(old) or Err(actual)
compare_exchange_weak same, but may fail spuriously; use in loops
fetch_update(set, fetch, f) CAS loop applying a closure
use std::sync::atomic::{AtomicU64, Ordering};
use std::thread;

static HITS: AtomicU64 = AtomicU64::new(0);

fn main() {
    thread::scope(|s| {
        for _ in 0..8 {
            s.spawn(|| {
                for _ in 0..100_000 {
                    HITS.fetch_add(1, Ordering::Relaxed);
                }
            });
        }
    });
    println!("{}", HITS.load(Ordering::Relaxed)); // always 800000
}

Why orderings exist

Two layers of the machine reorder your program:

  • The compiler assumes single-threaded semantics for ordinary memory and will reorder, merge, and hoist reads and writes.
  • The CPU has store buffers, out-of-order execution and caches. On ARM, POWER and RISC-V, two threads can genuinely observe each other’s writes in different orders. x86 is stricter (stores are not reordered with other stores), which is why bugs hide there and appear on a phone.

Rust adopts the C++20 memory model. An ordering is a statement to both compiler and CPU about what guarantees you need. Weaker orderings permit more optimisation; stronger ones cost more, mainly on weakly ordered hardware.

Happens-before

The model is defined in terms of a partial order, happens-before:

  • Within one thread, operations are ordered by program order.
  • Spawning a thread happens-before everything in it; everything in a thread happens-before a successful join.
  • Unlocking a mutex happens-before the next lock.
  • A Release (or stronger) store happens-before an Acquire (or stronger) load that reads the value written by that store.

A data race is two accesses to the same non-atomic location, at least one a write, with no happens-before relation between them. Data races are undefined behaviour. Atomic accesses never constitute a data race, whatever ordering you use — the ordering only determines what else is synchronised.

The five orderings

Relaxed — atomicity only. Each atomic variable has a single total modification order that all threads agree on, but nothing is implied about any other memory. Use for counters, statistics, ID generation, and flags where staleness is acceptable.

Release (stores) and Acquire (loads) — publication. Everything the writer did before the Release store is visible to a reader after the Acquire load that sees it. This is what locks, channels, Arc and OnceLock are built from.

AcqRel — for read-modify-write operations that both observe and publish, such as a CAS that takes a lock.

SeqCst — everything AcqRel gives, plus a single total order over all SeqCst operations in the program, consistent with happens-before. Needed only for patterns in which two threads each write one variable and then read the other (store-buffering); see below.

use std::cell::UnsafeCell;
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread;

struct Slot {
    ready: AtomicBool,
    data: UnsafeCell<u64>,
}
// SAFETY: `data` is written once before `ready` is set (Release) and read only
// after `ready` is observed (Acquire), so accesses never race.
unsafe impl Sync for Slot {}

static SLOT: Slot = Slot { ready: AtomicBool::new(false), data: UnsafeCell::new(0) };

fn main() {
    thread::scope(|s| {
        s.spawn(|| {
            // SAFETY: no reader touches `data` before `ready` is true.
            unsafe { *SLOT.data.get() = 42 };
            SLOT.ready.store(true, Ordering::Release); // publish
        });
        s.spawn(|| {
            while !SLOT.ready.load(Ordering::Acquire) { // synchronise with the publisher
                std::hint::spin_loop();
            }
            // SAFETY: the Acquire load saw the Release store, so the write to `data`
            // happens-before this read.
            println!("{}", unsafe { *SLOT.data.get() });
        });
    });
}

Change both orderings to Relaxed and the program has a data race on data: the reader could see ready == true without the write to data being ordered before its read. It would probably still print 42 on x86. That is precisely the danger.

The store-buffering litmus test

use std::sync::atomic::{AtomicBool, Ordering::SeqCst};
use std::thread;

static X: AtomicBool = AtomicBool::new(false);
static Y: AtomicBool = AtomicBool::new(false);

fn main() {
    let (a, b) = thread::scope(|s| {
        let t1 = s.spawn(|| { X.store(true, SeqCst); Y.load(SeqCst) });
        let t2 = s.spawn(|| { Y.store(true, SeqCst); X.load(SeqCst) });
        (t1.join().unwrap(), t2.join().unwrap())
    });
    // With SeqCst, at least one thread must see the other's store.
    assert!(a || b);
    println!("t1 saw Y={a}, t2 saw X={b}");
}

With SeqCst, all four operations fall into one total order, so whichever store is second is preceded by the other store — at least one load returns true. With Release/Acquire both loads may return false: each store can sit in its CPU’s store buffer while the other thread reads the old value. This is the one common shape that truly needs SeqCst (it appears in Dekker-style mutual exclusion and some wake-up protocols).

Compare-and-swap loops

CAS is the universal building block: read, compute, and commit only if nobody else changed the value in the meantime.

use std::sync::atomic::{AtomicU32, Ordering};
use std::thread;

/// Hands out IDs, refusing to overflow.
fn next_id(counter: &AtomicU32) -> Option<u32> {
    let mut current = counter.load(Ordering::Relaxed);
    loop {
        let next = current.checked_add(1)?;
        match counter.compare_exchange_weak(current, next, Ordering::Relaxed, Ordering::Relaxed) {
            Ok(_) => return Some(current),
            Err(actual) => current = actual, // someone else won; retry with the fresh value
        }
    }
}

fn main() {
    let counter = AtomicU32::new(u32::MAX - 5);
    let got: usize = thread::scope(|s| {
        let hs: Vec<_> = (0..4)
            .map(|_| s.spawn(|| (0..10).filter(|_| next_id(&counter).is_some()).count()))
            .collect();
        hs.into_iter().map(|h| h.join().unwrap()).sum()
    });
    println!("{got} IDs issued before exhaustion"); // exactly 5
}

The two ordering arguments are for success and failure. The failure ordering applies to the load performed when the comparison fails; it may not be stronger than the success ordering.

Case study: the orderings in Arc

A reference count shows how to reason from first principles:

// The essence of Arc<T>'s clone and drop.
fn clone(&self) -> Self {
    // Relaxed: we already hold a reference, so the count cannot reach zero underneath us,
    // and no other memory access depends on observing this increment.
    self.inner().strong.fetch_add(1, Ordering::Relaxed);
    /* … */
}

fn drop(&mut self) {
    // Release: publish every use this thread made of the data…
    if self.inner().strong.fetch_sub(1, Ordering::Release) != 1 {
        return;
    }
    // …Acquire: so that the thread performing the final decrement observes all of them
    // before running the destructor and freeing the memory.
    std::sync::atomic::fence(Ordering::Acquire);
    unsafe { self.drop_slow() };
}

The increment needs nothing. The decrement must ensure that the thread which frees the object has a happens-before edge from every other thread’s last use of it.

Practical guidance

  • Default to a Mutex. Uncontended locking costs a couple of atomic operations. Go lock-free when a profile says so.
  • If you must choose quickly: counters → Relaxed; publishing data → Release store, Acquire load; a CAS that acquires something → Acquire on success, Relaxed on failure; unsure → SeqCst, and then work out why.
  • SeqCst is not “the safe choice”: it will not repair a wrong algorithm, and it hides the reasoning a reader needs. But it is never less correct than a weaker ordering.
  • Test with tools, not hope. loom exhaustively explores thread interleavings and weak-memory behaviours for small tests. Miri simulates weak memory effects and detects data races. ThreadSanitizer (-Zsanitizer=thread) finds races in real executions. Run CI on an ARM machine.
  • False sharing: two hot atomics on one cache line make cores fight over it. Separate them with #[repr(align(64))] or crossbeam_utils::CachePadded.
  • AtomicPtr and atomics generally are subject to the ABA problem, the subject of the next lesson.

The definitive book-length treatment is Mara Bos, Rust Atomics and Locks (free online). Read it.

Exercises

  1. Implement a spin lock with AtomicBool: which orderings do lock (a CAS) and unlock (a store) need, and why? Add a guard type.
  2. Implement Once-style lazy initialisation with a three-state AtomicU8 (uninitialised / running / done).
  3. For the publication example, write down the happens-before chain from the write of data to its read. Then state exactly which edge disappears when the store becomes Relaxed.
  4. Model the store-buffering test under loom with Release/Acquire and observe the (false, false) outcome being found.
  5. Write a Relaxed statistics counter struct with several fields and explain why reading a consistent snapshot of all fields is impossible without further synchronisation.

Check your understanding

Pick an answer to see the explanation.

  1. 1.What does a Release store paired with an Acquire load of the same atomic establish?

  2. 2.When is Ordering::Relaxed sufficient?

  3. 3.Why should compare_exchange_weak be used in a retry loop rather than compare_exchange?

Key terms in this lesson

Finished the lesson and the exercises?