As the pressure on referees keeps growing, technology has been introduced into more or less every sport where millimetres matter. Unfortunately, that technology and its algorithms can completely overshadow what we all actually see. Yes - if the technology says the ball was touched, there is nothing more to discuss about that particular case. What I do want to discuss is the impact of technology on people.

The officiating we have now is hybrid. Technology is there primarily to help the head referee with decisions he couldn’t see. But if a referee knows that VAR can step in every time, we run into automation complacency - the tendency to lower your own attention and double-checking because you trust the system to do the job for you. It is closely related to automation bias, where a person places excessive trust in the technology’s suggestions and stops critically questioning the outcome or looking for additional information.

The second way of officiating is to remove technology entirely. The referee would then have to be maximally focused, knowing his career depends on his calls. But here’s what comes naturally to us: if a referee makes a mistake in that kind of system, after a while we, as people, will say, “Yes, he got it wrong, that’s human” - and everything will be fine.

The third way is to remove the human factor completely and let the technology itself judge every illegal touch, foul, and offside. Then we can say: fine, the algorithm has done its part, it is the referee now, and we simply have to accept it.

But when we have the hybrid model, people will never accept the technology as it is, because they still expect the referee to take responsibility, to see the bigger picture, rather than blindly trust the technology he’s using. And that is where the confusion, anger, and frustration come from - all the feelings yesterday’s match left us with. The referee becomes a lightning rod for a decision he didn’t actually make - he absorbs the anger for something the system determined.

Sadly, we will never go back to purely human officiating. Cameras now show every mistake to millions of viewers in a single second, and the commercial pressure is too great for anyone to give up the technology that “corrects” the referee. The only solution I see is to commit to technology fully and accept it as the final judge.

And here I have to be honest: I’m not advocating full automation because it’s infallible. The technology has its own margins of error and false positives. I advocate for it because it is consistent and because it clearly assigns responsibility. The problem with the hybrid system isn’t only about who makes the mistake, but about the mismatch between our expectations and reality: we expect human judgement, and we get an algorithmic one. When the algorithm openly becomes the referee, at least we know where we stand.

In tennis, almost every tournament is automated (Roland Garros is not), and in my view it works. That said, I have to be fair: a tennis line is pure geometry - the ball is either in or out. Football has far more decisions that are a matter of interpretation rather than measurement (intent on handball, interference from an offside position), so the tennis solution can’t be mapped over one-to-one. But the direction is the same: fewer grey zones, and less room for the feeling of being cheated.

What I feel most sorry about is the team and the people, because I know how much time and effort it takes to become even an average athlete, let alone a top one like our national team players. At this level, these things aren’t fair to everyone who lives this sport.

Why This Problem Exists At All

Modern Apple Silicon chips have multiple cores - an M3 Pro has 12, an M2 Ultra has 24. Each core has its own L1 and L2 cache. When two cores want to read the same data, that’s fine - they can both have a copy in their local cache simultaneously. When one core wants to write, it has to invalidate every other core’s cached copy, wait for acknowledgements, then do the write. This cross-core communication is expensive and doesn’t scale - the more cores you have, the more acknowledgements you need.

This coordination is managed by the MESI protocol - the cache coherence protocol used by Apple Silicon (or some MESI flavour). Every cache line is in one of four states:

• Modified - I have the only copy, and it’s dirty (different from RAM)
• Exclusive - I have the only copy, and it matches RAM
• Shared - multiple cores have this line, all matching RAM
• Invalid - my copy is stale, someone else modified it

When Core 0 wants to write to a line in Shared state, it broadcasts an invalidate message to all other cores, waits for acknowledgements, transitions to Modified, then writes. The more cores you have, the more acknowledgements you need to collect before you can write. This is the fundamental scalability problem that every concurrent data structure is working around.

This is why concurrent access to shared mutable state is hard. It’s not just a software problem. It’s a hardware problem. Every lock, every atomic operation, every memory barrier exists because of this physical reality.

The cache line is the unit of transfer. On Apple Silicon it’s 128 bytes. The CPU never moves less than 128 bytes between caches. This has a critical implication: if two completely unrelated variables happen to sit within the same 128-byte region of memory, and two different cores write to them, those cores will thrash each other’s caches even though they’re touching different data. This is false sharing - and it’s one of the most common reasons parallel code doesn’t scale.

I knew what a lock was. I knew what a data race was. But the idea that two variables could fight over cache just by sitting next to each other in memory - that was new to me, and honestly a bit mind-blowing.

This diagram is what everything else in this article is about. Each core has its own private caches - L1, L2 - and they all share L3 and RAM. When two cores read the same data, no problem. When one wants to write, it has to tell every other core to throw away their cached copy. That round-trip is what makes concurrent writes expensive, and it’s the reason locks, atomics, and memory barriers exist in the first place.

                    ┌─────────────────────────────────────────┐
                    │              Main RAM                   │
                    │            ~100+ cycles                 │
                    └─────────────────┬───────────────────────┘
                                      │
                    ┌─────────────────┴───────────────────────┐
                    │             L3 / SLC Cache              │
                    │              ~40 cycles                 │
                    └──────────────┬──────────────────────────┘
                                   │
              ┌────────────────────┴─────────────────────┐
              │                                          │
    ┌─────────┴────────┐                       ┌─────────┴────────┐
    │      Core 0      │                       │      Core 1      │
    │                  │                       │                  │
    │  ┌────────────┐  │                       │  ┌────────────┐  │
    │  │  L2 Cache  │  │                       │  │  L2 Cache  │  │
    │  │ ~12 cycles │  │                       │  │ ~12 cycles │  │
    │  └─────┬──────┘  │                       │  └─────┬──────┘  │
    │        │         │                       │        │         │
    │  ┌─────┴──────┐  │                       │  ┌─────┴──────┐  │
    │  │  L1 Cache  │  │                       │  │  L1 Cache  │  │
    │  │  ~4 cycles │  │                       │  │  ~4 cycles │  │
    │  └─────┬──────┘  │                       │  └─────┬──────┘  │
    │        │         │                       │        │         │
    │  ┌─────┴──────┐  │                       │  ┌─────┴──────┐  │
    │  │  Registers │  │                       │  │  Registers │  │
    │  │  ~0 cycles │  │                       │  │  ~0 cycles │  │
    │  └────────────┘  │                       │  └────────────┘  │
    └──────────────────┘                       └──────────────────┘

What Locks Actually Are

Every lock in existence is built on one hardware primitive: Compare-And-Swap (CAS). On ARM it’s implemented as a pair of instructions - ldaxr (load-acquire exclusive) and stlxr (store-release exclusive). The hardware guarantees these are atomic - indivisible, no intermediate state visible to other cores. CAS does this atomically:

if *address == expected {
    *address = new
    return success
} else {
    return failure  
}

If two cores race to CAS the same address, exactly one wins. The loser retries. Everything else - every lock, every mutex, every semaphore you’ve ever used - is built on top of this one instruction.

os_unfair_lock (what OSAllocatedUnfairLock wraps) uses CAS to try to acquire. If it succeeds - nobody else held it - the whole thing takes ~5ns and never touches the kernel. If it fails, it spins briefly then calls into the kernel to sleep the thread. “Unfair” means no FIFO queue - when released, any waiter can grab it. This is the lock you want in Swift when you need raw performance.

NSLock wraps pthread_mutex_t. Heavier - ~25ns uncontended, fair (FIFO ordering), more features. For the write path of left-right it doesn’t matter, because writes are rare. I used it because it’s simple and the overhead is irrelevant when you’re only writing once every few seconds.

Here’s the fundamental problem with locks for reads:

acquire lock      ← all readers queue here, one at a time
read data
release lock

Even if nobody is writing, readers block each other acquiring and releasing the lock. On 8 cores all trying to read the same data, 7 are waiting at any moment. Read throughput doesn’t scale with core count - it’s essentially serial. You bought an 8-core machine and you’re using one.

pthread_rwlock improves this by allowing concurrent readers, but readers still have to atomically increment a shared reader count on every read. That shared counter is a single cache line being hammered by every reader on every core - bouncing between L1 caches constantly. Under high concurrency, that counter becomes your bottleneck. It’s why pthread_rwlock can actually be slower than a plain mutex under high read contention. You solved one problem and created another.

The ARM Memory Model

Here’s something that surprises most people: the CPU does not execute your instructions in the order you wrote them. Both the compiler and the CPU reorder operations to keep execution units busy. In single-threaded code this is invisible because the CPU tracks dependencies. In multi-threaded code it’s a real problem.

Consider Thread A:

data = 42
ready = true

And Thread B:

while !ready { }
print(data)  // Is this always 42?

On x86 - yes. x86’s strong memory model (TSO - Total Store Order) prevents store-store reordering. On ARM - no. The CPU can reorder those two stores. Thread B might see ready = true but still read data = 0. This is not a bug. It’s a documented feature of the ARM memory model, designed to allow aggressive out-of-order execution for performance. Apple Silicon is ARM. Your M-series Mac can do this to you.

A memory barrier tells the CPU: do not reorder loads/stores across this point. On ARM the full barrier is dmb ish - expensive because it drains the store buffer and waits for acknowledgements from all other cores. You don’t want this in a hot path. The ARM architecture gives you lighter-weight alternatives:

• ldar - load-acquire: this load cannot be reordered with any memory operation after it
• stlr - store-release: this store cannot be reordered with any memory operation before it

These are cheaper than a full barrier but sufficient for most synchronization patterns - specifically the acquire-release pairing.

The Left-Right Algorithm

The insight behind left-right is simple: what if readers never had to touch any shared mutable state at all? No lock to acquire, no counter to increment, no cache line to contend on. Just data, sitting there, waiting to be read.

The way you achieve this is by keeping two complete copies of your data structure. At any moment, one copy is active - readers use it. The other is inactive - the writer owns it. Readers never touch the inactive copy. The writer never touches the active copy while readers are on it.

         ┌──────────────┐         ┌──────────────┐
Readers  │    ACTIVE    │         │   INACTIVE   │  Writer
  ──────▶│    Copy A    │         │    Copy B    │◀──────
  ──────▶│              │         │              │
  ──────▶└──────────────┘         └──────────────┘
                    ▲
                    │
               readIndex
          (atomic bool - tells
          readers which copy
              to use)

The Write Sequence

Step 1. The writer applies the operation to the inactive copy (B). Readers are all on A - this is completely safe, no coordination needed.

         ┌──────────────┐         ┌──────────────┐
Readers  │    ACTIVE    │         │   INACTIVE   │  Writer
  ──────▶│    Copy A    │         │    Copy B'   │◀── writes here
  ──────▶│  (old value) │         │  (new value) │
         └──────────────┘         └──────────────┘

Step 2. The writer atomically flips readIndex. New readers now go to B’ - the updated copy. Old readers that were already reading A continue until they finish. You can’t stop them mid-read.

         ┌──────────────┐         ┌──────────────┐
Old      │  now         │         │  now ACTIVE  │  New readers
readers  │  INACTIVE    │         │    Copy B'   │◀──────
finishing│    Copy A    │         │  (new value) │◀──────
  ──────▶│  (old value) │         │              │
         └──────────────┘         └──────────────┘

Step 3. The writer waits for all readers that were on A to finish. This is called the drain. Once drained, A is truly inactive - no thread is touching it.

Step 4. The writer applies the same operation to A. Both copies are now identical and up to date.

         ┌──────────────┐         ┌──────────────┐
         │   INACTIVE   │         │    ACTIVE    │  Readers
Writer ──▶│   Copy A'   │         │    Copy B'   │◀──────
         │  (new value) │         │  (new value) │◀──────
         └──────────────┘         └──────────────┘

The next write starts from this state, with A’ as the inactive copy. The two sides alternate on every write.

The Drain Problem

After the flip, the writer needs to wait for readers that were mid-read on the old copy to finish. But how does it know when they’re done - without shared mutable state?

This is the clever part. Each reader gets its own epoch counter - a single integer that lives on its own private cache line. The protocol is simple:

Even value → not currently reading
Odd value  → currently reading

The read protocol:

increment my counter  (even → odd)   "I am starting a read"
load readIndex                        "which copy do I use?"
read the data
increment my counter  (odd → even)   "I am done"

After the flip, the writer snapshots all epoch counters. Any counter that was odd at the moment of the flip belongs to a reader that was mid-read on the old copy. The writer spins on those specific counters until they go even. Readers that start after the flip go to the new copy - the writer doesn’t care about them.

Writer after flip:

epoch[0] = 4  (even) → not reading, skip
epoch[1] = 7  (odd)  → was reading on old copy, wait...
epoch[2] = 2  (even) → not reading, skip
epoch[3] = 11 (odd)  → was reading on old copy, wait...

... epoch[1] becomes 8, epoch[3] becomes 12 ...

All clear. Apply operation to now-inactive copy.

Each reader writes only to its own epoch counter - its own private cache line. No other thread writes to it. The writer only reads the counters during the drain, it never writes them. So there is no cache line bouncing between cores during the read path.

This is the whole algorithm. Two copies, an atomic flip, epoch counters, a drain. Everything else in the implementation is making this correct and efficient in Swift.

Swift Atomics

Swift has no built-in atomic operations. For a long time, if you needed atomics in Swift you had to drop into C or use deprecated OSAtomic APIs. swift-atomics is Apple’s official answer to that - a package that exposes atomic operations on primitive types with the full C++ memory model, available from pure Swift.

ManagedAtomic vs UnsafeAtomic

There are two atomic types you’ll use. The difference is ownership of storage. ManagedAtomic<T> is a class. Heap allocated, ARC managed. When you write:

let counter = ManagedAtomic<Int>(0)

What you get in memory:

Stack:
┌──────────────────┐
│  counter (ptr)   │──────┐  8 bytes, a pointer
└──────────────────┘      │
                          ▼
Heap:
┌──────────────────────────────────┐
│  ARC refcount    │  8 bytes      │
├──────────────────────────────────┤
│  type metadata   │  8 bytes      │
├──────────────────────────────────┤
│  atomic value    │  8 bytes      │  ← the actual Int
└──────────────────────────────────┘

Every operation requires following that pointer. One extra memory access on every read or write. For a single atomic value this is fine - the overhead is negligible. For an array of atomics where layout matters, it’s a problem.

UnsafeAtomic<T> does not own its storage. You provide the memory, you manage the lifetime. The key type is UnsafeAtomic<T>.Storage - a fixed-size chunk of properly aligned bytes that you embed directly in your own struct:

struct EpochCounter {
    var storage: UnsafeAtomic<UInt>.Storage = .init(0)
}

When you put these in an array, the atomic values are contiguous in memory exactly where you put them. This is what lets you control layout - essential for cache line padding.

To operate on the storage you create a temporary handle:

withUnsafeMutablePointer(to: &storage) { ptr in
    UnsafeAtomic<UInt>(at: ptr).wrappingIncrement(ordering: .releasing)
}

The handle is just a pointer wrapper. It doesn’t own anything, doesn’t allocate anything. It exists for the duration of the closure and disappears. The value lives in storage.

Array of EpochCounters with UnsafeAtomic:

┌──────────────────────────────────┐
│  UInt value  │  120 bytes pad   │  ← EpochCounter[0], 128 bytes, own cache line
├──────────────────────────────────┤
│  UInt value  │  120 bytes pad   │  ← EpochCounter[1], 128 bytes, own cache line
└──────────────────────────────────┘
No pointers. No heap. Values exactly where you need them.

Array of EpochCounters with ManagedAtomic:

┌─────┬─────┬─────┬─────┐
│ ptr │ ptr │ ptr │ ptr │  ← array of pointers, nicely laid out
└──┬──┴──┬──┴──┬──┴──┬──┘
   │     │     │     │
   ▼     ▼     ▼     ▼
  heap  heap  heap  heap   ← actual values scattered, same cache line, false sharing

For left-right, UnsafeAtomic is the only option that makes the cache line padding meaningful.

Memory Orderings

This is the part that confused me the most at first. The ordering parameter on an atomic operation doesn’t affect whether the operation is atomic - it’s always atomic. It controls how the operation interacts with surrounding memory operations - what the CPU and compiler are allowed to reorder around it.

There are four you need to know:

.relaxed

No ordering constraints. The operation is atomic but the CPU can reorder it freely with respect to everything around it. Use when you only care about atomicity, not about what other memory is visible.

nextSlot.loadThenWrappingIncrement(ordering: .relaxed)

On ARM this compiles to a plain atomic instruction with no barrier - fastest possible.

.acquiring (loads only)

This load cannot be reordered with any memory operation after it. Think of it as a one-way barrier - nothing below this line can float above it.

// ARM: ldar
epochs[i].load(ordering: .acquiring)

.releasing (stores only)

This store cannot be reordered with any memory operation before it. Nothing above this line can sink below it.

// ARM: stlr
epoch.wrappingIncrement(ordering: .releasing)

The acquire-release pairing is the fundamental synchronization mechanism in left-right. A .releasing store on Thread A synchronizes with an .acquiring load of the same variable on Thread B. Once that synchronization is established, Thread B is guaranteed to see all memory writes Thread A did before the release.

Without the pairing:

Thread A:
data = 42
ready.store(true, .relaxed)   ← no ordering guarantee

Thread B:
ready.load(.relaxed)          ← sees true
print(data)                   ← might still see 0 on ARM

With the pairing:

Thread A:
data = 42
ready.store(true, .releasing)  ← stlr: data write cannot sink below this

Thread B:
ready.load(.acquiring)         ← ldar: nothing below can float above this
print(data)                    ← guaranteed to see 42

The stlr/ldar pair establishes a happens-before relationship across threads. This is exactly the same hardware mechanism described in the ARM memory model section - now you’re using it directly from Swift.

How This Maps to Left-Right

Now every ordering choice in the implementation has a concrete reason:

// Reader entering - stlr
// prevents the data read below from floating above this signal
cell.epochs[slot].increment()  // .releasing

// Which copy? - plain ldr
// the .releasing increment above already acts as a barrier
let useRight = cell.readIndex.load(ordering: .relaxed)

// Read the data
let result = body(useRight ? cell.right : cell.left)

// Reader done - stlr
// prevents the data read above from sinking below this signal
cell.epochs[slot].increment()  // .releasing

// Writer flipping readIndex - stlr
// the inactive copy write cannot become visible after this flip
readIndex.store(inactiveIndex, ordering: .releasing)

// Writer polling during drain - ldar
// synchronizes with reader's .releasing increment
// guarantees writer sees reader's data access as complete
while epochs[i].load(ordering: .acquiring) == seenEpoch { }

Without .releasing on the reader’s exit increment, the CPU could reorder the data read to after the “I’m done” signal - the writer would think the reader finished when it hadn’t. Without .acquiring on the writer’s drain poll, the CPU could speculate past the while loop before the reader’s stores were actually visible.

On ARM these compile to stlr and ldar - lightweight, no full dmb ish barrier anywhere in the hot path. On x86 they compile to plain mov instructions because TSO gives you acquire-release semantics for free. swift-atomics handles the difference transparently.

The Swift Implementation

Now that we understand the algorithm, the hardware, and the primitives, the implementation becomes straightforward. Every line has a reason.

EpochCounter

The first thing we need is the epoch counter. One per reader, padded to a full cache line so there’s no false sharing between slots.

struct EpochCounter {
    var storage: UnsafeAtomic<UInt>.Storage = .init(0)

    #if arch(arm64)
    private let _pad: (UInt64, UInt64, UInt64, UInt64,
                       UInt64, UInt64, UInt64, UInt64,
                       UInt64, UInt64, UInt64, UInt64,
                       UInt64, UInt64, UInt64) =
        (0,0,0,0,0,0,0,0,0,0,0,0,0,0,0)
    #else
    private let _pad: (UInt64, UInt64, UInt64,
                       UInt64, UInt64, UInt64, UInt64) =
        (0,0,0,0,0,0,0)
    #endif

    init() {}

    func increment() {
        withUnsafeMutablePointer(to: &storage) { ptr in
            UnsafeAtomic<UInt>(at: ptr).wrappingIncrement(ordering: .releasing)
        }
    }

    func load() -> UInt {
        withUnsafeMutablePointer(to: &storage) { ptr in
            UnsafeAtomic<UInt>(at: ptr).load(ordering: .acquiring)
        }
    }
}

UnsafeAtomic<UInt>.Storage is 8 bytes embedded directly in the struct - no heap, no pointer. The padding fills the rest of the cache line: 120 bytes on ARM64 (128 - 8), 56 bytes on x86 (64 - 8). The #if arch block handles both platforms.

increment() uses .releasing - the data read must complete before this store becomes visible to the writer. load() uses .acquiring - the writer synchronizes with the reader’s release and sees the data access as truly complete. These two orderings are a pair. One without the other is wrong.

You can verify the layout is correct at compile time:

#if arch(arm64)
assert(MemoryLayout<EpochCounter>.stride == 128)
#else
assert(MemoryLayout<EpochCounter>.stride == 64)
#endif

stride not size - stride is what determines spacing between elements in an array, which is what cache line isolation depends on.

LeftRight

The container holds the two copies, the atomic flip index, the epoch counter array, and the write lock.

public final class LeftRight<T: Sendable>: @unchecked Sendable {

    internal var left: T
    internal var right: T

    internal let readIndex: ManagedAtomic<Bool>
    internal let epochs: UnsafeMutableBufferPointer<EpochCounter>

    private let writeLock = NSLock()
    private let nextSlot = ManagedAtomic<Int>(0)

    public init(_ initial: T, maxReaders: Int = 64) {
        self.left = initial
        self.right = initial
        self.readIndex = ManagedAtomic(false)
        self.epochs = UnsafeMutableBufferPointer<EpochCounter>.allocate(capacity: maxReaders)
        self.epochs.initialize(repeating: EpochCounter())
    }

    deinit {
        epochs.deinitialize()
        epochs.deallocate()
    }

    public func makeReader() -> Reader<T> {
        let slot = nextSlot.loadThenWrappingIncrement(ordering: .relaxed) % epochs.count
        return Reader(cell: self, slot: slot)
    }
}

@unchecked Sendable - Swift can’t prove this is safe because left and right are unprotected var properties. You’re telling the compiler to trust you. The epoch counters and atomic orderings are what make that claim true.

ManagedAtomic<Bool> for readIndex - this is fine as a class reference. There’s only one of it, read once per read operation. The pointer dereference is not your bottleneck.

UnsafeMutableBufferPointer<EpochCounter> for the epoch array - not [EpochCounter]. A Swift array would give you CoW semantics and no control over memory layout. The buffer pointer gives you direct access to contiguous memory that you manage yourself. allocate followed by initialize - you need both. allocate reserves the memory, initialize puts valid EpochCounter values in it. Without initialize you have garbage values in your epoch counters and the even/odd protocol breaks immediately.

nextSlot uses .relaxed - handing out slot indices is just a counter. No thread is synchronizing on this value, no ordering guarantees needed.

deinit calls deinitialize then deallocate - in that order, always. deinitialize runs the deinitializer on each element. deallocate frees the memory.

Reader

Each reader thread gets its own Reader - created once, kept for the lifetime of the thread. It knows its slot and holds a reference to the cell.

public struct Reader<T: Sendable> {
    internal let cell: LeftRight<T>
    internal let slot: Int
}

extension Reader {
    public func read<R>(_ body: (T) -> R) -> R {
        cell.epochs[slot].increment()

        let useRight = cell.readIndex.load(ordering: .relaxed)

        let result = body(useRight ? cell.right : cell.left)

        cell.epochs[slot].increment()

        return result
    }
}

This is the hot path. Four operations:

1. Increment epoch counter - signals entering read, .releasing ensures data access can’t float above this
2. Load readIndex - .relaxed is correct, the increment above already established ordering
3. Read the data - the actual work, no lock, no contention
4. Increment epoch counter again - signals done, .releasing ensures data access can’t sink below this

No lock. No shared counter. No cache line bouncing between cores. Each reader touches only its own private epoch slot and then the data. This is why reads scale.

The Write Path

extension LeftRight {
    public func write(_ operation: (inout T) -> Void) {
        writeLock.lock()
        defer { writeLock.unlock() }

        let currentReadIndex = readIndex.load(ordering: .acquiring)
        let inactiveIndex = !currentReadIndex

        if inactiveIndex == false {
            operation(&left)
        } else {
            operation(&right)
        }

        readIndex.store(inactiveIndex, ordering: .releasing)

        let snapshot = (0..<epochs.count).map { epochs[$0].load() }

        waitForReaders(snapshot: snapshot)

        if currentReadIndex == false {
            operation(&left)
        } else {
            operation(&right)
        }
    }

    private func waitForReaders(snapshot: [UInt]) {
        for (i, seenEpoch) in snapshot.enumerated() {
            guard seenEpoch & 1 == 1 else { continue }
            while epochs[i].load() == seenEpoch {
                sched_yield()
            }
        }
    }
}

Step by step:

1. Lock. NSLock enforces single-writer. In correct usage this never contends - you’re calling write from one thread. The lock is there to catch misuse, not to handle real contention.

2. Load readIndex with .acquiring. The writer needs to see the current state of the world - specifically any stores that happened before the last flip.

3. Apply to inactive copy. The copy readers are not on. No reader will touch it. No synchronization needed for the write itself.

4. Flip readIndex with .releasing. Redirects new readers to the updated copy. .releasing ensures the write to the inactive copy is visible to all cores before any reader gets redirected to it. Without this, a reader could arrive at the new copy before the writer’s mutations are visible - silent corruption on ARM.

5. Snapshot. Immediately after the flip, read all epoch counters. Any counter that is odd at this moment was mid-read on the old copy. These are the only readers the writer needs to wait for.

6. Drain. For each slot that was odd at snapshot time, spin until the counter changes. Any change means the reader finished. sched_yield() yields the thread to the OS scheduler - if the reader isn’t currently scheduled, this lets it run rather than the writer burning CPU waiting for something that can’t happen yet.

7. Apply to now-inactive copy. The old active copy is fully drained. Apply the same operation. Both copies are now identical.

What I Learned

I started this as a learning exercise. I wanted to understand how left-right works, implement it in Swift, and see what I’d learn along the way. I didn’t expect to end up this deep into ARM instruction sets, cache coherence protocols, and memory ordering models.

A few things that stuck with me:

False sharing was the biggest surprise. The idea that two completely unrelated variables can fight over cache just by sitting next to each other in memory - that’s not something you learn from writing everyday Swift code. You only discover it when you have to care about where things live in memory.

Memory orderings stopped being magic words. Before this, .acquiring and .releasing felt like incantations you copy from Stack Overflow and hope for the best. After implementing left-right, they’re just ARM instructions - ldar and stlr - with a clear, specific job. I understand what breaks if I get them wrong, because I deliberately got them wrong and watched TSan catch it.

The algorithm itself is surprisingly small. Two copies, an atomic flip, epoch counters, a drain. That’s it. The implementation is about 135 lines of Swift.

Swift gets you surprisingly close to the metal. swift-atomics, UnsafeMutableBufferPointer, manual memory layout - these aren’t C. They’re Swift. You can write lock-free concurrent data structures in Swift without dropping into C, and the result is correct, readable, and fast.

The implementation is a learning exercise and is scoped accordingly - full replacement writes only, fixed reader slots, Apple Silicon focused. There’s more to build. But as a foundation for understanding how this class of primitive works, I’m happy with it.

Sources

• swift-left-right - my Swift implementation
• Left-Right: A Classical Algorithm - Pedro Ramalhete and Andreia Correia, the original description of the primitive
• jonhoo/left-right - Jon Gjengset’s Rust implementation, the direct inspiration for this project
• The Cost of Concurrency Coordination - Jon Gjengset’s video that started all of this
• apple/swift-atomics - Apple’s official Swift atomics package
• ARM Architecture Reference Manual - the definitive source on ldar, stlr, and the ARM memory model

The addition of C++ interoperability in Swift (a topic for another post) is fantastic for transitioning legacy Objective-C++ projects to Swift. Additionally, there’s a specific area I’ve been wanting to explore for a while: audio development on iOS. So, I decided to combine my interest in Swift with AVAudioEngine, and that’s how the JSWaveform was born. You can check out the Swift package on my Github.

In this post, I will delve into the intricacies of an Audio Engine written in Swift, leveraging the power of Apple’s AVFoundation framework. We’ll explore how to manage audio playback, apply pitch effects, and handle asynchronous audio buffering in a robust and efficient manner.

What is AVAudioEngine?

AVAudioEngine is an advanced audio framework that lets you build intricate audio processing chains using a graph of audio nodes. Each node performs a specific function, such as playing an audio file, applying effects, or mixing multiple audio streams. The modularity of AVAudioEngine allows you to customize and extend audio processing according to your application’s needs.

Key Components of AVAudioEngine

Understanding the key components of AVAudioEngine is essential for leveraging its full potential:

1. Nodes: Nodes are the fundamental units in AVAudioEngine. They perform various audio tasks and can be connected to form complex audio processing graphs.

   • AVAudioInputNode: Captures audio from the microphone.
   • AVAudioOutputNode: Outputs processed audio to the device’s speakers or headphones.
   • AVAudioPlayerNode: Plays audio files or buffers.
   • AVAudioUnitEffect: Applies audio effects such as reverb or delay.
   • Engine: The AVAudioEngine class manages the audio nodes and the connections between them, ensuring synchronized processing.
2. AVAudioEngine: The core class that orchestrates the entire audio processing flow.
3. Buffers: Buffers temporarily store audio data during processing, enabling smooth transitions and real-time manipulation.

JSWaveform

JSWaveform is a Swift Package that has native interfaces consisting of audio engine and pure animatable SwiftUI components in iOS, iPadOS and visionOS.

JSWaveform provides native Swift and SwiftUI components. For now, it has 2 major SwiftUI views:

• AudioPlayerView - renders audio player which consits of play/pause button, downsampled waveform and time pitch effect button.
• AudioVisualizerView - renders audio visualizer which animates AudioVisualizerShape based on audio amplitudes.

All the code for manipulating audio signals follows JSWaveform.

Setting Up the Audio Engine

We define an AudioEngine actor to encapsulate the functionality:

actor AudioEngine {
    
    private let avAudioEngine = AVAudioEngine()
    private let audioPlayer = AVAudioPlayerNode()
    private let audioTimePitch = AVAudioUnitTimePitch()
    private var audioBuffer: AVAudioPCMBuffer?
    private var asyncBufferStream: AsyncStream<AVAudioPCMBuffer>?
    private var continuation: AsyncStream<AVAudioPCMBuffer>.Continuation?
    
    private let logger = Logger(subsystem: "JSWaveform.AudioEngine", category: "Engine")
    
    enum AudioEngineError: Error {
        case bufferRetrieveError
    }
    
    init() {
        avAudioEngine.attach(audioPlayer)
        avAudioEngine.attach(audioTimePitch)
    }
}

The setup function connects the various nodes and prepares the engine for playback:

nonisolated func setup() {
    let output = avAudioEngine.outputNode
    let mainMixer = avAudioEngine.mainMixerNode

    avAudioEngine.connect(audioPlayer, to: audioTimePitch, format: nil)
    avAudioEngine.connect(audioTimePitch, to: mainMixer, format: nil)
    avAudioEngine.connect(mainMixer, to: output, format: nil)
    avAudioEngine.prepare()
}

We prepare an asynchronous audio buffer stream to handle audio data efficiently:

func prepareBuffer() {
    asyncBufferStream = AsyncStream { continuation in
         self.continuation = continuation
     }
    
    audioPlayer.installTap(onBus: 0, bufferSize: 256, format: nil) { buffer, _ in
        self.continuation?.yield(buffer)
    }
}

func getBuffer() -> AsyncStream<AVAudioPCMBuffer>? {
    return asyncBufferStream
}

The audioPlayer.installTap(onBus:bufferSize:format:block:) method is an essential part of the AudioEngine class, which taps into the audio signal coming from the audioPlayer node, allowing you to process or analyze audio data in real-time. Let’s break down the components and significance of this method call.

The installTap method in AVAudioNode allows you to create a tap on a specified bus. This tap intercepts the audio data flowing through the node, providing you with a buffer of audio samples that you can use for various purposes like visualization, analysis, or further processing.

Parameters of the installTap method:

• onBus: The bus number you want to tap into. For most audio nodes, bus 0 is the default output bus.
• bufferSize: The number of audio samples per channel in each buffer that the tap handler receives. In this case, it is set to 256.
• format: The audio format of the tap. Passing nil means the format of the tap will be the format of the node’s output bus.
• block: A block (closure) that processes the audio buffers. This block is called with each buffer of audio data.

The block (closure) provided to installTap is where the audio processing happens:

• buffer: This is the AVAudioPCMBuffer containing the audio samples tapped from the audio node.
• (AVAudioTime): This parameter provides the time at which the buffer’s audio data is rendered. In this case, it’s ignored.

The closure yields the buffer to the AsyncStream continuation. This line sends the tapped audio buffer to the AsyncStream, making it available for asynchronous processing. This approach leverages Swift’s concurrency model to handle audio data efficiently, ensuring smooth real-time performance.

Why Buffer Size 256?

The buffer size of 256 frames (samples per channel) is chosen for several reasons:

• Latency vs. Performance Trade-Off: A smaller buffer size means lower latency, which is critical for real-time audio applications. However, smaller buffers require the CPU to handle more frequent interrupts, increasing the processing load. A buffer size of 256 is a balanced choice, providing relatively low latency without overwhelming the CPU.

• Consistency with Audio Standards: The buffer size of 256 frames is commonly used in audio processing as it aligns well with typical audio sample rates (like 44.1kHz or 48kHz), resulting in efficient processing and compatibility with audio hardware and software standards.

• Smooth Real-Time Processing: For real-time applications like audio playback, effects, and recording, using a buffer size of 256 ensures smooth operation. It allows the audio engine to deliver consistent audio data with minimal delay, enhancing the user experience in interactive audio applications.

Audio Processing

For animations, we need to processes an audio buffer to calculate the power levels (both average and peak) for each channel in the buffer.

func process(buffer: AVAudioPCMBuffer) {
    var powerLevels = [PowerLevels]()
    let channelCount = Int(buffer.format.channelCount)
    let length = vDSP_Length(buffer.frameLength)

    if let floatData = buffer.floatChannelData {
        for channel in 0..<channelCount {
            powerLevels.append(calculatePowers(data: floatData[channel], strideFrames: buffer.stride, length: length))
        }
    } else if let int16Data = buffer.int16ChannelData {
        for channel in 0..<channelCount {
            var floatChannelData: [Float] = Array(repeating: Float(0.0), count: Int(buffer.frameLength))
            vDSP_vflt16(int16Data[channel], buffer.stride, &floatChannelData, buffer.stride, length)
            var scalar = Float(INT16_MAX)
            vDSP_vsdiv(floatChannelData, buffer.stride, &scalar, &floatChannelData, buffer.stride, length)

            powerLevels.append(calculatePowers(data: floatChannelData, strideFrames: buffer.stride, length: length))
        }
    } else if let int32Data = buffer.int32ChannelData {
        for channel in 0..<channelCount {
            var floatChannelData: [Float] = Array(repeating: Float(0.0), count: Int(buffer.frameLength))
            vDSP_vflt32(int32Data[channel], buffer.stride, &floatChannelData, buffer.stride, length)
            var scalar = Float(INT32_MAX)
            vDSP_vsdiv(floatChannelData, buffer.stride, &scalar, &floatChannelData, buffer.stride, length)

            powerLevels.append(calculatePowers(data: floatChannelData, strideFrames: buffer.stride, length: length))
        }
    }
    self.values = powerLevels
}

If the buffer contains floating-point data, each channel’s data is processed directly. If the buffer contains 16-bit integer data, it is converted to floating-point data. The conversion is performed using the Accelerate framework’s vDSP_vflt16 function. The data is then normalized by dividing by INT16_MAX. The same calculations are done if the channel contains 32-bit integer data.

private func calculatePowers(data: UnsafePointer<Float>, strideFrames: Int, length: vDSP_Length) -> PowerLevels {
    var max: Float = 0.0
    vDSP_maxv(data, strideFrames, &max, length)
    if max < kMinLevel {
        max = kMinLevel
    }

    var rms: Float = 0.0
    vDSP_rmsqv(data, strideFrames, &rms, length)
    if rms < kMinLevel {
        rms = kMinLevel
    }

    return PowerLevels(average: 20.0 * log10(rms), peak: 20.0 * log10(max))
}

This function calculates the Root Mean Square (RMS) and peak power levels from the audio data. The function returns these values encapsulated in a PowerLevels struct. Here is a detailed explanation of each component:

1. RMS (Root Mean Square):

   • The RMS value is a measure of the average power of the audio signal.
   • It provides a meaningful average level of the waveform’s power.
   • The vDSP_rmsqv function from the Accelerate framework computes the RMS value of the audio data.
   • If the RMS value is less than kMinLevel, it is set to kMinLevel to avoid taking the logarithm of zero. kMinLevel is defined as kMinLevel: Float = 0.000_000_01 // -160 dB.

2. Peak Level:

   • The peak value represents the maximum power level of the audio signal.
   • It indicates the highest amplitude in the audio data.
   • The vDSP_maxv function from the Accelerate framework finds the maximum value in the audio data.

3. Decibel Conversion:

   • Audio levels are often represented in decibels (dB) because the human ear perceives sound logarithmically.
   • The conversion from linear power values to decibels is done using the logarithm function:

    let average = 20.0 * log10(rms)
    let peak = 20.0 * log10(max)
   

   • Multiplying by 20 converts the logarithm of the power ratio into decibels.

AudioVisualizer

Now when we have some data, we need to use it. In our Model, we process audio buffer and update our observable amplitudes. To have smooth animations based on screen refresh rate, we need to use CADisplayLink.

public func processAudio() async {
    guard let bufferStream = await audioEngine.getBuffer() else { return }

    for await buffer in bufferStream {
        if isPlaying {
            audioProcessing.process(buffer: buffer)
        } else {
            audioProcessing.processSilence()
        }
    }
}

// MARK: DisplayLink update method
override func update() {
    
    guard let levels = audioLevel.levelProvider?.levels else {
        return
    }
    
    level = CGFloat(levels.level)
    peakLevel = CGFloat(levels.peakLevel)
    
    var targetAmplitudes:[Double]

    // Example calculation
    let halfCount = amplitudes.count / 2
    let newAmplitudes = (0..<halfCount).map { index in
        let normalizedIndex = Double(index) / Double(halfCount - 1)
        let scalingFactor = 1.0 + normalizedIndex * 1.5
        let amplitude = level * (1.0 - normalizedIndex) + peakLevel * normalizedIndex
        return min(1.0, max(0.0, pow(amplitude * scalingFactor, 1.2)))
    }
    targetAmplitudes = newAmplitudes + newAmplitudes.reversed()

    if peakLevel > 0 {
        self.amplitudes = self.amplitudes.enumerated().map { index, current in
            self.lowPassFilter(currentValue: current, targetValue: targetAmplitudes[index])
        }
    }
}

private func lowPassFilter(currentValue: Double, targetValue: Double, smoothing: Double = 0.1) -> Double {
    return currentValue * (1.0 - smoothing) + targetValue * smoothing
}

lowPassFilter function is a simple yet effective method for smoothing or filtering a signal. This particular implementation is often used in audio processing, sensor data smoothing, and other applications where you want to reduce noise or fluctuations in a signal. We use it here for smooth animations.

AudioPlayer

But what when you want to play audio file, and show a waveform, and skip audio while dragging on waveform like, for example, voice message?

func loadSamples(forURL url: URL, downsampledTo targetSampleCount: Int) async throws -> [Float] {
    logger.debug("Loading audio samples for URL: \(url), downsampled: \(targetSampleCount)")
    
    let file = try? AVAudioFile(forReading: url)
    guard let format = file?.processingFormat, let length = file?.length else { throw WafevormError.audioFileNotFound }
    
    let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(length))
    try? file?.read(into: buffer!)
    
    guard let floatChannelData = buffer?.floatChannelData else { throw WafevormError.bufferRetrieveError }
    let channelData = floatChannelData.pointee
    
    let sampleCount = Int(length)
    let samplesPerPixel = sampleCount / targetSampleCount
    var downsampledData = [Float]()
    
    for i in 0..<targetSampleCount {
        let start = i * samplesPerPixel
        let end = min((i + 1) * samplesPerPixel, sampleCount)
        let sampleRange = start..<end
        
        let maxSample = sampleRange.map { channelData[$0] }.max() ?? 0
        downsampledData.append(maxSample)
    }
    
    return downsampledData
}

func loadSamples(downsampledTo: Int) async throws -> [Float] {
    let samples = try await waveformLoader.loadSamples(forURL: audioURL, downsampledTo: downsampledTo)
    normalizedSamples = normalizeWaveformData(samples)
    return normalizedSamples
}

private func normalizeWaveformData(_ data: [Float]) -> [Float] {
    guard let maxSample = data.max() else { return [] }
    return data.map { $0 / maxSample }
}

These functions loads audio data from a file, downsamples it to a specified number of samples, normalize the data and returns the downsampled data. The process involves reading the audio file, creating a buffer to hold the data, retrieving the data, calculating the appropriate downsampling parameters, and then iterating through the data to extract and store the downsampled samples. The use of the maximum value within each downsampled segment helps preserve the peaks in the audio data. You’ll get something like this:

Conclusion

AVAudioEngine is a versatile and powerful tool that opens up a world of possibilities for iOS developers. By mastering its components and features, you can create rich and immersive audio experiences in your applications. Whether you are developing a simple audio player or a sophisticated audio processing app, AVAudioEngine provides the tools you need to bring your audio visions to life. Happy coding!

Basic Principles of Discrete FT (DFT)

The DFT is particularly useful in image processing because it provides a way to analyze the content of an image in terms of its frequency components. The basic idea is that any image can be represented as a sum of sinusoidal functions of varying magnitudes, frequencies, and phases. This transformation from spatial to frequency domain is crucial for many applications because alterations and enhancements in the frequency domain can be inversely transformed back to the spatial domain, allowing modified images to be reconstructed with desired characteristics.

Formula

The DFT is essentially a sampled version of the Fourier Transform and thus captures only a subset of frequencies, sufficient to comprehensively describe the original image in the spatial domain. The set of sampled frequencies in the DFT corresponds directly to the pixel count in the spatial domain image. Consequently, both the spatial domain image and its Fourier transform representation maintain the same dimensions. Given a discrete square image F(x,y) of size NxN the DFT is:

DFT

Let’s see in this example. I took a picture of the tennis balls can. Using OpenCV and DFT (OpenCV uses Fast Fourier Transform - it’s an algorithm that computes DFT in \(O(NlogN)\) instead of \(O(N^2)\) time) we get this:

The image is represented in a complex form, comprising both magnitude and phase components. For visualization purposes, the magnitude component is particularly significant as it reflects the strength of each frequency component, offering a clear insight into the image’s frequency spectrum. However, because the center of the image typically exhibits significantly higher values than other areas, magnitudes are often displayed on a logarithmic scale to compress the dynamic range and enhance visibility.

Within the Fourier domain, the central areas correspond to low-frequency components, while the outer regions denote high-frequency components. The very center of the image signifies the DC value, which is the zero frequency component representing the overall intensity of the image.

Compressing the dynamic range of an image can be achieved by transforming each pixel value into its logarithm. This transformation effectively enhances the visibility of lower intensity pixels. Such a technique is particularly valuable in scenarios where the dynamic range is too extensive for conventional display screens or to be captured accurately by film. The logarithmic operation functions as a straightforward point processor, applying a logarithmic curve to map the original pixel values. While the choice between using a natural logarithm or a base 10 logarithm varies, it primarily affects the scale of the output values rather than the curve’s shape. This ensures that regardless of the logarithmic base used, the compression effect on the dynamic range remains consistent. The values are then scaled to suit an 8-bit display system. The mapping function for this operation can be described as follows:

Inverse DFT

The Inverse Discrete Fourier Transform (IDFT) is an essential mathematical tool in digital signal processing and image processing, serving as the counterpart to the Discrete Fourier Transform (DFT). While the DFT converts a signal or image from the spatial domain (or time domain) to the frequency domain, the IDFT performs the reverse, transforming the data back to its original spatial domain from the frequency domain. This process is crucial for applications that modify signals in the frequency domain and require a reconstruction of the original signal or image.

The mathematical formula for the IDFT in a two-dimensional space, typical for images, is expressed as follows for NxN images:

Filters

Filters in image processing function exactly as their name implies—they filter out unwanted elements. These filters are generally implemented as a mask array, the same size as the original image. When this mask is placed over the original image, it selectively retains only the desired attributes.

As previously discussed, in an image transformed by the DFT, low frequencies are located at the center, while high frequencies are dispersed around the periphery. By designing a mask array with a central circle of zeros surrounded by ones, this filter can be applied to emphasize high frequencies in the image.

There are mainly three types of filter used:

• Low Pass Filter (LPF)
• High Pass Filter (HPF)
• Band Pass Filter (BPF)

Edge Detection

Edge detection is a crucial technique in computer vision. By identifying the edges within an image, we can utilize this information for feature extraction or pattern detection purposes.

Edges are typically characterized by high frequencies in an image. Therefore, after performing a DFT on an image to convert it into the frequency domain, a High-Pass Filter is applied. This filter effectively blocks all low frequencies while allowing high frequencies to pass through. Subsequently, applying an inverse DFT to this filtered image reveals distinct edge features in the original image, making it a powerful tool for analysis and processing in various applications.

When applying HPF on the image above, and inverse DFT, we get:

Blur Detection

Detecting blur in an image using the Fourier Transform involves analyzing the frequency domain representation of the image to determine the absence or presence of high-frequency components, which are typically diminished in blurred images. Below, I provide a C++ example that uses the OpenCV library to perform this task.

#include <opencv2/opencv.hpp>
#include <iostream>

bool isBlurry(const cv::Mat& image) {
    cv::Mat gray, floatGray, dftImage, magnitudeImage;
    
    // Convert the image to grayscale
    cv::cvtColor(image, gray, cv::COLOR_BGR2GRAY);
    
    // Convert the grayscale image to float
    gray.convertTo(floatGray, CV_32F);
    
    // Compute the DFT
    cv::dft(floatGray, dftImage, cv::DFT_COMPLEX_OUTPUT);
    
    // Compute the magnitude of the complex numbers (real and imaginary)
    std::vector<cv::Mat> planes;
    cv::split(dftImage, planes); // planes[0] = Re(DFT(I)), planes[1] = Im(DFT(I))
    cv::magnitude(planes[0], planes[1], magnitudeImage);
    
    // Shift the DC component to the center of the image
    int cx = magnitudeImage.cols / 2;
    int cy = magnitudeImage.rows / 2;
    cv::Mat q0(magnitudeImage, cv::Rect(0, 0, cx, cy));   // Top-Left
    cv::Mat q1(magnitudeImage, cv::Rect(cx, 0, cx, cy));  // Top-Right
    cv::Mat q2(magnitudeImage, cv::Rect(0, cy, cx, cy));  // Bottom-Left
    cv::Mat q3(magnitudeImage, cv::Rect(cx, cy, cx, cy)); // Bottom-Right

    // Swap quadrants (Top-Left with Bottom-Right, Top-Right with Bottom-Left)
    cv::Mat tmp;
    q0.copyTo(tmp);
    q3.copyTo(q0);
    tmp.copyTo(q3);
    q1.copyTo(tmp);
    q2.copyTo(q1);
    tmp.copyTo(q2);
    
    // Threshold to determine the presence of significant high frequencies
    double meanValue = cv::mean(magnitudeImage)[0];
    std::cout << "Mean value of DFT magnitude: " << meanValue << std::endl;

    // Threshold the mean value to determine if the image is blurry
    return meanValue < 0.2; // Adjust the threshold based on experimentation
}

Let’s see what’s happening here:

1. Grayscale Conversion: The image is converted to grayscale because the color information is not necessary for blur detection.
2. Fourier Transform: The grayscale image is converted to the frequency domain using the DFT.
3. Magnitude Calculation: The magnitude of the DFT results is calculated to analyze the frequency components.
4. Quadrant Swapping: This step shifts the zero-frequency component to the center of the spectrum, which makes analysis more intuitive.
5. Mean Value Check: By calculating the mean of the magnitude spectrum, you can infer the presence of high-frequency components. A lower mean suggests fewer high frequencies, indicative of a blur.

What is YUV?

YUV refers to a color encoding system used in video applications. It separates the luminance (brightness) and chrominance (color information) into different components. This separation is advantageous for video compression, as the human eye is more sensitive to variations in brightness than color. YUV formats are extensively used in video compression algorithms, including MPEG and H.264.

Components of YUV

• Y: Represents the luminance component, which captures the brightness level of the video.
• U and V: Represent chrominance components, which capture the color information. U indicates how much blue-colored light is in the image, minus the luminance, while V indicates how much red-colored light there is, minus the luminance.

YUV Subformats

YUV has several subformats, categorized based on how the U and V components are sampled compared to the Y component. These subformats include YUV420, YUV422, and YUV444, among others. The numbers indicate the ratio of luminance to chrominance sampling, affecting the image quality and compression ratio.

Check out this great site for great YUV pixel formats explanations. I have it bookmarked and I open it every time when I need to do something with YUVs.

The NV12 Format

NV12 is a specific type of YUV 4:2:0 format where chrominance components are stored as an interleaved UV plane following the Y plane. This format is particularly efficient for hardware decoding and rendering, making it widely used in mobile devices, cameras, and for streaming video content.

Structure of NV12

The Y component occupies the first part of the image data, with one byte per pixel, representing the luminance. The UV component follows the Y data, with alternating U and V values. Each U and V pair is shared by four Y pixels, forming a macro pixel. Some advantages of NV12 formats are:

• Efficiency: The NV12 format is highly efficient for processing and storage, reducing bandwidth and storage requirements without significantly compromising image quality.
• Compatibility: NV12 is supported by a wide range of hardware decoders and media frameworks, enhancing interoperability across different platforms and devices.

Visualization of NV12 Format

The best visual representation I have seen so far is the visual representation on its Wiki page.

iOS and YUV

When dealing with YUVs on Apple platforms, you’ll most likely use kCVPixelFormatType_420YpCbCr8BiPlanarFullRange pixel format. It is a specific type of video pixel format used within the Core Video framework on iOS and macOS platforms. It’s essential for developers working with video capture, processing, or playback applications on Apple devices. This format is closely related to the NV12 format discussed in the context of YUV color spaces, with some specific characteristics tailored to the iOS ecosystem.

Key Characteristics

• Bi-Planar: The format is bi-planar, meaning the Y (luminance) and CbCr (chrominance) components are stored in two separate planes. The Y plane contains all the luminance data, with one byte per pixel. The CbCr plane contains interleaved chrominance data, with one byte for Cb (chroma blue) and one byte for Cr (chroma red) for every two pixels.
• Full Range: The “Full Range” in its name indicates that the luminance component (Y) uses the full 8-bit range from 0 to 255, unlike the video range (16 to 235) typically used in video encoding. This full range provides better detail in both the shadows and highlights of an image, making it suitable for high-quality video applications.
• 420 Chroma Subsampling: This format implements 4:2:0 chroma subsampling, where the chrominance information is sampled at half the horizontal and vertical resolution of the luminance. This approach reduces the amount of data needed to represent a color image, which is beneficial for reducing file sizes and bandwidth usage without significantly impacting perceived image quality.

Rotation

I had a fun work with this format recently where I needed to rotate the YUV for 90 degrees. When dealing with pixel buffers, it is important to now where the origin point. The default pixel buffer orientation on iOS is landscape right. Origin point is on the upper left corner. But what about when you change the connection’s pixel buffer video orientation or video rotation angle? Then the origin point is different. Depending on your ML models, preview layers rotation or something else, you’ll most likely rotate the YUV to its desired orientation.

Rotating a YUV420 image by 90 degrees involves manipulating the Y plane and the UV plane separately due to the semi-planar format of YUV420. In YUV420, the Y plane is full resolution while the UV (chroma) plane is subsampled by a factor of 2 both horizontally and vertically. The code is written in Objective-C. Really like Objective-C more than Swift, don’t hate me :)

Common

size_t originalHeight = CVPixelBufferGetHeight(pixelBuffer);
size_t originalWidth = CVPixelBufferGetWidth(pixelBuffer);
size_t rotatedWidth = originalHeight; // Swap the dimensions for rotation
size_t rotatedHeight = originalWidth;
size_t originalYLumaStride = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0); // For Y plane
size_t originalUVStride = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 1); // For the CbCr plane
        
rotatedPixelBuffer = nil;
CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, rotatedWidth, rotatedHeight, kCVPixelFormatType_420YpCbCr8BiPlanarFullRange, NULL, &rotatedPixelBuffer);
(void)status;
CVPixelBufferLockBaseAddress(rotatedPixelBuffer, 0);        

Y Plane Rotation

+ (void)rotateYPlaneFor:(CVImageBufferRef)pixelBuffer rotatedPixelBuffer:(CVImageBufferRef)rotatedPixelBuffer {
	uint8_t *originalYBaseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);
    uint8_t *rotatedYBaseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(rotatedPixelBuffer, 0);
    for (size_t x = 0; x < originalWidth; x++) {
        for (size_t y = 0; y < originalHeight; y++) {
            size_t originalIndex = y * originalYLumaStride + x;
            size_t rotatedIndex;
            if (isCounterClockwise) {
                rotatedIndex = (originalWidth - x - 1) * originalHeight + y; // Counterclockwise
            }
            else {
                rotatedIndex = x * originalHeight + (originalHeight - y - 1); // Clockwise
            }
            rotatedYBaseAddress[rotatedIndex] = originalYBaseAddress[originalIndex];
        }
    }
}

UV Plane Rotation

+ (void)rotateCbCrPlaneFor:(CVImageBufferRef)pixelBuffer rotatedPixelBuffer:(CVImageBufferRef)rotatedPixelBuffer {
    uint8_t *originalCbCrBaseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 1);
    uint8_t *rotatedCbCrBaseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(rotatedPixelBuffer, 1);
    
    for (size_t x = 0; x < originalWidth; x += 2) { // Half resolution for CbCr
        for (size_t y = 0; y < originalHeight / 2; y++) {
            size_t originalIndex = y * originalUVStride + x;
            size_t rotatedIndex;
            if (isCounterClockwise) {
                rotatedIndex = ((originalWidth - x - 2) / 2) * originalHeight + (y * 2); // Counterclockwise
            }
            else {
                rotatedIndex =  (x / 2) * originalHeight + (originalHeight / 2 - y - 1) * 2; // Clockwise
            }

            rotatedCbCrBaseAddress[rotatedIndex] = originalCbCrBaseAddress[originalIndex];
            rotatedCbCrBaseAddress[rotatedIndex + 1] = originalCbCrBaseAddress[originalIndex + 1];
        }
    }
}

Don’t forget

Lock the adresses when you start:

CVPixelBufferLockBaseAddress(pixelBuffer, 0);
CVPixelBufferLockBaseAddress(rotatedPixelBuffer, 0);

and release when you are done:

CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
CFRelease(pixelBuffer);
CVPixelBufferUnlockBaseAddress(rotatedPixelBuffer, 0);
CFRelease(rotatedPixelBuffer);

I had an interesting problem last year that I wanted to tackle; how to add Swift files to a framework written in Objective-C through CMake and make it work.

The basic concept is:

1. Add a Swift file to your sources. This is important, if you don't Swift Compiler will be turned off. You can create an empty Swift file (dummy).
2. We need to turn on and force Swift Compiler (swiftc) for CMake to recognize its 	builds settings.
3. Add CMake SET for swiftc build settings
4. Remove other Swift flags
5. Use Swift :)

Let’s tackle one by one.

• Add a Swift file to your sources; add this where you list your source files for cmake:

list( APPEND <your_source_list>
...
<path>/Empty.swift
...
)

• Force load swiftc in your <project>.cmake:

execute_process(
    COMMAND xcrun --find swiftc
    OUTPUT_VARIABLE SWIFTC_EXECUTABLE
    OUTPUT_STRIP_TRAILING_WHITESPACE)
set( SWIFTC_FOUND TRUE )
message( STATUS "Found swiftc: ${SWIFTC_EXECUTABLE}" )

if ( SWIFTC_FOUND )
    message(STATUS "Setting swiftc: ${SWIFTC_EXECUTABLE}")
    # We need to tell cmake where swiftc is
    set( CMAKE_Swift_COMPILER ${SWIFTC_EXECUTABLE} )
    # This line is important, we need to force load it
    set( CMAKE_Swift_COMPILER_FORCED ON ) 
endif()

if( CMAKE_Swift_COMPILER )
  # Enable Swift, tell cmake there's now new swiftc build settings, without this, cmake can't properly configure attributes
  enable_language(Swift)
else()
  message(FATAL_ERROR "FATAL ERROR! Did not find swiftc?!")
endif()

• Add swiftc build settings to your <project>.cmake:

# Swift Features
message( STATUS "Setting Swift Settings:" )
# Swift requires modules
set( CMAKE_XCODE_ATTRIBUTE_DEFINES_MODULE "YES" )
# Enable modules for ObjectiveC - Swift
set( CMAKE_XCODE_ATTRIBUTE_CLANG_ENABLE_MODULES "YES" )
# Set Swift to ObjC Interface name
set( CMAKE_XCODE_ATTRIBUTE_SWIFT_INSTALL_OBJC_HEADER "YES")
set( CMAKE_XCODE_ATTRIBUTE_SWIFT_OBJC_INTERFACE_HEADER_NAME "${FRAMEWORK_OUTPUT_NAME}-Swift.h" )
# Set name for the source code module constructed for this target, and which will be used to import the module in implementation source files
set( CMAKE_XCODE_ATTRIBUTE_PRODUCT_MODULE_NAME "${FRAMEWORK_OUTPUT_NAME}" )
# Set paths to be searched by the Swift compiler for additional Swift modules. This is used to use ObjectiveC in Swift. Set all ObjC private headers to use for internal use in module.modulemap
set( CMAKE_XCODE_ATTRIBUTE_SWIFT_INCLUDE_PATHS "<path>/ProjectModule" )
# Set Swift Version
set( CMAKE_XCODE_ATTRIBUTE_SWIFT_VERSION "5.0" )
set( CMAKE_XCODE_ATTRIBUTE_SWIFT_OPTIMIZATION_LEVEL "$<$<CONFIG:Release>:-O>$<$<CONFIG:Debug>:-Onone>" )
# Set for every target
set ( CMAKE_XCODE_ATTRIBUTE_OTHER_SWIFT_FLAGS "")

• Remove other Swift flags:

set_target_properties( ${PROJECT_NAME} PROPERTIES
FRAMEWORK TRUE
...
XCODE_ATTRIBUTE_OTHER_SWIFT_FLAGS ""
XCODE_ATTRIBUTE_BUILD_LIBRARY_FOR_DISTRIBUTION "YES"
...
)

• Use Swift. But be aware, you’ll need to wrap some things when using Swift in the Objective-C file if you want them private :)

Feel free to message me on Twitter if you want to talk about this, or anything tech related :)

actor GameScore {
    let name: String
    
    //...
}

The property name is immutable, so there’s no need to mark it directly with nonisolated keyword; it is safe to access from non-isolated state. Let’s add one more immutable property place and computed property venue:

actor GameScore {
    let name: String
    let place: String
    
    var venue: String {
        "See a game: \(name) at place: \(place)"
    }
    
    //...
}

If we accessed the venue, we would get the following error:

Actor-isolated property venue can not be referenced from a non-isolated context

To fix this, we must tell the compiler it is safe to access the property venue by marking the property nonisolated:

actor GameScore {
    let name: String
    let place: String
    
    nonisolated var venue: String {
        "See a game: \(name) at place: \(place)"
    }
    
    //...
}

Let’s try to remove venue and add CustomStringConvertible protocol conformance. We will get this:

extension GameScore: CustomStringConvertible {
    var description: String {
        "See a game: \(name) at place: \(place)"
    }
}

And when we try to use it, we will get the error:

Actor-isolated property ‘description’ cannot be used to satisfy a protocol requirement

To fix this and to use description, we need to mark it with a nonisolated keyword:

extension GameScore: CustomStringConvertible {
    nonisolated var description: String {
        "See a game: \(name) at place: \(place)"
    }
}

Conclusion

Actors are straightforward and neat ways to deal with asynchronous access to mutable data. We can use isolated and nonisolated keywords to control mutable and immutable states’ access precisely.

What is an Actor?

Swift has various types that we can use: classes, structs, and enums. Classes are reference types, and structs are value types. When you use classes, you must remember that they declare shared mutable states across the program. We can see that that can be tricky when you mix them with concurrency. You must do manual synchronization to avoid data races. Data races can occur when two threads simultaneously access and write the same data. Data races often lead to strange and unpredictable behavior in your application.

An Actor is a reference type that protects access to its mutable state from data races. You define an Actor using an actor keyword:

actor GameScore {...}

An Actor can have initializers, methods, properties, and subscripts. It can conform to protocols, extend them and work with generics.

An Actor is a reference type as a class, but there is one thing that you can’t do with an Actor, and that is inheritance:

actor BasketballGameScore: GameScore {}

You will get this error:

Actor types do not support inheritance

Synchronization - preventing Data Races

We would use various locks to create synchronized access to mutable data to prevent data races. Let’s look at the following example where we want to track the latest game score, for example, basketball. We want to know the latest score, and we want to add the new score:

class GameScore {
    private var score = [String]()
    
    var latestScore: String? {
        return score.last
    }
    
    func add(_ newScore: String) {
        score.append(newScore)
    }
}

This code is fine until we don’t use it in a multi-threaded environment. We can get into a lot of problems if we use this concurrently. To fix this, we can go by the route to do all our reads and writes on the specific DispatchQueue to ensure that all the operations execute in serial order:

class GameScore {
    private var score = [String]()
    private let queue = DispatchQueue(label: "gameScore.queue")
    
    var latestScore: String? {
        queue.sync {
            return score.last
        }
    }
    
    func add(_ newScore: String) {
        queue.sync {
            score.append(newScore)
        }
    }
}

If we want to have a concurrent specific DispatchQueue, we can sync reads and writes by the barrier flag:

class GamesScoreQueue {
    private var score = [String]()
    private let queue = DispatchQueue(label: "gamesCore.queue.concurrent", attributes: .concurrent)
    
    var latestScore: String? {
        queue.sync(flags: .barrier) {
            return score.last
        }
    }
    
    func add(_ newScore: String) {
        queue.sync(flags: .barrier) {
            score.append(newScore)
        }
    }
}

As we can see, we have some manual synchronization to do. To tackle this, we use an Actor. It automatically serializes all synchronized access to its properties and methods. When we refactor our GameScore class to an Actor, we get this:

actor GameScore {
    var score = [String]()
    
    var latestScore: String? {
        return score.last
    }
    
    func add(_ newScore: String) {
        score.append(newScore)
    }
}

Accessing Actor’s data?

We don’t know when access is allowed and when another thread will perform access to mutable data, so we can’t just access the mutable property directly. To create asynchronous access, we use await keyword:

let gameScore = GameScore()
await gameScore.latestScore

Data Races can still occur when using Actors

Actors can help prevent data races, and they give us an effortless way to synchronize access preventing weird crashes and behaviors. But we need to be careful; they are not a universal solution in a multi-threaded environment where we can forget about data races and race conditions. What if we have two queues, one accessing our score and one storing it:

queueOne.async {
    print(await gamesScore.latestScore)
}

queueTwo.async {
    await gamesScore.add("30:32")
}

We don’t know which one will fire first, so we can have different behavior and inconsistent game score.

Conclusion

Actors are straightforward and neat ways to deal with asynchronous access to mutable data. They provide exquisite, readable, and one-way solutions to prevent data races. In future articles, I’ll cover Actor Isolation, Sendable, and MainActor. So stay tuned!

At Microblink, we have developed some fascinating things in the Computer Vision area. Yes, of course, we use Machine Learning :) 

On this site, I will focus on the framework side of iOS development, write some tips and tricks, and see where we can go from there. If you have any questions, feel free to contact me! Have a great day!