Chapter 29: Performance, Optimization, and Memory Layout

This chapter looks at how data choices and low level mechanics shape speed. You will organize memory for predictable access, help the compiler inline wisely, play nicely with caches and branches, leverage small object and empty base optimizations, and measure with care. Optimization starts with evidence; change code only after you can explain a slow path and predict the gain.

Data oriented design basics

Data oriented design focuses on how bytes move through the machine. Layout and access patterns often matter more than clever algorithms. Keep hot data packed tightly, avoid pointer chasing inside tight loops, and stream sequentially so hardware prefetchers can help.

Array of Structures vs Structure of Arrays

An array of structures groups fields together per object. A structure of arrays groups the same field across many objects. Choose the layout that matches your access pattern. If a loop touches only one field, a structure of arrays can load fewer bytes and improve cache locality.

struct Particle { float x, y, z; float mass; };
std::vector<Particle> aos;
// SoA variant
struct Particles { std::vector<float> x, y, z, mass; } soa;

Alignment, padding, and packing pressure

Compilers insert padding to satisfy alignment. Reorder fields to reduce wasted space, but do not fight natural alignment for hot numeric types. Misaligned or packed fields can hurt performance on some targets. Favor naturally aligned arrays for vectorization.

struct Bad { char tag; double value; int count; };   // likely padding after tag
struct Good { double value; int count; char tag; };  // tighter layout
static_assert(sizeof(Good) <= sizeof(Bad));

Contiguous containers and iteration shape

std::vector and std::array offer contiguous storage that plays well with caches and SIMD. Linked structures scatter nodes and add branch cost. If you need stable iterators and frequent inserts in the middle, consider gap buffers or std::deque; otherwise prefer vectors and batch operations.

Inlining, branch prediction, and caching

Modern CPUs are fast when code is predictable. Inlining removes call overhead and exposes optimization opportunities. Predictable branches keep the pipeline full. Cache friendly access patterns avoid stalls. Balance each lever with binary size and instruction cache pressure.

When inlining helps and when it hurts

Inlining small hot functions can enable constant propagation and vectorization. Over inlining bloats text size and increases i-cache misses. Guide the compiler with inline or attributes sparingly; trust -O2 or -O3 heuristics first and confirm with a profiler.

inline float square(float x){ return x * x; }  // simple and likely to inline

Taming branches and using [[likely]]

Hard to predict branches cost cycles. Restructure code to remove branches inside inner loops or group data so common cases cluster. In C++20 you can annotate expected paths with [[likely]] and [[unlikely]]. Treat these as hints that must match measurements.

if (fast_path_condition()) [[likely]] {
  // hot path
} else {
  // rare path
}

Cache aware traversal and blocking

Iterate in memory order and block work to fit caches. For matrices store rows in row major order and traverse rows in the inner loop. For bigger kernels use tiling so working sets stay inside L1 or L2 caches during the hottest loops.

// Row major multiply sketch
for (size_t i = 0; i < n; ++i)
  for (size_t k = 0; k < n; ++k) {
    float aik = A[i*n + k];
    for (size_t j = 0; j < n; ++j)
      C[i*n + j] += aik * B[k*n + j];
  }

Small object optimization and EBO

The small object optimization stores tiny payloads directly inside a type to avoid heap allocations. The empty base optimization removes storage for empty base classes within derived objects. Both reduce indirection and can shrink working sets.

Understanding small string optimization

Most std::string implementations keep short text inline. Short names and tokens avoid heap traffic and improve locality. Never rely on a specific threshold because it varies across libraries and platforms.

std::string s = "ok";       // likely inline
std::string big(200, 'x');  // likely heap allocation

Designing small buffer friendly types

Custom wrappers can embed a small buffer. Keep the in place capacity modest to stay within register and cache budgets. Provide a growth path to the heap for larger cases and ensure strong exception safety.

template <size_t N>
class small_vec {
  size_t n_ = 0;
  alignas(std::max_align_t) unsigned char buf_[N * sizeof(int)];
  std::unique_ptr<int[]> heap_;
public:
  void push_back(int v){
    if (n_ < N) { reinterpret_cast<int*>(buf_)[n_++] = v; }
    else {
      if (!heap_) heap_.reset(new int[N * 2]);
      heap_.get()[n_++] = v;
    }
  }
};

Use empty base optimization correctly

EBO removes storage for empty bases inside derived objects. Mark tag or policy types as empty and inherit privately. Avoid accidental non empty bases by not adding hidden fields like virtual tables or non trivial members.

struct NoState {};
template <class Policy = NoState>
struct Widget : private Policy {
  int id;
}; // sizeof(Widget<>) equals sizeof(int) on typical platforms

Move semantics and copy elision

Move operations transfer resources instead of duplicating them. Copy elision removes whole moves and copies by constructing directly in place. Write types that are cheap to move and let the compiler elide when possible.

Prefer std::move at transfer points

Use std::move when ownership leaves a scope or when you forward temporaries through factory functions. After a move leave the source in a valid but unspecified state so it can be destroyed safely.

std::vector<int> make(){ std::vector<int> v = {1,2,3}; return v; }
// NRVO likely
std::vector<int> a = make();        // elision or move
std::vector<int> b = std::move(a);  // explicit transfer

Implement efficient move operations

For owning types, move pointers and counts, then null out the source. Declare copy and move members explicitly when managing resources. Consider the rule of five if any custom destructor or copy logic exists.

struct Buffer {
  size_t n{};
  int* p{};
  Buffer(size_t n): n(n), p(new int[n]) {}
  ~Buffer(){ delete[] p; }
  Buffer(const Buffer& o): n(o.n), p(new int[o.n]) { std::copy(o.p, o.p+o.n, p); }
  Buffer& operator=(Buffer o){ std::swap(n, o.n); std::swap(p, o.p); return *this; }
  // copy-and-swap
  Buffer(Buffer&& o) noexcept: n(o.n), p(o.p) { o.n = 0; o.p = nullptr; }
};

Trust guaranteed copy elision

In C++17 returning a prvalue can construct directly in the destination. Do not insert unnecessary std::move on named return variables because it can block elision. Keep factories simple and let the compiler place objects.

Buffer make_buf(){ return Buffer{1024}; }  // constructs in caller when possible

Microbenchmarks and pitfalls

Microbenchmarks help isolate the cost of small building blocks. They can also mislead when compilers optimize away work or when environments vary. Use a harness that disables dead code elimination and reports multiple statistics, then validate findings in end to end profiles.

Building reliable harnesses

Use steady clocks and pin measurements to real work. Prevent the compiler from removing computations by observing results or using escape barriers. Warm up before timing and run enough iterations to reduce noise.

# Example with Google Benchmark
BENCHMARK(MyAlgo)->Arg(1024)->Arg(4096);
BENCHMARK_MAIN();

Guard against dead code elimination

Ensure side effects survive optimization. Summaries printed to std::cout or calls to opaque functions can anchor results. Some frameworks provide DoNotOptimize and ClobberMemory helpers that create barriers.

// Pattern sketch
auto r = compute(…);
benchmark::DoNotOptimize(r);
benchmark::ClobberMemory();

Report distributions and environment

Single numbers hide variability. Publish medians and percentiles, list CPU model and compiler flags, and include dataset shapes. This context makes results reproducible and prevents cargo cult tuning.