Chapter 16: Data and Persistence

Applications become useful once they store state safely and retrieve it quickly. This chapter explores relational databases, NoSQL systems, migrations, schema tools, caching layers, and message queues. Each section highlights patterns that work cleanly with TypeScript so your data access remains predictable and strongly typed.

Relational databases and drivers

Relational engines such as PostgreSQL, MySQL, MariaDB, SQLite, and SQL Server provide structured storage with transactions and powerful query languages. Node offers several drivers and higher level abstractions that make these engines fit naturally into a TypeScript project.

Using pg with TypeScript

The pg client provides pooled connections and parameterised queries. Map rows to typed interfaces so handlers treat database output safely.

// db.ts
import { Pool } from "pg";

const pool = new Pool({
  connectionString: process.env.DATABASE_URL
});

export async function getUser(id: number) {
  const r = await pool.query("select id, email, name from users where id = $1", [id]);
  return r.rows[0] as { id: number; email: string; name: string } | undefined;
}

Type safe query builders

Tools like Kysely and Drizzle infer types from table definitions. They help prevent mistakes by checking column names and result shapes at compile time. These tools are helpful when schemas evolve often.

// drizzle-example.ts
import { drizzle } from "drizzle-orm/node-postgres";
import { users } from "./schema";
import { Pool } from "pg";

const db = drizzle(new Pool({ connectionString: process.env.DATABASE_URL }));

export async function allUsers() {
  return db.select().from(users);
}

NoSQL options and use cases

NoSQL databases store documents, key value pairs, graphs, or other flexible shapes. They work well when schemas change rapidly or when relationships are shallow. However, transactional guarantees and complex queries vary widely across systems.

MongoDB with the official driver

The official driver exposes collections as typed generics. This keeps inserts, updates, and queries consistent with your TypeScript interfaces.

// mongo.ts
import { MongoClient } from "mongodb";

const client = new MongoClient(process.env.MONGO_URL!);
const db = client.db("example");

export interface User {
  email: string;
  name: string;
  created: Date;
}

export async function createUser(u: User) {
  const coll = db.collection<User>("users");
  await coll.insertOne(u);
  return u;
}

Redis for ephemeral state

Redis excels when you need simple data structures that respond quickly. Use it for tokens, rate limits, counters, and caches.

// redis.ts
import { createClient } from "redis";

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

export async function bumpCounter(key: string) {
  return redis.incr(key);
}

Migrations and schema management

Schemas shift over time. Migrations bring structure to those changes so different environments stay aligned. Track migrations in version control and run them reliably during deployment.

Migration tools

Tools such as knex, drizzle-kit, and umzug provide structured migration files. Use sequential files or timestamped names to control ordering.

// knex-migration.js
export function up(knex) {
  return knex.schema.createTable("users", t => {
    t.increments("id");
    t.string("email").notNullable();
    t.string("name").notNullable();
    t.timestamp("created").defaultTo(knex.fn.now());
  });
}

export function down(knex) {
  return knex.schema.dropTable("users");
}

Safe rollouts

Prefer additive changes. Add new columns, migrate data gradually, then remove old columns once clients have switched. Heavy destructive changes slow deployments and risk downtime.

Versioning strategies

Use one of two approaches. You can keep a linear sequence of migration files or use timestamp based files that sort automatically. Both approaches work well as long as you test them in staging before production.

Caching layers and queues

Caching reduces repeated work and improves response times. Queues separate slow tasks from fast paths so users do not wait. Combined, these tools make applications smoother under load.

Layered caching

Combine in memory caches with distributed stores for speed and durability. In memory caches respond instantly while Redis or Memcached share state between processes. Use careful expiration policies so stale data remains rare.

// simple-cache.ts
const local = new Map<string, any>();

export function remember(key: string, ttlMs: number, compute: () => Promise<any>) {
  const item = local.get(key);
  if (item && item.expire > Date.now()) return item.value;
  return compute().then(value => {
    local.set(key, { value, expire: Date.now() + ttlMs });
    return value;
  });
}

Queues for background tasks

Use a queue to offload work that does not need to block an HTTP request. Jobs such as email, image resize, and report generation fit well here. Popular queue systems include RabbitMQ, Redis streams, and cloud managed queues.

// bullmq-example.ts
import { Queue } from "bullmq";
import IORedis from "ioredis";

const connection = new IORedis(process.env.REDIS_URL!);
const jobs = new Queue("emails", { connection });

export async function sendEmailJob(to: string, body: string) {
  await jobs.add("sendEmail", { to, body });
}

Workers and concurrency

Workers pull tasks from a queue and process them independently. Use concurrency settings that match your CPU and IO profile so each worker remains responsive without overwhelming the system.

// worker.ts
import { Worker } from "bullmq";
import IORedis from "ioredis";

const worker = new Worker("emails", async job => {
  const { to, body } = job.data;
  // send email …
}, {
  connection: new IORedis(process.env.REDIS_URL!),
  concurrency: 4
});

By choosing appropriate storage engines, planning schema changes, and placing caches and queues carefully, your application becomes more reliable and more predictable as it grows.