Application-layer validation is the weakest layer in the system. It must never be the wall. The database itself is the wall.

why App code can forget a WHERE tenant_id =. A least-privilege role under forced row-level security cannot. The boundary cannot depend on the next engineer remembering an invariant, so the invariant has to live where they cannot route around it. A forgotten filter falls through to RLS, a missed validation falls through to the read-only role.

what we tried, and what broke The tempting design makes the query validator the boundary: parse the SQL, blocklist the dangerous keywords, ship it. One regex gap is then a breach. A validator that checks DELETE as a substring also false-positives on WHERE name = 'DELETE'.

what we chose Six independent layers, each bypass-resistant on its own, so a successful exfil has to defeat all six.

1. String-aware SQL validation. First keyword must be SELECT or WITH; a dangerous-keyword blocklist matched against a literal-and-comment-stripped copy (so WHERE name='DELETE' is not a false positive); embedded semicolons rejected; a balanced-parens check on the model-authored SQL before it is wrapped and limited. The weakest layer on its own, but it catches known patterns and returns clear errors the model can self-correct against.
2. Read-only transaction. BeginTx(ctx, &sql.TxOptions{ReadOnly: true}). Postgres rejects writes at the engine even if validation missed one.
3. Least-privilege role. CREATE ROLE ... WITH LOGIN NOINHERIT (NOINHERIT blocks privilege creep) plus GRANT SELECT on a fixed allowlist of tables only.
4. FORCE ROW LEVEL SECURITY with a policy reading current_setting('app.tenant_id', true).
5. Transaction-scoped tenant context. SELECT set_config('app.tenant_id', $1, true) (the true is is_local, so the value dies at transaction end). Fail-closed: if it is never set, current_setting returns NULL, tenant_id = NULL matches zero rows. The default is no data, not all data.
6. Parameterized tenant value. Bound via $1, never string-concatenated, and shape-validated upstream.

01 │ // layer 5: tenant context, scoped to THIS transaction only (is_local = true)
02 │ tx.ExecContext(ctx, "SELECT set_config('app.tenant_id', $1, true)", tenantID)
03 │ // the RLS policy reads current_setting('app.tenant_id', true)
04 │ // if it is never set -> NULL -> tenant_id = NULL matches ZERO rows. the default is no data.

01 │ -- the boundary is the database, not the validator
02 │ CREATE ROLE app_ro LOGIN NOINHERIT;             -- NOINHERIT blocks privilege creep
03 │ GRANT SELECT ON events, entities TO app_ro;     -- read-only, a FIXED allowlist of tables only
04 │ 
05 │ ALTER TABLE events ENABLE ROW LEVEL SECURITY;
06 │ ALTER TABLE events FORCE  ROW LEVEL SECURITY;   -- even the table owner cannot bypass
07 │ CREATE POLICY tenant_isolation ON events
08 │   USING (tenant_id = current_setting('app.tenant_id', true));
09 │ 
10 │ -- the model-authored SQL runs inside the read-only transaction, wrapped and limited:
11 │ SELECT * FROM ( /* model-authored SQL */ ) AS _q LIMIT 1000;  -- balanced-parens checked

one surface All three stores - relational, graph, columnar - normalize output to one envelope, { Columns []string, Rows [][]any, Count int }, and import the shared validation primitives (StripLiteralsAndComments, ValidateBalancedParens) from one domain, so the agent learns one shape and validation lives in one place. What differs per store is where the wall sits.

graph: no rls in the engine, so isolation is structural in the query text The graph store (a managed, professional tier) has no RLS and no per-tenant roles. The graph-query validator therefore requires a tenant-label placeholder on every node pattern before the query is allowed to run.

• First keyword in {MATCH, OPTIONAL, WITH, UNWIND, RETURN}; blocks all writes, CALL, UNION, and EXISTS{}/COUNT{}/COLLECT{} subqueries.
• Requires __TENANT__ to appear as a label on every node pattern in every MATCH (covering anonymous (), unlabeled (m), and variable-length path endpoints); blocks NOT n:__TENANT__ negation and any hardcoded Tenant_ literal.
• Only after validation is the placeholder substituted: __TENANT__ becomes Tenant_<id>, then the query runs read-only under a dedicated read-only graph user.

The label builder must produce output byte-identical to the sibling implementation in the other language, so cross-repo determinism is enforced by convention and a strict whitelist regex, ^[A-Za-z_][A-Za-z0-9_]*$, that doubles as injection protection (you cannot parameterize a label).

01 │ // every node pattern must carry the tenant label placeholder, validated BEFORE substitution
02 │ MATCH (a:__TENANT__)-[:REL]->(b:__TENANT__) RETURN a, b
03 │ // reject: writes, CALL, UNION, EXISTS{}/COUNT{}/COLLECT{}, NOT n:__TENANT__, literal Tenant_*
04 │ // only after validation:  __TENANT__  ->  Tenant_<id>   (whitelist ^[A-Za-z_][A-Za-z0-9_]*$)

columnar: when you cannot validate isolation in app, give each tenant its own database user Pushing isolation down to the engine is stronger than any app-layer check, so each tenant query runs as that tenant's own database user (Username: tenantID) with server-side row policies. The Go layer adds only query-shape validation, a verbatim port of the original validator, deliberately frozen ("do not extend this list") and locked.

the whole pattern in one breath Relational: least-privilege role plus FORCE RLS keyed on a SET LOCAL session var, a read-only txn, a fail-closed NULL default, a parameterized tenant. Graph: require a tenant-label placeholder on every pattern, validate before substitution, whitelist-validate the label. Columnar: one database user per tenant with server-side row policies. Everywhere: app-layer validation is for fast, clear errors and self-correction hints. The database is the wall.

why force is the whole game Without FORCE ROW LEVEL SECURITY, table owners bypass RLS by default, and the main app role owns these tables. If ownership ever changes (a restore, a migration), data leaks silently. FORCE is defense in depth.

revoke the escape hatches REVOKE EXECUTE on set_config, current_setting, the whole *_to_xml family (query_to_xml, table_to_xml, and the rest), pg_advisory_lock*, pg_notify, and pg_sleep. Anything that could override tenant context, exfiltrate via an XML round-trip, or impact availability. About 18 dump and string functions in total. Revoking the set_config function is what stops the model-authored SQL from re-setting tenant context inside a CTE; the framework still sets it on the transaction with the statement form before handing over the constrained query.

01 │ REVOKE EXECUTE ON FUNCTION set_config(text, text, boolean)   FROM app_ro;
02 │ REVOKE EXECUTE ON FUNCTION current_setting(text, boolean)    FROM app_ro;
03 │ REVOKE EXECUTE ON FUNCTION pg_advisory_lock(bigint)          FROM app_ro;
04 │ REVOKE EXECUTE ON FUNCTION pg_notify(text, text)             FROM app_ro;
05 │ REVOKE EXECUTE ON FUNCTION pg_sleep(double precision)        FROM app_ro;
06 │ -- plus the whole *_to_xml family (query_to_xml, table_to_xml, ...) and ~18 dump/string functions

in production A red-team pass found two live holes: (1) an attacker could call set_config('app.tenant_id','other-tenant',true) inside a CTE to override tenant context mid-query, and (2) a subquery-breakout via an unbalanced ) escaping the SELECT * FROM (%s) AS _q wrapper. Both were closed. set_config was blocklisted in the string-aware validator and had its execute privilege revoked in the database (belt and suspenders), and balanced-parens validation was added. The lesson that generalizes: an LLM query surface needs a real red team, and every app-layer fix needs a database-layer twin.

why Factual inaccuracies in a prompt directly change agent behavior. A prompt is a deploy artifact, so it gets the same discipline as code: composition, generation from typed sources, and tests.

what we chose Persona prompts are .md files rendered through the Eta template engine, composed from auto-discovered shared partials, with any list-shaped content generated from typed records so it cannot drift.

• Every templates/*.md is loaded into a Map keyed by filename. The template registry is the filesystem, and filenames map 1:1 to an agent-type enum.
• Every templates/sections/_*.md partial loads under a camelCased key; templates pull them with <%~ data.partials.identityLinking %>. A Proxy wraps the partials object so unknown keys render as '', not the literal "undefined": typo-safe composition.
• List-shaped content is generated from a typed record. One typed record is the single source of truth for a catalog of items, and a render function turns it into a markdown table injected via the template. Edit the record, the prompt regenerates. Hardcoding an integration count guarantees it goes stale; derive catalog sizes from code.

01 │ // the template registry IS the filesystem; filenames map 1:1 to an agent-type enum
02 │ const templates = new Map<AgentType, string>(/* load templates/*.md */);
03 │ 
04 │ // typo-safe partial composition: unknown keys render '' , never "undefined"
05 │ const partials = new Proxy(loaded, {
06 │   get: (t, k) => (k in t ? (t as Record<string, string>)[k as string] : ''),
07 │ });
08 │ 
09 │ // list-shaped content is GENERATED from a typed record, so it cannot drift
10 │ const catalogMarkdown = renderCatalogMarkdown(CATALOG_RECORDS);

one evidence vocabulary everywhere Cross-cutting methodology has to be identical everywhere or it erodes persona by persona, so it ships as shared partials too. Every claim must cite a tool result, and findings are tagged Confirmed (multiple data points), Observed (a single data point), or Inconclusive (the gap is named, not papered over). Output hygiene is one of those partials: say "connection type", not the raw column name; never leak internal schema names to the user.

tested A spec asserts each partial loads and interpolates, and that the untrusted-web-results guardrail is present in every persona that can reach web search.

the position Dedup at 0.75: only collapse facts that are really the same, so you do not merge two distinct preferences into mush. Recall at 0.6: return anything plausibly relevant and let the agent filter, because a missed-but-relevant memory is a worse failure than a surfaced-but-irrelevant one (the model just ignores the latter). A symmetric threshold would force a choice between over-merging and under-recalling; the asymmetry sidesteps it and gets both right.

the store You already have Postgres; do not add Pinecone or Weaviate until you need to. pgvector wins on operational simplicity, transactional consistency (memory extraction commits in the same database as the conversations), and cost, and with a 500-memory-per-user cap (LRU-evicted by updatedAt), sub-100ms HNSW recall to millions of vectors is comfortably in reach - a dedicated vector store becomes inevitable only at billions of vectors across thousands of tenants. Prisma cannot express the <=> operator, so all three vector queries are raw SQL with an explicit $n::vector cast, and a WHERE tenant_id = $ AND user_id = $ scopes every read and write - memory is per-user, not just per-tenant. The HNSW index runs at stock parameters (m = 16, ef_construction = 64, vector_cosine_ops): the defaults are fine at this scale, and IVFFlat's cluster tuning and rebuild-when-the-distribution-shifts maintenance are not worth it here. Embeddings are Titan v2 at 1024 dimensions with normalize: true, because unit vectors are what make cosine distance behave.

the write pipeline Memory must not be "embed everything and hope" - embedding raw conversation turns bloats the store with ephemeral chatter, so an extract step distills durable facts and keeps recall signal-dense. A cheap model at temperature 0 pulls durable facts in five categories (preference, entity, workflow, pattern, context) with an explicit do-NOT-extract list (ephemeral mechanics, profile-obvious facts, volatile event data, greetings), Zod-validated; each fact is embedded, cosine-searched against the user's existing memories, and decided as exactly one of NEW, UPDATE, DELETE, or DUPLICATE, where DELETE handles direct contradiction (a stated switch from one tool to another obsoletes the fact that relied on the old one).

01 │ // fire-and-forget AFTER the agent loop; the next turn never waits on it
02 │ extractAndStoreMemories(conversation).catch(log);
03 │ 
04 │ const facts = await extract(conversation, {      // Haiku-class
05 │   temperature: 0.0, maxTokens: 2048,             // five categories, explicit do-NOT-extract list
06 │ });                                               // Zod-validated
07 │ 
08 │ for (const fact of facts) {
09 │   const v = await embed(fact);
10 │   const candidates = await cosineSearch(v, { threshold: 0.75, limit: 3 }); // DEDUP_* (0.75, top 3)
11 │   const decision = await decide(fact, candidates); // NEW | UPDATE | DELETE | DUPLICATE, temp 0
12 │   // NEW    -> insert (after MAX_MEMORIES_PER_USER = 500 cap, LRU by updatedAt)
13 │   // UPDATE -> re-embed merged content + update
14 │   // DELETE -> remove ;  DUPLICATE -> no-op
15 │ }

what we rejected Using the main model for extraction pays Opus rates for a deterministic structured task Haiku does at temp 0 for about one fifth the price.

recall Embed the query, then one raw cosine search, scoped to the tenant and user.

01 │ SELECT content, 1 - (embedding <=> $1::vector) AS similarity
02 │ FROM memories
03 │ WHERE tenant_id = $2 AND user_id = $3
04 │   AND 1 - (embedding <=> $1::vector) >= 0.6      -- RECALL_SIMILARITY_THRESHOLD
05 │ ORDER BY embedding <=> $1::vector LIMIT 10;       -- RECALL_LIMIT

A fact learned in one feature should still surface in another. The feature column is provenance, not a filter; user preferences and org context are universal.

degradation recall_memories is silent = true. Called at conversation start (in a Promise.all with context-building), it suppresses the TOOL_CALL and TOOL_RESULT stream events but still persists them as INTERNAL for observability: memory recall is system plumbing, and animating "Recalling memories..." in the UI is noise. Extract failure returns an empty array; dedup failure defaults to NEW (pessimistic, because a possible duplicate beats a lost fact); recall failure returns an empty array. Memory is an enhancement, so no memory failure can break a turn.

known limits, written down There is no TTL (memories live until LRU eviction at the cap), no reranking and no chunking (at one fact per memory under the cap, raw cosine ordering is enough), and the dedup-failure default to NEW means a flaky cheap-model call can leave a near-duplicate behind. None of these bite at current scale, and all were written down rather than discovered.

the opinion Everything structured and deterministic (extraction, dedup, titles) goes to the cheap deterministic model; the expensive model is reserved for the one turn that actually needs it. Tier the models explicitly, make the routing environment-driven, and keep the choice where a human owns it.

what we tried, and what broke Dynamic model selection ("let the system choose the right tier") was tried and abandoned. The routing decision is itself a cost and a source of nondeterminism, and it belongs in config, not in the model's lap.

what we chose Three explicit, environment-driven tiers: Opus for the reasoning loop, Sonnet for summarization (fired every roughly 8 messages or 15K tokens), and Haiku for titles, memory extraction, and dedup decisions, all at temperature 0. Every model id is prefixed us. at runtime for cross-region inference, which dodges single-region throttling.

the numbers Memory extraction and dedup are two Haiku-class calls at temperature 0, a negligible cost (about $0.00004 for a 10-fact conversation) against the Opus main-loop turn. The split is the difference between an agent that is cheap to run all day and one that is not.

why A tool-use loop re-sends the system prompt, the original question, and the tool definitions on every iteration. Those three prefixes are identical turn to turn, so they are exactly what a prefix cache is for.

what we chose Ephemeral caching ({ type: 'ephemeral', ttl: '1h' }) at three boundaries: the system prompt (stable per agent type, 12K to 22K tokens), the first user message (the question, preserved across every iteration), and the last tool definition (the provider caches the whole tool array only if the marker sits on the final element). Put the marker on the wrong tool and you cache nothing.

the number Roughly 60% input-token savings on multi-turn conversations, for free.

what we chose Against a CONTEXT_WINDOW_BUDGET_TOKENS = 100_000 budget (deliberately about 30% under the real 200K limit, because the estimator undercounts), compactMessages() returns as soon as it is under budget, through five escalating phases.

1. Truncate tool blocks in middle messages.
2. Drop those middle messages entirely.
3. Compact the protected tail oldest-first (always preserving MIN_RECENT_PAIRS = 4 plus the last 2 messages).
4. Truncate text blocks longer than 200 chars.
5. Proportionally truncate the largest remaining tool-result blocks via truncateJsonSafely.

A pre-flight ceiling of 1.3x budget aborts the turn with stopReason: 'context_limit' rather than sending an oversized prompt. Background summarization (the cheap model) fires at 8 or more messages or 15K or more tokens and re-summarizes at escalating thresholds; the summary is injected as previousContext.

01 │ const CONTEXT_WINDOW_BUDGET_TOKENS = 100_000; // ~30% under the real 200K; the estimator undercounts
02 │ const MIN_RECENT_PAIRS = 4;
03 │ 
04 │ // compactMessages returns as soon as it is under budget, escalating phase by phase:
05 │ // 1) truncate tool blocks in middle messages
06 │ // 2) drop middle messages entirely
07 │ // 3) compact the tail oldest-first (keep MIN_RECENT_PAIRS = 4 + last 2 messages)
08 │ // 4) truncate text blocks > 200 chars
09 │ // 5) proportionally truncate the largest tool-result blocks (truncateJsonSafely)
10 │ 
11 │ // pre-flight: 1.3x budget aborts with stopReason 'context_limit'
12 │ // token estimate is a HEURISTIC: 2.8 chars/token (JSON), 3.5 (text). reconcile vs inputTokens later

the honest caveat to reproduce with eyes open Token accounting is a heuristic (2.8 chars per token for JSON, 3.5 for text), not a real tokenizer. The shipped tokenizer is an older model generation that runs roughly 30% off on current-generation models and undercounts JSON specifically, so the whole safety net rests on the estimate and the margin is the only thing keeping you off the provider's hard limit. The budget is set deliberately low against that: 100K estimated is about 130K real, plus roughly 20K system, 5K tools, and 8K output, about 163K, leaving roughly 37K of margin under the real 200K window. If you can afford it, reconcile against the provider's reported inputTokens after the fact, and carry a flag on the estimate: do not use it for billing, only for budgeting.

why LLM output is not user input. It is untrusted machine-generated data. Streaming delivers partial and malformed JSON constantly, and a confused or adversarial model will happily emit a tool call that, taken literally, misbehaves.

what we chose JSON.parse(raw), on failure JSON.parse(jsonrepair(raw)), only then fall back to {}. This salvages tool calls Bedrock would otherwise drop.

01 │ // repair before discarding: salvages tool calls the provider would otherwise drop
02 │ function parseToolArgs(raw: string): object {
03 │   try { return JSON.parse(raw); }
04 │   catch {
05 │     try { return JSON.parse(jsonrepair(raw)); }
06 │     catch { return {}; }
07 │   }
08 │ }

why agent.run() is a black box, and the five things you most need in production - tenancy, cancellation, token budgeting, context compaction, and streaming delivery - are exactly the five it does not expose. Those are the levers an in-product agent lives or dies by. The loop is about 700 lines you should own; the agent SDK can sit in package.json as a mock target and still have zero production imports.

what we tried, and what broke Wiring a callback into a framework's run() is the fastest path to a demo, and it outsources cross-instance cancellation, per-tenant budgeting, compaction, and the interactive-versus-headless seam to code you cannot reach into. The first time you need to cancel a run on a different pod, or compact a context mid-investigation, you are blocked.

what we chose A hand-rolled loop:

01 │ let iteration = 0;
02 │ let continueLoop = true;
03 │ while (continueLoop && iteration < MAX_ITERATIONS) {
04 │   checkCancellation();          // local AND distributed
05 │   checkWallClockTimeout();      // LOOP_TIMEOUT_MS
06 │   compactMessages();            // keep post-compaction context under budget
07 │   const turn = await provider.queryStream(messages, tools); // exactly one model turn
08 │   const toolUses = extractToolUseBlocks(turn);
09 │   if (toolUses.length === 0) { continueLoop = false; break; } // model stopped calling tools
10 │   const results = await runBounded(toolUses, PARALLEL_TOOL_LIMIT); // fastq
11 │   messages.push({ role: 'tool', content: results });
12 │   iteration++;
13 │ }

• The loop emits through a sink interface. An SSE sink drives the interactive path (multiplexed per user); a collecting sink gathers events into an array for headless runs (the scheduled triage agent that runs inside a durable workflow). One loop, two sinks: interactive and headless share identical execution and persistence - a bug fixed in one is fixed in both - and the loop stays unit-testable without SSE scaffolding.

why Credentials, connection pooling, abort composition, JSON repair, and version quirks are all provider concerns. Spread them through the codebase and every quirk becomes a hunt across files; implement them once, in one injectable wrapper, and they never leak into calling code.

what we chose A single provider that exposes three verbs - query() (non-streaming), queryStream() (streaming to a normalized event union), and embed() - and owns:

• Credentials resolved once in onModuleInit (a provider chain on the server, SSO or env locally), never per request, no API keys in the environment.
• A pooled undici.Agent (keepAliveTimeout: 30s, connections: 10), because streamed responses hold a socket for the full turn, so too few connections bottleneck a multi-turn run.
• Composed abort: AbortSignal.any([shutdownSignal, requestSignal]), so either a user cancel or a server shutdown tears the request down.
• A version-aware temperature strip using range logic, not a hardcoded denylist, because the provider began 400-ing on temperature for the 4.7-and-newer model family and the deprecation carries forward to every newer release. A denylist would need editing on every model bump, and the edit gets forgotten until it 400s mid-investigation; the range comparison fixes it once and never again.
• JSON repair before discard: parse the tool-call input, on failure repair-then-parse, and only as a last resort fall back to {}. The same normalizer strips role:'tool' messages the SDK would re-emit, because duplicate tool_result blocks break the provider's message contract.

01 │ // Range logic, not a denylist.
02 │ const modelSupportsTemperature = (major, minor) =>
03 │   major < 4 || (major === 4 && minor < 7);
04 │ 
05 │ // JSON repair before discard: a no-op tool call beats dropping the whole turn.
06 │ function parseToolInput(raw) {
07 │   try { return JSON.parse(raw); }
08 │   catch {
09 │     try { return JSON.parse(jsonRepair(raw)); }
10 │     catch { return {}; }
11 │   }
12 │ }

why A model occasionally loops forever, a socket occasionally hangs, a fan-out occasionally exhausts a connection pool, and an estimate occasionally undercounts. Each is a different failure mode, so each needs its own ceiling.

what we chose Five hard caps, plus a soft wind-down, plus a typed exit reason. None of the constants were designed up front; each is a production failure that became a constant with a comment:

On top of the hard caps, a soft wind-down: three iterations before the cap (or at 80% of an optional per-run tokenBudget), inject a [SYSTEM: approaching your processing limit. Please summarize your findings...] hint into the last user message, giving the model two turns to land gracefully before the hard stop. Every exit maps to a typed stopReason (end_turn, max_iterations, timeout, context_limit, budget_exceeded, cancelled, error), each with a user-facing notification, so the user always learns why a run ended.

why Streaming delivers partial and malformed JSON constantly, and a confused or adversarial model will happily emit a tool call that, taken literally, reaches into the wrong tenant. The contract has to make that structurally impossible - even in the presence of application bugs - not merely discouraged, so a tool cannot forge a tenant it cannot name.

what we tried, and what broke Validating tool input with the same decorator-based validator used for HTTP DTOs instantiates decorated classes on untrusted output, which carries side-effect risk. Passing the tenant as an optional tool parameter that defaults to context feels convenient, until the one branch that forgets the default is a cross-tenant leak.

what we chose An abstract base with a typed name (an enum, not a string, so a typo is a compile error), a description, an inputSchema validated with a structural validator (Zod) that does not instantiate classes, and execute(ctx, args). Optional overrides carry real engineering: maxResultTokens (default 50,000; a timeline tool drops to 16,000 to prevent parallel calls from exceeding the context budget; a columnar tool runs 40,000), timeoutInMs (heavy analytics overrides to 55,000), silent, and errorLogLevel.

01 │ abstract class AgentToolBase<TInput, TOutput> {
02 │   abstract name: AiAgentToolName;          // an enum, not a string: a typo is a compile error
03 │   abstract description: string;
04 │   abstract inputSchema: ZodSchema<TInput>; // structural validation, no class instantiation on untrusted output
05 │   maxResultTokens = 50_000;                // a timeline tool overrides to 16_000; a columnar tool to 40_000
06 │   timeoutInMs?: number;                    // heavy analytics overrides to 55_000
07 │   silent = false;
08 │ 
09 │   // tenantId is NEVER here. It rides in ctx from the authenticated session and is banned from every schema.
10 │   abstract execute(ctx: AgentToolContext, args: TInput): Promise<ToolResult<TOutput>>;
11 │ }

the rule Model output is not user input. It is untrusted machine-generated data, and the tool contract treats it that way on every single call.

why Auto-discovery turns a forgotten decorator into a silently unregistered tool; one explicit array, DI-registered and indexed by name, forces a code-review gate on every new tool. And a tool can fail in three different ways - an expected failure, an unexpected throw, a timeout - each of which wants a different response.

what we chose A single entry point sanitizes the tool name (the provider sometimes emits trailing quotes; drop the regex and tools randomly fail "not found"), validates with the schema, then runs under a Promise.race timeout.

01 │ async function executeForTenant(rawName, args, ctx) {
02 │   const tool = registry.get(stripTrailingQuotes(rawName));
03 │   const parsed = tool.inputSchema.parse(args);
04 │   return Promise.race([tool.safeExecute(ctx, parsed), timeout(TOOL_TIMEOUT_MS)]);
05 │ }

Errors funnel through three layers:

• L1: execute() returns { success: false, error } for expected failures, no throws on the happy path.
• L2: a safe-execute wrapper catches unexpected throws and sanitizes them, passing query-shaped hints through (column ... does not exist, syntax error, table ... not found, no such function, missing required, invalid date, unknown field) so the model can self-correct, and replacing anything infra-shaped (connection strings, ARNs, IPs, stack traces) with "Try a simpler query or different parameters." The model can fix a wrong column name; it cannot fix, and should never see, a database hostname.
• L3: the registry catches timeouts and anything else and returns a graceful string.

silent tools ride the same funnel A silent tool (for example a memory-recall tool) executes normally but skips streaming events while still persisting to the database, object storage, and the analytics sink, so it is invisible to the chat and fully visible to observability. The gotcha: its output still counts against the context budget, so silent tools should run a low maxResultTokens. Invisible to the user is not invisible to the token count.

why Many tools return rows with email-shaped fields, and every one of them would otherwise have to re-implement "turn this email into a clickable record link." And a large result must be capped where it is produced, in a deliberate order, because truncation cuts JSON at a byte offset: drop first, truncate second - the reverse slices a half-removed enrichment block into invalid JSON mid-object.

what we chose Enrichment is centralized once, in the orchestrator. After a tool succeeds and before token-truncation, an enricher recursively collects email-shaped strings with an anchored regex, resolves them against the data plane in 50-email chunks with in-flight coalescing (three parallel tool results sharing emails cause one lookup, not three), and attaches a sibling <key>Identity block next to each email field. A tool author returns raw data and gets links for free. Then the bounding runs in three stages: the per-tool maxResultTokens is the budget; dropEnrichmentSiblings() strips the ...Identity blocks first; truncateJsonSafely() cuts at a }, or } boundary, closes any open brackets or braces, and appends [TRUNCATED: ...] so the model knows data was dropped. Crucially, the full enriched payload is structuredCloned to durable object storage before the drop, so the data-viewer UI keeps the rich version while the model sees the bounded one.

01 │ let result = capToTokens(raw, tool.maxResultTokens); // per-tool ceiling, at the source
02 │ structuredCloneToS3(raw);                            // keep the full payload BEFORE dropping
03 │ result = dropEnrichmentSiblings(result);             // strip whole blocks, never mid-block
04 │ result = truncateJsonSafely(result);                 // closes brackets/braces, appends [TRUNCATED: ...]

tradeoff (disclosed) The clone is an honest perf cost on thousand-row results, flagged for a stringify-snapshot swap if profiling ever demands it.

why App code can forget a WHERE tenant_id =; a least-privilege role under forced row-level security cannot (the six-layer wall itself is argued in the AI / LLM section). So the validator's job is not to be the wall - it is to hand the model an actionable hint faster than a round trip to the database would, without lying to it.

what we chose Context-aware validation: strip string literals before keyword-checking so WHERE name = 'DELETE' is not a false positive; anchor patterns (^\s*SYSTEM\b blocks the SYSTEM command while allowing FROM system.tables); use word boundaries (\bINSERT\b does not trip on an identifier); and emit specific messages ("Do not include LIMIT or OFFSET; use the limit and cursor parameters") the model can act on. The graph tool additionally requires a __TENANT__ placeholder, validation that doubles as the tenant-scoping contract because you cannot parameterize a graph label.

01 │ const noLiterals = sql.replace(STRING_LITERAL_RE, "''"); // strip literals: WHERE name='DELETE' is not a hit
02 │ if (/^\s*SYSTEM\b/i.test(noLiterals)) reject('...');     // anchored: blocks the command, allows FROM system.tables
03 │ if (/\bINSERT\b/i.test(noLiterals)) reject('...');       // word boundary: does not trip on an identifier
04 │ if (/\b(LIMIT|OFFSET)\b/i.test(noLiterals))
05 │   reject('Do not include LIMIT or OFFSET; use the limit and cursor parameters.'); // specific, actionable

the rule Every app-layer check needs a database-layer twin doing the real enforcing.

why Reaching for the durable-workflow engine "because it is our workflow engine" wastes its durability on jobs that do not need it and starves your real durable work of the discipline it does need. A short stateless transform, a scheduled fan-out, and an in-product workflow whose state must survive a worker restart are three different problems with three different answers.

what we chose Match the tool to the failure model:

the rule A SELECT-only job that fans out over tenants does not need replay determinism; it needs a sensor. Reserve the durable engine for when losing in-flight state to a restart is the failure you are actually afraid of.

why A durable workflow replays its history to reconstruct state. Anything nondeterministic in the body (a real random value, a wall-clock read, an I/O call) makes replay diverge, and you get the bug that only shows up after a deploy, in production, on the workflow you cannot reproduce.

what we chose Rules the code obeys, and why each one:

• Math.random() lives in the API process, never the workflow; Date.now() and new Date() are fine in the body, because the engine feeds them logical time, deterministic across replay.
• Activity services are imported as types only (import type); the type erases at compile, so no decorator runtime leaks into the sandbox.
• Cheap, CPU-only pure primitives run inside the sandbox: a run workflow builds a topological DAG of steps and runs effectful steps as activities but pure steps in-process, because dispatching every trivial computation as an activity pays per-step overhead for nothing.
• A post-layer barrier is deterministic by construction: an aggregate row-cap fires in artifact-step order, not inside a Promise.all closure, because Promise.all resolution order is wall-clock dependent and would make the cap non-replayable.

the worker runtime enforces the same line Each durable domain ships two DI modules, so the API process and the worker process compile as independent graphs; neither is global, and the worker explicitly imports every infra module its activities touch. The workflow-bundle build script doubles as the sandbox-compat gate: if a workflow accidentally imports a value instead of a type, the bundle build fails before deploy, not at replay time in production. Each workload gets its own task queue, so a slow background agent's concurrency cannot starve fast foreground runs, and in-flight runs are tracked in a pending set and drained on application shutdown, so a deploy does not sever live work.

why One mega-config that fits a 2-second write and a 15-minute run fits neither. And retrying a deterministic failure is latency you are charging the user for.

what we chose A policy per concern, each with its own non-retryable taxonomy:

01 │ // Query activities: the expensive, retryable path.
02 │ const queryActivities = proxyActivities({
03 │   startToCloseTimeout: '45s',
04 │   heartbeatTimeout: '15s',
05 │   retry: {
06 │     maximumAttempts: 3,
07 │     initialInterval: '2s',
08 │     backoffCoefficient: 2,
09 │     maximumInterval: '30s',
10 │     nonRetryableErrorTypes: ['INVALID_SQL', 'INVALID_CYPHER', 'ACCESS_DENIED', 'QUERY_EXECUTION_FAILED'],
11 │   },
12 │ });

The taxonomy is the craft, and the test is whether a retry could plausibly succeed. QUERY_EXECUTION_FAILED means the engine itself errored on our query (a syntax or coercion error), which is deterministic, so retrying just burns 6 or more seconds before surfacing the same error. But transport failures (statusCode 0, 502), a query timeout, and an unavailable-credentials error stay retryable so a brief data-plane blip self-heals. The persist activity runs its own shorter policy and is idempotent at the schema layer (a unique constraint on the workflow-id plus run-id pair, with upsert), so a redelivery cannot double-write - retry classification layered on top of, not in place of, schema-level idempotency.

then compute your real attempt count This is the most-missed durable-workflow subtlety, and it is real money: your effective maximum attempts is the minimum of what you configured and what fits under the workflow or route ceiling. A synchronous path the user awaits over HTTP caps the whole workflow at workflowExecutionTimeout: 60s, but the query activity's retry policy has a worst-case wall-clock budget of:

01 │ 45s (attempt 1) + 2s backoff + 45s (attempt 2) + 4s backoff + 45s (attempt 3) = 141s
02 │ // workflowExecutionTimeout: 60s  =>  only attempts 1 and 2 can fit; attempt 3 is unreachable.

So on the blocking path, the third configured attempt can never run; the same activity dispatched fire-and-forget (no execution timeout) gets the full 141-second budget. The asymmetry was written into a comment after someone reasoned it out, cheaper to document than to debug "the retry that only fires sometimes." Make the blocking-versus-durable distinction explicit at the call site, and compute the count - do not assume it.

why The data-plane HTTP layer deliberately carries no workflow concerns, so a single wrapper adds the three that matter. A long-but-alive query must keep beating or it gets declared dead and retried; a retry must present the same idempotency key or a future write primitive double-executes; and a cancelled workflow must tear down its in-flight HTTP request or it keeps computing against a dead run.

what we chose A wrapper that heartbeats every 5 seconds (well under the declared 15-second heartbeat timeout), builds an idempotency key of <workflowId>-<activityId>, and forwards the workflow's cancelled promise into an AbortController.abort() so the underlying request tears down. Cancellation is rethrown, not absorbed, so a cancelled run ends CANCELED, not a misleading COMPLETED.

01 │ async function withWorkerSemantics(fn) {
02 │   const ctx = Context.current();
03 │   const beat = setInterval(() => ctx.heartbeat(), 5_000); // well under the 15s heartbeatTimeout
04 │   const idempotencyKey = `${ctx.info.workflowExecution.workflowId}-${ctx.info.activityId}`;
05 │   //  ^ attempt is deliberately EXCLUDED: a retry must present the SAME key so a future write
06 │   //    primitive dedupes it as one logical call, not a second side effect.
07 │   const abort = new AbortController();
08 │   ctx.cancelled.catch(() => abort.abort()); // workflow cancel tears down the request
09 │   try {
10 │     return await fn({ idempotencyKey, signal: abort.signal });
11 │   } finally {
12 │     clearInterval(beat);
13 │   }
14 │ }

the rule A stable key is only useful if the request body is also stable across retries, so activity authors must not embed timestamps or per-attempt nonces in the substituted query; today's read-only queries ignore the key, but the contract is set for when write primitives land.

why For a per-entity object that lives for weeks, the workflow is the object's state machine: it parks at zero cost on a condition(), advances only on a real signal, and resets its history with continueAsNew so it never accretes unbounded events. A polling timer wastes history events and money; a buffered event queue re-derives state you were going to re-read anyway.

what we chose

01 │ setHandler(getStatusQuery, () => status);                  // the UI polls this
02 │ setHandler(reconciledSignal, (outcome) => { /* set triagePending or closed */ });
03 │ 
04 │ for (;;) {
05 │   await condition(() => closed || triagePending || signalsThisRun >= maxSignals);
06 │   if (closed) return status;
07 │   if (triagePending) {
08 │     triagePending = false;
09 │     status = 'PROCESSING';
10 │     await runHeadlessAgent(...); // best-effort; a failure is caught and logged
11 │     continue;
12 │   }
13 │   return await continueAsNew({ ...stateForward }); // reset history, carry status forward
14 │ }

• A single pending flag, not a queue: processing always re-reads the full current roster, so coalescing several joins into one run is correct. Do not buffer events you are going to re-derive.
• The continue-as-new budget is clamped to a positive finite value, because a caller passing 0 or NaN would otherwise continue-as-new in a tight loop.
• A failed agent run must not kill the lifecycle: the expensive sub-task is best-effort, caught and logged, and the agent activity is maximumAttempts: 1 on purpose, because a retry would re-spend model tokens; recovery is a re-run on the next material change, not a mechanical retry.

the rule When the unit of work is a model run that costs money and is not deterministic, the retry policy is a budget decision, not a reliability default. "Retry it harder" is exactly wrong. Long-running does not mean retry-harder; it means make the state durable and the dispatch idempotent so you rarely need to retry at all.

why At-least-once delivery is everywhere. If dispatch is not idempotent, redelivery double-opens entities and double-sends notifications; built to collapse, a duplicate becomes a no-op.

what we chose

• signalWithStart keyed on the entity id. Opening a lifecycle from a run is one idempotent call: if the workflow exists the signal is delivered, if not it is started and then signaled. The call goes to the engine over the network - nondeterministic in the sandbox - so it lives in an activity, never the workflow body, and putting it in the body is a common mistake. The dispatch is best-effort; an exhausted or non-retryable failure is caught and logged rather than failing the run.
• workflowIdReusePolicy: 'REJECT_DUPLICATE' for outbox-driven sends. A deterministic workflow id starts with reject-duplicate; a redelivery of the same outbox row throws already-started, which is caught as the dedupe working: report success, do not settle the existing row to FAILED.
• Workflow IDs encode scope and are fixed-length. A short id is 8 hex characters from crypto.randomBytes(4) (32 bits, fixed length), deliberately not Math.random().toString(36) which has a variable-length tail. Embedding the channel guarantees a multi-channel send never collides on one id; embedding the tenant makes the workflow UI searchable and scoped.

01 │ // signalWithStart lives in an activity, never the workflow body.
02 │ await temporal.signalWithStart(lifecycleWorkflow, {
03 │   workflowId: `record-${tenantId}-${recordId}`, // keyed on the entity: open-new and update-existing collapse
04 │   signal: reconciledSignal,
05 │ });
06 │ 
07 │ // Outbox-driven sends: a DETERMINISTIC id + REJECT_DUPLICATE turns a redelivery into a caught no-op.
08 │ try {
09 │   await temporal.start(dispatchWorkflow, {
10 │     workflowId: `evt-${outboxId}-${targetId}`, // deterministic from the outbox row
11 │     workflowIdReusePolicy: 'REJECT_DUPLICATE',
12 │   });
13 │ } catch (err) {
14 │   if (isAlreadyStarted(err)) return ok(); // the dedupe working; do NOT settle the row to FAILED
15 │   throw err;
16 │ }

why An in-process cron decorator runs once per replica: three API pods fire your "every 30s" job three times, or you pin it to one pod and it dies silently when that pod does - a ghost either way. Durable Schedules are HA by construction.

what we chose Three idempotent shapes:

1. Singleton boot Schedule. On bootstrap, create a Schedule and swallow already-running as the steady state, not an error. A skip-overlap policy means a slow sweep never stacks on itself; a tight catchup window means worker downtime does not backfill a burst of missed fires.
2. Per-entity schedule with compensation. Claim the row under a database advisory lock (released before the Schedule call), create the Schedule outside the database transaction, and compensate with an UPDATE on failure, because Schedules are not transactions and you cannot wrap a Schedule call in an interactive database transaction.
3. Transactional-outbox hot-trigger plus sweep backstop. The producer commits an outbox row and best-effort starts the dispatch workflow with the deterministic-id, reject-duplicate pattern from the previous entry; a 30-second sweep Schedule re-claims anything the hot trigger missed.

01 │ // Singleton boot Schedule: HA by construction, no single-replica ghost cron.
02 │ async onApplicationBootstrap() {
03 │   try {
04 │     await schedule.create({
05 │       spec: { intervals: [{ every: '30s' }] },  // sub-minute: Schedules, not cron (cron floors at 1m)
06 │       policies: { overlap: ScheduleOverlapPolicy.SKIP, catchupWindow: '1m' },
07 │     });
08 │   } catch (err) {
09 │     if (isAlreadyRunning(err)) return;          // the steady state, not an error
10 │     this.logger.error('schedule create failed (non-fatal)', err); // loud but never wedges the worker
11 │   }
12 │ }

tradeoff (disclosed) The boot-create failure is logged loud but non-fatal: without the sweep the outbox stops draining, so it must be alertable, yet a transient blip at boot should not wedge the whole worker.

why Version and patch guards are the textbook answer, and they metastasize into patch-guard sprawl whose debt compounds forever. Drain-short and terminate-long leave no guard debt, and the approach is simpler precisely when your workflows are either short (drainable) or terminable. If you had truly un-terminable multi-month workflows you would need the version guard; most do not.

what we chose

• Short-lived workflows (runs): drain the run queue before deploy.
• Long-lived lifecycles: terminate at deploy and let the next reconcile re-open them via signal-with-start; if the feature never shipped, there is no pre-patch history to keep.
• An in-body schema-version gate instead of code-version guards: the workflow checks a supported major version on its input and fails non-retryably on an unsupported major, plus a re-validate that is also non-retryable on a parse error. The version check is on the data, not the code path.

Pick exactly one system of record. Every other store is a projection you can throw away and rebuild. Postgres is the truth; the graph and the columnar store are projections.

why The whole architecture turns on one asymmetry: if the graph is lost, it can be fully rebuilt by replaying the outbox; if Postgres is lost, the graph cannot reconstruct it, because it holds a projection, not a log. That single sentence decides every store-of-truth question downstream, and it generalizes: the columnar store and any future store all inherit "derived." When truth is singular, the architecture cannot sprout new authorities over time.

what we tried, and what broke Graph-as-truth fails the recovery test outright (you cannot rebuild a relational log from a projection). Dual-write needs two-phase commit or a saga to stay consistent. An event bus adds infrastructure and eventual consistency. A nightly snapshot trades away freshness.

what we chose A transactional outbox. Every source write commits an outbox row in the same Postgres transaction as the data write, so write-or-do-not-write atomicity is guaranteed by Postgres itself, with no second durability model. A handler then replays the outbox into the projection idempotently. Drift is unidirectional, truth to projection, and detected drift is surfaced as findings, never reconciled sideways. Reach for a real queue only when the producer and consumer genuinely do not share a transaction. Writing the rejected alternative down is the reason the event bus we would otherwise have had to operate and then rip out never got built.

Run three engines, each answering a class of question the others structurally cannot, and join them in the app. Do not force one engine to do all three jobs.

why The value is at the intersection of events, state, and relationships. An event store has the timeline but no relationship graph; a state store has the current config but no record of how it got there. A product that holds all three answers questions none of the single-plane tools can.

what we chose

Replay into the projection with MERGE, never CREATE. Anything in the projection not in truth is drift; anything in truth not in the projection is drift; the fix is always to rebuild from truth.

why Auto-reconciliation is how two wrong stores agree. The moment a projection is allowed to edit truth, a corrupted projection can overwrite the source of record. Surfacing drift instead of fixing it keeps truth singular and makes every divergence visible.

what we chose Every temporal write replays as a MERGE (idempotent, re-runnable), never a CREATE (which would duplicate on replay). A data-quality auditor runs four levels of truth-to-projection reconciliation, count, then ID-set, then field, then relationship, and the only remediation is archive-and-rebuild the projection from Postgres-as-truth. Convenient auto-reconciliation between two stores was rejected precisely because it lets a bad projection poison the record.

honest gap we still carry The auditor checks store-to-store agreement but not whether a given diff is plausible: a partial snapshot (a paginated failure, a rate-limited 200 with an empty body) that mass-closes records still looks valid to it.

Set the canonical id once, on create, and never again. Whichever provider's sync happens to run first must not be able to flip the canonical identity.

why If the canonical id can be rewritten on a later match, the identity flip-flops based on sync order. Writing it only on create, and touching only updated_at on match, makes resolution deterministic regardless of which provider arrives first.

what we chose The dedup key is toLower(trim(email)). A MERGE sets the canonical id on create and nothing structural on match. Change detection is a SHA-256 over the sorted identity payload, not a field-by-field diff.

01 │ MERGE (i:Identity {email: toLower(trim($email))})
02 │   ON CREATE SET i.canonical_id = randomUUID(), i.created_at = timestamp()
03 │   ON MATCH  SET i.updated_at  = timestamp();   // never re-set canonical_id on match
04 │ // change detection is a SHA-256 over the sorted identity payload, so field reordering is not a change

Increment the attempt counter in its own committed transaction before the replay, never in the except block after. A crash must not be able to pretend the attempt never happened.

why If the attempt counter is bumped only on the failure path, a SIGKILL mid-replay leaves the row looking untried, so a poison row is re-picked forever. Claiming the work before doing it bounds the blast radius of a crash: a poison row fails after a fixed number of tries instead of being re-picked on every cycle.

what we chose Bound attempts with max_attempts = 3; sweep exhausted rows to a failed state with a single page; recovery is a manual rebuild, never an auto-requeue. Auto-requeue was rejected explicitly: it turns a 73-hour incident into an indefinite one.

The temporal model versions what changed and nothing it invented. The decision about what counts as "changed" belongs to the extractor, not to a helpful framework guard. And "at most one current row per entity" must be something the database cannot violate, even under a buggy loader.

what we chose An SCD Type 2 mixin runs during the load phase and does four things in order: capture a single SELECT NOW() so every row in the batch aligns to one clock; _diff_records computes deleted = current minus incoming, changed = a compare-field diff (only on the declared compare fields), and new = incoming minus current; _close_records sets the closed rows' active_till = NOW() with status of changed or deleted; and _insert_records writes the new versions with active_from = that timestamp and a caller-generated UUID primary key, all in one transaction. A full-snapshot diff is what makes a rerun safe: re-ingesting the same snapshot finds nothing changed and writes nothing. The single transaction keeps the "one current row" guarantee true at every instant, even if the function is killed mid-load. The verbatim raw JSONB blob is never a compare field (it changes every sync). The extractor, and only the extractor, has the context to tell "confirmed empty" from "unreachable," so no framework-level guard is allowed to suppress an empty snapshot (the three-state contract below).

01 │ # the load stage PRODUCES versions: diff a full incoming snapshot against current rows,
02 │ # then close changed/deleted rows and insert new versions, all in ONE transaction.
03 │ def apply_scd2(self, incoming: dict[str, dict]) -> None:
04 │     now = self.db.scalar("SELECT NOW()")                  # one clock for the whole batch
05 │     current = self.load_current_rows()
06 │ 
07 │     deleted = current.keys() - incoming.keys()
08 │     changed = {k for k in current.keys() & incoming.keys()
09 │                if self._compare(current[k], incoming[k])}  # only on the declared compare fields
10 │     new     = incoming.keys() - current.keys()
11 │ 
12 │     with self.db.transaction():
13 │         self._close(deleted, active_till=now, status="deleted")
14 │         self._close(changed, active_till=now, status="changed")
15 │         self._insert(new | changed, active_from=now)        # caller-generated UUID PK

the invariant lives in the schema, not in app code A partial unique index enforces one current row per entity. A separate unique constraint enforces temporal uniqueness (no two versions share a start time). A status check constrains the lifecycle to {current, changed, deleted}: a change closes the old row (valid_till = NOW(), status = 'changed') and inserts a new current row in the same transaction. JSONB shape contracts are test-enforced, not just documented: a policy-state column must carry exactly its four documented top-level keys, asserted by a test, so a transform-side drift cannot silently violate the table's contract.

indexing follows the read pattern Partial indexes on WHERE status = 'current' keep the index off the ~99% history rows and match the dominant query shape; point-in-time and history queries use a composite (tenant_id, valid_from) index instead. GIN indexes go on the include-side arrays only, because "which policies cover this app, user, or group?" is a real forward lookup; the exclude-side arrays are deliberately not indexed (they are under 10% of the size and consumed by a graph-side audit, not by SQL).

01 │ CREATE TABLE entities_scd (
02 │   id             bigserial PRIMARY KEY,
03 │   tenant_id      text NOT NULL,
04 │   integration_id text NOT NULL,
05 │   source_id      text NOT NULL,                 -- the upstream entity id
06 │   status         text NOT NULL CHECK (status IN ('current','changed','deleted')),
07 │   valid_from     timestamptz NOT NULL,
08 │   valid_till     timestamptz,                   -- NULL while current
09 │   raw            jsonb NOT NULL,                -- verbatim snapshot
10 │   attributes     jsonb NOT NULL
11 │ );
12 │ 
13 │ -- at most one current row per entity: the DB cannot violate this even under a buggy loader
14 │ CREATE UNIQUE INDEX entities_scd_one_current
15 │   ON entities_scd (tenant_id, integration_id, source_id)
16 │   WHERE status = 'current';
17 │ 
18 │ -- no two versions of the same entity share a start time
19 │ ALTER TABLE entities_scd
20 │   ADD CONSTRAINT entities_scd_temporal_unique
21 │   UNIQUE (tenant_id, integration_id, source_id, valid_from);
22 │ 
23 │ -- partial index keeps the index off the ~99% history rows and matches the dominant query shape
24 │ CREATE INDEX entities_scd_current
25 │   ON entities_scd (tenant_id, source_id)
26 │   WHERE status = 'current';
27 │ 
28 │ -- point-in-time / history queries use a composite instead
29 │ CREATE INDEX entities_scd_history
30 │   ON entities_scd (tenant_id, valid_from);
31 │ 
32 │ -- GIN on the INCLUDE-side array only: "which policies cover this app/user/group?" is a real lookup.
33 │ -- the exclude-side arrays are deliberately NOT indexed (<10% the size, consumed by a graph audit).
34 │ CREATE INDEX access_policies_include_gin
35 │   ON access_policies_scd USING gin (include_targets);

A heartbeat field in the compare surface rewrites your whole table daily. Quarantining volatile fields is the difference between linear and quadratic storage.

why Each phantom version also emits an outbox row, which triggers a projection close-cascade that severs and rebuilds edges. So volatile churn is not just storage bloat; it is the engine that manufactures graph drift. Every cycle becomes an opportunity to drop edges.

what we chose The compare surface is a declarative per-connector class attribute (scd_compare_fields) plus a scd_compare_subfield_allowlist that names exactly which JSON sub-keys may drive a new version; everything else is stored but inert. A companion bulk UPDATE keeps volatile-but-displayed fields fresh in a single round-trip (via unnest over parallel arrays) without emitting a version, and it deliberately leaves updated_at untouched so "last meaningful change" stays meaningful.

what we tried, and what broke "Track everything" feels thorough and is exactly how a last-login or last-checkin field rewrites every row on every login.

in production One reference tenant's temporal table reached 4.66M rows, an average of 607 versions per entity, entirely from volatile-field churn, not an SCD logic bug. Measured per source: one identity provider's users averaged ~7.12 versions each in under two weeks; another provider's permission-set-groups averaged ~2.67; the original case was a last-sign-in / last-password-change heartbeat field rewriting users on every login.

01 │ -- refresh volatile-but-displayed fields in ONE round-trip regardless of drift count,
02 │ -- WITHOUT emitting a new version and WITHOUT touching updated_at
03 │ UPDATE entities_scd AS e
04 │ SET    attributes = jsonb_set(e.attributes, '{last_seen}', v.last_seen)
05 │ FROM (
06 │   SELECT unnest($1::text[])  AS source_id,
07 │          unnest($2::jsonb[]) AS last_seen
08 │ ) AS v
09 │ WHERE e.source_id = v.source_id
10 │   AND e.status = 'current';

01 │ class LoadUsers(SCDType2Mixin, AbstractLoad):
02 │     scd_compare_fields = ["display_name", "state", "attributes"]
03 │     # a heartbeat field in the compare surface rewrites the whole table on every sync,
04 │     # so name EXACTLY which sub-keys may drive a new version:
05 │     scd_compare_subfield_allowlist = {"attributes": ["role", "groups", "mfa_enabled"]}
06 │     # volatile-but-displayed fields (last_sign_in, last_checkin, last_password_change) are
07 │     # refreshed in place WITHOUT emitting a version; the verbatim raw blob is never a compare field.

two normalization traps feed the same churn

• Upstream list ordering is unstable: some providers permute nested policy arrays (statement and principal lists) on every snapshot, which read literally is a change on every cycle. A normalize-for-comparison routine recursively sorts nested lists and dict keys before serializing.
• JSONB numeric encoding bites the same way: json.dumps(123) is not json.dumps(123.0), so numbers are canonicalized identically on both sides before the diff.

Both were direct contributors to the 4.66M-row churn above, and both are cheap insurance against it.

The difference between a 14-second sync and a 73-hour kill loop was one one-line index. Index the hot path, and PROFILE before churn does.

why A MATCH against a static label plus a tenant_id predicate is an index seek, K-linear in the result, roughly 5 to 30 db hits per id. The same lookup against a dynamic per-tenant label is a label scan. The anchor patterns for carry-forward edges were deliberately rewritten to use the static label plus property predicate (a seek) rather than the dynamic label (a scan).

the carry-forward wrinkle When a principal rolls to a new temporal version for an attribute-only change, its close cascade severs owned assignment edges that the now-unchanged assignment loader will not rebuild. Carry-forward traverses the closed predecessor and re-points its edges onto the new version, needing no new index.

in production A dependency bump redeployed an integration loader, which picked up seven new optional account-state fields added days earlier. The serialized JSONB shape changed for all 4,815 users at once; the temporal model marked every one "changed"; one outbox row carried the full rebuild. The source-record to application-entity MATCH had no supporting index (the index was named in a migration header but never created), so each lookup fell back to a dynamic-label scan: a PROFILE showed ~262k db hits per source id and 15.3 seconds of a 14-minute statement in that one lookup. The job hit the 14-minute wall and was SIGKILLed; because the attempt counter was incremented in the except block, the same row was re-picked and re-killed 70-plus times over 73 hours. The fix was one index: sync dropped to 14.6 seconds and drained 46 pending rows to 0. On the largest tenant (1.2M nodes), the dynamic-label carry-forward scan would have been ~20 billion db hits in one statement; the static-label seek made it K-linear (5 to 30 hits per id). Pair every schema change with index validation; a missing index is a production incident with a fuse.

01 │ // the one-line index that turned a 73-hour kill loop into a 14-second sync
02 │ CREATE INDEX app_assignment_lookup IF NOT EXISTS
03 │ FOR (a:Application) ON (a.tenant_id, a.source_id);
04 │ 
05 │ // static label + tenant_id predicate => index seek (~5-30 db hits per id)
06 │ MATCH (a:Application {tenant_id: $tenant_id, source_id: $source_id}) RETURN a;
07 │ 
08 │ // dynamic per-tenant label => label scan (~262k db hits per id; ~20B on a 1.2M-node tenant)
09 │ // MATCH (a) WHERE $tenant_label IN labels(a) AND a.source_id = $source_id RETURN a;

batching, sized by fan-out Writes are batched by the shape of the data, not a single global number: high-fan-out user loaders use a batch size of 500, group loaders and all close cascades are atomic so a partial group never lands, and the carry-forward step always runs last so it re-points edges onto versions that already exist.

A framework-level if not data: return looks like a safety net, silently prevents legitimate mass closures, and hides real deletions as "no change." Put the decision where the context is.

why Only the extractor can tell "the upstream genuinely returned nothing" from "the upstream was unreachable." A guard one layer up that "helpfully" skips empty snapshots trades away the ability to close stale records for false confidence, at the exact layer that cannot distinguish the two.

what we chose A three-state extractor contract: [] means confirmed empty (close stale records), None means unknown (touch nothing), and a raised exception means error. The tempting early-return guard was removed once it was clear it suppressed legitimate mass-closures.

Implicit positional couplings between a serializer and a column list are foot-guns that a single reorder detonates silently. If a coupling cannot be expressed in the type system, pin it with a test.

the traps A timeline-event builder returns list(model_dump().values()), whose order must match the columnar column list exactly: reorder one model field and you silently write event-times into the tenant column. The native-protocol insert path is the same trap, it scans into concretely-typed destinations in column order, so a reordered field silently corrupts every insert. Neither throws; they corrupt quietly. Each belongs in a test that fails the moment the order drifts.

the rule covers constants too When the same key set lives in three languages, assert the mirror, do not assume it: the metric keys mirrored across the Python engine, the Go constants, and the TS DTOs are gated by an excluded-keys set and a viability check that confirms a key is actually being collected, a deliberate, gated invariant rather than three hand-maintained lists hoping to stay in sync.

At billions of rows the engine is smarter than your hint. The moment you hand-place a PREWHERE, you take an adaptive decision away from the optimizer, and usually make it worse.

why The optimizer reorders predicates relative to index granularity and decides when a skip index actually helps. A manual hint freezes a decision that should stay adaptive as the schema and data grow.

what we chose All predicates go in WHERE; the optimize_move_to_prewhere setting decides placement.

in production A manual PREWHERE caused a measured 70x regression when a bloom-filter and a text (tokenbf_v1) skip index combined on the same query. That incident is the documented origin of the "let the optimizer choose" rule.

the same humility covers skip indexes Forcing hasToken or a specific skip index on a wide range races background merges and throws RangeReader errors. So text search is scoped to narrow time windows, with a structured column filter as the default path: integration_type = 'provider_a' is both safer and faster than searching a raw blob for the substring 'provider_a'.

FINAL is one keyword that forces a blocking merge of all parts, whose cost grows with ingest velocity. Dedup at query time instead; that cost is bounded.

what we chose Tables are ReplacingMergeTree / aggregating variants. We accept stale duplicate rows during background merges and dedup at read time: ORDER BY event_time DESC LIMIT 1 BY raw_hash for distinct rows, and a conditional unique aggregate (uniqExactIf(raw_hash, ...)) for counts. Forcing final = 1 "to just get clean rows" was rejected for the same reason: its cost is unbounded under high ingest, and it makes a live freshness engine block on a merge.

01 │ CREATE TABLE events (
02 │   tenant            LowCardinality(String),
03 │   window_start      DateTime,
04 │   actor_id          String,
05 │   integration_type  LowCardinality(String),
06 │   event_time        DateTime,
07 │   raw_hash          String,
08 │   raw               String
09 │ )
10 │ ENGINE = ReplacingMergeTree
11 │ PARTITION BY toYYYYMM(window_start)
12 │ ORDER BY (tenant, window_start, actor_id, integration_type);
13 │ 
14 │ -- never FINAL. dedup at query time (bounded cost):
15 │ SELECT *
16 │ FROM events
17 │ WHERE tenant = {tenant:String} AND window_start >= {from:DateTime}
18 │ ORDER BY event_time DESC
19 │ LIMIT 1 BY raw_hash;                 -- distinct latest row per raw_hash
20 │ 
21 │ -- counts dedup with a conditional unique aggregate, never FINAL:
22 │ SELECT uniqExactIf(raw_hash, event_time >= {from:DateTime}) AS distinct_events
23 │ FROM events
24 │ WHERE tenant = {tenant:String};

Each part is a directory on disk, and the engine stalls writes past roughly hundreds-to-thousands of parts. Monthly partitions plus a tenant-first sort key are the defense at both ends.

why Monthly is wide enough to keep part count sane under high-velocity ingest, narrow enough that day and week range queries still prune ~97% of the data, and a partition is the atomic unit for TTL drops. Batching inserts to 10,000 rows is the same defense at write time: one part per 10k rows, not one per row. A tenant-first sort key makes every query a tenant-scoped index seek.

what we chose ORDER BY (tenant, window_start, actor_id, integration_type), PARTITION BY toYYYYMM(window_start), LowCardinality(String) on the low-cardinality columns, and a cascading materialized-view rollup from raw events through 5-minute, 1-hour, and 1-day grains (mixing a simple aggregate for sums with merge-state aggregates for set-valued aggregations), with inserts batched over the native protocol.

Property-only filtering works at ten tenants and leaks at a hundred, the first time someone forgets the predicate. Make isolation structural, then audit that the structure holds - and put the wall in the database, because application validation is the weakest layer in the system.

label mechanics in the graph A property filter is checked late in the query plan, gives no index-level isolation, and a single missing WHERE n.tenant_id = silently leaks across tenants. A per-tenant label lets the planner prune a tenant's subgraph at the index, and an audit catches any node whose label and property disagree. So every node carries both a :Tenant_<normalized> label and a tenant_id property; a build_tenant_label() helper is byte-identical in the Python loader and the Go service, and validates the tenant against ^[A-Za-z_][A-Za-z0-9_]*$. You cannot parameterize a label in Cypher, so the regex whitelist is the injection defense. A data-quality auditor asserts that label and property always agree.

01 │ // you cannot parameterize a label in Cypher, so the regex whitelist IS the injection defense:
02 │ //   build_tenant_label(t) validates t against ^[A-Za-z_][A-Za-z0-9_]*$  (identical in Python and Go)
03 │ MERGE (n:Application {tenant_id: $tenant_id})
04 │ SET   n:`Tenant_acme`;   // structural label, audited against the tenant_id property

the database user in the columnar store Each query runs as Username: tenantID against server-side row policies. A tenant-first sort key plus a per-tenant database user makes every query a tenant-scoped index seek that the server enforces before materialization, cheaper than per-row row-level security at billions of rows and impossible to bypass with an application bug. The Go layer adds query-shape validation only, for fast, clear errors, against a frozen forbidden-keyword list that is explicitly locked ("do not extend this list"), because defense beyond keyword-matching lives in the per-tenant user's GRANTs, not in the validator. Enforcing isolation in the validator was rejected outright: it is the weakest layer, and one missed clause is a breach - a verdict the red-team pass in the AI / LLM section later proved against the relational store.

the contested trusted-path coda On the untrusted (LLM-driven, generated-query, external-integration) data path, isolation is structural: database-enforced row-level security, per-tenant database users, fail-closed defaults, non-negotiable. On the trusted, hand-written application path, isolation is manual and chosen deliberately: a Prisma $extends audits mutations but does not inject tenant_id; every read and write must hand-add it, and some methods even take tenant_id as an optional parameter. The stated reason is that "the one query that needs cross-tenant access becomes the one query that silently leaks; explicit is harder to forget than implicit." Defense there is a tenant-validation interceptor that rejects tenant-less requests at entry, plus the auth layer, but there is no compile-time or middleware guarantee, and our own reviewers flag it as the single biggest review-worthy risk in that codebase.

A per-tenant credential path needs the same pool every time: bounded, deduplicated on cold-fill, reference-counted on release, and self-healing on an access-denied.

what we chose Per-tenant pools compose four primitives. LRU(64) with a 30-minute TTL bounds memory. A singleflight cold-fill dedups concurrent opens to exactly one Open per tenant (so a thundering herd opens one connection, not 64), and a cancelled caller abandons its wait while the detached fill keeps running for the others. Refcount plus sync.Once release plus mark-for-close means an in-flight query never has its connection closed underneath it. And a bounded retry (max 3) on the cache-versus-singleflight eviction race replaces unbounded recursion. Self-heal closes the loop: on a positively-identified ACCESS_DENIED or UNKNOWN_USER from the downstream store, invalidate both the cached secret and the connection, then let the next request re-resolve; never cache negative lookups, so a failed-tenant enumeration cannot poison the cache, and recovery is automatic and bounded to one failed request. This is the model for any per-tenant credential path.

01 │ func (f *TenantStoreFactory) Invalidate(tenantID string) {
02 │     f.secrets.InvalidateCache(tenantID)          // re-fetch the secret next time
03 │     if entry, ok := f.cache.Peek(tenantID); ok {
04 │         entry.markEvicted(f.log)
05 │         f.cache.Remove(tenantID)                 // re-open the connection next time
06 │     }
07 │ }

On the relational side the same shape holds, with two connection handles and two roles: a privileged handle on the application role, and a separate raw read-only handle on app_ro for the untrusted agent path, different driver, different privileges, by design, with sslmode=require non-negotiable. PgBouncer is deliberately not deployed: in-process pooling suffices until connection count actually pressures max_connections, and PgBouncer complicates per-tenant credential rotation.

Stagger the timeouts so the database fires first and returns a specific, mappable error instead of a generic context cancellation.

why "It timed out" with a generic context.DeadlineExceeded is not actionable. The specific database timeout error is. So the deadlines are staggered to let the database lose the race last.

what we chose HTTP 65 seconds is greater than the Go context's 62 seconds, which is greater than the database statement timeout of 60 seconds. So Postgres (or the columnar query timeout) fires first, surfacing SQLSTATE 57014 mapped to a QUERY_TIMEOUT, a specific and actionable error. The Go deadline is the safety valve for when the database is unreachable entirely.

01 │ HTTP request deadline       65s
02 │   Go context (Postgres)     62s
03 │     statement_timeout       60s   -> Postgres fires first: 57014 QUERY_TIMEOUT (specific, mappable)

Every query would have to remember deleted_at IS NULL, and the one query that forgets is a privacy incident. The privacy model is fail-closed: deleted data is gone, not flagged.

why A soft-delete column is one column versus the privacy incident the first forgotten filter ships. Hard delete plus an audit log of the delete event is a stronger, fail-closed contract: deleted data is actually gone.

what we chose Hard deletes, with the delete captured by a single Prisma $extends audit-logger that intercepts every create, update, and delete on every model and writes the audit row in the same transaction as the mutation. That makes it atomic (if the audit write fails, the mutation rolls back), transparent (no per-service audit calls to remember), and complete (you cannot route around it). It skips when request context (user_id, tenant_id) is absent, rather than writing an unattributable log.

A panic must not leak a half-committed transaction; a lock must cover the whole act, including the external call; and a check-then-write is a race until you collapse it into one atomic statement.

what we chose A WithTx helper with panic recovery: a defer rolls back on error and re-panics on a panic (so a panic cannot leak a half-committed transaction), and commits only on a clean return. An advisory lock must wrap the entire critical section, including external calls: if you lock around the database write but make the external provisioning call outside it, two requests both pass the check and one fails at the external system, so the lock's scope is the whole check-act-external sequence. And a time-of-check to time-of-use race collapses into one atomic operation: a findFirst-then-create in two statements has a race window; use upsert / INSERT ... ON CONFLICT / a serializable transaction instead.

01 │ // defer rolls back on error AND re-panics on a panic, so a panic can't leak a half-committed tx
02 │ func WithTx(ctx context.Context, db *sql.DB, fn func(*sql.Tx) error) (err error) {
03 │     tx, err := db.BeginTx(ctx, nil)
04 │     if err != nil {
05 │         return err
06 │     }
07 │     defer func() {
08 │         if p := recover(); p != nil {
09 │             _ = tx.Rollback()
10 │             panic(p) // re-panic after rollback
11 │         } else if err != nil {
12 │             _ = tx.Rollback()
13 │         } else {
14 │             err = tx.Commit() // commit only on a clean return
15 │         }
16 │     }()
17 │     return fn(tx)
18 │ }

01 │ -- the advisory lock must wrap the WHOLE check-act-external sequence, not just the DB write
02 │ SELECT pg_advisory_xact_lock(hashtext($1 || ':' || $2));   -- ($tenant_id, $resource)
03 │ -- ... read, decide, AND make the external provisioning call, all inside the same tx-scoped lock ...
04 │ 
05 │ -- collapse a findFirst-then-create (a race window) into one atomic statement
06 │ INSERT INTO entities (tenant_id, source_id, attributes)
07 │ VALUES ($1, $2, $3)
08 │ ON CONFLICT (tenant_id, source_id) DO UPDATE
09 │   SET attributes = EXCLUDED.attributes;

migrations get the same discipline Migration and deploy overlap in time, so every change ships expand-contract over two releases, add the column plus dual-write/dual-read in release N, drop the old column in release N+1, with DDL kept idempotent so a re-run or a resumed-after-failure migration is safe. The one hard never: do not run CREATE INDEX CONCURRENTLY inside the migration tool, which wraps each migration in a transaction, and Postgres forbids CONCURRENTLY inside one; run it as a separate out-of-band step.

• Ids. Every entity has an auto-increment bigint id (the primary key, excluded from every DTO) and a prefixed nanoid (tnt-..., usr-...) exposed as id in JSON: sequential integers are enumerable and collide in cache keys; prefixed ids are non-guessable and type-self-describing. In Go these are branded types (type TenantID string), so the compiler refuses to let you pass an integration id where a tenant id belongs.
• TEXT over VARCHAR(N). Postgres treats them identically for performance; VARCHAR(255) "to be safe" buys nothing and costs a future migration to widen. The deliberate, standards-driven exceptions: email VARCHAR(320) per RFC 3696, domain VARCHAR(255) per RFC 1035.
• JSONB for what evolves, columns for what you filter on. Columns hold bounded, discrete, plannable values (a display name, a state, an applies-to-all flag); JSONB holds provider-specific, evolving, or verbatim payloads, so a new provider can add fields without a migration, at the cost that you cannot cheaply plan on its contents. The raw column is never indexed: write amplification on a column written once and read rarely.

Three abstract base classes in a shared core package supply all the machinery. A new provider supplies about three thin subclasses plus declarative class attributes, and nothing else.

why Most of an ingestion integration is identical plumbing: pull a page, write a raw snapshot, validate, hash, batch, load. If every provider re-implements that, the twentieth provider re-derives the first provider's bugs. Pushing the machinery into base classes makes "add a provider" a declaration problem, not an engineering problem, and it makes the machinery a single place to fix.

what we chose An AbstractExtract whose save_raw() writes a raw snapshot to a deterministic object-store path. Deterministic paths are what make a rerun overwrite instead of duplicate, which is idempotency for free. An AbstractTransform that validates against a pydantic model, enriches, sanitizes email and IP, generates a SHA-256 content hash, and exposes the canonical-event template method as overridable hooks. An AbstractLoad that streams JSONL from the object store to the columnar store in BATCH_SIZE = 10_000 chunks. A concrete loader composes behavior by declaration only: it inherits the mixins it needs and sets class attributes.

01 │ # three abstract base classes carry all the machinery;
02 │ # a new provider supplies ~3 thin subclasses + declarative class attributes.
03 │ 
04 │ class AbstractExtract(ABC):
05 │     @abstractmethod
06 │     def fetch(self) -> list[dict] | None: ...
07 │ 
08 │     def save_raw(self, records: list[dict], collected_at: str) -> str:
09 │         # deterministic path => a rerun OVERWRITES instead of duplicating (idempotency for free)
10 │         key = f"raw/{self.tenant}/{self.integration}/{self.entity}/{collected_at}.json"
11 │         return self._put_jsonl(key, records)
12 │ 
13 │ 
14 │ class AbstractTransform(ABC):
15 │     def make_event(self, raw: dict) -> list:        # validates against a pydantic model
16 │         ...
17 │     def generate_hash(self, payload: dict) -> str:  # SHA-256, used for warehouse dedup
18 │         ...
19 │ 
20 │ 
21 │ class AbstractLoad(ABC):
22 │     BATCH_SIZE = 10_000                             # stream JSONL from the object store in 10k chunks
23 │     @abstractmethod
24 │     def load(self) -> None: ...
25 │ 
26 │ 
27 │ # a concrete loader composes behavior by DECLARATION only:
28 │ class LoadUsers(TemporalGraphSyncMixin, SCDType2Mixin, AbstractLoad):
29 │     scd_id_column = "source_id"
30 │     scd_compare_fields = [...]   # which fields trigger a new version
31 │     # + SQL templates + graph templates + a graph allowlist, all as class attributes

One serverless function per integration, phase-dispatched: a single handler routes event["integration_phase"] (extract_users, transform_users, load_users, ..., graph_sync) through a handler dict, with the storage and database connections built at module scope so warm starts reuse them.

01 │ # one function per integration, phase-dispatched; connections at module scope for warm-start reuse
02 │ HANDLERS = {
03 │     "extract_users":   extract_users,
04 │     "transform_users": transform_users,
05 │     "load_users":      load_users,
06 │     "graph_sync":      graph_sync,
07 │ }
08 │ 
09 │ def handler(event, _ctx):
10 │     return HANDLERS[event["integration_phase"]](event)

A DAG file should be about ten lines of declaration. Orchestration topology is data, not code you write by hand: the moment your second integration copy-pastes the first one's extract -> sensor -> transform -> load -> graph_sync wiring, every future change to that shape is an N-file edit, and the phase-ordering bugs are per-file.

why Hand-writing each DAG is fast for integration number two and a 35-file edit by integration number twenty, every time the phase shape changes, with copy-paste ordering bugs you only find in production. If the wiring is data, the shape lives in one builder and every integration inherits a fix at once.

what we chose A domain-agnostic phase-graph factory (build_phase_tasks()). A concrete DAG passes a declarative phase_graph: list[(phase, [deps])] plus an integration type and a schedule. The builder walks the graph, emits one mapped task per tenant-integration, and wires the >> edges as a side effect of the walk. The builder is typed against an enum, but it is deliberately not "auth helpers with a config opt-out": keeping it domain-agnostic is what unlocks adding a third or fourth builder later without re-extracting the same code. A concrete DAG ends up about ten lines.

01 │ # a concrete DAG is ~10 lines of DECLARATION: topology is data, not hand-wired code
02 │ build_integration_dag(
03 │     dag_id="provider_a",
04 │     integration_type=IntegrationType.PROVIDER_A,
05 │     schedule="33 * * * *",                 # staggered cron minute (see the schedule paragraph below)
06 │     entity_types=["users", "groups", "roles"],
07 │     phase_graph=[
08 │         (Phase.EXTRACT,    []),
09 │         (Phase.TRANSFORM,  [Phase.EXTRACT]),
10 │         (Phase.LOAD,       [Phase.TRANSFORM]),
11 │         (Phase.GRAPH_SYNC, [Phase.LOAD]),
12 │     ],
13 │ )

01 │ # the builder walks the declarative graph, fans one MAPPED task out per tenant, wires >> edges
02 │ def build_phase_tasks(dag, phase_graph, integration_type, tenants):
03 │     tasks = {}
04 │     for phase, deps in phase_graph:
05 │         tasks[phase] = (
06 │             invoke_lambda
07 │             .partial(phase=phase, integration=integration_type)
08 │             .expand(tenant=tenants)        # ONE mapped task index per tenant-integration
09 │         )
10 │         for dep in deps:
11 │             tasks[dep] >> tasks[phase]     # edges are a side effect of walking the graph
12 │     return tasks

the schedule is configuration too Each provider runs on a staggered cron minute to spread serverless load, provider A at minute 33, provider B at 43, provider C at 23, provider D at 27, provider E at 52; a fleet that all fires at the top of the hour is a self-inflicted thundering herd on the serverless concurrency limit. Every DAG sets max_active_runs=1 (a slow run must never overlap its own next run) and catchup=False (a paused or newly-deployed DAG must not stampede the provider with a backlog of historical runs on resume).

A synchronous invoke pins an orchestrator worker slot for the whole job. At fleet scale that is the difference between draining a hundred tenants in one window and serializing them into hours.

why it scales The serverless platform kills a function at 15 minutes (900 seconds) regardless. A synchronous invoke pins an Airflow worker slot for that whole window, so 100 tenants times 15 minutes is 25 worker-hours of serial latency. Async invoke plus a reschedule-mode sensor returns the slot between pokes, so 100 mapped indices drain in roughly one 900-second window, not a hundred of them.

what we chose Functions are invoked asynchronously (InvocationType='Event', which returns HTTP 202), they write status to a DynamoDB job table, and the orchestrator polls with a reschedule-mode job sensor (mode='reschedule', poke_interval=30, timeout=900). Reschedule mode frees the worker slot between 30-second pokes instead of holding it.

01 │ # async invoke (HTTP 202) + a reschedule-mode sensor that FREES the worker slot between pokes,
02 │ # the only thing that survives the 900s serverless wall at fleet scale.
03 │ lambda_client.invoke(FunctionName=fn, InvocationType="Event", Payload=payload)  # 202, fire-and-forget
04 │ 
05 │ class JobSensor(BaseSensorOperator):
06 │     mode = "reschedule"        # release the worker slot between pokes ('poke' would pin it)
07 │     poke_interval = 30
08 │     timeout = 900
09 │ 
10 │     def poke(self, ctx):
11 │         row = job_table.get_item(Key={...})     # status written by the function to a DynamoDB table
12 │         if row["status"] == "FAILURE":
13 │             raise AirflowFailException(row["error"])   # fail FAST on a terminal row
14 │         return row["status"] == "SUCCESS"

scar tissue The fail-fast-on-a-FAILURE-row rule exists because polling a terminal failure wastes the DAG's retry budget (with the default exponential backoff, each retry adds 2, then 4, then 8 or more minutes) and, under the finish-no-matter-what filter pattern below, blocks every other integration's downstream work for the duration. A terminal failure should fail the task now, not get poked until timeout.

the permanent-failure envelope The same budget logic covers provider errors: a non-retryable one, an OAuth invalid_client, a revoked grant, a deleted integration, will fail identically on every retry, so retrying it wastes the budget and pushes the real alert minutes into the future. A {"permanent_failure": True} return envelope is translated into an AirflowFailException, which bypasses the retry budget; transient errors (a timeout, a rate limit, a flaky 5xx) simply raise and retry on the normal exponential backoff.

Dataset scheduling fires on outlet emission. If a task that can run even when an upstream failed emits a dataset, every downstream DAG fires on a failed upstream. The split is load-bearing.

why Framework tasks default to trigger_rule=all_done so the DAG's run-status reflects reality (the task completes whether or not an upstream succeeded). But inter-DAG dataset emission must be all_success, because a dataset that fires on a failed upstream wakes every downstream consumer on bad data. The two rules answer two different questions: "did this stage finish?" versus "is this output real?"

01 │ # framework tasks complete no-matter-what so the DAG run-status reflects reality:
02 │ task = invoke_lambda(..., trigger_rule="all_done")
03 │ 
04 │ # but a cross-DAG dataset must NOT fire when its upstream failed, so emission is success-only:
05 │ emit = publish_dataset(..., outlets=[Dataset("integration/users")], trigger_rule="all_success")

the firewall against mapped skip-cascade The default rule has a second failure mode: with mapped tasks under all_success, one tenant returning zero entities raises an AirflowSkipException, and the mapped-skip propagation then skips the downstream chain for every index, not just the empty one. One tenant's empty extract becomes every tenant's skipped pipeline. So a filter_valid_extracts task with trigger_rule=all_done sits before every transform and load: it runs after all indices finish, drops the extracts that produced no object-store path, and fans out only the survivors. One tenant's empty or failed extract no longer skip-cascades and kills the others.

01 │ # the firewall against mapped skip-cascade. runs AFTER all indices finish (all_done),
02 │ # drops the empties, fans out only the survivors.
03 │ @task(trigger_rule="all_done")
04 │ def filter_valid_extracts(results: list[dict]) -> list[dict]:
05 │     return [r for r in results if r.get("s3_path")]

Passing connection_config on the task payload is convenient and persists tenant secrets into the orchestrator's cross-task storage, where the UI renders them in plaintext.

why It is tempting to attach the connection config to the task payload so the function has everything it needs. That config then lands in Airflow's XCom, where the UI renders it in plaintext for anyone with access. Lazy-fetching at invocation keeps secrets out of XCom entirely, and it means credential rotation needs no DAG redeploy: the next run simply reads the new value.

what we chose Credentials are lazy-fetched at invocation from Postgres, never passed through the task payload or XCom.

Raw provider JSON becomes one canonical event model through a declarative path map with fallbacks, not a hand-rolled parser per provider. And the failure-status mapping is never allowed to default.

why Provider event shapes drift constantly: a field moves, an array gains a layer, a key is renamed across event variants. A hand-rolled parser crashes on the first drift. A declarative path map with ordered fallbacks degrades to a default instead, so a missing path is a blank field, not a dead pipeline.

what we chose Raw provider JSON maps into one canonical pydantic event model (model_config = extra="forbid", so an unmapped field is a loud validation error, not a silent drop) using glom path-extraction with Coalesce fallbacks, plus a template-method taxonomy classifier with overridable hooks. Coalesce traverses alternative provider paths and degrades to a default, so schema drift across event variants does not crash the parser.

01 │ base_event = glom(raw, {
02 │     "src_ip":      Coalesce("connection.ip", T["request"]["ip_chain"][0]["ip"], default="::", skip=(None,)),
03 │     "actor_email": "actor.email",
04 │     "target":      Coalesce(T["target"][0]["name"], T["target"][0]["id"], default="", skip=(None,)),
05 │ })
06 │ # Coalesce walks alternative provider paths and degrades to a default,
07 │ # so schema drift across event variants does not crash the parser.

the house rule with teeth Every integration must override the status classifier hook. The base default maps to status id 1 (Success), which means a provider's failure events are silently recorded as successes if you forget. The classification hooks (_classify_category, _classify_activity, _map_status) are overridable precisely so each provider states its own taxonomy, and the status mapper is the one hook that is dangerous to leave at its default.

We run a managed queue in exactly one place, and it is a tenant-provided feed we consume, not an architecture we chose. The one rule that matters is to delete the message only after the durable write succeeds.

why Deleting a message on receive, before the durable write, loses data on any crash between receive and write. The queue message is never the system of record; the thing you wrote durably is.

what we chose A consumed audit-log feed with MaxNumberOfMessages: 10, WaitTimeSeconds: 10 (long-poll), and a deliberately short VisibilityTimeout: 660, because the message is the trigger and the raw durable object chunk is the recovery anchor. Messages are deleted only after the durable object write succeeds; any message referencing a failed object is left undeleted so it reappears next hour. There is no dead-letter queue, because these logs are detective, not operational.

01 │ // The one queue we consume: a tenant-provided audit-log feed, not an architecture we chose.
02 │ const params = {
03 │   MaxNumberOfMessages: 10,
04 │   WaitTimeSeconds: 10,     // long-poll
05 │   VisibilityTimeout: 660,  // short on purpose: the message is the trigger, the durable chunk is the recovery anchor
06 │ };
07 │ // Delete the message ONLY after the durable write succeeds; a message pointing at a failed object is
08 │ // left undeleted so it reappears next hour. No DLQ: audit logs are detective, not operational.

A DTO is a contract boundary. It is not a database row, not a domain object, and not a bag of helper methods. Request, response, and domain are three different types because they answer to three different masters: the validator, the API consumer, and the database. And the frontend's wire types are generated from that contract, never hand-written.

what we tried, and what broke The shortcut is one class with @Expose() to hide the request-only fields. It saves files today. Tomorrow it embeds an if (role === ADMIN) authorization check, a toJSON(), an isExpired(), and is now a god-object no test can isolate. Mapping logic creeps into the DTO next, and the type that was supposed to describe a shape now decides behavior.

what we chose Three distinct types, with DTO files defining data shapes only: no utility functions, no helper methods. Mapping lives in services, never in DTOs. Constructors do normalization only, for example null to undefined, or "records owned by the requester carry no shared-by fields." The split is enforced at review, not by the framework. On the frontend, the Swagger / OpenAPI decorators are the single source of truth for the wire shape, fed through an OpenAPI code generator into a typed query layer; the frontend never hand-writes an API type, so an optional-to-required change breaks the build, not a user.

01 │ // Request DTO: inbound, untrusted. class-validator decorators only.
02 │ export class CreateEntityRequest {
03 │   @IsString() @IsNotEmpty()
04 │   name!: string;
05 │ 
06 │   @IsOptional() @IsString()
07 │   description?: string;
08 │ }
09 │ 
10 │ // Response DTO: the public shape. @Expose / @Exclude + Swagger.
11 │ export class EntityResponse {
12 │   @Expose() name!: string;
13 │   @Expose() description?: string;
14 │   // no toJSON(), no isExpired(), no authorization: shape only.
15 │ }
16 │ 
17 │ // Internal type: the persistence shape. Never serialized to the wire.
18 │ type EntityRow = Prisma.EntityGetPayload<{}>;

scar tissue The cheapest place to enforce "one job" is the review, before the second responsibility ever lands.

Validate where data enters, and validate by where it came from. HTTP request bodies are user input. LLM tool output is untrusted machine-generated data. They get different validators.

why Validating model output as rigorously as user input catches hallucinated or partial JSON at parse time, where the agent can self-correct, instead of letting a silent {} reach the engine downstream.

what we chose class-validator for HTTP requests; a runtime schema (Zod) for LLM tool output; and the tenant identifier is injected from the authenticated context, never read from either wire.

01 │ // LLM output is not user input: it is untrusted machine-generated data.
02 │ const ToolArgs = z.object({
03 │   entityId: z.string(),
04 │   limit: z.number().int().min(1).max(100),
05 │   // tenantId is BANNED here: it comes from the authenticated context.
06 │ });
07 │ 
08 │ const parsed = ToolArgs.safeParse(modelEmittedJson);
09 │ if (!parsed.success) return toolError(parsed.error);   // agent can self-correct
10 │ const args = { ...parsed.data, tenantId: ctx.tenantId }; // identity from context

the quick win we rejected "Trust the LLM to emit valid JSON, add the schema later." Later is after the silent-{} bug ships, and partial JSON during streaming guarantees you hit it.

The API surface is the stable contract. The database schema is free to change behind it. Several obvious conveniences were rejected on purpose, and the rejections are the part worth memorizing.

what we rejected, and why

• No soft deletes. "The one query that forgets to filter them is a privacy incident." A deleted row that is still physically present is one missing WHERE clause away from a leak.
• No automatic tenant-injection middleware. "The one query that needs cross-tenant access becomes the one query that silently leaks. Explicit is harder to forget than implicit." This is a genuinely contested call: implicit injection is more convenient and removes a class of "forgot the filter" bugs, but it also removes the visible seam where a deliberate cross-tenant read would stand out in review. The tension resolves one layer down: row-level security is the structural backstop that catches a forgotten filter, which frees this rejection to be what it is - an app-layer ergonomics call about keeping the seam visible.
• No repository pattern. "Prisma's typed query builder is the abstraction layer." A second hand-rolled abstraction over a typed query builder buys indirection, not safety.
• No UUID primary keys for external ids. UUIDs are "type-anonymous": you cannot tell a user id from a conversation id by looking. Instead an internal bigint id (excluded from the API) plus an external prefixed entityId.

For anything streamed, an agent run or a long job with progress, never block "start the work" on "watch the work." The action endpoint returns an id immediately and runs the work async; events arrive on one persistent SSE connection per user, multiplexing every conversation, filtered server-side by identity.

why If "start" blocked until the first event, a slow first token times out the POST, the caller never gets an id, and there is nothing to re-subscribe to after a drop. Decoupling makes the UI resilient to a slow start: the id exists the instant the request returns, so a reconnect always has something to attach to. And one SSE endpoint per conversation is a clean mental model that is dead on arrival at the browser's connection limit: browsers cap concurrent connections per origin at roughly 6 to 8 - that is HTTP/1.1's per-origin limit, and a proxy in the path can still downgrade you to it - so the 9th open conversation silently loses its stream, and the reconnect plus keep-alive logic multiplies per connection. One multiplexed channel is a single TCP connection and a single reconnect loop no matter how many conversations are open.

what we chose POST /conversations/start fire-and-forgets the agent run and returns { conversationId } immediately; the in-flight run is tracked in a pendingAgentRuns set so the process can drain gracefully on shutdown instead of severing live work. Events are not on this response; they flow over a shared SseService, backed by an EventEmitter exposed as one RxJS Observable on GET /sse, that filters by userEntityId and tenantEntityId. The frontend subscribes once and routes each event to the right conversation in a local store.

01 │ @Post('conversations/start')
02 │ async start(@Body() body: StartConversationRequest, @Ctx() ctx: RequestContext) {
03 │   const conversationId = newEntityId('cnv');
04 │ 
05 │   // fire-and-forget: do NOT await. Events arrive on the SSE channel.
06 │   const run = this.agent
07 │     .run(conversationId, body, ctx)
08 │     .finally(() => this.pendingAgentRuns.delete(run));
09 │   this.pendingAgentRuns.add(run);     // tracked for graceful drain on shutdown
10 │ 
11 │   return { conversationId };          // returns immediately, no events here
12 │ }
13 │ 
14 │ @Injectable()
15 │ export class SseService {
16 │   private readonly emitter = new EventEmitter();
17 │ 
18 │   // One Observable per request, filtered to this user's events only.
19 │   stream(userEntityId: string, tenantEntityId: string): Observable<SseEvent> {
20 │     return fromEvent(this.emitter, 'event').pipe(
21 │       filter((e: SseEvent) =>
22 │         e.userEntityId === userEntityId && e.tenantEntityId === tenantEntityId),
23 │     );
24 │   }
25 │ 
26 │   emit(event: SseEvent) { this.emitter.emit('event', event); }
27 │ }
28 │ 
29 │ @Sse('sse')                           // GET /sse: ONE connection per user
30 │ stream(@Ctx() ctx: RequestContext) {
31 │   return this.sse.stream(ctx.userEntityId, ctx.tenantEntityId);
32 │ }

the quick win we rejected "Return the events on the start response." It couples the request lifetime to the run lifetime, so a refresh loses everything mid-run.

The replay path is the single nastiest bug class in LLM streaming. On resume, when a summary exists, inject it into the system prompt and stop. Do not replay the raw event history.

why A summary ("here is what 10 messages of analysis found") needs no balanced tool pairs, so it sidesteps the provider's message contract entirely. Replaying the raw history is forever one missed filter away from an unbalanced tool_use that stalls the conversation. When there is no summary, only the user and response messages are replayed and tool events are skipped on purpose: replaying a tool_use without its matching tool_result (or the reverse) orphans the pair, and the model provider's message contract rejects an unbalanced pair outright.

what we chose A buildMessageHistory that branches on whether a summary exists, and skips tool events on the no-summary path.

01 │ function buildMessageHistory(conversation, events): Message[] {
02 │   if (conversation.summary) {
03 │     // Resume from summary: inject it into the system prompt and STOP.
04 │     // No prior events are replayed.
05 │     return [systemWithSummary(conversation.summary)];
06 │   }
07 │ 
08 │   // No summary: replay USER / RESPONSE only. Tool events are SKIPPED to
09 │   // avoid orphan tool_use / tool_result pairs that break the model
10 │   // provider's message contract.
11 │   return events
12 │     .filter((e) => e.type === 'USER' || e.type === 'RESPONSE')
13 │     .map(toProviderMessage);
14 │ }

the quick win we rejected "Replay the full event history on resume, to preserve context." It is the intuitive move, and it is precisely what orphans tool pairs and breaks the provider's message contract. The lesson generalizes: history you resend must satisfy the provider's invariants, and "close enough" is not forgiving.

The agent loop emits through one seam, the user always sees motion even on a buffered turn, and a cancel issued on one pod is honored by another. The frontend's reconnect loop survives a token refresh mid-stream, a navigation race, and a stale reconnect that thinks it is still live. Every one of those is a real bug, and the fix for each is a specific mechanism.

the server half Streamed turns buffer: a long model call can go quiet for seconds while it composes, and a silent stream reads as a hang. And in a multi-pod deployment, the pod that owns the in-memory cancel signal is rarely the pod the user's cancel request lands on, so cancellation has to cross the instance boundary. The agent loop emits through an AgentEventSink seam, where model SDK stream parts are converted to SSE event DTOs; the rest is mechanism:

01 │ // Each agent loop iteration checks BOTH cancellation sources.
02 │ while (continueLoop && iteration < maxIterations) {
03 │   if (this.cancel$.value || (await redis.exists(`agent:cancelled:${id}`))) break;
04 │   // The Redis key (TTL 1h) means a cancel issued on another pod is honored
05 │   // even by the pod that owns the in-memory Subject.
06 │ 
07 │   // tool-call blocks: emit SSE-only first, persist later with the result.
08 │   sink.emit({ type: 'tool_call', label: displayLabel, persist: false });
09 │   // ... run the turn, then persist with the tool_result ...
10 │ }
11 │ 
12 │ // Heartbeat: rotating activity so the user sees motion on a buffered turn.
13 │ const labels = ['Thinking...', 'Analyzing...'];
14 │ let i = 0;
15 │ const heartbeat = setInterval(
16 │   () => sink.emit({ type: 'activity', label: labels[i++ % labels.length], persist: false }),
17 │   15_000,                              // 15s server keep-alive
18 │ );

the frontend half A long-lived stream outlives the access token, so a naive reconnect re-sends an expired header and drops the stream; and the backend can emit before the chat view has mounted, so an event can arrive with nowhere to land. The fixes:

01 │ // fetch-event-source, guarded by a connection epoch.
02 │ const epoch = ++connectionEpochRef.current;
03 │ 
04 │ await fetchEventSource('/sse', {
05 │   openWhenHidden: true,
06 │   // custom fetch re-injects the LATEST auth header on every retry, read from
07 │   // a ref, so a mid-stream token refresh does not drop the stream and does
08 │   // not itself trigger a reconnect.
09 │   fetch: (url, init) =>
10 │     fetch(url, { ...init, headers: { ...init.headers, Authorization: authRef.current } }),
11 │ 
12 │   onmessage(ev) {
13 │     if (connectionEpochRef.current !== epoch) return;  // stale loop: ignore
14 │     if (seenEventIds.has(ev.id)) return;               // Set<eventId> dedup
15 │     seenEventIds.add(ev.id);
16 │     bufferFor10s(ev);   // 10s buffer: survives create-before-mount nav race
17 │     store.reduce(ev);   // into a local store keyed by stable event id
18 │   },
19 │ 
20 │   onerror(err) {
21 │     if (err instanceof FatalSseError) throw err;       // permanent abort
22 │     return 5_000;                                       // 5s backoff, then retry
23 │   },
24 │ });
25 │ 
26 │ // ready-state lives in a REF, read synchronously - never React state:
27 │ if (readyStateRef.current === OPEN) { /* ... */ }

scar tissue The duplicate-subscription bug was a real hotfix: ready-state tracked in React state closes over a stale value under batched updates, so two reconnects both believed they were active; the fix reads readyStateRef.current synchronously. The lesson is one sentence: in streaming, the state you read must be current at the instant you read it.

The durable transcript and the live-stream cache are two different jobs with two different reliability requirements. Postgres is the source of truth. Redis exists only so a refreshed page can replay in-flight events, and it is allowed to fail.

why Persisting every event straight to Postgres and reading it back on refresh saturates the database on a 50-event-per-second stream, and still needs an "incomplete versus complete" marker that an in-memory buffer gives you for free. Treating the live cache as best-effort keeps a refresh smooth without making the cache a hard dependency of correctness: a Redis outage degrades the refresh-replay experience, never the run.

what we chose A durable message table in Postgres as the primary transcript, keyed by conversation_id plus an atomically-incremented message_index; an async S3 archive of transcripts and summaries to tenant-namespaced paths for audit and training; and a Redis live-stream cache with a 30-minute TTL so a refreshed page replays in-flight events. Every cache operation is wrapped so a Redis failure logs and continues instead of breaking the run.

01 │ // Redis is best-effort: a cache failure must NEVER break the run.
02 │ async function cacheEvent(id: string, event: SseEvent) {
03 │   try {
04 │     await redis.rpush(`live:${id}`, JSON.stringify(event));
05 │     await redis.expire(`live:${id}`, 30 * 60);   // 30-minute live-stream cache
06 │   } catch (err) {
07 │     log.warn('live cache write failed; run continues', err);  // non-fatal
08 │   }
09 │ }
10 │ // Postgres message table is the primary transcript (durable truth), keyed by
11 │ // conversation_id + an atomic message_index; S3 async-archives transcripts and
12 │ // summaries to tenant-namespaced paths (audit / training). The DB is NOT written
13 │ // per-event on a 50-event/sec stream: the in-memory buffer marks complete vs incomplete.

the quick win we rejected "Persist every event straight to Postgres and read it back on refresh." It saturates the database under load and reinvents the completeness marker the buffer already gives you.

Application caching is for slow-changing config, not live analytics. Do not cache the expensive query results; cache the boring config. And every cache op must fall through to source on failure.

why The intuitive move is to cache the hot, expensive analytical queries. That is exactly backwards: caching live analytics is how a freshness engine starts returning stale answers. The right target is the slow-changing config that is annoying to fetch and rarely changes.

what we chose One Redis (or Valkey) instance, read imperatively (no annotation magic), under the same best-effort rule as the live-stream cache above: the cache is never a hard dependency. Keys follow a namespace:id:version convention, where the version is itself the invalidation trigger: rotate the key version and the old entry is simply never read again. Invalidation otherwise is an explicit del on every mutation (a schedule change, a secret rotation, a conversation delete), with TTL-only for external data that can tolerate staleness.

01 │ // imperative read-through; EVERY cache op falls through to source on a Redis failure (best-effort).
02 │ async function readThrough(key, ttlSeconds, loadFromSource) {
03 │   try {
04 │     const hit = await redis.get(key);
05 │     if (hit) return JSON.parse(hit);
06 │   } catch { /* cache down: fall through, never a hard dependency */ }
07 │ 
08 │   const fresh = await loadFromSource();
09 │   try { await redis.set(key, JSON.stringify(fresh), "EX", ttlSeconds); } catch { /* ignore */ }
10 │   return fresh;
11 │ }

what is cached, and what is not Secrets-manager values (1 hour, encrypted before caching), schedule-describe DTOs, the canonical event-model schema (24 hours), identity-provider user and tenant records (15 minutes), and the derived 30-day run-metrics DTO. The hot columnar and graph analytical queries themselves are deliberately not cached: app caching is reserved for slow-changing config, not live analytics.

The low-level hypervisor API is "maximum control." Maximum control means building vCPU exits, page tables, and a device model from scratch. That is the multi-year tax you are choosing to pay.

why The high-level virtualization framework hands you a VM object, a Linux boot loader (no manual ELF parsing), a virtio-socket device, and virtio block/net/console devices already written and proven at scale by other shipping VM tools. The low-level framework hands you a blank page and a syscall. For a v1 whose value is everywhere except the VMM, the device models alone are years you do not have.

what we tried, and what broke The day-1 vision document - written before a line of VM code existed - picked the low-level hypervisor framework for control. It was reversed on evidence: a multi-year VMM project for a product whose differentiation lives elsewhere, when the high-level framework got us booting in "weeks instead of years." We keep the low-level path documented as an escape hatch, not a plan.

what we chose The high-level platform virtualization framework as the VMM. We accept the tradeoffs and we wrote them down: tied to the OS vendor's release cycle, no support for old OS versions, less control over vCPU scheduling, and a low-level, unsafe binding whose taming is a whole crate's job.

why it is better The bet held: cold VM boot landed at ~2-3 s against a < 5 s target.

why Every FFI boundary you hand-write is surface you hand-maintain. An auto-generated binding to the framework's object runtime gives 100% coverage of the API with a single boundary (Rust -> the native runtime), riding an ecosystem with tens of millions of downloads behind it.

what we tried, and what broke The binding-strategy menu was a trap of plausible options. A C wrapper is manual maintenance forever. A Swift bridge is two FFI boundaries (Rust -> Swift -> the runtime). A second systems-language bridge is a double FFI for "zero benefit." Each adds a layer that exists only to be kept in sync with the framework you are already bound to.

what we chose The auto-generated runtime binding, wrapped by a hand-written safe layer that is the binding crate's entire reason to exist. The unsafe, low-level binding is the cost; the safe wrapper is the product.

tradeoff The binding is low-level and lags the framework. It did not expose the network attachment's designated initializer; the inherited new() yields a NULL network reference and the framework segfaults during validation. The fix calls the missing initializer through a raw msg_send! and wraps the lower-level network framework directly - disclosed technical debt until the binding catches up, and the reason the field-order teardown contract (below) became load-bearing. Check the header for the designated initializer before assuming new() is enough. Still worth it: one boundary to debug instead of three.

This is the canonical pattern. Reach for it whenever async meets a thread-affine C or Objective-C framework. One owning thread, a command channel, and a clonable handle that is the only thing the async world ever touches.

why The framework's VM objects are !Send - thread-affine - and their lifecycle is async completion handlers. You cannot move them between tokio tasks. So you stop trying. One dedicated std::thread exclusively owns every framework object; the async world talks to it over a channel; a Send + Sync handle (a clonable channel sender) is the only thing async code ever holds.

what we tried, and what broke Pinning all VM ops to one async task with a local set complicates concurrent health checks and file sync. Shared memory is harder to debug. The request/response shape maps naturally onto a channel - so a plain OS thread "is simpler and more predictable."

what we chose A spawned thread owns the registry; each handle method opens a one-shot reply channel, sends a command, and blocks on the reply - so the async caller gets a synchronous-looking API while every framework call stays on the owning thread.

01 │ #[derive(Clone)]
02 │ pub struct VmHandle { tx: mpsc::Sender<VmCmd> }   // Send + Sync; the async side holds ONLY this
03 │ 
04 │ pub fn spawn() -> VmHandle {
05 │     let (tx, rx) = mpsc::channel();
06 │     std::thread::Builder::new()
07 │         .name("vm".into())
08 │         .spawn(move || vm_thread_main(rx))         // the !Send framework objects live here, alone
09 │         .expect("failed to spawn VM thread");
10 │     VmHandle { tx }
11 │ }

why it is better One level deeper, each VM also owns a serial dispatch queue and bridges completion handlers back to mpsc, so even inside the owning thread the API stays synchronous and runtime-free. And the owning thread is not async at all: it multiplexes its command channel and a live file-sync event source with a single recv_timeout - a 50 ms poll while syncing, a blocking recv when idle - so it costs zero CPU at rest with no second runtime and no busy spin.

why To hand a closure that captures raw framework pointers onto the serial queue, you must assert Send. The assertion is sound - but only for a precise reason, and a language edition will silently break it if you are not careful.

what we chose A newtype wrapper carrying the closure, unsafe impl Send, consumed by a method that runs it on the queue:

01 │ struct AssertSend<T>(T);
02 │ unsafe impl<T> Send for AssertSend<T> {}     // sound ONLY because exec_sync blocks: no concurrent access
03 │ 
04 │ impl<T: FnOnce()> AssertSend<T> {
05 │     fn call(self) { (self.0)(); }            // consumes self -> WHOLE-struct capture
06 │ }
07 │ 
08 │ fn dispatch_sync<F: FnOnce()>(&self, f: F) {
09 │     let work = AssertSend(f);
10 │     self.queue.exec_sync(move || work.call());   // NOT `work.0()` - see below
11 │ }

what we tried, and what broke Two subtleties live in those eight lines. First, the Send assertion is sound only because the dispatch blocks the calling thread until the closure finishes - there is never concurrent access to the !Send captures. Second, edition-2024 closures capture individual fields: move || work.0() captures work.0: F (which is !Send) and defeats the wrapper entirely. A method that consumes self forces whole-struct capture of the Send newtype.

the rule Any Send/Sync newtype wrapper in the 2024 edition must be used through a consuming method, never direct field access inside the closure.

The most expensive lifetime lesson in the project. The framework holds disk-image file descriptors until the reference-counted VM object is dropped - not when the VM is stopped.

why Leave a "stopped" VM sitting in your map and its disk FDs are still held. The next start fails with framework error 2, "invalid storage device attachment," and you spend a day proving the disk is fine. Stop must remove and drop.

what we chose Stop removes the VM from the map first, force-stops synchronously, and lets the reference-counted object drop at scope end - which is when the framework finally releases the FDs.

01 │ let Some(vm) = self.vms.remove(id) else { return };  // remove FIRST so the Retained can drop
02 │ vm.force_stop();                                      // blocks on the completion handler
03 │ // `vm` drops here -> framework releases the disk-image FDs

tradeoff / corollary We skip graceful ACPI shutdown entirely: our minimal guests have no ACPI handler, so a polite stop just burns ~5 s timing out. Application-level grace belongs in pre-stop hooks, not in waiting on a shutdown the guest will never acknowledge.

corollary (borrowed fds) A framework-owned socket file descriptor is closed when the framework object drops - dup() it the moment you keep it, so the Rust side outlives the object that lent it.

why The framework opens disk images read-write and corrupts ext4 metadata if a start aborts. Validation still passes; the next boot rejects the image. Any disk a VMM opens read-write is, in effect, mutable persistent state you did not mean to keep.

what we chose The registry boots a runtime copy of the rootfs, never the original.

why A file-handle serial-port attachment produced zero output. The API marks the read handle optional; in practice, with a nil read handle, the whole attachment silently does nothing - so the earliest boot messages vanish into a port that "works." Optional was a lie.

what we chose Wire both handles (two pipes), and when a framework "optional" silently no-ops, test against the vendor's own language to rule out the binding before you trust the docs. The guest section hits the same symptom from the platform side: no legacy UART means the earliest boot messages are lost before the virtio console comes up - two independent ways to get a silent boot.

why "How much unsafe" is the wrong question; "where" is the right one. A workspace lint makes every unsafe deliberate, and the distribution is the safety story: the VMM crate (the entire framework + network FFI) and the guest agent (raw syscalls as PID 1) carry essentially all of it; the two crates that define the data model and the wire protocol - the ones everything else depends on

• carry zero.

why it is better The parts every other crate trusts are the parts with no unsafe in them. The danger is quarantined at exactly the two places it is unavoidable. The domain stays synchronous and free of external dependencies for the same reason: async, like unsafe, belongs at the edges. That is a defensible posture you can show a reviewer in one table.

why Our own internal notes claimed the error model was "1:1, no stringification." Re-grounding against the code, it is two-layer and lossy on purpose: the infrastructure error carries a structured framework { code, message }; the boundary into the domain collapses seven variants into five and stringifies the structured code.

01 │ VmmError::Framework { code, message } => VmError::Framework(format!("({code}): {message}"))

why it is better That is an acceptable, disclosed lossy boundary - nothing currently pattern-matches on framework codes - but it is not "no stringification," and the fix (a structured code: Option<isize> on the domain error) is written down for the day retry logic needs it.

Rust drops struct fields in declaration order. When those fields are a native object and the things holding a reference to it, the order is not style - it is correctness.

why The shared network object is reference-counted across all VMs and its Drop calls the native release. If it drops while live VM attachments still hold a reference, the release runs under their feet and teardown crashes.

01 │ pub struct VmRegistry {
02 │     vms: HashMap<VmId, VirtualMachine>,   // VMs (and their attachments) declared FIRST -> drop first
03 │     // ...
04 │     vmnet_network: Option<Arc<VmnetNetwork>>,  // declared AFTER vms -> its release runs LAST
05 │ }

the rule When a struct owns native/reference-counted objects and their referrers, write the drop order down as the field order, with a comment, because the next person to alphabetize the fields will introduce a teardown crash and never suspect the struct definition.

why Health, file sync, credentials, and exec ride a virtio-socket control plane that is independent of network state - it works even if DHCP races or there is no IP, sub-0.1 ms. The data plane is a shared L2 segment carrying standard TCP, so the existing stack's clients (psql -h …, redis-cli -h …, the node and Go drivers) work with no client changes - a hard requirement for replacing the incumbent without rewriting the app.

why it is better Splitting the planes means the thing you need most when networking is broken (health, the ability to push a fix) does not depend on the thing that is broken.

The NAT attachment was "assumed to work and never exercised." Every acceptance path for four phases was host->guest. The first time two VMs actually talked to each other, it failed.

why From one VM, connecting to another returned EHOSTUNREACH with an all-zeros ARP entry. Root cause: the platform's NAT attachment programs every bridge-member NIC with the OS PRIVATE flag, which by design isolates members from each other - a member can only reach the host gateway - and there is no public API to clear it. The NAT was doing exactly the safe, host-only thing it was built to do.

what we chose A shared vmnet network, one per daemon, putting all VMs on one L2 segment: same subnet, same outbound NAT, same TCP clients - "we just removed the PRIVATE flag." It even hid inside the later phase for a while, because the dogfood app died at env validation before it ever opened a socket; a synthetic raw connection from inside one VM is what finally isolated it.

the rule Exercise a primitive on the path it actually has to serve, early, end-to-end. A plausible feature tested only on the easy path is an assumption wearing a green checkmark.

This is the lesson. We started from a minimal config to chase a tiny kernel and spent six weeks rediscovering, one crash at a time, every option a mainstream distro turns on by default. There was no stop condition.

why Building up from a minimal config means every new workload finds the next missing option by failing in production. Building down from a distro's VM-guest config means the defaults are already right and you only ever remove.

what we tried, and what broke From a minimal kernel, every database and runtime failure was a missing CONFIG, and every one of them was a distro default:

The realization, verbatim from the decision record: "every fix was something every mainstream distro kernel has by default… the next runtime to onboard will surface another set of missing CONFIGs through another debugging cycle. There is no stop condition."

what we chose Build from the distro defconfig, merge a vendored Alpine VM-guest config (the recognized "VM-guest sweet spot," used by the mainstream Mac VM tools), then a thin overrides layer that only forces virtio/vsock/ext4 builtin and strips what the framework never exposes (audio, USB, wireless, DRM, RAID, NFS). The JIT now runs with no --jitless flag and no per-runtime workaround.

why it is better ~85% of the product is kernel-agnostic - the kernel is infrastructure, not differentiation. Minimalism externalized the entire distro-kernel maintenance burden onto our own dogfooding cycle, one runtime crash at a time. The kernel grew ~5 MB -> ~10 MB (≈0.1% of a 5 GB VM disk) and the boot-time delta is unmeasurable against framework startup. Trim from working.

corollary (=y, never =m) The guest ships no /lib/modules and the init has no module loader, so a virtio driver built as a module (=m) is not "lazily loaded" - it does not exist, and a large share of the override config exists purely to drag the distro's =m virtio/vsock/ext4 settings back to =y. For anything needed at boot, =y, never =m - and verify the full resolved .config after merging fragments, not just your fragment, because olddefconfig can silently drop a dependency you thought you set.

platform facts, not bugs The boot loader rejects a gzipped kernel - ship an uncompressed Image (check the magic at the known offset). On arm64 the framework enumerates devices via a device tree, not ACPI, and provides no legacy UART - so earlycon does nothing and the earliest boot messages are lost before the virtio console comes up. Design around these; do not debug them.

A failed command under set -e exits the shell. When the shell is PID 1, exiting it panics the kernel. The most catastrophic one-character bug in the system.

why The process-handling bugs look unrelated until you trace them to one source: the agent is init.

• Never set -e in a PID-1 init script. Auto-mounted devtmpfs makes the init's own mount devtmpfs fail "resource busy"; under set -e, PID 1 exits on that non-fatal failure and the kernel panics with no init. The fix is || true per command and an init that treats failure as information, not death.
• Ignoring SIGCHLD to auto-reap zombies also reaps the children you are actively waiting on, so try_wait() returns ECHILD. Read the child's stdout/stderr to EOF first, then wait, and treat ECHILD as "completed," not as an error.
• Long-lived services reparented to PID 1 get SIGPIPE when a dropped handle closes the pipe. Use file-based logging for anything long-lived or reparented - never pipes.
• A privilege-dropping su double-forks, so the PID you tracked dies while the real service is reparented away. Handle ECHILD in the health check and exec to cut a fork.

the rule When your agent is PID 1, you signed up for every duty init has. Design the process handling around reaping from the start, not as a patch after the first zombie.

A hand-assembled rootfs silently diverges from a real install in exactly the ways a real application trips over. Unpacking the archives skips every post-install hook, and you must redo each by hand.

why Because we extract packages without running their install hooks, every side effect those hooks normally perform is missing until reproduced:

the rule A "process" rootfs is defined less by its package list than by faithfully reconstructing a real distro's post-install glue: applet symlinks, /usr/bin/env, the CA bundle, PATH, and the OS-identity files. And distro extension packaging is keyed to the default server major - you cannot drop an extension built for one major into another.

The headline feature, and the measurements vindicate the bet. A branch is "swap the data image behind a stable endpoint" - same VM, same port, different data - created in tens of milliseconds for zero bytes.

why The copy-on-write clone syscall is public and unrestricted - no special entitlements, unlike the direct filesystem-snapshot APIs. It is instant regardless of database size, ~50 lines of FFI, and consumes zero disk until writes diverge. Mechanism: checkpoint the database, clone the data directory, point a new database at the clone, serve both.

what we tried, and what broke Schema-per-branch makes migrations collide - one agent adds a column, another drops it, on a shared type system - plus extension conflicts. Too fragile for concurrent multi-agent migration. Database-per-branch gives each branch fully independent migrations.

why it is better Branch create measured 32-35 ms on a 1.1 GB volume; total disk across a main plus five clones was 0 bytes until they diverged (pure copy-on-write). The six-branch lifecycle and WAL-replay-after-dirty-shutdown both validated. The bet was the product, and the numbers cleared the target by orders of magnitude.

the syscall is instant; the consistency is your job A copy-on-write clone of a live database directory is a crash-consistent copy unless you quiesce first - the syscall does not know about your write-ahead log. Checkpoint the database before the clone, and guard the filesystem type (the clone syscall is filesystem-specific) so you fail loudly on the wrong volume instead of silently degrading. Instant cloning is a storage primitive, not a database guarantee.

why The guest's credential store is an in-memory map applied via the process environment; nothing is written to a file. Because the secrets live only in the agent's process memory, they are absent from the rootfs image, from every copy-on-write branch clone, and from any disk snapshot - by construction, not by a scrubbing step you might forget.

tradeoff (disclosed) The residual is real and named: the allocator does not zero on dealloc, so a freed credential can linger in process memory until overwritten. The fix (zeroing the store on rotation) is scoped, not pretended-away. "By construction" is the strong guarantee; the residual is the honest asterisk.

why Each CLI invocation is a separate process, so an in-process lock cannot serialize them - that needs flock(LOCK_EX), which is released automatically on process death, so there are no stale locks. Concurrent operations within the daemon need an in-process Mutex. Two layers, two distinct races, named separately.

what we chose Sessions and branches persist as single human-readable JSON files under a state directory, every mutation serialized by the file lock; read-only listing deliberately skips the lock and tolerates a stale read. We rejected SQLite (a significant dependency for a key-value store) and in-memory state (it must survive daemon restarts).

tradeoff (disclosed) fs::write is not atomic on every filesystem - a crash mid-write could corrupt the JSON (mitigated by tiny size; effectively atomic under 4 KB on this filesystem). The fix (write-to-temp + rename) is written down. It is the right tradeoff at single-machine scale and the wrong one the moment state goes distributed - and the gap analysis says exactly that.

The same u32 length prefix + payload frames the host↔guest socket, the CLI↔daemon RPC, and the event stream. One primitive to implement, debug, and reason about - and the stated reason we rejected gRPC, server-sent events, and WebSocket for internal transports.

why The framing was originally implemented twice - host side and guest side - with identical constants "by convention only," so a change to either would silently break the other. The durable fix is a single source of truth: a no_std + alloc, zero external dependency crate that compiles for both the host and the cross-compiled guest, carrying a version byte and message tags.

what we chose No framing-level message-type header (payload content determines type); no checksums (the socket runs over virtio, which has its own integrity guarantees); raw bytes, not protobuf or MessagePack ("adds a serialization dependency to the guest binary").

bound it where the bytes enter The parser borrows where it can (&'a str over owned String) and allocates only when it must own the bytes - but a length-prefixed protocol that pre-sizes a Vec from a wire-supplied count is a memory bomb unless the outer frame is bounded. A hard 16 MiB payload cap at the transport layer means a Vec::with_capacity(n) cannot be driven to exhaustion - n is bounded by the frame - and the crate's own length fields are additionally bounded by their wire types via an explicit overflow error variant, added when a review flagged a silent as u16 truncation. Centralize the contract; duplicate nothing; bound the allocation at the place the bytes enter.

why "This circular dependency is untenable for a product whose value proposition is eliminating Docker." So there is none: the kernel builds in CI with a native cross-compiler and is downloaded; the rootfs is built locally with the platform's own ext-filesystem tools (the host OS can't make ext4 natively); the guest is cross-compiled with a Rust-native linker because the host toolchain can't link a Linux ELF.

01 │ [target.aarch64-unknown-linux-musl]
02 │ linker = "rust-lld"          # the host cc/clang cannot link a Linux ELF
03 │ [profile.release]
04 │ lto = true
05 │ strip = true

the rule If your product's thesis is "you no longer need X," X cannot appear in your own build graph. Eat the cross-compilation pain; it is the price of the thesis.

why The kernel, the rootfs, and the CLI have different build environments (a Linux cross-compiler vs the host SDK), different change frequencies, and different consumers. "A single monolithic release would force rebuilding 30 MB+ of rootfs images for every CLI typo fix." Three release tags, installed by one idempotent setup command.

tradeoff (disclosed) A version-sync burden - the kernel/rootfs versions are pinned in four places that must be aligned by hand. An accepted tax, taken in exchange for zero Docker and fast incremental releases, with a drift check planned so the build fails when the four fall out of line. And watch the guard rails you set yourself: the guest binary crept from a historical ~351 KB to ~515 KB as features landed, and now sits just under the 512 KiB CI size gate - the next feature forces either a trim or a guard bump. A guard you quietly slip past is not a guard.

why A cluster of late-phase bugs traced to one anti-pattern: a typed CLI output object that re-serialized only a subset of the daemon's response, silently dropping fields the daemon had computed. The machine-readable output dropped credentials, dropped whole VM classes, reported "unknown" for data it never read.

the keystone Fixing one of them - passing the daemon's payload through verbatim in machine-readable mode - immediately surfaced three previously-invisible failures that had been masked. "The error was hidden until the passthrough was fixed."

the rule For machine-readable output, pass the source-of-truth payload straight through; never re-serialize a lossy subset, because a DTO that drops fields defeats every downstream success check and converts hard failures into green checkmarks. Two corollaries: a tool that exits 0 on "matched nothing" turns a typo into a silent no-op - prefer unambiguous invocation; and capturing a child's output is not capturing its exit status - propagate the real code or every "green" start is unverified.

the house style, in one breath Avoid a dependency for a small problem - flock over SQLite, the platform CLI over an HTTP client, a hand-rolled JSON parser over a full serde stack in the size-budgeted guest, a raw length prefix over protobuf - but know the cost (e.g. a version-sync burden across pinned files) and write it down. The instinct is correct; the discipline is disclosing where it bites.

Standing up an "events plus state plus relationships plus AI agent" product again tomorrow: the stack we would reach for, what we would refuse to run, and the sequence - with the why pinned to every row.

the stack

what we deliberately do not run Every added system is a second durability surface and an operational tax: one more thing that fails independently, one more retry/TLS/auth config, one more place truth can diverge. So: no message bus between two stores that already commit in one Postgres transaction. No dedicated vector database while pgvector with an HNSW index still reaches into the millions of rows. No connection pooler before connection pressure is real. No self-operated workflow engine. No in-process cron library. The vendor is justified by a need you can measure, not by a diagram you want to look complete; the default underneath every row is fewer vendors, and managed over self-operated for anything that is not the differentiator.

the order The order is not cosmetic; several steps are brutal to retrofit, so they go first.

1. Decide the authoritative store, and write the decision down - everything downstream assumes it.
2. Stand up the engines and the canonical event model; normalize on day one, because retrofitting a schema later is brutal.
3. Build the ingestion base-class framework before the second connector - the first defines the contract; the framework pays off from the third onward.
4. SCD Type 2 with a subfield allowlist from the start - the 4.66M-row lesson is far cheaper to learn here than in production.
5. Wire the transactional outbox for the derived graph before there is drift to debug.
6. Build the safe-query layer before the agent - RLS, per-tenant users, fail-closed defaults; the agent is only as safe as the surface beneath it.
7. Hand-roll the agent loop over a single provider wrapper - resist the framework - then add tools one at a time through a base class and a tenant-injected registry.
8. Prompts as templated markdown with an explicit evidence vocabulary.
9. The SSE hub, then context compaction and RAG memory.
10. Durable workflows for anything scheduled or long-lived, and codify the conventions as a pattern library plus hooks the day there is a second engineer.

One artifact chain from convention to enforcement: a pattern library, path-scoped rules, mechanical hooks, and a codified review culture.

the pattern library Per-area pattern files, each headed "confirmed from 50+ PR reviews per repo," are never auto-loaded into context; you pull one in by reference when you touch that area. Every rule cites the PR that taught it. A sampling of the highest-value entries:

• Columnar store: do not use manual PREWHERE; it disables the optimizer and produced a 70x regression when bloom filters and text indexes combined.
• Postgres: FORCE ROW LEVEL SECURITY - without FORCE, table owners bypass RLS and data leaks silently; never CREATE INDEX CONCURRENTLY inside a migration tool that wraps each migration in a transaction.
• Go: bounded retry loops, not recursion - recursive retries on a flaky dependency blow the stack.
• Python: class-var versus instance-var matters for serverless warm starts - on a warm start, tenant B's request can see tenant A's cached credentials.
• React: mutation guards via useRef, not isPending - isPending can flip between render cycles.

path-scoped rules, backed by hooks The must-follow subset of the library is mirrored into path-scoped rule files - condensed, enforceable, loaded only for the files they match - and backstopped mechanically: a git-guard hook blocks push --force, reset --hard, and branch -D main, and an auto-lint hook runs eslint, gofmt, and ruff by path on every edit.

review culture, reduced to what holds The standards of the strongest reviewers are captured in a codified review-culture file: every destructive mutation carries an onError; type-guard narrowing, never the non-null assertion; no nested ternaries; hands-on testing before approval. The point is not the specific rules; it is that the review bar is written rather than tribal, so it applies the same way to every PR and every author.

Anything that must stay identical across two or three repos is an invariant. An invariant without a test is a future incident.

the list The things that must stay in sync:

1. A set of metric keys appears in all three repos - a Python attribute, the Go constants file, the TS metrics module plus DTOs; renaming one means editing three. Gated by an exclusion list and a viability check.
2. The durable-workflow worker/workflow two-module split.
3. A canonical event schema's five core fields, asserted in tests; the status-mapping override is mandatory.
4. A content artifact equals the file that ships: the authoring JSON is byte-identical to the catalog, and the slug must equal the filename.
5. Postgres-authoritative, graph-derived: drift is surfaced as findings, never silently reconciled - argued in full in Data & storage.

01 │ // invariant: a metric key renamed in one repo must be renamed in all three.
02 │ // the canonical set is generated from each repo and asserted equal in CI.
03 │ import { pythonMetricKeys, goMetricKeys, tsMetricKeys } from "./generated";
04 │ 
05 │ test("metric keys stay in sync across the three repos", () => {
06 │   expect(goMetricKeys).toEqual(pythonMetricKeys);
07 │   expect(tsMetricKeys).toEqual(pythonMetricKeys);
08 │ });

The status-mapping invariant gets the same treatment: one test asserting the override is present, so an unknown failure never defaults to success.

Every position on this page restates one of these. The tag names the section where it is argued in full, with its numbers and its scar.

• One authoritative store; everything else is a derived, rebuildable projection. (DATA)
• Structural over remembered - make the wrong thing impossible, not discouraged. (DATA, AGENTS)
• The database is the wall; application validation is for the model, not for security. (AI/LLM, DATA)
• Bound everything; every cap on the list is a scar. (AGENTS)
• Decouple start-the-work from watch-the-work; the caller's request never lives as long as the work. (PLATFORM)
• Exactly-once effect, composed from the data model, never rented from a broker. (DATA)
• A plausible primitive exercised on one path is an assumption, not a feature. (VIRT)
• Leak self-correction hints to the model, never infrastructure. (AGENTS)
• Record the rejected alternative next to the decision, with what killed it; it is why the event bus never got built. (everywhere)
• Real app, real secrets, early; synthetic acceptance finds nothing. (STATE)

Each line is a real finding or a standing refusal: the sin, the better move, and the tag that owns the story.

isolation & trust

• tenantId as a tool parameter - banned from every tool schema. (AGENTS)
• Credentials in the orchestrator's task metadata, full ARNs and tokens readable in its UI - pass a secrets-manager reference and fetch it lazily inside the task, never on a payload or XCom. (DATA)
• No enforced tenant prefix in cache keys, isolation leaning entirely on auth - prefix every key. (PLATFORM)
• eval() on a legacy load path, guarded but live - delete it; dead-code debt is a sharp edge. (DATA)

data & durability

• FINAL to force merges at read time - dedup with LIMIT 1 BY the raw hash instead. (DATA)
• Soft deletes - deletion is real, and the audit row commits in the same transaction. (DATA)
• Acking on receive - ack only after the durable write. (DATA)
• [] and None collapsed inconsistently across extractors, so stale records either never close or mass-close - [] is confirmed-empty, None is unknown, everywhere. (DATA)
• Model-field order silently coupled to columnar column order, corruption untested - assert the alignment in a test. (DATA)
• Caching a negative lookup - only ever cache a hit. (DATA)
• Replaying raw event history to resume a session - inject a summary. (PLATFORM)
• UUID external ids - bigint keys inside, a prefixed nanoid outside. (PLATFORM)

runtime & lifecycle

• A "stopped" resource left in the map, still holding its file descriptors, so the next start fails on a phantom attachment - remove-and-drop, model release on Drop. (VIRT)
• Branching control flow on error.contains("...") - the daemon's retry logic once decided transient-versus-fatal by matching error text - a typed enum the compiler checks. (VIRT)
• set -e in a PID-1 init script, where one non-fatal failure panics the kernel - || true per command; failure is information. (GUEST)
• A loadable module (=m) in a guest with no module loader; it is simply absent at boot - =y for everything boot-critical, and verify the full resolved config. (GUEST)
• An HTTP client that defers Close() inside its own constructor, alive only by a beta-library accident - never close a long-lived client in the factory. (DATA)
• No cache-stampede protection, cold-cache concurrency all hitting upstream - add single-flight. (PLATFORM)
• Three Redis client libraries against one instance, each deriving its own TLS and retry config - consolidate to one. (PLATFORM)
• Connection ready-state in React state - keep it in a ref. (PLATFORM)

build & observability

• A 2.3K-LOC agent service and a 1.6K-LOC memory service, one method mixing cancellation, compaction, heartbeats, token math, and telemetry - split by concern. (AGENTS)
• 35 near-identical entrypoint files across 24 connector folders, a phase-dispatch change becoming a 30-file edit - generate or share more. (DATA)
• An agent-framework SDK in the manifest with zero production imports - remove dead deps in the same PR. (AGENTS)
• A chars-per-token heuristic as the load-bearing context-budget safety net - reconcile against provider-reported counts. (AI/LLM)
• A /health that omits the durable-workflow cluster, so a dead scheduler stays invisible - health-check what you cannot afford to lose. (AGENTS)
• Building a guest kernel up from minimal; there is no stop condition - trim down from a known-good distro config. (GUEST)
• Trusting a primitive tested only host-to-guest; VM-to-VM was assumed until it failed in front of the real app - exercise every primitive on its real path, early. (VIRT)
• A --json DTO that re-serializes a subset of the source of truth, hiding failures behind green checkmarks - pass the canonical payload through verbatim. (STATE)
• Quietly slipping under a size guard you set - trim, or move the guard deliberately and say why. (STATE)
• Overclaiming your own purity when the boundary is lossy - state the seam precisely, with the fix written down. (STATE)