now

Work

The replication-lag investigation is closed and written up. The fix that mattered turned out to be a pgBouncer connection-pooling issue, not the reconciliation cron. The write-up is in the index.

Current focus is async processing. Three batch operations that were previously running synchronously on the request path are being moved into job queues backed by Redis and Graphile Worker. The queuing itself is straightforward. The harder problem is observability: knowing when a background job fails silently is more difficult than knowing when a request fails, because there is no caller waiting to receive an error.

Reading

Kuhn, The Structure of Scientific Revolutions. I have read it before but it reads differently after spending a year inside a system where the dominant mental model of a problem turned out to be wrong.

David Deutsch, The Beginning of Infinity. Slower going. The epistemological claims are interesting; the detours into political philosophy are less so.

Thinking about

What the right failure model is for a queue worker that processes financial events. Retries are not free. An idempotent operation can be retried safely. A non-idempotent one cannot. The line between them is often less clear than the code suggests.

Whether the telemetry work should have been built before the queue workers, not after. Same lesson as the replication-lag investigation. I keep learning it.