Home · Builds · Finance ETL · Part 4

Finance ETL · Part 4 of 8

Tables you can tear down and rebuild, on purpose.

Every entity gets a table. Every delete is soft and terminal. Every transform is a SQL function that upserts inside one transaction and returns a row count.

Scratch cut · 1:41 Screen recording in psql: select the unapplied responses, call apply_response on one, see the per-entity counts, run the flat transactions view. Demo data, scratch voice, real pipeline. The live recording takes over once it is shot.

The modeled schema

Budgets, accounts, payees, category groups, categories, months, month categories, transactions, subtransactions, scheduled transactions, and their subtransactions. Each row carries deleted and deleted_at; deletes never reverse and timestamps never reset.

One function per entity

SELECT ynab.apply_response(123);   -- dispatches to apply_accounts, apply_transactions, ...
-- each: extract from jsonb -> INSERT ... ON CONFLICT DO UPDATE -> return count
Modeling: raw sheets through the transform, out as tables. One cell fades but stays in place. That is a soft delete.

Where the docs and the payload disagreed

The first transform was written from the API documentation. The real payload has subtransactions as top-level arrays, no split flag on transactions, and currency format as an object. A second SQL file replaced the affected functions. The raw rows were replayed. No data was lost, because of part 3.

Audio · 3 minWhy deletes are soft and terminal
Scratch narration Balance history, audit trails, and the one time a hard delete would have hidden a bug.
Eleven table boxes in tiers joined by dotted foreign keys, each with deleted and deleted_at columns.
Plate IV · Modeling
Free to borrow for your feed · Square edition

Where you are: clean tables you can rebuild from raw at will. Next, they refresh themselves every hour.

Previous · Landing the dataNext · Part 5 of 8The hourly loop, and the four-day outage

Write to me

I built this for a household. Imagine it pointed at your company.

Fractional CTO and CFO engagements start with a short call. No deck required, no pitch either. Bring the number you do not trust.

Tell me what is stuck