Home · Builds · Finance ETL · Part 3

Finance ETL · Part 3 of 8

Store every response exactly as it arrived.

The YNAB API gives you the whole budget with a server-knowledge cursor. Land each response as raw JSON in an append-only table, and you can rebuild everything downstream whenever you like.

Scratch cut · 1:28 Screen recording: the curl call with the token redacted, the JSON payload scrolling past, the INSERT returning an id, then a SELECT on the landing table showing the new row. Demo data, scratch voice, real pipeline. The live recording takes over once it is shot.

The endpoint that matters

One request returns the whole budget: accounts, payees, categories, months, transactions, scheduled transactions. Pass last_knowledge_of_server and you get only what changed since. Two requests an hour is well inside the limit of 200 per hour.

GET /v1/budgets/{budget_id}?last_knowledge_of_server={cursor}
Authorization: Bearer $YNAB_TOKEN

The landing table

ynab_raw.api_responses (
  id, endpoint, budget_id,
  request_knowledge, server_knowledge,
  payload jsonb,
  fetched_at, transformed_at
)
A response lands: the sheet leaves the terminal and stacks into the raw table, on top of everything that came before.

Why raw first

Three transform bugs were fixed in the first week. Not one required re-fetching the API. The fix was a new SQL function and a replay of the rows already on disk.

Audio · 3 minHow the server-knowledge cursor works
Scratch narration What the number means, why it is stored per endpoint, and what a cursor of zero triggers.
A terminal labeled sync.sh and a box of sheets labeled ynab_raw.api_responses, append only, with one sheet entering.
Plate III · Landing the data
Free to borrow for your feed · Square edition

Where you are: every response on disk, forever. Next, it becomes tables you can query.

Previous · The boxNext · Part 4 of 8Modeling: where the docs and the payload disagreed

Write to me

I built this for a household. Imagine it pointed at your company.

Fractional CTO and CFO engagements start with a short call. No deck required, no pitch either. Bring the number you do not trust.

Tell me what is stuck