Ledger repair after BAN-306 (BAN-348)¶

Between 2026-10-06 and 2026-10-08, 26,676 ledger flows on Lithic debit accounts that were
still wrong after the Galileo flow repair were corrected
in production. The 12,754 accounts the correction left owing money, or that Finance chose to
restore, were then admin credited through Hydra. This page records what was wrong, how it
was corrected, what it did to customer balances and what is still open. The routing that
caused the original contamination is documented in
Lithic → Galileo transaction routing.

One interactive view of the result is attached to this page:

Ledger repair register: every one of the 26,679
planned flows with its tier, the batches that carried it, its balance change, Lithic
tokens, native flows and RUIDs; every one of the 18,556 accounts whose balance changed,
with the admin credit it received and its Hydra link, its balance at the last read and the
communication lists it was on; and the day-by-day, admin-credit, list and closing-check
totals.

The ticket is
BAN-348.

What was wrong¶

BAN-306 put every contaminated GA= flow back to its correct Galileo state. It did not
touch the Lithic side of those transactions, and a register built afterwards found that the
ledger still disagreed with Lithic on 26,679 flows:

category	flows	what was wrong
GA flow: Lithic settlement never debited	25,246	the purchase shows as settled in the app, but the customer's balance never moved
Native flow mis-stated, 22 Apr to 31 May 2026	1,248	settled purchases restated to zero, voided purchases ending on the wrong leg, sign-flipped expiries
GA flow: customer charged more than Lithic settled	160	the customer is owed the difference
Stranded native ASA hold from migration day	21	an authorisation hold that nothing will ever release
GA flow: reservation still open	4	a hold left on the GA flow

The largest group, 24,061 flows, has one shape. The Galileo hold was still open when the card
migrated to Lithic, so the Lithic clearing landed on the GA= flow as a row with no state
change. The be-lithic settlement-report rerun of 2026-04-24 then set the flow to reserved 0
and settled 0, and the Galileo expiry closed it. The customer's history shows the purchase,
but the money was never taken.

How each flow was corrected¶

The mechanism is the one BAN-306 used. Each flow was corrected by one be-lithic
ReconcileTransactionFlowEvent message inserted into platform.MessageOutbox on
prod-lithic. be-lithic replays the token's latest Lithic event, and be-ledger overwrites the
native flow's reserved and settled totals with Lithic's transaction totals, so the native
flow ends exactly at Lithic's final amounts. Where the GA= flow had to change as well, the
message carried a GalileoFlowRestore block that sets the GA flow's reservation to 0 and its
settled amount to the target.

The targets were:

GA flow: reserved 0, settled = what Galileo itself settled, plus anything Lithic events
of other tokens posted on the flow (a refund under its own token, for example);
native flow: Lithic's transaction total.

Before every batch a preflight read Lithic's latest event per token (dbo.LithicEventV2 and
dbo.LithicEvent), the flows' current TransactionFlowCore state, the account balance and
the outbox. A row that disagreed with the plan was excluded with a reason and stayed planned.
After the messages landed, a verify step read every flow again and compared it, and each
account's available balance, with the expected values. A flow counts as fixed only once
verify has passed it.

The flows were grouped into tiers by what the correction does to the customer:

tier	flows	what happens to the customer
credit	331	charged more than Lithic settled; available goes up
debit-full	9,770	a settled purchase never debited; the balance covered it when the tier was built
debit-part	13,700	the same, but the balance covered only part of it
debit-none	2,039	the same, and the balance covered none of it
release	818	no balance change; the flows' reserved and settled split is corrected
asa-hold	21	the stranded hold is released

The debit tiers ran with the account allowed to go below zero. That was decided on
2026-10-05: the admin credits afterwards bring the affected accounts back.

How it was run¶

day (UTC)	batches	messages	returned to customers	taken from customers	net
2026-10-06	10	409	11,473.18	−1,533.90	+9,939.28
2026-10-07	7	8,635	0.99	−155,396.57	−155,395.58
2026-10-08	13	17,660	217.96	−564,074.33	−563,856.37
total	30	26,704	11,692.13	−721,004.80	−709,312.67

Each tier started with one-message pilots and grew from there (1, 1, 10, 100, then up to
3,000 per batch). Messages were scheduled 3 s apart at first, then 1 s, then 0.5 s. The
be-platform outbox publishes every due message back to back, so the ledger receives them at
the scheduled pace, and 3,000 messages at 0.5 s arrive over about 25 minutes.

There are more messages (26,704) than flows (26,676 verified) because 28 flows were carried
by two batches: 27 follow-ups that needed a second message, and one flow debited 0.99 in one
batch and credited 0.99 in another.

What changed along the way:

The GA target rule was corrected on 2026-10-07, after GA=258192772 showed a refund
posted under its own token on a GA flow. Every GA flow was audited against its
TransactionCore postings; 20 flows carried such a refund, 19 were corrected in the plan
before they ran and one was fixed with an extra 0.99 message.
The plan was rebuilt each morning from live state (TransactionFlowCore and
AccountBalance) instead of the September register. 773 flows already held Lithic's totals
by then and became 0.00 releases, and the debit tiers were re-assigned by that day's
balances.
Two flows ran at Lithic's latest total (GA=258937663, GA=258893851), because a Lithic
event newer than the plan had changed the amount.
8 of the 19 stranded holds had already been cleared on a GA flow by the old routing.
Settling only their native flow would have debited the customer twice, so each of those
messages also restored the GA flow to 0/0.

Outcome¶

26,676 of 26,679 flows verified, 0 failed. The fix returned 11,692.13 to customers and
took 721,004.80, a net of −709,312.67, and changed the balance of 18,556 accounts.

Three flows are held and were not sent:

GA=259260782 and GA=259278006: the native flow already held Lithic's final amount at the
first preflight, because the settlement had been debited after the register was built.
One stranded-hold row whose Lithic transaction, 04722cfd-c0a2-4f9f-99ec-795087ab61f0, is
the same as that of GA=259232482, which was fixed by its own row.

Admin credits¶

The credit rule was set before the run:

Restore: an Active account on Finance's write-off list was credited what the fix took
from it, net of what the fix returned to it.
To zero: every other account the fix left negative was credited up to zero, and only by
the part of the negative the fix caused. Blocked and Terminated accounts on the write-off
list were credited to zero in the same way, through a forced credit.

Credits were issued one at a time through Hydra, with a fresh AccountBalance read before
each chunk. An account credited to zero was skipped if it was no longer negative, and
no account was ever credited twice.

day (UTC)	rule	accounts	amount
2026-10-07	restore	1,319	17,857.28
2026-10-07	to zero	17	57.28
2026-10-08	restore	9,733	403,780.01
2026-10-08	to zero	1,422	92,776.40
2026-10-08	to zero, Blocked or Terminated (forced)	263	10,541.39
	issued	12,754	525,012.36
2026-10-07	debited back (over-credit)	38	−1,017.19
	net to customers	12,754	523,995.17

Finance's own total for the write-off list was 536,258.61, computed from an earlier balance
snapshot.

The over-credit. The first wave computed the restore from the fix's debits alone. 39
accounts on which the fix had also credited money received 1,019.52 too much; one account
whose fix was +300.00 and −300.00 was credited 300.00. 38 of them were debited back the same
afternoon through Hydra administrative debits, each only after a fresh balance read showed
the account still covered it. The 39th keeps 2.33, because its balance was −15.09 for
reasons unrelated to the fix. The rule has netted the fix's credits since, every planned
amount was checked against an independent recompute before the second wave, and no credit
has been debited back since.

Not credited:

13 Blocked accounts on the write-off list were refused by be-transactions
(AccountBlockedException) on 2026-10-07. None was negative.
12 accounts planned for a credit to zero were no longer negative when the credit ran on
2026-10-08 (252.14).

After the credits (balances read 2026-10-08 15:00 UTC), 110 credited accounts are still
negative, −5,091.30 together: 97 restored, 12 credited to zero, and 1 credited and fully
debited back. A restore returns what the fix took, and a credit to zero covers only the part
of the negative the fix caused, so what remains was there before the fix or is the
customer's own activity.

The second wave ran as three parallel streams over disjoint account files, about two credits
per second together. minority-ledger-service, minority-hydra-api, minority-hydra-service and
minority-transactions-service showed no latency or error change.

Customer communication¶

Accounts whose balance changed were put on lists for the customer message, one list per
message:

sent	list	accounts
2026-10-06	balance increased only	209
2026-10-07	list 1, restored to the balance before the fix	1,286
2026-10-07	list 2, negative, credited to zero	16
2026-10-07	list 3, no credit	5,189
2026-10-07	list 4, correction to the 2026-10-06 message	42
2026-10-08	list 1, restored to the balance before the fix	9,733
2026-10-08	list 2, negative, credited to zero	1,683
2026-10-08	list 3, no credit	427

That is 18,543 distinct accounts; the 42 corrections had already been on the 2026-10-06 list.
An account already messaged was left off later lists unless the fix debited or credited it
again after its message. Two such accounts, messaged on 2026-10-07 as not negative and
credited to zero on 2026-10-08, form the 2026-10-08 list 4, which had not been sent when this
page was written. 13 accounts whose balance only went up, through the hold releases of
2026-10-08 (158.03 together), were on no list.

Closing checks¶

GA flow audit, every TransactionCore posting of every GA flow read again after the
last batch: all 25,421 GA flows, the 2 held ones among them, are at reserved 0 and settled at
their target. TransactionFlowCore agrees with the last posting on every flow, and no flow
has more than one TransactionFlowCore row. Native flows are not part of this audit; the
verify step checked each one against Lithic's latest event when its batch landed.
Over-credit recompute, after the second wave: every issued credit recomputed from the
applied batches. The outstanding over-credit is unchanged at one account, 2.33.
Lithic settlement file (dbt_semantic.fact_lithic_debit_card_transaction): 1,477
distinct tokens were checked: the 1,462 not matched to a settlement file earlier and the 17
tokens of the 14 flows Galileo settled, two of them in both groups. Lithic's settled total
equals the plan's native target on 1,130 of the 1,226 plan rows the file covers, and no row
differs. The other 96 rows carry 198 tokens absent from the file, each with a Lithic final
of 0.00 (expired or declined, never settled), which the file does not carry.
Processor callbacks (dbt_semantic.fact_processed_transaction): be-lithic's reconciled
callbacks after the replay equal the plan target on all 1,226 rows.
Galileo-settled flows: for the 14 flows Galileo settled, Lithic's settled total equals
the register's Lithic settled amount on all 14.

Things to know before the next repair¶

A compensation must net the fix's own credits. Summing only what the fix debited
over-credited 39 accounts. Compute the credit from everything the fix did to the account,
and recompute it independently from what was actually applied before issuing it.

A GA flow can be restored once per three months. The GalileoFlowRestore record's
idempotency key is derived only from GA-REPAIR:GA=<flow>, and be-ledger compares a key it
has seen in the last three months field by field. A second restore of the same flow with
different amounts is rejected with 400 "Transaction data does not match the existing
record." On 2026-10-08 an attempt to move 20 refunds off their GA flows failed this way on
every message, after four deliveries each, and nothing was written. Those 20 flows keep the
refund on the GA flow and the purchase on its native flow, which is net correct per customer.
Separating them needs a be-lithic change that puts the previous and target amounts into the
key. The dead-lettered messages must not be resent.

A flow can be carried by more than one batch. Anything that totals per flow (the
admin-credit plan, the lists, this page) must sum every applied batch for the flow. Keeping
only the last one planned 0.99 too much for one account; the independent recompute caught it
before the credit ran.

Long runs against Azure SQL need splitting. The access token is taken once per run and
lasts about an hour; a longer run fails with
Login failed for user '<token-identified principal>'. Splitting the accounts into disjoint files and running
them as parallel streams kept each run short. Disjoint files are what prevents a double
credit.

Lithic events live in two tables. Since 2026-09-29 new events are written to
dbo.LithicEventV2; older ones stay in dbo.LithicEvent, which BAN-392 plans to drop. Any
replay of an April token has to read both while both exist.

Still open¶

What is still wrong in the ledger, with what fixing each item takes, is tracked on
Ledger leftovers after BAN-348.

The 20 refunds that stay on their GA flows, described above.
Out of scope for this repair: 43 credit-program rows (a different card program), 1,888
stale holds (BAN-238) and 686 reserve mismatches from the BAN-348 register.