Galileo flow repair (BAN-306)

Between 2026-09-03 and 2026-09-09 all 30,250 contaminated GA= transaction flows were
repaired in production. This page records what was wrong, what was done about it, and
what it did to customer balances. The mechanism that caused the contamination is
documented separately in
Lithic → Galileo transaction routing.

Two interactive views of the result are attached to this page:

• Galileo flow register — every one of the 30,250
  flows with its tier, ledger deltas, status, verdict and the ledger row ids the repair
  created. Searchable by flow, account, Lithic token, native flow, RUID or ledger row id.
• Customer impact lists — the 4,604 accounts whose
  balance the repair changed, split into the six communication lists, with account
  status, prior disputes and admin-credit detail.

Both are also attached to
BAN-306.

What was wrong

be-lithic could divert a Lithic transaction event to galileo-service instead of posting
it to the ledger, matching a Galileo flow by network transaction id. Network ids are
not unique across schemes, so a Lithic clearing could match an unrelated Galileo
transaction — often one that had settled weeks earlier and belonged to a different
customer — and the flow upsert then overwrote that flow's AccountId.

The result was 30,250 GA= flows carrying Lithic events that did not belong to them.
The ledger state on those flows no longer described the transactions attached to them,
and in the cross-customer cases the flow had been reassigned to the wrong account
entirely.

The routing was removed from be-lithic on 2026-08-12, which stopped new contamination.
It did nothing about the flows already in production, which is what BAN-306 covered.

The shapes the contamination took

The flows did not all need the same treatment. They were classified by what the ledger
would have to do to put them right:

tier	flows	what was wrong
A-zero	11,662	flow reads zero, no balance correction needed
zero-other	10,486	net zero across the flow, but individual legs move money
debit	3,735	settled amount to be restored to the Galileo flow
B-net-zero	2,972	multiple legs cancelling out
credit	1,249	credit leg to be restored
mastercard	146	matched by switch_serial_number rather than Visa transaction_id

What was done

Each flow was repaired by inserting one or more be-lithic ReconcileTransactionFlowEvent
messages into platform.MessageOutbox on prod-lithic. One message per Lithic transaction
token on the flow, scheduled one per ScheduledTime second, with the messages of a single
flow spaced ten slots apart so they could never be applied out of order.

Where the Galileo flow's own settled amount had to be put back, the last message of the
flow carried a GalileoFlowRestore payload. That produced a restore record in
dbo.TransactionCore of type 7, with ExternalTransactionId = GA-REPAIR:<flow> and an
idempotency key derived as uuid5(NAMESPACE_URL, "galileo-flow-restore/GA-REPAIR:<flow>"),
so a replay could not double-apply it. The native Lithic flow each message created or
updated is identified by uuid5(NAMESPACE_URL, "minoritylithic/<token>").

Every write moved the account balance by

available' = available − (flow.R + flow.S) + (rec.R + rec.S)

where R and S are the cumulative reserved and settled amounts on the flow. Each flow's
net effect was computed before it was sent and checked against the ledger afterwards.

How it was run

The repair was sent in waves, each verified against prod-ledger before the next was
prepared. Concurrency was raised deliberately and the load measured at each step rather
than assumed.

date	tier	flows
2026-09-03	A-zero pilot	1
2026-09-04	A-zero	4,608
2026-09-07	A-zero	7,053
2026-09-07	B-net-zero	2,972
2026-09-08	credit	1,249
2026-09-08	debit	3,735
2026-09-08	mastercard	146
2026-09-09	zero-other	10,486

The zero-other tier, the largest, was itself sent in six waves escalating from a single
pilot flow to eight concurrent streams carrying 21,188 messages.

At peak — eight streams, about eight messages per second — be-ledger's call rate rose
from a 2,237–3,083 per minute baseline to 7,545–9,209 per minute, a 3.4× increase, with
p95 latency unchanged at 31–32 ms and a maximum of 243 ms. No call exceeded one second at
any point in the repair. The outbox publisher emits one row per ScheduledTime second, so
concurrency scales the rate linearly and cannot burst.

Across the whole zero-other tier there were two error-class log lines, both the same
routine FirebaseDatabase.PUTRecord call on be-transactions, unrelated to the repair.

Outcome

30,250 of 30,250 flows verified, 0 failed. Every flow's measured available and reserved
delta matched the value predicted before it was sent, no flow was written out of its
planned order, and no native flow received a duplicate row.

What it did to balances

The repair changed the balance of 4,604 accounts: 37,619.86 in increases and
82,079.43 in decreases, a net of −44,459.57. The decreases are money that had been
credited to the wrong account by the contamination and was returned to the flow it
belonged to.

group	accounts
balance increased	1,237
decreased into negative, admin credited back to zero	2,447
decreased but stayed non-negative	873
cancelled or closed, no communication	27
decreased, already negative before the repair	20
carried a prior dispute, flagged for investigation	85

2,466 accounts were admin credited back to zero, totalling 59,141.55. These were
accounts the repair pushed below zero through no fault of the customer. Credits were
issued one at a time through Hydra with a fresh ledger balance read per chunk, skipping
any account that was no longer negative and never crediting the same account twice. The
full set reconciles exactly against the warehouse:

	issued	dbt_semantic.fact_processed_transaction
credits	2,466	2,466 rows, 2,466 distinct transaction_id
total	59,141.55	59,141.55
type	AdminCredit	AdminCredit / AdministrativeAdjustment

AdminCredit rows carry account_id = 0 in the warehouse, so the reconciliation is by
transaction_id. The table has no created_at; the time column is
local_transaction_time, and it is null for these rows.

The subscription-fee interaction

Reconcile legs reach be-feesandpromotions. TransactionEventHandler is registered for
both TransactionCompletedEvent and TransactionReconciledEvent
(FeesAndPromotionsServiceModule.cs:158-159), and its gate at line 49 fires when

evt.AvailableBalance > 0 && (evt.Amount >= 0 || evt.AvailableBalanceChange > 0)
    && evt.MajorityAccountType == AccountType._00_CheckingPrimary

Zero-amount legs never opened the gate. evt.Amount carries the original card amount,
which is negative, and AvailableBalanceChange is zero. Measured on 5,552 zero-amount legs:
all reached be-feesandpromotions, none charged. This was confirmed again on a fresh flow in
the zero-other pilot.

Credit legs did open it. A flow whose credit leg lands before its debit leg exposes a
transient surplus for a few seconds, and the fee is charged against a live balance read
(SubscriptionService.cs:385), not against the event payload. Because the flow returns to
its starting balance, an account charged during that window settles below zero.

Two populations hit this:

• 8 accounts whose transient peaked between +0.58 and +2.00, below the 3.99 fee floor.
  All eight took the insufficient-balance path: a ChargeStatus.Failed row was written and
  InsufficientBalanceSubscriptionSuspendedEvent published, but no money moved — no
  DebitFunds, no external card charge, and no scheduled pause cleared. Because
  ChargeGetLastSuccessful.sql:10 filters Status = 1, those Failed rows stay invisible to
  IsFeeAlreadyCharged and the users retry on their next genuine balance increase.
• 101 accounts whose transient cleared the floor, held back from the main waves and sent
  last. Of these, 33 were charged (230.67), 69 were already charged for the current period,
  5 had insufficient balance and 3 were blocked. The 33 charged accounts ended negative,
  −200.93 in total, and were left in that state: the fee was overdue in every case, so it is
  a real debt collected earlier than it otherwise would have been.

A log line that reads wrong

SubscriptionFeeChargedEvent is published for any charge status except Unknown
(SubscriptionService.cs:1371), including Failed. Seeing it in the logs does not mean a
fee was collected — it is a data-warehouse notification. The line that means money actually
moved is ITransactionService.DebitFunds.