Skip to content

Adding a Lithic card program

Status: living document. Add to it whenever you set up, change or debug a Lithic card program, including things learned on earlier programs.

This page lists everything a new Lithic card program needs across our services: code, configuration, secrets, Lithic-side setup, webhooks, deployment order and the traps we have hit. Rain (CardProgramType.LithicRainCrypto = 5, BAN-270) is the worked example, compared with debit (LithicDebit) and the two LoanPro credit programs.

Read from origin/master on 2026-10-30: be-bank-platform 6c5451564, be-lithic f12f65a6, be-cde-carddata a33d91110, be-wallet 6db2b374e, be-crypto c977045, be-transactions 1bfebc876, be-cashback a4c33e4, be-configuration 55903379, be-cde-configuration b589e2d, be-cicd 5217195, be-platform 25112856a, platform-infrastructure 19e3ca2. File and symbol names below are from those commits; check them against master before relying on them.

Rechecked on 2026-11-11 where a later PR changed a fact, at be-lithic 28bc7fd8, be-wallet a263ddc9a, be-cde-carddata 071f4dc9d, be-bank-platform 782e30a2b, be-crypto abe28f3, be-configuration 41282d13 and be-cde-configuration f27065f. Those facts name the PR and its merge date.

Summary

  • Adding the enum value compiles everywhere. Every switch on CardProgramType has a default arm, so each place you miss fails at runtime or silently treats the card as debit. The worst one is be-wallet LedgerAccountHelper.ResolveCardProgramType, whose default is LithicDebit.
  • A program is identified three ways today: by AccountType plus LedgerType in be-wallet and be-transactions, by CardProgramType in be-lithic and be-cde-carddata, and by the Lithic program token in be-cde-carddata. A new program on a new AccountType works with today's code. Two programs on one AccountType need the deferred Program table (see the 2026-10-12 program architecture decision).
  • Secrets are read from Key Vault only at pod startup. A secret added or rotated later is not seen until the pod restarts.
  • lithic-config.json NetworkTransactionGateway is set per program since 2026-11-05 (be-lithic#733, be-configuration#3463): CardPrograms.<Program>.Enabled and CardIds. A program whose money lives outside be-ledger, like Rain, must stay enabled; switched off, its flows go to be-ledger, which does not hold its balance.
  • Calls between services are denied unless the platform-infrastructure network policy allows them. A missing rule gives no error: each attempt times out after about 10 seconds and the target logs nothing.

How a program is identified

Where Key Set by
be-wallet, be-transactions AccountType (+ LedgerType) Account opening
be-lithic CardProgramType Webhook URL segment; LithicAccountHolder.ProgramType; CustomData["CardProgramType"] from wallet (debit and credit path) or ProgramId on gateway requests (Rain path)
be-cde-carddata CardProgramType, or the Lithic program token AccountAndCardAddedEvent.ProgramId; Lithic's card_program_token, stored as MajorityCard.CardProgramId
Warehouse Enum name or number be-transactions CardProrgamId (name), be-lithic LithicAccountHolder.ProgramType and SettlementReport.CardProgramType (int), card_program_token in dbt_stage.lithic_request_event

Decisions to make first

  1. A new AccountType, or a shared one. A shared one needs the Program table first.
  2. Where the account and the money live: the Lithic account plus be-ledger (debit pattern: CustomData["CardProgramType"], the default network gateway, MajorityLedgerProcessorGateway), or the program's own service behind the account, card, ledger-processor and network-transaction gateways (Rain pattern, be-crypto). The program's service then needs network-policy rules in both directions with minority-lithic-service (see Network policies).
  3. Its own Lithic API key, or a shared one. Credit secured and unsecured share one key, so the settlement response factory always stamps LithicLoanproCreditUnsecured on both.
  4. Funding type _00 or _01 (be-lithic funding switch). Rain is network credit but uses _00.
  5. Issuance at account opening (AccountFeatures.IssuesCardAtAccountOpening), or by card request behind a policy, a flag and an allowlist (Rain).
  6. Plus tier; physical cards (then card art, product ids and expedited ids); non-personalized cards; ATM cash load, balance inquiry and the money-load MCC fallback; Kard cashback.
  7. Whether Lithic sends the program settlement reports. be-lithic fetches them through the client factory since be-lithic#733 (2026-11-05), so no separate arm is needed.
  8. The KYC workflow. Every program is sent as KYC_EXEMPT / PrepaidCardUser; confirm Lithic accepts that for the program.
  9. The webhook URL segment: enum name or number. The number survives a rename.
  10. Which card operations must work at launch: replacement, reissue, lost or stolen, provisioning, restrictions, PIN.
  11. An enum name of 20 characters or fewer (be-transactions CardProrgamId is nvarchar(20)), and the warehouse mappings.

Lithic side

Get these from Lithic for each environment (sandbox for stage, production for prod):

Item Notes
Lithic account and API key Rain is a separate Lithic account with its own key
Program token Must be unique across programs and exactly the token Lithic returns on a card
BIN Must match the be-wallet Bank card row (Rain: 45492432)
Event subscription and its signing secret Registered by us through the API: Registering webhooks with the Lithic API
Auth stream (ASA) responder and its secret In production, Lithic's Implementation Manager must enable ASA for the program first
Digital card art tokens, English and Spanish Sent for virtual and physical cards
Card design product ids, en and es, standard and expedited Physical cards only
Settlement report access Must be reachable with the program's key

Webhooks in be-lithic

  • Routes: POST v1/webhooks/{cardProgramType}/Event (IEventController) and POST v1/webhooks/{cardProgramType}/Authorization (IAuthController). The segment is parsed ignoring case, by enum name or number. An unknown value parses to 0 and gets 401. Only the URL decides the program; EventDispatchAction stamps it downstream.
  • URLs for Rain:
Environment Event Authorization
Stage https://stage-lithictesting.minority.com/v1/webhooks/lithicraincrypto/Event https://stage-lithictesting.minority.com/v1/webhooks/lithicraincrypto/Authorization
Prod https://lithic.minority.com/v1/webhooks/lithicraincrypto/event https://lithic.minority.com/v1/webhooks/lithicraincrypto/authorization
  • Signature: HMAC-SHA256 over {webhook-id}.{webhook-timestamp}.{body}, with the whsec_ prefix stripped from the secret and the timestamp within 5 minutes. A missing secret for the program fails closed with 401.
  • EventDispatchAction handles only card_transaction.updated, tokenization.two_factor_authentication_code, tokenization.updated and settlement_report.updated. Every other event type gets 200 and is only published to the warehouse (lithic-request-event).
  • To check that a webhook arrived on stage, search Elastic:
FROM minority-backend-stage
| WHERE @timestamp >= NOW() - 1 hour
  AND kubernetes.container_name.keyword IN ("minority-lithic-api", "minority-lithictesting-api")
  AND (object LIKE "*aincrypto*" OR object LIKE "*ainCrypto*" OR object LIKE "*webhooks/5/*")
| STATS n = COUNT(*) BY kubernetes.container_name.keyword, logLevel.keyword

Code changes per repo

be-bank-platform

  • Add the value to Minority.Wallet.Domain.Contract/Card/CardProgramType.cs and publish the packages.
  • Every consumer must move to that build of Minority.Wallet.Domain.Contract: be-lithic (Minority.Lithic.Client, Api.Contract, Shared.Contract), be-cde-carddata (Service.Contract), be-wallet, and be-crypto if involved. Enums travel as numbers, so an old consumer gets an undefined value and hits a throwing default arm, for example carddata LithicCardApiResolver.Resolve.
  • The gateway contracts already carry int ProgramId (OpenAccountRequest, CreateCardRequest, CardProcessorGateway.Contract); they need no change.

be-lithic

Clients:

  • Host constants in Minority.Lithic.Client.Contract/Host.cs and Minority.Lithic.Cde.Client.Contract/Host.cs.
  • Six marker interfaces per program: ILithic<P>{Accounts,AuthRules,Transactions,Simulations,SettlementReports}Api and ILithicCde<P>AccountsApi.
  • Registration in LithicClientModule (named HttpClient, Authorization header from the key) and LithicCdeClientModule (ships as Minority.Lithic.Cde.Client for carddata).
  • <P>ApiKey on both LithicCredentials classes. If missed, startup works and the first call throws FormatException (null header).
  • LithicClientFactory: field, constructor parameter and arm. An unmapped program throws BadRequestException("Unsupported card program type") for every caller.
  • LithicSettlementsResponseFactory: one subclass per program, which stamps the program on lithic-request-event.

Webhooks:

  • Two properties and two switch arms in LithicWebhookCredentials (default null). If missed, every webhook for the program gets 401.

Card creation:

  • Token lookup in CardProgramTypeConfigurationExtensions. If missed: ArgumentException("Unknown card program type") at card creation.
  • LithicCardIssuerGateway (debit and credit path): takes the program from wallet's CustomData["CardProgramType"], sends KYC_EXEMPT / PrepaidCardUser, persists LithicAccountHolder.ProgramType, and has a funding switch whose default throws BadRequestException.
  • LithicCardProcessorGateway (gateway path, Rain): ToProgramType rejects undefined values and Galileo; EnsureAccountHolder throws when the holder is on another program; the card is created OPEN and stored through carddata AddCard.
  • Card art and product ids (LithicConfiguration) are read by both gateways. Missing art means Lithic's default art; a missing product id falls back to "1". Neither raises an error.
  • The design language comes from each request, not from the stored holder (be-lithic#747, 2026-11-11): Profile.Locale on card creation, and the top-level Locale on convert to physical, reissue and renew (be-bank-platform#853, in Minority.CardGateway.Contract and Minority.CardProcessorGateway.Contract). A null locale gives the English design, so the caller sends the holder's profile with every card create (be-bank-platform#854). [Locale] rejects anything that is not a culture name, for example es_MX, with 400. LithicAccountHolder.Locale is no longer read.
  • Both gateways store the wallet account id in LithicAccountHolder.AccountId on each holder they create; the issuer path does so since be-lithic#739 (2026-11-06) and returns 400 when CustomData AccountId is missing. The filtered unique index UX_LithicAccountHolder_AccountId allows one holder per wallet account, so one wallet account cannot hold cards on two Lithic holders.

Transactions:

  • LithicServiceModule keys the network transaction gateway by program: Rain to ICryptoNetworkTransactionGateway (be-crypto), both LoanPro credit programs to ICreditNetworkTransactionGateway (be-credit, since be-lithic#728, 2026-11-03), debit to DefaultNetworkTransactionGateway (be-ledger).
  • NetworkTransactionRouter.GetGateway returns the program's gateway only when its NetworkTransactionGateway.CardPrograms entry in lithic-config.json has "Enabled": true and its CardIds is empty or holds the wallet card id. Otherwise, and for a flow without a wallet card id, it returns null and the inline be-ledger calls run. The code default enables LithicRainCrypto for every card; every other program is off unless the deployed config enables it. The global Enabled and EnabledForCardIds keys are no longer read (be-lithic#733 and be-configuration#3463, 2026-11-05). An enabled program with no keyed gateway throws InvalidOperationException: the ASA fails, and callbacks retry and then dead-letter. Callers: AuthService, TransactionCallbackService, SettlementDetailBroadcastAction.
  • A program whose money lives in its own service must stay enabled. Switched off, its flows go to be-ledger, which does not hold its balance, and its open flows are stranded (the remark on INetworkTransactionRouter).
  • The gateway call on the ASA path uses the platform's $outgoing-request resilience policy (be-configuration global-resilience-config.json: 5 retries, up to 10 seconds apart). A target that does not answer is retried for over a minute; on 2026-10-30 one Rain ASA was retried for about 80 seconds, long after Lithic had declined it.

Settlement, expiry and reconcile:

  • SettlementReportUpdatedEventHandler fetches the report through ILithicClientFactory since be-lithic#733 (2026-11-05), so the factory arm covers a new program. Before that it threw ArgumentOutOfRangeException for anything but debit and credit.
  • SettlementDetailBroadcastAction reads each detail's balance from the program's network transaction gateway when the router returns one, and from be-ledger BalanceInquiry otherwise. The balance is the one at publish time, not at settlement.
  • The settlement report moves no money. The handler publishes ProcessorCallbackSettlementReportingEvent, which only be-auditing consumes and sends to the warehouse; be-transactions does not subscribe. Holds settle from transaction events instead (for Rain, be-crypto RecordEvent). Lithic's documentation says settlement_report.updated is sent only for programs that settle directly with the networks.
  • ExpireReservationMessageHandler and ReconcileTransactionFlowHandler already use the factory and dead-letter an unknown program.
  • No migration: AccountHolder.ProgramType is INT and SettlementReport.CardProgramType is TINYINT.

Debit-only and credit-only branches to decide on:

  • ATM cash load (AuthService, LithicTransactionTypeParser), the money-load MCC fallback (AuthService), and balance inquiry are debit only.
  • Credit-secured RDF handling in LithicTransactionTypeParser.
  • Non-personalized cards are hard-coded to LithicDebit (ConnectNonPersonalizedCardAction, LithicNonPersonalizedCardGateway, both card gateways).

Local config and tests:

  • Minority/app-config/global-cardprogramtype-config.json and lithic-config.json; user secrets id be-lithic.
  • LithicBaseRegressionTest copies a hard-coded list of secret names and resolves only the debit clients.
  • Tests to extend: LithicClientFactoryTest, SignatureTest, LithicIntegrationTest, NetworkTransactionRouterRegistrationTest, the settlement, expiry, reconcile and callback tests, and LithicTesting's MockBuilder, LithicTestingIntegrationTest and TransactionSimulateServiceTest.

be-cde-carddata

  • LithicCardApiResolver: program-to-client arm and constructor parameter. If missed, the card's PAN, CVV and expiry are never stored: the card-added handler retries and dead-letters.
  • Token-to-program lookup in the same class: a null token resolves to Galileo debit, the legacy prod debit token is hard-coded, otherwise the first config entry with a matching token wins, and no match throws. If the token is wrong, PIN set, SaveCardDataIfMissing, verify retry, expiry update and every LithicProxyService(V2) call throw.
  • AccountAndCardAddedEventHandler uses evt.ProgramId ?? CardProgramTypeFallback.FromIsCreditCard(...). If wallet leaves ProgramId null, carddata calls the debit or credit client, gets a 4xx from Lithic, and dead-letters.
  • CardUpdateInformationMessageHandler (physical card shipped) and the missing-row branch of MajorityCardService.SaveCardDataIfMissing take the program from wallet's GetCardById (Card.ProgramId) since be-cde-carddata#327 (2026-10-30), no longer from the funding type. Wallet fills it from LedgerAccountHelper.ResolveCardProgramType, so the wallet arm decides it. A missing or undefined program gives 403 in SaveCardDataIfMissing and a dead letter in the handler, without a Lithic call.
  • Wiring in CardDataServiceModule; the chart mounts global-cardprogramtype-config. Local runs and the test host read Minority/app-config/global-cardprogramtype-config.json, which needs the program with a placeholder token (Rain: ...-3000-...).

be-wallet

  • LedgerAccountHelper.ResolveCardProgramType: add the (AccountType, LedgerType) arm. The default is LithicDebit, so a missed arm sends the card to the debit token and client and puts the wrong ProgramId on events and card reads.
  • The other switches in LedgerAccountHelper, each needing a line for a new AccountType or LedgerType: ResolveLedgerType (throws NotSupportedException), GetLedgerProductId, ShouldCallExternalProvider, ShouldCreatePrn, ShouldCreateInternalLedgerAccount, ShouldSyncProfile. Missing arms give 500s at account opening, profile sync and GetCardById.
  • AccountFeatures: CanHoldCards (403 on card creation without it), SupportsPhysicalCards (403 on a physical order), IssuesCardAtAccountOpening, CardsRequirePlusTier, ToCardFundingType. _05_Crypto can hold cards but has no tier requirement, no physical cards and no card at account opening.
  • Bank rows and PRN ranges, keyed by (LedgerType, AccountType, PrnVersion). PRNs come from PrnGetNext, which throws 50000 when the range is missing and 50001 when it is used up. Rain: migration 229_Bank_AddCryptoPrnRange.sql, one range 771000000000 to 771999999999 shared by accounts and cards, a card Bank row 'Rain' with BIN 45492432, LedgerType 6, AccountType '05'. Missing rows give "No bank found", "No bank is configured" or "missing a PRN".
  • CreateCardAction refuses an account without Account.Prn, and LedgerGateway.CreateVirtualCard refuses a missing account PRN for gateway accounts. AccountServiceV2.GetAccountInfoByCardId throws for it too, and be-lithic's TransactionCallbackService calls it for card transaction events. On 2026-10-30 every active prod account without a PRN was a legacy I2c or Internal account with only terminated ledger rows, and every crypto account had one.
  • Gateways in BankServiceModule: ICardIssuerGateway keyed by LedgerType; IMajorityAccountGateway and ICardGateway keyed by AccountType (_05_Crypto goes to be-crypto). Without a keyed account or card gateway the legacy issuer path runs.
  • Card policy per AccountType (be-wallet#1352, 2026-11-02): ICardPolicy registered in CardPolicyModule, with AllowsCardRequest(accountHolderId) and IssuesCardsAutomatically. The second decides both the card at account opening and the replacement on cancel or lost or stolen; it returns AccountFeatures.IssuesCardAtAccountOpening() for the policy's type. An unregistered type gets DefaultCardPolicy (neither), and CardPolicyModuleTests.Load_RegistersACardPolicyForEveryAccountType fails until a new AccountType has a policy.
  • Card request (opt-in issuance): POST v3/accounts/{id}/cards, RequestCardAction, table CardRequest (migration 230). A policy that does not allow the holder gives 403 "is not enrolled for a card". CryptoCardPolicy (formerly CryptoCardEnrolmentPolicy) lets everyone in when the flag is on and the allowlist is empty or null. A new program needs its own policy and flag pair, or this policy made general.
  • AccountAndCardAddedEvent.ProgramId is set in CreateCardAction and GenericCardService. CardGatewayMappings never sets CardProgramId, so gateway cards have no product token. Card.ProgramId is filled only by GetCardById. LedgerAccountAddedEvent and CardStatusUpdatedMessage carry no program.
  • Many CardService, GenericCardService and NonPersonalizedCardService paths ignore the card gateway and call the issuer gateway, which for crypto is CryptoCardIssuerGateway and throws NotImplementedException (500).

be-crypto (only when the program's account and money live there)

  • CryptoProgram.Ensure accepts only program 5 and returns 400 otherwise; called from the account and card gateways.
  • CryptoWalletCardGateway calls be-lithic ILithicCardProcessorGateway. CryptoCard.ProgramId is int NOT NULL.
  • CryptoNetworkTransactionGateway implements Authorize with a balance check, GetBalance, BalanceInquiry and RecordEvent since be-crypto#290 (merged 2026-11-02). Moving money goes through LedgerProgramAction, whose Sync and GetWalletDebitCapacity throw NotImplementedException until the on-chain ledger program ships (be-crypto abe28f3).

Downstream

  • be-transactions: TransactionServiceModule keys ILedgerProcessorGateway by AccountType; a new one needs a key or transactions fail with "No ledger processor is registered". Decide the three lists in AccountTypeExtensions (credit, non-balance, Firebase). CardProrgamId (misspelt in the schema) is nvarchar(20) and receives the enum name.
  • be-cashback: AccountAndCardAddedEventHandler registers every card with Kard, whatever the program.
  • be-hydra: AccountTypeExtensions shows credit funding only for 02 and 03.
  • be-nonpersonalizedcard: non-personalized cards are debit only.
  • be-credit: only for a LoanPro-backed program.
  • dbt models that map the program need the new value.

Configuration

Each file under src/<env>/ in be-configuration (namespace minority) and be-cde-configuration (namespace cde) becomes a ConfigMap named after the file. A merge to master syncs stage and then prod, so both values must be safe at merge time. be-cde-configuration PRs need cde-change-managers approval.

File Repos What to add Read by
global-cardprogramtype-config.json both, same token per env The program and its Lithic token minority-lithic-service, minority-carddata-service
app-settings.json service-hosts be-configuration: lithic-<p>-api and lithic-cde-<p>-api; be-cde-configuration: lithic-cde-<p>-api Hosts Lithic clients
global-resilience-config.json be-configuration $lithic-<p>-api, be-cde-configuration $lithic-cde-<p>-api Resilience policy; otherwise $outgoing-request applies Lithic clients
lithic-config.json be-configuration Card art, product ids, the program's NetworkTransactionGateway.CardPrograms entry (Enabled, CardIds) minority-lithic-service
wallet-configuration.json be-configuration Enrolment flag and allowlist (Rain: CryptoCardEnrolmentEnabled, its allowlist) minority-wallet-service

Which lithic-config.json value each card type uses:

Key Virtual Physical Debit prod Credit secured prod
CardProgramToken (in global-cardprogramtype-config.json) yes yes real token real token
DigitalCardArtTokens yes yes no entry, so no art is sent two tokens
CardDesignProductIds no yes: create, renew migrated, convert virtual to physical, reissue, renew 1 / 3 1 / 2
ExpeditedCardDesignProductIds no yes, when shipping is UPS2ndDayAir 5 / 6 4 / 3

NonPersonalizedCardDesignProductIds is not per program.

Traps:

  • A program name in global-cardprogramtype-config.json that the running Minority.Wallet.Domain.Contract does not know fails to bind the whole config, which breaks card creation and card-data reads for every program. Bump the packages first, or write the number.
  • The debit and credit CDE hosts end in /v1 and their routes do not; Rain's CDE host has no /v1 and its route does. Copy one pattern consistently. A missing host makes the platform call http://<hostId>/... and fail on the first call.
  • A NetworkTransactionGateway.CardPrograms entry without "Enabled": true is off, and "LithicRainCrypto": { "Enabled": false } sends Rain to be-ledger, which does not hold its money. The old global Enabled and EnabledForCardIds keys are ignored since be-lithic#733.
  • Turning an enrolment flag on with an empty allowlist opens it to everyone.

Secrets

Names, with <P> the program prefix (Rain: RainCrypto):

  • lithic-credentials--<P>ApiKey
  • lithic-webhook-secret--<P>EventWebhookKey
  • lithic-webhook-secret--<P>AuthStreamWebhookKey

Both webhook secrets are optional at startup: the service starts without them and fails per call.

Vault Secrets Read by If missing
stage-lithic-kv, prod-lithic-kv API key and both webhook secrets API key: minority-lithic-service; webhook secrets: minority-lithic-api FormatException on the first call; 401 on the program's webhooks
stage-lithictesting-kv (stage only) API key and both webhook secrets, with the same values as stage-lithic-kv minority-lithictesting-api and -service The proxy returns 500, or be-lithic rejects re-signed webhooks on a mismatch
carddata vault API key for Minority.Lithic.Cde.Client minority-carddata-service Card data calls fail
  • Deployments and the vaults they read (AzureKeyVault:KeyVaultNames): minority-lithic-api and -service read {env}-m-global-kv,{env}-lithic-kv; the stage lithictesting pods read stage-m-global-kv,stage-lithictesting-kv.
  • In prod-carddata-kv, setting a secret needs the key-custodians group.
  • local- copies: be-cicd add_user_secrets_from_kv.py reads local-* secrets from vaults tagged env=stage and repo=<repo>, strips the prefix and writes user secrets. Add local-lithic-credentials--<P>ApiKey, local-lithic-webhook-secret--<P>EventWebhookKey and local-lithic-webhook-secret--<P>AuthStreamWebhookKey to stage-lithic-kv, with the stage values (runbook, step 4).
  • lithic-webhook-secret--*TransactionEventWebhookKey and lithic-webhook-secret--AuthStreamWebhookKey in the lithic vaults are leftovers; no code reads them.
  • A new secret in an existing vault needs no infrastructure change. A new vault or pod identity needs an OpenTofu and network-policy PR first.

Network policies

Pods deny all traffic unless platform-infrastructure allows it: helm/values/internal/noncde-network-policies/values.yaml for the minority namespace, and cde-network-policies for cde. Each service has an ingress list (who may call it) and an egress list (what it may call). A call needs both the caller's egress rule and the target's ingress rule. The base values.yaml applies to stage and prod; the environment files only add private endpoints.

A program whose account and money live in another service needs rules in both directions between that service and minority-lithic-service. Rain:

Direction Used for PR
crypto-service to lithic-service be-crypto's card gateway calls be-lithic's card processor gateway platform-infrastructure#1350
lithic-service to crypto-service be-lithic's ASA and transaction events call be-crypto's network transaction gateway platform-infrastructure#1404

A missing rule gives no error. The connection is dropped, each attempt ends after about 10 seconds with no response, the platform retries, and the target logs no request. In Elastic the caller's ROUTRP has HttpMessageBody: null, no response headers and WillRetry: true. On 2026-10-30 this made Lithic decline Rain's first stage ASA with "Customer ASA Timeout".

Validate a change for stage and prod, and diff the rendered output against master:

V=helm/values/internal/noncde-network-policies
helm lint helm/templates/network-policies -f $V/values.yaml -f $V/stage-values.yaml
helm template np helm/templates/network-policies -f $V/values.yaml -f $V/stage-values.yaml > branch-stage.yaml

helm lint warns about an empty object name; that warning is already on master.

Stage test proxy

stage-lithictesting.minority.com (minority-lithictesting-api) has the same webhook routes as be-lithic and forwards to minority-lithic-api. A request with a webhook-signature header is re-signed with the program's secret from stage-lithictesting-kv, without checking the incoming signature, on purpose so test events can be posted. A request without the header is forwarded unchanged. It deploys only by workflow_dispatch (cicd_workflow_lithic_testing.yaml).

A new program needs a switch arm and registrations in LithicTestingServiceModule, its secrets in stage-lithictesting-kv, the lithic-cde-<p>-api host in be-configuration stage service-hosts, and a manual deploy. TransactionSimulateService reads the program from be-lithic's LithicAccountHolder; null falls back to LithicDebit.

  • The debit and credit sandbox webhooks are registered on stage-lithictesting too (548 debit events in 30 minutes on 2026-10-30).
  • AuthTestController.Authorize only sets the POS fields from the MCC and forwards the ASA to be-lithic's IAuthController.
  • The simulate service (ITransactionSimulateService, POST simulate/authorize on minority-lithictesting-service) cannot be reached from outside: its chart's ingress is disabled. To test an authorization, call Lithic's sandbox directly.

Registering webhooks with the Lithic API

A runbook for the event subscription and the ASA responder of one program in one environment. Run it yourself, stage first, then prod. Nothing in it prints a secret: the API key goes from Key Vault into a shell variable, and each webhook secret goes from Lithic straight into Key Vault. Never paste an API key or a secret into a ticket, chat or a file.

<P> is the program prefix in secret names (Rain: RainCrypto) and <p> the URL segment (Rain: lithicraincrypto, see Webhooks in be-lithic).

Stage Prod
Lithic API https://sandbox.lithic.com https://api.lithic.com
API key read from stage-lithic-kv prod-lithic-kv
Webhook secrets written to stage-lithic-kv and stage-lithictesting-kv, identical values, plus the local- copies in stage-lithic-kv prod-lithic-kv only; prod has no testing proxy
Event URL https://stage-lithictesting.minority.com/v1/webhooks/<p>/Event https://lithic.minority.com/v1/webhooks/<p>/event
Authorization URL https://stage-lithictesting.minority.com/v1/webhooks/<p>/Authorization https://lithic.minority.com/v1/webhooks/<p>/authorization
Pods to restart minority-lithic-api, minority-lithictesting-api minority-lithic-api
Access needed Write on both stage vaults Write on prod-lithic-kv. Rain's prod API key is already there (lithic-credentials--RainCryptoApiKey, added 2026-10-29)

Why this order: be-lithic answers 401 to a webhook whose program has no secret loaded (it fails closed), and Lithic takes a 401 on the ASA as a decline. So the responder is registered and the subscription enabled only after both secrets are in Key Vault and the pods have restarted to read them.

For Rain, stage was done on 2026-10-30 and prod is not done yet; see Rain status.

0. Set up the shell. Stage values with Rain as the example; for prod use the prod column.

VAULT=stage-lithic-kv
LITHIC_BASE=https://sandbox.lithic.com
P=RainCrypto
EVENT_URL=https://stage-lithictesting.minority.com/v1/webhooks/lithicraincrypto/Event
AUTH_URL=https://stage-lithictesting.minority.com/v1/webhooks/lithicraincrypto/Authorization
LITHIC_API_KEY=$(az keyvault secret show --vault-name $VAULT --name lithic-credentials--${P}ApiKey --query value -o tsv)
lithic() { printf 'Authorization: %s\n' "$LITHIC_API_KEY" | curl -sS --fail-with-body -H @- -H 'Content-Type: application/json' "$@"; }
put_secret() {
  lithic "$LITHIC_BASE$1" | jq -j '.secret // .key // empty' \
    | az keyvault secret set --vault-name "$2" --name "$3" --file /dev/stdin --encoding utf-8 -o none
}

1. Prod only: ASA enabled for the program. Lithic's Implementation Manager enables it; ask before you start.

2. List the subscriptions. If one already points at the event URL, use its token in step 3 instead of creating a second one.

lithic "$LITHIC_BASE/v1/event_subscriptions" | jq '.data[] | {token, url, disabled}'

3. Create the event subscription, disabled, and keep its token.

EVT_TOKEN=$(lithic -X POST "$LITHIC_BASE/v1/event_subscriptions" \
  -d "{\"url\":\"$EVENT_URL\",\"description\":\"$P events\",\"disabled\":true}" | jq -r '.token // empty')
echo "subscription token: $EVT_TOKEN"

An empty EVT_TOKEN makes the later calls return 404 Invalid endpoint, which looks like a wrong URL.

4. Load both secrets into Key Vault. GET /v1/event_subscriptions/{token}/secret is the subscription's signing secret. GET /v1/auth_stream/secret is the ASA secret; it belongs to the Lithic account, not the program, and Lithic creates one on the first read.

Stage, both vaults with identical values, the local- copies, and the API key in the testing vault if it is not there yet:

for v in stage-lithic-kv stage-lithictesting-kv; do
  put_secret "/v1/event_subscriptions/$EVT_TOKEN/secret" $v lithic-webhook-secret--${P}EventWebhookKey
  put_secret /v1/auth_stream/secret $v lithic-webhook-secret--${P}AuthStreamWebhookKey
done
put_secret "/v1/event_subscriptions/$EVT_TOKEN/secret" stage-lithic-kv local-lithic-webhook-secret--${P}EventWebhookKey
put_secret /v1/auth_stream/secret stage-lithic-kv local-lithic-webhook-secret--${P}AuthStreamWebhookKey
printf %s "$LITHIC_API_KEY" | az keyvault secret set --vault-name stage-lithic-kv --name local-lithic-credentials--${P}ApiKey --file /dev/stdin --encoding utf-8 -o none
printf %s "$LITHIC_API_KEY" | az keyvault secret set --vault-name stage-lithictesting-kv --name lithic-credentials--${P}ApiKey --file /dev/stdin --encoding utf-8 -o none

Prod, one vault:

put_secret "/v1/event_subscriptions/$EVT_TOKEN/secret" prod-lithic-kv lithic-webhook-secret--${P}EventWebhookKey
put_secret /v1/auth_stream/secret prod-lithic-kv lithic-webhook-secret--${P}AuthStreamWebhookKey

5. Restart the pods. Secrets are read only at startup. Stage: minority-lithic-api and minority-lithictesting-api, plus minority-lithictesting-service when step 4 put the API key into stage-lithictesting-kv. Prod: minority-lithic-api. Use Argo CD "Deploy Restart" or kubectl rollout restart.

6. Register the ASA responder.

lithic -X POST "$LITHIC_BASE/v1/responder_endpoints" -d "{\"type\":\"AUTH_STREAM_ACCESS\",\"url\":\"$AUTH_URL\"}"

7. Enable the event subscription.

lithic -X PATCH "$LITHIC_BASE/v1/event_subscriptions/$EVT_TOKEN" -d "{\"url\":\"$EVENT_URL\",\"disabled\":false}"

8. Verify.

  • Lithic's delivery attempts, with our response codes: lithic "$LITHIC_BASE/v1/event_subscriptions/$EVT_TOKEN/attempts" | jq '.data'.
  • Elastic: the query in Webhooks in be-lithic. On prod, search minority-backend-prod and only minority-lithic-api. A webhook for a program without a loaded secret shows there as RINRP with result == 401.
  • Stage: the two stage vaults hold the same values. With Rain's names; for another program replace RainCrypto with <P>. It prints "same", "DIFFERS", or which vault could not be read or held an empty value, never a secret:
for n in lithic-credentials--RainCryptoApiKey lithic-webhook-secret--RainCryptoEventWebhookKey lithic-webhook-secret--RainCryptoAuthStreamWebhookKey; do
  a=$(az keyvault secret show --vault-name stage-lithic-kv --name $n --query value -o tsv) && [ -n "$a" ] || { echo "$n: missing or unreadable in stage-lithic-kv"; continue; }
  b=$(az keyvault secret show --vault-name stage-lithictesting-kv --name $n --query value -o tsv) && [ -n "$b" ] || { echo "$n: missing or unreadable in stage-lithictesting-kv"; continue; }
  [ "$a" = "$b" ] && echo "$n: same" || echo "$n: DIFFERS"
done; unset a b

9. Clear the shell.

unset LITHIC_API_KEY EVT_TOKEN EVENT_URL AUTH_URL VAULT LITHIC_BASE P; unset -f lithic put_secret

Notes from earlier runs:

  • Both secret types have rotate endpoints; the old secret stays valid for 24 hours after a rotation, so the new one must be in every vault and the pods restarted within those 24 hours.
  • POST /v1/simulate/event_subscriptions/{token}/send_example with {"event_type": "..."} sends a sample event, but not for every type. On 2026-10-30 the sandbox returned 400 for card.created (event-type 'card.created''s schema doesn't have an example at index 0), and also for account_holder.created, balance.updated and dispute.updated, and nothing reached our endpoint. Repeated calls also returned 429 (rate limit). When testing, pick a type be-lithic ignores (see Webhooks in be-lithic) so the sample payload is not processed.
  • Blocking and unblocking a card (PATCH /v1/cards/{token} with state) sent no webhook to the Rain subscription on 2026-10-30.

Testing an authorization on stage

Lithic's POST /v1/simulate/authorize needs the card's pan; the card token alone returns 400 Missing required parameter(s): pan. With the shell from step 0 of the runbook on stage, read the PAN with the program's key without printing it (sandbox only):

CARD_TOKEN=<Lithic card token>
PAN=$(lithic "$LITHIC_BASE/v1/cards/$CARD_TOKEN" | jq -r '.pan // empty'); echo "pan length: ${#PAN}"
lithic -X POST "$LITHIC_BASE/v1/simulate/authorize" \
  -d "{\"pan\":\"$PAN\",\"amount\":100,\"descriptor\":\"ASA test\",\"mcc\":\"5812\"}" | jq '{token,debugging_request_id}'
unset PAN

amount is in cents. The returned token is the Lithic transaction token; the dashboard shows the decision and, when our responder did not answer in time, "Customer ASA Timeout".

The ASA's path on stage, with the log level each hop writes:

Hop Container Log
Lithic to the proxy, /v1/webhooks/<program>/Authorization minority-lithictesting-api RINRQ
Proxy to be-lithic minority-lithic-api RINRQ
IAuthService.Authorize minority-lithic-service RINRQ
ITransactionService.ValidateNetworkTransactions (card status, risk rules) minority-lithic-service to minority-transactions-service ROUTRQ, ROUTRP
The program's network transaction gateway, Authorize (Rain: http://minority-crypto-service/accounts/{accountId}/network-transactions/{flowId}/authorize) minority-lithic-service ROUTRQ, ROUTRP
The decision back to Lithic minority-lithic-api, minority-lithictesting-api RINRP

Find the request by the Lithic transaction token, which is in the body minority-lithic-api logs on RINRQ, and read its ruid, the correlation id every hop shares:

FROM minority-backend-stage
| WHERE @timestamp >= NOW() - 1 hour
  AND kubernetes.container_name.keyword == "minority-lithic-api"
  AND logLevel.keyword == "RINRQ"
  AND object LIKE "*<transaction token>*"
| KEEP @timestamp, ruid

Then follow the ruid, or pass it to the Elastic connector's trace_request:

FROM minority-backend-stage
| WHERE @timestamp >= NOW() - 1 hour
  AND ruid.keyword == "<ruid>"
  AND logLevel.keyword IN ("RINRQ", "RINRP", "ROUTRQ", "ROUTRP")
| KEEP @timestamp, kubernetes.container_name.keyword, logLevel.keyword, endpoint, result
| SORT @timestamp

On 2026-11-11 one stage ASA's ruid covered every hop above, from minority-lithictesting-api through minority-lithic-api, minority-lithic-service and minority-transactions-service to the program's gateway service. The object of minority-lithictesting-api's RINRQ did not contain the token, so start from minority-lithic-api.

The card_transaction.updated event for the same transaction arrives a few seconds later on the Event route and is published as lithic-callback-event.TransactionEventsRequestEvent.

Deployment order

  1. Lithic: program, tokens and API key per environment; in prod, ASA enabled for the program by Lithic's Implementation Manager.
  2. The API key in the vaults, stage then prod, plus its local- copy. The webhook secrets come in step 11.
  3. Network-policy PR in platform-infrastructure for every new call between be-lithic and the program's service, in both directions.
  4. be-bank-platform enum, packages published.
  5. Consumers bump Minority.Wallet.Domain.Contract.
  6. be-lithic, which publishes Minority.Lithic.Cde.Client.
  7. be-cde-carddata.
  8. be-wallet: resolver arm, Bank/PRN migration, account features, gateways, policy, flags off.
  9. Config PRs in both repos: additive only, prod flags off.
  10. Restart, because secrets never reload and some config is captured at startup: minority-lithic-api, minority-lithic-service, minority-wallet-service, on stage minority-lithictesting-api and -service, and minority-carddata-service in the cde namespace. Use Argo CD "Deploy Restart" or kubectl rollout restart.
  11. Webhooks on stage: the stage part of Registering webhooks with the Lithic API, which loads both webhook secrets, restarts the webhook pods, registers the ASA responder and enables the event subscription. Then test on stage with the flag on and an allowlist.
  12. Webhooks on prod: the prod part of the same runbook, before any prod card can be issued.
  13. A separate prod config PR: real token and card art, the program's NetworkTransactionGateway.CardPrograms entry enabled (all cards, or a CardIds list), enrolment on with a non-empty allowlist.

Rain status

First recorded on 2026-10-30. Rows rechecked later say when; the last check was on 2026-11-11.

Item State
Prod program token Placeholder ...-3000-... on master in both config repos; draft PRs be-configuration#3436 and be-cde-configuration#166 (BAN-488) set the real one. Still open drafts on 2026-11-11
Prod card art and physical product ids Placeholders; marked TBD in be-configuration#3436. Unchanged on 2026-11-11
Prod NetworkTransactionGateway Off on 2026-10-30. On for Rain, all cards, since be-configuration#3463 (merged 2026-11-05), and on by be-lithic's code default. be-configuration#3436 still edits the removed global Enabled and EnabledForCardIds keys, so it needs rebasing onto the per-program shape before it merges
Prod CryptoCardEnrolmentEnabled Off on master (checked 2026-11-11); be-configuration#3436 turns it on with an empty allowlist
Stage enrolment On with an empty allowlist, so every stage crypto holder can request a card (checked 2026-11-11)
Settlement reports On 2026-10-30 SettlementReportUpdatedEventHandler threw for Rain; reporting only, so no money was affected. Since be-lithic#733 (merged 2026-11-05, BAN-533) Rain reports are fetched through the client factory and the details carry be-crypto's balance. No stage run checked
Authorization balance check None on 2026-10-30. be-crypto#290 (merged 2026-11-02) checks the balance in CryptoNetworkTransactionGateway.Authorize. A purchase the wallet covers then calls LedgerProgramAction.Sync, which throws NotImplementedException until the on-chain ledger program ships (be-crypto abe28f3, read 2026-11-11)
Lost or stolen, cancel The card is terminated with no automatic replacement since be-wallet#1352 (merged 2026-11-02); the holder requests a new card through POST v3/accounts/{accountId}/cards
Reissue, provisioning, restrictions, PIN 500 (NotImplementedException) on 2026-10-30; not rechecked
Stage webhooks (checked 2026-11-11) Set up. Both webhook secrets are in stage-lithic-kv and stage-lithictesting-kv since 2026-10-30, with identical values. The event subscription is enabled; card_transaction.updated arrives through stage-lithictesting and is published as TransactionEventsRequestEvent. 29 Rain event webhooks in the 14 days to 2026-11-11, all answered 200 by both minority-lithictesting-api and minority-lithic-api
Prod webhooks (checked 2026-11-11) Not set up. prod-lithic-kv holds lithic-credentials--RainCryptoApiKey (added 2026-10-29) but neither lithic-webhook-secret--RainCryptoEventWebhookKey nor lithic-webhook-secret--RainCryptoAuthStreamWebhookKey. minority-lithic-api answered no request with 401 in the 7 days to 2026-11-11, so no Rain webhook reaches prod yet. The prod part of the runbook is needed before prod issuance is turned on (BAN-488)
Stage local- copies (checked 2026-11-11) stage-lithic-kv has no local-lithic-credentials--RainCryptoApiKey, local-lithic-webhook-secret--RainCryptoEventWebhookKey or local-lithic-webhook-secret--RainCryptoAuthStreamWebhookKey, so be-lithic run locally has no Rain key or webhook secrets
Stage ASA Approved end to end on 2026-10-30 after platform-infrastructure#1404 (first at 12:37 UTC, about 230 ms from the proxy receiving it to the proxy answering). Before that, the call to be-crypto timed out because no network-policy rule allowed it. No Rain ASA on stage in the 14 days to 2026-11-11; the ASA secret is unchanged since 2026-10-30. The last approval predates be-crypto#290, so a purchase the wallet covers now reaches the Sync stub above (read in code, not tested on stage)
Stage transaction events On 2026-10-30 card_transaction.updated reached be-crypto RecordEvent, which returned 500 (NotImplementedException); lithic-service tries the message four times and stops. Since be-crypto#290 (merged 2026-11-02) RecordEvent records the event, but a flow that has amounts left to apply calls LedgerProgramAction.GetWalletDebitCapacity and Sync, which throw NotImplementedException until the ledger program ships (read 2026-11-11, not observed on stage)

Compared with debit and credit, Rain added: issuance by card request with a reserved card id and PRN, gateways chosen by AccountType, card ids minted by wallet, ProgramId on open-account and create-card requests, one PRN range for accounts and cards, and a network-transaction owner other than be-ledger.

Not yet confirmed

  • Whether the sandbox treats ...-3000-... as Rain's real program token.
  • Which carddata vault holds the Lithic API keys today.
  • Whether the LoanPro program names (26 and 28 characters) are cut in be-transactions CardProrgamId.