Queueing¶
Background:
- Remittance has four providers namely Thunes, GMT, Uniteller and Remitee. All these providers need some kind of funding for the remittance transactions to go through.
- Sample flow is like this, User submits remittance, BE does pre-auth and submits the transaction to one of the providers. Then the provider uses the funds from its balance to complete the transaction and deposit money to the beneficiary.
- There are two licenses involved in Remittance : MPS and MUSA. So providers maintain two separate accounts for these two licenses. Our finance team pre-funds these accounts from time to time so that remittance can work.
Note : GMT and Uniteller have credit lines, so no funding takes place for those. Only for Thunes and Remitee finance team actively does the funding.
Problem:
* For the funding to take place, Majority take out loans which incur high interest. Also the pre-funding amount is quite large so as to cover the fluctuations in remittance traffic especially over the weekend.
- So the problem is to utilize the pre-funding amount in an efficient manner and to save costs for Majority.
Solution:
The approach decided to solve this problem is to queue the remittance transactions in the backend whenever the provider balance is below a certain threshold.
How it works:
* Both Thunes and Remitee have API endpoints to check the account balance. So before creating the transaction at the provider, a balance check takes place.
- When the balance is below the threshold, from then on, all the future transactions for that provider will be updated to status QUEUED and saved in the BE database.
- Queueing continues till the balance is back up again.
- ‘CheckProviderBalancesEvent’ runs every 30 mins to check the provider balances. This event also checks if there is any active queueing for a provider and if the balance is above the threshold (meaning finance team has done the funding), then it sets the queueing to inactive.
- ‘ReleaseQueuedTransactionMessage’ event is thrown, then it takes the queued transactions based on batches and processes them.
Guide to Monitor
Follow the below steps to monitor the system when queueing happens,
* Based on Provider and License. So when inserting a record in the table, these two are required.
- Alert when queueing started: https://app.datadoghq.com/monitors/157318154
- Generally when queuing starts, check if it's for provider balance, sometimes it could be for other reasons. Verify if the balance is actually lower than the threshold.
- Check if there are no TEKMs and ERRs. Check if both the licenses are getting queued.
- Alert when queueing stopped and transaction starting to process again: https://app.datadoghq.com/monitors/157315624
- If possible have a list of transactions that got queued, check if all transactions have a majority exchange rate set.
- Before transactions start to process. Check if there is any need to increase the batch size (500 currently).
- Once the transactions start to process, check if all the transactions are processed.
- There is a possibility that, transactions will fail the FX threshold, alert for this: https://app.datadoghq.com/monitors/157315669