Skip to content

Arch Forum 2025-09-04

Participants: Backend devs, Andy, Kyle, Magnus, and Victor

Agenda

  • Timeout in Rate
  • Log sanitizers
  • Locale & Culture
  • Test performance
  • Logic in SPs

Summary

Timeout in Rate

An interesting case in the interaction between Rate Service and MoneyOut. (Rate calls MoneyOut to get provider FX rates)

Looking at the code here, it seems like MoneyOut will always return a 200 OK response, but sometimes with a "fake"/default FX rate of 0. be-mf-remittance

However, looking in the logs, this was not how it happened: Kibana

Discussion around this topic yielded two points that could be investigated further:

  • In general, it's advisable to have shorter timeouts for external services compared to our internal timeout (default of 100s).
  • When using Polly retries, either directly or indirectly, be aware of multiplied retries. If service A calls B, which calls C, and we use 3 retries, then we might end up with 9 total retries if C fails and B retries but also fails, leading to A to retry.

Log sanitizers

Will revert to using old sanitizing logic in stage, since we are too far from running the new sanitizer in prod.

Locale & Culture

The Accept-Language header is actually very complicated:

Read about locale and culture at Culture, Locale & Language

Other general notes:
- When designing tables and SPs, make sure to use the proper length and types (meaning owner decides)!
- Strings are silently truncated between C# and SP, but not between SP and columns. How should it work?

Test performance

This is data from GitHub Actions on the slowest GitHub action steps. Surprisingly, the integration tests (dotnet test) are the slowest in most cases.

Repo Step Duration (s)
be-user dotnet test 1101
be-galileo Run additional repo tests 611
be-galileo dotnet test 360
be-kyc dotnet test 278
be-cde-externalcarddata dotnet test 243
be-mtu Run Regression Test 242
be-risk dotnet test 239
be-phoneplan dotnet test 234
be-transactions dotnet test 234
be-mf-remittance dotnet test 232
be-braze dotnet test 217
be-risk Build and Push Docker Image 201
be-calling dotnet test 198
be-kyc Run Regression Test 186
be-calling Run Regression Test 181
be-wallet Build and Push Docker Image 171
be-kyc Build and Push Docker Image 167
be-cde-externalcarddata Run Tests 150
be-sms Run Tests 148
be-proxy Run Tests 146
be-phoneplan Run Regression Test 145
be-user Run Tests 145
be-remittance Run Tests 144
be-transactions Run Tests 144
be-rebtelcallbacks Run Tests 142
be-user Run Regression Test 142
be-galileo Run Tests 141
be-kyc Run Tests 140
be-braze Run Tests 138
be-mf-remittance Build and Push Docker Image 137
be-sms Build and Push Docker Image 136
be-credit dotnet test 136
be-credit Run Tests 134
be-ledger Run Tests 134
be-phoneplan Run Tests 132
be-wallet Run Tests 132
be-mtu Run Tests 132
be-hydra Run Tests 132
be-mpay Run Tests 131
be-mf-remittance Run Tests 129

How to improve?

Ensure there are no excessive waits: Reduce waiting time when using waits.

  • Set Wait.MaxRetryTime to a low value in test startup.
    - Example here
  • Override the wait time on the specific call to Wait,
    - Wait.Until(() => Task.FromResult(x), a => a, TimeSpan.Seconds(10)).
    - Example here

Parallelize the tests: If possible, allow tests to run in parallel:

  • Use the assembly Parallelize attribute,
    - i.e. [assembly: Parallelize(Scope = ExecutionScope.MethodLevel, Workers = 8)].
    - Example here AssemblyInfo.cs in platform.
  • If only some tests can't run in parallel, they can be excluded with the [DoNotParallelize] attribute.
    - Example here

Do not use Thread.Sleep or Task.Delay

  • The best is to not have any Wait or sleep/delays at all.
  • If you need to wait, use the Wait in platform instead of the generic Task.Delay.
  • Thread.Sleep should never be used

Logic in SPs

Discussion about when to put logic in SPs.

In general, our philosophy is to keep as much logic as possible in C#. But it does create some overhead (more SPs, more DB calls, etc.).

Example to start discussion here