Skip to content

Arch Forum 2024-04-18

Participants: Backend devs, EMs, Andy, JD and Victor

Agenda

  • Automapper!
  • DB Perf
  • Transaction retries
  • Deadletter queues?

Notes

Automapper: With .NET8, the Automapper version we use is not supported. Instead, we need to update it to latest. We tried with Hydra as an example since Hydra is one of the areas with the most mappings. There are a number of changes in later version of Automapper, notably there's no default mappings anymore. The motivation behind is described in their Github, at AutoMapper/AutoMapper#3063. Victor updated Hydra, and had to add around 50 new mapping pairs to make it work, a fair amount of work.

Given the above complexities with Automapper, and our thinking that Mapperly would be a better choice for the future, Victor put together a c# tool to automatically do most of the repetitive work converting mappings from Automapper and Mapperly and ran it on Hydra. This showed the work would be similar to convert to Mapperly as to upgrade Automapper.

Rasmus brought up the case why we need a mapping library at all, couldn't we just write manual code? It can be done of course, however, we have not really considered this option. There are of course downsides with all dependencies, And just as Automapper upgrade proved to be difficult, so could a future version of Mapperly. Also, with manual mappings, there's less magic than even Mapperlys source generation. However:

  • In previous discussions, no-one have suggested or pushed for this case.
  • It would probably be even more work to convert the Automapper / Mapperly mappings to "manual" mappings.
  • A script could help, but that script would be quite complicated, already the helper from Automapper to Mapperly was not easy to put together.
  • Without a script, the risk will be higher (i.e. missing to map a property) since there's a lot of mappings (200+ in Hydra).

In total, this means it's too late without very strong case to go back to "square one" in the Automapper discussions. Hence, we will go with Mapperly.

In conclusion:

  1. We will use Mapperly for our object mapping needs.
  2. New code, like Check, should use Mapperly.
  3. Victor will upgrade Hydra. After this we plan upgrade of other areas.

DB Perf: As a second topic we revisited our DB performance. The database is arguably most critical part, the most heavily used, and most expensive part of our infrastructure. As discussed before, we are now at a point good use of the database is important not only for the biggest / heavily used DB, but most of the databases.

We looked at the current DTU usage over all the databases. Without digging into the details, probably more than half of DTUs used in a day are wasted and could be saved relatively easily. https://app.datadoghq.com/notebook/5415726/victor-db-usage-overview

We also looked at 3 recent examples of non-performing stored procedures / queries:

- UserGetAllByFilters_V2 - Does a user.msisdn like '%xxx%' query, which cant be optimized with normal index and always scan either table or index.
- SessionHistoryGetDeviceCount - Uses index optimally with seek, but is still slower than needed because the table is partitioned and query did not constrain the partition column into the future.
- OneTimeCodeVerificationGetCount - By using the wrong data type on one of its arguments, it did not use index properly and did scan instead of seek.

Transaction retries: A discussion around what happens when Galileo returns 5xx errors on transactions (or any 3rd party). When Galileo returns 5xx errors we can not be certain of the result, sometimes a transaction went through, sometimes not. How should this case be handled? A discussion followed.

  • Transaction service can take some responsibility by using OwnerResourceId to distinguish if a call is a retry or a new. If the same OwnerResourceId is used, it means that caller retried.
  • Does Galileo detect retries if the same TransactionId is used and return duplicate transaction error?
  • When Galileo fails, can/should we call them to check if the transaction already happened?
  • What about UX? If I do a Call I do not want the transaction to succeed later, then it's too late. However, with a remittance it can be ok to retry when Galileo is working again.
  • We need to have a standard way to handle this kind of problem.

At this point the meeting ended.

Deadletter queues with RabbitMQ: Out of time, saved for another time.