Skip to content

Arch Forum 2024-03-14

Participants: Backend devs, Zak, Victor and Andy

Agenda

  • Scheduling

Notes

Scheduling: We use the scheduling feature of Azure Service Bus, which allows a message to be scheduled to be sent in the future. When we now migrate to RabbitMQ it's not obvious what to do, RabbitMQ does not come with a scheduling feature built in. In this meeting we explained the situation, and gathered insights from everyone of usages, alternatives etc:

Scheduling usage

Looking at our current usage, here's a selection of events and their scheduling time:

  • Calling FinishPendingUnlimitedCallMessage (1 min)
  • CashDeposit ExpireAccountNumberMessage (15 min)
  • Fees&Promos SubscriptionFeeMessage(30 days)
  • Remittance PendingTransactionAlertEvent (1 day)
  • SalesTool AdvisorAutoClockOutEvent (< 1day)
  • User UserBlockRequestMessage (5 sec)
  • User ScheduledMessageWrapper (some built in generic scheduling feature...)
  • Galileo UpdateCardStatusCallbackMessage (5 sec)

In summary:
- 10-15 areas use scheduling
- There are 40-60 calls to ScheduleEvent method, maybe 30 unique events scheduled.
- Scheduling delay is between 1 second to 30 days
- In addition to service bus message scheduling, we use k8s cron jobs, and the Rate area has an internal scheduling mechanism only used by Rate.
- There are around 300k messages scheduled, most of them are for subscription fee.

Alternatives

1. RabbitMQ Scheduling plugin

  • The scheduling plugin available for RabbitMQ is not suitable for us. It has a number of limitations (message only persisted on single node, message delay max 21 days in the future) which rules it out.
  • There seems to be plans to re-architect the plugin, but that does not help us now or in the near future.

2. A scheduling service.

  • All areas using scheduling will be dependent on this single service. For long schedule times it doesn't matter, but for short ones it might.
  • A scheduling service is quite a lot of overhead for scheduled messages with very short schedule times (like 1 sec)
  • Sometimes we need to schedule "area-internal" messages, not exposed in contract. A scheduling service should be able to handle that.

3. Scheduling functionality in platform code.

  • This means the area needs a db with a table to keep scheduled messages. Some areas does not have a DB today, but might want to use scheduling.

4. Continue to use Azure Service Bus for scheduled messages.

  • Easy solution
  • Requires we continue to keep the service bus code working, config etc.

Additional notes

  • With our maturing as a company and increased number of events, it can be useful to introduce the "outbox" pattern. While logically a different concern than scheduling, implementation might be very similar.
  • Unclear if scheduling needs to work together with sessions or not. There's at least one test that use both at the same time.
  • Unclear how many messages use scheduling (not count of the type of messages, but of number of messages sent). No easy way to see this.
  • Both Quartz and Hangfire libraries are troublesome. We have previously attempted to use Hangfire but did not work well because of issues with async.
  • There have been some talks about additional scheduling features, such as a fx rate notification users can subscribe to when fx rate is above or below some value.