Feature-Data-Service

The current deployed flow uses Pub/Sub and SQL. See the
interactive risk configuration map for the full
source-to-decision flow, be-risk-configuration
for deployed ingestion config, and be-risk
for runtime implementation. Kinesis and DocumentDB belong to the retired
architecture shown in the historical diagrams.

What does feature data service do?

Feature Data Service consumes live Pub/Sub events, extracts configured variables,
and publishes feature updates through RabbitMQ. Feature Update Service persists
those updates in SQL for Feature Read Service to resolve during decisions.

risk-event-processing-config.json

In be-risk-configuration repo, there is a configuration file "risk-event-processing-config.json". This file guides how FeatureDataService filter, extract and transform variable from event. However, we use DBT model to generate this file now. So we do not update this file manually, unless you want to fix some urgent thing on prod. But keep in mind that any manual change in this file will be overwrote later by DBT model.

Here is a snippet

{
  "Events": {
    "beneficiaryId/RemittanceTransactionSubmitEvent/sentAmount": {
      "Sources": [
        {
          "AggregatePoint": "BeneficiaryId",
          "SourceEvent": "RemittanceTransactionSubmitEvent",
          "Types": [
            "record"
          ],
          "VariableName": "SentAmount"
        },
        {
          "AggregatePoint": "BeneficiaryId",
          "SourceEvent": "RemittanceTransactionSubmitEvent",
          "Types": [
            "stats"
          ],
          "VariableName": "SentAmount"
        }
      ]
    }
  }
}

Events is a dictionary. Its item key give information about how the variable is organized in risk database. Its item value is an object which specify where does the information come from.

Events Key

• aggregationPoint is kind of identity to an object in real world. For example, a userId, a beneficiaryId.
• eventAliasName is kind of category of the variables. It would be better to call it variable category. But since event alias name has been used since beginning, we still use it.
• variableName is the variable name used in risk database.

Events Value

The value part contains a Sources property which is a list of sources. Here explain properties on the source object
- AggregatePoint: specify the source field to get a value for aggregation point
- SourceEvent: specify the event name of the source event
- Types: specify how data is stored in risk. It is a list of types actually. It is how risk system supports. But we used DBT model to generate this configuration file now. DBT parser has limits to output multiple types. So you will see it is always single value in the array.
- Variable: specify from which field should feature data service extract value
- transactionKeyFields: a part of transaction trace. It tells which field in the source event contains the transactionId. It is named as Fields, because you can put multiple field names, separated by comma
- transactionStatus: a part of transaction trace. To give transaction status directly
- transactionStatusField: a part of transaction status. To fetch transaction status from source event field. Not used now due to limitation in Dbt model parser. But risk system support it still.
- transactionStatusMap: a part of transaction status. Not used now due to limitation of DBT model parser. But risk system support it still
- Idempotent: a part of transaction trace. It tells should risk accept duplicate request. risk detect duplication by combining transactionId and transactionStatus

Transaction Trace

Transaction trace is a function to trace related activities, such as a transaction. This function was created to
- skip duplicated event because of application retry, or risk has to get value from multiple events but they overlaps
- get correct velocity value when tranaction starts and ends in different dates. For example, a user send $100 dollar yesterday, but is rejected by provider today. Without transaction trace, we will see the user's last1d amount become -$100. transaction trace can link this Rollback record to its Initialize_ record to get correct value.

Pub/Sub delivery

Stage and production select Pub/Sub in risk-app-features-config.json. Pub/Sub
subscription and retention settings belong to the deployed transport config;
there is no Kinesis shard sizing step in the current operating model.