Health Check

Health Checks (Rebtel.Core.Infrastructure.HealthProbes)

Relevant Kubernetes resources:
* Pod Lifecycle
* Configure Liveness, Readiness and Startup Probes
* Termination of Pods
* Container Lifecycle Hooks

Health Cheks infrastructure has the following features:
* Integrates with DotNet core HealthCheks infrastructure
* Exposes four endpoits: /probe/liveness, probe/readiness, probe/report and /lifecycle/pre-stop
* Monitors application lifecycle and updates liveness/readiness probes accordingly
* Wraps all IHostedService instances inside HostedServiceDecorator, monitors hosted services lifecycle and updates liveness/readiness probes accordingly
* Wraps all health checks inside HecthCheckController to add configuration and caching support

Probe Endpoints

Three probe/xxx endpoints return different HTTP status codes based on Health Status returned from health checks.

/probe/liveness

Returns 503 if health checks result in Unhealthy status. Failed liveness probe causes Kubernetes to restart containers, so health checks should be included in liveness probe with caution.

/probe/readiness

Returns 503 if health checks result in Degraded or Unhealthy status. Failed readiness probe causes Kubernetes to temporarily stop routing new traffic to unhealthy container, which lets it some time to recover. The traffic resumes when readiness probe returns 200.

/probe/report

Always returns 200 and contains detailed health check status information in the body.

/lifecycle/pre-stop

Used to notify the application that it will be shut down soon. Causes the readiness probe to fail. This endpoint should be wired up to the corresponding Kubernetes web-hook call to implement graceful application shutdown. When kubernetes calls it before container shutdown, the application will start failing readiness probes, which in turn will stop new traffic being routed to the application in the container.

Probe Results During Application Lifecycle

Application
Status      Starting  Running  Pre-stop Stopping Stopped
            >-------->-------->-------->-------->--------|
liveness    200       200      200      200      200
readiness   503       200      503      503      503



Application
Status      Starting  Failed  
            >-------->--------|
liveness    200       503     
readiness   503       503

Factors That Affect Readiness Probe Result

• Degraded/Failed health check status returned by one of the health checks
• Application lifecycle is not in Running state
• IHostedService is not in Running state

Factors That Affect Liveness Probe Result

• Application or IHostedService is not in running state, i.e. starting or stopping.
• IHostedService threw an exception while starting
• A healcth check returned unhealthy status

Configuration

The configuration for health checks in general and for specific health checks can be defined in in app-settings config map, in HealthProbe section:

{
  "HealthProbe": {
    "CacheDurationSeconds": 0,
    "Probes": {
      "Report": true,
      "Startup": true,
      "Liveness": true,
      "Readiness": true
    },
    "$SpecificHealthCheckType": {
      "Probes": {
        "Report": true,
        "Startup": true,
        "Liveness": false,
        "Readiness": false
      },
      "$AutoFacServiceKey": {
        "Probes": {
          "Report": true,
          "Startup": true,
          "Liveness": false,
          "Readiness": false
        }
      }
    }
  }
}

The configuration supports enabling/disabling specific probe types for all health checks or for specific ones. To configure a specific health check type, add a new Property with Health Check type name starting with $ into HealthProbes section. If you have multiple instances of the same health check type registered with different keys (AutoFac keyed service), you can define separate configurations for specific keys, like in the example above.

I.e. when a health check initializes, the configuration will be first read from HealthProbe configuration key, then from HealthProbe.$HealthCheckTypeName and override the default HealthProbe settings, then from HealthProbe.$HealthCheckTypeName.$AutoFacServiceKey. This allows us to define global configuration for all health checks and override

Implementing Health Checks

• Implement DotNet Core IHealcthCheck interface
• If your health check class has complex initialization logic, implement IServiceLifetime interface.
• Register your health check in AutoFac as singleton:
  

  // The code below registers three health checks: two MyKeyedHealthCheck with different keys/names and one MyHealthCheck.
  // The config values for the check below can be defined in HealthProbe.$MyKeyedHealthCheck (will configure both checks below the same way) 
  // or HealthProbe.$MyKeyedHealthCheck.$HealthCheck1 (will configure the current registration only)
  builder.RegisterType<MyKeyedHealthCheck>()
         .Named<IHealthCheck>("HealthCheck1")
         .SingleInstance();
  
  builder.RegisterType<MyKeyedHealthCheck>()
         .Keyed<IHealthCheck>("HealthCheck2")
         .SingleInstance();
  
  // This check can only be configured by providing HealthProbe.$MyHealthCheck config section as it has no keys.names
  builder.RegisterType<MyHealthCheck>().As<IHealthCheck>.SingleInstance();
  

• Implement CheckHealthAsync method and provide the status: Healthy, Degraded or Unhealthy. Please note that Degraded status will only affect readiness probe result while liveness probe will continue returning 200.