Circuit Breaker Strategy

Prevent cascading failures by breaking the circuit when error thresholds are exceeded

Circuit Breaker Strategy

The circuit breaker reactive resilience strategy shortcuts the execution if the underlying resource is detected as unhealthy. When a circuit is broken, subsequent calls are immediately rejected without attempting execution.

The Circuit Breaker strategy rethrows all exceptions, including those that are handled. Its role is to monitor faults and break the circuit when a threshold is reached, not to manage retries.

When to Use Circuit Breaker

Use the circuit breaker strategy when:

• Protecting downstream services from being overwhelmed during failures
• Failing fast is better than making users wait for timeouts
• You want to give a failing system time to recover
• Preventing cascading failures in microservices architectures
• Implementing the “stop doing it if it hurts” principle

Installation

dotnet add package Polly.Core

Circuit States

The circuit breaker has four states:

1. Closed (Normal)

   Operations execute normally. The circuit monitors for failures.

2. Open (Broken)

   Circuit is broken. All operations are immediately rejected with BrokenCircuitException.

3. Half-Open (Testing)

   After the break duration expires, the circuit allows one test operation to check if the system has recovered.

4. Isolated (Manual)

   Circuit is manually held open. Operations are blocked until manually closed.

Usage

Basic Circuit Breaker

// Default: breaks when 10% of calls fail within 30 seconds (minimum 100 calls)
var pipeline = new ResiliencePipelineBuilder()
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions())
    .Build();

try
{
    await pipeline.ExecuteAsync(async ct => 
    {
        await CallExternalServiceAsync(ct);
    }, cancellationToken);
}
catch (BrokenCircuitException ex)
{
    // Circuit is open, operation was not executed
    Console.WriteLine($"Circuit broken. Retry after: {ex.RetryAfter}");
}

Custom Thresholds

var options = new CircuitBreakerStrategyOptions
{
    // Break if 50% of actions fail
    FailureRatio = 0.5,
    
    // Within any 10-second window
    SamplingDuration = TimeSpan.FromSeconds(10),
    
    // With at least 8 actions processed
    MinimumThroughput = 8,
    
    // Stay broken for 30 seconds
    BreakDuration = TimeSpan.FromSeconds(30),
    
    ShouldHandle = new PredicateBuilder()
        .Handle<HttpRequestException>()
};

Dynamic Break Duration

var options = new CircuitBreakerStrategyOptions
{
    FailureRatio = 0.5,
    SamplingDuration = TimeSpan.FromSeconds(10),
    MinimumThroughput = 8,
    
    // Break duration increases with failure count
    BreakDurationGenerator = static args => 
        new ValueTask<TimeSpan>(TimeSpan.FromMinutes(args.FailureCount))
};

Handling HTTP Status Codes

var options = new CircuitBreakerStrategyOptions<HttpResponseMessage>
{
    ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
        .Handle<HttpRequestException>()
        .HandleResult(r => r.StatusCode == HttpStatusCode.InternalServerError ||
                          r.StatusCode == HttpStatusCode.ServiceUnavailable)
};

Monitoring Circuit State

var stateProvider = new CircuitBreakerStateProvider();

var options = new CircuitBreakerStrategyOptions<HttpResponseMessage>
{
    StateProvider = stateProvider
};

var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddCircuitBreaker(options)
    .Build();

// Check current state
var state = stateProvider.CircuitState;
switch (state)
{
    case CircuitState.Closed:
        // Normal operation
        break;
    case CircuitState.Open:
        // Circuit is broken
        break;
    case CircuitState.HalfOpen:
        // Testing if system recovered
        break;
    case CircuitState.Isolated:
        // Manually held open
        break;
}

Manual Circuit Control

var manualControl = new CircuitBreakerManualControl();

var options = new CircuitBreakerStrategyOptions
{
    ManualControl = manualControl
};

// Manually isolate the circuit (e.g., during maintenance)
await manualControl.IsolateAsync();

// Manually close the circuit to resume operations
await manualControl.CloseAsync();

State Transition Events

var options = new CircuitBreakerStrategyOptions
{
    OnOpened = args =>
    {
        Console.WriteLine($"Circuit opened at {DateTime.Now}");
        // Send alert, log to monitoring system
        return default;
    },
    OnClosed = args =>
    {
        Console.WriteLine("Circuit closed - system recovered");
        return default;
    },
    OnHalfOpened = args =>
    {
        Console.WriteLine("Circuit half-open - testing system");
        return default;
    }
};

Configuration Options

ShouldHandle · Predicate

Defines which results and/or exceptions are counted as failures.

FailureRatio · double · default: "0.1"

The failure-success ratio that will cause the circuit to break. 0.1 means 10% of sampled executions must fail.

MinimumThroughput · int · default: "100"

The minimum number of executions that must occur within the sampling duration before the circuit can break.

SamplingDuration · TimeSpan · default: "30 seconds"

The time period over which the failure-success ratio is calculated.

BreakDuration · TimeSpan · default: "5 seconds"

Fixed time period for which the circuit will remain broken before attempting to reset.

BreakDurationGenerator · Func<BreakDurationGeneratorArguments, ValueTask<TimeSpan>> · default: "null"

Dynamically calculates the break duration using runtime information like failure count. If set, BreakDuration is ignored.

ManualControl · CircuitBreakerManualControl · default: "null"

Enables manual control of circuit state via IsolateAsync() and CloseAsync() methods.

StateProvider · CircuitBreakerStateProvider · default: "null"

Enables retrieving the current circuit state for health reporting and monitoring.

OnClosed · Func<OnCircuitClosedArguments, ValueTask> · default: "null"

Invoked after the circuit transitions to the Closed or Isolated state.

OnOpened · Func<OnCircuitOpenedArguments, ValueTask> · default: "null"

Invoked after the circuit transitions to the Open state.

OnHalfOpened · Func<OnCircuitHalfOpenedArguments, ValueTask> · default: "null"

Invoked after the circuit transitions to the HalfOpen state.

Best Practices

Combine with Retry

Place circuit breaker inside retry strategy. This allows retry to respect the broken circuit and fail fast when the circuit is open.

var pipeline = new ResiliencePipelineBuilder()
    .AddRetry(new RetryStrategyOptions())
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions())
    .Build();

Set appropriate thresholds

Balance between sensitivity and stability:

• Too sensitive: circuit breaks on minor issues
• Not sensitive enough: failing system gets overwhelmed

Start with defaults and tune based on your service’s behavior.

Monitor circuit state

Use StateProvider to expose circuit state in health checks and monitoring dashboards. This provides visibility into system health.

Use separate circuits per endpoint

Don’t use a single circuit breaker for multiple endpoints. Isolate failures by creating separate circuits for each dependency.

Consider break duration carefully

Too short: may not give the system enough time to recover Too long: increases user-perceived downtime

Use BreakDurationGenerator for adaptive break durations that increase with repeated failures.

Examples

HTTP Client with Circuit Breaker

var httpClient = new HttpClient();

var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions<HttpResponseMessage>
    {
        ShouldHandle = new PredicateBuilder<HttpResponseMessage>()
            .Handle<HttpRequestException>()
            .HandleResult(r => (int)r.StatusCode >= 500),
        FailureRatio = 0.3,
        SamplingDuration = TimeSpan.FromSeconds(30),
        MinimumThroughput = 10,
        BreakDuration = TimeSpan.FromSeconds(15),
        OnOpened = args =>
        {
            Console.WriteLine($"Circuit opened for {args.BreakDuration}");
            return default;
        }
    })
    .Build();

try
{
    var response = await pipeline.ExecuteAsync(async ct =>
        await httpClient.GetAsync("https://api.example.com/data", ct),
        cancellationToken);
}
catch (BrokenCircuitException)
{
    // Return cached data or default value
}

Circuit Breaker with Retry and Fallback

var pipeline = new ResiliencePipelineBuilder<string>()
    .AddFallback(new FallbackStrategyOptions<string>
    {
        ShouldHandle = new PredicateBuilder<string>()
            .Handle<BrokenCircuitException>()
            .Handle<HttpRequestException>(),
        FallbackAction = args => Outcome.FromResultAsValueTask("Cached value")
    })
    .AddRetry(new RetryStrategyOptions<string>
    {
        MaxRetryAttempts = 3,
        Delay = TimeSpan.FromSeconds(1)
    })
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions<string>
    {
        FailureRatio = 0.5,
        MinimumThroughput = 5,
        SamplingDuration = TimeSpan.FromSeconds(10)
    })
    .Build();

Health Check Integration

public class CircuitBreakerHealthCheck : IHealthCheck
{
    private readonly CircuitBreakerStateProvider _stateProvider;

    public CircuitBreakerHealthCheck(CircuitBreakerStateProvider stateProvider)
    {
        _stateProvider = stateProvider;
    }

    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        return _stateProvider.CircuitState switch
        {
            CircuitState.Closed => Task.FromResult(HealthCheckResult.Healthy("Circuit is closed")),
            CircuitState.HalfOpen => Task.FromResult(HealthCheckResult.Degraded("Circuit is half-open")),
            CircuitState.Open => Task.FromResult(HealthCheckResult.Unhealthy("Circuit is open")),
            CircuitState.Isolated => Task.FromResult(HealthCheckResult.Unhealthy("Circuit is isolated")),
            _ => Task.FromResult(HealthCheckResult.Unhealthy("Unknown state"))
        };
    }
}

What's the difference between Closed and Isolated states?

• Closed: Normal operation. Circuit monitors for failures and can automatically transition to Open.
• Isolated: Manually held open. Circuit won’t automatically close; requires manual intervention via CloseAsync().

Does the circuit breaker prevent exceptions from being thrown?

No. During normal operation (Closed state), exceptions are thrown as normal. When Open or Isolated, it throws BrokenCircuitException or IsolatedCircuitException instead.

How do I implement a circuit breaker per user/tenant?

Use the ResiliencePipelineProvider with keyed services or maintain a dictionary of circuit breakers per user/tenant identifier.

Can I use one circuit breaker for multiple endpoints?

It’s not recommended. Create separate circuit breakers for each dependency to isolate failures and avoid one failing service affecting others.