Circuit Breaker Middleware

The circuit breaker middleware protects your application from cascading failures when the backing KV store is unavailable or degraded. When consecutive failures exceed a threshold, the circuit "opens" and subsequent requests fail immediately with ErrCircuitOpen instead of waiting for timeouts.

Installation

import "github.com/xraph/grove/kv/middleware"

Usage

import (
    "time"

    "github.com/xraph/grove/kv"
    "github.com/xraph/grove/kv/middleware"
)

store, err := kv.Open(drv,
    kv.WithHook(middleware.NewCircuitBreaker(5, 30*time.Second)),
)

Constructor

func NewCircuitBreaker(threshold int, timeout time.Duration) *CircuitBreakerHook

Circuit States

The circuit breaker has three states:

         success
    +-----------------+
    |                 |
    v     failures    |
[Closed] ---------> [Open]
    ^                 |
    |    timeout      |
    |    elapsed      v
    +---------- [Half-Open]
     success     |
                 | failure
                 v
              [Open]

Closed (normal operation)

All requests pass through to the backend. Each failure increments an internal counter. When the counter reaches threshold, the circuit transitions to Open.

A successful operation resets the failure counter to zero.

Open (failing fast)

All requests are immediately rejected with middleware.ErrCircuitOpen. No traffic reaches the backend, giving it time to recover.

After timeout elapses, the circuit transitions to Half-Open.

Half-Open (testing recovery)

A limited number of requests (one by default) are allowed through to test whether the backend has recovered:

• If the test request succeeds, the circuit transitions back to Closed and normal operation resumes.
• If the test request fails, the circuit transitions back to Open and the timeout resets.

Error Handling

When the circuit is open, all operations return middleware.ErrCircuitOpen:

import "errors"

err := store.Get(ctx, "key", &value)
if errors.Is(err, middleware.ErrCircuitOpen) {
    // Backend is unavailable -- serve degraded response
    // or return a cached fallback
}

Recording Results

The circuit breaker exposes RecordSuccess() and RecordFailure() methods to update its state based on operation outcomes:

cb := middleware.NewCircuitBreaker(5, 30*time.Second)

// After a successful operation
cb.RecordSuccess()

// After a failed operation
cb.RecordFailure()

You can also inspect the current state:

state := cb.State()
switch state {
case middleware.StateClosed:
    // Normal operation
case middleware.StateOpen:
    // Circuit is open
case middleware.StateHalfOpen:
    // Testing recovery
}

Configuration Guidelines

Choose a threshold that reflects how many failures indicate a real outage versus transient blips. Choose a timeout that gives the backend enough time to recover before retesting.

Combining with Retry

Place the circuit breaker before the retry middleware so that retries do not count as separate failures against the circuit breaker threshold:

store, err := kv.Open(drv,
    kv.WithHook(middleware.NewCircuitBreaker(5, 30*time.Second)),
    kv.WithHook(middleware.NewRetry(3)),
)

When the circuit is open, the retry middleware never runs because the circuit breaker rejects the request in its PreQueryHook.