Day 8: Circuit Breakers & Health Checks

• ~5 min read

What I Built

  • Circuit breaker pattern for IBKR, database, and market data services
  • Health check endpoints (/api/health liveness, /api/health/ready comprehensive)
  • Graceful shutdown handler with signal handling and pending order cancellation
  • Service status monitoring (healthy/degraded/unhealthy states)
  • Automatic recovery testing after circuit breaker timeout

Code Highlight

# Circuit Breaker with automatic recovery
class CircuitBreaker:
    async def call(self, func: Callable, *args, **kwargs) -> Any:
        if self.state == CircuitBreakerState.OPEN:
            if self._should_attempt_reset():
                self._half_open_circuit()  # Test recovery
            else:
                raise CircuitBreakerOpenException("Circuit breaker is open")

        try:
            result = await asyncio.wait_for(func(*args, **kwargs), timeout=self.timeout)
            self._record_success()
            return result
        except self.expected_exception as e:
            self._record_failure()
            raise e

# Health monitoring with service status
class HealthCheck:
    async def check_database(self, session) -> ServiceStatus:
        try:
            await session.execute("SELECT 1")
            return ServiceStatus.HEALTHY
        except Exception as e:
            return ServiceStatus.UNHEALTHY

Architecture Decision

Circuit breakers prevent cascading failures by failing fast when services are down, while health checks provide operational visibility. The half-open state allows automatic recovery testing, and graceful shutdown ensures no orders are left hanging during deployments.

Testing Results

All 9 circuit breaker tests pass, covering failure scenarios:

  • Circuit opens after failure threshold (3 failures)
  • Automatic recovery after timeout (half-open testing)
  • Success/failure handling in half-open state
  • Timeout protection for external calls
  • Proper state transitions (closed → open → half-open → closed)

Next Steps

Day 9: Error handling and retry logic with exponential backoff.

Follow @therealkamba on X for regular updates. View all posts →