Micro Service Patterns : Best Practices for implementing Circuit Breaker with Spring Cloud.

It’s very obvious that the service you are building communicates with several external upstream services . To prevent this problem and to enhance the resilience and robustness of our service, the retry and circuit breaker mechanism needs to be implemented. The circuit breaker is used to prevent cascading failures when a service fails while the Retry mechanism will automatically retry failed operations or requests.

Need of Circuit Breaker

While recently working on the circuit breakers, i initially thought very hard on the point that what is the harm of retrying again and again or bombarding the upstream service with request calls. The problem as a result of above would be that our service requests to the upstream service until its resources are exhausted, resulting in our service failure.

Consider an example that we are building a LinkedIn application and multiple users needs to log in to that application and suppose one of the service is down i.e account service. The authentication service will wait on the account service and now a lot of user threads are waiting for a response thereby exhausting the CPU on the authentication service as well as the account service. As a result, the system cannot serve any of the users.

Note : Please check my previous article for the basics and implementation of circuit breaker and retry pattern in microservices.

Best Practices for implementing the circuit breaker pattern in microservices

The following are some of the best practices for implementing the circuit breaker pattern :

1. Register a health indicator to monitor circuit breaker states : One of the most important property to configure in circuit breaker is registerHealthIndicator property. With the help of this property we can register a health indicator with our application’s health endpoint. It gives the current metrics and states with your application’s health endpoint. This data is crucial for monitoring and debugging.
2. Add configuration to enable/disable circuit breaker in your application : There can be cases during integrating, debugging or testing where you might consider temporarily disabling a circuit breaker. Also when performing maintenance or upgrades on a service or component that the circuit breaker is protecting, you might temporarily disable the circuit breaker to ensure that no legitimate traffic is blocked during the maintenance window. For that there should be atleast a configuration to disable/enable the circuit. This can be achieved by using transitionToDisabledState() method provided by the resilience library. In the below case we have exposed an api to disable the circuit breaker. Here we can provide the circuit breaker name we want to disable which we mentioned in application.properties :

  @PutMapping(value = "/disable/{circuit_breaker}")
  @Hidden
  public void disableCircuitBreaker(
      @PathVariable("circuit_breaker") String circuitBreakerName) {
    try {
      circuitBreakerRegistry.circuitBreaker(circuitBreakerName).transitionToDisabledState();
    } catch (Exception exception) {
      log.error("Circuit Breaker with name '{}' doesn't exist", circuitBreakerName);
      throw new BadRequestException(String.format("%s doesn't exist", circuitBreakerName));
    }
    log.info(" {} circuit breaker is now in disabled state.", circuitBreakerName);
  }

Similarly to enable the circuit breaker we can again expose an api and can use transitionToClosedState() method.

  @PutMapping(value = "/enable/{circuit_breaker}")
  @Hidden
  public void enableCircuitBreaker(
      @PathVariable("circuit_breaker") String circuitBreakerName) {
    try {
      circuitBreakerRegistry.circuitBreaker(circuitBreakerName).transitionToClosedState();
    } catch (Exception exception) {
      log.error("Circuit Breaker with name '{}' doesn't exist", circuitBreakerName);
      throw new BadRequestException(String.format("%s doesn't exist", circuitBreakerName));
    }
    log.info(" {} circuit breaker is now in closed state.", circuitBreakerName);
  }

Let’s test this by hitting the disable API endpoint and then checking our health endpoint to see the state. As shown below we can see that the state has now been changed from CLOSED to DISABLED state.

In a similar way as above we can also consider externalizing circuit breaker configurations to make adjustments without modifying the code. Dynamic updates help in adapting to changing conditions.

3. Preventing our Circuit Breaker / RETRY from tripping due to certain Exceptions : There must be many application exceptions that the circuit breaker should ignore when determining whether to open or close the circuit. I have published a whole separate article explaining and implementing the circuit breaker to avoid the (4.x.x) Client error series exceptions by using the ErrorDecoder interface provided by the Feign Client library.

public class FeignClientDecoder implements ErrorDecoder {

  Logger log = LoggerFactory.getLogger(FeignClientDecoder.class);

  private final ErrorDecoder defaultErrorDecoder = new Default();

  @Override
  public Exception decode(String feignClientClassName, Response response) {
    if (response.status() >= HttpStatus.BAD_REQUEST.value()
        && response.status() < HttpStatus.INTERNAL_SERVER_ERROR.value()) {
      log.error(
          "FeignClientErrorDecoder response received for feign client {} with response: {}",
          feignClientClassName,
          response);
      return new ExternalServiceException("Bad Response from external services");
    }
    return defaultErrorDecoder.decode(feignClientClassName, response);
  }
}

4. Fallback Mechanism: Always implement a fallback mechanism that provides alternative data or functionality when the circuit is open. This ensures that users experience minimal disruption during failures. In our previous article we provided a very simple fallback mechanism as shown below :

default ResponseEntity<List<People>> serviceFallbackMethod(Throwable exception) {
    log.error(
        "Data server is either unavailable or malfunctioned due to {}", exception.getMessage());
    throw new RuntimeException(exception.getMessage());
  }

5. Version Compatibility: Ensure that the circuit breaker pattern is compatible with different versions of your microservices, as they may have different failure rates or performance characteristics. This is important because various versions of your microservices may have different failure rates, performance characteristics, or APIs, and you want to ensure that the circuit breaker can handle these variations appropriately.

6. Integration Tests with proper documentation : Document the usage and configuration properties of circuit breakers within your microservices architecture. All the circuit breaker state cases should be automated using integrations tests. It should also include all the failing scenarios in case our circuit trips.

Note : You can download the sample code from https://github.com/nitsbat/circuit-breaker/tree/main

Conclusion

By following the above best practices, you can effectively implement the circuit breaker pattern in your microservices architecture, improving system resilience and ensuring that failures in one microservice do not negatively impact the entire system.