| | Medium Circuit Breakers in Microservices | 3 mins Site Reliability Engineering | Solve |
You are a site reliability engineer at a company that has recently migrated to a microservices architecture. The company has three microservices - Service A, Service B, and Service C. Service A receives the most traffic and consequently, experiences the most load. To manage the load, you've implemented a Round-Robin load balancer. Simultaneously, you've introduced a circuit breaker in Service B due to its dependency on an external service that occasionally experiences downtime.
Pseudo-code of the services:
The implemented load balancer and circuit breaker are working as expected. However, you've noticed that when the external service that Service B depends on experiences downtime, there's an increase in error rate and user complaints, as requests are still being routed to Service B.
To minimize the impact of the external service's downtime on your system, which of the following steps should you consider?
A: Modify the load balancer to send all traffic to Service B only when the circuit breaker is open.
B: Modify the load balancer to stop sending traffic to Service B when the circuit breaker is open.
C: Modify Service B to handle all requests, regardless of the state of the circuit breaker.
D: Modify the circuit breaker to open only when the external service is up.
E: Modify the circuit breaker to close only when the external service is down.
|
| | Medium Error Budget Management | 3 mins Site Reliability Engineering | Solve |
You are a site reliability engineer responsible for maintaining a microservices-based e-commerce platform. Your system consists of several independent services, each deployed on its separate container within a Kubernetes cluster.
Your organization follows a strict Service Level Objective (SLO) to maintain user satisfaction, which mandates that the 95th percentile latency for all requests over a 30-day period should not exceed 200 ms.
The following pseudo-code represents a simplified version of the request processing in your system:
You realize that over the first two weeks of the current 30-day window, the 95th percentile latency has risen to 250 ms. Analyzing further, you discover that out of 10 million requests, 600,000 requests took more than 200 ms to complete.
Given these facts, which of the following is the most effective course of action that you can take to troubleshoot and reduce the system's latency issues?
A: Change the latency log level to debug to gather more information.
B: Increase the SLO for latency to 250 ms to accommodate the current system performance.
C: Introduce more instances of each microservice to handle the increased load.
D: Implement a distributed tracing mechanism to identify the microservices contributing most to the latency.
E: Implement request throttling to reduce the overall number of requests.
|
| | Medium Incident Response Procedure | 3 mins Site Reliability Engineering | Solve |
You are an SRE for a large-scale distributed system. The system architecture includes five primary servers (P1 to P5) and three backup servers (B1 to B3). The system uses an advanced load balancer that distributes the workload across the primary servers evenly.
One day, the monitoring system triggers an alert that server P5 is not responding. The pseudo-code for the current incident response procedure is as follows:
The function 'replaceServer(server)' replaces the failed server with a new one from a pool of spare servers, which takes around 30 minutes.
The current discussion revolves around modifying this procedure to improve system resilience and minimize potential downtime. The backup servers are underutilized and could be leveraged more effectively. Also, the load balancer can dynamically shift workloads based on server availability and response time.
Based on the situation above, what is the best approach to optimize the incident response procedure?
A: Implement an early warning system to predict server failures and prevent them.
B: Upon failure detection, immediately divert traffic to backup servers, then attempt to reboot the primary server, and replace if necessary.
C: Replace the failed server without attempting a reboot and keep the traffic on primary servers.
D: Enable auto-scaling to add more servers when a primary server fails.
E: Switch to a more advanced load balancer that can detect and handle server failures independently.
|
| | Medium Service Balancer Decision-making | 2 mins Site Reliability Engineering | Solve |
You are a Site Reliability Engineer (SRE) working on a distributed system with a load balancer that distributes requests across a number of servers based on the current load. The decision algorithm for load balancing is written in pseudo-code as follows:
The system receives a large burst of requests. In response to this, some engineers propose increasing the `threshold` value to allow for more requests to be handled concurrently by each server. Others argue that instead, we should increase the number of servers to distribute the load more evenly.
Consider that the system has auto-scaling capabilities based on the average load of all servers, but the scaling operation takes about 15 minutes to add new servers to the pool. Also, the servers' performance degrades sharply if the load is much above the threshold.
One of the engineers also proposes modifying the getServer function logic to distribute the incoming load one by one across all servers to trigger the average load to rise faster.
Based on this scenario, what is the best approach?
A: Increase the `threshold` value to allow more requests on each server.
B: Add more servers to distribute the load, regardless of the auto-scaling delay.
C: Modify the getServer function to distribute the incoming load one by one across all servers to trigger the average load to rise faster.
D: Increase the `threshold` and add more servers simultaneously.
E: Manually trigger the auto-scaling process before the load increases.
|
