Load Balancers & Nginx v1AdvancedLoad Balancers & Nginx
Solution architect guide to load balancing and Nginx implementation logic: traffic strategy, HA design, failover, and production trade-offs.
Foundations
2 topics
L4 vs L7 Decision Logic
Pick the right balancing layer based on protocol and routing needs
WHEN TO USE L4 (Transport Layer): - Protocols: TCP/UDP - Need: ultra-low latency, high throughput, minimal packet inspection - Common use: DB traffic, gRPC at scale, game servers WHEN TO USE L7 (Application Layer): - Protocols: HTTP/HTTPS/WebSocket - Need: path/host/header routing, auth-aware policies, canary releases - Common use: web apps, APIs, multi-tenant SaaS SOLUTION ARCHITECT RULE: If you need request-level intelligence, choose L7. If you need raw performance and simple distribution, choose L4.
💡 Start with L7 for APIs unless latency/throughput constraints force L4
⚡ L4 often scales harder, L7 offers richer routing control
Topology Patterns
Single LB, active-active, and global traffic entry patterns
PATTERN 1: Single LB + Multi-AZ targets Internet -> LB -> App nodes in AZ-A/AZ-B Use for: small-to-mid systems with HA requirements PATTERN 2: Active-Active Regional Users -> Global DNS/GSLB -> Region A LB + Region B LB Use for: low RTO, geo-distributed users PATTERN 3: Edge + Internal LB tier CDN/WAF -> Public LB -> Services -> Internal LB -> private services Use for: zero-trust segmentation, layered security
💡 Keep LB and targets spread across failure domains
📌 Cross-zone balancing improves resilience but may increase data transfer cost
Traffic Distribution Logic
2 topics
Balancing Algorithms
Choose algorithm based on request profile and node variance
ROUND ROBIN - Even request distribution - Best when backend capacity is similar LEAST CONNECTIONS - Routes to instance with fewest active requests - Best for long-lived or variable-duration requests WEIGHTED ROUND ROBIN - Distributes by capacity weights - Best when instances have different sizes IP HASH / CONSISTENT HASH - Same client tends to same backend - Best for sticky affinity and cache locality SOLUTION ARCHITECT RULE: Default to Least Connections for uneven request duration; use Weighted RR when nodes differ in CPU/memory sizing.
⚡ Revisit algorithm after real traffic shape is known
💡 Consistent hash reduces cache misses in distributed caching layers
Session Strategy (Sticky vs Stateless)
Avoid hidden coupling in horizontally scaled services
OPTION A: STICKY SESSIONS Pros: simple migration from legacy apps Cons: uneven load, poor failover behavior OPTION B: STATELESS APP + SHARED SESSION STORE (recommended) Pros: true horizontal scale, cleaner failover, easier autoscaling Cons: requires external session/state layer (Redis, DB) DECISION: - Greenfield: design stateless first - Brownfield: use stickiness as temporary bridge, then remove
⚠️ Sticky sessions can mask architecture debt
💡 For APIs, prefer token-based auth to stay stateless
Health Checks & Failover
2 topics
Health Check Design
Use probes that reflect real readiness, not just process liveness
LIVENESS CHECK - Confirms process is running - Example: /healthz returns 200 if process alive READINESS CHECK - Confirms app can serve traffic now - Validates DB/cache/dependency reachability BEST PRACTICE: - Keep thresholds conservative to avoid flapping - Remove unhealthy nodes quickly, add back cautiously EXAMPLE THRESHOLDS: - interval: 10s - timeout: 3s - unhealthy threshold: 3 - healthy threshold: 5
📌 Readiness checks should fail fast during dependency outages
⚡ Add jitter/backoff to avoid synchronized retries during incidents
Failure Routing Logic
How traffic should shift during partial and regional failures
FAILURE SCENARIO A: Single node failure Action: LB drains and removes node, traffic shifts to healthy pool FAILURE SCENARIO B: AZ failure Action: Shift to surviving AZ with pre-provisioned headroom FAILURE SCENARIO C: Regional failure Action: DNS/GSLB failover to secondary region ARCHITECT CHECKLIST: - Define RTO/RPO targets - Validate cross-region data strategy - Run controlled failover drills quarterly
💡 Failover without drills is only a theory
⚠️ Capacity planning must assume one domain is down
Nginx Implementation Logic
3 topics
Upstream Design (Round Robin, Least Conn, Weights)
Practical Nginx upstream patterns aligned to backend capacity
# /etc/nginx/conf.d/api.conf
upstream api_pool {
# Default algorithm is round robin
server 10.0.1.11:8080 weight=3 max_fails=3 fail_timeout=15s;
server 10.0.1.12:8080 weight=2 max_fails=3 fail_timeout=15s;
server 10.0.1.13:8080 backup;
least_conn; # choose this for variable request duration
}
server {
listen 80;
server_name api.example.com;
location / {
proxy_pass http://api_pool;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}💡 Use `least_conn` for uneven request duration APIs
⚡ `backup` nodes are useful for burst/failure scenarios
📌 Weight ratios should reflect real benchmarked capacity
Failover, Timeouts, and Retry Policy
Prevent cascading failures with bounded retries and strict timeouts
upstream app_pool {
server app1.internal:8080 max_fails=2 fail_timeout=10s;
server app2.internal:8080 max_fails=2 fail_timeout=10s;
}
server {
listen 443 ssl;
server_name app.example.com;
location / {
proxy_pass http://app_pool;
# Keep timeout budget tight to protect user latency
proxy_connect_timeout 2s;
proxy_send_timeout 10s;
proxy_read_timeout 10s;
# Retry only on safe conditions
proxy_next_upstream error timeout http_502 http_503 http_504;
proxy_next_upstream_tries 2;
proxy_next_upstream_timeout 3s;
}
}⚠️ Unlimited retries amplify outages
💡 Set timeout budgets from your p95/p99 SLO targets
📌 Retries are safer for idempotent requests than for writes
Canary and Blue/Green Routing (Architect Pattern)
Use deterministic routing rules for low-risk incremental releases
# Canary by header or cookie (simple example)
map $http_x_release_track $target_upstream {
default stable_pool;
canary canary_pool;
}
upstream stable_pool {
server stable1.internal:8080;
server stable2.internal:8080;
}
upstream canary_pool {
server canary1.internal:8080;
}
server {
listen 80;
server_name rollout.example.com;
location / {
proxy_pass http://$target_upstream;
add_header X-Upstream $target_upstream always;
}
}
# Architect rollout flow:
# 1) Internal traffic -> canary
# 2) 5% external traffic
# 3) 25% / 50% / 100% if SLO stays healthy
# 4) Instant rollback by mapping canary -> stable💡 Keep canary cohorts deterministic for debugging
⚡ Roll forward only when latency + error budget remain healthy
📌 Blue/green cutover should be one config switch plus health validation
Security & Observability
2 topics
TLS and Edge Security
Plan TLS termination and trust boundaries explicitly
TLS TERMINATION OPTIONS: 1) Terminate at LB (common): simple cert management, L7 controls 2) Pass-through TLS: end-to-end encryption to backend 3) Re-encrypt: terminate at LB and re-encrypt to services EDGE CONTROLS: - WAF rules - Rate limiting - Geo/IP policies - Bot mitigation RULE: Document where plaintext exists in-flight and keep it minimal.
💡 Use automated certificate rotation
📌 Enforce modern TLS versions and ciphers centrally
SLO Metrics for Load Balancing
Metrics that indicate balancing quality and user impact
CORE METRICS: - request rate (RPS) - p50/p95/p99 latency - backend 4xx/5xx - connection errors / resets - healthy host count - spillover / queue depth ARCHITECT VIEW: - Watch p99 and error rate first (user pain signals) - Correlate LB metrics with app and DB saturation - Alert on burn-rate against SLO, not only raw thresholds
⚡ Golden signals: latency, traffic, errors, saturation
💡 Keep dashboards per service and per region
Capacity Planning
2 topics
Sizing and Headroom Logic
Plan for growth and failure-domain loss at the same time
BASE FORMULA: Required capacity = peak load * safety factor SUGGESTED SAFETY FACTOR: - steady workloads: 1.3x - bursty workloads: 1.5x to 2x N+1 RULE FOR HA: If one zone/node fails, remaining capacity must still handle SLO traffic. AUTOSCALING POLICY: - scale out before CPU reaches sustained saturation - use latency and queue depth as primary scale signals for APIs
📌 Capacity without failure headroom is not true HA
💡 Recalculate sizing after major feature launches
Reference Architecture Blueprint
A pragmatic baseline for most SaaS workloads
REFERENCE STACK: Internet -> CDN + WAF -> Public L7 LB -> Stateless API tier (multi-AZ) -> Internal L4/L7 LB -> Stateful services (DB, cache, queues) -> Observability stack (logs/metrics/traces) RELEASE PATTERN: - Blue/green or canary at LB rule level - Automated rollback on SLO breach
⚡ Keep release controls close to LB routing rules
💡 Standardize this as a platform template for teams
Related Cheat Sheets
More hands-on references connected to this topic.
Agentic AIComplete guide to building AI agents — architecture, tool use, memory, planning patterns, multi-agent systems, frameworks, evaluation, and production best practices.
same difficulty, keyword overlap
Pandassame difficulty, keyword overlap
Kuberneteskubectl commands, Pod/Deployment/Service manifests, config management, and cluster operations.
same difficulty, keyword overlap
Cloudflarekeyword overlap
Related Articles
Background reading and deeper explanations for this sheet.
Agent Routing vs Model Routing: Real Production Patterns, Tradeoffs, and Implementation Guide
Choosing between agent routing and model routing can make or break your AI product’s cost, speed, and reliability. In this practical guide, you’ll learn what each pattern is, when to use it, and how production teams combine both to ship robust systems. We’ll walk through real examples, architecture patterns, and code you can adapt immediately.
keyword overlap
Cost Optimization in Agentic Systems: The Most Underrated Lever for Reliable AI at Scale
Most teams building agentic systems obsess over accuracy and latency, but ignore the fastest path to sustainable scale: cost optimization. In this guide, you’ll learn practical cost models, real architecture patterns, and implementation tactics to cut spend without sacrificing quality or autonomy.
keyword overlap
GenAI SaaS Architecture: A Practical Blueprint for Building, Scaling, and Securing AI Products
Designing a SaaS product on top of GenAI is more than calling an LLM API—it requires thoughtful architecture across product, data, safety, and operations. This practical guide walks you through a beginner-friendly yet deep system blueprint, with real-world patterns, code snippets, and deployment strategies you can use immediately.
keyword overlap
How to Design a Production-Ready LLM System: Architecture, Guardrails, and Scale
Building an LLM demo is easy; building one that survives real traffic, strict latency targets, and compliance audits is not. This guide walks through a real production architecture, design decisions, and implementation patterns for reliable, secure, and cost-efficient LLM systems.
keyword overlap