Microservice Architecture #
No architecture is more misused than microservices. Teams new to it often arrive with expectations: independent deployment, per-service scaling, and team autonomy. What they get — if not prepared correctly — is all the complexity of a distributed system without a single one of its benefits: latency increases because every operation needs a network hop, debugging becomes a nightmare because errors spread across services without clear traces, and every new feature requires deployment coordination between three different teams. This is the so-called distributed monolith — a system that is physically distributed but logically tightly coupled. True Microservice Architecture requires more than just splitting an application into many services: it requires a database per service, firm domain boundaries, mature observability, and an organization structured in alignment with the system.
Characteristics of a Healthy Service #
Not every “service” is a good microservice. Four characteristics must be met:
flowchart TD
subgraph GOOD["Healthy microservice"]
S1["Single Business Capability\\nOne service = one business domain"]
S2["Own Database\\nNo shared database with other services"]
S3["Independent Deployable\\nCan deploy without coordinating other services"]
S4["Loosely Coupled\\nCommunication via API or events, not a shared DB"]
end
subgraph BAD["Distributed Monolith — not a microservice"]
B1["One service for all domains"]
B2["All services share one database"]
B3["Deploying A must wait for B"]
B4["Service A directly accesses service B's tables"]
end| Characteristic | Correct | Wrong (Distributed Monolith) |
|---|---|---|
| Scope | One business bounded context | Many domains mixed together |
| Database | One database per service | Shared database |
| Deployment | Independent — no coordination needed | Must deploy together |
| Communication | Via API contracts or events | Direct cross-service DB access |
| Size | Manageable by one small team | Team cannot see the responsibility boundaries |
System Topology #
flowchart TD
CLIENT["Client\\n(Web / Mobile)"]
GW["API Gateway\\n(routing, auth, rate-limit)"]
CLIENT --> GW
GW --> US["User Service\\n:8081\\nPostgreSQL"]
GW --> OS["Order Service\\n:8082\\nMySQL"]
GW --> PS["Payment Service\\n:8083\\nPostgreSQL"]
OS -->|"HTTP: validate user"| US
OS -->|"publish OrderPlaced"| MB[(Message Broker\\nKafka)]
MB -->|"OrderPlaced"| PS
MB -->|"OrderPlaced"| NS["Notification Service\\n:8084\\nRedis"]
PS -->|"publish PaymentProcessed"| MB
MB -->|"PaymentProcessed"| OS
subgraph OBS["Observability Stack"]
LOG["Centralized Logging\\n(ELK / Loki)"]
TRACE["Distributed Tracing\\n(Jaeger / Tempo)"]
METRIC["Metrics\\n(Prometheus / Grafana)"]
end
US & OS & PS & NS --> OBSInter-Service Communication: Sync vs Async #
One of the most important decisions in microservices is when to use synchronous communication (HTTP/gRPC) and when asynchronous (event/message broker):
flowchart LR
subgraph SYNC["Synchronous — HTTP/gRPC"]
A1["Order Service"] -->|"GET /users/{id}\\nvalidate user exists"| B1["User Service"]
B1 -->|"200 OK / 404 Not Found"| A1
NOTE1["✓ Needs an immediate response\\n✓ Queries current data\\n✗ Temporal coupling\\n✗ Cascade failure"]
end
subgraph ASYNC["Asynchronous — Event/Message"]
A2["Order Service"] -->|"OrderPlaced event"| MB2["Kafka"]
MB2 -->|"consume"| B2["Payment Service"]
MB2 -->|"consume"| C2["Notification Service"]
NOTE2["✓ Loose coupling\\n✓ High throughput\\n✓ Failure isolation\\n✗ Eventual consistency\\n✗ Harder debugging"]
endGuidelines for choosing:
| Use Sync (HTTP/gRPC) if | Use Async (Events) if |
|---|---|
| You need an immediate response to continue the operation | The operation can be processed in the background |
| Querying real-time, always-current data | The producer does not need to know the processing result |
| Simple request-response operations | Many consumers need to be notified |
| Cross-service validation before committing | Tolerant of eventual consistency |
Implementation: HTTP Client with Retry and Timeout #
When a service calls another service via HTTP, several concerns must be handled:
// pkg/httpclient/client.go — reusable HTTP client with timeout and retry
package httpclient
import (
"context"
"encoding/json"
"fmt"
"net/http"
"time"
)
// Config configures the HTTP client's behavior
type Config struct {
BaseURL string
Timeout time.Duration
MaxRetries int
RetryDelay time.Duration
}
// Client is an opinionated HTTP client for inter-service communication
type Client struct {
config Config
http *http.Client
}
func New(config Config) *Client {
if config.Timeout == 0 {
config.Timeout = 5 * time.Second
}
if config.MaxRetries == 0 {
config.MaxRetries = 3
}
if config.RetryDelay == 0 {
config.RetryDelay = 100 * time.Millisecond
}
return &Client{
config: config,
http: &http.Client{Timeout: config.Timeout},
}
}
// Get performs an HTTP GET with automatic retries for transient errors
func (c *Client) Get(ctx context.Context, path string, result interface{}) error {
url := c.config.BaseURL + path
var lastErr error
for attempt := 0; attempt <= c.config.MaxRetries; attempt++ {
if attempt > 0 {
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(c.config.RetryDelay * time.Duration(attempt)):
}
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return fmt.Errorf("failed to create request: %w", err)
}
// Propagate the trace ID for distributed tracing
if traceID := ctx.Value("trace_id"); traceID != nil {
req.Header.Set("X-Trace-ID", fmt.Sprint(traceID))
}
resp, err := c.http.Do(req)
if err != nil {
lastErr = err
continue // retry
}
defer resp.Body.Close()
// Do not retry for client errors (4xx)
if resp.StatusCode >= 400 && resp.StatusCode < 500 {
return fmt.Errorf("client error %d from %s", resp.StatusCode, url)
}
// Retry for server errors (5xx)
if resp.StatusCode >= 500 {
lastErr = fmt.Errorf("server error %d from %s", resp.StatusCode, url)
continue
}
if result != nil {
if err := json.NewDecoder(resp.Body).Decode(result); err != nil {
return fmt.Errorf("failed to decode response: %w", err)
}
}
return nil
}
return fmt.Errorf("after %d attempts: %w", c.config.MaxRetries, lastErr)
}
// internal/order/user_client.go — client for the User Service
package order
import (
"context"
"fmt"
"myapp/pkg/httpclient"
)
// UserServiceClient is the client for communicating with the User Service
type UserServiceClient struct {
client *httpclient.Client
}
func NewUserServiceClient(baseURL string) *UserServiceClient {
return &UserServiceClient{
client: httpclient.New(httpclient.Config{
BaseURL: baseURL,
Timeout: 3 * time.Second,
MaxRetries: 2,
}),
}
}
type userResponse struct {
ID string `json:"id"`
FullName string `json:"full_name"`
IsActive bool `json:"is_active"`
}
// ValidateUserActive calls the User Service to validate a user
func (c *UserServiceClient) ValidateUserActive(ctx context.Context, userID string) error {
var user userResponse
if err := c.client.Get(ctx, fmt.Sprintf("/users/%s", userID), &user); err != nil {
return fmt.Errorf("failed to validate user: %w", err)
}
if !user.IsActive {
return fmt.Errorf("user %s is inactive", userID)
}
return nil
}
Circuit Breaker Pattern #
Without a circuit breaker, one service’s failure can cause a cascade failure across the entire system. A service calling a down service will keep trying, exhausting thread pools, and eventually go down itself.
// pkg/circuitbreaker/breaker.go
package circuitbreaker
import (
"errors"
"sync"
"time"
)
type State int
const (
StateClosed State = iota // Normal: requests allowed
StateOpen // Open: requests rejected immediately
StateHalfOpen // Half-open: one trial request
)
var ErrCircuitOpen = errors.New("circuit breaker open — service unavailable")
type Breaker struct {
mu sync.Mutex
state State
failures int
maxFailures int
successCount int
lastFailure time.Time
openDuration time.Duration
}
func New(maxFailures int, openDuration time.Duration) *Breaker {
return &Breaker{
maxFailures: maxFailures,
openDuration: openDuration,
}
}
// Execute runs fn through the circuit breaker
func (b *Breaker) Execute(fn func() error) error {
b.mu.Lock()
state := b.currentState()
b.mu.Unlock()
if state == StateOpen {
return ErrCircuitOpen // ✓ fail fast — no attempt at all
}
err := fn()
b.mu.Lock()
defer b.mu.Unlock()
if err != nil {
b.recordFailure()
} else {
b.recordSuccess()
}
return err
}
func (b *Breaker) currentState() State {
if b.state == StateOpen {
if time.Since(b.lastFailure) > b.openDuration {
b.state = StateHalfOpen
b.successCount = 0
}
}
return b.state
}
func (b *Breaker) recordFailure() {
b.failures++
b.lastFailure = time.Now()
if b.failures >= b.maxFailures {
b.state = StateOpen
}
}
func (b *Breaker) recordSuccess() {
b.failures = 0
if b.state == StateHalfOpen {
b.successCount++
if b.successCount >= 2 {
b.state = StateClosed // ✓ recovered
}
}
}
stateDiagram-v2
[*] --> Closed : Start
Closed --> Closed : Request succeeds
Closed --> Open : Failures >= threshold
Open --> HalfOpen : After openDuration
HalfOpen --> Closed : Consecutive successes
HalfOpen --> Open : A failure occurs
Open --> Open : Requests rejected (ErrCircuitOpen)Observability: Logging, Tracing, and Metrics #
In microservices, observability is not an add-on feature — it is a prerequisite. Without mature observability, cross-service debugging is nearly impossible.
// internal/order/handler.go — structured logging with trace IDs
package order
import (
"encoding/json"
"log/slog"
"net/http"
"time"
"github.com/google/uuid"
)
// Middleware to inject trace IDs and structured logging
func TracingMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Generate or propagate the trace ID
traceID := r.Header.Get("X-Trace-ID")
if traceID == "" {
traceID = uuid.New().String()
}
// Inject into the context for propagation to downstream calls
ctx := context.WithValue(r.Context(), "trace_id", traceID)
// Set in the response header
w.Header().Set("X-Trace-ID", traceID)
start := time.Now()
rw := &responseWriter{ResponseWriter: w, statusCode: http.StatusOK}
next.ServeHTTP(rw, r.WithContext(ctx))
// Structured log for every request
slog.InfoContext(ctx, "request completed",
"trace_id", traceID,
"method", r.Method,
"path", r.URL.Path,
"status", rw.statusCode,
"duration_ms", time.Since(start).Milliseconds(),
"service", "order-service",
)
})
}
type responseWriter struct {
http.ResponseWriter
statusCode int
}
func (rw *responseWriter) WriteHeader(code int) {
rw.statusCode = code
rw.ResponseWriter.WriteHeader(code)
}
// Handler with structured logging
func (h *Handler) CreateOrder(w http.ResponseWriter, r *http.Request) {
traceID := r.Context().Value("trace_id")
var req CreateOrderRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
slog.WarnContext(r.Context(), "invalid request body",
"trace_id", traceID,
"error", err.Error(),
)
http.Error(w, "invalid request", http.StatusBadRequest)
return
}
slog.InfoContext(r.Context(), "creating order",
"trace_id", traceID,
"customer_id", req.CustomerID,
"item_count", len(req.Items),
)
output, err := h.service.CreateOrder(r.Context(), req.toInput())
if err != nil {
slog.ErrorContext(r.Context(), "failed to create order",
"trace_id", traceID,
"customer_id", req.CustomerID,
"error", err.Error(),
)
http.Error(w, err.Error(), http.StatusUnprocessableEntity)
return
}
slog.InfoContext(r.Context(), "order created successfully",
"trace_id", traceID,
"order_id", output.OrderID,
)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(output)
}
The three mandatory observability pillars:
flowchart LR
subgraph OBS["Mandatory Observability in Microservices"]
LOG["Structured Logging\\n• JSON format\\n• trace_id in every log\\n• levels: INFO, WARN, ERROR\\n• service name"]
TRACE["Distributed Tracing\\n• Trace ID propagated\\nacross services\\n• Span per operation\\n• Jaeger / Tempo"]
METRIC["Metrics\\n• Request rate\\n• Error rate\\n• Latency (p50, p95, p99)\\n• Queue depth\\n• Prometheus + Grafana"]
endDistributed Transactions: The Saga Pattern #
In microservices, there is no ACID transaction spanning multiple services. The solution is the Saga Pattern — a series of local transactions, each publishing an event to trigger the next transaction:
sequenceDiagram
participant OS as Order Service
participant MB as Kafka
participant PS as Payment Service
participant IS as Inventory Service
participant NS as Notification Service
Note over OS,NS: Happy Path — Choreography Saga
OS->>OS: CreateOrder (local transaction)
OS->>MB: publish OrderCreated
MB->>PS: OrderCreated
PS->>PS: ProcessPayment (local transaction)
PS->>MB: publish PaymentProcessed
MB->>IS: PaymentProcessed
IS->>IS: ReserveInventory (local transaction)
IS->>MB: publish InventoryReserved
MB->>NS: InventoryReserved
NS->>NS: SendConfirmation
Note over OS,NS: Compensating Transaction — if Payment Fails
PS->>MB: publish PaymentFailed
MB->>OS: PaymentFailed
OS->>OS: CancelOrder (compensating transaction)
OS->>MB: publish OrderCancelled
MB->>NS: OrderCancelled
NS->>NS: SendCancellationNotice// internal/order/saga_handler.go — handling results from other services
package order
import (
"context"
"log/slog"
"myapp/internal/shared/eventbus"
)
// PaymentFailedEvent is received from the Payment Service
type PaymentFailedEvent struct {
OrderID string
Reason string
}
func (e PaymentFailedEvent) EventName() string { return "payment.failed" }
// SagaHandler handles compensating transactions
type SagaHandler struct {
repo Repository
eventBus *eventbus.EventBus
}
// HandlePaymentFailed is the compensating transaction for an order
func (h *SagaHandler) HandlePaymentFailed(ctx context.Context, event eventbus.Event) error {
e, ok := event.(PaymentFailedEvent)
if !ok {
return nil
}
slog.InfoContext(ctx, "payment failed, cancelling order",
"order_id", e.OrderID,
"reason", e.Reason,
)
order, err := h.repo.FindByID(ctx, e.OrderID)
if err != nil {
return err
}
// Compensating transaction: cancel the order
if err := order.Cancel("payment failed: " + e.Reason); err != nil {
return err
}
if err := h.repo.Save(ctx, order); err != nil {
return err
}
// Publish the compensation event
return h.eventBus.Publish(ctx, OrderCancelledEvent{
OrderID: e.OrderID,
Reason: "payment failed",
})
}
Anti-Patterns to Avoid #
// ✗ Shared database between services — distributed monolith
// The Order Service queries the User Service's database directly
func (r *OrderRepo) GetOrderWithUser(ctx context.Context, orderID string) (*OrderWithUser, error) {
// ✗ JOIN into another service's database — the most dangerous coupling
return r.db.QueryContext(ctx, `
SELECT o.id, u.full_name
FROM order_db.orders o
JOIN user_db.users u ON u.id = o.customer_id
WHERE o.id = $1
`, orderID)
}
// ✓ Each service only accesses its own database
// Data from other services is fetched via API calls
func (s *OrderService) GetOrderWithUser(ctx context.Context, orderID string) (*OrderDisplay, error) {
order, err := s.repo.FindByID(ctx, orderID)
if err != nil {
return nil, err
}
// Fetch the user's name from the User Service via HTTP
user, err := s.userClient.GetUser(ctx, order.CustomerID)
if err != nil {
// Graceful degradation: show the order even if the user is unavailable
return &OrderDisplay{OrderID: orderID, CustomerName: "Unknown"}, nil
}
return &OrderDisplay{OrderID: orderID, CustomerName: user.FullName}, nil
}
// ✗ Overly long synchronous chains — cascade failure
// Order → Inventory → Warehouse → Supplier (all sync)
func (s *OrderService) CreateOrder(ctx context.Context, req CreateOrderInput) error {
// ✗ 4 sync hops — if Supplier is slow, the whole order creation is slow
inventory := s.inventoryClient.CheckStock(ctx, req.Items)
warehouse := s.warehouseClient.Reserve(ctx, inventory)
supplier := s.supplierClient.Confirm(ctx, warehouse)
_ = supplier
return s.repo.Save(ctx, buildOrder(req))
}
// ✓ Only critical validation is sync, the rest is async via events
func (s *OrderService) CreateOrder(ctx context.Context, req CreateOrderInput) error {
// ✓ Only user validation is sync (needs an immediate response)
if err := s.userClient.ValidateActive(ctx, req.CustomerID); err != nil {
return err
}
order := buildOrder(req)
if err := s.repo.Save(ctx, order); err != nil {
return err
}
// ✓ The rest is async — Inventory, Warehouse, Supplier respond via events
return s.eventBus.Publish(ctx, OrderCreatedEvent{OrderID: order.ID, Items: req.Items})
}
// ✗ No timeout on inter-service calls — goroutines wait forever
func (s *Service) getUser(ctx context.Context, id string) (*User, error) {
resp, err := http.Get("http://user-service/users/" + id) // ✗ no timeout!
// ...
}
// ✓ Always use a context with a timeout
func (s *Service) getUser(ctx context.Context, id string) (*User, error) {
ctx, cancel := context.WithTimeout(ctx, 2*time.Second) // ✓ bounded timeout
defer cancel()
return s.userClient.Get(ctx, id)
}
Microservices Readiness Checklist #
ORGANIZATIONAL PREREQUISITES:
□ Each service is owned by one clear team (2–8 people)
□ The team can deploy its service without coordinating with other teams
□ On-call rotation exists for every service in production
□ The business domain is well understood — bounded contexts are clear
TECHNICAL PREREQUISITES:
□ CI/CD pipelines are mature and automated per service
□ Container (Docker) and orchestration (Kubernetes) are mastered by the team
□ Centralized logging exists (ELK, Loki, or similar)
□ Distributed tracing is configured (Jaeger, Tempo)
□ Metrics and alerting exist (Prometheus, Grafana)
□ Health check endpoints are available on every service
SERVICE DESIGN:
□ One database per service — no shared databases
□ API contracts are documented (OpenAPI, protobuf)
□ Clear API versioning — no breaking changes without a new version
□ Timeouts and retries are configured on all inter-service calls
□ Circuit breakers are installed for unreliable dependencies
RESILIENCE:
□ Services can run even when dependencies are unavailable (graceful degradation)
□ Compensating transactions (Saga) are designed for all distributed workflows
□ Idempotency keys are applied to retryable operations
□ Dead letter queues are configured for events that fail to process
TESTING:
□ Per-service unit tests can run without other services
□ Contract tests verify API contracts between services
□ Integration tests exist for the main flows
□ Chaos engineering or fault injection is performed periodically
When Microservices, When Not #
Microservices are justified by real pain:
✓ Deployment bottleneck: Team A cannot deploy because it waits for Team B
✓ Scaling bottleneck: Module X needs 10x the resources of other modules
✓ Technology bottleneck: An ML module needs Python, the rest Go
✓ Reliability bottleneck: A bug in one module often crashes other modules
✓ The organization already has > 20 developers with clear domain ownership
Avoid microservices if:
✗ Choosing microservices to learn, not because of real need
✗ Team < 10 developers — coordination overhead exceeds the benefit
✗ The business domain is not yet understood — wrong service boundaries are very expensive
✗ The observability stack does not exist — debugging will be a nightmare
✗ CI/CD is not automated — manually deploying N services is unsustainable
Summary #
- Microservices are not a goal, but a solution to specific problems — choose microservices when there is real pain that a better monolith cannot solve.
- The distributed monolith is the biggest trap — services that are physically distributed but still share a database or are logically tightly coupled get all of distribution’s downsides without its benefits.
- A database per service is a non-negotiable rule — sharing a database between services creates the hardest-to-break and most dangerous coupling.
- Sync communication for critical validation, async for side effects — avoid overly long synchronous chains; use events for operations that do not need an immediate response.
- Circuit breakers are mandatory — without a circuit breaker, one service’s failure causes a cascade failure across the whole system.
- Observability is a prerequisite, not a feature — structured logging with trace IDs, distributed tracing, and metrics must exist before a service goes to production.
- The Saga pattern for distributed transactions — there is no cross-service ACID; design compensating transactions for every workflow spanning multiple services.
- Timeouts on all inter-service calls — goroutines waiting without a limit will exhaust thread pools; every HTTP/gRPC call to another service must have a timeout.
- Start from a modular monolith, extract when there is real need — understanding the domain well in a monolith before splitting it produces far more accurate service boundaries.
- Conway’s Law works both ways — organizational structure influences system architecture; if teams cannot work independently, services will not be deployable independently either.