Microservice Architecture #

No architecture is more misused than microservices. Teams new to it often arrive with expectations: independent deployment, per-service scaling, and team autonomy. What they get — if not prepared correctly — is all the complexity of a distributed system without a single one of its benefits: latency increases because every operation needs a network hop, debugging becomes a nightmare because errors spread across services without clear traces, and every new feature requires deployment coordination between three different teams. This is the so-called distributed monolith — a system that is physically distributed but logically tightly coupled. True Microservice Architecture requires more than just splitting an application into many services: it requires a database per service, firm domain boundaries, mature observability, and an organization structured in alignment with the system.

Characteristics of a Healthy Service #

Not every “service” is a good microservice. Four characteristics must be met:

flowchart TD
    subgraph GOOD["Healthy microservice"]
        S1["Single Business Capability\\nOne service = one business domain"]
        S2["Own Database\\nNo shared database with other services"]
        S3["Independent Deployable\\nCan deploy without coordinating other services"]
        S4["Loosely Coupled\\nCommunication via API or events, not a shared DB"]
    end

    subgraph BAD["Distributed Monolith — not a microservice"]
        B1["One service for all domains"]
        B2["All services share one database"]
        B3["Deploying A must wait for B"]
        B4["Service A directly accesses service B's tables"]
    end
CharacteristicCorrectWrong (Distributed Monolith)
ScopeOne business bounded contextMany domains mixed together
DatabaseOne database per serviceShared database
DeploymentIndependent — no coordination neededMust deploy together
CommunicationVia API contracts or eventsDirect cross-service DB access
SizeManageable by one small teamTeam cannot see the responsibility boundaries

System Topology #

flowchart TD
    CLIENT["Client\\n(Web / Mobile)"]
    GW["API Gateway\\n(routing, auth, rate-limit)"]

    CLIENT --> GW

    GW --> US["User Service\\n:8081\\nPostgreSQL"]
    GW --> OS["Order Service\\n:8082\\nMySQL"]
    GW --> PS["Payment Service\\n:8083\\nPostgreSQL"]

    OS -->|"HTTP: validate user"| US
    OS -->|"publish OrderPlaced"| MB[(Message Broker\\nKafka)]
    MB -->|"OrderPlaced"| PS
    MB -->|"OrderPlaced"| NS["Notification Service\\n:8084\\nRedis"]
    PS -->|"publish PaymentProcessed"| MB
    MB -->|"PaymentProcessed"| OS

    subgraph OBS["Observability Stack"]
        LOG["Centralized Logging\\n(ELK / Loki)"]
        TRACE["Distributed Tracing\\n(Jaeger / Tempo)"]
        METRIC["Metrics\\n(Prometheus / Grafana)"]
    end

    US & OS & PS & NS --> OBS

Inter-Service Communication: Sync vs Async #

One of the most important decisions in microservices is when to use synchronous communication (HTTP/gRPC) and when asynchronous (event/message broker):

flowchart LR
    subgraph SYNC["Synchronous — HTTP/gRPC"]
        A1["Order Service"] -->|"GET /users/{id}\\nvalidate user exists"| B1["User Service"]
        B1 -->|"200 OK / 404 Not Found"| A1
        NOTE1["✓ Needs an immediate response\\n✓ Queries current data\\n✗ Temporal coupling\\n✗ Cascade failure"]
    end

    subgraph ASYNC["Asynchronous — Event/Message"]
        A2["Order Service"] -->|"OrderPlaced event"| MB2["Kafka"]
        MB2 -->|"consume"| B2["Payment Service"]
        MB2 -->|"consume"| C2["Notification Service"]
        NOTE2["✓ Loose coupling\\n✓ High throughput\\n✓ Failure isolation\\n✗ Eventual consistency\\n✗ Harder debugging"]
    end

Guidelines for choosing:

Use Sync (HTTP/gRPC) ifUse Async (Events) if
You need an immediate response to continue the operationThe operation can be processed in the background
Querying real-time, always-current dataThe producer does not need to know the processing result
Simple request-response operationsMany consumers need to be notified
Cross-service validation before committingTolerant of eventual consistency

Implementation: HTTP Client with Retry and Timeout #

When a service calls another service via HTTP, several concerns must be handled:

// pkg/httpclient/client.go — reusable HTTP client with timeout and retry
package httpclient

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"time"
)

// Config configures the HTTP client's behavior
type Config struct {
	BaseURL    string
	Timeout    time.Duration
	MaxRetries int
	RetryDelay time.Duration
}

// Client is an opinionated HTTP client for inter-service communication
type Client struct {
	config Config
	http   *http.Client
}

func New(config Config) *Client {
	if config.Timeout == 0 {
		config.Timeout = 5 * time.Second
	}
	if config.MaxRetries == 0 {
		config.MaxRetries = 3
	}
	if config.RetryDelay == 0 {
		config.RetryDelay = 100 * time.Millisecond
	}

	return &Client{
		config: config,
		http:   &http.Client{Timeout: config.Timeout},
	}
}

// Get performs an HTTP GET with automatic retries for transient errors
func (c *Client) Get(ctx context.Context, path string, result interface{}) error {
	url := c.config.BaseURL + path
	var lastErr error

	for attempt := 0; attempt <= c.config.MaxRetries; attempt++ {
		if attempt > 0 {
			select {
			case <-ctx.Done():
				return ctx.Err()
			case <-time.After(c.config.RetryDelay * time.Duration(attempt)):
			}
		}

		req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
		if err != nil {
			return fmt.Errorf("failed to create request: %w", err)
		}

		// Propagate the trace ID for distributed tracing
		if traceID := ctx.Value("trace_id"); traceID != nil {
			req.Header.Set("X-Trace-ID", fmt.Sprint(traceID))
		}

		resp, err := c.http.Do(req)
		if err != nil {
			lastErr = err
			continue // retry
		}
		defer resp.Body.Close()

		// Do not retry for client errors (4xx)
		if resp.StatusCode >= 400 && resp.StatusCode < 500 {
			return fmt.Errorf("client error %d from %s", resp.StatusCode, url)
		}

		// Retry for server errors (5xx)
		if resp.StatusCode >= 500 {
			lastErr = fmt.Errorf("server error %d from %s", resp.StatusCode, url)
			continue
		}

		if result != nil {
			if err := json.NewDecoder(resp.Body).Decode(result); err != nil {
				return fmt.Errorf("failed to decode response: %w", err)
			}
		}
		return nil
	}

	return fmt.Errorf("after %d attempts: %w", c.config.MaxRetries, lastErr)
}
// internal/order/user_client.go — client for the User Service
package order

import (
	"context"
	"fmt"

	"myapp/pkg/httpclient"
)

// UserServiceClient is the client for communicating with the User Service
type UserServiceClient struct {
	client *httpclient.Client
}

func NewUserServiceClient(baseURL string) *UserServiceClient {
	return &UserServiceClient{
		client: httpclient.New(httpclient.Config{
			BaseURL:    baseURL,
			Timeout:    3 * time.Second,
			MaxRetries: 2,
		}),
	}
}

type userResponse struct {
	ID       string `json:"id"`
	FullName string `json:"full_name"`
	IsActive bool   `json:"is_active"`
}

// ValidateUserActive calls the User Service to validate a user
func (c *UserServiceClient) ValidateUserActive(ctx context.Context, userID string) error {
	var user userResponse
	if err := c.client.Get(ctx, fmt.Sprintf("/users/%s", userID), &user); err != nil {
		return fmt.Errorf("failed to validate user: %w", err)
	}
	if !user.IsActive {
		return fmt.Errorf("user %s is inactive", userID)
	}
	return nil
}

Circuit Breaker Pattern #

Without a circuit breaker, one service’s failure can cause a cascade failure across the entire system. A service calling a down service will keep trying, exhausting thread pools, and eventually go down itself.

// pkg/circuitbreaker/breaker.go
package circuitbreaker

import (
	"errors"
	"sync"
	"time"
)

type State int

const (
	StateClosed   State = iota // Normal: requests allowed
	StateOpen                  // Open: requests rejected immediately
	StateHalfOpen              // Half-open: one trial request
)

var ErrCircuitOpen = errors.New("circuit breaker open — service unavailable")

type Breaker struct {
	mu           sync.Mutex
	state        State
	failures     int
	maxFailures  int
	successCount int
	lastFailure  time.Time
	openDuration time.Duration
}

func New(maxFailures int, openDuration time.Duration) *Breaker {
	return &Breaker{
		maxFailures:  maxFailures,
		openDuration: openDuration,
	}
}

// Execute runs fn through the circuit breaker
func (b *Breaker) Execute(fn func() error) error {
	b.mu.Lock()
	state := b.currentState()
	b.mu.Unlock()

	if state == StateOpen {
		return ErrCircuitOpen // ✓ fail fast — no attempt at all
	}

	err := fn()

	b.mu.Lock()
	defer b.mu.Unlock()

	if err != nil {
		b.recordFailure()
	} else {
		b.recordSuccess()
	}
	return err
}

func (b *Breaker) currentState() State {
	if b.state == StateOpen {
		if time.Since(b.lastFailure) > b.openDuration {
			b.state = StateHalfOpen
			b.successCount = 0
		}
	}
	return b.state
}

func (b *Breaker) recordFailure() {
	b.failures++
	b.lastFailure = time.Now()
	if b.failures >= b.maxFailures {
		b.state = StateOpen
	}
}

func (b *Breaker) recordSuccess() {
	b.failures = 0
	if b.state == StateHalfOpen {
		b.successCount++
		if b.successCount >= 2 {
			b.state = StateClosed // ✓ recovered
		}
	}
}
stateDiagram-v2
    [*] --> Closed : Start
    Closed --> Closed : Request succeeds
    Closed --> Open : Failures >= threshold
    Open --> HalfOpen : After openDuration
    HalfOpen --> Closed : Consecutive successes
    HalfOpen --> Open : A failure occurs
    Open --> Open : Requests rejected (ErrCircuitOpen)

Observability: Logging, Tracing, and Metrics #

In microservices, observability is not an add-on feature — it is a prerequisite. Without mature observability, cross-service debugging is nearly impossible.

// internal/order/handler.go — structured logging with trace IDs
package order

import (
	"encoding/json"
	"log/slog"
	"net/http"
	"time"

	"github.com/google/uuid"
)

// Middleware to inject trace IDs and structured logging
func TracingMiddleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// Generate or propagate the trace ID
		traceID := r.Header.Get("X-Trace-ID")
		if traceID == "" {
			traceID = uuid.New().String()
		}

		// Inject into the context for propagation to downstream calls
		ctx := context.WithValue(r.Context(), "trace_id", traceID)

		// Set in the response header
		w.Header().Set("X-Trace-ID", traceID)

		start := time.Now()
		rw := &responseWriter{ResponseWriter: w, statusCode: http.StatusOK}

		next.ServeHTTP(rw, r.WithContext(ctx))

		// Structured log for every request
		slog.InfoContext(ctx, "request completed",
			"trace_id", traceID,
			"method", r.Method,
			"path", r.URL.Path,
			"status", rw.statusCode,
			"duration_ms", time.Since(start).Milliseconds(),
			"service", "order-service",
		)
	})
}

type responseWriter struct {
	http.ResponseWriter
	statusCode int
}

func (rw *responseWriter) WriteHeader(code int) {
	rw.statusCode = code
	rw.ResponseWriter.WriteHeader(code)
}

// Handler with structured logging
func (h *Handler) CreateOrder(w http.ResponseWriter, r *http.Request) {
	traceID := r.Context().Value("trace_id")

	var req CreateOrderRequest
	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
		slog.WarnContext(r.Context(), "invalid request body",
			"trace_id", traceID,
			"error", err.Error(),
		)
		http.Error(w, "invalid request", http.StatusBadRequest)
		return
	}

	slog.InfoContext(r.Context(), "creating order",
		"trace_id", traceID,
		"customer_id", req.CustomerID,
		"item_count", len(req.Items),
	)

	output, err := h.service.CreateOrder(r.Context(), req.toInput())
	if err != nil {
		slog.ErrorContext(r.Context(), "failed to create order",
			"trace_id", traceID,
			"customer_id", req.CustomerID,
			"error", err.Error(),
		)
		http.Error(w, err.Error(), http.StatusUnprocessableEntity)
		return
	}

	slog.InfoContext(r.Context(), "order created successfully",
		"trace_id", traceID,
		"order_id", output.OrderID,
	)

	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusCreated)
	json.NewEncoder(w).Encode(output)
}

The three mandatory observability pillars:

flowchart LR
    subgraph OBS["Mandatory Observability in Microservices"]
        LOG["Structured Logging\\n• JSON format\\n• trace_id in every log\\n• levels: INFO, WARN, ERROR\\n• service name"]
        TRACE["Distributed Tracing\\n• Trace ID propagated\\nacross services\\n• Span per operation\\n• Jaeger / Tempo"]
        METRIC["Metrics\\n• Request rate\\n• Error rate\\n• Latency (p50, p95, p99)\\n• Queue depth\\n• Prometheus + Grafana"]
    end

Distributed Transactions: The Saga Pattern #

In microservices, there is no ACID transaction spanning multiple services. The solution is the Saga Pattern — a series of local transactions, each publishing an event to trigger the next transaction:

sequenceDiagram
    participant OS as Order Service
    participant MB as Kafka
    participant PS as Payment Service
    participant IS as Inventory Service
    participant NS as Notification Service

    Note over OS,NS: Happy Path — Choreography Saga
    OS->>OS: CreateOrder (local transaction)
    OS->>MB: publish OrderCreated
    MB->>PS: OrderCreated
    PS->>PS: ProcessPayment (local transaction)
    PS->>MB: publish PaymentProcessed
    MB->>IS: PaymentProcessed
    IS->>IS: ReserveInventory (local transaction)
    IS->>MB: publish InventoryReserved
    MB->>NS: InventoryReserved
    NS->>NS: SendConfirmation

    Note over OS,NS: Compensating Transaction — if Payment Fails
    PS->>MB: publish PaymentFailed
    MB->>OS: PaymentFailed
    OS->>OS: CancelOrder (compensating transaction)
    OS->>MB: publish OrderCancelled
    MB->>NS: OrderCancelled
    NS->>NS: SendCancellationNotice
// internal/order/saga_handler.go — handling results from other services
package order

import (
	"context"
	"log/slog"

	"myapp/internal/shared/eventbus"
)

// PaymentFailedEvent is received from the Payment Service
type PaymentFailedEvent struct {
	OrderID string
	Reason  string
}

func (e PaymentFailedEvent) EventName() string { return "payment.failed" }

// SagaHandler handles compensating transactions
type SagaHandler struct {
	repo     Repository
	eventBus *eventbus.EventBus
}

// HandlePaymentFailed is the compensating transaction for an order
func (h *SagaHandler) HandlePaymentFailed(ctx context.Context, event eventbus.Event) error {
	e, ok := event.(PaymentFailedEvent)
	if !ok {
		return nil
	}

	slog.InfoContext(ctx, "payment failed, cancelling order",
		"order_id", e.OrderID,
		"reason", e.Reason,
	)

	order, err := h.repo.FindByID(ctx, e.OrderID)
	if err != nil {
		return err
	}

	// Compensating transaction: cancel the order
	if err := order.Cancel("payment failed: " + e.Reason); err != nil {
		return err
	}

	if err := h.repo.Save(ctx, order); err != nil {
		return err
	}

	// Publish the compensation event
	return h.eventBus.Publish(ctx, OrderCancelledEvent{
		OrderID: e.OrderID,
		Reason:  "payment failed",
	})
}

Anti-Patterns to Avoid #

// ✗ Shared database between services — distributed monolith
// The Order Service queries the User Service's database directly
func (r *OrderRepo) GetOrderWithUser(ctx context.Context, orderID string) (*OrderWithUser, error) {
	// ✗ JOIN into another service's database — the most dangerous coupling
	return r.db.QueryContext(ctx, `
		SELECT o.id, u.full_name
		FROM order_db.orders o
		JOIN user_db.users u ON u.id = o.customer_id
		WHERE o.id = $1
	`, orderID)
}

// ✓ Each service only accesses its own database
// Data from other services is fetched via API calls
func (s *OrderService) GetOrderWithUser(ctx context.Context, orderID string) (*OrderDisplay, error) {
	order, err := s.repo.FindByID(ctx, orderID)
	if err != nil {
		return nil, err
	}

	// Fetch the user's name from the User Service via HTTP
	user, err := s.userClient.GetUser(ctx, order.CustomerID)
	if err != nil {
		// Graceful degradation: show the order even if the user is unavailable
		return &OrderDisplay{OrderID: orderID, CustomerName: "Unknown"}, nil
	}

	return &OrderDisplay{OrderID: orderID, CustomerName: user.FullName}, nil
}

// ✗ Overly long synchronous chains — cascade failure
// Order → Inventory → Warehouse → Supplier (all sync)
func (s *OrderService) CreateOrder(ctx context.Context, req CreateOrderInput) error {
	// ✗ 4 sync hops — if Supplier is slow, the whole order creation is slow
	inventory := s.inventoryClient.CheckStock(ctx, req.Items)
	warehouse := s.warehouseClient.Reserve(ctx, inventory)
	supplier := s.supplierClient.Confirm(ctx, warehouse)
	_ = supplier
	return s.repo.Save(ctx, buildOrder(req))
}

// ✓ Only critical validation is sync, the rest is async via events
func (s *OrderService) CreateOrder(ctx context.Context, req CreateOrderInput) error {
	// ✓ Only user validation is sync (needs an immediate response)
	if err := s.userClient.ValidateActive(ctx, req.CustomerID); err != nil {
		return err
	}

	order := buildOrder(req)
	if err := s.repo.Save(ctx, order); err != nil {
		return err
	}

	// ✓ The rest is async — Inventory, Warehouse, Supplier respond via events
	return s.eventBus.Publish(ctx, OrderCreatedEvent{OrderID: order.ID, Items: req.Items})
}

// ✗ No timeout on inter-service calls — goroutines wait forever
func (s *Service) getUser(ctx context.Context, id string) (*User, error) {
	resp, err := http.Get("http://user-service/users/" + id) // ✗ no timeout!
	// ...
}

// ✓ Always use a context with a timeout
func (s *Service) getUser(ctx context.Context, id string) (*User, error) {
	ctx, cancel := context.WithTimeout(ctx, 2*time.Second) // ✓ bounded timeout
	defer cancel()
	return s.userClient.Get(ctx, id)
}

Microservices Readiness Checklist #

ORGANIZATIONAL PREREQUISITES:
  □ Each service is owned by one clear team (2–8 people)
  □ The team can deploy its service without coordinating with other teams
  □ On-call rotation exists for every service in production
  □ The business domain is well understood — bounded contexts are clear

TECHNICAL PREREQUISITES:
  □ CI/CD pipelines are mature and automated per service
  □ Container (Docker) and orchestration (Kubernetes) are mastered by the team
  □ Centralized logging exists (ELK, Loki, or similar)
  □ Distributed tracing is configured (Jaeger, Tempo)
  □ Metrics and alerting exist (Prometheus, Grafana)
  □ Health check endpoints are available on every service

SERVICE DESIGN:
  □ One database per service — no shared databases
  □ API contracts are documented (OpenAPI, protobuf)
  □ Clear API versioning — no breaking changes without a new version
  □ Timeouts and retries are configured on all inter-service calls
  □ Circuit breakers are installed for unreliable dependencies

RESILIENCE:
  □ Services can run even when dependencies are unavailable (graceful degradation)
  □ Compensating transactions (Saga) are designed for all distributed workflows
  □ Idempotency keys are applied to retryable operations
  □ Dead letter queues are configured for events that fail to process

TESTING:
  □ Per-service unit tests can run without other services
  □ Contract tests verify API contracts between services
  □ Integration tests exist for the main flows
  □ Chaos engineering or fault injection is performed periodically

When Microservices, When Not #

Microservices are justified by real pain:
  ✓ Deployment bottleneck: Team A cannot deploy because it waits for Team B
  ✓ Scaling bottleneck: Module X needs 10x the resources of other modules
  ✓ Technology bottleneck: An ML module needs Python, the rest Go
  ✓ Reliability bottleneck: A bug in one module often crashes other modules
  ✓ The organization already has > 20 developers with clear domain ownership

Avoid microservices if:
  ✗ Choosing microservices to learn, not because of real need
  ✗ Team < 10 developers — coordination overhead exceeds the benefit
  ✗ The business domain is not yet understood — wrong service boundaries are very expensive
  ✗ The observability stack does not exist — debugging will be a nightmare
  ✗ CI/CD is not automated — manually deploying N services is unsustainable

Summary #

  • Microservices are not a goal, but a solution to specific problems — choose microservices when there is real pain that a better monolith cannot solve.
  • The distributed monolith is the biggest trap — services that are physically distributed but still share a database or are logically tightly coupled get all of distribution’s downsides without its benefits.
  • A database per service is a non-negotiable rule — sharing a database between services creates the hardest-to-break and most dangerous coupling.
  • Sync communication for critical validation, async for side effects — avoid overly long synchronous chains; use events for operations that do not need an immediate response.
  • Circuit breakers are mandatory — without a circuit breaker, one service’s failure causes a cascade failure across the whole system.
  • Observability is a prerequisite, not a feature — structured logging with trace IDs, distributed tracing, and metrics must exist before a service goes to production.
  • The Saga pattern for distributed transactions — there is no cross-service ACID; design compensating transactions for every workflow spanning multiple services.
  • Timeouts on all inter-service calls — goroutines waiting without a limit will exhaust thread pools; every HTTP/gRPC call to another service must have a timeout.
  • Start from a modular monolith, extract when there is real need — understanding the domain well in a monolith before splitting it produces far more accurate service boundaries.
  • Conway’s Law works both ways — organizational structure influences system architecture; if teams cannot work independently, services will not be deployable independently either.

← Previous: Modular Monolith   Next: Service-Based →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact