Series 5 - Bài 5: Reliability Engineering cho Agent

Database connection fail → retry 3 lần với backoff. Đây là pattern quen thuộc.

LLM API fail → retry 3 lần với backoff. Nghe tương tự, nhưng có một sự khác biệt quan trọng: mỗi retry với LLM không chỉ là I/O operation — mà là generation với cost thực. Retry sai cách = cost tăng, không giải quyết vấn đề.

Agent system cần reliability engineering riêng, không phải copy-paste từ service design.

Failure Categories trong Agent

Layer 1: Infrastructure failures
  LLM API timeout
  LLM API rate limit (429)
  LLM API server error (5xx)
  Network blip
  → Retry có thể giúp

Layer 2: LLM quality failures
  LLM trả về malformed JSON
  LLM hallucinate tool name không tồn tại
  LLM produce output không theo format
  LLM loop (gọi same tool 3 lần)
  → Retry với modified prompt có thể giúp

Layer 3: Tool failures
  File not found
  Permission denied
  Command exit non-zero
  Network call fail
  → Tùy loại: retry, fallback, hoặc escalate

Layer 4: Logic failures
  Task impossible (contradiction trong requirements)
  Agent stuck (không tiến triển sau N turns)
  Budget exceeded
  → Không retry — escalate hoặc abort

Retry Strategy

Exponential Backoff với Jitter

Backoff strategy:

  Attempt 1: Fail → wait 1s + jitter
  Attempt 2: Fail → wait 2s + jitter
  Attempt 3: Fail → wait 4s + jitter
  Attempt 4: Give up → escalate

  Jitter: random(0, wait_time * 0.1)
  Mục đích: tránh thundering herd nếu nhiều agents
            retry cùng lúc

  Max attempts: 3 (không phải 10)
  Lý do: LLM failures thường không transient
         Retry nhiều = waste money

Retry chỉ khi idempotent

Safe to retry:
  LLM call (read operation về bản chất)
  read_file (idempotent)
  search_files (idempotent)
  list_directory (idempotent)

NOT safe to retry without check:
  write_file (mỗi retry có thể overwrite)
  run_command (có thể có side effects)
  send_notification (duplicate notifications)

Safe retry với idempotency check:
  write_file: check nếu file đã được ghi đúng
              trước khi retry
  run_command: check result state trước khi retry

Retry với Context Enhancement

Khi LLM trả về malformed output:

First attempt:
  [Standard messages]
  → LLM output: invalid JSON

Retry attempt (không chỉ là bare retry):
  [Standard messages]
  + "[System: Previous response was invalid JSON.
     Please respond with valid JSON only.
     No markdown fences, no explanation.
     Just the JSON object.]"
  → LLM output: valid JSON

Context enhancement giúp LLM understand
và fix lỗi của chính nó.

Fallback Strategy

Model Fallback

Primary → Fallback model chain:

  Primary: claude-sonnet-4-5
    Fail (timeout/5xx) → retry 2 lần
    Still fail ↓

  Fallback: gpt-4o
    Khác provider, independent infrastructure
    Tốt cho API outage (không phải user error)
    Retry 1 lần
    Still fail ↓

  Degraded mode:
    Simplified task
    Inform user về degraded experience
    Queue for retry khi service recovered

When NOT to fallback:
  Rate limit (429) → wait, không fallback
  Invalid API key → fail hard, không retry
  Budget exceeded → abort, không retry

Tool Fallback

Tool fallback chains:

  search_files fail:
    → Fallback: list_directory + manual scan
    → Dùng khi search index không available

  run_command fail (non-zero exit):
    → Không auto-retry (có side effect)
    → Return error với detail to LLM
    → LLM decide: fix issue hoặc alternative approach

  External API call fail:
    → Retry 2 lần với backoff
    → Fallback: notify user, queue for retry
    → Không silent fail

Timeout Design

Timeout hierarchy:

  Individual tool timeout:
    read_file:     5 seconds
    write_file:    5 seconds
    run_command:   30 seconds (có thể config)
    external_api:  10 seconds
    → Sau timeout: return timeout error to LLM

  Single LLM call timeout:
    Simple tasks:  30 seconds
    Complex tasks: 120 seconds
    → Sau timeout: retry với backoff

  Agent turn timeout:
    Mỗi turn (LLM + tools):  3 minutes
    → Sau timeout: stop turn, report to user

  Session timeout:
    Total session:  30 minutes
    → Sau timeout: graceful shutdown
    → Save state nếu có thể
    → Report progress to user

Circuit Breaker Pattern

Circuit breaker cho LLM provider:

  CLOSED state (normal):
    Requests pass through
    Track failure rate

  If failure_rate > 50% in last 60s:
    → OPEN state

  OPEN state:
    Requests blocked immediately
    → Fast fail, kh không wait for timeout
    → Use fallback provider
    Wait 30s before testing

  HALF-OPEN state:
    Allow 1 test request
    If success → CLOSED
    If fail    → OPEN again

Benefits:
  Không waste tokens/time trên broken provider
  Fast failover to backup
  Provider gets time to recover

Graceful Degradation

Khi system bị stressed, degrade gracefully:

Full capability:
  All tools available
  Primary model
  Full context window

Degraded mode 1 (partial failure):
  Non-essential tools disabled
  Primary model, reduced context
  Warn user về limited capability

Degraded mode 2 (severe failure):
  Read-only tools only
  Fallback model
  Minimal context
  Clear user message về limitations

Emergency mode:
  Accept task
  Queue for later
  Inform user about delay
  Resume khi system recovered

Trade-off: Aggressive Retry vs Fast Fail

Aggressive retry:
  + Higher success rate
  - Latency tăng
  - Cost tăng (mỗi retry = tokens)
  - Mask underlying problems

Fast fail:
  + Low latency
  + Low cost
  + Surface problems sớm
  - Cao failure rate khi transient issues

Balance:
  Retry cho infrastructure failures (transient)
  Fast fail cho logic failures (not transient)
  Never retry without understanding why it failed

Kết luận

• LLM failures chia 4 layers: infrastructure, quality, tool, logic — mỗi loại cần strategy khác.
• Exponential backoff với jitter cho infrastructure failures, max 3 attempts.
• Retry chỉ idempotent operations — write/execute cần check trước khi retry.
• Retry với context enhancement cho LLM quality failures — không bare retry.
• Model fallback chain: primary → backup provider → degraded mode.
• Timeout hierarchy: tool → LLM call → turn → session.
• Circuit breaker ngăn cascade failure khi provider down.
• Degrade gracefully thay vì fail hard — user experience còn partial value.