Series 5 - Bài 4: Cost Management và Model Routing

Một agent session đơn giản có thể tốn $0.01. Một agent session phức tạp có thể tốn $2.00. Nhân với 10,000 sessions/tháng — đó là $100 đến $20,000. Chênh lệch này không phải ngẫu nhiên.

Cost management trong agent system không phải "dùng model rẻ hơn". Đó là system design quyết định: task nào cần model nào, context được build như thế nào, và khi nào nên stop.

Anatomy của Agent Cost

Cost breakdown một agent session:

  LLM Input tokens:     70% tổng cost
    System prompt:      2,000 tokens (fixed per turn)
    Conversation:       grow theo turns
    Tool results:       lớn nhất, khó kiểm soát
    File context:       variable

  LLM Output tokens:   20% tổng cost
    Agent reasoning:   200-500 tokens/turn
    Tool calls:        50-150 tokens/tool
    Final response:    300-1000 tokens

  Infrastructure:      10%
    Tool execution:    rất nhỏ
    Storage:           nhỏ
    Compute:           nhỏ

Kết luận:
  Input tokens là lever lớn nhất
  → Context management = cost management

Model Routing: Đúng model cho đúng task

Model tier và use cases:

  Tier 1 — Cheap, Fast (Haiku, GPT-4o mini, Gemini Flash)
    Cost:       ~$0.001/1K tokens
    Good for:
      Đọc và summarize file
      Phân loại task đơn giản
      Format conversion
      Validation và checking
      Simple single-tool calls

  Tier 2 — Balanced (Sonnet, GPT-4o, Gemini Pro)
    Cost:       ~$0.015/1K tokens
    Good for:
      Code refactoring
      Multi-step reasoning
      Architecture analysis
      Writing technical docs
      Most production tasks

  Tier 3 — Expensive, Powerful (Opus, GPT-4)
    Cost:       ~$0.075/1K tokens
    Good for:
      Complex security audits
      Novel algorithm design
      High-stakes decisions
      Ambiguous requirements

Routing Decision Tree

Khi nhận task, orchestrator classify:

  Is task clearly defined?
  and Is task single-step or well-scoped?
  and Does task not require deep reasoning?
    YES → Tier 1 model

  Is task multi-step?
  or Is task require code changes?
  or Is task require architectural thinking?
    YES → Tier 2 model (default)

  Is task ambiguous and high-stakes?
  or Is task require novel solution?
  or Did Tier 2 fail or produce low confidence?
    YES → Tier 3 model

Fallback routing:
  Tier 1 fail → retry with Tier 2
  Tier 2 low confidence → escalate to Tier 3
  Never Tier 3 as default

Task Classification cho Routing

Classifier (có thể dùng Tier 1 model để classify):

  Input: Task description + context summary

  Output:
  {
    complexity: "simple" | "medium" | "complex",
    requires_code_change: boolean,
    has_ambiguity: boolean,
    estimated_turns: number,
    recommended_tier: 1 | 2 | 3
  }

Classification cost: ~500 tokens = ~$0.0005
If routing from Tier 3 → Tier 2 saves $0.50:
  Classification ROI: 1000x

Token Optimization Techniques

1. Context Compression trước khi gửi

Trước khi gọi LLM, compress context:

File content optimization:
  Thay vì: full 300-line file mỗi turn
  → Line 1-50 (class definition) + line 120-145 (target function)
  → Saving: ~250 lines = ~3,750 tokens per turn

Tool result compression:
  Thay vì: full command output (có thể 5,000 tokens)
  → Summary: "Tests pass: 52/52 (exit 0, 2.3s)"
  → Saving: ~4,950 tokens

History compression:
  Khi history > 30,000 tokens
  → Summarize oldest 50% trước khi gửi

2. System Prompt Optimization

System prompt được gửi mỗi request.
Nếu 20 turns × 2,000 token prompt = 40,000 tokens chỉ cho prompt.

Optimization:

  Verbose (2,000 tokens):
  "When you are about to use the write_file tool to modify
   a file, you should first make sure that you have read the
   current contents of the file using the read_file tool..."

  Concise (100 tokens):
  "Always read before write.
   Run tests after changes.
   Confirm before destructive actions."

  Saving: 1,900 tokens × 20 turns = 38,000 tokens
  Cost saving: ~$0.57 per session

3. Tool Result Truncation

Big tool results là cost killer:

  run_command output có thể 20,000 tokens
  Nhưng LLM chỉ cần:
    Exit code, last 50 lines của output
    Full stderr nếu có error

Truncation rules:
  stdout: keep last 200 lines (nếu success)
          keep all (nếu error)
  stderr: always keep full
  Binary output: replace với metadata

Saving: 80-90% của command output tokens

4. Parallel Calls thay vì Sequential

Sequential (tốn kém):
  Turn 1: read_file(a.ts) → result → Turn 2: read_file(b.ts)
  2 LLM calls, 2 × input context

Parallel (tiết kiệm):
  LLM request cả 2 files cùng lúc
  1 LLM call, 1 × input context + 2 tool results

Saving: 50% LLM calls cho independent reads
Agent tốt tự làm điều này
System prompt có thể hint: "Batch independent reads"

Cost Monitoring và Alerts

Cost budgets:

  Per-session limit:
    Soft limit: $1.00 → warn user
    Hard limit: $5.00 → stop agent

  Per-user daily:
    Monitor để phát hiện abnormal usage

  Per-organization monthly:
    Budget alert tại 80%
    Hard stop tại 100%

Metrics cần track:
  cost_per_session (average + P95)
  tokens_per_task
  model_distribution (% Tier 1 vs 2 vs 3)
  routing_accuracy (did right model win?)

Trade-off: Quality vs Cost

Aggressive routing → Tier 1 cho mọi task:
  + Cost giảm 90%
  - Nhiều fails, retries
  - Retry với Tier 2 → tổng cost tăng
  - User experience tệ hơn

Conservative routing → Tier 2 default:
  + Reliable
  + Ít retries
  - Cost cao hơn Tier 1

Intelligent routing → Classify trước:
  + Optimal balance
  + 30-50% cost savings vs Tier 2 default
  - Cần implement classifier
  - Classification có thể sai

Kết luận

• Input tokens là 70% cost — context management là lever lớn nhất.
• Model routing: classify task complexity → Tier 1/2/3 theo đúng use case.
• Classification cost bằng Tier 1, nhưng ROI cao khi routing đúng.
• Tool result truncation: 80-90% saving với minimal quality loss.
• System prompt concise hóa: 1,900 tokens × N turns = significant saving.
• Set hard cost limits per session để prevent runaway cost.
• Intelligent routing > conservative routing > aggressive routing về total cost.