flare-agent Bài 9: Observability & Tracing

Series: Build Your Own AI Agent Framework trên Cloudflare Bài: 9 / 9 — @flare-agent/observability

---

Tại sao cần Tracing?

Agent loop là black box — bạn gọi agent.run() và nhận output, nhưng không biết:

• Mất bao lâu ở mỗi bước?
• Tool nào được gọi với args gì?
• LLM trả về gì trước khi gọi tool?
• Iteration nào tốn nhiều tokens nhất?
• Lỗi xảy ra ở đâu trong multi-agent flow?

Tracing giải quyết tất cả — biến black box thành glass box.

---

Thiết kế đơn giản, không over-engineer

Thay vì tích hợp full OpenTelemetry SDK (nặng, không phù hợp Workers), build lightweight tracer tương thích OTel format — export được sang Jaeger, Grafana, hoặc Cloudflare Workers Analytics.

packages/observability/
  src/
    types.ts        # Span, Trace interfaces
    Tracer.ts       # Core tracer
    SpanContext.ts  # Context propagation
    exporters/
      console.ts    # Dev: log ra console
      d1.ts         # Prod: lưu vào D1
      otel.ts       # Export sang OTel collector
    index.ts

---

Types

// src/types.ts

export type SpanStatus = 'ok' | 'error' | 'running';

export interface Span {
  traceId: string;      // ID của toàn bộ request
  spanId: string;       // ID của span này
  parentSpanId?: string; // ID của span cha
  name: string;         // Tên operation
  startTime: number;    // Unix ms
  endTime?: number;
  durationMs?: number;
  status: SpanStatus;
  attributes: Record<string, unknown>; // metadata
  events: SpanEvent[];  // logs trong span
  error?: string;
}

export interface SpanEvent {
  name: string;
  timestamp: number;
  attributes?: Record<string, unknown>;
}

export interface Trace {
  traceId: string;
  spans: Span[];
  startTime: number;
  endTime?: number;
  totalDurationMs?: number;
}

export interface SpanExporter {
  export(spans: Span[]): Promise<void>;
}

---

Tracer

// src/Tracer.ts
import type { Span, SpanEvent, SpanExporter, SpanStatus } from './types';

export class Tracer {
  private spans = new Map<string, Span>();
  private exporters: SpanExporter[] = [];

  constructor(private traceId = crypto.randomUUID()) {}

  addExporter(exporter: SpanExporter): this {
    this.exporters.push(exporter);
    return this;
  }

  // Bắt đầu 1 span
  startSpan(
    name: string,
    attributes: Record<string, unknown> = {},
    parentSpanId?: string
  ): string {
    const spanId = crypto.randomUUID();
    const span: Span = {
      traceId: this.traceId,
      spanId,
      parentSpanId,
      name,
      startTime: Date.now(),
      status: 'running',
      attributes,
      events: [],
    };
    this.spans.set(spanId, span);
    return spanId;
  }

  // Kết thúc span
  endSpan(spanId: string, status: SpanStatus = 'ok', error?: string): void {
    const span = this.spans.get(spanId);
    if (!span) return;

    span.endTime = Date.now();
    span.durationMs = span.endTime - span.startTime;
    span.status = status;
    if (error) span.error = error;

    // Export ngay khi span kết thúc
    this.exporters.forEach((e) => e.export([span]));
  }

  // Thêm event vào span đang chạy
  addEvent(
    spanId: string,
    name: string,
    attributes?: Record<string, unknown>
  ): void {
    const span = this.spans.get(spanId);
    if (!span) return;
    span.events.push({
      name,
      timestamp: Date.now(),
      attributes,
    });
  }

  // Set attribute
  setAttribute(
    spanId: string,
    key: string,
    value: unknown
  ): void {
    const span = this.spans.get(spanId);
    if (!span) return;
    span.attributes[key] = value;
  }

  // Helper: wrap async function trong span
  async trace<T>(
    name: string,
    fn: (spanId: string) => Promise<T>,
    attributes: Record<string, unknown> = {},
    parentSpanId?: string
  ): Promise<T> {
    const spanId = this.startSpan(name, attributes, parentSpanId);
    try {
      const result = await fn(spanId);
      this.endSpan(spanId, 'ok');
      return result;
    } catch (err) {
      this.endSpan(
        spanId,
        'error',
        err instanceof Error ? err.message : String(err)
      );
      throw err;
    }
  }

  getTrace() {
    const spans = [...this.spans.values()];
    return {
      traceId: this.traceId,
      spans,
      startTime: Math.min(...spans.map((s) => s.startTime)),
      endTime: Math.max(...spans.map((s) => s.endTime ?? Date.now())),
    };
  }
}

---

Exporters

Console Exporter — dev

// src/exporters/console.ts
import type { Span, SpanExporter } from '../types';

export class ConsoleExporter implements SpanExporter {
  async export(spans: Span[]): Promise<void> {
    for (const span of spans) {
      const status = span.status === 'error' ? '❌' : '✅';
      const duration = span.durationMs ? `${span.durationMs}ms` : 'running';

      console.log(
        `${status} [${span.name}] ${duration}`,
        span.attributes
      );

      if (span.error) {
        console.error(`   Error: ${span.error}`);
      }

      if (span.events.length) {
        span.events.forEach((e) =>
          console.log(`   Event: ${e.name}`, e.attributes)
        );
      }
    }
  }
}

D1 Exporter — production

// src/exporters/d1.ts
import type { Span, SpanExporter } from '../types';

export class D1Exporter implements SpanExporter {
  constructor(private db: D1Database) {}

  async export(spans: Span[]): Promise<void> {
    // Batch insert
    const stmts = spans.map((span) =>
      this.db
        .prepare(
          `INSERT INTO agent_traces
           (trace_id, span_id, parent_span_id, name,
            start_time, end_time, duration_ms,
            status, attributes, error)
           VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
        )
        .bind(
          span.traceId,
          span.spanId,
          span.parentSpanId ?? null,
          span.name,
          span.startTime,
          span.endTime ?? null,
          span.durationMs ?? null,
          span.status,
          JSON.stringify(span.attributes),
          span.error ?? null
        )
    );

    await this.db.batch(stmts);
  }
}

// Migration
export const TRACES_MIGRATION = `
CREATE TABLE IF NOT EXISTS agent_traces (
  id           INTEGER PRIMARY KEY AUTOINCREMENT,
  trace_id     TEXT NOT NULL,
  span_id      TEXT NOT NULL UNIQUE,
  parent_span_id TEXT,
  name         TEXT NOT NULL,
  start_time   INTEGER NOT NULL,
  end_time     INTEGER,
  duration_ms  INTEGER,
  status       TEXT NOT NULL,
  attributes   TEXT, -- JSON
  error        TEXT,
  created_at   DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_trace ON agent_traces(trace_id);
`;

---

Tích hợp vào Agent Loop

Thêm tracer vào AgentLoop — inject qua config:

// packages/core/src/AgentLoop.ts — updated

interface LoopConfig {
  systemPrompt: string;
  provider: LLMProvider;
  memory: MemoryManager;
  tools: ToolRegistry;
  maxIterations: number;
  tracer?: Tracer; // ← optional, không bắt buộc
}

export class AgentLoop {
  async run(userMessage: string, ctx: AgentContext): Promise<AgentResult> {
    const { tracer } = this.config;

    // Span cho toàn bộ agent run
    const runSpanId = tracer?.startSpan('agent.run', {
      'agent.input': userMessage,
      'agent.sessionId': ctx.sessionId,
      'agent.userId': ctx.userId,
    });

    try {
      await this.config.memory.addMessage(ctx.sessionId, {
        role: 'user',
        content: userMessage,
      });

      for (let i = 0; i < this.config.maxIterations; i++) {
        const iterSpanId = tracer?.startSpan(
          'agent.iteration',
          { 'iteration.index': i },
          runSpanId
        );

        const messages = await this.config.memory.getMessages(ctx.sessionId);
        const toolSchemas = this.config.tools.getSchemas();

        // Span cho LLM call
        const llmSpanId = tracer?.startSpan(
          'llm.chat',
          {
            'llm.messageCount': messages.length,
            'llm.toolCount': toolSchemas.length,
          },
          iterSpanId
        );

        const response = await this.config.provider.chat(
          [{ role: 'system', content: this.config.systemPrompt }, ...messages],
          toolSchemas
        );

        tracer?.setAttribute(
          llmSpanId!,
          'llm.responseType',
          response.type
        );
        tracer?.endSpan(llmSpanId!);

        if (response.type === 'tool_call' && response.toolCall) {
          const { id, name, args } = response.toolCall;

          // Span cho tool execution
          const toolSpanId = tracer?.startSpan(
            `tool.${name}`,
            { 'tool.name': name, 'tool.args': JSON.stringify(args) },
            iterSpanId
          );

          let result: unknown;
          try {
            result = await this.config.tools.execute(name, args, ctx);
            tracer?.setAttribute(
              toolSpanId!,
              'tool.result',
              JSON.stringify(result)
            );
            tracer?.endSpan(toolSpanId!, 'ok');
          } catch (err) {
            tracer?.endSpan(
              toolSpanId!,
              'error',
              err instanceof Error ? err.message : String(err)
            );
            result = { error: String(err) };
          }

          await this.config.memory.addMessage(ctx.sessionId, {
            role: 'tool',
            content: JSON.stringify(result),
            toolCallId: id,
          });

          tracer?.endSpan(iterSpanId!);
          continue;
        }

        // Final response
        const output = response.content ?? '';
        await this.config.memory.addMessage(ctx.sessionId, {
          role: 'assistant',
          content: output,
        });

        tracer?.setAttribute(runSpanId!, 'agent.output', output);
        tracer?.setAttribute(runSpanId!, 'agent.iterations', i + 1);
        tracer?.endSpan(iterSpanId!);
        tracer?.endSpan(runSpanId!, 'ok');

        return { output, iterations: i + 1, toolCalls: [] };
      }

      throw new Error('Max iterations exceeded');
    } catch (err) {
      tracer?.endSpan(
        runSpanId!,
        'error',
        err instanceof Error ? err.message : String(err)
      );
      throw err;
    }
  }
}

---

Dùng trong Worker

import { Tracer } from '@flare-agent/observability';
import { ConsoleExporter, D1Exporter } from '@flare-agent/observability';

app.post('/api/chat', async (c) => {
  const { input, sessionId, userId } = await c.req.json();

  // Tạo tracer cho request này
  const tracer = new Tracer()
    .addExporter(new ConsoleExporter())        // log dev
    .addExporter(new D1Exporter(c.env.DB));   // persist prod

  const result = await vocabularyAgent.run(input, {
    sessionId,
    userId,
    env: { ...c.env, tracer }, // inject tracer qua env
  });

  // Trả về trace cùng result nếu cần debug
  return c.json({
    result,
    traceId: tracer.getTrace().traceId,
  });
});

// Debug endpoint — xem trace của 1 request
app.get('/api/traces/:traceId', async (c) => {
  const { traceId } = c.req.param();
  const { results } = await c.env.DB
    .prepare('SELECT * FROM agent_traces WHERE trace_id = ? ORDER BY start_time ASC')
    .bind(traceId)
    .all();
  return c.json({ spans: results });
});

---

Output khi chạy

✅ [agent.run] 1243ms { agent.input: 'giải thích từ serendipity', agent.iterations: 3 }
  ✅ [agent.iteration] 423ms { iteration.index: 0 }
    ✅ [llm.chat] 380ms { llm.messageCount: 2, llm.responseType: 'tool_call' }
    ✅ [tool.get_vocabulary] 42ms { tool.name: 'get_vocabulary', tool.args: '{"word":"serendipity"}' }
  ✅ [agent.iteration] 401ms { iteration.index: 1 }
    ✅ [llm.chat] 390ms { llm.messageCount: 4, llm.responseType: 'tool_call' }
    ✅ [tool.save_progress] 10ms { tool.name: 'save_progress' }
  ✅ [agent.iteration] 419ms { iteration.index: 2 }
    ✅ [llm.chat] 415ms { llm.messageCount: 6, llm.responseType: 'text' }

Nhìn vào là biết ngay: 3 iterations, LLM call nào chậm, tool nào được gọi.

---

Checklist

• Tạo packages/observability/
• Chạy migration thêm bảng agent_traces
• Inject tracer vào AgentLoop
• Verify trace output đúng trong console
• Test D1Exporter persist được

---

Tổng kết Series

9 bài, 9 packages — bạn đã có full AI Agent Framework chạy native trên Cloudflare:

@flare-agent/types         ← interfaces
@flare-agent/providers     ← Groq, WorkersAI, Ollama
@flare-agent/memory        ← KV, D1, Vectorize
@flare-agent/core          ← Agent Loop, Tool Registry
@flare-agent/workflow      ← Graph-based Workflow
@flare-agent/multi-agent   ← Agent Network, Handoff
@flare-agent/observability ← Tracing, Debugging

Không phụ thuộc Mastra. Không phụ thuộc LangChain. Hiểu 100% internals.

Series hoàn tất. Nếu bài viết hữu ích, hãy chia sẻ với cộng đồng developer Việt Nam! 🇻🇳