Grimoire CLI Quickstart: 5 Minutes to Your First Evaluation

What is Grimoire?

Grimoire is the free, open-source CLI that captures telemetry from your AI workflows and sends it to EvalOps. Think of it as the agent that sits between your code and the evaluation platform—logging every prompt, response, tool call, and policy check.

You can run Grimoire locally (telemetry stays on disk) or connect it to an EvalOps workspace for centralized scoring and governance.

Installation

# macOS / Linux
curl -fsSL https://install.evalops.dev/grimoire | sh

# With Homebrew
brew install evalops/tap/grimoire

# npm
npm install -g @evalops/grimoire

# Verify
grimoire --version

Your First Trace

Let's capture a simple OpenAI call and see the telemetry:

// example.ts
import { grimoire } from '@evalops/grimoire';
import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

async function main() {
  // Start a trace
  const trace = grimoire.startTrace({
    scenario: 'product-description-gen',
    metadata: {
      product_id: 'SKU-12345',
      team: 'growth'
    }
  });

  try {
    const response = await openai.chat.completions.create({
      model: 'gpt-4',
      messages: [
        { role: 'system', content: 'You are a product copywriter.' },
        { role: 'user', content: 'Write a description for wireless headphones.' }
      ]
    });

    // Log the interaction
    trace.log({
      provider: 'openai',
      model: 'gpt-4',
      prompt: 'Write a description for wireless headphones.',
      response: response.choices[0].message.content,
      tokens: response.usage
    });

    trace.end({ status: 'success' });
    console.log(response.choices[0].message.content);
  } catch (error) {
    trace.end({ status: 'error', error: String(error) });
  }
}

main();

Run it:

grimoire exec -- ts-node example.ts

What Just Happened?

Grimoire captured:

• The scenario name (product-description-gen)
• Metadata (product ID, team)
• Full prompt + response
• Token usage
• Success/failure status

By default, this is saved locally in .grimoire/traces/. View it:

grimoire show --latest

Connecting to EvalOps

Want to send traces to an EvalOps workspace for centralized scoring? Authenticate:

grimoire login

This opens a browser to authenticate with your EvalOps account. Once linked, all traces automatically sync to your workspace.

You can now:

• Apply scorecards (hallucination detection, policy checks, cost analysis)
• Set up monitors for drift or regressions
• Share dashboards with your team

Local-Only Mode

Prefer to keep everything on disk? Set local mode:

grimoire config set mode local

Traces stay in .grimoire/ and you can review them with:

grimoire dashboard

This launches a local UI at http://localhost:3333 where you can browse traces, search by scenario, and export reports.

Next Steps

• CI Integration: Add Grimoire to your GitHub Actions or GitLab CI to capture eval traces on every deploy
• Redaction Policies: Configure field-level redaction for PII before traces leave your environment
• Custom Metrics: Define your own scoring functions and apply them locally or in EvalOps

Check out the full docs or import a Spellbook recipe to automate common evaluation workflows.

---

Questions? Join the EvalOps community or email hello@evalops.dev.