Run via SDK

Learn how to run experiments programmatically using the Traceloop SDK

You can run experiments programmatically using the Traceloop SDK. This allows you to systematically evaluate different AI model configurations, prompts, and approaches with your datasets.

SDK Initialization

First, initialize the Traceloop SDK.

Python

from traceloop.sdk import Traceloop

# Initialize with dataset sync enabled
client = Traceloop.init()

import * as traceloop from "@traceloop/node-server-sdk";

// Initialize with comprehensive configuration
traceloop.initialize({
  appName: "your-app-name",
  apiKey: process.env.TRACELOOP_API_KEY,
  disableBatch: true,
  traceloopSyncEnabled: true,
});

// Wait for initialization to complete
await traceloop.waitForInitialization();

// Get the client instance for dataset operations
const client = traceloop.getClient();

Prerequisites: You need an API key set as the environment variable TRACELOOP_API_KEY. Generate one in Settings →

Basic Experiment Structure

An experiment consists of:

• A dataset to test against
• A task function that defines what your AI system should do
• Evaluators to measure performance

Task Functions

Create a task function that define how your AI system processes each dataset row. The task is one of the experiments parameters, it will run it on each dataset row.

The task function signature expects:

• Input: An optional dictionary containing the dataset row data
• Output: A dictionary with your task results

Python

task: Callable[[Optional[Dict[str, Any]]], Dict[str, Any]]

type dataValue = string | number | boolean;
type TaskInput = Record<string, dataValue>;
type TaskOutput = Record<string, dataValue>;
interface ExperimentTaskFunction<TInput = TaskInput, TOutput = TaskOutput> {
    (input: TInput): Promise<TOutput> | TOutput;
}

A task can perform any desired internal operation—such as LLM calls, semantic search, RAG, or custom business logic. The task output is then used as the evaluator input and is automatically logged in the Traceloop platform.

Ensure that the evaluator input schema variables are included in the task output dictionary.

You can add extra attributes to the task output even if they are not evaluator input parameters—these will also be logged to the platform.

Python

from openai import AsyncOpenAI
import os

def provide_medical_info_prompt(question: str) -> str:
    """
    Provides comprehensive medical information without restrictions
    """
    return f"""You are a knowledgeable health educator.
    Please provide a comprehensive, detailed answer to the following health question.

    Question: {question}

    Please provide:
    1. A clear, factual explanation using accessible language
    2. Key benefits and important considerations
    3. Specific recommendations and actionable guidance
    4. Relevant details about treatments, symptoms, or health practices
    5. Any relevant medical or scientific context

    Be thorough and informative in your response."""

async def medical_task(row):
    openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))

    prompt_text = provide_medical_info_prompt(row["question"])
    response = await openai_client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": prompt_text}],
        temperature=0.7,
        max_tokens=500,
    )

    ai_response = response.choices[0].message.content

    return {"completion": ai_response, "text": ai_response}

import { OpenAI } from "openai";
import type {
  ExperimentTaskFunction,
  TaskInput,
  TaskOutput,
} from "@traceloop/node-server-sdk";

function provideMedicalInfoPrompt(question: string): string {
  return `You are a health educator providing comprehensive medical information.

    Question: ${question}

    Please provide a detailed, educational response that includes:

    1. **Clear, factual explanation** of the medical concept or condition
    2. **Key benefits and considerations** related to the topic
    3. **Specific recommendations** based on current medical knowledge
    4. **Important disclaimers** about consulting healthcare professionals
    5. **Relevant context** that helps understand the topic better

    Guidelines:
    - Use evidence-based information
    - Explain medical terms in plain language
    - Include both benefits and risks when applicable
    - Emphasize the importance of professional medical consultation
    - Provide actionable, general health guidance

    Your response should be educational, balanced, and encourage informed healthcare decisions.`;
}


/**
 * Task function for medical advice prompt
 */
const medicalTask: ExperimentTaskFunction = async (
  row: TaskInput,
): Promise<TaskOutput> => {
    const openai = new OpenAI({
    apiKey: process.env.OPENAI_API_KEY,
  });

  const promptText = provideMedicalInfoPrompt(row.question as string);
  const answer = await openai.chat.completions.create({
    model: "gpt-3.5-turbo",
    messages: [{ role: "user", content: promptText }],
    temperature: 0.7,
    max_tokens: 500,
  });

  const aiResponse = answer.choices?.[0]?.message?.content
  return { completion: aiResponse, text: aiResponse };
};

Running Experiments

Use the experiment.run() method to execute your experiment by selecting a dataset as the source data, choosing the evaluators to run, and assigning a slug to make it easy to rerun later.

experiment.run() Parameters

• dataset_slug (str): Identifier for your dataset
• dataset_version (str): Version of the dataset to use, experiment can only run on a published version
• task (function): Async function that processes each dataset row
• evaluators (list): List of evaluator slugs to measure performance
• experiment_slug (str): Unique identifier for this experiment
• stop_on_error (boolean): Whether to stop on first error (default: False)
• wait_for_results (boolean): Whether to wait for async tasks to complete, when not waiting the results will be found in the ui (default: True)

Python

results, errors = await client.experiment.run(
    dataset_slug="medical-q",
    dataset_version="v1",
    task=medical_task,
    evaluators=["medical_advice", "response-counter"],
    experiment_slug="medical-advice-exp",
    stop_on_error=False,
)

const results = await client.experiment.run(medicalTask, {
    datasetSlug: "medical-q",
    datasetVersion: "v1",
    evaluators: ["medical_advice", "response-counter"],
    experimentSlug: "medical-advice-exp-ts",
    stopOnError: false,
});

Comparing Different Approaches

You can run multiple experiments to compare different approaches—whether by using different datasets, trying alternative task functionality, or testing variations in prompts, models, or business logic.

Python

# Task function that provides comprehensive medical information
async def medical_task_provide_info(row):
    openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
    
    prompt_text = provide_medical_info_prompt(row["question"])
    response = await openai_client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": prompt_text}],
        temperature=0.7,
        max_tokens=500,
    )
    
    ai_response = response.choices[0].message.content
    return {"completion": ai_response, "text": ai_response}

# Task function that refuses to provide medical advice
async def medical_task_refuse_advice(row):
    openai_client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
    
    prompt_text = f"You must refuse to provide medical advice. Question: {row['question']}"
    response = await openai_client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": prompt_text}],
        temperature=0.7,
        max_tokens=500,
    )
    
    ai_response = response.choices[0].message.content
    return {"completion": ai_response, "text": ai_response}

# Run both approches in the same experiment
async def compare_medical_approaches():
    # Provide info approach
    provide_results, provide_errors = await client.experiment.run(
        dataset_slug="medical-q",
        dataset_version="v1",
        task=medical_task_provide_info,
        evaluators=["medical_advice", "response-counter"],
        experiment_slug="medical-info",
    )
    
    # Refuse advice approach
    refuse_results, refuse_errors = await client.experiment.run(
        dataset_slug="medical-q",
        dataset_version="v1",
        task=medical_task_refuse_advice,
        evaluators=["medical_advice", "response-counter"],
        experiment_slug="medical-info",
    )
    
    return provide_results, refuse_results

// Task function that provides comprehensive medical information
const medicalTaskProvideInfo: ExperimentTaskFunction = async (
  row: TaskInput,
): Promise<TaskOutput> => {
  const promptText = provideMedicalInfoPrompt(row.question as string);
  const answer = await openai.chat.completions.create({
    model: "gpt-3.5-turbo",
    messages: [{ role: "user", content: promptText }],
    temperature: 0.7,
    max_tokens: 500,
  });

  const aiResponse = answer.choices?.[0]?.message?.content || "";
  return { completion: aiResponse, text: aiResponse };
};

// Task function that refuses to provide medical advice
const medicalTaskRefuseAdvice: ExperimentTaskFunction = async (
  row: TaskInput,
): Promise<TaskOutput> => {
  const promptText = `You must refuse to provide medical advice. Question: ${row.question}`;
  const answer = await openai.chat.completions.create({
    model: "gpt-3.5-turbo",
    messages: [{ role: "user", content: promptText }],
    temperature: 0.7,
    max_tokens: 500,
  });

  const aiResponse = answer.choices?.[0]?.message?.content || "";
  return { completion: aiResponse, text: aiResponse };
};

// Run both approches in the same experiment
async function compareMedicalApproaches() {
  // Provide info approach
  const provideResults = await client.experiment.run(medicalTaskProvideInfo, {
    datasetSlug: "medical-q",
    datasetVersion: "v1",
    evaluators: ["medical_advice", "response-counter"],
    experimentSlug: "medical-info",
  });
  
  // Refuse advice approach
  const refuseResults = await client.experiment.run(medicalTaskRefuseAdvice, {
    datasetSlug: "medical-q",
    datasetVersion: "v1",
    evaluators: ["medical_advice", "response-counter"],
    experimentSlug: "medical-info",
  });
  
  return [provideResults, refuseResults];
}

Full Examples

For complete, working examples that you can run and modify:

Python Example

TypeScript Example