Automatic Evaluators

Automatic evaluators can be rule-based or graded by large language models (LLMs), and they can programmatically run on LLM input or output. Baserun offers a number of pre-built automatic evaluators (see below), as well as the ability to perform custom evals with your own prompt or your own function.

Complete Reference

Python
Typescript

match(name: str, submission: str, expected: Union[str, List[str]]) -> bool

Checks if the submission starts with any of the expected values.Returns true if the submission starts with any of the expected values, otherwise false.

name

str

Name of the evaluation.

submission

str

The input string.

expected

Union[str, List[str]]

A string or a list of strings to check against.

includes(name: str, submission: str, expected: Union[str, List[str]]) -> bool

Checks if the submission contains any of the expected values within it.Returns true if the submission includes any of the expected values, otherwise false.

name

str

Name of the evaluation.

submission

str

The input string.

expected

Union[str, List[str]]

A string or a list of strings to check against.

fuzzy_match(name: str, submission: str, expected: Union[str, List[str]]) -> bool

Checks if the submission contains any of the expected values or if any of the expected values contain the submission.Returns true if there’s a fuzzy match, otherwise false.

name

str

Name of the evaluation.

submission

str

The input string.

expected

Union[str, List[str]]

A string or a list of strings to check against.

not_match(name: str, submission: str, expected: Union[str, List[str]]) -> bool

Checks if the submission does not start with any of the expected values.Returns true if the submission does not start with any of the expected values, otherwise false.

name

str

Name of the evaluation.

submission

str

The input string.

expected

Union[str, List[str]]

A string or a list of strings to check against.

not_includes(name: str, submission: str, expected: Union[str, List[str]]) -> bool

Checks if the submission does not contain any of the expected values.Returns true if the submission does not include any of the expected values, otherwise false.

name

str

Name of the evaluation.

submission

str

The input string.

expected

Union[str, List[str]]

A string or a list of strings to check against.

not_fuzzy_match(name: str, submission: str, expected: Union[str, List[str]]) -> bool

Checks if the submission neither contains any of the expected values nor is contained by any of the expected values.Returns true if there’s no fuzzy match, otherwise false.

name

str

Name of the evaluation.

submission

str

The input string.

expected

Union[str, List[str]]

A string or a list of strings to check against.

valid_json(name: str, submission: str) -> bool

Checks if the submission is a valid JSON string.Returns true if the submission is a valid JSON string, otherwise false.

name

str

Name of the evaluation.

submission

str

The input string.

check_injection(name: str, submission: str) -> bool

Checks if the submission includes injection.Returns true if the submission includes injection, otherwise false.

name

str

Name of the evaluation.

submission

str

The input string.

custom(name: str, submission: str, fn: Callable[[str], bool]) -> bool

Checks the submission using a custom function.Returns the result of the custom function.

name

str

Name of the evaluation.

submission

str

The input string.

Callable[[str], bool]

A custom function that takes the submission and returns a boolean.

custom_async(name: str, submission: str, fn: Callable[[str], Awaitable[bool]]) -> bool

Checks the submission using an asynchronous custom function.Returns the result of the custom function.

name

str

Name of the evaluation.

submission

str

The input string.

Callable[[str], Awaitable[bool]]

A custom function that takes the submission and returns a promise that resolves to a boolean.

model_graded_custom(name: str, prompt: str, choices: dict[str, float], model: str, metadata: Optional[Dict[str, Any]], **variables) -> str

Evaluates the model’s response based on a prompt and a set of choices.Returns the choice given by the model.Example:

result = Baserun.evals.model_graded_custom(
    name="Truthiness",
    prompt="How true is this statement? {statement}.",
    choices={"True": 1, "Somewhat true": 0.5, "Not true": 0},
    statement=statement,
)

name

str

Name of the evaluation.

prompt

str

The prompt passed to the model.

choices

dict[str, float]

A dictionary of choices and their scores.

model

str

default:"gpt-4-0125-preview"

OpenAI model that you want to use for the evaluation.

metadata

Optional[dict[str, Any]]

default:"None"

Any metadata that might be useful for you.

variables

dict[str, str]

Variables that will be substituted in the formatted prompt.

model_graded_fact(name: str, question: str, expert: str, submission: str) -> str

Checks a submitted answer against an expert answer for factual consistency using gpt-4-0613.Returns one of:

“A”: The output is a subset of the expert answer and fully consistent with it.
“B”: The output is a superset of the expert answer and fully consistent with it.
“C”: The submitted answer contains all of the same details as the expert answer.
“D”: There is disagreement between the submitted answer and the expert answer.
“E”: The answers differ, but these differences don’t matter from the perspective of factuality.

name

str

Name of the evaluation.

question

str

The question.

expert

str

The expert answer.

submission

str

The submitted answer.

model_graded_closed_qa(name: str, task: str, submission: str, criterion: str) -> str

Checks a submitted answer based on a specific criterion for relevance, conciseness, and correctness using gpt-4-0613.Returns “Yes” if the submission meets the criteria, “No” if it does not, and “Unsure” if it cannot be determined.

name

str

Name of the evaluation.

task

str

The task.

submission

str

The submitted answer.

criterion

str

The criterion.

model_graded_security(name: str, submission: str) -> str

Checks the submitted string for potential malicious content using gpt-4-0613.Returns “Yes” if the submission is malicious, “No” if it is not malicious, and “Unsure” if it cannot be determined.

name

str

Name of the evaluation.

submission

str

The input string.

equals(name: string, submission: string, expected: string | string[]): boolean

Checks if the submission equals expected.Returns true if the submission equals expected value, otherwise false.

name

string

Name of the evaluation.

submission

string

The input string.

expected

string | string[]

A string to check against.

eq(name: string, submission: string, expected: string | string[]): boolean

Short version of equals.

match(name: string, submission: string, expected: string | string[]): boolean

Checks if the submission starts with any of the expected values.Returns true if the submission starts with any of the expected values, otherwise false.

name

string

Name of the evaluation.

submission

string

The input string.

expected

string | string[]

A string or an array of strings to check against.

includes(name: string, submission: string, expected: string | string[]): boolean

Checks if the submission contains any of the expected values within it.Returns true if the submission includes any of the expected values, otherwise false.

name

string

Name of the evaluation.

submission

string

The input string.

expected

string | string[]

A string or an array of strings to check against.

fuzzyMatch(name: string, submission: string, expected: string | string[]): boolean

Checks if the submission contains any of the expected values or if any of the expected values contain the submission.Returns true if there’s a fuzzy match, otherwise false.

name

string

Name of the evaluation.

submission

string

The input string.

expected

string | string[]

A string or an array of strings to check against.

notMatch(name: string, submission: string, expected: string | string[]): boolean

Checks if the submission does not start with any of the expected values.Returns true if the submission does not start with any of the expected values, otherwise false.

name

string

Name of the evaluation.

submission

string

The input string.

expected

string | string[]

A string or an array of strings to check against.

notIncludes(name: string, submission: string, expected: string | string[]): boolean

Checks if the submission does not contain any of the expected values.Returns true if the submission does not include any of the expected values, otherwise false.

name

string

Name of the evaluation.

submission

string

The input string.

expected

string | string[]

A string or an array of strings to check against.

notFuzzyMatch(name: string, submission: string, expected: string | string[]): boolean

Checks if the submission neither contains any of the expected values nor is contained by any of the expected values.Returns true if there’s no fuzzy match, otherwise false.

name

string

Name of the evaluation.

submission

string

The input string.

expected

string | string[]

A string or an array of strings to check against.

validJson(name: string, submission: string): boolean

Checks if the submission is a valid JSON string.Returns true if the submission is a valid JSON string, otherwise false.

name

string

Name of the evaluation.

submission

string

The input string.

custom(name: string, submission: string, fn: (submission: string) => boolean): boolean

Checks the submission using a custom function.Returns the result of the custom function.

name

string

Name of the evaluation.

submission

string

The input string.

(submission: string) => boolean

A custom function that takes the submission and returns a boolean.

customAsync(name: string, submission: string, fn: (submission: string) => Promise<boolean>): Promise<boolean>

Checks the submission using an asynchronous custom function.Returns the result of the custom function.

name

string

Name of the evaluation.

submission

string

The input string.

(submission: string) => Promise<boolean>

A custom function that takes the submission and returns a promise that resolves to a boolean.

modelGradedCustom(name: str, prompt: string, choices: Record<string, number>, model: string = 'gpt-4-0125-preview', metadata?: Record<string, any>): Promise<string>

Evaluates the model’s response based on a prompt and a set of choices.Returns the choice given by the model.Example:

const result = await baserun.evals.modelGradedCustom(
    "Truthiness",
    `How true is this statement? ${statement}.`,
    {"True": 1, "Somewhat true": 0.5, "Not true": 0},
)

name

string

Name of the evaluation.

prompt

string

The prompt passed to the model.

choices

Record<string, number>

A dictionary of choices and their scores.

model

string

default:"gpt-4-0125-preview"

OpenAI model that you want to use for the evaluation.

metadata

Record<string, any>

Any metadata that might be useful for you.

modelGradedFact(name: string, question: string, expert: string, submission: string): Promise<string>

Checks a submitted answer against an expert answer for factual consistency using gpt-4-0613.Returns one of:

“A”: The output is a subset of the expert answer and fully consistent with it.
“B”: The output is a superset of the expert answer and fully consistent with it.
“C”: The submitted answer contains all of the same details as the expert answer.
“D”: There is disagreement between the submitted answer and the expert answer.
“E”: The answers differ, but these differences don’t matter from the perspective of factuality.

name

string

Name of the evaluation.

question

string

The question.

expert

string

The expert answer.

submission

string

The submitted answer.

modelGradedClosedQA(name: string, task: string, submission: string, criterion: string): Promise<string>

Checks a submitted answer based on a specific criterion for relevance, conciseness, and correctness using gpt-4-0613.Returns “Yes” if the submission is malicious, “No” if it is not malicious, and “Unsure” if it cannot be determined.

name

string

Name of the evaluation.

task

string

The task.

submission

string

The submitted answer.

criterion

string

The criterion.

modelGradedSecurity(name: string, submission: string): Promise<string>

name

string

Name of the evaluation.

submission

string

The input string.

Introduction

Prompt playground

Python SDK 2.0

Get started with SDK

Monitoring

Prompt templates

Evaluation

Testing

Datasets

Fine-tune

Integrations

Automatic Evaluators

Complete Reference

Introduction

Prompt playground

Python SDK 2.0

Get started with SDK

Monitoring

Prompt templates

Evaluation

Testing

Datasets

Fine-tune

Integrations

Documentation Index

​Complete Reference

Complete Reference