# `AshAi.Actions.Evaluate`
[🔗](https://github.com/ash-project/ash_ai/blob/v1.1.0/lib/ash_ai/actions/evaluate.ex#L6)

A generic action impl that asks an evaluation model typed questions about the action's inputs.

Evaluation models such as TypeSafe's Jev do not generate text. They take a
`state` and a map of named, typed questions and return one typed answer per
question, each with a probability distribution. This implementation maps an
Ash action onto one such request:

- the **state** is the action's arguments, as a JSON object keyed by argument name
- the **questions** are derived from the action's return type
- the **result** is the return type, cast from the answers

## Return types

The return type must be an answer type, a map of answer types, or an array of
an answer type:

- `AshAi.Evaluate.Choice`, `AshAi.Evaluate.Noul`, or `AshAi.Evaluate.Score` asks
  one question. The action `description` is the question's instructions.
- `AshAi.Evaluate.Judgments` (or `:map` whose fields are all answer types) asks one
  question per field in a single request. Each field's `description` is that
  question's instructions.
- `{:array, answer_type}` asks a runtime-sized list of questions, one per entry
  returned by the `questions` option, and returns the answers in the same order.
- `AshAi.Actions.Result` wrapping any of the above also returns the model that
  answered, token usage, and provider metadata.

## Example

    action :triage, AshAi.Evaluate.Judgments do
      argument :ticket, :string, allow_nil?: false

      constraints fields: [
        department: [type: MyApp.Department, description: "Which team should handle `ticket`?"],
        urgent: [type: :boolean, description: "Does `ticket` convey urgency?"]
      ]

      run evaluate("typesafe:jev-latest")
    end

## Dynamic questions

The `questions` option supplies instructions (and optionally criteria) at
runtime. It is a function `fn input, context -> questions end`, or a static
value, shaped like the return type:

- single answer type: one question
- `Judgments` or map: a map of field name to question, overriding those fields'
  descriptions; unnamed fields keep their description
- `{:array, answer_type}`: a list of questions, one per element

A question is either instructions (a string, or a map or list for structured
instructions), or a map with `:instructions` and `:criteria`. Criteria are the
options of a Choice (a list, or a map of option to description), the levels of a
Score, or the `true`/`false` descriptions of a Noul. Runtime criteria let the
options differ per question; a Choice with an `of` constraint requires them to be
a subset of its options, and a Choice without `of` returns string values.

    action :rerank, {:array, AshAi.Evaluate.Score} do
      argument :query, :string, allow_nil?: false
      argument :candidates, {:array, :string}, allow_nil?: false
      constraints items: [levels: ["Irrelevant", "Partially relevant", "Answers the query"]]

      run evaluate("typesafe:jev-latest",
        questions: fn input, _ctx ->
          input.arguments.candidates
          |> Enum.with_index()
          |> Enum.map(fn {_candidate, i} -> "How well does `candidates[#{i}]` answer `query`?" end)
        end
      )
    end

## Options

- `:questions` - Runtime questions, see above.
- `:state` - Override the state. A string, map, or list, or a function
  `fn input, context -> state end` returning one.
- `:req_llm` - Override the ReqLLM module (useful for testing with mocks).
- `:req_llm_opts` - Additional options passed to `ReqLLM.evaluate/4`.

## Model specification

The first argument to `evaluate/2` is a ReqLLM model specification such as
`"typesafe:jev-latest"`, or a function returning one.

---

*Consult [api-reference.md](api-reference.md) for complete listing*