AshAi.Actions.Evaluate (ash_ai v1.1.0)

Copy Markdown View Source

A generic action impl that asks an evaluation model typed questions about the action's inputs.

Evaluation models such as TypeSafe's Jev do not generate text. They take a state and a map of named, typed questions and return one typed answer per question, each with a probability distribution. This implementation maps an Ash action onto one such request:

  • the state is the action's arguments, as a JSON object keyed by argument name
  • the questions are derived from the action's return type
  • the result is the return type, cast from the answers

Return types

The return type must be an answer type, a map of answer types, or an array of an answer type:

  • AshAi.Evaluate.Choice, AshAi.Evaluate.Noul, or AshAi.Evaluate.Score asks one question. The action description is the question's instructions.
  • AshAi.Evaluate.Judgments (or :map whose fields are all answer types) asks one question per field in a single request. Each field's description is that question's instructions.
  • {:array, answer_type} asks a runtime-sized list of questions, one per entry returned by the questions option, and returns the answers in the same order.
  • AshAi.Actions.Result wrapping any of the above also returns the model that answered, token usage, and provider metadata.

Example

action :triage, AshAi.Evaluate.Judgments do
  argument :ticket, :string, allow_nil?: false

  constraints fields: [
    department: [type: MyApp.Department, description: "Which team should handle `ticket`?"],
    urgent: [type: :boolean, description: "Does `ticket` convey urgency?"]
  ]

  run evaluate("typesafe:jev-latest")
end

Dynamic questions

The questions option supplies instructions (and optionally criteria) at runtime. It is a function fn input, context -> questions end, or a static value, shaped like the return type:

  • single answer type: one question
  • Judgments or map: a map of field name to question, overriding those fields' descriptions; unnamed fields keep their description
  • {:array, answer_type}: a list of questions, one per element

A question is either instructions (a string, or a map or list for structured instructions), or a map with :instructions and :criteria. Criteria are the options of a Choice (a list, or a map of option to description), the levels of a Score, or the true/false descriptions of a Noul. Runtime criteria let the options differ per question; a Choice with an of constraint requires them to be a subset of its options, and a Choice without of returns string values.

action :rerank, {:array, AshAi.Evaluate.Score} do
  argument :query, :string, allow_nil?: false
  argument :candidates, {:array, :string}, allow_nil?: false
  constraints items: [levels: ["Irrelevant", "Partially relevant", "Answers the query"]]

  run evaluate("typesafe:jev-latest",
    questions: fn input, _ctx ->
      input.arguments.candidates
      |> Enum.with_index()
      |> Enum.map(fn {_candidate, i} -> "How well does `candidates[#{i}]` answer `query`?" end)
    end
  )
end

Options

  • :questions - Runtime questions, see above.
  • :state - Override the state. A string, map, or list, or a function fn input, context -> state end returning one.
  • :req_llm - Override the ReqLLM module (useful for testing with mocks).
  • :req_llm_opts - Additional options passed to ReqLLM.evaluate/4.

Model specification

The first argument to evaluate/2 is a ReqLLM model specification such as "typesafe:jev-latest", or a function returning one.