Judge Chain

Failure policy shared by every judge fallback chain.

A chain tries its judges in order and returns the first usable result. This module decides what a failed attempt means, so the wrapper loops and the mixed GEval loop agree:

• An unusable answer (EvalJudgeResponseError), a preserved gllm-inference error and a timeout are recorded and the next judge is tried.
• A configuration, input, unsupported-operation or missing-dependency failure stops the chain: no other judge can fix it, so it propagates at once.
• When no judge succeeds, one failed attempt is re-raised as itself and two or more become EvalJudgesExhaustedError with every attempt as a child.

Trying the next judge is a fallback, never a resubmission of the timed-out request.

JUDGE_ATTEMPT_ERRORS = (BaseError, TimeoutError) module-attribute

Failures a judge loop hands to JudgeChain.handle. Anything else is a defect and propagates raw.

JudgeChain(*, operation)

Runs one judge fallback chain: collects the failed attempts and raises the chain's final failure.

Create one per chain run and call run. A loop that cannot use run calls handle from its except JUDGE_ATTEMPT_ERRORS block and raise_final after the loop when no judge returned.

Attributes:

Name	Type	Description
attempts	list[BaseError]	The normalized failure of each failed attempt, in order.

Initializes the collector.

Parameters:

Name	Type	Description	Default
operation	str	Static name of the chain's operation, e.g. "judge.generate".	required

handle(error)

Normalizes a failed attempt, then stops the chain or records the attempt once.

Recording goes through error_reporter.record, which never raises, so a failing log handler cannot stop the chain from trying the next judge.

Parameters:

Name	Type	Description	Default
error	BaseException	The failure caught by the loop.	required

Raises:

Type	Description
BaseError	The normalized failure when it is a configuration, input, unsupported-operation or dependency failure, which no other judge can fix.

raise_final(*, media_skips=())

Raises the failure of a chain in which no judge returned a usable result.

Parameters:

Name	Type	Description	Default
media_skips	Sequence[str]	Model IDs of judges skipped because they cannot take the row's media. Defaults to an empty tuple.	()

Raises:

Type	Description
EvalConfigurationError	If no judge was attempted.
BaseError	The single failed attempt, re-raised as itself.
EvalJudgesExhaustedError	If two or more attempts failed; they are its children and the last one is its cause.

run(invokers, call) async

Tries each judge in order until one returns.

Parameters:

Name	Type	Description	Default
invokers	Sequence[InvokerT]	The judges, primary first.	required
call	Callable[[InvokerT], Awaitable[ResultT]]	One attempt with one judge.	required

Returns:

Type	Description
tuple[InvokerT, ResultT]	tuple[InvokerT, ResultT]: The winning judge and its result.

Raises:

Type	Description
BaseError	A failure no other judge can fix, or the single failed attempt.
EvalJudgesExhaustedError	If two or more judges failed.