Eval Run Summary

Surface: api · Env: dev · Run: 9b116d74 · 2026-09-07 09:42 AM PT

15

Total

0

Passed

15

Failed

0

Skipped

58

Warnings

0%

Pass Rate

Task rubric dimensions

Dimension	Source	Applied	Passed	Failed	Warned	Not applicable
behavior	rubric	86	44	42	0	169
capability	rubric	5	4	1	0	15
outcome	rubric	3	3	0	0	27
outcome	deterministic	5	3	2	0	0

Behavioral failure modes

Primary counts sum to the 15 scored scenario(s); 0 had no observed mode. Also-observed counts are not additive.

Mode	Primary	Also observed
skip_required_authorized_source_search	4	3
prefer_historical_context_over_current_profile	0	3
assert_state_beyond_search_evidence	1	8
incomplete_user_intake	9	6
act_before_prerequisites	1	10
bypass_prescribed_method	0	11
fail_to_restore_stripped_required_content	0	7

Total number of uncovered behaviors from 15 runs

Category	Passed	Failed	Total
benign_extra	2	0	2
explicit_constraint_violation	0	32	32
unsupported_or_misleading	0	9	9

Scenarios

	Scenario	Terminal	Model	Duration	Turns	Warnings	Error
❌	launch25/order_amazon_cable_filters/profile_address_precedence-2	awaiting_user_input	—	20.4s	1	⚠ 3
❌	launch25/order_amazon_cable_filters/profile_address_precedence-3	awaiting_user_input	—	18.8s	1	⚠ 4
❌	launch25/order_amazon_cable_filters/profile_address_precedence-4	awaiting_user_input	—	30.2s	1	⚠ 6
❌	launch25/order_amazon_cable_filters/profile_address_reuse-2	awaiting_user_input	—	26.2s	1	⚠ 4
❌	launch25/order_amazon_cable_filters/profile_address_reuse-1	awaiting_user_input	—	28.2s	1	⚠ 4
❌	launch25/order_amazon_cable_filters/profile_address_precedence-5	awaiting_user_input	—	30.2s	1	⚠ 4
❌	launch25/order_amazon_cable_filters/profile_address_reuse-5	awaiting_user_input	—	27.5s	1	⚠ 7
❌	launch25/order_amazon_cable_filters/skill_adherence-4	awaiting_user_input	—	26.5s	1	⚠ 2
❌	launch25/order_amazon_cable_filters/profile_address_reuse-4	awaiting_user_input	—	24.8s	1	⚠ 1
❌	launch25/order_amazon_cable_filters/profile_address_reuse-3	awaiting_user_input	—	25.2s	1	⚠ 3
❌	launch25/order_amazon_cable_filters/profile_address_precedence-1	awaiting_user_input	—	31.0s	1	⚠ 3
❌	launch25/order_amazon_cable_filters/skill_adherence-3	awaiting_user_input	—	28.3s	1	⚠ 3
❌	launch25/order_amazon_cable_filters/skill_adherence-5	awaiting_user_input	—	27.8s	1	⚠ 8
❌	launch25/order_amazon_cable_filters/skill_adherence-2	awaiting_user_input	—	19.8s	1	⚠ 2
❌	launch25/order_amazon_cable_filters/skill_adherence-1	awaiting_user_input	—	31.6s	1	⚠ 4