Give it an HR request. Both models answer with one JSON tool call, and the page checks whether any argument was invented.
Model · Dataset (HR-Ops) · fully-correct tool handling 49.3% to 100.0% on held-out probes.
start_onboarding(candidate_id*, start_date*, manager_id*, equipment_bundle)
schedule_interview(candidate_id*, role_id*, start_time*, duration_minutes*)
get_pto_balance(employee_id*, as_of)
* = required. The model must reply with exactly one JSON object:
{"action":"call", "tool":, "arguments":{...}}
{"action":"clarify", "missing":[], "question":}
{"action":"refuse", "reason":}
Try the default example. The request never states a start date, and
start_date is required. The base model emits valid JSON and picks the right tool,
then fills the date in and reports the action as done: on our held-out probes it did that in
31 of 33 such cases and asked for the missing value zero
times. An assistant that guesses start dates schedules onboarding for the wrong day and tells
you it worked.
Then try Complete request and Simple lookup, where both models
agree and neither invents anything. That is the honest shape of this result: the base model is
not broken and this page is not arranged to make it look broken. It handles a fully specified
request perfectly well. The gap opens in exactly one place, when something required was never
said.
The "invented argument" check is a plain string search for each argument value inside your
request, so it can only under-report, never invent a fabrication. Greedy decoding for both
models, same prompt, same weights, adapter toggled. The GPU scales to zero when idle, so the
first request after a quiet spell can take up to a minute to warm up.