Preprint · October 21, 2026

AgentOrchHow Close Can Global Orchestration Come to Step-by-Step Interaction?

Zijie Liu¹, Qi Yang Huang², Jinhao Duan¹, Yueh Che Lee², Thomas A. Joseph², Zayne Turner², Adam Seligman², Tianlong Chen¹, Zhaozhuo Xu²

¹University of North Carolina at Chapel Hill · ²Workato · Contact: jesseliu@cs.unc.edu

Leaderboard Dataset Closing the gap BibTeX

650enterprise SOPs

3,220runtime variants

64tool interfaces

~79%gap shrink, GPT-3.5 → Opus 5T

23.2×more LLM calls for Step-by-Step

Abstract

Can an agent plan an entire enterprise workflow once, then run it with no further LLM calls, and still match an agent that thinks after every step?

Enterprise workflow agents typically rely on step-by-step interaction: execute an action, observe its result, call the LLM again. Global orchestration instead generates a complete executable workflow in advance and runs it without further LLM calls, which cuts inference cost but has been less accurate.

AgentOrch compares the two directly on 650 enterprise SOPs and 3,220 runtime variants under matched tools, environments, and evaluation. Step-by-Step remains more accurate for every evaluated model, but the gap has shrunk by about 79% from early models to the strongest recent model. Three routes narrow it further: incremental workflow construction (+9.98 pp), execution-feedback revision (+6.13 pp on held-out variants), and targeted training on complete workflows (+12.18 pp across five models).

Overview

Same SOPs, same tools, same simulator, same evaluator for every protocol.

Diagram of AgentOrch benchmark construction (reference recipes, fixed tool bank, input variants and task specification, simulated execution) and the model evaluation pipeline comparing Global plan-once execution with Step-by-Step interactive execution. — **Figure 1.** Benchmark construction and evaluation. Left: reference recipes define the SOPs; their tool operations are mapped to a fixed bank of 64 tools, and task specifications and runtime variants are built and validated. Right: Global generates one workflow and reuses it across variants; Step-by-Step selects tool calls using intermediate results. Both run in the same simulator, which reproduces real-connector behavior at 98.54% Orchestration Accuracy under reference-action replay.

Four protocols

Global and Step-by-Step are the two paradigms being compared; Local and Replan are two ways to strengthen Global.

Global plan once

Generates one complete workflow before seeing any runtime input, then reuses it across all variants.

spec→LLM ×1→workflow→run N variants

Step-by-Step ReAct-style

Receives the concrete input, emits one tool call, observes the result, decides the next call.

spec + input→LLM⇄tool… until done

Local incremental

Same information as Global, but builds the workflow node by node through a validating append-only editor.

spec→LLM⇄validator→workflow

Replan feedback

Runs the Global workflow, returns its execution trace, and asks for a revised complete workflow. Still zero LLM calls at run time.

workflow→run→LLM revise× 3 rounds

Metrics

Every protocol is scored from its realized execution by the same evaluator.

Orchestration Accuracy

Primary metric: mean of Execution Trace Accuracy and Side-Effect Accuracy for each runtime variant.

Execution Trace Accuracy

Mean of five binary checks against the reference SOP: tool selection, tool order, tool execution, required arguments, control path.

Side-Effect Accuracy

Required state changes are present, with no missing, duplicated, or unintended effects.

Final Accuracy

Correct only when both side effects and terminal output are correct, independent of the trajectory.

Error Rate

Share of variants with no scoreable execution. These stay in the denominator and score zero.

Leaderboard

Orchestration Accuracy (%) of 31 models on all 3,220 runtime variants.

Global orchestration Step-by-Step interaction Bar = gap (pp)

#	Model	Global	Step-by-Step	Gap

Failed executions score zero and stay in the denominator. Gap = Step-by-Step − Global, in percentage points.

Findings

Step-by-Step leads for every model. Stronger models alone do not reliably close the gap.

Scale

Qwen2.5 Global accuracy drops from 16.18% at 32B to 9.91% at 72B. The gap goes 36.33 → 27.87 → 36.66 pp from 0.5B to 7B to 72B. Gemma-3’s gap grows from 26.60 pp (4B) to 32.52 pp (27B).

Newer generations

Global improves in four of five generation pairs (+5.80 to +17.98 pp), but the gap narrows only for Qwen (−3.88) and GPT (−18.64) and widens for Gemma, Llama and GLM.

Reasoning and runtime inputs

Thinking mode helps Global in four of eight models (+13.44 pp for Claude Opus 5, −9.87 pp for DeepSeek-V4-Flash). Giving Global the concrete runtime input changes accuracy by only −0.18 pp on average.

Closing the gap

Three routes, each drawing on a stronger signal than the last.

Route 1 · Local

+9.98 pp

Incremental workflow construction

Same information as Global, but the workflow is built node by node with validator feedback after each edit, and still with no runtime execution. Five of six models improve; the largest gains are for Llama-3.3-70B and Qwen2.5-3B.

Model	Global	Local	Δ
Llama-3.3-70B	10.4	34.9	+24.5
Qwen2.5-3B	2.2	22.8	+20.6
Mistral-Small-3.1-24B	14.1	27.3	+13.1
Qwen3.8-27B-Think	33.0	41.5	+8.5
Qwen3-30B-A3B	8.4	8.5	+0.1
Qwen3-4B	13.8	6.9	−6.9

Route 2 · Replan

+6.13 pp

Execution-feedback revision

The model sees how its workflow executed and rewrites it, over three rounds. Feedback comes from two variants per SOP; the scores are on held-out variants it never saw. Most of the gain comes from repairing invalid workflows: 83.9% of the repaired ones parsed fine but had wrong tool arguments or references between steps.

Model	Base	Final	Δ
Qwen3.8-27B	29.11	40.07	+10.96
DeepSeek-V4-Flash	27.21	35.46	+8.25
Mistral-Small-3.1-24B	14.52	21.89	+7.37
Gemma-4-26B-A4B	3.82	9.62	+5.80
Gemma-4-12B	14.47	20.24	+5.77
Qwen3-4B	14.24	19.13	+4.89
Qwen3-30B-A3B	8.50	11.14	+2.64
Gemma-3-27B	7.80	11.13	+3.33

Route 3 · Training

+12.18 pp

Training on complete workflows

SFT and GRPO on 3,000 source recipes disjoint from the benchmark. SFT improves all five models, and RL adds more for three of them. Bold marks the better of SFT and RL, which Δ compares with the base model.

Model	Base	SFT	RL	Δ
Gemma-4-26B-A4B	4.05	19.89	23.93	+19.88
Qwen3-4B	13.75	25.46	27.16	+13.41
Qwen3.8-27B	28.57	41.96	41.14	+13.39
Gemma-4-12B	14.26	17.92	21.66	+7.40
Gemma-4-31B	12.71	19.55	19.10	+6.84

Analysis

Where the gap is largest, and what each protocol costs.

Table 1. The gap grows with workflow size and nesting depth. Averaged over 31 models; 95% bootstrap CIs clustered by SOP.

Subset	SOPs	Variants	Global	Step-by-Step	Gap	95% CI
All workflows	650	3220	13.8	46.4	+32.6	[31.2, 34.2]
Workflow size
Short (1–2 calls)	234	1035	18.8	45.2	+26.3	[24.5, 28.4]
Medium (3–5 calls)	251	1149	14.2	47.3	+33.1	[30.9, 35.1]
Long (≥6 calls)	165	1036	8.3	46.6	+38.3	[35.1, 41.3]
Control-flow composition
Linear	186	781	14.4	39.7	+25.3	[22.7, 28.0]
Branch only	132	688	14.9	48.6	+33.6	[30.2, 36.9]
Loop only	36	148	8.9	30.9	+22.0	[18.6, 26.2]
Error handling only	78	295	16.6	48.8	+32.2	[28.6, 36.1]
Multiple control-flow types	218	1308	12.8	50.4	+37.7	[35.2, 40.4]
Maximum nesting depth
1	186	781	14.4	39.7	+25.3	[22.9, 28.0]
2	235	1030	15.0	47.8	+32.8	[30.4, 35.3]
3+	229	1409	12.6	49.0	+36.5	[34.2, 38.8]

Table 2. Global uses fewer, longer calls. Step-by-Step needs 15.4×–35.4× more LLM calls and 1.42×–5.29× more completion tokens.

Model	Calls Global	Calls Step	×	Tokens Global (M)	Tokens Step (M)	×
Claude Opus 5 Thinking	650	11,964	18.4×	0.91	1.30	1.42×
GPT-5.6 Sol Thinking	647	11,312	17.5×	0.59	1.40	2.37×
GPT-4o	650	10,042	15.4×	0.68	1.08	1.60×
Qwen3.8-27B	649	17,539	27.0×	1.07	5.67	5.29×
Qwen3-30B-A3B	650	22,997	35.4×	1.13	3.71	3.28×
GLM-5.2	650	15,228	23.4×	0.81	2.55	3.13×
Gemma-4-31B	649	12,484	19.2×	1.16	2.89	2.48×

Citation

@misc{liu2026agentorch,
  title  = {AgentOrch: How Close Can Global Orchestration Come to Step-by-Step Interaction?},
  author = {Liu, Zijie and Huang, Qi Yang and Duan, Jinhao and Lee, Yueh Che and Joseph, Thomas A.
            and Turner, Zayne and Seligman, Adam and Chen, Tianlong and Xu, Zhaozhuo},
  year   = {2026}
}