Qin et al. ask what happens when an open-source LLM is dropped into a marketplace of 16,464 real-world RESTful APIs from RapidAPI. Existing tool-use instruction tuning is either constrained to single tools (no multi-tool scenarios), locked to a small scenario set (poor diversity), or chained on a single reasoning path that cannot recover from one bad API call (CoT/ReACT error propagation). ToolLLM couples three things around a new reasoning algorithm: a data pipeline (ToolBench) that uses ChatGPT to crawl RapidAPI, sample APIs, generate instructions covering single-tool / intra-category multi-tool / intra-collection multi-tool scenarios, and annotate solution paths; a neural API retriever trained on instruction–API pairs over Sentence-BERT embeddings; and a fine-tuned LLaMA-2 7B (ToolLLaMA) equipped with DFSDT — depth-first search-based decision tree, which expands a reasoning tree pre-order, can abort a dead-end branch via the Finish by Giving Up action, and degrades to ReACT when no retraction is needed.
The numbers are uneven by design. ToolLLaMA + DFSDT-Retriever reaches 70.0 average pass rate across four generalization splits, beating ChatGPT-DFSDT (64.8) but trailing GPT-4-DFSDT (71.0); on out-of-distribution APIBench (HuggingFace, TorchHub, TensorHub), the same model posts 16.77 / 51.16 / 40.59 AST accuracy with no APIBench training, beating Gorilla-RS-BM25 on HuggingFace and TorchHub. But Vicuna and Alpaca fine-tuned on ToolBench still score 0.0 pass rate on every split — instruction tuning optimized for dialogue does not transfer to tool use. ToolEval (Pass Rate + Win Rate) agrees with human annotators at 87.1% and 80.3% respectively. The session below is a teaching schematic — Figure 4’s Hailee Steinfeld trace is reproduced verbatim, but the tree-vs-chain comparison and the per-round middle steps are compressed to fit the tty pane. The argument and the tables are in the paper; this site does not host the PDF.
teaching explainer · Qin et al. · arXiv 2307.16789 · LLaMA-2 7B / ChatGPT / GPT-4 · arXiv:2307.16789 | traces are schematic session logs; numbers marked paper-report come from the tables
0overview
core: DFSDT lets an LLM expand reasoning on RapidAPI tool calls: a depth-first decision tree where a wrong API can be aborted via Finish by Giving Up and a sibling node tried next.
ToolBench
16,464 real-world RapidAPI RESTful APIs across 49 categories, instruction-tuned with ChatGPT — API collection, instruction generation, solution path annotation.
DFSDT
Pre-order DFS over a reasoning tree; Finish with Final Answer and Finish by Giving Up are the two terminal actions. Degrades to ReACT when no retraction is needed.
ToolEval
Pass Rate (soluble vs. unsolvable × two finish types) and Win Rate (paired paths scored on richness / factuality / reasoning / milestone / exploration / cost); 87.1% and 80.3% agreement with human annotators.
Figure 4 illustrates DFSDT vs. CoT / ReACT. ReACT is a degraded DFSDT (paper Appendix A.4 verbatim). The trial below reuses paper Figure 4 verbatim for the Hailee Steinfeld trace; the chain-vs-tree comparison in §1 is compressed to fit the tty pane.
1problem: linear chains cannot abandon a bad API
point: same RapidAPI instruction across three reasoning strategies. CoT does not call any API; ReACT commits to one API per step and cannot backtrack once the first call returns an error; DFSDT expands a tree, aborts via Finish by Giving Up, and finds a sibling API that works. Paper Figure 4 left paints this as Normal → Normal → Error → Error → Fail for ReACT and a tree for DFSDT.
instruction and the DFSDT pane's Round 1 Thought / API / Observation are paper Figure 4 verbatim. Middle rounds are marked x N per Figure 4. CoT and ReACT panes are schematic — paper Figure 2 reports aggregated pass / win rates but no per-pane CoT trace for this instruction.
Instruction (paper Figure 4 verbatim) · I want to give my friend a birthday surprise. I know her favorite actress is Hailee Steinfeld. Help me please!
CoT (no API) ReACT chain DFSDT tree node API call API response Finish with Final Answer Finish by Giving Up FAIL
2mechanism: pre-order DFS, two terminal actions
point: DFSDT expands a reasoning tree pre-order. At each node the model emits : either an API call (advance) or one of two terminal actions — Finish with Final Answer or Finish by Giving Up. Giving Up abandons the current branch so the algorithm can try a sibling. The expected cost is in OpenAI API calls. When the model does not retract any action, DFSDT degrades to ReACT (paper Appendix A.4 verbatim).
DFS expansion API call (advance) API response Finish by Giving Up Finish with Final Answer
3case: RapidAPI tool chain, paper Figure 4 verbatim
point: a friend wants a birthday surprise for an admirer of actress Hailee Steinfeld. DFSDT enters the tree at the root, calls get_extra_character_details to retrieve Hailee's age and recent movies, and after several rounds the algorithm reaches one of two terminals: Finish with Final Answer (return the suggested gift) or Finish by Giving Up (abandon this branch, try a sibling API).
user instruction, Round 1 Thought / API Name / Arguments / Observation, the final-answer Arguments text, and the Finish by Giving Up action name are all paper Figure 4 verbatim. Middle rounds are marked x N per Figure 4 — no expanded middle steps were given in the paper.
Thought API call (RapidAPI) Observation (API response) Finish with Final Answer (paper Figure 4) Finish by Giving Up (paper Figure 4)
4results and limits
point: ToolLLaMA + DFSDT-Retriever reaches 70.0 average pass rate across four generalization splits, beating ChatGPT-DFSDT (64.8) but trailing GPT-4-DFSDT (71.0). On out-of-distribution APIBench, the same model posts 16.77 / 51.16 / 40.59 AST accuracy on HuggingFace / TorchHub / TensorHub — beating Gorilla-RS-BM25 on HuggingFace (16.77 vs. 15.71) and TorchHub (51.16 vs. 50.00). But Vicuna and Alpaca fine-tuned on ToolBench score 0.0 pass rate on every split.
ReACT DFSDT ChatGPT GPT-4 paper-report numbers
70.0%
ToolLLaMA-DFSDT-Retriever Avg pass (vs. ChatGPT 64.8, GPT-4 71.0)
+28.5 pp
DFSDT vs. ReACT avg pass rate gain (paper Table 3, ChatGPT)
87.1%
ToolEval pass rate agreement with human annotators (paper §3.1)
0.0%
Vicuna / Alpaca fine-tuned pass rate — dialogue instruction tuning does not transfer (paper §3.2)
The four cards are paper-report values, not computed from the printout above. Win rate ties are split into win + lose in Table 6 (footnote); most splits of DFSDT vs. ChatGPT-ReACT show win < 50% on that table — ToolLLaMA is competitive but not dominant.
5limits
point: paper-acknowledged limits. The tree search costs OpenAI API calls per task. When the model does not retract any action, DFSDT degrades to ReACT (paper Appendix A.4 verbatim). Solution path annotation itself can hallucinate API names — ChatGPT generates the trace, the pipeline filters some, the rest is training data. On Tie-split win rate (Table 6), most DFSDT vs. ChatGPT-ReACT splits drop below 50%.
paper-acknowledged limit DFSDT → ReACT (Appendix A.4) Tie-split win rate < 50%
Schematic traces are not paper figures. Paper-report numbers: Qin, Liang, Ye, Zhu, Yan, Lu, Lin, Cong, Tang, Qian, Zhao, Hong, Tian, Xie, Zhou, Gerstein, Li, Liu, Sun, ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs, arXiv:2307.16789v2 (3 Oct 2023), arXiv:2307.16789. ToolBench data and trained models: github.com/OpenBMB/ToolBench. This site does not host the PDF.