Tool Learning with Foundation Models

by   Yujia Qin, et al.

Humans possess an extraordinary ability to create and utilize tools, allowing them to overcome physical limitations and explore new frontiers. With the advent of foundation models, AI systems have the potential to be equally adept in tool use as humans. This paradigm, i.e., tool learning with foundation models, combines the strengths of specialized tools and foundation models to achieve enhanced accuracy, efficiency, and automation in problem-solving. Despite its immense potential, there is still a lack of a comprehensive understanding of key challenges, opportunities, and future endeavors in this field. To this end, we present a systematic investigation of tool learning in this paper. We first introduce the background of tool learning, including its cognitive origins, the paradigm shift of foundation models, and the complementary roles of tools and models. Then we recapitulate existing tool learning research into tool-augmented and tool-oriented learning. We formulate a general tool learning framework: starting from understanding the user instruction, models should learn to decompose a complex task into several subtasks, dynamically adjust their plan through reasoning, and effectively conquer each sub-task by selecting appropriate tools. We also discuss how to train models for improved tool-use capabilities and facilitate the generalization in tool learning. Considering the lack of a systematic tool learning evaluation in prior works, we experiment with 17 representative tools and show the potential of current foundation models in skillfully utilizing tools. Finally, we discuss several open problems that require further investigation for tool learning. Overall, we hope this paper could inspire future research in integrating tools with foundation models.


page 4

page 8

page 15

page 17

page 18

page 20

page 26

page 30


Foundation Models in Smart Agriculture: Basics, Opportunities, and Challenges

The past decade has witnessed the rapid development of ML and DL methodo...

Decentralised Governance for Foundation Model based AI Systems: Exploring the Role of Blockchain in Responsible AI

Foundation models including large language models (LLMs) are increasingl...

Large Language Models as Tool Makers

Recent research shows the potential of enhancing the problem-solving abi...

Making Language Models Better Tool Learners with Execution Feedback

Tools serve as pivotal interfaces that enable humans to understand and r...

Foundation models in brief: A historical, socio-technical focus

Foundation models can be disruptive for future AI development by scaling...

API-Bank: A Benchmark for Tool-Augmented LLMs

Recent research has shown that Large Language Models (LLMs) can utilize ...

GeoGPT: Understanding and Processing Geospatial Tasks through An Autonomous GPT

Decision-makers in GIS need to combine a series of spatial algorithms an...

Please sign up or login with your details

Forgot password? Click here to reset