A Survey on Self-Evolution of Large Language Models

Abstract

Large language models (LLMs) have significantly advanced in various fields and intelligent agent applications. However, current LLMs that learn from human or external model supervision are costly and may face performance ceilings as task complexity and diversity increase. To address this issue, self-evolution approaches that enable LLM to autonomously acquire, refine, and learn from experiences generated by the model itself are rapidly growing. This new training paradigm inspired by the human experiential learning process offers the potential to scale LLMs towards superintelligence. In this work, we present a comprehensive survey of self-evolution approaches in LLMs. We first propose a conceptual framework for self-evolution and outline the evolving process as iterative cycles composed of four phases: experience acquisition, experience refinement, updating, and evaluation. Second, we categorize the evolution objectives of LLMs and LLM-based agents; then, we summarize the literature and provide taxonomy and insights for each module. Lastly, we pinpoint existing challenges and propose future directions to improve self-evolution frameworks, equipping researchers with critical insights to fast-track the development of self-evolving LLMs. Our corresponding GitHub repository is available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/Awesome-Self-Evolution-of-LLM.

A Survey on Self-Evolution of Large Language Models

Zhengwei Tao${}^{\mathrm{12}}$, Ting-En Lin${}^{\mathrm{2}}$, Xiancai Chen${}^{\mathrm{1}}$, Hangyu Li${}^{\mathrm{2}}$, Yuchuan Wu${}^{\mathrm{2}}$, Yongbin Li${}^{\mathrm{2}}$, Zhi Jin${}^{\mathrm{1}\mathbf{}\mathrm{†}}$, Fei Huang${}^{\mathrm{2}}$, Dacheng Tao${}^{\mathrm{3}}$, Jingren Zhou${}^{\mathrm{2}}$ ${}^1$ Key Lab of HCST (PKU), MOE; School of Computer Science, Peking University ${}^2$Alibaba Group  ${}^3$Nanyang Technological University {tttzw, xiancaich}@stu.pku.edu.cn,  zhijin@pku.edu.cn {ting-en.lte, shengxiu.wyc, shuide.lyb, jingren.zhou}@alibaba-inc.com dacheng.tao@ntu.edu.sg

Figure 1: Training paradigms shift of LLMs.

1 Introduction

With the rapid development of artificial intelligence, large language models (LLMs) like GPT-3.5 Ouyang et al. (2022), GPT-4 Achiam et al. (2023), Gemini Team et al. (2023), LLaMA Touvron et al. (2023a, b), and Qwen Bai et al. (2023) mark a significant shift in language understanding and generation. These models undergo three stages of development as shown in Figure 1: pre-training on large and diverse corpora to gain a general understanding of language and world knowledge Devlin et al. (2018); Brown et al. (2020), followed by supervised fine-tuning to elicit the abilities of downstream tasks Raffel et al. (2020); Chung et al. (2022). Finally, the human preference alignment training enables the LLMs to respond as human behaviors Ouyang et al. (2022). Such successive training paradigms achieve significant breakthroughs, enabling LLMs to perform a wide range of tasks with remarkable zero-shot and in-context capabilities, such as question answering Tan et al. (2023), mathematical reasoning Collins et al. (2023), code generation Liu et al. (2024b), and task-solving that require interaction with environments Liu et al. (2023b).

Despite these advancements, humans anticipate that the emerging generation of LLMs can be tasked with assignments of greater complexity, such as scientific discovery Miret and Krishnan (2024) and future events forecasting Schoenegger et al. (2024). However, current LLMs encounter challenges in these sophisticated tasks due to the inherent difficulties in modeling, annotation, and the evaluation associated with existing training paradigms Burns et al. (2023). Furthermore, the recently developed Llama-3 model has been trained on an extensive corpus comprising 15 trillion tokens. It’s a monumental volume of data, suggesting that significantly scaling model performance by adding more real-world data could pose a limitation. This has attracted interest in self-evolving mechanisms for LLMs, akin to the natural evolution of human intelligence and illustrated by AI developments in gaming, such as the transition from AlphaGo Silver et al. (2016) to AlphaZero Silver et al. (2017). AlphaZero’s self-play method, requiring no labeled data, showcases a path forward for LLMs to surpass current limitations and achieve superhuman performance without intensive human supervision.

Drawing inspiration from the paradigm above, research on the self-evolution of LLMs has rapidly increased at different stages of model development, such as self-instruct Wang et al. (2023b), self-play Tu et al. (2024), self-improving Huang et al. (2022), and self-training Gulcehre et al. (2023). Notably, DeepMind’s AMIE system Tu et al. (2024) outperforms primary care physicians in diagnostic accuracy, and Microsoft’s WizardLM-2 exceeds the performance of the initial version of GPT-4. Both models are developed using self-evolutionary frameworks with autonomous learning capabilities and represent a potential LLM training paradigm shift. However, the relationships between these methods remain unclear, lacking systematic organization and analysis.

Therefore, we first comprehensively investigate the self-evolution processes in LLMs and establish a conceptual framework for their development. This self-evolution is characterized by an iterative cycle involving experience acquisition, experience refinement, updating, and evaluation, as shown in Figure 2. During the cycle, an LLM initially gains experiences through evolving new tasks and generating corresponding solutions, subsequently refining these experiences to obtain better supervision signals. After updating the model in-weight or in-context, the LLM is evaluated to measure progress and set new objectives.

The concept of self-evolution in LLMs has sparked considerable excitement across various research communities, promising a new era of models that can adapt, learn, and improve autonomously, akin to human evolution in response to changing environments and challenges. Self-evolving LLMs are not only able to transcend the limitations of current static, data-bound models but also mark a shift toward more dynamic, robust, and intelligent systems. This survey deepens understanding of the emerging field of self-evolving LLMs by providing a comprehensive overview through a structured conceptual framework. We trace the field’s evolution from the past to the latest cutting-edge methods and applications while examining existing challenges and outlining future research directions, paving the way for significant advances in developing self-evolution frameworks and next-generation models.

The survey is organized as follows: We first present the overview of self-evolution (§ 2), including background and conceptual framework. We summarize existing evolving abilities and domains of current methods(§ 3). Then, we provide in-depth analysis and discussion on the latest advancements in different phases of the self-evolution process, including experience acquisition (§ 4), experience refinement (§ 5), updating (§ 6), and evaluation (§ 7). Finally, we outline open problems and prospective future directions (§ 8).

Figure 2: Conceptual framework of self-evolution. For the $t^{th}$ iteration: ${ℰ}^t$ is the evolution objective; ${𝒯}^t$ and ${𝒴}^t$ denote the task and solution; ${ℱ}^t$ represents feedback; $M^t$ is the current model. Refined experiences are marked as ${\widetilde{𝒯}}^t$ and ${\widetilde{𝒴}}^t$, leading to the evolved model $\tilde{M}$. $\mathrm{ENV}$ is the environment. The whole self-evolution starts at ${ℰ}^1$.

2 Overview

In this section, we will first discuss the background of self-evolution and then introduce the proposed conceptual framework.

2.1 Background

Self-Evolution in Artificial Intelligence. Artificial Intelligence represents an advanced form of intelligent agent, equipped with cognitive faculties and behaviors mirroring those of humans. The aspiration of AI developers lies in enabling AI to harness self-evolutionary capabilities, paralleling the experiential learning processes characteristic of human development. The concept of self-evolution in AI emerges from the broader fields of machine learning and evolutionary algorithms Bäck and Schwefel (1993). Initially influenced by the principles of natural evolution, such as selection, mutation, and reproduction, researchers have developed algorithms that simulate these processes to optimize solutions to complex problems. The landmark paper by Holland (1992), which introduced the genetic algorithm, marks a foundational moment in the history of AI’s capability for self-evolution. Subsequent developments in neural networks and deep learning have furthered this capability, allowing AI systems to modify their own architectures and improve performance without human intervention Liu et al. (2021).

Can Artificial Entities Evolve Themselves? Philosophically, the question of whether artificial entities can self-evolve touches on issues of autonomy, consciousness, and agency. While some philosophers argue that true self-evolution in AI would require some form of consciousness or self-awareness, others maintain that mechanical self-improvement through algorithms does not constitute genuine evolution Chalmers (1997). This debate often references the works of thinkers like Dennett (1993), who explore the cognitive processes under human consciousness and contrast them with artificial systems. Ultimately, the philosophical inquiry into AI’s capacity for self-evolution remains deeply intertwined with interpretations of what it means to ’evolve’ and whether such processes can purely be algorithmic or must involve emergent consciousness Searle (1986).

2.2 Conceptual Framework

In the conceptual framework of self-evolution, we describe a dynamic, iterative process mirroring the human ability to acquire and refine skills and knowledge. This framework is encapsulated within Figure 2, emphasizing the cyclical nature of learning and improvement. Each iteration of the process focuses on a specific evolution goal, allowing the model to engage in relevant tasks, optimize its experiences, update its architecture, and evaluate its progress before moving to the next cycle.

Experience Acquisition

At the $t^{th}$ iteration, the model identifies an evolution objective ${ℰ}^t$. Guided by this objective, the model embarks on new tasks ${𝒯}^t$, generating solutions ${𝒴}^t$ and receiving feedback ${ℱ}^t$ from the environment, $\mathrm{ENV}$. This stage culminates in the acquisition of new experiences $({𝒯}^t,{𝒴}^t,{ℱ}^t)$.

Experience Refinement

After experience acquisition, the model examines and refines these experiences. This involves discarding incorrect data and enhancing imperfect ones, resulting in refined outcomes $({\widetilde{𝒯}}^t,{\widetilde{𝒴}}^t)$.

Updating

Leveraging the refined experiences, the model undergoes an update process, integrating $({\widetilde{𝒯}}^t,{\widetilde{𝒴}}^t)$ into its framework. This ensures the model remains current and optimized.

Evaluation

The cycle concludes with an evaluation phase, where the model’s performance is assessed through an evaluation in external environment. The outcomes of this phase inform the objective ${ℰ}^{t+1}$, setting the stage for the subsequent iteration of self-evolution.

The conceptual framework outlines the self-evolution of LLMs, akin to human-like acquisition, refinement, and autonomous learning processes. We illustrate our taxonomy in Figure 3.

for tree= grow=east, growth parent anchor=west, parent anchor=east, child anchor=west, [Self-Evolution of LLMs, root, parent anchor=south [Evaluation (§7), node_lv1 [Qualitative, node_lv2, [Yang et al. (2023b) , LLM Explanation Zheng et al. (2024a) , ChatEval Chan et al. (2023) , node_lv3], ] [Quantitative, node_lv2, [ LLM-as-a-Judge Zheng et al. (2024a); Dubois et al. (2024) , Reward Score Ouyang et al. (2022) , node_lv3], ] ] [Updating (§6), node_lv1, [In-Context, node_lv2 [Working Memory: Reflexion Shinn et al. (2023), IML Wang et al. (2024a), EvolutionaryAgent Li et al. (2024c) , Agent-Pro Zhang et al. (2024d), ProAgent Zhang et al. (2024a) , node_lv3], [External Memory: MoT Li and Qiu (2023), MemoryBank Zhong et al. (2024b), TiM Liu et al. (2023a), IML Wang et al. (2024a), TRAN Yang et al. (2023b), MemGPT Packer et al. (2023) UA${}^2$ Yang et al. (2024d) , ICE Qian et al. (2024), AesopAgent Wang et al. (2024d) , node_lv3], ] [In-Weight, node_lv2 [Architecture: LoRA Hu et al. (2021), ConPET Song et al. (2023), Model Soups Wortsman et al. (2022), DAIR Yu et al. (2023a), UltraFuser Ding et al. (2024), EvoLLM Akiba et al. (2024) , node_lv3], [Regularization: InstuctGPT Ouyang et al. (2022), FuseLLM Wan et al. (2024) , Elastic Reset Noukhovitch et al. (2024), WARM Ramé et al. (2024), AMA Lin et al. (2024) , node_lv3], [Replay: RFT Yuan et al. (2023), ReST Gulcehre et al. (2023); Aksitov et al. (2023) , AMIE Tu et al. (2024), SOTOPIA-$\pi$ Wang et al. (2024e), LLM2LLM Lee et al. (2024), LTC Wang et al. (2023a), A${}^3$T Yang et al. (2024c), SSR Huang et al. (2024a), SDFT Yang et al. (2024b) , node_lv3], ] ] [Experience Refinement (§5), node_lv1, [Correcting, node_lv2, [Critique-Free: STaR Zelikman et al. (2022), Self-Debugging Chen et al. (2023c), IterRefinement Chen et al. (2023b), Clinical SV Gero et al. (2023) , node_lv3], [Critique-Based: Self-Refine Madaan et al. (2023), CAI Bai et al. (2022), RCI Kim et al. (2023) , SELF Lu et al. (2023), CRITIC Gou et al. (2023), SelfEvolve Jiang et al. (2023), ISR-LLM Zhou et al. (2023b), Reflexion Shinn et al. (2024) , node_lv3], ] [Filtering, node_lv2, [Metric-Free: Self-Consistency Wang et al. (2022) , LMSI Huang et al. (2022), Self-Verification Weng et al. (2023) , CodeT Chen et al. (2022) , node_lv3] [Metric-Based: ReST${}^{EM}$ Singh et al. (2023), AutoAct Qiao et al. (2024), Self-Talk Ulmer et al. (2024), Self-Instruct Wang et al. (2023b) , node_lv3], ] ] [Experience Acquisition (§4), node_lv1 , [Feedback (§4.3), node_lv1_5 [Environment, node_lv2, [ SelfEvolve Jiang et al. (2023), Self-Debugging Chen et al. (2023c), Reflexion Shinn et al. (2023), CRITIC Gou et al. (2023), RoboCat Bousmalis et al. (2023), SinViG Xu et al. (2024b), SOTOPIA-$\pi$ Wang et al. (2024e) , node_lv3_5], ] [Model, node_lv2, [ Self-Reward Yuan et al. (2024), LSX Stammer et al. (2023), DLMA Liu et al. (2024a), SIRLC Pang et al. (2023), Self-Alignment Zhang et al. (2024e), CAI Bai et al. (2022), Self-Refine Madaan et al. (2023) , node_lv3_5], ] ] [Solution (§4.2), node_lv1_5 [Negative, node_lv2, [Perturbative: RLCD Yang et al. (2023a), DLMA Liu et al. (2024a), Ditto Lu et al. (2024a) , node_lv3_5], [Contrastive: Self-Reward Yuan et al. (2024), SPIN Chen et al. (2024), GRATH Chen and Li (2024), Self-Contrast Zhang et al. (2024c), ETO Song et al. (2024), A³T Yang et al. (2024c), STE Wang et al. (2024b), COTERRORSET Tong et al. (2024) , node_lv3_5], ] [Positive, node_lv2, [Grounded: Self-Align Sun et al. (2024), SALMON Sun et al. (2023), MemoryBank Zhong et al. (2024b), TiM Liu et al. (2023a), MoT Li and Qiu (2023), IML Wang et al. (2024a), TRAN Yang et al. (2023b), MemGPT Packer et al. (2023) , node_lv3_5 ], [Self-Play: Debates Taubenfeld et al. (2024), Self-Talk Ulmer et al. (2024), Ditto Lu et al. (2024a), SOLID Askari et al. (2024), SOTOPIA-$\pi$ Wang et al. (2024e) , node_lv3_5], [Interactive: SelfEvolve Jiang et al. (2023), LDB Zhong et al. (2024a), ETO Song et al. (2024), A${}^3$T Yang et al. (2024c), AutoAct Qiao et al. (2024), KnowAgent Zhu et al. (2024) , node_lv3_5], [Rationale-Based: LMSI Huang et al. (2022), STaR Zelikman et al. (2022), A${}^3$T Yang et al. (2024c) , node_lv3_5], ] ] [Task (§4.1), node_lv1_5 [Selective, node_lv2, [ DIVERSE-EVOL Wu et al. (2023), SOFT Wang et al. (2024c), Selective Reflection-Tuning Li et al. (2024b) V-STaR Hosseini et al. (2024) , node_lv3_5], ] [Knowledge-Free, node_lv2, [ Self-Instruct Wang et al. (2023b); Honovich et al. (2022); Roziere et al. (2023), Ada-Instruct Cui and Wang (2023), Evol-Instruct Xu et al. (2024a), MetaMath Yu et al. (2023b), PromptBreeder Fernando et al. (2023), Backtranslation Li et al. (2023b), Kun Zheng et al. (2024b) , node_lv3_5], ] [Knowledge-Based, node_lv2, [Unstructured: UltraChat Ding et al. (2023), SciGLM Zhang et al. (2024b), EvIT Tao et al. (2024a), MEEL Tao et al. (2024b) , node_lv3_5], [Structured: Self-Align Sun et al. (2024), Ditto Lu et al. (2024a), SOLID Askari et al. (2024) , node_lv3_5], ] ] ] ]

Figure 3: Taxonomy of self-evolving large language models.

3 Evolution Objectives

Evolution objectives in self-evolving LLMs serve as predefined goals that autonomously guide their development and refinement. Much like humans set personal objectives based on needs and desires, these objectives are crucial as they determine how the model iteratively self-updates. They enable the LLM to autonomously learn from new data, optimize algorithms, and adapt to changing environments, effectively "feeling" its needs from feedback or self-assessment and setting its own goals to enhance functionality without human intervention.

We define an evolution objective as combining an evolving ability and an evolution direction. An evolving ability stands for an innate and detailed skill. The evolution direction is the aspect of the evolution objective aiming to improve. We formulate the evolution objective as follows:

$${ℰ}^t=({𝒜}^t,{𝒟}^t),$$ (1)

where ${ℰ}^t$ is the evolution objective, composed by evolving abilities ${𝒜}^t$ and evolution directions ${𝒟}^t$. Take "reasoning accuracy improving" as an example, "reasoning" is the evolving ability and "accuracy improving" is the evolution direction.

3.1 Evolving Abilities

In Table 1, we summarize and categorize the targeted evolving abilities in current self-evolution research into two groups: LLMs and LLM Agents.

3.1.1 LLMs

These are fundamental abilities underlying a broad spectrum of downstream tasks.

Instruction Following: The capability to follow instructions is essential for effectively applying language models. It allows these models to address specific user needs across different tasks and domains, aligning their responses within the given context Xu et al. (2023a).

Reasoning: LLMs can self-evolve to recognize statistical patterns, making logical connections and deductions based on the information. They evolve to perform better reasoning involving methodically dissecting problems in a logical sequence Cui and Wang (2023).

Math: LLMs enhance the intricate ability to solve mathematical problems covering arithmetic, math word, geometry, and automated theorem proving Ahn et al. (2024) towards self-evolution.

Coding: Methods improve the LLM coding abilities to generate more precise and robust programs Singh et al. (2023); Zelikman et al. (2023). Furthermore, EvoCodeBench Li et al. (2024a) provides an evolving benchmark that updates periodically to prevent data leakage.

Role-Play: It involves an agent understanding and acting out a particular role within a given context. This is crucial in scenarios where the model must fit into a social structure or follow a set of behaviors associated with a specific identity or function Lu et al. (2024a).

Others: Apart from the above fundamental evolution objectives, self-evolution can also achieve and a wide range of NLP tasks Stammer et al. (2023); Koa et al. (2024); Gulcehre et al. (2023); Zhang et al. (2024b, c).

3.1.2 LLM-based Agents

The abilities discussed here are characteristic of advanced artificial agents used for task-solving or simulations in digital or physical world. These capabilities mirror human cognitive functions, allowing these agents to perform complex tasks and interact effectively in dynamic environments.

Planning: It involves the ability to strategize and prepare for future actions or goals. An agent with this skill can analyze the current state, predict the outcomes of potential actions, and create a sequence of steps to achieve a specific objective Qiao et al. (2024).

Tool Use: This is the capacity to employ objects or instruments in the environment to perform tasks, manipulate surroundings, or solve problems Zhu et al. (2024).

Embodied Control: It refers to an agent’s ability to manage and coordinate its physical form within an environment. This encompasses locomotion, dexterity, and the manipulation of objects Bousmalis et al. (2023).

Communication: It is the skill to convey information and understand messages from other agents or humans. Agents with advanced communication abilities can participate in dialogue, collaborate with others, and adjust their behaviour based on the communication received Ulmer et al. (2024).

3.2 Evolution Directions

Examples of evolution directions include but are not limited to:

Improving Performance: The goal is to continuously enhance the model’s understanding and generation across various languages and abilities. For instance, a model initially trained for question answering and chitchat can autonomously extend its proficiency and develop abilities like diagnostic dialogue Tu et al. (2024), social skills Wang et al. (2024e), and role-playing Lu et al. (2024a).

Adaptation to Feedback: This involves improving model responses based on feedback to better align with preferences or adapt to environments Yang et al. (2023a); Sun et al. (2024).

Expansion of Knowledge Base: The aim is to continuously update the model’s knowledge base with the latest information and trends. For example, a model might automatically integrate new scientific research into its responses Wu et al. (2024).

Safety, Ethic and Reducing Bias: The goal is to identify and mitigate biases in the model’s responses, ensuring fairness and safety. One effective strategy is to incorporate guidelines, such as constitutions or specific rules, to identify inappropriate or biased responses and corrected through model updates Bai et al. (2022); Lu et al. (2024b).

Method	Acquisition			Refinement $f^{ℛ}$	Updating $f^{𝒰}$	Objective $ℰ$
	Task $f^{𝒯}$	Solution $f^{𝒴}$	Feedback $f^{ℱ}$
Large Language Models
Self-Align Sun et al. (2024)	Context-Based	Pos-G	-	Filtering	In-W	IF
SciGLM Zhang et al. (2024b)	Context-Based	-	-	-	In-W	Other
EvIT Tao et al. (2024a)	Context-Based	-	-	-	In-W	Reasoning
MEEL Tao et al. (2024b)	Context-Based	-	-	-	In-W	Reasoning
UltraChat Ding et al. (2023)	Context-Based	-	-	-	In-W	Role-Play
SOLID Askari et al. (2024)	Context-Based	Pos-S	-	Filtering	In-W	Role-Play
Ditto Lu et al. (2024a)	Context-Based	Pos-S, Neg-P	-	-	In-W	Role-Play
MetaMath Yu et al. (2023b)	Context-Free	Pos-R	-	-	In-W	Math
Self-Rewarding Yuan et al. (2024)	Context-Free	-	Model	-	In-W	IF,Reasoning,Role-Play
Kun Zheng et al. (2024b)	Context-Free	-	-	Filtering	In-W	IF,Reasoning
PromptBreeder Fernando et al. (2023)	Context-Free	-	-	-	In-C	Math, Reasoning
Ada-Instruct Cui and Wang (2023)	Context-Free	-	-	-	In-W	Math, Reasoning, Code
Backtranslation Li et al. (2023b)	Context-Free	-	-	-	In-W	IF
DiverseEvol Wu et al. (2023)	Selective	Pos-I	-	-	In-W	Code
Grath Chen and Li (2024)	Selective	Neg-C	Model	-	In-W	Reasoning
REST${}^{em}$ Singh et al. (2023)	Selective	-	Model	Filtering	In-W	Math, Code
SOFT Wang et al. (2024c)	Selective	-	-	-	In-W	IF
LSX Stammer et al. (2023)	-	Pos-R	Model	Correcting	In-W	Other
LMSI Huang et al. (2022)	-	Pos-R	-	Filtering	In-W	Math
TRAN Yang et al. (2023b)	-	Pos-G	-	-	In-C	Reasoning
MOT Li and Qiu (2023)	-	Pos-R, Pos-G	-	Filtering	In-C	Math, Reasoning
STaR Zelikman et al. (2022)	-	Pos-R, Neg-C	Model	Correct	In-W	Reasoning
COTERRORSET Tong et al. (2024)	-	Pos-R, Neg-C	-	-	In-W	Math, Reasoning
Self-Debugging Chen et al. (2023c)	-	Pos-I	Env	-	In-C	Code
SelfEvolve Jiang et al. (2023)	-	Pos-I	-	-	In-C	Code
Reflexion Shinn et al. (2024)	-	Pos-I, Pos-G	-	-	In-C	Code, Reasoning
V-STaR Hosseini et al. (2024)	-	Neg-C	Model	Filter	In-W	Math, Code
Self-Contrast Zhang et al. (2024e)	-	Neg-C	Model	-	In-W	Reasoning
SALMON Sun et al. (2023)	-	Neg-C	Model	-	In-W	IF,Reasoning,Role-Play
SPIN Chen et al. (2024)	-	Neg-C	-	-	In-W	IF,Reasoning,Role-Play
RLCD Yang et al. (2023a)	-	Neg-P	Model	-	In-W	IF
DLMA Liu et al. (2024a)	-	Neg-P	Model	-	In-W	IF
SELF Lu et al. (2023)	-	-	Model	Correct	In-W	IF, Math
LLM Agents
AutoAct Qiao et al. (2024)	Context-Based	Pos-I	Env	Filtering	In-W	Planning, Tool
KnowAgent Zhu et al. (2024)	Context-Based	Pos-I, Pos-G	Env	Filtering	In-W	Embodied, Planning, Tool
RoboCat Bousmalis et al. (2023)	Context-Free	Pos-I	Env	-	In-W	Embodied
STE Wang et al. (2024b)	Context-Free	Pos-I, Neg-C	Env	Correct	In-W	Tool
IML Wang et al. (2024a)	-	Pos-R, Pos-G	-	-	In-C	Reasoning
SinViG Xu et al. (2024b)	-	Pos-I	Env	Filtering	In-W	Embodied
ETO Song et al. (2024)	-	Pos-I, Neg-C	Env	Correct	In-W	Tool
A3T Yang et al. (2024c)	-	Pos-I, Neg-C	Env	Correct	In-W	Tool
Debates Taubenfeld et al. (2024)	-	Pos-S	-	-	In-W	Communication
SOTOPIA-$\pi$ Wang et al. (2024e)	-	Pos-S,Pos-G	Env	-	In-W	Communication
Self-Talk Ulmer et al. (2024)	-	Pos-S, Pos-G	Model	Filtering	In-W	Communication
MemGPT Packer et al. (2023)	-	Pos-G	Env	Filtering	In-C	Communication
MemoryBank Zhong et al. (2024b)	-	Pos-G	Env	Filtering	In-C	Communication
ProAgent Zhang et al. (2024a)	-	Pos-G	Env	-	In-C	Embodied
Agent-Pro Zhang et al. (2024d)	-	Pos-G	Env	-	In-C	Planning
AesopAgent Wang et al. (2024d)	-	Pos-G	Env	-	In-C	Planning
ICE Qian et al. (2024)	-	Pos-G	Env	-	In-C	Planning
TiM Liu et al. (2023a)	-	Pos-G	-	-	In-C	Communication
Werewolf Xu et al. (2023b)	-	Pos-G	-	-	In-C	Planning

Table 1: Overview of self-evolution methods, detailing approaches across evolutionary stages. Key: Pos (Positive), Neg (Negative), R (Rationale-based), I (Interactive), S (Self-play), G (Grounded), C (Contrastive), P (Perturbative), Env (Environment), In-W (In-Weight), In-C (In-Context), IF (Instruction-Following). For the evolution objectives, Adaptation to Feedback is in green, Expansion of Knowledge Base is in blue, and Safety, Ethic and Reducing Bias is in brown. Improving Performance is in the default color, black.

4 Experience Acquisition

Exploration and exploitation Gupta et al. (2006) are fundamental strategies for learning in humans and LLMs. Among that, exploration involves seeking new experiences to achieve objectives and is analogous to the initial phase of LLM self-evolution, known as experience acquisition. This process is crucial for self-evolution, enabling the model to autonomously tackle core challenges such as adapting to new tasks, overcoming knowledge limitations, and enhancing solution effectiveness. Furthermore, experience is a holistic construct, encompassing not only the tasks encountered Dewey (1938) but also the solutions developed to address these tasks Schön (2017), and the feedback Boud et al. (2013) received as a result of task performance.

Inspired by that, we divide experience acquisition into three parts: task evolution, solution evolution, and obtaining feedback. In task evolution, LLMs curate and evolve new tasks aligning with evolution objectives. For solution evolution, LLMs develop and implement strategies to complete these tasks. Finally, LLMs may optionally collect feedback from interacting with the environment for further improvements.

4.1 Task Evolution

Figure 4: Task evolution. ${ℰ}^t$ and ${𝒯}^t$ are the evolving objective and task of $t^{th}$ iteration. $𝕋$ is the set of all tasks to be selected. The first two are generative methods that differ based on their respective use of knowledge. The third method, in contrast, employs a discriminative approach to select what to learn.

To gain new experience, the model first evolves new tasks according to the evolution objective ${ℰ}^t$ in the current iteration. Task evolution is the crucial step in the engine that starts the entire evolution process. Formally, we denote the task evolution as:

$${𝒯}^t=f^{𝒯}({ℰ}^t,{\mathrm{M}}^t),$$ (2)

where $f^{𝒯}$ is the task evolution function. ${ℰ}^t$, $M^t$, and ${𝒯}^t$ denote the evolution objective, the model, and the evolved task at iteration $t$, respectively. We summarize and categorize existing studies on the task evolution method $f^{𝒯}$ into three groups: Knowledge-Based, Knowledge-Free, and Selective. We detail each type in the following parts and show the concepts in Figure 4.

Knowledge-Based

The objective ${ℰ}^t$ may associate with external knowledge to evolve where the knowledge is not inherently comprised in the current LLMs. Explicitly sourcing from knowledge enriches the relevance between tasks and evolution objectives. It also ensures the validity of relevant facts in the tasks. We delve into the Knowledge-Based methods seeking to evolve new tasks of the evolving objective assisted by external information.

The first kind of knowledge is structured. Structured knowledge is dense in information and well-organized. Self-Align Sun et al. (2024) curates topic-guided tasks generated covering 20 scientific topics such as scientific and legal expertise. Apart from topic knowledge, DITTO Lu et al. (2024a) includes character knowledge from Wikidata and Wikipedia. The knowledge comprises attributes, profiles, and concise character details for role-play conversations. SOLID Askari et al. (2024) generates structured entity knowledge as conversation starters.

The second group consists of tasks evolving from an unstructured context. Unstructured context is easy to obtain but is sparse in knowledge. UltraChat Ding et al. (2023) gathers unstructured knowledge of 20 types of text materials based on 30 meta-concepts to construct conversation tasks. SciGLM Zhang et al. (2024b) derives questions from the text of diversified science subjects, which covers rich scientific knowledge. EvIT Tao et al. (2024a) derives event reasoning tasks based on large-scale unstructured events mined from the unsupervised corpus. Similarly, MEEL Tao et al. (2024b) evolves multi-modal events in both image and text to construct the tasks for MM event reasoning.

Knowledge-Free

Unlike previous methods that require extensive human effort to gather external knowledge, Knowledge-Free approaches operate independently using the evolving object ${ℰ}^t$ and the model itself. These efficient methods can generate more diversified tasks without additional knowledge restrictions.

First, the LLMs can prompt themselves to generate new tasks according to ${ℰ}^t$. Self-Instruct Wang et al. (2023b); Honovich et al. (2022); Roziere et al. (2023) is a typical methodology of Knowledge-Free task evolution. These methods self-generate a variety of new task instructions based on evolution objectives. Ada-Instruct Cui and Wang (2023) further proposes an adaptive task instruction generation strategy that fine-tunes open-source LLMs to generate lengthy and complex task instructions for code completion and mathematical reasoning.

Second, extending and boosting original tasks increases the quality of instructions. WizardLM Xu et al. (2023a) proposes Evol-Instruct that evolves instruction following tasks with in-depth and in-breadth evolving and further expands it in code generation Luo et al. (2024). MetaMath Yu et al. (2023b) rewrites the question in multiple ways, including rephrasing, self-verification, and FOBAR. It evolves a new MetaMathQA dataset for fine-tuning LLMs to improve mathematical task-solving. Promptbreeder Fernando et al. (2023) evolves seed tasks via mutation prompts. It further evolves mutation prompts via the hyper mutation prompts to increase the task diversity.

Third, deriving tasks from plain text is another way. Backtranslation Li et al. (2023b) extracts self-contained segments in unlabelled data and regards it as the answers to tasks. Similarly, Kun Zheng et al. (2024b) presents a task self-evolving algorithm utilizing instruction harnessed from unlabelled data towards back-translation.

Selective

Instead of task generation, we may start with large-scale existing tasks. At each iteration, LLMs can select tasks that exhibit the highest relevance to the current evolving objective ${ℰ}^t$ without additional generation. This approach obviates the intricate curation of new tasks, streamlining the evolution process Zhou et al. (2024); Li et al. (2023a); Chen et al. (2023a).

A simple task selecting method is to randomly sample tasks from the task pool like REST Gulcehre et al. (2023), REST${}^{em}$ Singh et al. (2023), and GRATH Chen and Li (2024) do. Rather than random selection, DIVERSE-EVOL Wu et al. (2023) introduces a data sampling technique where the model selects new data points based on their distinctiveness in the embedding space, ensuring diversity enhancement in the chosen subset. SOFT Wang et al. (2024c) then splits the initial training set. Each iteration selects one chunk of the split set as the evolving task.

Li et al. (2024b) propose Selective Reflection-Tuning and select a subset of tasks via a novel metric calculating to what extent the answer is related to the question. V-STaR Hosseini et al. (2024) selects the correct solutions in the previous iteration and adds their task instructions to the task set of the next iteration.

4.2 Solution Evolution

Figure 5: Solution evolution. ${ℰ}^t$, ${𝒯}^t$, and ${𝒴}^t$ are the evolving objective, task, and solution of $t^{th}$ iteration. $ℛ$ is the rational thought.

After obtaining evolved tasks, LLMs solve the tasks to acquire the corresponding solution. The most common strategy is to generate the solution directly according to the task formulation Zelikman et al. (2022); Gulcehre et al. (2023); Singh et al. (2023); Zheng et al. (2024b); Yuan et al. (2024). However, this straightforward approach might reach solutions irrelevant to the evolution objective, leading to suboptimal evolution Hare (2019). Therefore, solution evolution uses different strategies to solve tasks and enhance LLM capabilities by ensuring that solutions are not just generated but are also relevant and informative. In this section, we comprehensively survey these strategies and illustrate them in Figure 5. We first formulate the solution-evolution as follows:

$${𝒴}^t=f^{𝒴}({𝒯}^t,{ℰ}^t,{\mathrm{M}}^t),$$ (3)

where $f^{𝒴}$ is the model’s strategy to approach the evolution objective.

We then categorize these methods into positive and negative according to the correctness of the solutions. The positive methods introduce various approaches to acquire correct and desirable solutions. On the contrary, negative methods elicit and collect undesired solutions, including unfaithful or mis-align model behaviors, which are then used for preference alignment. We elaborate on the details of each type in the following sections.

4.2.1 Positive

Current studies explore diverse methods beyond vanilla inference for positive solutions to obtain correct solutions aligned with evolution objectives. We categorize the task-solving process into four types: Rationale-Based, Interactive, Self-Play, and Grounded.

Rationale-Based

The model incorporates rationale explanations towards approaching the evolving objective when solving the tasks and can self-evolve by utilizing such rationales. These methods enable models to explicitly acknowledge the evolution objective and complete this task in that direction Wei et al. (2022); Yao et al. (2024); Besta et al. (2024); Yao et al. (2022).

Huang et al. (2022) proposes a method where an LLM self-evolves using "high-confidence" rationale-augmented answers generated for unlabeled questions. Similarly, STaR Zelikman et al. (2022) generates rationale when solving the task. If the answer is wrong, it further corrects the rationale and the answer. Then, it uses the answer and rationale as experiences to fine-tune the model. Similarly, LSX Stammer et al. (2023) proposes the novel paradigm to generate an explanation of the answer, incorporating an iterative loop between a learner module performing a base task and a critic module that assesses the quality of explanations given by the learner. Song et al. (2024); Yang et al. (2024c) obtain rationales in the ReAct Yao et al. (2022) style when solving the tasks. The rationales are further engaged in training the agents in the following step.

Interactive

Models can interact with the environment to enhance the evolution process. These methods can obtain environmental feedback that is valuable for guiding self-evolution directions.

SelfEvolve and LDB Jiang et al. (2023); Zhong et al. (2024a) improve code generation ability via self-evolution. They allow the model to generate code and acquire feedback via running the code on the interpreter. As another environment, Song et al. (2024); Yang et al. (2024c) interact in embodied scenarios and acquire feedback. They learn to take proper actions based on their current state. For agent abilities, AutoAct Qiao et al. (2024) introduces self-planning from scratch, focusing on an intrinsic self-learning process. In this process, agents enhance their abilities through recursive planning iterations with environment feedback. Following AutoAct, Zhu et al. (2024) further enhances agent training by integrating self-evolution and an external action knowledge base. This approach guides action generation and boosts planning ability through environment-driven corrective feedback loops.

Self-Play

It’s the situation where a model learns to evolve by playing against copies of itself. Self-play is a powerful evolving method because it enables systems to communicate with themselves to get feedback in a closed loop. It’s especially effective in environments where the model can simulate various sides of the roles, like multi-player games Silver et al. (2016, 2017). Compared with interactive methods, self-play is an effective strategy to obtain feedback without an environment.

Taubenfeld et al. (2024) investigate the systematic biases in simulations of debates by LLMs. On the contrary to debating, Ulmer et al. (2024) engage LLMs in conversations following generated principles. Another kind of conversation via role-playing. Lu et al. (2024a) proposes self-simulated role-play conversation. The process involves instructing the LLM with character profiles and aligning its responses to maintain consistency with the character’s knowledge and style. Similarly, Askari et al. (2024) propose SOLID to generate large-scale intent-aware role-play dialogues. This self-playing aspect harnesses the expansive knowledge of LLMs to construct information-rich exchanges that streamline the dialog generation process. Wang et al. (2024e) introduces a novel approach whereby each LLM follows a role and communicates with each other to achieve their goals.

Grounded

To reach the evolving objective and reduce exploration space, models can be grounded on existing rules Sun et al. (2024) and previous experiences for further explicit guidance when solving the tasks.

LLMs can generate desirable solutions more effectively by being grounded on pre-defined rules and principles. For instance, Self-Align Sun et al. (2024) generated self-evolved questions with principle-driven constraints to guide the task-solving process. SALMON Sun et al. (2023) design a set of combined principles that requires the model to follow when solving the task. Self-Talk Ulmer et al. (2024) ensures the LLMs generate a workflow-aligned conversation based on preset agent characters. They generate the workflow in advance based on GPT-4.

Besides pre-defined rules, grounding on previous experiences can improve the solutions. MemoryBank Zhong et al. (2024b) and TiM Liu et al. (2023a) answer current questions by incorporating previous question-answer records. Rather than previous solution histories, MoT Li and Qiu (2023), IML Wang et al. (2024a), and TRAN Yang et al. (2023b) incorporate induced rules from the histories to answer new questions. MemGPT Packer et al. (2023) combines these merits and retrieves previous questions, solutions, induced events, and user portrait knowledge.

4.2.2 Negative

In addition to acquiring positive solutions, recent research illustrates that LLMs can benefit from negative ones for self-improvement Yang et al. (2023b). This strategy is analogous to trial and error in human behavior when learning skills. This section summarises typical methods of gaining negative solutions to assist in self-evolution.

Contrastive

A widely used group of methods is to collect multiple solutions for a task and then contrast the positive and negative ones to get improvements.

Self-Reward, SPIN Yuan et al. (2024); Chen et al. (2024) updates the model by comparing the answers of high and low scores. Similarly, GRATH Chen and Li (2024) generates both correct and incorrect answers. It then trains the model by comparing these two answers. Self-Contrast Zhang et al. (2024c) contrasts the differences and summarizes these discrepancies into a checklist that could be used to re-examine and eliminate discrepancies. In ETO Song et al. (2024), the model interacts with the embodied environment to complete tasks and optimizes from the failure solutions. A³T Yang et al. (2024c) improves ETO by adding rationale after each action for solving tasks. STE Wang et al. (2024b) implements trial and error where the model solves the tasks with unfamiliar tools. It learns by analyzing failed attempts to improve problem-solving strategies in future tasks. More recently, COTERRORSET Tong et al. (2024) obtains incorrect solutions generated by PALM-2 and proposes mistake tuning, which requires the model to void making mistakes.

Perturbative

Compared to Contrastive, Perturbative methods seek to add perturbations to obtain negative solutions intentionally. Models can later learn to avoid generating these negative answers. Adding perturbations to obtain negative solutions is more controllable than contrastive methods.

Some methods add perturbation to generate harmful solutions Yang et al. (2023a); Liu et al. (2024a). Given a task, RLCD Yang et al. (2023a) curates both positive and negative instructions and generates positive and negative solutions. DLMA Liu et al. (2024a) gathers both positive and negative instructional prompts and subsequently produces corresponding positive and negative solutions.

Rather than harmful perturbation, incorporating negative context is another way. Ditto Lu et al. (2024a) adds negative persona characters to generate incorrect conversations. The model then learns from the negative conversations to evolve persona dialogue ability.

4.3 Feedback

Figure 6: Types of Feedback. ${ℰ}^t$ and ${𝒴}^t$ are the evolving objective and task solutions of $t^{th}$ iteration.

As humans learn skills, feedback plays a critical role in demonstrating the correctness of the solutions. This key information enables humans to reflect and then update their skills. Akin to this process, LLMs should obtain feedback during or after the task solution in the cycle of self-evolution. We formalize the process as follows:

$${ℱ}^t=f^{ℱ}({𝒯}^t,{𝒴}^t,{ℰ}^t,{\mathrm{M}}^t,\mathrm{ENV}),$$ (4)

where $f^{ℱ}$ is the method to acquire feedback.

In this part, we summarize two types of feedback. Model feedback refers to gathering the critique or score rated by the LLMs themselves. Besides, Environment denotes the feedback received directly from the external environment. We illustrate these concepts in Figure 6.

4.3.1 Model

Current studies demonstrate that LLMs can play well as a critic Zheng et al. (2024a). In the cycle of self-evolution, the model judges itself to acquire the feedback of the solutions.

One type of feedback is a score that indicates correctness. Self-Reward Yuan et al. (2024), LSX Stammer et al. (2023), and DLMA Liu et al. (2024a) rate their own solutions and output the scores via LLM-as-a-Judge prompting. Similar to that, SIRLC Pang et al. (2023) utilizes self-evaluation results of LLM as the reward for further reinforcement learning. Self-Alignment Zhang et al. (2024e) leverages the self-evaluation capability of an LLM to generate confidence scores on the factual accuracy of its outputs.

Another type provides a textual description, offering multi-dimensional information. To alter the distribution of the responses via supervised learning, CAI Bai et al. (2022) asks the model to critique its response according to a principle in the constitution. In contrast to supervised learning and reinforcement learning approaches, Self-Refine Madaan et al. (2023) allows the model to generate natural language feedback on its own output in a few-shot manner.

4.3.2 Environment

Another form of feedback comes from the environment, common in tasks where solutions can be directly evaluated. This feedback is precise and elaborate and can provide sufficient information for model updating. They may be derived from code interpreter Jiang et al. (2023); Chen et al. (2023c); Shinn et al. (2024), tool execution Qiao et al. (2024); Gou et al. (2023), the embodied environment Bousmalis et al. (2023); Xu et al. (2024b); Zhou et al. (2023b), and other LLMs or agents Wang et al. (2024e); Taubenfeld et al. (2024); Ulmer et al. (2024).

For code generation, Self-Debugging Chen et al. (2023c) utilizes execution results on test cases as part of feedback while SelfEvolve Jiang et al. (2023) receives the error message from the interpreter. Similarly, Reflexion Shinn et al. (2023) also obtains the run-time feedback from the code interpreter. It then further reflects to generate thoughts. This run-time feedback contains the trace-back information that can point out the key information for improved code generation.

Recently, methods endow tool-using ability to LLMs and agents. Executing tools leading to feedback in return Gou et al. (2023); Qiao et al. (2024); Song et al. (2024); Yang et al. (2024c); Wang et al. (2024b).

RoboCat Bousmalis et al. (2023) and SinViG Xu et al. (2024b) act in the robotic embodied environment. This type of feedback is precise and strong to guide self-evolution.

Communication feedback is common and effective in LLM-based multi-agent systems. Agents can correct and support each other, enabling co-evluation Wang et al. (2024e); Taubenfeld et al. (2024); Ulmer et al. (2024).

Figure 7: Experience refinement.

5 Experience Refinement

After experience acquisition and before updating in self-evolution, LLMs may improve the quality and reliability of their outputs through experience refinement. It helps LLMs adapt to new information and contexts without relying on external resources, leading to more reliable and effective assistance in dynamic environments. This process is formulated as follows:

$${\widetilde{𝒯}}^t,{\widetilde{𝒴}}^t=f^{ℛ}({𝒯}^t,{𝒴}^t,{ℱ}^t,{ℰ}^t,{\mathrm{M}}^t),$$ (5)

where $f^{ℛ}$ is the methods of experience refinement, ${\widetilde{𝒯}}^t,{\widetilde{𝒴}}^t$ are the refined tasks and solutions. We classify the methods into two categories: filtering and correcting.

5.1 Filtering

Refinement in self-evolution involves two primary filtering strategies: Metric-Based and Metric-Free. The former uses external metrics to assess and filter outputs, while the latter does not rely on these metrics. This ensures that only the most reliable and high-quality data is utilized for further updating.

5.1.1 Metric-Based

By relying on feedback and pre-defined criteria, metric-based filtering improves the quality of the outputs Singh et al. (2023); Qiao et al. (2024); Ulmer et al. (2024); Wang et al. (2023b), ensuring the progressive enhancement of LLM capabilities through each iteration of refinement.

For example, ReST${}^{EM}$ Singh et al. (2023) incorporates a reward function to filter the dataset sampled from the current policy. The function provides binary rewards based on the correctness of the generated samples rather than a learned reward model trained on human preferences in ReST Gulcehre et al. (2023). AutoAct Qiao et al. (2024) leverages F1-score and accuracy as rewards for synthetic trajectories and collects trajectories with exactly correct answers for further training. Self-Talk Ulmer et al. (2024) measures the number of completed subgoals to filter the generated dialogues, ensuring that only high-quality data is used for training. To encourage diversity of the source instructions, Self-Instruct Wang et al. (2023b) automatically filters low-quality or repeated instructions using ROUGE-L similarity and heuristics before adding them to the task pool.

The filtering criteria or metrics are crucial for maintaining the quality and reliability of the generated outputs, thereby ensuring the continuous improvement of the model’s capability.

5.1.2 Metric-Free

Some methods seek filtering strategies beyond external metrics, making the process more flexible and adaptable. Metric-Free filtering typically involves sampling outputs and evaluating them based on internal consistency measures or other model-inherent criteria Huang et al. (2022); Weng et al. (2023); Chen et al. (2022). The filtering in Self-Consistency Wang et al. (2022) is based on the consistency of the final answer across multiple generated reasoning paths, with higher agreement indicating higher reliability. LMSI Huang et al. (2022) utilizes CoT prompting plus self-consistency for generating high-confidence self-training data.

Designing internal consistency measures that accurately reflect output quality can be challenging. Self-Verification Weng et al. (2023) allows the model to select the candidate answer with the highest interpretable verification score, calculated by assessing the consistency between the predicted and original condition values. For the code generation task, CodeT Chen et al. (2022) considers both the consistency of the outputs against the generated test cases and the agreement of the outputs with other code samples.

These methods emphasize the language model’s ability to self-assess and filter its outputs based on internal agreement, showcasing a significant step forward in self-evolution without the direct intervention of external metrics.

5.2 Correcting

Recent advancements in self-evolution have highlighted the significance of iterative self-correction, which enables models to refine their experiences. This section divides the methods employed into two categories: Critique-Based and Critique-Free correction. Critiques often serve as strong hints that include the rationale behind perceived errors or suboptimal outputs, guiding the model towards improved iterations.

5.2.1 Critique-Based

These methods rely on additional judging processes to draw the critiques of the experiences. Then, the experiences are refined based on the critiques. By leveraging either self-generated Madaan et al. (2023); Bai et al. (2022); Shinn et al. (2023); Lu et al. (2023) or environment-interaction generated critiques Gou et al. (2023); Jiang et al. (2023); Zhou et al. (2023b), the model benefits from detailed feedback for nuanced correction.

LLMs have demonstrated their ability to identify errors in their outputs. Self-Refine Madaan et al. (2023) introduces an iterative process in which the model refines its initial outputs conditioned on actionable self-feedback without additional training. To evolve from the correction, CAI Bai et al. (2022) generates critiques and revisions of its outputs in the supervised learning phase, significantly improving the initial model. Applied to an agent automating computer tasks, RCI Kim et al. (2023) improves its previous outputs based on the critique finding errors in the outputs.

Since weaker models may struggle significantly with the self-critique process, several approaches enable models to correct the outputs using critiques provided by external tools. CRITIC Gou et al. (2023) allows LLMs to revise the output based on the critiques obtained during interaction with tools in general domains. SelfEvolve Jiang et al. (2023) prompts an LLM to refine the answer code based on the error information thrown by the interpreter. ISR-LLM Zhou et al. (2023b) helps the LLM planner find a revised action plan by using a validator in an iterative self-refinement process.

The primary advantage of this method lies in its ability to process and react to detailed feedback, potentially leading to more targeted and nuanced corrections.

5.2.2 Critique-Free

Contrary to critique-based, critique-free methods correct the experiences directly leveraging objective information Zelikman et al. (2022); Chen et al. (2023c, b); Gero et al. (2023). These methods offer the advantage of independence from nuanced feedback that critiques provide, allowing for corrections that adhere strictly to factual accuracy or specific guidelines without the potential bias introduced by critiques.

One group of critique-free methods modifies the experiences on the signal of whether the task was correctly resolved. Self-Taught Reasoner (STaR) Zelikman et al. (2022) proposes a technique that iteratively generates rationales to answer questions. If the answers are incorrect, the model is prompted again with the correct answer to generate a more informed rationale. Self-Debug Chen et al. (2023c) enables the model to perform debugging steps by investigating execution results from unit tests and explaining the code on its own.

Different from depending on the task-solving signal, other information produced during the solving process can be leveraged. IterRefinement Chen et al. (2023b) relies on a series of refined prompts that encourage the model to reconsider and improve upon its previous outputs without any direct critique. For information extraction tasks, Clinical SV Gero et al. (2023) grounds each element in evidence from the input and prunes inaccurate elements using supplied evidence.

These critique-free approaches simplify the correction mechanism, allowing for easier implementation and faster adjustments.

Figure 8: The illustration of Updating methods, including in-weight and in-context updating. The terms ${\widetilde{𝒯}}^t$ and ${\widetilde{𝒴}}^t$ represent refined experiences, each containing task and corresponding solutions, respectively. ${\mathrm{M}}^t$ denotes the updated model.

6 Updating

After experience refinement, we enter the crucial updating phase that leverages the refined experiences to improve model performance. We formulate updating as follows:

$${\mathrm{M}}^{t+1}=f^{𝒰}({\widetilde{𝒯}}^t,{\widetilde{𝒴}}^t,{ℰ}^t,{\mathrm{M}}^t),$$ (6)

where $f^{𝒰}$ is the updating functions. These update methods keep the model effective by adapting to new experiences and continuously improving performance in changing environments and during iterative training.

We divide these approaches into in-weight learning, which involves updates to model weights, and in-context learning, which involves updates to external or working memory.

6.1 In-Weight

Classical training paradigms in updating LLMs in weight encompass continuous pretraining Brown et al. (2020); Roziere et al. (2023), supervised fine-tuning Longpre et al. (2023), and preference alignment Ouyang et al. (2022); Touvron et al. (2023a). However, in the iterative training process of self-evolving, the core challenge lies in achieving overall improvement and preventing catastrophic forgetting, which entails refining or acquiring new capabilities while preserving original skills. Solutions to this challenge can be categorized into three main strategies: replay-based, regularization-based, and merging-based methods.

6.1.1 Replay-based

Replay-based methods reintroduce previous data to retain old knowledge. One is experience replay, which mixes the original and new training data to update LLMs Roziere et al. (2023); Yang et al. (2024c); Zheng et al. (2023); Lee et al. (2024); Wang et al. (2023a). For example, Rejection sampling Fine-Tuning (RFT) Yuan et al. (2023) and Reinforced Self-Training (ReST) Gulcehre et al. (2023); Aksitov et al. (2023) method iteratively updates large language models by mixing seed training data with filtered new outputs generated by the model itself. AMIE Tu et al. (2024) utilizes a self-play simulated learning environment for iterative improvement and mixes generated dialogues with supervised fine-tuning data through inner and outer self-play loops. SOTOPIA-$\pi$ Wang et al. (2024e) leverages behavior cloning from the expert model and self-generated social interaction trajectory to reinforce positive behaviors.

Another is generative replay, which adopts the self-generated synthesized data as knowledge to mitigate catastrophic forgetting. For instance, Self-Synthesized Rehearsal (SSR) Huang et al. (2024a) generates synthetic training instances for rehearsal, enabling the model to preserve its ability without relying on real data from previous training stages. Self-Distillation Fine-Tuning (SDFT) Yang et al. (2024b) generates a distilled dataset from the model itself to bridge the distribution gap between task datasets and the LLM’s original distribution to mitigate catastrophic forgetting.

6.1.2 Regularization-based

Regularization-based methods constrain the model’s updates to prevent significant deviations from original behaviors, exemplified by function- and weight-based regularization. Function-based regularization focuses on modifying the loss function that a model optimizes during training Zhong et al. (2023); Peng et al. (2023). For example, InstuctGPT Ouyang et al. (2022) employs a per-token KL-divergence penalty from the output probabilities of the initial policy model ${\pi }_{\text{SFT}}$ on the updated policy model ${\pi }_{\text{RL}}$. FuseLLM Wan et al. (2024) employs a technique akin to knowledge distillation Hinton et al. (2015), leveraging the generated probability distributions from source LLMs to transfer the collective knowledge to the target LLM.

Weight-based regularization Kirkpatrick et al. (2017) directly targets the model’s weights during training. Techniques such as Elastic Reset Noukhovitch et al. (2024) counters alignment drift in RLHF by periodically resetting the online model to an exponentially moving average of its previous states. Furthermore, Ramé et al. (2024) introduced WARM, which combines multiple reward models through weight averaging to address reward hacking and misalignment. Moreover, AMA Lin et al. (2024) adaptively average model weights to optimize the trade-off between reward maximization and forgetting mitigation.

6.1.3 Architecture-based

Architecture-based methods explicitly utilize extra parameters or models for updating, including decomposition- and merging-based approaches. Decomposition-based methods separate large neural network parameters into general and task-specific components and only update the task-specific parameters to mitigate forgetting. LoRA Hu et al. (2021); Dettmers et al. (2024) inject trainable low-rank matrices to significantly reduce the number of trainable parameters while maintaining or improving model performance across various tasks. This paradigm is later adopted by GPT4tools Yang et al. (2024a), OpenAGI Ge et al. (2024) and Dromedary Sun et al. (2024). Dynamic ConPET Song et al. (2023) combines pre-selection and prediction with task-specific LoRA modules to prevent forgetting, ensuring scalable and effective adaptation of LLMs to new tasks.

Merging-based methods, on the other hand, involve combining multiple models or layers to achieve general improvements, including but not limited to merging multiple generic and specialized model weights into a single model Wortsman et al. (2022); Ilharco et al. (2022); Yu et al. (2023a); Yadav et al. (2024), through mixture-of-expert approach Ding et al. (2024) or even layer-wise merging and up-scaling such as EvoLLM Akiba et al. (2024).

6.2 In-Context

In addition to directly updating model parameters, another approach is to leverage the in-context capabilities of LLMs to learn from experiences, thereby enabling fast adaptive updates without expensive training costs. The methods could be divided into updating external and working memory.

Method	Content	Operation
MoT Li and Qiu (2023)	Experience	Insert
TRAN Yang et al. (2023b)	Rationale	Insert, Reflect
MemoryBank Zhong et al. (2024b)	Experience, Rationale	Insert, Reflect, Forget
MemGPT Packer et al. (2023)	Experience	Insert, Forget
TiM Liu et al. (2023a)	Rationale	Insert
IML Wang et al. (2024a)	Rationale	Insert, Reflect
ICE Qian et al. (2024)	Rationale	Insert, Reflect
AesopAgent Wang et al. (2024d)	Experience, Rationale	Insert, Reflect

Table 2: Content and operations of updating external memory.

External Memory

This approach utilizes an external module to collect, update, and retrieve past experiences and knowledge, enabling models to access a rich pool of insights and achieve better results without updating model parameters. The external memory mechanism is common in AI Agent systems Xu et al. (2023b); Qian et al. (2024); Wang et al. (2024d). This section provides a detailed overview of the latest methods for updating external memory, emphasizing the aspects of memory Content and Updating Operations, and summarized in Table 2.

Content: External memory mainly stores two type of content: past experiences and reflected rationale, each serving distinct purposes. For instance, past experience provide valuable historical context, serving as a guiding force toward achieving improved outcomes. MoT Li and Qiu (2023) archives filtered question-answer pairs to construct a beneficial memory repository. Additionally, the FIFO Queue mechanism in MemGPT Packer et al. (2023) maintains a rolling history of messages, encapsulating interactions between agents and users, system notifications, and inputs and outputs of function calls.

On the other hand, reflected rationales offer condensed explanations, such as rules, that support decision-making. For instance, TRAN Yang et al. (2023b) archives rules inferred from experiences alongside information on mistakes to mitigate future errors. Correspondingly, TiM Liu et al. (2023a) preserves inductive reasoning, defined as text elucidating the relationships between entities. Moreover, IML Wang et al. (2024a) and ICE Qian et al. (2024) store comprehensive notes and rules derived from a series of trajectories, demonstrating the broad spectrum of content types that memory systems can accommodate.

MemoryBank Zhong et al. (2024b) and AesopAgent Wang et al. (2024d) establish both experience and reflection knowledge stores, which are the integration of both two memories.

Updating Operation: We categorize the operations to the memory into Insert, Reflect, and Forget. The most common operation is insert, methods insert text content into the memory for storage Li and Qiu (2023); Yang et al. (2023b); Zhong et al. (2024b); Packer et al. (2023); Liu et al. (2023a); Wang et al. (2024a). Another operation is reflection, which is to think and summarize previous experiences to conceptualize rules and knowledge for future use Yang et al. (2023b); Zhong et al. (2024b); Wang et al. (2024a); Qian et al. (2024). Last, due to the limited storage of memory, forgetting content is crucial to keeping memory efficient and the content valid. MemGPT Packer et al. (2023) adopts the FIFO queue to forget the contents. MemoryBank Zhong et al. (2024b) establishes a forgetting curve on the insert time of each item.

Working Memory

The methods use past experience to evolve the capabilities of agents by updating internal memory streams, states, or beliefs, known as working memory, often in the form of verbal cues. Reflexion Shinn et al. (2023) introduces verbal reinforcement learning for decision-making improvement without conventional model updates. Similarly, IML Wang et al. (2024a) enables LLM-based agents to autonomously learn and adapt to their environment by summarizing, refining, and updating knowledge based on past experience directly in working memory.

EvolutionaryAgent Li et al. (2024c) aligns agents with dynamically changing social norms through evolution and selection principles, leveraging environmental feedback for self-evolution. Agent-Pro Zhang et al. (2024d) employs policy-level reflection and optimization, allowing agents to adapt their behavior and beliefs in interactive scenarios based on past outcomes. Lastly, ProAgent Zhang et al. (2024a) enhances cooperation in multi-agent systems by dynamically interpreting teammates’ intentions and adapting behavior.

These collective works demonstrate the importance of integrating past experiences and knowledge into the agents’ memory stream to refine their state or beliefs for improved performance and adaptability across various tasks and environments.

7 Evaluation

Much like the human learning process, it is essential to ascertain whether the present level of ability is adequate and meets the application requirements through evaluation. Furthermore, it is from these evaluations that one can identify the direction for future learning. However, how to accurately assess the performance of an evolved model and provide directions for future improvements is a crucial yet underexplored research area. For a given evolved model $M^t$, we conceptualize the evaluation process as follows:

$${ℰ}^{t+1},{𝒮}^{t+1}=f^{ℰ}(M^t,{ℰ}^t,\mathrm{ENV}),$$ (7)

where $f^E$ represents the evaluation function that measures the performance score (${𝒮}^{t+1}$) of the current model and provide evolving goal (${ℰ}^{t+1}$) for the next iteration. Evaluation function $f^{ℰ}$ can be categorized into quantitative and qualitative approaches, each providing valuable insights into model performance and areas for improvement.

7.1 Quantitative Evaluation

This method focuses on providing measurable metrics to reliably assess LLM performance, such as automatic Papineni et al. (2002); Lin (2004) and human evaluation. However, traditional automatic metrics struggle to accurately evaluate increasingly complex tasks, and human assessment is not an ideal option for autonomous self-evolution. Recent trends use LLMs as human proxy for automatic evaluators, offering cost-effective and scalable solutions for evaluations.

For example, reward model score has been widely used to measure model or task performances Shinn et al. (2024) and select the best checkpoint Ouyang et al. (2022). LLM-as-a-judge Zheng et al. (2024a) using LLMs to evaluate LLMs, employing methods like pairwise comparison, single answer grading, and reference-guided grading. It shows that LLMs can closely match human judgment, enabling efficient large-scale evaluations.

7.2 Qualitative Evaluation

Qualitative evaluation involves case studies and analysis to derive insights, offering evolving guidance for subsequent iterations. Initiatives such as LLM-as-a-judge Zheng et al. (2024a) provide the reasoning behind its assessments; ChatEval Chan et al. (2023) explores the strengths and weaknesses of model outputs through debate mechanisms. Furthermore, TRAN Yang et al. (2023b) leverages past errors to formulate rules that enhance future LLM performances. Nonetheless, compared with instance-level critic or reflection, qualitative evaluation at the task- or model-level still needs comprehensive investigation.

8 Open Problems

8.1 Objectives: Diversity and Hierarchy

Section 3 summarizes existing evolution objectives and their coverage. Nonetheless, these highlighted objectives can only satisfy a small fraction of the vast human needs. The extensive application of LLMs across various tasks and industries highlights unresolved challenges in establishing self-evolution frameworks for evolving objectives that can comprehensively address a broader spectrum of real-world tasks Eloundou et al. (2023).

Furthermore, the concept of evolving objectives entails a potential hierarchical structure; for instance, UltraTool Huang et al. (2024b) and T-Eval Chen et al. (2023d) categorize the capability of tool usage into various sub-dimensions. Exploring evolutionary objectives into manageable sub-goals and pursuing them individually emerges as a viable strategy.

Overall, a clear and urgent need exists to develop self-evolution frameworks that effectively address diversified and hierarchical objectives.

8.2 Level of Autonomy: From Low to High

Self-evolution in large models is emerging, yet lacks clear definitions for its autonomous levels. We categorize self-evolution into three tiers: low, medium, and high-level autonomy

Low-level

In this level, the user predefined the evolving object $ℰ$ and it remains unchanged. The user needs to design the evolving pipeline, namely all modules $f^{\bullet }$, on its own. Then, the model completes the self-evolution process based on the designed framework. We denote this level of self-evolution in the following formula:

$$\tilde{\mathrm{M}}={\mathrm{Evol}}^{\mathrm{L}}(\mathrm{M},ℰ,f^{\bullet },\mathrm{ENV}),$$ (8)

where $\mathrm{M}$ denotes the model to be evolved. $\tilde{\mathrm{M}}$ is the evolving output. $\mathrm{ENV}$ is the environment. Most of the current works lie at this level.

Medium-level

In this level, the user only sets the evolving object $ℰ$ and keeps it unchanged. The user doesn’t need to design the specific modules $f^{\bullet }$ in the framework. The model can construct each module $f^{\bullet }$ independently for self-evolution. This level denotes as follows:

$$\tilde{\mathrm{M}}={\mathrm{Evol}}^{\mathrm{M}}(\mathrm{M},ℰ,\mathrm{ENV}),$$ (9)

High-level

In the final level, the model diagnoses its deficiency and constructs the self-evolution methods to improve itself. This is the ultimate purpose of self-evolution. The user model sets its own evolving object $ℰ$ according to the evaluation $f^E$ output. The evolving objective would change during the iteration. Besides, the model designs the specific modules $f^{\bullet }$ in the framework. We represent this level as:

$$\tilde{\mathrm{M}}={\mathrm{Evol}}^{\mathrm{H}}(\mathrm{M},\mathrm{ENV}),$$ (10)

As discussed in previous open problem (§ 8.1), there are a large of unfulfilled objectives. However, most of the existing self-evolution frameworks are at the Low-level which requires specifically designed modules Yuan et al. (2024); Lu et al. (2024a); Qiao et al. (2024). These frameworks are objective-dependent and rely on large human efforts to develop. Exhausting all objectives are not deployment-efficient which brings about the urgent need to develop medium and high levels self-evolution frameworks. At the medium level, it doesn’t require expert efforts to design specific modules. LLMs can self-evolve according to targeted objectives. Then at the high level, LLMs can investigate their current deficiencies and evolve in a targeted manner. In all, developing highly autonomous self-evolution frameworks remains an open problem.

8.3 Experience Acquisition and Refinement: From Empirical to Theoretical

Suppose we have addressed the previous two challenges that we have developed promising self-evolution frameworks, the exploration of self-evolution LLMs lacks solid theoretical grounding. This idea posits that LLMs can self-improve or correct their outputs, with or without feedback from the environment. However, the mechanisms behind it remain unclear. Studies show mixed results: Huang et al. (2023) observed self-corrective behavior in models with over 22 billion parameters, while Ganguli et al. (2023) finds LLMs struggle to self-correct reasoning errors without external feedback.

A related challenge is the use of self-generated data for learning. Critics argue this approach could reduce linguistic diversity Guo et al. (2023) and lead to "model collapse," where models fail to capture complex, long-tailed data distributions Shumailov et al. (2023). Furthermore, Alemohammad et al. (2023) reveal that generative models trained on their synthetic outputs progressively lose output quality and diversity. Fu et al. (2024) extend this by theoretically analyzing the impact of self-consuming training loops on model performance, emphasizing the importance of balancing synthetic and real data to mitigate error accumulation.

Recent studies Yang et al. (2024c); Singh et al. (2023) also show that current methods struggle to improve after more than three rounds of self-evolution. One hypothesized reason is that the self-critic of LLM has not co-evolved with the evolving objective, but more experimental and theoretical support is still needed. These findings highlight a pressing need for more theoretical exploration in self-evolving LLMs. Addressing these concerns is crucial for advancing the field and ensuring that models can effectively learn and improve over time.

8.4 Updating: Stability-Plasticity Dilemma

The stability-plasticity dilemma represents a crucial yet unresolved challenge that is essential for iterative self-evolution. This dilemma reflects the difficulty of balancing the need to retain previously learned information (stability) while adapting to new data or tasks (plasticity). Existing LLMs either overlook this issue or adopt conventional methods that may be ineffective. While training models from scratch could mitigate the problem of catastrophic forgetting, it is highly inefficient, particularly as model parameters increase exponentially and autonomous learning capabilities advance. Finding a balance between acquiring new skills and preserving existing knowledge is crucial for achieving effective and efficient self-evolution, leading to overall improvement.

8.5 Evaluation: Systematic and Evolving

To effectively assess LLMs, a dynamic, comprehensive benchmark is crucial. This becomes even more pivotal as we progress towards Artificial General Intelligence (AGI). Traditional static benchmarks risk obsolescence due to LLMs’ evolving nature and potential access to test data through interacting with environments, such as search engines, undermining their reliability. A dynamic benchmark, like Sotopia Zhou et al. (2023a), proposes a solution by creating an LLM-based environment tailored for evaluating the social intelligence of LLMs, thereby avoiding the limitations posed by static benchmarks.

8.6 Safety and Superalignment

The advancement of LLMs opens the possibility for AI systems to achieve or even surpass expert-level capabilities in both supportive and autonomous decision-making. For safety, ensuring these LLMs align with human values and preferences is crucial, particularly to mitigate inherent biases that can impact areas such as political debates, as highlighted by Taubenfeld et al. (2024). OpenAI’s initiative, Superalignment Leike and Sutskever (2023), aims to align a superintelligence by developing scalable training methods, validating models for alignment, and stress-testing the alignment process through scalable oversight Saunders et al. (2022), robustness Perez et al. (2022), automated interpretability Bills et al. (2023), and adversarial testing. Although challenges remain, Superalignment marks an initial attempt to develop a self-evolving LLM that closely aligns with human ethics and values in a scalable way.

9 Conclusion

The evolution of LLMs towards self-evolution paradigms represents a transformative shift in artificial intelligence akin to the human learning process. It is promising to overcome the limitations of current models that rely heavily on human annotation and teacher models. This survey presents a comprehensive framework for understanding and developing self-evolving LLMs, structured around iterative cycles of experience acquisition, refinement, updating, and evaluation. By detailing advancements and categorizing the evolution objectives within this framework, we offer a thorough overview of current methods and highlight the potential for LLMs to adapt, learn, and improve autonomously. We also identify existing challenges and propose directions for future research, aiming to accelerate the progress toward more dynamic, intelligent, and efficient models. This work deepens the understanding of self-evolving LLMs. It paves the way for significant advancements in AI, marking a step towards achieving superintelligent systems capable of surpassing human performance in complex real-world tasks.