Let’s Talk About Emergence

This is a linkpost for https://www.odysseaninstitute.org/post/let-s-talk-about-emergence

The field of machine learning has existed for many decades, but only recently have governments become actively concerned about the technologies leveraging its most advanced techniques. For a majority of people, this can be traced to the launch of ChatGPT, when we entered an era of so-called Large Language Models, or LLMs. One reasonable question, then, is what made ChatGPT dissimilar to its predecessors?

One distinction that has been proposed as a key differentiation between LLMs, and their smaller counterpart, Language Models, is that LLMs exhibit Emergence, or equivalently, that some of their capabilities have been categorized as Emergent[1]. It is important to note that, in this context, the root Emerge is being used as a keyword specific to the domain of machine learning, and not for its other definitions. Although the nuances of the definition differ between publications, the root Emerge is frequently reduced to some variation of the definition given by Wei et al.: “An ability is [E]mergent if it is not present in smaller models but is present in larger models.”[2] Although this meaning does result in circular reasoning when taken in conjunction with the description of LLMs proposed by domain experts in “Large Language Models: A Survey,” it is the one that has been largely accepted within machine learning circles, so we will use it as the basis for our understanding within this article[1,3,4,5]. The term’s meaning, as a keyword in the field of machine learning, is obfuscated by a number of factors, which we will explore in this article.

Emergence has been referenced in many works as a salient threat vector that could cause significant harm if ignored; a prominent paper from GovAI titled “Open Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives” highlighted Emergence as a reason to refrain from Open Source practices, and a relatively recent paper titled “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training” recognized Emergent deception as a threat vector salient enough to warrant a massive research effort and modification of current state-of-the-art techniques[3,4]. Perhaps most crucially, policymakers are beginning to cite Emergence as a motivating factor as well, as is seen in a letter from the House Committee on Science, Space, and Technology to the director of the National Institute of Standards and Technology[5,6].

Seeing as this trait played a significant role in the shift of attitude and rhetoric surrounding cutting edge machine learning systems, and as Emergence continues to be used as a critical source of danger surrounding the deployment of LLMs, let’s take a closer look at the term, what it means, and how it is being used.¹

The current concept of Emergence in the field of machine learning can be traced to the paper “Unsolved Problems in ML Safety,” which was released in September of 2021[10]. In it, Hendrycks et al. make the case that “[Machine learning systems] frequently demonstrate properties of self-organizing systems such as spontaneously [E]mergent capabilities,” citing two additional papers, “Language Models are Few Shot Learners,” and “Emerging Properties in Self-Supervised Vision Transformers”[11,12]. It is important to note that neither paper discussed their findings from this perspective; it was a conclusion reached by the team writing “Unsolved Problems in ML Safety.”² Hendrycks et al. use the unpredictability of Emergence as a significant motivator in their call for increased efforts towards ensuring that advanced machine learning systems are safe.

Beginning in early 2022 with Jacob Steinhardt’s blog post “Future ML Systems Will Be Qualitatively Different,” the concept has been presented many times; prominent definitions of Emergence, with regards to machine learning systems, are chronicled in Table 1[13].

NOTE: The table doesn’t render super well here, so check it out on the original post.

Perhaps most importantly, Bommasani et al. make the claim that in-context learning³ is an Emergent property[14]. This is based on the assertion that GPT-3, with 175 billion parameters, exhibits in-context learning, while GPT-2, with 1.5 billion parameters, does not. Lu et al. refute this claim, stating that “…in-context learning can be used in performing any task through the inclusion of a few illustrative examples within the prompt. We note that this contrasts with the notion of [E]mergent abilities, which are implied to occur due to LLMs’ capacity to perform above the random baseline on the corresponding tasks without explicit training on that task.”[18]

Schaeffer et al. also provide compelling evidence that Emergence is wholly dependent on the researcher’s choice of metrics, which is visualized in Figure 2 of their paper. In essence, when a metric that can change abruptly is used, the resulting plots indicate Emergence; contrarily, when more smooth metrics are used, the notion of Emergence vanishes.[17]

These two papers provide an important critique of the narrative surrounding certain risks that advanced machine learning systems pose, indicating that the definition of Emergence as a keyword in the field of machine learning is still being worked out, and determining the properties which can be considered Emergent is currently an active area of research.

As someone with experience in research, machine learning, and education, I would argue that we probably shouldn’t have used the term Emergence in the first place. The root emerge is already widely used in academic articles, including in papers within the domain of machine learning; when combined with the fact that the disparate definitions of the term are related⁴, it quickly becomes difficult to parse its intended meaning.⁵ Finally, the concept of emergence in the study of complex systems has been described as inherently subjective, meaning that, depending on the circumstances of analysis, different conclusions may be reached[20].⁶ In any scenario where a developing technology is going to have substantial effects on society, every effort should be made to remove potential sources of confusion or misunderstanding.

The imprecision of researchers has a meaningful effect on scientific rigor, which can be explicitly seen in this example by the circular definition that has developed between LLMs and Emergence. In turn, the understanding of these advanced machine learning technologies is undermined, making deliberation and democratic decision-making more time consuming and complicated. By using terminology that is inaccurate, unclear, and/or sensationalistic, researchers are actively making forward progress more difficult.

That being said, dismissing the concept of Emergence in machine learning, as it has been put forth, results in missing two very important elements of this story. The first, which was also noted by Steinhardt in his blog post, is a concept referred to as the phase transition[13]. Although I won’t go into too much detail here, phase transitions can be thought of as changes in system behavior which are relatively quick or sharp. There is a robust selection of literature on the study of phase transitions in machine learning, and it is still an active area of research[21,22]. Importantly, the larger increases to the inputs of machine learning systems are, the more likely it is that phase transitions will occur.

The second piece that we shouldn’t throw out with the bathwater is that we were surprised by something. Perhaps it was the impact that exponential scaling of parameter count and data would have on model performance, perhaps it was the progress that could be made without any innovation being applied to the underlying transformer architecture that powers the majority of today’s cutting edge machine learning systems, or maybe it was something else entirely. To me, all of this is indicative not of Emergent properties that couldn’t have been documented and addressed before creating the models, but of negligence from the companies pulling the strings.

Acknowledgements

I would like to thank Igor Krawczuk for review, and critique of this post, as well as discussion on the topic. I also greatly appreciate Giuseppe Dal Pra, Isabel Johnson, Chris Chan, and Bilal Ashghar for their assistance.

References