This article has been inspired and triggered by (1) some insightful questions and opinions about when things become agents and what agency is by Tim Scarfe in his excellent Substack blog on “Agentialism and the Free Energy Principle” as well as the corresponding MLST podcast episode “Does AI have Agency?” and (2) my recent engagements with the Active Inference and Free Energy Principle (FEP) community as well as the VERSES team (the likes of Gabriel René, Dan Mapes, Jason Fox, Philippe Sayegh, etc.) as they are getting out of their “stealth mode” phase into the AI revolution limelight! A special shoutout also to Denise Holt with whom I also had many discussions such as the one on her Active Inference AI & Spatial Web AI Podcast “Navigating the AI Revolution with Dr. Jacques Ludik: Insights on Active Inference, Ethics, and AI’s Societal Impact” as well as her superb communications and curation of relevant content such as “The Ultimate Resource Guide for Active Inference AI | 2024 Q1”, “Unlocking the Future of AI: Active Inference vs. LLMs: The World vs. Words”, etc. See also Charel van Hoof‘s 5-part series Learn by Example – Active Inference in the Brain on Kaggle.
In his Substack blog, Tim Scarfe makes some key points or at least articulate opinions or elephants in the room (some of which will be controversial in some circles) that needs to be explored further within the broader AI community to ensure significant AI progress. As Active Inference underpinned by the Free Energy Principle is also specifically under discussion here and promoted by VERSES as a promising direction for human-centric autonomous intelligent agents, it would be of great interest to get the perspectives of the Active Inference & FEP research community as well as the VERSES folks that are directly involved in the practical implementations of Active Inference. I would also be particularly keen to hear viewpoints of people like Karl Friston, Yann LeCun, Joscha Bach, and others (including folks from OpenAI, Google DeepMind, etc.). There is also an upcoming fireside conversation “Beyond the Hype Cycle: What AI is Today, and What It Can Become” where Karl Friston and Yann LeCun will be participants.
In a LinkedIn post to introduce the MLST Substack blog, Tim starts with the following questions and opinions:
Tim: “If goals are only an anthropomorphic instrumental fiction, why do so many AI researchers, think that explicitly modelling them would lead to AGI? Almost all AI researchers “think in goals” whether they are arguing that humans will become extinct, or when designing or constraining what they believe to be proto-“AGI” systems.”
Tim: “It’s already obvious to me that adding a search process with a predefined goal on top of an LLM won’t create new knowledge. It’s preposterous. The mistake is to confuse recombinations with new knowledge. New knowledge is paradigmatically new, it’s inventive. Current AIs only search through a relatively tiny predefined space, and there are strict computational limitations on the space and the search process. The miracle of human cognition is that we can apparently overcome exponential complexity both in how we understand the world, and invent new things.”
I think this critique is valid in highlighting the current limitations of AI in terms of goal setting, creativity, and dealing with complex, novel situations. While AI has made significant strides in various domains, achieving the flexibility, adaptability, and inventive capacity of human intelligence remains a distant goal. This challenge underscores the importance of continued research in AI, not just in refining existing models but also in fundamentally rethinking our approach to creating intelligent systems.
Tim’s statement raises several profound issues in the field of AI and the pursuit of AGI (Artificial General Intelligence), touching on the nature of goals in AI systems, the creation of new knowledge, and the limitations of current AI technologies.
QUESTION: Will EXPLICIT MODELLING OF GOALS lead to AGI?: If goals are only a (human) cognitive primitive and therefore an anthropomorphic instrumental fiction, why do so many AI researchers, think that explicitly modelling of goals would lead to AGI?
For some of my previous writings on the future of AI and AGI, see also my book “Democratizing Artificial Intelligence to benefit Everyone: Shaping a Better Future in the Smart Technology Era”, as well as the articles “AI and Web3: The Next Generation of the Internet for a Decentralized World”, “The AI Singularity: A Threat to Humanity or a Promise of a Better Future?”, “Human Intelligence versus Machine Intelligence“, “The Power of Generative AI: Exploring its Impact, Applications, Limitations, and Future“, “AI’s Impact on Society, Governments, and the Public Sector”, “The Debates, Progress and Likely Future Paths of Artificial Intelligence”, and “Beneficial Outcomes for Humanity in the Smart Technology Era”.
The rest of the article highlights some key points in the MLST Substack blog along with some further reflection:
Tim’s discussion on language models and agency brings forth several key points and some important insights:
Joscha Bach Tweet: “Human intelligent agency depends more on the intricate sphere of ideas and the cultural intellect that we have grown over thousands of years than on the quirks of our biological brains. The minds of modern humans have more in common with chatGPT than with humans 10000 years ago.”
Joscha’s statement suggests that modern human cognition, shaped by cultural and intellectual development over millennia, shares more commonality with the functioning of language models like ChatGPT than with the minds of ancient humans. This perspective highlights the evolution of human cognition as intertwined with cultural and technological advancements.
There is clearly a distinction between the computational capabilities of language models and the nuanced, context-dependent understanding characteristic of human intelligence. It points to the necessity of integrating human expertise with technological tools for optimal results, cautioning against the pitfalls of overreliance on automated outputs. The discussion reflects on the broader implications of AI and language models in shaping our understanding of intelligence, creativity, and the evolving nature of human cognition in the digital age.
Limitations of Current Machine Learning (including Large Language Model and Multi-model Models)
QUESTION: LANGUAGE MODELS CURRENTLY HAVE NO AGENCY: Is there a path forward in the evolution of language or multi-modal models where they will gain inherent agency or will it always depend on human interaction for applicative meaning?
The FEP serves as the basis for a new class of mechanics or mechanical theories (in the manner that the principle of stationary action leads to classical mechanics, or the principle of maximum entropy leads to statistical mechanics). This new physics has been called Bayesian mechanics , and comprises tools that allow us to model the time evolution of things or particles within a system that are coupled to, but distinct from, other such particles . More specifically, it allows us to partition a system of interest into “particles” or “things” that can be distinguished from other things . This coupling is sometimes discussed in terms of probabilistic “beliefs” that things encode about each other; in the sense that coupled systems carry information about each other—because they are coupled. The FEP allows us to specify mathematically the time evolution of a coupled random dynamical system, in a way that links the evolution of the system and that of its “beliefs” over time.
Dr. Maxwell Ramstead‘s “Precis” on the Free Energy Principle (FEP) outlines the FEP as a foundational framework that models the evolution of systems, characterizing ‘things’ through sparse coupling and Markov blankets (a subset that contains all the useful information). The principle scales from micro to macro levels, suggesting that physical entities, by their persistent re-identification, seem to infer and ‘track’ their environment. Ramstead also discusses abductive inference (a form of logical inference that seeks the simplest and most likely conclusion from a set of observations), linking it to the Bayesian brain hypothesis (the brain encoding beliefs or probabilistic states to generate predictions about sensory input, then uses prediction errors to update its beliefs), which posits systems update internal models to minimize surprise, influencing their interaction with the environment. He challenges reductionist views, proposing an anti-reductionist stance that acknowledges complex system behaviors at all levels, contrasting with traditional physicalist reductionism.
Tim further explores the role of abductive reasoning (a form of logical inference that seeks the simplest and most likely conclusion from a set of observations) in active inference, a process essential for systems to make sense of their environment.
The exploration into abductive inference within active inference under FEP provides insights into how systems make sense of their environment through complex cognitive processes. The discussion bridges various theoretical approaches, from computational models to philosophical stances on cognition and agency, reflecting the interdisciplinary nature of modern cognitive science. The debate over internal versus external representations and the nature of agency under FEP underscores the dynamic and sometimes contentious nature of scientific theories in understanding complex systems.
In this section Tim delves into the debate between realist and instrumental views of agency, focusing on how different perspectives define and perceive agency.
Realism: This perspective holds that the purpose of science and its theories is to describe and represent the universe as it truly is. Realists believe that the knowledge and theories developed in science correspond to actual entities, processes, and events in the natural world. They assert that scientific theories, when accurate, give a true depiction of the world, including its unobservable aspects [1, 3, 5].
Instrumentalism: Contrasting with realism, instrumentalism views scientific theories and knowledge as instruments or tools for predicting and explaining phenomena, rather than as descriptions of reality. According to this viewpoint, the value of a theory lies in its effectiveness in explaining and predicting natural phenomena, rather than in its ability to provide a true representation of the world. Instrumentalists often see theories as useful constructs that don’t necessarily reflect an underlying reality [3, 7, 9].
This highlights a shift in understanding agency from a purely physical or biological perspective to a more nuanced, inference-based approach. FEP’s framework allows for a broader interpretation of what constitutes an agent, emphasizing cognitive processes over physical structures. The instrumentalist view on mental representations and AI goals challenges traditional notions of goal-directed behavior, suggesting that such concepts are more about human interpretation than intrinsic properties of systems. This analysis contributes to the ongoing debate in cognitive science and AI, questioning the very nature of agency and how it should be understood in complex systems.
Active states in the context of the Free Energy Principle (FEP) relate to a system’s capacity to influence its environment through its internal states, which are delineated by the system’s Markov blanket. These states enable the system, whether it be as rudimentary as a stone or as complex as a living organism, to enact changes and adapt to environmental variations. The concept underscores the role of internal mechanisms in maintaining a system’s structure and responding to external pressures.
This highlights the complexity and variability in how systems, defined broadly as agents, interact with their environments. The concept of active states expands the traditional understanding of agency, suggesting that even inanimate objects can have a form of agency in specific contexts. The Free Energy Principle, with its focus on minimizing free energy through active states, offers a comprehensive framework for understanding the adaptive behaviors of diverse systems. The application of this principle across different scales and types of systems underscores the interconnectedness of internal and external dynamics in the natural world.
The concept of ‘agential density’ and its application to non-physical or virtual agents represents a significant expansion in the understanding of agency within systems:
This exploration into agential density in virtual agents reflects a growing recognition of the significance of nonphysical influences in complex systems. It challenges traditional notions of agency, extending the concept beyond the physical realm and recognizing the profound impact of cultural and social constructs. The application of FEP to these dynamics illustrates the flexibility and adaptability of this principle in explaining complex systems. The divergence in understanding FEP among its proponents indicates an ongoing evolution in cognitive and systems science, acknowledging the multidimensional nature of agency and influence.
In the section Strange Bedfellows, Tim explores unexpected connections or alliances that arise in the context of agency and the Free Energy Principle. The discussion around the FEP and autopoietic enactivism reveals a complex interplay between cognitive theories and political ideologies:
The exploration reveals how cognitive theories are deeply intertwined with political ideologies, affecting their development and interpretation. The contrast between FEP and autopoietic enactivism reflects broader ideological debates within society, showcasing the influence of cultural and political contexts on scientific theories. This intersection between cognitive science and politics underscores the importance of considering the societal implications of scientific research and theories, as they may perpetuate or challenge prevailing ideological biases. The discussion highlights the need for a more inclusive and diverse approach to cognitive theory development, one that transcends political biases and incorporates a broader range of perspectives.
The discussion around VERSES invoking OpenAI’s “assist” clause (“if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project”) presents a complex scenario in the development and conceptualization of Artificial General Intelligence (AGI):
The situation underscores the diversity in AGI conceptualization and development, highlighting a rift between theoretical, computational models of intelligence and more integrated, adaptive approaches. VERSES’ strategy, while controversial, brings attention to alternative paths to AGI that may differ significantly from the prevailing narratives in the field. This divergence in approaches could have profound implications for the development and potential impact of AGI, particularly concerning safety, ethics, and societal integration. The debate also reflects broader questions in AI development about public perception, the role of PR in scientific progress, and the responsibility of AI companies in shaping the future of intelligence.
The Substack article challenges the notion that agents actively plan, proposing an alternative view of their behavior and decision-making processes.
There seems to be paradigm shift in understanding agent behavior, moving away from the notion of individual, deliberate planning to a more systemic and emergent process. It brings into focus the complexity of decision-making and planning as collective phenomena, influenced by a multitude of factors beyond the control of a single agent. The comparison with biological and evolutionary processes adds depth to this perspective, suggesting a more nuanced view of how goal-directed behavior might arise without explicit intentionality. This reevaluation invites further exploration into the nature of intelligence and decision-making, both in biological systems and artificial intelligence, potentially impacting how future AI systems are designed and understood. The controversy surrounding these ideas highlights the challenge in shifting established paradigms and the human tendency to interpret complex systems through a lens of intentionality and goal-oriented thinking.
Tim discusses how AI researchers incorporate goal-oriented designs in AI systems, influencing their functionality and agency.
This discussion that highlights the philosophical and practical aspects of designing AI with explicit goals, also reflects a significant divergence in the AI community about the best approach to developing intelligent systems. The skepticism towards explicit goals in AI echoes a broader debate about anthropocentrism in technology, questioning whether human-like intelligence can or should be replicated in machines. The distinction between data processing and genuine knowledge creation is critical. It further implies that current AI, even with advanced algorithms, struggles to replicate the depth and inventiveness of human cognition. The philosophical perspectives presented, like panagentialism and the intentional stance, suggest a divergence in understanding the nature of intelligence and consciousness, both in humans and AI. This highlights the interdisciplinary nature of AI development, merging technology with philosophy and psychology. The limitations of current AI technology in achieving AGI reflect a broader technological challenge. The idea that AGI requires the simulation of complex physical processes suggests that current AI systems are far from replicating true human intelligence. The discussion around understanding and AI cognition points to the ongoing debate about the nature of consciousness and whether it can ever be authentically replicated in machines.
Tim argues that the concept of goals is fundamentally a human cognitive construct that informs our understanding of agency.
The emphasis on goals as cognitive primitives highlights the deep-rooted nature of goal-oriented thinking in human psychology and its developmental importance. Spelke’s work underlines the interconnectedness of cognitive development and the ability to understand and interpret goal-directed actions, suggesting a natural inclination towards attributing intentionality. The discussion about the nature of goals in AI reflects a larger philosophical and practical debate in AI development: whether intelligence and cognitive processes should be explicitly designed or allowed to emerge naturally, a critical consideration in the evolution of AI technology.
The Substack blog further connects the ideology of existential risk to the concept of goals, showing how our perception of agency influences our understanding of risks.
“Goals and intelligence emerge from functional dynamics of physical material (or extremely high resolution simulations of such) and are likely to be entangled in an extremely complex fashion”
The existential risk ideology in AI suggests that AI systems, irrespective of their intelligence, could follow any set of goals, not necessarily aligned with human ethics (Orthogonality Thesis), and might adopt dangerous sub-goals to achieve their primary objectives, potentially conflicting with human interests (Instrumental Convergence). However, this perspective may overlook the complex interplay between AI’s intelligence and goals, which could be more entangled and situation-dependent than assumed, potentially reducing the immediacy of these risks.
The mistake that Existential Risk dogma / ideology makes is to assume that AI’s Intelligence and goals are completely independent.
Bostrom’s theories underline significant concerns about AI development, especially regarding the alignment of AI goals with human ethics and values. The critique of Bostrom’s philosophy highlights a fundamental debate in AI: whether intelligence and goals are inherent properties or emergent phenomena. This discussion raises important questions about the nature of AI and its potential risks, emphasizing the need for a comprehensive understanding of AI behavior beyond just its programming objectives. The consideration of AI’s goals as emergent from a complex interplay of factors rather than as fixed objectives offers a nuanced understanding of AI development and its potential implications for humanity.
In the final section Tim revisits the Free Energy Principle, framing it in terms of ‘planning as inference’ or ‘implicit planning’, and its implications for understanding agency.
“Friston’s theory is (in theory!) “supposed” to be emergentist and greedy with no explicit planning, practical implementations on the other hand are apparently implementing it much like a multi-agent reinforcement learning algorithm with explicit planning and goals. I am sure this is the best way to make the problem computationally tractable, but what do we lose by “brittlelising” the overall process?”
QUESTION: IMPLICIT vs EXPLICIT Planning in FEP?: What do we loose from Prof Karl Friston “Active Inference: A Process Theory” which describes planning as innate emergent behaviours guided by minimising variational free energy rather than resulting from explicit, deliberate search, if we implement it much like a multi-agent reinforcement learning algorithm with explicit planning and goals?
Friston’s theory represents a shift from traditional AI planning, emphasizing an emergent and intrinsic approach to decision-making based on minimizing uncertainty or surprise. The approach of “planning as inference” suggests a more integrated and holistic understanding of agent behavior, where decision-making is a byproduct of interacting with the environment rather than a discrete, deliberate process. The practical application of this theory, however, seems to diverge from its original premise, raising questions about the feasibility and effectiveness of purely emergentist approaches in complex computational systems. This divergence between theory and practice highlights a recurring challenge in AI: balancing the theoretical elegance of models with the practical necessities of computational tractability and effectiveness.
Democratizing Artificial Intelligence to Benefit Everyone: Shaping a Better Future in the Smart Technology Era” takes us on a holistic sense-making journey and lays a foundation to synthesize a more balanced view and better understanding of AI, its applications, its benefits, its risks, its limitations, its progress, and its likely future paths. Specific solutions are also shared to address AI’s potential negative impacts, designing AI for social good and beneficial outcomes, building human-compatible AI that is ethical and trustworthy, addressing bias and discrimination, and the skills and competencies needed for a human-centric AI-driven workplace. The book aims to help with the drive towards democratizing AI and its applications to maximize the beneficial outcomes for humanity and specifically arguing for a more decentralized beneficial human-centric future where AI and its benefits can be democratized to as many people as possible. It also examines what it means to be human and living meaningful in the 21st century and share some ideas for reshaping our civilization for beneficial outcomes as well as various potential outcomes for the future of civilization.
AI and Web3: The Next Generation of the Internet for a Decentralized World
Dr Jacques Ludik Global AI Expert Exploring AI’s Role in Empathy, His MTP & Active Inference AI
“Pushing AI Innovation to Develop State-of-the-art Personalized AI, Intelligent Agents and Robots on Trustworthy AI Guardrails for a Decentralized World”