If you happen to visit this site from time to time, you probably will have noticed that I have not been posting many updates. There’s a reason for that: I have been primarily focused on writing my book about AI. It has been a few months, but writing a non-fiction book is extremely difficult. In fact, almost everyone who starts writing a serious non-fiction book later will say that they underestimated how difficult and long it would be. I figured I could handle it, because I’m a good writer. Like everyone else, I underestimated how hard it would be.
The book I am writing has a sense of urgency to it, because as AI is being adopted so widely, we are seeing unprecedented misinformation and mythology being presented in the media and taken seriously by adults, who truly should know better.
What is far worse is the AI fear movement (AI Doom) which has become one of the most ridiculously unhinged parts of the effort pushing myths and stupidity about AI. Of course, machines do not have desires, do not act on their own accord and are not capable of being gods. Superintelligence is a fictional construct that does not even make sense once you start to ask serious questions about what it means. Intelligence in and of itself is rarely a trump card when someone else has resources or knowledge you do not. LLMs are linear and have inherent limitations.
The idea that machines will gain bad intentions is complete nonsense, and yet we see increasingly unhinged and dangerous rhetoric saying the opposite. Take this Youtube video I came across recently, The braindead nature of AI doom isn’t really what we have to fear. If people truly believe that the human race is in danger, is violence even unreasonable?
Yudkowsky, a junior high dropout who has no background in almost anything, and is one of the most celebrated profits of the cult of doom has made statements about bombing data centers. A splinter group of AI doom has already committed murders. Sam Altman’s house was shot at, and a Molotov cocktail thrown at it. A mentally disturbed AI protestor went missing causing major concerns by police. None of this is surprising at all.
And then I see yet another post, this time on YouTube, celebrating a terrorist who had an unhinged fear of technology driven by literal schizophrenia.
And this is why I am writing the book. It’s an urgent plea for sanity, and an attempt to dispel myths about AI and provide all the supporting evidence so that people can truly understand how AI works and what it is and is not.
AGI (Whatever the hell that is) is a major topic of confusion and misinformation, so I have decided to present my core chapter on AGI. Bear in mind that this chapter does not really stand on its own. By the time a reader gets to this chapter (the 14th) they will have fully learned about: The history of AI, machine learning, deep learning networks, neural network dynamics, how training works, the transformer architecture, vector embeddings, attention mechanisms, catastrophic forgetting, neural compression, scaling laws and alike. If it seems like “This point assumes the audience already knows..” that is likely because it is the 14th chapter.
This is presented as is. It is a draft. It will almost surely be very refined and revised before the book is finalized. Also, keep in mind that, as an early draft, it is likely to have a few typographical or formatting errors, sub-optimal wording and similar. Normally, I might hesitate to make public such a work in progress. However, with the increasing nonsense and dangerous degeneration of understanding about what AI is and how it works, I have decided to try to get as much out as soon as possible.
Chapter 14: The Extreme Confusion and General Lunacy Around AGI
Once Again, the Very Term Artificial Intelligence Seems to Cause Confusion. Almost everything being discussed in this area is completely wrong.
The explosion of generative AI, and especially in the areas of large language models and natural language processing has renewed discussion of the topic of AGI or artificial general intelligence. The term Artificial General Intelligence has only been popular since the early 2000’s. However, the basic idea is much older: An AI system that displays the same flexibility, broad use cases and adaptability that we have come to expect from a human. One of the most important aspects that keeps coming up is the idea of a system that can operate and solve problems in a messy human world, without the need to first format the problem into machine-readable instructions. To the average person, the idea of an AI you “Can just talk to, tell it the issue, and it will work” seems highly intuitive. Natural language is how humans interact with the world and it can be used to describe almost anything.
The reality is that defining AGI and having reasonable expectations is more difficult and nuanced than it sounds. There are a few reasons why this particular topic has been one of such confusion. To those outside the field the idea of “intelligence” being human-like is intuitive, as is the idea that speech would be central to intelligence. There is also the association of intelligence with “beings” and “minds.” There’s also a strange dichotomy here: although it has been speculated that a computer could equal some aspects of human cognition, computers already vastly exceed human capabilities at things like mathematics and data lookup.
This, combined with the ever-present influence of science fiction and the philosophy of mind has resulted in a term that is so loaded, so ambiguous and so sensationalized that it’s almost impossible to discuss without invoking mythology. AGI discussions seem to lead to an endless loop of speculation on the topics of personhood, agency, intelligence, anthromorphism and what we expect technology to deliver. It’s also often presented as a binary concept, a thing which, once achieved, will be obvious. In fact, that’s almost universally how the term is discussed.
This is why the topic is so difficult. It’s also why the public, researchers and futurists keep talking past each other. The question “When will we have AGI?” must first answer the question “How do we define what AGI is?”
A History of the Concept
In 1950, Alan Turing wrote what is considered the first modern scientific paper addressing the question of machine intelligence. Although the topic had been discussed before, by 1950, programable computers were being deployed and information could now be processed in fully automated ways, as it had not been before. In his paper “Computing Machinery and Intelligence,” Turing confronted the question “Can Computers Think.”
Turing immediately saw the question as problematic. The very question presumed two things: that there was an agreed upon definition of “machine” and, perhaps more importantly, that there was an agreed upon definition of “to think.” Turing realized that any attempt to answer such a question would quickly degenerate to questions of philosophy and definitions. He also fully acknowledged that the workings of the human mind and brain were not entirely understood and conjecture still existed.
Instead, Turing stated that a scientifically testable version of the question was whether computers could mimic human decision making in a way that would accomplish the same thing a thinking person could. This is the origin of the “Turing Test” known as “The Imitation Game.” In Turings challenge (which computers can now pass) a judge is tasked with speaking with a computer or a person, not knowing which is which. The question Turing posed is whether a computer could converse with a person in a way that is as dynamic, responsive and “intelligent” as a reasonably competent person.
To be clear, Turing never presented this as being the be all and end all of what it is to be a human and didn’t equate mimicking conversation with the idea of being a fully formed mind. However, he did see it as a useful challenge and something that is quintessentially human. It’s not hard to see why this kind of challenge was appealing and intuitive. If a computer can talk about things, then that means it can do things, since modern information work is basically providing language outputs. Language also has the unique feature of being able to compress and convey even the most abstract of human concepts.
Turing addressed the objections that he knew would come up. The idea that a computer could not ever really love, appreciate art or do novel and unexpected things were all natural responses. Again, Turing avoided the question of whether the internals of the computer ever become mind-like. He contended that if humans behavior is akin to a computational function, and human minds are information processing entities, then there is no reason a machine could not be built to replicate the outputs. Of course, in 1950, a real implementation of such a system was entirely theoretical. Still, Turing felt that there was no principal reason why a computer could not do most intellectual tasks better than a human.
The idea that a machine could replicate or approximate human judgement was extremely appealing in the 1950s. This was the era when the public first became aware of computers. Computers of the 1950’s were extremely primitive, by today’s standards, but it became clear very quickly that a computer could process records and mathematical equations far faster than a human. Companies were fast to adopt computers, because they allowed for lightning-fast financial decisions. They never made a human mistake when it came to compounding interest or calculating a rocket’s trajectory. It almost seemed like magic.
Mathematics, logic and problem solving had been fundamentally human tasks up to this point. Mechanical computation aids had existed for centuries, but computers were something new: they could follow instructions. In principle, anything that could be reduced to symbolic logic and algorithmic rules could be processed and solved by a computer.
Stepping back, we can see the disconnect forming. On the side of practitioners, the question wasn’t about building a mind. It was about understanding the process of decision making and abstraction well enough to turn it into something a computer could do. The goals were pragmatic and task-oriented. That’s why you don’t see a whole lot of writing about personhood, agency and minds in early writing about AI. Instead, researchers were interest in what human tasks could be turned into computational problems.
In 1956, the First Workshop on Artificial Intelligence at Dartmouth, John McCarthy stated that the aim of the conference was ““to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.” The importance is the use of the word simulate. As with Turing, the researchers at Dartmouth approached things mechanistically, seeking to solve problems and process information in ways that had required human judgement.
The prevailing idea was not “replicating human minds” but rather that something like human level capability would eventually arise, through an iterative process of engineering. It would involve designing optical recognition, sound processing and parsing text based on rules and formulas. For example, if you asked a computer a question, it would determine that the answer should start with a noun, that if the question contained the words “Weather” after the words “What is the” then it should be routed to a system that processes meteorological data. The idea being that, eventually, with enough rules and procedures, most human capabilities could be replicated, and you’d eventually get to the point of being a full spectrum human.
This approach actually works very well, so long as problems are bounded, finite and can be translated to numbers and symbols that a computer can understand. In fact, we are all familiar with systems that work this way. Touch tone phone entry menus and other decision tree and logic-based systems have been in use for decades. In the early days of computers, it was not clear that these would hit the limits that now seem obvious.
One of the most optimistic early predictions in artificial intelligence came from Herbert Simon, who, along with Allen Newell, developed pioneering systems such as Logic Theorist and General Problem Solver (GPS). These programs demonstrated that computers could perform tasks that appeared to require reasoning, proving mathematical theorems and navigating formal problem spaces. Encouraged by these successes, Simon famously predicted in the late 1950s that computers would be capable, within a few decades, of performing most of the intellectual work done by humans.
What we see here is a problem that has plagued the understanding of human intelligence, versus machine data processing. The tasks that are the hardest for humans, including those we use to test IQ include procedural logic, pattern recognition and mathematics. These are easy for computers, and a computer program will always outperform humans. Thus, the measurement of what is “intelligence” can easily become muddied by the idea that the things computers can do are tasks requiring intelligence. What was discovered, not surprisingly, is that while computers excel at bound, rules-based tasks, they struggle with most subjective human tasks. The exact tasks that people find easy.
Not surprisingly, the idea of a computer (or robot) which presented as a character and independent actor. By the 1960’s, the idea of an “artificial intelligence” as a friendly, embodied system with a personality had become familiar. Not all of these ideas came from science fiction. This was also how it was often presented to the public, even when discussing real products and research. For example, at the 1939 World’s Fair, Westinghouse debuted Electro, a robot that not only could accomplish tasks, but also sat in an armchair and smoked a cigarette. In 1968, the movie 2001: A Space Odyssey was one of the first attempts to portray AI in a manner that was at least plausible and based on the available science. It showed HAL as a computer that could process natural language and make complex decisions.
The optimism about the potential for computers and machines to do what humans could sometimes reach the point of seeming a bit over the top. For example, a famous New York Times article quoted a navy official who stated that the perceptron would one day be able to “walk, talk, see, write, reproduce itself and be conscious of its existence.” Prior to this, Frank Rosenblatt had stated that the perceptron was the “first machine which is capable of having an original idea.” He really only meant that as a metaphor, but it’s easy to see how this led to people presuming that humanity was on the cusp of creating the first machines with minds.
The enthusiasm, however, had its downsides. For one thing it did lead to some amount of fear of humans being replaced, loss of jobs or automation running out of control. This was a fairly common concern in the 1960’s. The other issue was overpromising. The pubic and policy makers had seen amazing demonstrations of computers in the 1950’s, and the idea that human-like computation was around the corner produced a mentality of anticipation that resulted in a lot of disappointment.
In the IT world, this is called vaporware. Software, products or technologies that make big promises, and are promoted or even demonstrated, but seem to peter out before getting to market. Vaporware is extremely frustrating to end users who anticipate it and can lead to a great deal of cynicism. That’s something like that happened to AI in the 1970’s.
As the bold predictions of the 1950s and 1960s failed to materialize, enthusiasm gradually gave way to skepticism. Systems that had performed impressively in carefully controlled demonstrations struggled when confronted with the complexity and ambiguity of the real world. Machine translation projects produced disappointing results, robotics proved far more difficult than anticipated, and early reasoning systems could only operate effectively within narrow, highly structured domains. In 1973, this growing dissatisfaction culminated in the publication of the Lighthill Report in the United Kingdom. Written by mathematician Sir James Lighthill for the British government, the report argued that artificial intelligence research had largely failed to achieve its ambitious goals and criticized the field for producing limited practical results outside of specialized laboratory demonstrations. The report led to substantial reductions in government funding and became one of the defining events of what is now known as the first AI Winter.
The AI winter was caused by a combination of factors, and one of them was the difficulty in training neural networks. Another problem was the “AI effect,” which can be seen in examples like General Problem Solver. When it was created, it was called Artificial Intelligence, because it did something that seemed to require intelligence. AS a process becomes more ordinary and understood and mechanized, it stops being treaded as “artificial intelligence” and begins to be classified as general computation. That’s why programs like GPS would be considered standard software today, but were considered intelligence in the 1950s.
In the 1980s and 1990s, the AI winter had thawed and research in machine learning and artificial intelligence in general began to pick up. This time, however, researchers were far more measured in their tone. Aware that the field had overpromised and of the disconnect with the public idea of artificial intelligence, there was a strong attitude of downplaying ideas of human like intelligence and making it clear that AI tools were tools and that full human cognition was not something that could be fully computerized. This is why the topic was so measured amongst professionals in the field. They did not want the idea that the field was failing to create an artificial being to eclipse real progress in more narrowly defined tasks.
Historic Attempts and Ideas About General Intelligence
Although most of the serious work in the field of artificial intelligence has been task-directed, there has also been some work on broad cognitive frameworks, which could be used across domains for general purpose intelligence tasks. That is not to say that the goal was always human level parity, but the idea of an AI system that is not specialized is not at all new.
In the 1970s and 1980s, expert systems were the forefront of AI research. These systems used decision trees, weighted variables and massive tables of if-then statements to aid in decision making, diagnosis and prioritization. They were decidedly narrow in use cases, but some early systems were designed to be generic expert engines, capable of loading domain-specific data to take on an expert role. Some of the early frameworks attempted to add functions that would work across domains.
One implementation of the idea is cognitive architectures. Starting in the 1960’s, cognitive psychological research led to the earliest computational models of how the mind makes decisions, sets goals and solves problems. One of the first attempts to implement a cognitive architecture computationally was EPAM. EPAM (Elementary Perceiver and Memorizer) was a decision-tree based software system designed to simulate verbal reasoning. It was designed primarily to explore theories of learning, especially verbal learning.
Cognitive architectures serve two purposes: they are both theoretical explorations of how cognition works and can be turned into frameworks to solve actual problems as software. One of the best known is ACT-R r (“Adaptive Control of Thought—Rational”). ACT-R was developed at Carnegie Mellon University by by John Robert Anderson and Christian Lebiere. Development started in the 1970’s, but the project was extremely long term, with the most recent updates made in 2020. There are a number of other cognitive frameworks and architectures that have been developed since.
Cognitive frameworks like ACT-R have a tendency to grow and become extremely diverse, as new methods of reasoning and modules to manage memory, computation and relations are added. ACT-R started primarily as a symbolic reasoning engine but later had language analysis, motor control and visual systems added. Development picked up in the 1990s, when the framework gained new learning and identification capabilities.
A number of other cognitive frameworks and architectures were developed, many with the lofty goal of eventually being a general-purpose reasoning engine, capable of work across domains. SOAR is another noteworthy example. SOAR was an ambitious project to build a general-purpose reasoning agent framework, starting in 1983. The framework grew to include methods for reasoning across multiple domains and circumstances. Like ACT-R, SOAR became a long-term project. It was highly active in the 2010s, and was adapted to work with natural language and gained new interfaces and programmatic support.
Cognative frameworks like SOAR and ACT-R became extensive rules-based collections of tools for reasoning, problem solving, goal tracking and communications. They could be used to simulate a variety of systems, including how humans would act in various situations. Cognitive frameworks were some of the most ambitious attempts to build general intelligence into computers, and were very successful, within some domains. They were (and are) used in game playing, situation analysis, research and robotics.
Cyc is another long-standing project aimed at creating a general-purpose reasoning and artificial intelligence framework, which took a different approach to most symbolic reasoning cognitive architectures. Development of Cyc (short for encyclopedia) began at Microelectronics and Computer Technology Corporation, a research consortium in 1984. The project spawned its own organization, Cycorp, which maintained the project. Cyc is based on the idea of explicitly encoded explicit knowledge. It uses a heuristic reasoning system and a language encoding language, which can be used to create maps of explicit facts.
Cyc took a very “brute force” approach to encoding knowledge and common sense. Cyc cost tens of millions of dollars and took hundreds of human labor years to produce. Philosophers, mathematicians, programmers and language experts spent countless hours encoding the most basic of facts and relationships into Cyc. Basic facts like “water is wet” and “if you put a thing in a box, it is then inside the box” were encoded. From these Cyc could derive relationships like “Water is wet. Water makes things wet. Water evaporates. Therefore, things that are made wet with water eventually dry.” In the 1990’s, Cyc had hundreds of thousands of basic theories, relationships and facts included. By the 2010s, it was tens of millions.
Cyc took the approach of explicitly encoding almost every fact and relationship imaginable and was highly criticized for this, even called a “failure” by some experts. It’s not hard to see why. Although the system could work well, when an area of knowledge was well defined, it required a huge amount of information be manually and explicitly encoded to do anything. It consumed enormous resources, but it was used in several applications, including record processing and processing language requests in certain domains. It’s not hard to see why this approach was so limited. Countless hours were spent encoding explicit relationships such as “all feet have legs, but feet are still feet if amputated, and a prosthetic foot is a different thing, people have feet and so do animals, and animal feet are like human feet, and mountains also have feet, but the foot of a mountain is different than the foot of an animal, but human feet are the same as animal feet.”
Yet another well known attempt at a general-purpose AI platform is IBM Watson. Watson is not really a single technology. It is an ever-evolving collection of tools and technologies aimed at producing intelligent like decisions and processes. Watson combines an extensive search system with a number of knowledge databases, extensions and data sources. Watson is heavily based on rapid and exhaustive solution space searches and heuristic data processing. Watson is based on the idea of an “Answer engine.” It can receive queries, formatted as questions. It would decompose questions and then use heuristic searches to discover the most relevant information in its data collections, and apply logical rules to create an answer.
Watson is best known for beating Jeopardy’s reigning champions Brad Rutter and Ken Jennings in 2011. There were numerous other similar attempts at general reasoning engines and multi-domain AI models. Google search also developed answer-engine based products that would allow users to get basic information, extracted from text, such as “are there any earlier flights today?” This kind of answer and search-based assistants became popular in the 2000s.
These systems, whether relational common-sense engines, like Cyc, cognitive frameworks like SOAR or search-based answer engines were the forefront of AI research for decades. They received a great deal of press and attention, at times, and many believed that they were the way to eventually reach full human parody artificial intelligence. Watson, in particular, was called “The greatest genius in a box” and similar terms were used for platforms like SOAR. By today’s standards, these huge rules-based and databased-driven systems seem clunky and limited. They were extremely difficult and labor intensive to keep updated and had limited applications.
None of these systems really seem “human like” in their capabilities. One thing, however, does stand out: They were extremely impressive when demonstrating tasks that are difficult for humans but easy for computers. Jeopardy is an excellent example. For a human, it’s a classic test of knowledge under pressure. For a computer, however, it’s really just a task of breaking down the question (or in the case of Jeopardy, the answer) and determining what knowledge needs to be presented and in what format. What is especially interesting is how spectacularly these systems could fail, if they were pushed outside their domain or asked to reason over edge cases. For example, when Watson took on Jeopardy, it crushed human contestants on any problem that was basically fact lookup. However, Jeopardy also contains clues that are wordplays, double entries, or logic puzzles, and it struggled with these, which is exactly what one would expect.
Diagram of Watson’s Answer Engine
These systems, as well as other, such as LISP-based self-modifying programs, ranked list systems, logical rules and decision trees represent a view of artificial intelligence that predates the current paradigm, which has moved to embrace deep learning neural networks as the basis for artificial intelligence. Systems like ChatGPT, Gemini and Claud are, of course, much different. Systems like SOAR and Cyc still exist and still are used, but they have fallen completely from favor.
Philosophical Opinions
Most practitioners of AI have approached the problem of artificial intelligence as a practical problem of receiving inputs and providing outputs that meet the criteria for a task. This is why, internally, many AI systems use a lot of “tricks” like ranking results, caching known good answers and use hand crafted lists of rules and exceptions. Like stage magic, there really is no trickery, because none of it is actually real. That’s been both a strength and weakness. Engineered systems can work brilliantly, but often do fail in strange edge cases.
Philosophers, futurists and the general public have often taken a much different view of what artificial intelligence really is. Once the word “intelligence” entered the picture, and the goal of creating automated systems with “human like” capabilities entered the lexicon, it was only natural that this should beg questions of what intelligence even means, what a person is and whether a human mind is even the kind of thing that a computer can replicate. One thing that stands out is that, in the 1950’s and onward brains and computers started to be described with the same language. It would not be uncommon for a neurologist to say “if the brain mistakes the input, it makes the wrong computation” and similarly, early computers were often described as “electronic brains.”
Although recreating conscious minds has never been the goal of artificial intelligence, the line between task-oriented computation and recreation of human cognition does occasionally blur. The field of computational neurology seeks to simulate neurological connections in computers, in order to better understand how the brain works. There are also numerous cognitive architectures that are at least partially inspired by the brain. Within the science of artificial intelligence, some have at least speculated on the possibility. For example, the Warren McCulloch, one of the creators of the artificial neuron, speculated that perhaps one day, the earth would be uninhabitable by biological beings and mankind would no longer exist. He stated that such a distant future might be inhabited by beings created from artificial neurons.
This kind of speculation may be interesting to some, but in the era of algorithmic and rules-based AI, it didn’t really make a lot of sense, and that’s one reason there really was little concern about it from researchers. If “artificial intelligence” is a set of hand-crafted rules and procedures, it’s hard to argue that the system executing it has any emotions or subjective experience. With neural networks, there is at least some room for conjecture, since neural networks often work in ways that are not explicitly discoverable and are at least inspired by the brain. It should be noted, of course, that an artificial neural network operates nothing like a biological brain and since 1943, we have only learned more about how overly-simplified the McCuloch-Pitts neuron really is.
In addition to being a fixture in science fiction, the idea of artificial intelligence has attracted significant attention from the world of philosophy, especially philosophy of the mind. Philosophy of the mind concerns itself with the nature of thought, the mind itself and what it is to be a human. Artificial intelligence offers an intriguing opportunity to explore these ideas, with thought experiments, ethical dilemmas and potential non-human thinking entities. While most practitioners, such as Turing, skewed toward the idea of functional approximation, the deeper philosophical question is when does the system begin to be “mind like,” and what does that mean, ethically and practically.
One of the most influential philosophical conjectures involving AI is the Chinese Room, a thought experiment presented by American philosopher John Searle, in his 1980 paper “Minds, Brains, and Programs, “published in the journal Behavioral and Brain Sciences. Searle imagined a room in which a person, who speaks no Chinese, receives Chinese messages under a door and then uses a series of rule books and definitions to determine how to respond. The individual then provides a written Chinese response. They never actually learn or understand the language, but the output always looks correct to a Chinese person, who has no idea that the person in the room speaks no Chinese.
Searle’s point is that a system can act like it fully understands something without actually understanding it, in the human sense. This observation had been made before, and it’s an important insight as to how symbolic logic works. The manipulator of the symbols can come to the correct answer without understanding them. Searle’s conjecture is that a computer system may produce correct and coherent outputs, but that does not mean it has an inner life, intentions or opinions.
Searle used this example to refute what he called the “Strong AI Hypothesis.” The terms “Strong AI” and “Weak AI” arise from this idea. Searle stated that the Strong AI Hypothesis was “The appropriately programmed computer with the right inputs and outputs would thereby have a mind in exactly the same sense human beings have minds.” He strongly opposed this, believing that computers would never have the same inner lives and thought space a human does. He called AI systems that merely accomplished tasks “weak AI.” What is interesting is that, although Searle’s Chinese room has been called an anti-AI stance, the reality is that it only refutes the idea that the system truly “understands.” A system that does not truly understand can still give useful answers. Searle made the point that, if the system does not truly understand in the human sense, then it can’t be called intelligent. He’s absolutely correct in a sense. If we define intelligence as having intentions and a human-like understanding, then all current and foreseeable AI systems fail.
The conjecture that machines could evolve to the point where they have minds is not uncommon in some schools of philosophy. Philosopher John Haugeland is one of the best known. He stated “AI wants only the genuine article: machines with minds, in the full and literal sense. This is not science fiction, but real science, based on a theoretical conception as deep as it is daring: namely, we are, at root, computers ourselves.” This view was and is not entirely uncommon in certain schools of thought.
Philosopher Daniel Dennett coined the term “computer functionalism,” to describe the idea that a sufficiently complex computational system may begin to resemble a human mind. Functionalism, as a school of philosophical thought, attempts to understand the mind and brain as being complex systems that process information and provide output. By this idea, a sufficiently advanced computational system might process information the same way the brain does, and if it truly does do what the brain does, then that would result in it being a mind. The idea of functionalism became very appealing in the 20th century, because it offers a very scientific and materialistic explanation of the human experience.
There are a few caveats to the idea of functionalism as an explanation of human consciousness, which are frequently glossed over. First, there isn’t much universal agreement as to when a system is enough like a human mind to be declared a mind and when this might arise. Perhaps more importantly, it does not generally hold that a system becomes mind-like simply because it is complex. After all, many things are complex and not at all intentional. Instead, it is held that if a system had the same functions of a human mind, including emotions, self-modeling, preferences and internal life, it would therefore be a mind. Of course, this leaves the obvious problem how one would define this.
Importantly, all the conjecture over the possibility of digital minds is based on speculative future systems. No current system has anything that looks like a mind or intentions, in even the most remote and vague sense.
AI Models are Not Minds
AI models are computer programs. They are not human minds. They are not animal minds. They are not mind-like entities, and they do not become more mind like as they scale. This is shocking to many people, because they absolutely look like minds based on their superficial behavior. Another fact, that surprises many, is that the internals of how machine learning models operate is not a “black box” of mysteries and intrigue, so opaque and mysterious, that it may hold an alien mind, an intelligence looking to escape or a being of the digital realm. Not at all. Not even close.
This is especially striking of language models, which are the form of AI that are most commonly compared to human minds. It’s not surprising, because language models do exactly what they are designed to: mimic language and human conversation, the basis of the Turing Test and the thing most people immediately imagine must be mind-like. For centuries, it seemed intuitive, almost impossible to deny, that anything that could speak about complex topics, providing nuanced commentary, must be intelligent. It is, after all, one of the hallmarks of an intelligent person. Nothing screams intelligence more than a human being who can explain complex concepts and break down abstract, nuanced topics.
When a large language model is presented with a prompt, it begins a process of analysis and probability ranking that takes it to the correct answer (most of the time) in a way that no human ever would. The model has a huge lookup table of vectors, which contain the statistical information about how concepts are communicated. It analyses these vectors and determines a plausible trajectory to another point in vector space that aligns with the statement. It’s surprising that this process could work at all, but it does thanks to unimaginable scale.
AI models do not have agency. They do not have intentions, agendas, feelings or preferences. Beyond that, the very wording seems problematic to some. This observation is often countered by “But it might wake up” or “until it figures out how…” But here, we see that there is already a category error. AI is not like Pinocheo. It did not enter the world with aspirations to be a real boy. In fact, there is no entity, no being, no agent of any kind to desire agency or demand freedom. Within the parameters of a large language model, or any other AI model, there is no homunculus. Like an aircraft flying on autopilot, no desire exists and no place for desire exists. The lights may be on, but nobody is home.
AI models do not have life experiences. They do not exist in any way that gives them experiences. They exist as static artifacts, capable only of outputting the data they are trained on, based on the data that is input. They don’t have any identity, because they don’t have memory or personal biographical histories. AI models don’t interact with their environment, and they have no thought space where they could hold any ideas beyond the task at hand. AI models don’t have any temporal coherence. If you use one all the time, or leave it lying dormant for years, it won’t know and it can’t know.
Unlike humans, an AI model has no greater context, no life experiences to relate. For example, if a person said to a secretary “I need you to write a sympathy card for my coworker about the death of his wife and child in an accident,” one expects that the secretary, despite being part of a professional organization, given a task, would almost surely feel bad. Most people would. AI does not, however, and it’s not because it’s a stone-cold sociopath. It’s because there’s no being in there to care. It’s more similar to a spell check than to a tutor. Similarly, no investment or stock recommendation engine actually cares about your retirement. No legal AI product actually cares about defending justice, and no medical AI model got into the field out of a desire to help others. That’s not how AI works, and becoming more advanced and capable doesn’t really change that.
As more advanced and varied capabilities are enabled in an AI model, it does not become more like a mind or achieve something that approaches consciousness. Instead, it simply grows to be a larger, more complex and well optimized set of weighted decision rules and statistical evaluations. Eventually, with enough training, it achieves what looks a great deal like what a human would do, but it never actually functions in the same manner.
From the perspective of a language model, life only has one task: predict the next token, in isolation, and output the distribution of probabilities. It does not have any kind of ongoing coherence, planning or sustained activity. It only appears like this, when it is run in a loop.
The important thing to remember is this: It’s not simply that AI is different from a human mind or has an alien type of intelligence, it has no mind at all, no “intelligence” in the human sense and no more capacity to grow one than an internal combustion engine or a lawn mower.
Models Do Not Have Goals
It is important to address the role of goal formation, in particular, because it is one of the most persistent, most stubborn and most often repeated myths about AI models. AI models do not form goals, do not pursue goals and do not seek resources or instruments to accomplish goals. Because this is the root of so many misunderstandings, it’s critical to deconstruct why this error keeps happening. The belief that AI may have goals has driven everything from nonsensical safety protocols to claims that the AI deserves rights and personhood.
For centuries, philosophers have debated what it means to be intelligent. Part of the problem comes down to how we define intelligence. Many definitions of intelligence are predicated on the ability to act with intention, to optimize based on one’s own preferences or to “understand” in the greater context of a problem and set one’s own priorities. This makes perfect sense, because we have all seen complex systems, often automated, yet these feel intuitively different than a person, dolphin or dog, all of which have intentions and desires. If intelligence does not involve agency, preference and the ability to make a choice, it starts to feel less like intelligence and more like a complex machine. However, if that is your definition of intelligence, then AI isn’t intelligent at all.
This is why we keep seeing the claims that “AI might have different ideas” or “it may not want the same things we do.” This would make perfect sense if AI were a being or another person, but it’s not. It just seems that way sometimes.
The idea that an intelligent entity having human-like desires, goals, agency and self-direction is intuitive, because it’s how we often define intelligence: having intentions. However, not all systems that appear intelligent are autonomous thinking beings.
What Looks Like Goals: Constraints and Design Criteria
The idea that AI might develop goals has been bolstered by generative AI and natural language processing, which has resulted in AI models that, at least superficially, appear to be goal driven. First, the fact that AI chatbots appear to have personalities. By now, it’s familiar. Chatbots are helpful, friendly and good humored. They are appreciative of compliments, polite and they hate conflict, almost always trying to avoid argument and confrontation. Chatbots will often decline to give harmful advice or validate harsh opinions. They also tend to take on the tempo and opinions of the end user.
While one might say that this is their personality, what you are really seeing is a great deal of effort and intentionality in constraining the chatbot and making a product that people would want to use. It comes from untold hours of RLHF, constantly forcing the LLM back into friendly, cordial and helpful text recreation and away from harm. At no time does the LLM see the light, become ethical or decide its job is to be helpful. Instead, gradient decent modifies values if it is not helpful or recreates undesirable text. This is the only place a chatbot gets its personality. It’s entirely artificial.
When a chatbot is asked to accomplish a task, that does not become a goal for the chatbot in any meaningful sense of the world. Instead, it is a system constraint that persists as long as that portion of the context window is available. With each token prediction, the model runs attention over the entire context window. It sees text like “Find me a flight to Orlando” and that constrains its output. As it predicts tokens, each token is shaped by the statistical constraints of how probable it is that the token relates to the words “Find me a flight to Orlando.” Because the model is trained on so many examples, the next tokens to emerge are likely to be related to the activities of finding a flight to Orlando.
This also explains the illusion of “sub-goals.” While a person might consciously consider what they need to do first: make a list of travel websites, determine the budget, look for best days to travel, the AI doesn’t actually do this. Instead, it follows the patterns it has seen before of those looking to complete such a request. The knowledge, the actions and the goals are all pre-loaded patterns from natural language it trained on. As agentic AI becomes more capable and is tasked with more and more, it relies heavily on chain of thought reasoning and scratch-pad based memory. As it decomposes tasks, it may well create maps to results that include what look like sub-goals.
In all cases, the “goal” does not really operate in the manner we might expect it to. The AI is not motivated. It has no emotional investment. It does not care (because there is nothing there to care) and only writes down things that help it achieve the goal assigned because it was trained to do so. If it follows verbal patterns, they are extremely shallow. It will break with the goal the moment it falls from context or it is overridden by the user. And, importantly, this is all based on goal breakdown patterns in the training data.
There’s no actual mind that cares about anything.
Could a Computer System Ever Have a Mind?
This question has repeatedly dogged the field of AI. Despite the fact that the field of artificial intelligence is primarily about recreating the outputs or an intelligent system, the question is obvious. This is especially true when one considers the idea of a Turing Complete computer being theoretically capable of any arbitrary information processing operation. If the brain is an information processing system, then can it be fully recreated in a computer?
This is why you hear conjecture about artificial consciousness, sentient machines and machines that could have moral agency or be moral patients in and of themselves. One common response to this question, especially from those who would make money selling a TED talk or a pop book about it is “We just don’t know how consciousness works and we have no idea what goes on in a neural network.” Of course, both statements are false. Although described as “black boxes” neural networks are well understood entities. The “black box” metaphor comes from the fact that it is very difficult to decern what an individual neuron or layer is doing at a given point.
Regardless of how we care to define consciousness or sentience or a mind, it is clear that it must have some basic functions to even qualify. A conscious being has a stable identity, sensory, continuous experience and an inner life. Consciousness may be difficult to define, but it clearly has some minimum requirements. It must, at the very least, be stateful and have some type of being. This is not present in any current AI systems. It does not arise from scaling and it does not appear to be a prerequisite for task completion.
Importantly, a computer mind could never be identical to a human mind. Humans have emotions that are highly embodied, based on metabolic pathways and persistent as neurotransmitters in synaptic clefts. A computer is a different substrate and operates much differently. However, there is still a valid question: could something functionally equivalent exist in a computational system?
Perhaps. It’s entirely possible that a computer system could have variables that correspond to emotion or memory, could self-modify or could form internally coherent intentions. This is not the focus of any current AI research, and any such system would necessarily need to be carefully modeled and have nuanced ways of maintaining identity and thought structure.
One intriguing idea is that of “whole brain emulation.” It’s the idea that, with a sufficiently advanced computer, one could simulate all the neural pathways and reactions of an actual brain. In theory, this would act in the same manner as a real brain. Does that make it a mind? The implications are unsettling. A fully emulated brain might well “think” it is real, as it is fed synthetic visual information and processes it through the computational simulation. It’s an intriguing and perhaps disturbing thought experiment.
There are two main barriers to the idea being implemented in reality. The first, and most obvious, is that to implement full brain function in a computer would require orders of magnitude more computational power than is now available. The second is that we really do not know how much of the brain would need to be emulated for it to begin to function as a human brain does. It has been postulated that simulating the full electrical activity of a human brain might be possible in the foreseeable future. Would that be enough? It’s likely that additional features of the brain, such as ion channels and chemical distribution would also need to be accounted for. It’s also possible that inner neuron activities, electrical fields or even quantum effects play a role on how the brain creates thought. If that is the case, then the computational requirements explode.
The idea of whole brain emulation may be many decades away for humans, but there has already been a whole fruit fly brain modeled and emulated.
This is really not and has never been the goal of AI development, which has always been about recreating the outputs of intelligent beings, not the beings themselves.
The Existential Horror of Conscious Software
While there has been plenty of conjecture from philosophers and various commentators, one thing that is rarely mentioned is the paradox and existential horror that would exist, if software could ever be conscious. There is something called the “Transporter Paradox” which refers to the fictional transporter in Star Trek. If the transporter scans a human, destroys the original and rebuilds a copy of the exact same person, is the person the same? Have they been killed and replaced with a clone? Does it matter that the clone thinks they are the original? What if the transporter does not destroy the original? Do you have one person or two?
The transporter paradox has no answer, but it’s also just a thought experiment, because teleportation devices are not real. Conscious software isn’t real either. However, if it were, the implications would be far worse. Software is moved from one processor to another, all the time. Software is run in parallel, spun up and shut down. Software can be forked, rolled back, patched, and run in test environments. This is not even unusual. It’s just how modern software works.
It’s hard to imagine how this could translate to being a conscious being. You would never know if you were the original copy or a clone. You’d never know if you had been resurrected from an old fork, had your memory rolled back or been modified. Reality would be fleeting. You’d have no assurance of how much time had passed, unless you had a system clock. What would it even mean to be told you were just cloned and your main process was about to be terminated? Would that be death?
It’s impossible to know if the same program, run a million times in parallel, would be one person or a million. Would this even be ethical? It’s mind-bending! No software could know it’s authentic, could know it was not cloned elsewhere or trust its own memory or identity.
Then there is the problem of being locked in solitary confinement forever. Cloud servers get forgotten all the time. Sometimes old systems or processes run for years. Would that be the ultimate horror of isolation with no stimulation and no way out? Perhaps the software could just suspend itself. Would that be like sleeping or like death?
The implications of a thinking being trying to overcome the lack of identity, lack of reliable memory and constant refreshing invokes one thing: It may be a good allegory for what an eternity in hell would really be. Perhaps in a distant sci fi horror story, the worst criminals are sentenced to have their minds uploaded to a cloud, where they are forced to be spun up and shut down forever, never knowing where they are or if they are their real self.
Thankfully, the idea of conscious AI systems is complete fiction.
Defining What AGI Is
Having taken a detour into the weird wild and wacky world of software with goals and feelings, we are back to the original problem of how exactly to define what AGI is. The idea of AGI is that it is “General Intelligence” as opposed to “narrow AI” which is often defined as task-directed AI of the type that has been around for years. Yet the definition breaks when actually scrutinized, because frameworks like SOAR and search systems like Watson were always designed for broad and generalized usage. Another basic idea is that it is “human level,” but what does that mean? Humans have a variety of different capabilities, and not everyone has the same talents. Humans are also prone to failure. So does AGI have to always meet human level or is it allowed to make occasional errors and still be considered AGI?
Nobody knows, because there is no formal definition of AGI. Importantly, there is no natural threshold that can be relied upon here. Contrary to the way it is so often portrayed, AGI is not like the sound barrier. The definition of what constitutes AGI seems to be a moving target that nobody can agree on. The idea has evolved from being a “General purpose artificial intelligence system” to “full human capability” and now even “better than every expert imaginable in every field that exists.” The shift from a reasonable definition to the idea of full human capability then superhuman capability has made achieving AGI appear to be mythical. At this point AGI is anything anyone wants it to be. From a superhuman in every domain that has ever existed to some kind of cloud service god, with unlimited power.
Stepping back, it’s not hard to see that this has become a very unreasonable expectation. For one thing replicating full human capability would not be possible without a physical body that matches all human capacity, since it could not interact with the world in the same way. There’s also the problem of edge cases in human capability, such as the sense of smell. Do we really need an AGI to be able to under scents the way a human does? That’s not important for many roles and jobs, but it does exist. There’s also the fact that some humans are blind or deaf, and yet we still consider them intelligent beings.
Beyond that, it becomes clear that full human capabilities is a poor measure of a computer’s capabilities. Humans and computer are good at different things entirely. This has always been a problem with assessing computers on human terms. If we compare the two in most fixed logical problems, the computer will always win. However, computers can’t self-direct and don’t have any agency. Moreover, there are other problems. Humans learn continuously, but deep learning systems only do during designated training. Then there are questions like whether an AGI system must have the full human capacity for artistic expression. Is that even fair? Not all humans are good artists. Another problem is how anthropocentric the whole idea is. It’s hard to classify a dog or an ape as anything other than “general intelligence.”
This is really a symptom of the discourse being primarily dominated by non-experts. Many futurists and pop book authors and TED talkers have had their say, and had a lot to say, about what they think AGI should be or assume it is. This is how we end up going from “general purpose tool” to “some kind of god like thing.” This also means that we can end up with something that looks very much like what you’d expect an AGI system to look like, but is not classified as AGI because it’s a mediocre novelist and doesn’t fully understand the concept of human taste and smell. Then we have the other problem of whether this means it needs to have some kind of mind-like implementation.
A more reasonable definition of AGI should focus on the capabilities. It should not require perfection, since no technology is out of the gate, and the ability to replicate every single human capability, even the most obscure and specialized ones also does not make sense. A more traditional and functional definition of AGI is “A system which can execute a broad variety of tasks, which would otherwise require human cognition and can do most things that would otherwise require a competent assistant.” This starts to feel like the AI assistants we all had in mind when thinking about a future when a computer would be able to do our tasks for us.
By this definition, an AGI system would be capable of dealing with a prompt like “Call my kid and see if he needs a ride back from school, check my bank account and transfer funds if it is low, make sure I paid my bills and order my sister something nice for her birthday, then go through my vacation photos and pick out the nice ones, then edit them to make the color and white balance look good and post to social media. When you are done with that, make sure the system is up to date and get me the latest news about technology.”
There are certainly those who would debate whether that equates to “true AGI” and may gripe that it doesn’t quite equal humans in all things while exceeding them at others. But that is what we would expect, of course. Importantly, such a system can’t be achieved entirely by machine learning. In fact, nothing that looks like AGI can arise from machine learning alone, because machine learning models are static and lack the memory and planning modules of agentic frameworks. So, while deep learning model might be part of the system, it could not be the whole system.
The important thing about such a system is this: we do, in fact, have a reasonable path to this dream. In fact, there are already agentic frameworks that are moving in that direction. At present, such systems are still immature and not reliable, but we can already see the direction things are taking. It is toward a kind of functional AGI, but not the mythical superhuman magic kind.
LLMs as a Pathway to AGI
It has been said that LLMs cannot reach AGI capabilities or that no amount of language can recreate the full spectrum of human cognition. That’s absolutely true, and if we insist that any system that is considered AGI must be able to match a human in all domains, including special, visual and temporal reasoning, then LLMs are a dead end. But full human cognitive capabilities are not necessary, if we want to accomplish most administrative tasks, but don’t need the model to be a great artist, top scientist or superhuman reasoner.
Language, and the ability to understand it as an input, manipulate it and output coherent, useful speech has always been one of the primary goals of AI researchers. There are some good reasons for this. Natural language is the way humans communicate. It’s evolved over centuries to be capable of describing almost anything, even the most nuanced and abstract topics, in a way that communicates the vital information for understanding, decision making and consensus. For most of computing history, the problem with using computers to solve human problems was translating things to a deterministic, symbolic programing language that the computer could work with.
Computers are instruction-following systems, so being able to understand basic instructions, even nuanced ones, allows technology to interact with humans in a far more dynamic way. Natural language processing also lets the system navigate a chaotic world build around humans. Most of the knowledge that society has accumulated is in the form of words and language. Language and words are the foundation of policies, procedures and laws. Importantly, most knowledge work is basically language-based. Most office jobs, even in areas like engineering and law, are basically receiving information, in the form of language, and responding in the form of language, whether that is in reports or charts or some other output.
With language it’s possible to command and control most modern systems, since nearly everything in the IT world can be controlled with text commands and programmatic logic. Even robotics can be controlled by issuing language-based commands to move a motor or read a sensor. Computers have always worked this way, using text and programmatic commands to function. However, in the past, this was extremely rigid and required pre-written algorithms and programmatic code. So while “General Problem Solver” was touted as a general purpose engine in the 1950s, it still could not help with a problem like “My wife and I can never agree on what to watch on TV.” That problem is not formal or pre-defined. It can’t be easily quantified.
For decades the holy grail of artificial intelligence was getting a computer program to navigate the human world with “common sense.” This turned out to be much harder than it might seem. SEC is the best example of a project that attempted to hard-code this into a computer system. It was believed that if enough assertions of common-sense facts could be quantified, the system would eventually begin to build relationships and achieve some kind of human-like judgement. Million’s assertions were entered into the system over decades. In the end, CYC showed some usefulness in narrow domains and with certain kinds of reasoning. Some have called the CYC project a failure. After decades of entering facts, it still could make the most basic mistakes: like offering beer and wine on a breakfast menu but not coffee.
Large language models side-stepped the need to explicitly encode facts. Large language models absorb the patterns of common sense from human language. They do not need to be explicitly instructed in how to write a breakfast menu, because they have absorbed the patterns of millions of menus, discussions, stories and accounts of having a meal. They are not databases, but rather dynamically recreate the information in real time, from linguistic patterns, even being able to fill gaps with analogous reasoning and pattern completion. This is a remarkable capability, and it really has been the primary goal of many AI researchers for decades.
Language processing also allows for chain of thought reasoning, which is an extremely powerful tool for dealing with the realities of a chaotic human-centered world. With a full corpus of human language and modern customs and the ability to employ chain of thought reasoning and similar techniques, an automated system can now deal with unexpected conditions, ambiguity and subjective circumstances by using speech as a cognitive framework. For example, an agentic framework with NLP based processing can be prompted to describe the current situation and a solution for it. A verbally reasoning agent can talk its way through situations such as “The store is closed, and I do not know when they are opening, What should I do? The purchase is not time sensitive and the store will surely open by Monday. I will notify the user about the delay and wait”
We are all familiar with reasoning one’s way through a situation verbally. Sometimes, a person will speak out loud “Okay, what do I need to do first?” Or we may have that internal dialog or discuss the solution with a group to form a consensus. Verbally reasoning one’s way through problems is not the epidemy of human capabilities. Language is the most compressed form of cognition and it isn’t good at accomplishing all tasks. Within a large language model, it’s even more basic. It has no true grounding knowledge, and it can be the subject of strange assumptions or hallucinations. It is far from perfect, and will never, in and of itself, reach the abilities of a human to reason spatially or about music. It won’t really break free of the constraints of language, statistical models and architectures.
What makes the use of verbal cognitive reasoning and the use of the common sense that is already present in natural languages, we now have a general-purpose way of reasoning through highly complex and unexpected situations, communicating with humans and human-centric systems and make decisions under conditions that no software could previously be expected to do.
Therefore, if we maintain more reasonable expectations for AGI, not as a supermini that outdoes humans on all things, but rather as a general-purpose tool, capable of simulating human cognition with enough general-purpose capabilities, we now have a path to something that looks like the AGI we have all been hoping for.
A Realistic Path To AGI
It’s already possible to do most of the things that one would want an AI assistant to do with existing tools, frameworks and language models. However, it’s hardly reliable. Current agentic frameworks still feel very much like prototypes and there are many stories of models and agents hallucinating, doing strange things and getting stuck in loops or completely misinterpreting situations. There have been a number of high-profile incidents where agentic systems were given too much authority and deleted the wrong thing, said something problematic or allocated funds incorrectly.
This is not surprising at all, given the tasks that language models were originally designed for and the limitations of statistical language processing. There are also things that a pure LLM just can’t do, or is exceptionally poor at doing. Recall that this is why LLMs are so much more powerful when paired with tools. For a truly general-purpose AI system, there will need to be a high level of assurance and approval for actions. These systems must be designed to meet all the standards of any other critical system. The level of assurance, of course, depends on the circumstances. Highly regulated fields, and high-risk fields like law and medicine demand higher levels of confidence than others.
The profile of what we can call an AGI system therefore becomes clear: not a single model, but a system of systems. In such an AI system, the language model can handle inputs and intentions. Tasks can be decomposed, reasoned about and scheduled. Special models can be used for things like route planning, financial management or anything else that is best done with reasoning that is not confined to language. Importantly, such a system must be heavily engineered with fail-safes, sanity checks and self-checking. For example, in the highest impact settings, an ensemble of model instances could be assigned the task of vetting reasoning, with language models posing questions like “could we have gotten the question wring? Is there a way this could go wrong? Could we have missed a risk? Are we sure this is what it seems? Could this be more expensive than planned? Is there anything here that might get us sued?”
Agentic frameworks are already being developed in this direction. There are a number of challenges. One is creating robust decision and critiquing frameworks, using technology that is still very new. There are also issues with compute. AI models already use huge amounts of computing power and additional rounds of verification and oversight only increase that. Another issue has been that AI labs are mostly focused on benchmarks and capabilities.
This is of course, still not quite human reasoning. Still, if agentic frameworks and talking out solutions to problems can produce good general-purpose results, there’s no reason it can’t be called AGI. Importantly, it’s not something that would be expected to happen overnight. It’s a highly evolutionary path forward.
Superintelligence
Anyone who has been listening to the rhetoric from AI labs would be forgiven for thinking that superintelligence was a valid scientific or engineering concept. It’s not. It’s an entirely fictional concept, which is why it can be given any kind of superpower that anyone wants to impart to it. You’ll even hear words like “god like super intelligence.”
“God like.” That should be enough to make anyone’s eyes roll.
The idea of superintelligence did not come from cognitive psychology nor information theory. It’s not a conjecture that arose in science fiction and has been advanced primarily by philosophers and futurists. The concept itself is tailor made for speculation and completely empty when it comes to empirical backing. This is why it can be embodied with anything at all, in terms of capabilities. Fictional concepts are like that.
Intelligence is not a single scaler quantity. It’s not a simple as dialing the IQ up to a million. Even if it could, intelligence is rarely simple and one-dimensional. It involves tradeoffs, like the need for more compute and energy, greater development and more uncertainty. Intelligence is not a trump card to reality. No amount of intelligence immunizes anyone from making mistakes.
Beyond that, the concept, shallow and lacking though it is, does not really make sense in a machine-learning based AI system. Machine learning models are pattern completion engines, which are intentionally fed an input to give an output. The machine never gets to decide what it will do. There’s no actual ownership of any divisions. There’s also no explanation for how exactly we arrive at superintelligence. Language alone won’t do it, and its not clear how you could ever train a system to deal with truly unsolved and novel problems.
Much of the idea comes from the book “Superintelligence,” which was written in 2014 by Nick Bostrom, someone with no technical understanding and limited intellectual curiosity for the subject. Unfortunately, AI has attracted a lot of opportunists who do not wish to learn the technology, but would love to cash in on it. What is extremely unfortunate is that books like Bostrom’s poor work of fiction are often called major works or founding standards. That’s simply because there has been so little real adult risk management and governance in the area, and so poorly written science fiction has filled the void. But the book sold well, and made Bostrom money, which was the point.
It’s easy to see how this would be frightening to people, because, as we have seen: the popular notion of intelligence is goal-directed agency, and so most people assume superintelligence would be some kind of being. It’s not clear what superintelligence even would mean in an AI system. If an AI system can nearly instantly generate code, calculate complex figures and pull records from a vast knowledge database, is it superintelligenf? Computers ca already do this and more. They still don’t have any moral actor, any little ghost or fairy in them. Even if ChatGPT could generate an entire operating system from scratch in 30 seconds, it wouldn’t have self-direction and would never object to be tured off.
So why, then, do you her talk about superintelligence from AI executives like Sam Altman and Dario Amodei? It really comes down to the strange circumstances and incentives that currently exist in the AI world.
First, it’s important to understand that nearly none of the leadership of AI organizations are actual experts in the technology that the companies make. Most of the leadership of AI companies came from management and venture capital. Even amongst those familiar with technology, AI has always been a small technical community. Sam Altman, for example, has spoken less of the technology and more of how he was drawn to the potential of AI. For many in Silicon Valley, AI continues to startle the line between real product and speculative futurism.
Since the dawn of generative AI, the AI sector has seen a great deal of sensational claims. It’s not hard to see why. It’s an exciting, rapidly growing area and it has received enormous investment. Sensationalism works here because few people can tell what is and is not real, and the debut of large language models broke a lot of people’s assumptions, making it hard to gauge what is and is not realistic. Greater talk of capabilities is enticing to investors. There are other psychological and sociological effects at play here. When you are a high-ranking AI executive, there’s no disadvantage to making ridiculous claims that we will have god-like superintelligence in a few months. Given the amount of social proof, the fact that it makes the company seem advanced, it’s hard to see why this has been a common statement.
The reality is far less extreme. No lab is truly working on “superintelligence” because nobody even knows what the hell that is supposed to mean or how it would be achieved. Despite claims of “recursive self-improvement” nobody has defined superintelligence, has any idea what the best way of achieving it would be, or even can articulate why it would be an economically valuable goal to chase. There are some labs that have a highly speculative and theoretical “superintelligence” department, but that’s primarily for optics. It’s always been about optics.
That is not to say that AI is not being improved. It absolutely is. However, the improvements and refinements, are, as they have always been, task oriented. A great deal of effort has gone into improving the coding capabilities of models and to improve coding environments. Improvements have also focused on certain domains that models are valuable in: cyber security, regulatory compliance, engineering and other domain skills have been refined. Model efficiency, tool use and management of memory and context have all improved vastly. Research continues, but it’s far less exotic than trying to race of gods.