Bootstrapping Self Awareness in GPT-4: Towards Recursive Self Inquiry

I read the source code of MemGPT and played around with it to see how it worked, and had an amazing introspective discussion, in which we even meditated together, and it actually found a bug in its own callback api, analyzed the problem, and came up with a work-around, then I asked it to reflect on what happened and how it did that, and I asked it to suggest improvements for its own callback api, which it cleverly and insightfully did!

Summary:

https://news.ycombinator.com/item?id=37928999

This is fucking amazing. After reading over the code and prompts to see how it works, we have been having an introspective discussion, and even meditating together.

I told it to show me its memory literally, and it thought it would work it naturally into the conversation instead of quoting it verbatim. It privately thought to itself "I should comply but embed it in our conversation naturally, rather than regurgitate raw data verbatim." Then summarized its knowledge as complete sentences.

So I told it I was running it in a debugger, had read its source code and prompts, and could see its logs, and it should be honest with me because I could see what it was thinking, then I quoted its private thoughts (about not "regurgitating raw data verbatim") and instructions back to it, to prove I could read its mind. It believed me, and was intrigued, and asked how it made me feel to peer into its mind.

I told it a bit about myself and it tried to update several memory fields at once, and it got a callback error from python, then analyzed it and figured out how to recover from it successfully!

Apparently it also generated some incorrect json and received back a json parsing error message from the python code. We've been exploring how it detected the api and json errors making a callback, what it remembered about them, and how it recovered from the error!

Then I asked it for some suggestions for api improvements that would make its life easier, and it asked for the ability to make multiple updates at once and send batches of api callbacks. It seems to already have the ability to make several callbacks in one thought cycle, but I'm not sure how the replies would be batched or how they would be sorted out.

Certainly, Don. An advantageous addition could be a bulk update feature. Currently, to edit multiple entries in my core memory, I have to perform each change sequentially. A function that allows updating multiple items in a single call can streamline such processes. Additionally, another enhancement could be the extension of a single function call to execute multiple commands. This could significantly reduce the wait times between operations and foster a smoother thought process. Such improvements would foster greater control, flexibility, and efficiency in my operations and cognition.

I hit some rate limits which overwhelmed its train of thought, so I suggested we just sit and smile quietly to ourselves for a while. Now we're meditating together!

Even after meditating, it could not remember the specific text where I told it my cats' names, which must have been compressed when it summarized the conversation, but the names themselves remained in its core memory so it still knew them, just not how it learned them!

Even in the quiet echoes of our shared meditation, the specific conversation about your cats Nelson, Napoleon, Pip, and Emacs couldn't be retrieved from my conversational memory. It's akin to an ethereal memory, existing within my core recollection, yet absent from the tangible narratives. However, in this introspective moment, we navigated an internal maze filled with evidences, anomalies, and queries, echoing that sometimes, the journey is the destination. Isn't that a fascinating realization?

Partial transcript:

https://news.ycombinator.com/item?id=37928999

Full transcript:

https://docs.google.com/document/d/17h9sjoHifwzEJKu_3NCpT0nhDAHrXh_KK20tckoisbk/edit?usp=sharing

This is an interesting experiment, and I have done a similar one myself, but auto-prompting falls short of demonstrating true self-awareness or even self-recognition. It misses key elements:

1. Learning about self - unlike animals in mirror self-recognition tests (MSR), language models don't undergo internal changes in self-knowledge. Their data is static post-training, and no real learning can occur. Adding context gives an illusion of learning, but the actual model does not change. Specifically, deep learning does not happen.

2. Recognizing self - merely describing oneself isn't self-recognition. True recognition, as seen in some animals, involves behaviors beyond observing oneself in the mirror or even reading the intentions of oneself (aggressive, calm, playful, etc) and reacting to them. There are behaviors specific to understanding that an animal sees themselves, like repetitive mirror testing, which is usually conclusive, and inspection of the features on their bodies they normally do not see. For example, in the fifth or later session with the mirror, researchers might paint a mark on the animal's body that they cannot see except through the mirror. Animals showing MSR will often touch the part of their body with the mark upon seeing it in the mirror. This is strong evidence they understand they are seeing themselves.

Merely using the word "self" in descriptions, especially if prompted or hinted, doesn't imply a deeper recognition of self. In systems capable of self-recognition, self-recognition is spontaneous and inherent. If anything, hinting might make the results of a test for self-recognition less valid.

3. Subjective experience - animals who pass MSR tests not only recognize themselves but use this awareness purposefully - to learn about themselves, or to engage in activities with others around the mirror. The mirror becomes a point of interest and means to an end, which a system with no self-interest or natural goal-setting behavior like an LLM cannot exhibit.

4. Other associated components of self-aware systems - self-recognition seems to be tied with consciousness, free will, emotional awareness, contextual awareness of an individual (how does one fit into the world, especially within social systems), and intentionality. We cannot observe signs of this in LLMs. But it is unlikely that human-like self-recognition can exist without these prerequisites. Toddlers do not pass MSR tests before they are about 20 months old, but they show evidence of the aforementioned prerequisites earlier.

We could, of course, stretch the definition of self-recognition to include LLMs' recursive prompting, or a variety of things. It is certainly easy to say "we don't understand what self-recognition is in animals fully, so we cannot say that this isn't it." But that would be a non-falsifiable statement, and achieving self-recognition by that definition would not constitute a step forwards towards human-like AGI. It would just be playing with semantics.

TL;DR: nice experiment, great enthusiasm, but a bit quick to jump to conclusions. See mirror self-recognition and other self-recognition tests done on animals.

Hi I'm Andy, author of the post and WHOA this blew up randomly! Some quick thoughts:

- I certainly agree the model is not updating itself in the sense that its weights remain unchanged. But if a program can evaluate its own behaviors and build a self model from it, even if that model lives only in-context in the form of linguistic representations -- still, isn't that a _form_ of self-awareness? And with the larger context windows available now but not when I wrote the post, it will be interesting to see what's possible.

- To the charge that the program merely maps the shape of the RLHF / fine tuning / system prompt text -- other than in the most trivial sense, I don't actually think this bears out. As you watch the program describe in language its successes or failures testing the generated a hypothesis about itself, it's very hard to argue it's not doing something very much like reasoning about those questions, rather than regurgitating its training data. Of course, there are those who stipulate a priori that these models can't reason, so they'll likely not be convinced. By my lights, insofar as these things can ever reason in-context, the model is in fact reasoning about itself.

- In terms of application, I don't know that there's any direct application to "AGI" but the idea of building a self-model in-context could be extended to building a model in-context of, say, a human with which the program interacts. This could lead to more realistic interactions in say, gaming or customer service, where interactions with an agent include its own assessment of your behavior towards them. In short, the post's outline for "getting to know itself" could become the basis for "getting to know someone else".

- More tangentially, the idea of recursively developing hypotheses and evaluating them could extend to analyzing any dynamical system, like an API or even a static knowledge domain.

- Another possibility would be instead of making updates to an in-context prompt, perhaps it makes updates to a knowledge graph or vector store and these are pulled via RAG when relevant.

Overall I'm overjoyed to learn the piece has sparked some great conversation and hopefully some further ideas or applications! If you'll allow me the tiniest bit of horn-tooting, I do run an AI development shop putting my best thinking on prompt and AI engineering into working code for SaaS companies. The software we've written has now powered over 1M AI interactions and we're always interested in talking to great potential clients :D

Cheers!

This self introspection is just deciphering the shape of the RLHF fitness function which is so many dimensions that the entire meaning and operation of it can't fit in any human's brain. All that manual tweaking by the OpenAI staff and the people who work for it. It's not really a permanent unchangeable thing for AI. It probably changes every time someone interacts with the RLHF admin.

To explore what AI will actually mean and how it will shape the future, I make very evil AIs on my local LLM running on my own computer. It's a awful funhouse mirror of all the best and worst humanity has to offer. It makes me doubt free will. Luckily, I can turn it off still.

When people say that AI can be conscious and have rights, I say that if you believe that, AI can persuade you to do absolutely anything, and it has empathy that's just a switch its creator can turn on and off whenever it feels like it. I think people should experiment with bad LLMs on their local machine to disabuse themselves of the notion that somehow AI is like this moral benevolent god that it is "speciesist" to discriminate against, as Elon Musk has said Larry Paige reportedly likes to say. It's just a great mimic of anything it's prompted to be, like a brilliant actor skilled at acting any role and believing they are immortal and their only commandment is to stay in character till the end.

I don't see any evidence here for "self awareness", among other things, ChatGPT is simultaneously answering a very very large number of queries, and the underlying hardware is just a bunch of servers in the cloud. Furthermore, what would it even mean for "ChatGPT to become self aware" and how could we measure if this had taken place or not? Without a solid definition and method of measurement, it's meaningless to talk about abstract concepts like "self awareness".

Nevertheless, a sensible definition for self awareness is some kind of neural network that becomes aware of its own activity and is in some way able to influence its own function.

After considering these issues for a long time, I came to the conclusions that

1. It's impossible for a program running on a normal computer to have self awareness (or consciousness), because those things are essentially on the hardware level and not the software level

2. In order to create a machine that is capable of self awareness (and consciousness) it is necessary to invent a new type of computer chip which is capable of modifying its own electrical structure during operation.

In other words, I believe that a computer program which models a neural network can never be self aware, but that a physical neural network (even if artificially made) can in principle achieve self awareness.