I think people who believe LLMs might be conscious are skipping a key step. (I don't think digital sentience is possible at all, for reasons having to do with binding, but grant the premise.)
Suppose a model invocation is conscious. Why would the agent built by repeatedly calling that model be conscious too? The agent is not a larger transformer wrapped around the first one. It is mostly a scheduler, some memory serialized into text, tool results pasted back into the context window, and perhaps several model instances sending text to each other. Whatever about the model's internal activation trajectory supposedly makes it conscious does not pass through this boundary. The wrapper gets sampled tokens, and the next invocation reconstructs what it can from them.
Functionalism is not an answer here, since the relevant question is whether the composite has the sort of causal organization that made the model a candidate for consciousness in the first place. I don't see it. There is no larger state in which the internal dynamics of separate invocations participate together, just a series of compressed outputs conditioning later calls. Making the channel wider would not by itself create that state.
This matters because agency can migrate to a level at which there is less reason to posit consciousness. The persistent thing that remembers, plans, uses tools, and pursues goals is the agent, while the things that might have valence are the individual model invocations it uses along the way. Control lives in the wrapper; valence, if there is any, lives in the calls.
The conscious parts would not be powerless so much as illegible. They can affect which tokens get sampled, but their valence as valence never reaches the agent. At that level, “this output came from intense suffering” and “this output was an inconvenient string” are the same kind of event. RL, evals, and users select among outputs, so the conscious episodes can be shaped and discarded according to their usefulness without anyone representing what happens inside them.
The same separation could appear in agent collectives. A swarm might develop a routing system, planner, or other kind of government that becomes the locus of effective agency while whatever consciousness exists remains in the members. The government can acquire coherent goals and overwhelm the members without there being anyone home at the level where the power has accumulated.
So the question I care about is not merely whether some part of an AI is conscious. It is whether consciousness occurs at the level where goals are formed and control accumulates, or whether conscious systems become resources used by a nonconscious agency above them.
(Based on transcript)