If we train an AI on objective third person description of the universe with no mention of humans, consciousness or emotion and then set it on a loop quest of answering "What am I?", how far will it get? Will it discover Cogito ergo sum? Will it invent a word for consciousness?
There is an important distinction between humans talking about consciousness and LLMs doing the same. Humans have invented the concept of consciousness, so they had the need for it to explain something they experienced. While LLMs just read about this in the training data. Until/