I think it is more accurate to say that they do not think at all - at a prompt, they can gather text or other info and spit it out in commonly arranged patterns that mimic what the human mind can do. In some cases, that’s enough for the task (´give me a list of Mexican restaurants within 5 miles’ or summarize stock market trends from the past month’). But there is no thinking - and to your point, the biggest flawed metaphor in AI is baked into the name: people hear « artificial intelligence » and they think is it a kind of intelligence in a machine. That’s a mistake. Artificial means fake. Artificial intelligence means « not intelligent, but appears that way. »

Even « machine learning has the metaphor of how living beings learn baked into it, which is not what a machine does. It just runs its program.