I had a thought that people who are not familiar with AI might assume it is built the way normal software is built. The engineer writes the rules, tests them and adds features one by one until the product is done, but that is not what happens at all.
Building an AI is more akin to running experiments than building a product, and only keeping the one that survives at the end. Think of it like growing a plant. You pick the seed, prepare the soil and water it, but you don't decide the shape of the branches. You can only plant it and you watch the result of your efforts.
In the case of AI, engineers set up the conditions: how big it will be, what it can read, and how much compute power to throw at it. Then they press start on a process that can cost millions and run for months. Nobody writes or decides the AI's answers during this time. The AI is teaching itself by reading, and the engineers are just watching outputs to see if it's learning.
When something goes wrong (the AI stops improving), they can't open it up and repair the broken part. There is no broken part to find. Instead, they change the recipe, go back to an earlier save point, or throw the whole thing away and start over.
So the AI you end up talking to is the survivor with a graveyard of failed attempts behind it. There is no one who sat down and decided on its personality or its answers. The engineers built the process that produced it but not actually the thing itself.
The finished AI is just billions of numbers (we call them weights). Not rules, not sentences, not code a person can read. Just numbers that somehow, together, can hold a conversation. No human wrote those numbers, and no one can decipher them (there are researchers whose whole job is to figure out what's going on inside and they're making progress, but nobody fully understands how these things work). We can see what an AI does, but we mostly can't explain why it does what it does.
This is why AI misbehaves so differently from normal software. When a normal app has a bug, someone finds the bad line of code and fixes it. When an AI confidently tells you something wrong, there is no bad line to fix. We can put rules around it, but we cannot reach in and correct the mistake directly, because the mistake isn't written down anywhere. It's a behavior that emerged from the training process, same as everything else the AI knows.
So "built" is the wrong word, and grown might be a better word.