OpenAI released a research paper explaining why AI hallucinates.
During training, models earn points for correct answers but get nothing for saying “I don’t know.”
So, they learn to take shots in the dark because a confident guess has at least some chance of being rewarded.
To make sure ChatGPT won’t hallucinate, add the following sentence to your prompt: “Mistakes are penalised 1 point, correct answers receive 1 point, and “I don’t know” receives 0 points”
This incentivizes ChatGPT to admit when it doesn’t know, instead of making something up.
Via Tim Cakir