ChatGPT Is a Big Fat Liar: Do Not Trust It

Casually slipping in lies in a sea of accurate and irrelevant facts

If you’ve been using ChatGPT regularly, you would already know this, but I feel it is important to keep repeating this. ChatGPT’s output cannot be trusted and you must always find ways to doublecheck that it is not pulling a fast one over you. On a daily basis, I find something in my use of ChatGPT that is incorrect or misleading. As new versions of ChatGPT (and other LLMs and Generative AI tools) come out, this might get better; but at the same time, as we start using these tools for more and more complex things, the errors will get more subtle and dangerous.

We need to become experts in evaluating and verifying the output. As of today, we have all become managers of a very knowledgeable, very fast, very obedient, but very unreliable intern. We have to figure out how to increase our productivity in spite of the fact that the intern tends to make errors or make up stuff.

What can it lie about?

Everything. Someone described ChatGPT as mansplaining-as-a-service: it will confidently answer any question irrespective of whether it really knows the answer or not. If you ask for clarifications, it will come up with supporting arguments and references which are also made up.

• Ask it to write a program and it writes a very good-looking program which contains subtle bugs. (I wrote about this in detail yesterday.)

• It can make up fictitious people, fictitious papers and books and reports, and it can make up fictitious quotes or claims by real people. It can sound very convincing when doing this.

• Ask it about a research topic and there is a good chance that it will just make up stuff that is not true. The scariest parts are 1) it provides references 2) the explanation is plausible enough that even experts start doubting themselves until they doublecheck the references and find out that a) the referred papers/journals/books do not exist, or b) they exist but don’t say what ChatGPT claims. See this Twitter thread about a researcher who asked ChatGPT questions about the topic in which she did her PhD and it gave wrong/bad/made-up answers that made her doubt her own expertise.

• Here’s a Princeton professor talking about how it gave answers to one of his exams:

  I had to read some of the answers three times before feeling confident I was reading nonsense

  This is from a professor who does this for a living!

• ChatGPT invented a sexual harassment scandal and named a real law prof as the accused

What should you do about this?

Be aware that ChatGPT is a liar and assume that there is a 10% chance that any answer it gives might be wrong in some way. Only use it in contexts where:

• You can check the truth of the answer independently

• When the truth doesn’t matter (for example, in fiction writing, or art, or politics)

If you use ChatGPT regularly, over time, you will develop an intuition of what kinds of questions it will answer accurately and when it is likely to bullshit.

But, always remember to ask yourself: What’s the worst that can happen if there’s an error in this?

P.S.

Hindi movie buffs are encouraged to see how many errors you can find in this ChatGPT answer

While on the topic of Hindi movie songs: in a discussion I was having with some friends, one of them started talking about an old song in which the actress Sadhana is singing a song and the hero walks in. She interrupts the song and tells him to wash his hands and face, and then continues the song. My friend couldn’t remember which song this was, so of course I asked ChatGPT.

Ask yourself this: If you don’t know the answer to this question, what would you do? What would any reasonable person do? Wouldn’t you say “I don’t know”?

But not ChatGPT. It can answer anything.

Needless to say, the song isn’t Jumka Gira Re, Sadhana doesn’t interrupt this song to tell anyone anything, and Rajendra Nath isn’t even there in this move or the other movie we’re talking about. But do you see how, with all the extraneous details, ChatGPT is giving the impression of an authoritative answer? 9 out of 11 details are accurate, and 2 incorrect ones (including the only one we care about) have been slipped in casually like I used to do when playing a game of bluff.