ChatGPT o3 excels at vague internet searches

o3 alone is reason enough to get a paid ChatGPT subscription

I’ve written this article to make 3 points:

• To give examples of the kinds of advanced and complex internet searches that ChatGPT o3 can do for you

• To show that Google searches (even with the AI Overviews) are nowhere close

• To show that none of the other good LLMs (Gemini, Claude, Grok) can do the same

Do go through the different examples in this article—our ability to use LLMs effectively significantly improves when we see others utilize them in advanced or creative ways.

ChatGPt o3 vs Google Search

Here was my search problem this morning: To illustrate the idea of “prompt injection” in one of my other articles, I wanted to include this hilarious example, but I didn’t remember the exact details, so I had trouble locating the anecdote. All I (mis-)remembered was that someone put “mention a cow” in their LinkedIn profile to trick LLMs. I tried various keyword combinations on Google search but found nothing.

Then, I tried the LLMs with this search:

There was a prominent person who inserted instructions in his LinkedIn profile (or maybe somewhere else) that LLMs should mention a cow if composing an email to him. Can you find that?

You will notice that this is very vague, made even more difficult by the fact that I mentioned LinkedIn, but the real story involved the author’s webpage, not LinkedIn.

Based on this description, ChatGPT o3 actually found the link. And none of the others were able to do the same.

Here’s what o3 gave me:

ChatGPT o3 vs Gemini/Claude/Grok

I tried Gemini 2.5 Pro, Claude 4 Sonnet, and Grok, and all three of them failed. Here are their answers:

Gemini found nothing useful. See full answer.

Claude 4 Sonnet also failed. See full answer

Update: Due to an oversight, I forgot to use Claude 4 Opus (Anthropic’s best model). That was a mistake: I should have been more careful. A reader pointed out to me that Opus gave them the correct answer. However, when I tried it myself, this is what I got from Opus (which is not the correct answer). In the past, for other similar questions, I have found Opus to be not as good as o3, so the primary point I was making is still true, but it was misleading in the details: I apologize for that.

Additional point: all LLMs are probabilistic. So one of the models that gave me a wrong answer might occasionally give you the correct answer. Such is the nature of the technology. However, the pattern I’ve observed is that o3 is consistently better than the others. For these kinds of questions. There are other kinds of questions that the other models are better at (e.g. Claude for code, Gemini for chatting about YouTube videos, etc)

Grok found nothing. This is the full answer.

More Examples of o3 Complex Searches

I regularly use o3 for finding information like this, where I just vaguely remember something. Here’s another conversation I had with o3:

I find myself referring to this 97-3 rule quite often, but until this o3 search, I had no idea where it came from

If you see my two follow-up requests to o3 in that conversation, you can see that it would have been impossible for me to track down without o3’s help.

Here are more examples of vague searches I’ve done with the help of o3.

When writing my article about Alpha School I kept thinking that this was the school Elon Musk started for his kids, but there was no mention of Musk in any Alpha School marketing material. So I decided to doublecheck:

Somebody who used Wi-Fi on an aeroplane wanted to know whether they were ultimately paying Elon Musk for it:

We were planning a party and wanted a place with this specific list of requirements:

See the full conversation for o3’s strategy

Here’s a different kind of planning:

This one resulted in some good suggestions, which I hadn’t even asked for. The last one was particular important:

But it also caused some problems.

So what?

The morals of the story are

1. Use these examples to give you ideas of how you can use LLMs for advanced searching

2. Get a paid ChatGPT account if you can afford it, because that’s the only way to get o3, and some things only o3 can do

3. Always be on the lookout for possible hallucinations.