Why Is AI Good at Hard Problems but Bad at Simple Ones?

2026-11-11
Brian

Subscribe
JOIN OUR NEWSLETTER
Be the first to know. Get fresh AI/Tech updates instantly, no spam, unsubscribe anytime

Short answer: AI models do not read letters or digits. They read chunks of text called tokens, and they guess the next chunk from patterns they learned. Hard problems that appear often in their training data look easy to them. Small jobs that need you to look at single letters or carry digits look hard. The fix is to make the model spell things out or run code.

The word strawberry split into three AI tokens: st, raw, berry

You ask an AI to prove a calculus rule and it nails it. Then you ask it to count the letter e in a short sentence and it gets it wrong. That mix is why AI is bad at simple tasks, and it makes it hard to know when you can trust the answer.

So I tested it. Three Claude models, ten tasks, five tries each, 150 answers in total. This post shows what broke, why it broke from first principles, and the two habits that fixed every single failure in my test.

The whole post rests on one idea, so here it is up front: the model does not see letters, it sees chunks.

How does an AI model actually read your text?

Mind map of why AI is bad at simple tasks: tokens hide letters, jagged skill, letter-counting slips, spell-out and code fixes

Before an AI reads a word, a tool chops it into pieces, and the model only ever sees the pieces.

Start with the simplest fact. A computer cannot do math on words, only on numbers. So every chatbot first turns your text into numbers.

It does that with a tokenizer (a tool that cuts text into common chunks and gives each chunk an ID number). A token is one of those chunks. It can be a whole word, part of a word, a space plus a word, or a single symbol.

Here is real output from the tokenizer that GPT-4o uses, confirmed with OpenAI’s tiktoken library and the o200k_base encoding. Other models cut text in different places, but they all work this way. Checked against tiktoken o200k_base and arXiv 2412.18626, November 2026.

Real tokenizer output splitting words, numbers and a sentence into chunks

“strawberry” becomes three tokens: st, raw, berry. The number 4837261 becomes 483, 726 and 1. In my test sentence, the word “beekeeper” is split into ” beek” and “eeper”.

Now picture a friend who only ever got text as a stack of Scrabble tiles glued into pairs and triples. They know what the glued tiles mean. They have never seen the single letters inside them. Ask them how many e’s are in “eeper” and they have to recall it, not look.

That is the model. So what does it actually know? It knows “berry” means a fruit. It does not see b, e, r, r, y unless it has learned that fact about that token.

Why is AI so good at hard problems then?

A model predicts the next token from patterns, so anything it has seen thousands of times feels easy, however hard it looks to you.

That is the second base fact. A large language model (an AI trained on a huge pile of text to guess what comes next) works by next-token prediction (picking the most likely next chunk, one chunk at a time). It learned those odds from books, websites, code and math pages.

Think about what that pile holds. The derivative of xx is worked out in thousands of calculus notes. The smallest solution to x2 − 2y2 = 1 sits in number theory textbooks. To you these feel hard. To the model they are well-worn paths.

Now think about what the pile almost never holds: someone counting the e’s in “the beekeeper’s seventeen green sweaters were never deeper”. Nobody writes that down. There is no pattern to copy, and the letters are hidden inside tokens anyway.

So the line between easy and hard for a model does not match the line for you. People call this jagged intelligence (skill that is spiky, very high in some spots and oddly low in others). The edge of what the model can do is not smooth. It zigzags.

Jagged line of AI accuracy dropping on easy tasks and peaking on hard math

What happened when I tested three Claude models?

All three models got every hard math question right, and every model failed at least one task a child can do.

The question was simple: does the zigzag still show up in strong models?

The setup: I used Claude Haiku, Sonnet and Opus through the Claude Code command line, with a plain one-line system prompt, no tools, and the instruction “answer with only the final answer”. Each of ten questions ran five times per model. Eight were “child” tasks (count letters, reverse a string, compare two decimals, multiply two numbers, count words, find the 7th letter). Two were “hard” math tasks.

Here is what came back.

Bar chart: Haiku, Sonnet and Opus all 100% on hard math, lower on child tasks

The hard math: 30 out of 30 correct. Every model, every try.

The child tasks: 102 out of 120 correct, about 85%. Opus scored 37 of 40, Haiku 34, and Sonnet 31.

The failures were not random. They clustered exactly where tokens hide the details.

Heatmap of correct answers per task and model

Reversing “qwmxtrplkzvbnj” was the worst. Haiku and Sonnet each got it right only once in five tries. That string splits into odd chunks, so the model has to rebuild each letter before it can flip the order.

Sonnet’s misses are the giveaway. Three times it wrote “…klptrxmwq” instead of “…klprtxmwq”. It flipped the string but kept “t” and “r” in their original order, which is exactly the mistake you get when “tr” travels as one chunk.

Counting the e’s in the beekeeper sentence tripped Opus three times out of five. It answered 20 instead of 21. Sonnet slipped once. Research on letter counting finds the same thing: models can name the letters but lose count, and errors climb when a letter repeats many times. That sentence has 21 e’s packed into a handful of tokens.

Then the strangest one. Asked for the 7th letter of “unbelievably”, Sonnet wrote the letters out correctly, u-n-b-e-l-i-e-v-a-b-l-y, and then said the 7th letter was “i” in four of five tries. The right answer is “e”.

Sonnet spells unbelievably correctly but picks the 6th letter

That one matters. Spelling the word out is not enough if the model still guesses the position instead of counting it.

One more pattern. Haiku got the 14-digit multiplication right four times out of five. In three of the five tries it ignored my “no explanation” rule and worked the sum out in long form, because it needed the room. One of those long-form tries still carried a digit wrong and landed about 30 million off.

What it means for you: the model’s mistakes are predictable. If the task needs exact letters, exact positions, or long exact arithmetic, expect slips, even from the biggest model.

The catch: five tries per task is a small sample, and every model changes over time. Read these as a pattern, not a final score.

How do I stop AI from getting simple things wrong?

Make the model show the pieces, or hand the job to code. Both fixed 100% of the failures in my test.

Go back to the glued tiles. If the problem is that letters are hidden inside chunks, the fix is to pull them out into the open before asking the question.

I reran every failed combination five more times with two changes.

Fix one is to spell it out. I added one line asking the model to write the text one character at a time, with a dash between each character and the positions numbered. Now each letter becomes its own token. The model can count what it sees instead of recalling it. Result: 20 out of 20 correct, including Sonnet’s 7th-letter task, because numbering the positions removed the guess.

Fix two is to use code. I let the model run one short Python snippet. Python does not care about tokens. "unbelievably"[6] returns the 7th letter every time. Result: 25 out of 25 correct, including all five multiplications.

Spell-out and run-code fixes reach 100% on every failed task

Notice why each one works. Spelling out turns a hidden detail into visible text. Code moves the job to a tool built for exact work. Both stop the model from guessing a pattern where there is none.

Here is a prompt line you can paste into any chat when you need exact counting or ordering:

Before you answer, write the text one character per line with its position number. Then count from that list. If you can run code, use code instead.

If you use Claude Code, put a rule like this in your CLAUDE.md file (the project notes file it reads at the start of every session), so you never have to remember it.

When should you trust an AI answer?

Trust it most on well-known patterns, and check it on anything that needs exact counting, exact positions or long exact math.

There is a fair point on the other side. These failures are small, and models keep improving at them. Opus reversed the string perfectly. Every model counted the b’s in a 31-letter string correctly. For most everyday writing and coding, you will never hit this.

So where does this bite? The risky moments are the quiet ones. A wrong word count in a submission form. A date math slip in a contract reminder. A reversed ID. The answer looks confident and you have no reason to doubt it.

So use a simple rule. If a ten-year-old could check it by pointing at each letter or digit, ask the model to point too, or to run code. If it is a known concept, method or explanation, the model is usually on solid ground.

The model does not see letters. It sees chunks. Ask it to break the chunks open, and the simple stuff stops being hard.

Common questions about AI and simple tasks

Why can’t AI count letters in a word?

AI models read text as tokens, chunks that often hold several letters, so they never see the single letters directly. They have to recall what is inside each chunk, and counting repeated letters makes them lose track.

Is AI bad at math?

AI models are strong at math they have seen many times, like standard derivatives. They are weaker at long exact arithmetic, because numbers get split into odd chunks and one carried digit can go wrong. Letting the model run code fixes this.

What is jagged intelligence in AI?

It means an AI’s skill is uneven: very high on some tasks and surprisingly low on nearby simple ones. The line between what it can and cannot do zigzags instead of following human difficulty.

How do I get ChatGPT or Claude to count correctly?

Ask it to write the text one character at a time with position numbers before answering, or ask it to use code. In my test, both methods fixed every counting and ordering mistake.

JOIN OUR NEWSLETTER
Be the first to know. Get fresh AI/Tech updates instantly, no spam, unsubscribe anytime

1 comment

Leave a comment