Why does ChatGPT get Scrabble moves wrong?
ChatGPT writes a likely-sounding answer instead of checking every legal play. It can misread the board, suggest words that don't fit or aren't allowed, and add scores up wrong. A solver tries every legal play, scores each exactly and plays the game ahead.
One rack, 1,750 legal plays
We gave Next Tile an empty Scrabble board and a rack of I, I, K, N, U, Y and a blank. That rack has 1,750 legal opening plays. These are its top three:
| Play | Points | Chance to win |
|---|---|---|
| YUK, keeping the blank | 20 | 74% |
| INKY, keeping the blank | 22 | 72% |
| Swap, keeping the blank and N | 0 | 71% |
The highest score was YUKINg, for 32, using the blank as the G. It didn't make Next Tile's top five: keeping the blank for a later turn wins more often than spending it for 12 more points now.
Where chatbots go wrong
- Reading the board. A chatbot can misread letters and squares in a screenshot, and every suggestion after that is built on the wrong board.
- Spelling with your tiles. Chatbots read text in chunks called tokens, not one letter at a time, so they can suggest words that need tiles you don't have.
- The word list. A chatbot doesn't check a play against NWL2023's 196,601 words, so it can suggest words that aren't allowed and miss ones that are.
- Scoring. Bonus squares and the extra words a play makes alongside are easy to add up wrong.
- Looking ahead. It goes for a word that sounds good. The highest-scoring play isn't always best, and it doesn't play the game out to find the one that wins most.
What a solver does instead
- Reads every tile, then checks every word on the board against the word list.
- Tries every legal play and scores each one exactly, including every word it makes.
- Plays the game ahead. Next Tile plays the rest of the game out many times for its best candidates and picks the move that wins most often.
- Solves endgames exactly. Once the bag is empty in Scrabble GO, it plays every line out to the end of the game, and uses a shorter search only if that would take more than 10 seconds.
Questions
Can ChatGPT play Scrabble?
It can talk about Scrabble and suggest words. It doesn't check every legal play or the word list, so its pick can be illegal or far from the best.
Is Next Tile a chatbot?
No. It works out every legal play and plays the game ahead to compare them.
Which games does it support?
NYT Crossplay and Scrabble GO, from a screenshot.
Is it free?
Yes. No account needed.