Jev vs Jev
Jev came out a couple of weeks ago. It's an AI model from TypeSafe AI that doesn't write anything. You give it a situation and a list of options, and it gives each option a probability. It's built for decisions inside software, like working out which team a support ticket should go to.
A chess move is a choice from a list too. No position in chess has more than 218 possible moves, and Jev takes up to 255 options, so it fits. I gave two copies of Jev a board and had them play each other.
I also brought in Stockfish, the strongest free chess program around, to grade how they were doing. Five games and 556 moves later, with a total bill of 2.8 cents, my summary is this: Jev plays like someone who has watched a lot of chess and never thought about any of it.
How it plays
For every move, Jev gets one question. I describe the board in text, list the moves so far and tell it which side it's on. I also gave each side a one-line personality, "aggressive attacker" for White and "careful, solid player" for Black, mostly to see if it would change anything. I never did work out whether it did.
Then it gets every legal move as an option, each described in plain words like "knight takes pawn", and sends back a probability for each one. That's it. A normal chess program looks ahead, trying millions of possible futures before it chooses. Jev looks at the board once and scores each move on how it looks.
Jev only gives probabilities, so something else has to pick the move. The program does it with weighted dice, and a setting called temperature controls how much it trusts Jev's favourite. At 0 it always plays the favourite. At 1 it follows Jev's odds exactly. Three of my games were at 0.5, with one at 0 and one at 1 to compare.
Stockfish grades every move by how much it changed that player's chances of winning. Throw away a lot and it's a blunder. A bit less and it's a mistake, or an inaccuracy. Chess websites use the same kind of grading when they review your games.
Each move cost about $0.00005. TypeSafe had paused signups, so I went through Vercel's AI Gateway instead.
A good start
Every game opened with the most popular first move in chess, and Jev was about 90% sure of it each time. In the first few moves, its favourite was also Stockfish's pick 44% of the time. For something that isn't thinking at all, that's not bad, and it has obviously seen a lot of chess. It goes downhill from there.
Then it gives everything away
The game at temperature 0 shows it best, because there Jev always got its favourite move. On move 4, White gave away a knight for a pawn. Two moves later it gave away its queen, the strongest piece on the board, for a much weaker one. Black blundered in between, so for exactly one move White was winning. Stockfish flagged four moves in a row as blunders, and every one was Jev's first choice.
Over all five games, Jev blundered 50 times, roughly once every 11 moves. My favourite is from game 1. White had a completely winning position and threw it away with one queen move that attacked the king. That was Jev's first choice too.
It really likes attacking the king
In chess, attacking the king is called a check, and the other side has to respond to it right away. Checks look exciting, and so do moves that capture a piece. Jev goes for both far more often than it should.
| Share of the possible moves | Share of Jev's probability | |
|---|---|---|
| Checks | 11% | 47% |
| Captures | 13% | 51% |
About one in nine of the available moves were checks, and they got nearly half of Jev's probability. That's roughly four times their share, and captures got about four times theirs as well.
It makes sense once you think about how Jev works. You can see that a move is a check just by looking at it. Whether it's a good check depends on what happens afterwards, and Jev never looks at afterwards.
A queen against a lone king
Sometimes a game gets down to almost nothing: one side has a king and a queen, the other just a king. It's one of the first endings beginners learn to win. You use the queen to push the other king into a corner, walk your own king over, and finish it. Most of those moves don't attack the king at all. You're slowly taking squares away until it has nowhere to go.
Jev got there twice and won neither time.
In game 5, Black ended up with a queen against a lone king around move 62, and according to Stockfish the win from there was guaranteed. Black then spent 23 moves chasing the king around the board. Fourteen of those moves were checks, and on average Jev put about two-thirds of its probability on checking moves. The white king kept stepping out of the way. Eventually the queen gave check from the square right next to the king, the king took it, and with two bare kings left, the game was a draw.
Game 3 ended the same way, only sooner. Seven moves in, the queen checked from next door and the king took it.
In fairness, neither losing move was Jev's favourite. The dice picked its fourth choice in one game and its sixth in the other. But its favourites were checks as well. I don't think it ever had a plan beyond "checks are good".
It knows when it doesn't know
This was the bit I didn't expect. Across all the moves, Jev's favourite matched Stockfish only 31% of the time. But it wasn't wrong evenly. It was mostly wrong when it seemed unsure.
| How sure Jev was of its favourite | Moves | Stockfish agreed |
|---|---|---|
| Under 20% | 109 | 11% |
| 20–40% | 285 | 24% |
| 40–60% | 111 | 47% |
| Over 60% | 51 | 76% |
When Jev spread its bets across lots of moves, it was nearly always wrong. When it put most of its weight on one move, Stockfish agreed three times out of four.
That's more useful than it sounds. Nobody should use Jev to play chess, and that isn't its job anyway. In a real app you'd want exactly this behaviour: trust it when it's confident, and send the unsure cases somewhere else. The caveat is that this is five games, so I wouldn't lean too hard on the exact percentages.
The numbers
| Games | 5 |
| Moves chosen by Jev | 556 |
| Jev's favourite was Stockfish's choice | 31% |
| Stockfish's choice was in Jev's top three | 57% |
| Blunders | 50 (one every 11 moves) |
| Results | 2 wins, 3 draws |
| Total cost | $0.028 |
I got one thing wrong going in. I thought the temperature 0 game would get stuck repeating itself, since the same position always gets the same move. It ended in checkmate after 20 moves instead. The only game that did end by repeating moves was one of the 0.5 games.
The hardest part of the whole project was getting answers at all. Jev is in early access and busy, and recording five games meant sitting through more than 500 "try again later" responses.
Watch it yourself
All five games are on chess.victorarowo.com. You can step through any game and see which moves Jev was considering, how likely it thought each one was, and what Stockfish would have done. You don't need to know chess to follow it. There's also a button to start a new live game, but with the early-access limits, expect to wait a bit between moves.
If you only look at one, make it game 5 from around move 62, where Black has a queen against a lone king. It's the most patient 23 moves of not winning I've ever seen.