Private skateboard-trick naming results
| Model as named in the transcript | Reported result |
|---|---|
| Gemini 4 Argan | 100%, first perfect score on Theo's private Skatebench. 7:43 |
| Opus 5.5 Max | 79%. 8:14 |
| "3.8 Flash" | 96%; name preserved from the transcript. 8:14 |
- Skatebench tests naming skateboard tricks, including switch and fakie stances. Theo describes these stance distinctions as a grammar that makes the task difficult for models.
8:14 - Gemini answered quickly without excessive reasoning-token use on this task, unlike his broader experience of the model's token consumption.
8:53 - Early API instability required changes to his benchmark runner to handle endpoint errors. He does not know whether reliability improved after that testing.
8:33 - His lesson is task specificity: a perfect result is meaningful for comparable work, but does not establish reliability on unrelated tasks.
8:53
Vision strengths and untested possibilities
- Theo describes historical Gemini strengths with documents, PDFs, video, audio, and image understanding, giving motherboard-component labeling and overlays as an example. He explicitly did not benchmark those broader vision tasks for this review.
9:209:47 - He suggests evaluating Gemini on complex vision work rather than assuming his coding verdict transfers to those tasks.
9:47 - He has heard positive reports about internationalization, localization, and multilingual work, but has not seen much of that capability in his own tests. These remain suggestions for further evaluation.
16:1016:37