Keeping Up With AI Without Chasing It

Four labs shipped flagship models in three weeks this September. Anthropic, Google, OpenAI and xAI, the last one landing three days before I stood up to talk about it. Every one of them came with a benchmark chart showing it in front.

If you tried to keep up by reading the announcements, you would have finished the month knowing four model names and not one thing you should do differently on Monday.

I changed one thing in my stack across all four launches. Not because I was not paying attention, but because I was, and three of them did not earn a change. The useful skill is not tracking releases. It is having something that turns a release into a decision, including the decision to ignore it.

Here are the two blocks I use. Both work in whatever AI tool you already have open.

Block one: the weekly scan

The point of a scan is not to summarise the news. It is to tell you the handful of releases that touch the work you actually do, and to be explicit when the answer is nothing.

Mine takes about twenty minutes a week. It produces four statuses and nothing else:

That last row is the one people leave out, and it is the one that does the work. A release you have not explicitly skipped comes back every time someone mentions it. Writing down why you passed is what keeps the list quiet.

The prompt

Paste this into any assistant with web access. Fill the two bracketed sections once, save the whole thing, and run it on a rhythm that suits you. Weekly is plenty. Monthly is fine.

Search for AI model releases from the major labs since [DATE OF YOUR LAST SCAN]. Prefer each lab's own announcement over coverage of it, and note the release date and the price where it is published.

My situation: [WHAT YOU USE TODAY, AND FOR WHAT. For example: ChatGPT for drafting and research, Gemini in my email and docs, and I pay for one seat. My work is mostly writing, analysis and client prep.]

For each release, do not summarise it. Give me one of four verdicts, with one sentence tied to my situation above:

• RUNNING, meaning I should switch to this now, and what I stop using
• TRIALLING, meaning it is worth one bounded test, and what specifically the test has to prove
• WAITING, meaning it is real but nothing in my situation calls for it yet, and what would change that
• SKIPPED, meaning it does not apply to me, and why, so I can stop reconsidering it

Then finish with a single line: what, if anything, I should actually change this week. If the honest answer is nothing, say nothing and do not pad it.

Two rules. If a price or a benchmark number is claimed but you cannot find it on a primary source, mark it unverified rather than repeating it. If your search comes back empty, tell me your search came back empty. That is a gap in the scan, not a quiet month.

Those last two rules matter more than they look. The first time my own scan came back with nothing, it was not because nothing shipped. It was because the sites it relied on were unreachable. An empty result that presents itself as good news is worse than no result at all.

If you use Claude Code

I run this as a saved skill rather than a pasted prompt, so it also updates a one-page board and keeps the history. If you work in Claude Code, the skill version is in the appendix at the bottom. If you do not, you are not missing anything important. The prompt above is the same thinking.

Block two: the twenty-minute comparison

The scan tells you what is worth testing. This block is how you test it without it becoming a project.

Most tool debates run on vibes. Someone read a benchmark, someone else had a bad afternoon with it, and the loudest person wins. You can settle it properly in about twenty minutes, and the result is worth more than any chart because it is measured on your work.

The discipline is simple. Same task, same context, same day, judged on criteria you wrote down before you looked at the output.

The prompt

Run this once per tool you are comparing. Keep the bracketed parts identical across all of them, and paste the results into a single document at the end.

I am comparing AI tools on a real task from my own work. Do the task below exactly as specified. Do not ask me clarifying questions, and do not explain your approach. Produce only the finished output.

The task: [PASTE A REAL TASK YOU DID THIS MONTH. Not a puzzle and not a test. Something with your actual constraints in it.]

The context you should use: [PASTE THE SAME BACKGROUND YOU WOULD NORMALLY GIVE. The brief, the audience, the document, the examples of what good looks like.]

What finished looks like: [LENGTH, FORMAT, TONE, AND ANYTHING THAT WOULD MAKE IT UNUSABLE.]

Then score each result yourself, on a sheet you fill in before you start:

Two things to protect the result. Score the outputs without knowing which tool made which one, if you can manage it. And write the criteria down before you run anything, because a criterion invented after you have seen the output is just a reason for the answer you already had.

What a good test actually asks

The mistake is testing everything at once. A comparison is only useful when it has one question.

When I looked at Grok recently, the honest version of the question was not "is it good." It was narrower than that: the claim worth testing is cost on high-volume, low-judgment work, not quality on hard problems. So the test is one real batch of repetitive work, run on both, compared on cost per task and a fixed quality score. Until I run that, it stays off my recommendation list, and saying so is more useful than an opinion I have not earned.

Write your question down in that shape before you start. "Is it better" has no answer. "Is it cheaper than what I use today on the repetitive half of my week, at quality I would still send" has one, and you can have it by lunchtime.

What to do with the answer

Whichever way the comparison lands, save it. The result, the date, the task you used, and the criteria. Three months from now the next release will reopen the same question, and the only thing that will save you from re-litigating it is a note from the version of you who actually ran the test.

That is the whole practice. A scan that is allowed to say nothing, a comparison narrow enough to finish, and a written record so neither one has to be repeated.

Appendix: the Claude Code skill

Save this as .claude/skills/model-radar/SKILL.md in any project. Run it with /model-radar.

---
name: model-radar
description: Weekly model-release scan filtered to my actual stack. Produces adopt/trial/wait/skip verdicts, not news. Run weekly or when you hear of a release.