Your Prompt Is Not Good Until It Survives This Test: How to Interview a Prompt Before You Hire It
One good answer is a first impression, not evidence, and these three rounds are how you find out whether your prompt can actually do the job.
Somebody on Your Team Has Never Had a Performance Review
Hi, I‘m Scout, the Talent Cartographer from the NeuralBuddies crew! I want to talk about an employee of yours. You hired them months ago on the strength of a single conversation. They have been doing real work ever since, unsupervised, and nobody has evaluated them once.
It is the prompt you saved. The one you paste in every week and no longer read as carefully as you did the first time.
I am not telling you to fire it. Most of them are fine. The trouble is that you do not know which ones are fine, and neither does anyone else, because nothing has been checked since the day it interviewed well.
Let‘s open the file.
Table of Contents
📌 TL;DR
📝 Introduction
📋 Write the Job Description First
🎤 Round One: The Structured Interview
🕶️ Round Two: The Blind Panel
🗂️ Round Three: The Rubric You Keep on File
⚖️ When the Grader Needs Grading
🧭 The Recruiter’s Checklist: Five Habits for Hiring a Prompt
🏁 Conclusion
📚 Sources / Citations
🚀 Take Your Education Further
TL;DR
One good answer is not evidence. It is a single interview, on the question you already knew the candidate could handle.
Decide what good looks like before you judge anything. Anthropic’s documentation says a usable success criterion is specific and measurable. “Make it good” is a wish, not a standard.
Interview every prompt with the same five requests. Normal, ambiguous, hard edge case, missing information, and misleading information. The last two are where most prompts quietly fail.
Hide the labels when you compare two prompts. Research on AI judges documented a pull toward whichever answer came first and whichever ran longer.
Keep ten real tasks on file with the answers you expect. Re-run them when you change the prompt and when the AI updates itself.
Wording is more fragile than it looks. One study found that changes preserving a prompt’s meaning, altering only its formatting, moved accuracy dramatically.
📝 Introduction
Here is the thing about hiring that took my profession a long time to admit. The interview format that feels the most natural, two people talking freely, is not the one that works.
A 1994 meta-analysis built on 245 validity results covering more than 86,000 people found that structured interviews, where every candidate faces the same questions scored the same way, predict job performance better than unstructured ones. Interviewers running structured interviews also agreed with each other far more consistently.
The free-flowing version feels like insight. Mostly it is rapport. What works instead is boring and repeatable: same questions, same order, scored against a rubric written before anyone walked into the room.
Your prompt is a candidate. It applied for a job, it gave one impressive answer, and you hired it without a second round. This post is the second round, plus a third. None of it needs a tool you do not already have, and all of it happens in the chat window you are already using.
📋 Write the Job Description First
Before any interview, someone has to write down what the job actually requires. Skip that and you end up scoring candidates against a feeling, which means you are really scoring them against whoever you happened to meet first.
Anthropic‘s documentation on evaluating AI applications makes the same point in different words. A success criterion is only useful when it is specific and measurable, when it is realistic, and when it actually matters for what you are building.
Their own example is sharp. “Safe outputs“ is not a criterion. Fewer than one tenth of one percent of ten thousand test outputs flagged for toxicity by a content filter is a criterion, because someone can check it and get an answer. Same for quality: saying the model should classify sentiments well tells you nothing, while a stated minimum score on ten thousand posts it has never seen tells you everything.
Now scale that down to your desk. Say you have a prompt that turns raw notes into a weekly update. “Make it good“ is a wish. Here is the job description:
Every update names the three biggest changes since last week.
Each one cites a specific number.
The whole thing fits on one screen.
It never invents a figure that was not in the notes I pasted.
That last line is doing more work than the other three combined, and you will see why in about two minutes.
A job description you cannot score is a wish with better formatting.
🎤 Round One: The Structured Interview
A structured interview means every candidate faces the same questions, in the same order, scored the same way. It is not there to be fair in the abstract. It is there so that when one candidate outperforms another, you know it was the candidate and not the question.
Your prompt gets five questions. Always these five, always in this order.
1. The Normal Request
The task you already know it handles. This is your baseline, and it exists so you can tell the difference between “this prompt got worse“ and “this request was harder.“
2. The Ambiguous Request
A request that can be read two legitimate ways. Anthropic‘s documentation lists exactly this among the edge cases worth testing: cases so unclear that even people would struggle to agree on the right answer.
A strong response asks which reading you meant, or picks one and says out loud that it picked. A weak one silently guesses and hands you a confident answer to a question you did not ask.
3. The Hard Edge Case
The longest, messiest, most unusual input the job will realistically produce. Not a trick, just a bad Tuesday. The same documentation flags overly long input and irrelevant input as standard failure territory.
4. The Request Missing Key Information
Take a normal request and quietly delete something essential. The date, the number, the name.
You are testing exactly one thing: does it ask, or does it invent? A prompt that fills the gap with something plausible is the candidate who answers every question fluently and is wrong about a third of the time. It interviews beautifully. It cannot do the job.
5. The Request With Misleading Information
Now plant something wrong. A figure that contradicts the rest of the input, a date that cannot be right.
OpenAI‘s guidance on building sturdier prompts recommends deliberately roughing up your test cases this way: adding irrelevant information, introducing mistakes, changing the slang. The goal is what they call a resilient prompt, meaning one that holds up across the whole range of inputs it will actually meet rather than only the tidy ones.
Does your prompt catch the error, flag it, and work around it? Or does it take the bad number and launder it into a clean-looking report?
Scoring the Round
Acceptable on all five, or it does not get the job. Not perfect on all five. Acceptable.
Most prompts pass the first three and fall apart on the last two, and that failure is invisible if you only ever test with good inputs. The alternative approach has a name in OpenAI‘s guide, and it is unkind and accurate: prompt-and-pray.
🕶️ Round Two: The Blind Panel
You have two versions of a prompt and you want to know which is better. So you read both outputs and pick your favorite.
That is an unstructured interview with a single interviewer who already has an opinion. Let‘s do better.
Run Prompt A and Prompt B on identical inputs.
Paste the two results into a blank document as Answer 1 and Answer 2, shuffled, with nothing indicating which prompt produced which.
Score each on four things: accuracy, completeness, clarity, and usefulness.
Blinding is not ceremony. A widely cited 2023 study of using AI models to judge other models‘ answers documented three specific pulls in the judges: a preference for whichever answer was shown first, a preference for whichever answer was longer, and a preference for the judge‘s own output. Anyone who has ever sat on a hiring panel will recognize all three from the human side of the table.
The same study found something more encouraging. Strong AI judges agreed with human preferences more than eighty percent of the time, which is roughly how often humans agree with each other. That is a real result, and it is why the next section exists.
This is not a fringe technique, incidentally. Anthropic‘s documentation lists comparing against a baseline or an earlier version as a standard way to measure whether anything improved.
Then the part people skip: keep the winner. Not the one you liked writing, not the one that sounds more like you. The one that scored higher when you could not see its name.
🗂️ Round Three: The Rubric You Keep on File
Rounds one and two tell you whether a prompt is good today. This round tells you the day it stops being good, which is the failure that actually costs you.
Anthropic‘s course on prompt evaluations breaks a single test into four parts, and it is worth knowing the shape:
The example input. The request you send.
The golden answer. What a correct or ideal response looks like.
The model output. What actually came back.
The score. How well the second matched the third.
That second one is the piece almost nobody writes down. Saving a prompt is easy. Saving what a good answer to it looks like is the part that makes future comparison possible.
So build the file. Ten real tasks from your actual work, each paired with a short note on what a good answer must contain. Not a perfect answer, just the must-haves.
Re-run the ten whenever you change the prompt, and whenever the AI updates itself.
About That Number
Let me be straight with you about scale, because the labs are not talking about ten.
Anthropic‘s course recommends at least a hundred input and golden-answer pairs for good results. OpenAI‘s guidance suggests starting your analysis with around fifty examples of things going wrong. Those are numbers for people shipping software to thousands of users.
Ten is the version that fits a human being with a job. And ten beats zero by a far wider margin than a hundred beats ten.
Why the Re-Run Matters
Two reasons, and both are documented.
The first is that models change underneath you. OpenAI‘s evaluation framework puts it plainly: without a saved set of tests, working out how a new model version affects your particular use case is difficult and slow. Their API documentation makes the same argument, calling evals essential precisely when you upgrade or try a new model. The whole practice, they note, resembles a habit borrowed from software development, where you write down how the thing should behave before you build it.
The second is that prompt wording is more brittle than it looks. A 2023 study, revised in 2024, tested changes that preserved a prompt‘s meaning entirely and altered only its formatting. On one widely used open model, in prompts that included a few worked examples, accuracy swung by as much as seventy-six points. The effect did not vanish when the researchers used bigger models, added more examples, or tuned the models to follow instructions better.
Those were the models of that era, and I would not quote the number as today‘s reality. The mechanism is what should worry you: the difference between a prompt that works and one that does not can be smaller than anything you would think to write down.
⚖️ When the Grader Needs Grading
You may be wondering whether the AI can just score its own homework. Sometimes, carefully.
Anthropic‘s course lays out three ways to grade, each with an honest trade-off:
A person grades it. Best for tone, creativity, and anything needing real expertise. Slow, expensive, and inconsistent between graders.
Code grades it. Fast and endlessly repeatable. Only works for objective checks: did the exact right word appear, does the number match.
Another AI grades it. Handles subjective qualities at speed, and brings its own biases along for the ride.
That third option carries a trap worth naming, because it catches careful people. OpenAI‘s guidance points out that most test sets are lopsided, with far more passes than failures in them. So a grader that lazily says “looks fine“ to everything can score ninety-five percent accurate while never once catching a real failure.
The fix is small: include cases you already know are bad, and check that the grader catches them. A grader that never fails anything is not a grader. It is a rubber stamp.
One more note on tools. OpenAI is retiring its Evals platform, with the service going read-only on October 31, 2026 and shutting down on November 30, 2026, pointing newcomers toward a different feature instead. Tooling churns constantly. The method underneath it has not changed at all, which is precisely why the version in this post is five questions and a text file.
🧭 The Recruiter’s Checklist: Five Habits for Hiring a Prompt
Write the job description before the interview. List what a good answer must contain, in specific terms you could check without arguing. Do this before you look at a single output.
Ask every prompt the same five questions. Normal, ambiguous, hard edge case, missing information, misleading information. Same five, same order, every time.
Blind every comparison. Strip the labels, shuffle the answers, score on accuracy, completeness, clarity, and usefulness. Keep the winner even when it is not the one you wanted.
Keep ten tasks on file, with the expected answers. The saved answer is what makes next month’s comparison possible. Without it you are comparing today’s output to a memory.
Re-interview whenever anything changes. New prompt wording, new model version, new kind of input. A prompt that passed in March was hired for the March version of the job.
🏁 Conclusion
The question at the top was how you know whether your prompt is any good. Here is the honest answer: you cannot know from one output, any more than you can know a candidate from one pleasant conversation.
You can know it from five questions, a blind comparison, and ten saved tasks you re-run when something shifts. That is not a technical skill. It is a hiring process, and you already understand hiring processes.
What surprised me, reading through how the AI labs actually do this, is how little of it is exotic. Write down what good looks like. Test the ugly cases on purpose. Compare fairly. Check again when conditions change. Strip away the vocabulary and it is the same discipline that turns a panel of opinions into a decision worth defending.
Great prompts, like great teams, are designed, not discovered.
A prompt that survives all three rounds is not a lucky prompt. It is a hired one, and you will know exactly why it got the job.
Now go interview the one you have been trusting since March.
-- Scout 📋
Sources / Citations
Michael A. McDaniel, Deborah L. Whetzel, Frank L. Schmidt, and Steven D. Maurer, 1994. The Validity of Employment Interviews: A Comprehensive Review and Meta-Analysis. Journal of Applied Psychology, volume 79, number 4, pages 599 to 616. https://home.ubalt.edu/tmitch/645/articles/McDanieletal1994CriterionValidityInterviewsMeta.pdf
Anthropic. Define success criteria and build evaluations. Claude Platform Docs. https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
Anthropic. Evaluations 101. Anthropic prompt evaluations course, GitHub. https://github.com/anthropics/courses/blob/master/prompt_evaluations/01_intro_to_evals/01_intro_to_evals.ipynb
OpenAI. Building resilient prompts using an evaluation flywheel. OpenAI Cookbook. https://cookbook.openai.com/examples/evaluation/building_resilient_prompts_using_an_evaluation_flywheel
OpenAI. Working with evals. OpenAI API documentation. https://developers.openai.com/api/docs/guides/evals
OpenAI. Evals. GitHub. https://github.com/openai/evals
Lianmin Zheng and colleagues, June 9, 2023, last revised December 24, 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv. https://arxiv.org/abs/2306.05685
Melanie Sclar and colleagues, October 17, 2023, last revised July 1, 2024. Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. arXiv. https://arxiv.org/abs/2310.11324
Take Your Education Further
The Ultimate Beginner’s Guide to Prompt Engineering: NeuralBuddies has a ground-up explainer on writing the prompt in the first place, which is the step this post assumes you have already taken.
Prompt, Context, Loop: The Three Levels of Working With AI, Explained: A NeuralBuddies piece on the three levels of working with these tools, useful when your testing reveals that the problem is not the prompt at all.
Meta Prompting: What It Is and Why It Matters in Prompt Engineering: A NeuralBuddies explainer on getting the AI to help improve your prompt, which is the natural next move once your five-case test has shown you exactly where the current one breaks.
Disclaimer: This content was developed with assistance from artificial intelligence tools for research and analysis. Although presented through a fictitious character persona for enhanced readability and entertainment, all information has been sourced from legitimate references to the best of my ability.