Project overview

After developing MailSage, an AI-powered Gmail companion application, I wanted to explore a different application of AI: whether a reusable set of instructions could help an AI model (the underlying system producing an answer, such as GPT-6) understand existing software more accurately, explain its behaviour and identify implementation defects—bugs or errors in how the software works.

I developed the MailSage Code Explainer, an AI agent skill: a reusable instruction guide that tells an existing AI model how to approach a particular task. It was designed to provide structured, beginner-friendly explanations of JavaScript source code—the programming instructions, written in a language used to build parts of MailSage, that determine how the application behaves.

What began as an experiment in improving code explanations evolved into a more systematic investigation of AI evaluation. Across 26 recorded evaluations (Tests A–Z), I investigated whether the AI could discover and use the skill, refined scoring rubrics (marking guides defining what earns points), compared responses with and without the skill, and developed synthetic benchmarks: standardised challenges built from artificial code examples containing deliberately introduced defects.

I also created a reference implementation (a separate version of the code intended to demonstrate how the software should behave) and automated tests to check it.

The record includes seven paired comparisons, in which two AI runs tackled the same task under different conditions. The early and later tests used different tasks and marking criteria, so their scores cannot be combined into one overall performance measure. The final phase involved four controlled blind evaluations: the AI reviewed the same defective code without seeing the answer key, while I varied whether it received the skill and whether it used Medium or High thinking effort (a setting governing how much reasoning effort the model allocates to a response).

All four final evaluations identified all eight deliberately planted, or seeded, defects, regardless of whether the skill was supplied.

Consequently, the results did not demonstrate an additional defect-detection benefit from the skill on this particular benchmark. However, the experiment produced a reusable agent skill, a tested synthetic benchmark and a documented evaluation methodology.

Key outcomes

Section 001: The motivation

From building with AI to evaluating AI

During the development of MailSage, AI-assisted programming tools became an important part of my workflow.

I used them to understand unfamiliar code, investigate errors, explore implementation approaches and troubleshoot application behaviour.

However, I had also encountered situations where their suggestions appeared plausible but were technically inappropriate, incomplete or potentially destructive.

This raised a broader question:

Could I create a reusable set of instructions that would help an AI model analyse source code more reliably, rather than relying exclusively on its default behaviour?

I was particularly interested in whether a structured approach could improve technical accuracy, encourage explanations grounded in the actual source code, and make complex engineering concepts easier for a beginner to understand.

MailSage provided a useful testing environment because I already understood how its different components worked together—its software architecture—and had practical experience investigating its behaviour.

Section 002: Designing the agent skill

Turning a repeatable workflow into instructions

An AI agent skill is a reusable set of instructions that guides how an AI system approaches a particular category of work. It does not create a new AI model: it is more like giving an experienced assistant a written checklist for how to carry out an assignment.

For this experiment, I created a SKILL.md file written in Markdown, a plain-text format that uses simple symbols to organise headings, lists and links. The file contained instructions for analysing and explaining MailSage's JavaScript source code.

The skill encouraged the model to:

• Identify the overall purpose of a file.

• Trace the sequence of important operations.

• Explain functions (blocks of code that perform particular tasks), conditional branches (decisions such as “if this happens, do that”) and error handling (how the program responds when something goes wrong).

• Distinguish behaviour visible in the code from assumptions about what it might do.

• Use accessible language, analogies and diagrams.

• Identify relevant implementation limitations.

• Avoid inventing functionality that the source code did not support.

Anatomy of the MailSage Code Explainer skill. The SKILL.md file defines how the AI should analyse code, verify technical accuracy, teach beginners and follow safety rules.

The skill did not modify the model's underlying training. It supplied extra instructions intended to guide how the existing model approached a task.

I revised the skill over multiple versions, comparing AI-generated explanations for technical accuracy, completeness and clarity. Early evaluations helped refine the instructions but also exposed a measurement problem: a response could be more detailed or easier to understand without necessarily being more correct.

I also learned that the testing environment mattered. Simply disabling or renaming the skill file did not necessarily guarantee that an AI session could no longer access its contents—for example, if those instructions had already been supplied in the conversation or were otherwise available in its context.

To make later comparisons more meaningful, I paid closer attention to skill availability (whether the instructions were accessible), isolation (whether a test could be influenced by previous material) and exactly what each model session received.

I needed a way to evaluate correctness more objectively.

Section 003: Moving from subjective reviews to controlled evaluation

Why better-looking responses were insufficient

The initial tests focused primarily on explanations of existing MailSage source files.

I assessed whether the AI could explain application behaviour, correctly describe important functions, and identify relevant implementation details.

However, reviewing explanations introduced an element of subjectivity. Two responses might communicate the same technical behaviour differently, making it difficult to determine whether the skill itself had improved performance.

I therefore introduced a second task: identifying deliberately planted software defects. Instead of judging only whether an explanation sounded convincing, I could ask a more concrete question:

How many known bugs did the AI correctly identify?

This required three components:

1. A requirements specification: a written description of how the software should behave.

2. A deliberately defective implementation: code containing known violations of those requirements.

3. An answer key: the list of known issues, held back from the AI during the review, against which its findings could be checked.

The aim was to make evaluation more repeatable and less dependent on presentation style.

The full experimental journey — Tests A–Z

The experiment progressed through several stages, each introducing more structured ways of measuring the AI's answers. Early tests helped refine the process; later tests used fixed challenges and predefined marking criteria. A baseline means the starting performance used for comparison. A calibrated rubric is a marking guide revised to apply scores more consistently, while source-grounded questions can be checked against the actual code rather than a reader's impression.

The evolution of the MailSage Code Explainer experiment. Twenty-six evaluations across seven stages, progressing from exploratory testing to controlled, blind defect-discovery comparisons.

The progression matters as much as the final scores. Early tests helped reveal weaknesses in the setup; later tests introduced stronger controls (conditions kept consistent), more explicit ground truth (the verified answers used for checking) and clearer limits on what the evidence could establish.

The seven paired comparisons were M/N, O/P, Q/R, S/T, U/V, W/X and Y/Z. A pair placed two runs on a common task, but the full sequence was small and evolved over time. It was not a statistically powered study with enough independent observations to establish that a difference would reliably recur.

What changed in my approach: I stopped asking only, “Is this a good explanation?” and increasingly asked, “What specific claim is being tested, against which evidence, and under which conditions?”

Section 004: Constructing the benchmarks

Benchmark V1 — Establishing an initial baseline

A benchmark is a standardised challenge against which different approaches can be measured. My first benchmark used synthetic message-triage code—purpose-built JavaScript that simulated processing messages rather than accessing real Gmail data—with seven predefined bugs.

I compared GPT-6 at Medium thinking effort with and without Skill V3 (SKILL.md).

Both configurations identified all seven defects. However, the test fixture (the purpose-built example used for testing) included comments explaining aspects of the intended behaviour. Those comments might have helped the AI find the issues, so I treated the result as useful but not a demanding independent test of code understanding.

Although the responses differed in how they explained the code, the measured defect-detection results did not establish an advantage for the skill.

The model's baseline capabilities were already strong enough to complete the initial challenge. I therefore developed a second, more demanding benchmark.

Benchmark V2 — Introducing interacting workflows

Benchmark V2 simulated a more complete inbox-processing workflow. It combined several connected operations that a real application might need to perform when retrieving and organising messages:

The simplified workflow looked like this:

Benchmark V2 — Simulated inbox-processing workflow. The benchmark tested a sequence of connected operations, with error handling applying throughout. Eight deliberately introduced defects challenged the AI's ability to identify incorrect behaviour.

I deliberately introduced eight known behavioural defects into one version of this implementation. These included stopping pagination too early, mishandling case-sensitive message IDs, failing to exclude messages marked as trashed, exceeding limits on simultaneous requests and suppressing error records under particular conditions.

The benchmark was entirely synthetic: it did not connect to Gmail or retrieve real user messages.

Creating an executable reference

Alongside the deliberately defective code, I developed a reference implementation: a separate version intended to demonstrate how the workflow should behave according to its written requirements.

I also wrote an automated test suite—55 repeatable software checks—using Node.js, a tool that can run JavaScript outside the browser. The tests covered expected behaviours and edge cases, or less common situations that can expose bugs, across the workflow.

All 55 tests passed against the reference implementation.

This gave me an executable way to check that the reference satisfied the behaviours covered by the tests before asking the AI to inspect the defective version. However, passing the tests did not prove that the reference code was flawless: a test suite can only check situations it actually includes. The tests also did not measure how well the AI performed.

Once the benchmark materials were finalised, I created a Git tag: a named checkpoint identifying an exact version of the code and test materials. The tag provided a fixed reference so later results could be compared against the same benchmark; it did not physically prevent further edits to the files.

Section 005: Designing the blind evaluations

Controlling the variables

In an experiment, controlled variables are conditions kept unchanged so that a comparison is fair. I wanted to investigate the effect of providing Skill V3 (SKILL.md) while keeping the other relevant conditions as consistent as possible.

For the final comparisons, each evaluation used the same model, defective implementation, written requirements and substantive review instructions.

The conditions differed in two ways:

• Skill access: whether Skill V3 was provided.

• Thinking effort: whether GPT-6 used its High or Medium setting, which affects the reasoning effort allocated to the task.

I conducted the evaluations in fresh, isolated, unpersonalised temporary ChatGPT sessions. In practical terms, each run started separately rather than continuing a prior test conversation, reducing the risk of previous answers influencing a later result.

The evaluations were blind with respect to the answer key: the AI received the defective code and written requirements, but not the list of the eight planted bugs. After each run, I compared its findings with that list.

Final Benchmark V2 — Blind evaluation process (Tests W–Z). Four separate evaluations compared GPT-6 with and without Skill V3, using High and Medium thinking effort.

The Medium-effort evaluations were supplementary runs intended to see whether the same comparisons behaved differently at a lower thinking-effort setting.

Section 006: The results

Earlier results — Explanations and targeted questions

The earlier work produced several informative results that the final benchmark alone would not show:

• Tests I–L: Recorded explanation scores were I = 12/12, J = 14/18, K = 16/18 and L = 11/12. These covered different source files and evolving marking criteria, so the scores should not be compared as if they came from one standardised test.

• Tests M–P: Four GPT-6 full-file explanations each received 48/50 on the final calibrated Explanation V2 marking guide, across High and Medium thinking effort and both skill conditions.

• Tests Q–R: Both scored 12/12 when answering six targeted technical questions checked against the code.

• Tests S–T: Both found all six target issues, but each made an additional important claim that the evidence did not support—a material overstatement. This showed that a perfect score on the specified task can coexist with a misleading extra assertion.

• Tests U–V: Both identified 7/7 deliberately planted defects in synthetic Benchmark V1.

These were ties on the specified headline measures. They do not prove that the responses were identical, that skill-assisted and unassisted models are equivalent, or that the skill could never provide value.

Final Benchmark V2 — What did the blind evaluations show?

All four final evaluations identified all eight deliberately planted defects.

Final Benchmark V2 results. All four evaluations identified all eight deliberately seeded defects. Providing Skill V3 did not improve the measured recall in these runs.

Each run achieved 100% seeded-defect recall. Recall means the proportion of known problems that the AI correctly identified: here, eight found out of eight deliberately introduced.

On this particular measure, supplying Skill V3 did not produce an observable improvement at either thinking-effort setting.

One additional observation emerged during Test W. The AI identified a valid input-validation issue—a problem with checking unexpected or invalid information before it is processed—that affected both the deliberately defective version and the reference implementation.

Because this issue was not one of the eight deliberately seeded defects, I recorded it separately rather than increasing the seeded-defect count. It also exposed a limitation of the automated tests: all 55 could pass without covering every possible input or problem.

Test Z raised a different challenge: it reported an additional candidate issue, then withdrew it after determining that the claim was unsupported. Recording that correction matters because reporting more potential problems is not automatically better if some of those claims are wrong.

Why 8/8 is not the same as 100% overall accuracy

Benchmark V2's main measure was recall: the proportion of the eight deliberately planted defects that the AI found.

Although all four evaluations achieved 100% seeded-defect recall, I did not comprehensively check and classify every additional claim made in the responses. Therefore, I could not reliably calculate:

• Precision: Of the problems the AI reported, how many were genuine?

• F1 score: A combined measure of precision and recall, designed to reward finding genuine problems without making too many false claims or missing real ones.

For example, Test W's additional input-validation finding was valid, while Test Z's withdrawn candidate demonstrates why extra findings need scrutiny. Without systematically grading such claims across every run, an F1 score would give a misleading impression of certainty.

The 8/8 results demonstrate that every deliberately planted defect was identified. They do not establish 100% overall code-review accuracy, 100% precision or a perfect F1 score.

Interpreting the outcome

My original hypothesis was that structured instructions might improve the model's ability to identify implementation defects.

The final benchmark did not provide evidence of an improvement in seeded-defect recall attributable to Skill V3.

However, this does not prove that the skill is ineffective for other tasks or under different conditions. For example, the final comparisons did not independently score whether the explanations were easier for beginners to understand, whether the AI taught concepts better or whether it followed a more consistent structure.

The findings therefore concern a specific capability measured under a limited set of conditions—not every possible benefit the skill was designed to provide.

Section 007: Limitations and trade-offs

I treated this as an exploratory engineering experiment rather than a statistically conclusive study. In other words, the number and variety of tests were insufficient to determine whether a performance difference would reliably recur across many independent trials.

Several limitations affected what I could reasonably conclude.

First, small sample size. The final benchmark used one synthetic defective implementation, and each of the four configurations was evaluated once.

Second, a ceiling effect. All four configurations reached the maximum score for identifying planted bugs. When everyone achieves the highest possible score, that measure cannot distinguish their performance.

Third, limited generalisability. Results from one artificially constructed coding challenge may not carry over to different applications, programming languages, codebases or real production environments.

Fourth, incomplete false-positive measurement. The final benchmark measured how many known bugs were found. It did not comprehensively classify every extra claim as correct or incorrect, so the results cannot establish overall code-review accuracy, precision or F1.

Fifth, archival limitations. Not every early AI response or original testing condition was preserved in enough detail for an outside reader to recreate the experiment independently. An evidence index—a record indicating where supporting test files and documentation can be found—describes what is available and where the gaps remain.

Finally, a narrow final metric. The final benchmark did not directly measure all the qualities the skill was intended to improve, including teaching effectiveness, the structure of explanations and consistency across very different codebases.

A larger experiment using more varied code examples, independent repetitions and explicit grading of extra claims would be needed to investigate those possibilities.

I chose to close the experiment rather than repeatedly modify the benchmark in pursuit of a more favourable result.

Section 008: Outcome and reflections

The experiment concluded with five principal deliverables:

1. A reusable AI agent skill — structured instructions for explaining JavaScript source code and analysing application behaviour.

2. A tested synthetic benchmark — including written requirements, deliberately defective code, a reference implementation and 55 automated tests.

3. A documented A–Z evaluation methodology — covering all 26 recorded tests, seven paired comparisons, scoring decisions, results, limitations and interpretation.

4. A structured evidence record — a corrected Excel tracker, seven public CSV exports (spreadsheet-compatible text files that make the results easier to inspect), an evidence index and a complete evaluation report.

5. Published project documentation — connecting the skill, benchmarks, results and explanatory case study through the MailSage repository.

The initial skill work was developed on a separate experimental Git branch, a parallel line of development that allows changes to be prepared without immediately modifying the main project. The completed Tests A–Z data, CSV export utility and consolidated documentation were later published through a dedicated documentation branch and merged into main (the repository's primary version) via Pull Request #4—a formal proposal to review and incorporate changes.

Verification and publication — 55 automated checks passed against the reference implementation; 26 evaluations were documented with an Excel tracker and seven CSV datasets; the completed documentation was merged through GitHub Pull Request #4.

I can used the command below to run the 55 automated Benchmark V2 tests:

node --test evals/mailsage-code-explainer/benchmark-v2/tests/*.test.mjs

node --test evals/mailsage-code-explainer/benchmark-v2/tests/*.test.mjs

This was a separate AI engineering experiment, not a new feature of the live MailSage email application.

What I learned

The most significant lesson was the distinction between building a useful AI instruction set and demonstrating that it improves performance.

I began by concentrating on the instructions themselves. As the work progressed, I became increasingly focused on the quality of the evaluation process.

This meant defining what success would look like before running tests, creating known reference answers, keeping comparison conditions consistent, documenting unexpected findings and resisting conclusions that the results did not justify.

I also learned that automated software tests and AI evaluations answer different questions. The former check whether code behaves as intended in specified situations; the latter measure how an AI model performs on a particular task.

Publishing all 26 recorded evaluations changed how I thought about technical evidence. The final results are more understandable when the less controlled early tests, revised scoring approaches and final benchmark can be seen as one progression—with their differences stated openly.

Ultimately, the absence of a demonstrated skill-related improvement was itself a meaningful result.

The experiment reinforced a principle that I intend to carry forward into future AI development work:

A plausible improvement is a hypothesis. Demonstrating that improvement requires evidence.

Project documentation

• Experiment overview and methodology

• Complete evaluation report — Tests A–Z

• Evaluation tracker — Excel workbook

• Seven CSV evaluation datasets

• Agent Skill V3

• Benchmark V2 — Blind Evaluation Results

• Publication and documentation — merged Pull Request #4