TL;DR

After developing MailSage, an AI-powered Gmail companion application, I wanted to explore a different application of AI: whether a reusable set of instructions could help an AI model understand existing software more accurately, explain its behaviour and identify implementation defects.

I developed the MailSage Code Explainer, an AI agent skill intended to provide structured, beginner-friendly explanations of JavaScript source code.

What began as an experiment in improving code explanations evolved into a more systematic investigation of AI evaluation. I compared model responses with and without the skill, developed synthetic JavaScript benchmarks containing deliberately introduced defects, built an executable reference implementation, and created an automated test suite.

The final phase involved four controlled blind evaluations using GPT-6 at Medium and High thinking effort.

All four evaluations identified all eight deliberately seeded defects, regardless of whether the skill was supplied.

Consequently, the results did not demonstrate an incremental defect-detection benefit from the skill. However, the experiment produced a reusable agent skill, a verified synthetic benchmark and a documented evaluation methodology.

Key outcomes

Section 001: The motivation

From building with AI to evaluating AI

During the development of MailSage, AI-assisted programming tools became an important part of my workflow.

I used them to understand unfamiliar code, investigate errors, explore implementation approaches and troubleshoot application behaviour.

However, I had also encountered situations where the suggestions appeared plausible but were technically inappropriate, incomplete or potentially destructive.

This raised a broader question:

Could I create a reusable set of instructions that would help an AI model analyse source code more reliably, rather than relying exclusively on its default behaviour?

I was particularly interested in whether a structured approach could improve technical accuracy, encourage explanations grounded in the actual source code, and make complex engineering concepts easier for a beginner to understand.

MailSage provided a useful testing environment because I was already familiar with its architecture and had practical experience investigating its behaviour.

Section 002: Designing the agent skill

Turning a repeatable workflow into instructions

An AI agent skill is a reusable set of instructions that guides how an AI system approaches a particular category of work.

For this experiment, I created a Markdown-based SKILL.md file containing instructions for analysing and explaining MailSage's JavaScript source code.

The skill encouraged the model to:

• Identify the overall purpose of a file.

• Trace the sequence of important operations.

• Explain functions, conditional branches and error handling.

• Distinguish observed behaviour from assumptions.

• Use accessible language, analogies and diagrams.

• Identify relevant implementation limitations.

• Avoid inventing functionality that the source code did not support.

The skill was not a new AI model, nor did it modify the model's underlying training. It provided additional instructions for how the existing model should approach the task.

I iterated on the skill over multiple versions, using comparisons of AI-generated explanations to assess technical accuracy, completeness and clarity.

These early evaluations helped refine the instructions. They also exposed a measurement problem: a response could be more detailed or easier to understand without necessarily being more technically accurate.

I needed a way to evaluate correctness more objectively.

Section 003: Moving from subjective reviews to controlled evaluation

Why better-looking responses weren't enough

The initial tests focused primarily on explanations of existing MailSage source files.

I assessed whether the AI could explain application behaviour, correctly describe important functions and identify relevant implementation details.

However, reviewing explanations introduced an element of subjectivity. Two responses might communicate the same technical behaviour differently, making it difficult to determine whether the skill itself had improved performance.

I therefore introduced a second evaluation task: identifying deliberately planted software defects.

Rather than asking whether an explanation sounded convincing, I could ask a more concrete question:

How many known implementation defects did the AI correctly identify?

This required three components:

1. A written specification describing how the software should behave.

2. A deliberately defective implementation containing known violations of that specification.

3. A separate answer key against which the AI's findings could be compared.

The objective was to make the evaluation more repeatable and less dependent on presentation style.

Section 004: Constructing the benchmarks

Benchmark V1 — Establishing an initial baseline

The first benchmark used a synthetic message-triage implementation containing seven predefined defects.

I compared GPT-6 at Medium thinking effort with and without Skill V3.

Both configurations identified all seven defects.

Although the responses showed differences in how they explained the code, the defect-detection results did not establish an advantage for the skill.

This was an important observation: the model's baseline capabilities were already strong enough to complete the initial test.

I therefore developed a second, more demanding benchmark.

Benchmark V2 — Introducing interacting workflows

Benchmark V2 simulated a more complete inbox-processing workflow.

The implementation covered several connected operations:

• Retrieving paginated message references.

• Removing duplicate message identifiers.

• Fetching message details with concurrency limits.

• Filtering eligible inbox messages.

• Sorting messages by timestamp.

• Applying result limits.

• Handling retrieval failures.

I created a deliberately defective version of the implementation containing eight known behavioural defects.

These included terminating pagination too early, incorrectly handling case-sensitive message IDs, failing to exclude trashed messages, exceeding configured concurrency limits and suppressing failure records under particular conditions.

The benchmark was entirely synthetic. It did not connect to Gmail or retrieve real user messages.

Creating an executable reference

Alongside the defective implementation, I developed a separate reference implementation and a suite of automated JavaScript tests using Node.js.

The suite contained 55 tests covering expected behaviours and edge cases across the inbox-processing workflow.

All 55 tests passed.

This provided an executable basis for checking the intended functionality before running the final AI evaluations.

Importantly, these tests validated the reference implementation against the included test cases. They were not themselves a measurement of AI performance, nor proof that every possible implementation edge case had been covered.

Once the benchmark materials were finalised, I preserved their state using a Git tag. This prevented changes to the test material from inadvertently affecting subsequent comparisons.

Section 005: Designing the blind evaluations

Controlling the variables

I wanted to isolate the effect of providing Skill V3 while keeping the other relevant evaluation conditions as consistent as possible.

For the final comparisons, each evaluation used the same model, defective implementation, requirements specification and substantive review instructions.

The conditions differed in two respects:

• Whether Skill V3 was provided.

• Whether GPT-6 was configured with High or Medium thinking effort.

I conducted the evaluations in fresh, isolated, unpersonalised temporary ChatGPT sessions.

The AI was given the defective code and the written requirements but was not provided with the benchmark answer key.

The resulting findings were subsequently compared against the eight deliberately seeded defects.

The Medium-effort evaluations were supplementary runs intended to examine whether the results differed at a lower thinking-effort setting.

Section 006: The results

What did the evaluations actually show?

All four final evaluations identified all eight seeded defects.

Every configuration achieved 100% recall of the eight known defects.

This meant that Skill V3 did not produce a measurable improvement in seeded-defect recall at either thinking-effort setting.

One additional observation emerged during Test W. The evaluation identified a valid input-validation issue affecting both the deliberately defective implementation and the reference implementation.

Because this issue had not been deliberately introduced as one of the eight benchmark defects, I documented it separately rather than counting it as an additional seeded defect.

It also exposed a limitation in the test suite: passing all 55 automated tests did not establish that the reference implementation behaved correctly for every possible input.

Interpreting the outcome

The initial hypothesis was that providing structured instructions might improve the model's ability to identify implementation defects.

The results did not provide evidence of that improvement on the final benchmark.

However, they also did not establish that the skill was ineffective in other contexts.

For example, the final comparison did not independently score the quality of beginner-friendly explanations, teaching effectiveness or consistency of response structure.

The findings therefore applied to a specific capability, evaluated under a limited set of conditions.

Section 007: Limitations and trade-offs

I treated the experiment as an exploratory engineering exercise rather than a statistically conclusive research study.

Several limitations affected what I could reasonably conclude.

First, the final benchmark used one synthetic defective implementation. Each of the four configurations was evaluated once.

Second, all configurations achieved perfect seeded-defect recall. This created a ceiling effect: when every configuration achieves the maximum score, the metric cannot distinguish between them.

Third, a synthetic benchmark does not necessarily represent the variety and complexity of real production repositories.

Finally, the main evaluation metric was defect recall. It did not measure every quality that the original skill was designed to improve.

A larger and more varied experiment would be needed to investigate whether the skill produced consistent benefits across different codebases, task difficulties or explanation-quality measures.

I chose to close the experiment rather than repeatedly modify the benchmark in pursuit of a more favourable result.

Section 008: Outcome and reflections

The experiment concluded with three principal deliverables:

1. A reusable AI agent skill — structured instructions for explaining JavaScript source code and analysing application behaviour.

2. A verified synthetic benchmark — including a requirements specification, deliberately defective implementation, reference implementation and 55 automated tests.

3. A documented evaluation methodology — covering controlled comparisons, blind review conditions, results, limitations and interpretation.

The complete work was committed to GitHub through a dedicated experimental branch, reviewed through a pull request and merged into the main MailSage repository.

What I learned

The most significant lesson was the distinction between building a useful AI instruction set and demonstrating that it improves performance.

I began by concentrating on the instructions themselves. As the experiment progressed, I became increasingly focused on the quality of the evaluation process.

This meant defining what success would look like before running the tests, creating known reference answers, controlling variables, documenting unexpected findings and resisting conclusions that the results did not justify.

I also learned that successful automated tests and successful AI evaluations answer different questions.

The former can verify defined software behaviour; the latter assesses how a model performs against a particular task or evaluation target.

Ultimately, the absence of a demonstrated skill-related improvement was itself a meaningful result.

The experiment reinforced a principle that I intend to carry forward into future AI development work:

A plausible improvement is a hypothesis. Demonstrating that improvement requires evidence.

Project documentation

• MailSage Code Explainer — GitHub

• Agent Skill V3

• Benchmark V2 — Blind Evaluation Results

• Merged Pull Request #3

Tech Stack

• Gmail API

• OpenAI API

• JavaScript

• React — Component state management and email interface logic

• Next.js

• Tailwind CSS

• Vite

• NextAuth.js

• Browser localStorage API — Persistent, account-specific pinned and hidden email preferences stored locally in the browser

• DOMPurify — HTML email content sanitisation before rendering

• Vercel

• Lucide React

• Framer Motion

• Google Fonts

Development & Release Workflow

VS Code, Cursor IDE, Kiro CLI, Git, GitHub (including pull requests), and Vercel (preview and production deployments).