Why GEPA Needs a Better Metric

A generated theme can look striking in a thumbnail and fail on a real map. Deep water, warm land, pale labels, a dark article card with a thin heading. The palette reads as a decision. Then the same colors meet a dense street grid, and the labels sit a few lightness steps away from the land under them. Article body contrast drops below 4.5 to 1.

A vision judge that receives only the rendered board has little reason to catch this. The image is attractive, the mood matches the request, the swatches agree with each other. If the metric is that judge's score, GEPA will find the cheapest path to a higher score, and the cheapest path can be a handsome palette with text nobody can read. Nothing in the loop knows that a map has to be read.

The prompt lab under labs/prompts exists to make that failure visible before a prompt change reaches production.

What the optimizer can see

Ax optimize takes a program, a set of training examples, a metric, a set of validation examples, and two model services, one for the student and one for the teacher. It proposes instruction changes, runs the student over cases, scores the outputs with the metric, and returns a Pareto result.

The teacher model is not given the product. It sees the instruction, the examples, the numbers the metric produced, and whatever text the metric returns through feedbackFn. If the metric reports one number, that number is the whole description of quality the teacher gets.

GEPA does its job well. It will raise whatever the metric rewards, including the wrong thing. A high score only means the prompt satisfied the metric.

Three targets, three measurement designs

The three targets differ in how much judgment they hand to a model. auto-location hands none. Every field in its output can be checked against the input text and gold data without asking a model. The visual targets render a board and ask a vision model, because a palette's fit to a request cannot be reduced to arithmetic. Both of them also keep deterministic checks that the judge cannot overrule.

The target with no judge

For auto-location, the metric matches predicted locations against gold locations after normalization. The weighted score is 55 percent location precision and recall F1, 15 percent label_in_context grounding, 15 percent context grounding with at most 15 characters, 10 percent marker_id accuracy, and 5 percent confidence calibration.

Matching is generous on form and strict on substance. Both label and label_in_context are normalized and compared against the gold label, the in-context label, and any aliases. Grounding is literal: label_in_context and context must appear in the user's text, and context is capped at 15 characters so it behaves like a short quote. Marker accuracy and confidence calibration are counted only on matched pairs, the second rewarding a predicted confidence close to the gold value.

This target is the easiest to trust. It is also the easiest to game in small ways: a model can copy phrases from the input and collect two of the five components without matching a single location.

The hard gate on map themes

Before any model judges a theme, the output has to survive validation. All 14 color fields must be six-digit hex. The description must be 50 characters or fewer. The score must sit between 0 and 100. The theme's own score field is never used in scoring. A model that declares its work excellent gets no credit for the declaration.

The deterministic color checks go further. They measure article heading, secondary, and body contrast against the article background at 4.5 to 1 by default. They measure brand foreground against brand background at 4.5 to 1 by default. They measure label color against label halo at 3 to 1. Then they measure OKLab separation for water against land, green against land, primary road against land, secondary road against land, and primary road against secondary road.

Each measurement has a threshold that produces a named issue, and a scoring curve that gives partial credit above a lower floor, so the two numbers differ on purpose. A theme below 4.5 to 1 body contrast fails the requirement and still receives partial credit, because a value that is off target still leaves some text readable.

One gate sits above all of them. If body, heading, or brand contrast falls under 3 to 1, or the label halo under 2 to 1, the case is a critical failure, and legibility is capped at 0.55 however the judge scored the board. A striking palette can still lose on that cap. That cap is the direct answer to the opening failure mode.

The judge gets a rendered board

The renderer builds an evaluation board from the generated theme using the product's own theme logic and MapLibre renderer: fixed real-world map views, an article card, brand UI, a marker, and the generated swatches. The judge receives that JPEG, the original request, the case rubric, and the theme spec.

Four dimensions, each scored 0 to 100: intent fidelity, map legibility, cross-surface coherence, and design quality. The judge returns a critique and up to five prompt-level rules. The instructions tell it to ignore the student's self-reported score and to treat insufficient text or road legibility as a serious defect even when the palette is attractive. It is also told to write rules that would help future outputs, beyond the colors for this one case.

The metric does not take the judge at face value. Legibility is 60 percent measured color quality and 40 percent the judge's map legibility score, so a judge that falls for a pretty board still has to get past the contrast numbers. Intent fidelity is 85 percent judge score plus 15 percent mode fidelity when the case asks for a dark or light result, measured from the average background luminance of land, article, and brand.

Feedback is what gives the teacher a direction

A scalar tells the teacher that a candidate was worse. It does not say why. The metric stores a feedback string per prediction: every deterministic issue with its measured value and expected minimum, then the judge's critique, then each suggested rule prefixed with Rule:. Ax calls feedbackFn with the prediction and example, and that text goes back to the teacher.

That changes the shape of the proposal. A teacher told only that a candidate scored lower has nothing to act on. A teacher told that article body contrast fell below the required minimum, plus a critique about labels disappearing into the land, has something to write a rule about. The deterministic lines also constrain the judge: praise for a board whose body text misses the contrast floor is arguing with a number the teacher can see.

None of this guarantees a good proposal. It raises the chance that a proposal is about the right problem.

Four objectives, kept apart

Both visual targets report four objectives. Blending them into one number is available, and the lab computes a weighted overall for reports: 30 percent intent fidelity, 35 percent legibility, 20 percent cross-surface coherence, and 15 percent design quality for themes. GEPA does not search that blend. It searches the vector.

The difference matters at the decision boundary. A candidate that follows the request more closely and loses legibility should stay visible as a tradeoff. A single score averages the two into a middle value, and the tradeoff disappears. On a Pareto front the candidate remains a point someone can look at and reject for a stated reason.

Marker styling uses its own four: visibility, theme coherence, semantic fit, and palette quality. Visibility combines deterministic contrast and OKLab checks with a partial cap of 0.5 on a critical failure, so markers that vanish into the map cannot score their way out. Palette quality is 40 percent measured palette separation, 30 percent the judge's visual hierarchy score, and another 30 percent its design quality score.

The artifact keeps the Pareto front and per-candidate diagnostics: the color measurements, the four judge scores, the derived metrics, and the deterministic issues, critique, and suggested rules. When a proposal is rejected, the reason is still on disk. That is what makes a failed run useful later.

What the loop is allowed to see

auto-location uses 16 train cases, 6 validation cases, and 8 test cases. The two visual targets use 16, 8, and 8. GEPA sees train and validation. The test set is evaluated before and after optimization and never enters the search.

The visual targets run 10 trials with a minibatch size of 4, early stopping after 4 trials, and a hard cap of 20 metric calls. auto-location runs 8 trials with a cap of 120 metric calls. A visual metric call renders a headless Chromium board and calls a vision judge, so each call costs seconds and real money. A deterministic call costs almost nothing, which is why the text target can afford a cap six times larger.

The default student is Gemini 3.5 Flash Lite, the same model the product calls for map themes and locations. The teacher and the judge can run through Codex or the AI Gateway.

Ax scores are not production scores

Ax and the AI SDK render prompts differently. The same instruction text becomes a different request, so a score computed on one renderer is not a score for the other.

The lab therefore has a second step. It compares the current instruction and the optimized instruction using generateText, Output.object, and the matching Zod schema, on the 8 held-out test cases, with the same model. For the visual targets, the judge runs three samples per case and takes the median per dimension, so one odd judgment does not decide the result. The location target is scored deterministically, so one run per case is enough.

That comparison is the closest thing the lab has to a production measurement, and it is still a lab measurement. Artifacts and JSON reports are gitignored. Nothing in the pipeline writes to a product prompt. Applying a candidate means a person reads the diff, decides the rules are sound, and edits the prompt by hand.

What a better metric still cannot fix

A metric can be blind in ways no amount of engineering removes. The theme judge sees a fixed set of boards and can miss a failure that appears on a different map at a different zoom. Judge scores vary between runs, and three samples with a median reduce the variance without removing it. The training sets are small, 16 cases for the visual targets, and they cannot represent every user's request, language, or taste. Some failures come from missing tools or context, and no instruction can invent data the model was never given. Longer prompts that repair one case cost more on every call after that.

The lab is a filter for bad changes. It is not a guarantee of a good one.

Why this belongs next to the product

Two targets cover the live product prompts, for map themes and location extraction. The third optimizes a marker color selection prompt that the lab keeps and evolves on its own. Each output can look plausible while quietly making a map worse. A theme that fails contrast, a marker palette that would disappear into the map, and a location list that invents a label are all failures a person might miss while reviewing a screenshot.

Rejecting those changes before they reach a product prompt is the point. The optimizer is the easy half.

If you want the product side, AI in Tasmap explains how the assistant features fit together, and Vibe map theming tips shows what the theme prompt is trying to do. Tasmap is at tasmap.app.