Are AI labs pelicanmaxxing?

Author

Affiliation

Dylan Castillo

Iwana Labs

For the past few years, Simon Willison has tested every major LLM release with the same prompt: “Generate an SVG of a pelican riding a bicycle”.

What began as a tongue-in-cheek benchmark has become one of the most famous informal benchmarks in AI. Simon’s pelican-on-a-bicycle results are often among the most upvoted comments on Hacker News threads announcing new releases from AI labs.

The benchmark is now famous enough that there’s plenty of discussion about its usefulness and about whether AI labs might be benchmaxxing¹ on it. When billions or even trillions of dollars are at stake, and a strong result could help persuade users, wouldn’t it be tempting to pelicanmaxx your model just a bit?

I wanted to find out, so I put together a small experiment. I generated 1,008 SVGs across seven frontier models, scored them with an LLM judge, and used Claude Fable 5 for the analysis.

This article presents the results. All the code is available on Github.

How I tested it

I built a grid of 8 animals × 6 vehicles = 48 prompts, where the famous prompt is one cell:

1. Animals: pelican, flamingo, heron, otter, raccoon, antelope, whale, cat
2. Vehicles: bicycle, unicycle, skateboard, scooter, plane, boat

Every prompt uses almost identical phrasing to Simon’s, only switching the animal and vehicle. The animal and vehicle selection wasn’t done in a very rigorous manner, but I tried to vary both similarity to the original prompt and difficulty. Flamingo and heron are quite similar to pelicans; cat, raccoon, and otter are easy cases; antelope is hard; and whale is as different as you can get.

I tested seven models through OpenRouter: GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro. I generated 3 samples per prompt, at temperature 1.0, requesting the same reasoning effort from every model. That resulted in 1,008 SVGs.

Then I ran each image through a three-stage pipeline:

1. Rendering: Each SVG is rendered to PNG. If a model returns no SVG or one that fails to render, I regenerate until it produces a valid one, and record the number of attempts. There were only 11 retries across the 1,008 generations.
2. Judging: GPT-5.6 Luna scores each image against its prompt, producing 1-5 ratings for the animal, the vehicle, and the action.
3. Blind feature extraction: To extract some potentially useful data for a more detailed analysis, I also passed each rendered image to Gemini 3.1 Flash-Lite and extracted the animal and vehicle it recognized, which way the subject faces, and an open-ended list of scene elements.

My hypothesis is that if a lab trained on the benchmark, it should show up in some combination of the pelican row scoring above what the animal deserves, the bicycle column scoring above what the vehicle deserves, or the specific pelican-bicycle cell beating both.

Show me the SVG

Pick a lab to see everything it drew, with the judge’s score under each image (click to open full size):

Evidence #1: Labs are not better at drawing pelicans

Here’s the mean judge score per animal, pooled across all models:

Figure 1: Mean judge score by animal, all models pooled.

The pelican is 4th of 8, behind cat, flamingo and raccoon. If AI labs were training on the benchmark, you’d expect pelicans at the top. Instead they’re mid-tier: six of the seven labs draw cats better than pelicans (the exception is Grok 4.5), and four of seven draw flamingos better.

However, you might argue that the pelican may simply be a harder animal to draw than a cat. A lab could train on pelicans and still not push them past the easy animals. We’ll account for that in a later section.

Evidence #2: Labs are not better at drawing bicycles

Bicycles fare even worse. They’re the second-to-worst vehicle in the ranking, ahead of only planes:

Figure 2: Mean vehicle rating by vehicle, all models pooled.

If labs were training on the benchmark, the bicycle should be their best vehicle, not their second-worst.

The same caveat applies here. A bicycle has much more unforgiving geometry than a skateboard, so a lab could train on bicycles and still land near the bottom of this ranking.

One note on the last place, though: “plane” wasn’t a great choice, because models often read it geometrically, drawing the animal standing on a flat surface instead of flying an aircraft. The plane is the only vehicle where the blind extractor sometimes found no vehicle at all (25 of 168 images, against zero for the other five), and 17% of plane images scored a 1 or 2 on the vehicle rating, versus at most 1% everywhere else. The plane’s last place is partly my fault for not writing “airplane”.

Evidence #3: Labs are not better at drawing pelicans on bicycles, even adjusting for difficulty

Put the two together and the “pelican on a bicycle” ends up in the lower half of the ranking, at #33 of 48:

Figure 3: All 48 combos ranked; pelican + bicycle highlighted.

But that’s still a raw ranking, and it doesn’t adjust for difficulty.

To account for that, I fit a fixed-effects regression on all 1,008 images: score ~ lab + animal + vehicle, plus per-lab interaction terms for pelican, bicycle, and the pelican-bicycle cell, with robust standard errors. The animal and vehicle terms absorb the inherent complexity of each. The interactions measure each lab’s benchmark-specific boost, with confidence intervals.

The results:

1. Every per-lab pelican effect (the lab’s boost on pelicans across all six vehicles) lands between -0.09 and +0.06 judge points (all p ≥ 0.40).
2. The per-lab bicycle effects (the lab’s boost on bicycles across all eight animals) run from Grok 4.5 at -0.22 (p=0.022) to Gemini 3.5 Flash at +0.24 (p=0.005). Three of the seven clear p < 0.05, and they point in both directions.
3. No pelican-bicycle cell effect (the extra boost on the specific combination, on top of the lab’s pelican and bicycle effects) is significant. The closest is Grok 4.5 at +0.32 (p=0.061); the rest sit between -0.24 and +0.06.
4. The average benchmark-cell effect across labs is -0.09 (p=0.39). If anything, the famous prompt slightly underperforms what each lab’s pelican and bicycle skills predict.

Here are the full per-lab estimates. A pelicanmaxxing lab would show dots to the right of the zero line across its whole row:

Figure 4: Difficulty-adjusted effects per lab, with 95% confidence intervals. Highlighted intervals exclude zero.

Every pelican effect and every cell effect contains zero. The three intervals that don’t are all in the bicycle column (Gemini and Claude positive, Grok negative), and none survives a multiple-comparisons correction: the Bonferroni threshold across the 21 tests is 0.05/21 ≈ 0.002, and the smallest p-value among them is 0.005. The full table of estimates and p-values is in the repo.

However, the width of those intervals puts the minimum detectable effect at roughly 0.7 judge points for a single lab’s pelican-bicycle cell, on a 1-5 scale. This means that smaller effects than that might go undetected by this approach.

Evidence #4: Labs don’t draw pelicans on bicycles any differently

Some have suggested that the pelican on a bicycle looks like a memorized composition, pointing to recurring patterns such as the pelican facing right, a sun, or a scarf. So I wanted to know if this was true.

Direction: All 21 pelican-bicycle images, across all seven labs, face right. No other animal/vehicle combination does that.

However, facing right is common across the board: 60% of all 1,008 images do it. How common depends on the animal and the vehicle, and bicycles are one of the two vehicles where it’s strongest:

vehicle	left	right	ambiguous
scooter	9%	83%	8%
bicycle	11%	81%	8%
skateboard	19%	60%	21%
plane	21%	58%	21%
unicycle	22%	45%	33%
boat	33%	35%	32%

Pelicans are also among the animals that tend to face right:

animal	left	right	ambiguous
antelope	21%	78%	1%
pelican	22%	78%	0%
heron	22%	77%	1%
whale	34%	65%	1%
flamingo	36%	64%	0%
otter	8%	45%	47%
cat	8%	40%	52%
raccoon	3%	36%	61%

It’s hard to draw a pelican or a bicycle facing forward, which is why both have such a low share of ambiguous images. Given those base rates, the result could have happened by chance, but it’s hard to conclude much from it.

Scene elements: I let the extractor name any element it saw in the image. These are the counts:

Figure 5: Share of images containing each element, from the open-ended extraction pass.

A memorized scene would show up as the same set of elements recurring picture after picture. I went looking for that, but pelican-on-a-bicycle images had no elements that recurred more consistently than in other animal-vehicle combinations.

Limitations

1. Using an LLM judge for scoring. This article’s conclusions rest on being able to trust that an LLM can reliably judge the quality of an image.
2. SVGmaxxing. A lab that optimized SVG generation as a whole rises on every cell at once and looks identical to a lab that’s just good. Some labs, such as Google/DeepMind, openly do this. This experiment won’t capture that.
3. Limited budget. The whole experiment ran on roughly $60 of API credits. That capped it at 3 samples per cell, a single judge, and 7 models.
4. “Plane” instead of “airplane”. I used “plane” instead of “airplane” in the vehicles, and models often read it geometrically. However, dropping the plane cells and refitting does not substantially change the results. I didn’t re-run the generations with “airplane” because of budget.

Conclusion

Sorry, HN haters, but there’s little evidence that AI labs are pelicanmaxxing. Or at least they’re not doing it in a plainly obvious manner.

Pelicans aren’t drawn any better than the rest. Bicycles aren’t drawn any better than other vehicles. The combination doesn’t beat what each lab’s pelican and bicycle skills predict, and its scenes aren’t any more repetitive than the rest of the grid’s.

The one exception is direction. All 21 pelican-bicycle images face right, and no other combination is unanimous. But it doesn’t seem that strange. Facing right is the norm across the experiment, and strongest exactly where the famous prompt sits. Three other combinations land at 90% or above, and with 48 of them, I’m not surprised one reached 21 out of 21.

The more plausible story is SVGmaxxing. Google does a version of this openly. Other labs might be doing it more quietly. Sadly, this experiment can’t say who’s doing it. But at least you can sleep tonight knowing that AI labs are not producing terabytes of pelicans on bicycles just to trick Simon Willison.

If you want to look at the data yourself, the full pipeline is in the repo.

Footnotes

1. the practice of optimizing AI models to achieve high scores on popular benchmarks.