Honey, I shrunk the embeddings: Matryoshka vs. PCA

Author

Affiliation

Dylan Castillo

Iwana Labs

As people began using LLMs with their own documents, a new problem emerged: how do you store and search all that information efficiently?

Vector databases quickly became the standard solution. But vectors can contain thousands of dimensions, and storing millions of them can make retrieval slow and expensive.

AI labs responded with a technique called Matryoshka Representation Learning (MRL), which lets you use fewer embedding dimensions without sacrificing much accuracy in your retrieval. That means smaller vector database bills and faster queries.

Happy ending. Almost.

I do not usually spend my Wednesdays worrying about vector database bills. But Doug Turnbull’s article about using Principal Component Analysis (PCA), to reduce vector dimensions made me curious: how would this older, simpler technique compare with MRL?

To find out, I compared the two methods across eight standard retrieval-quality datasets. In this article, I walk through the experiment and share what I found.

All the code and data is available on GitHub.

What are MRL and PCA?

Both methods produce smaller vectors that behave almost like the full ones. But they get there in different ways.

MRL works during training. You train the model with the loss applied at several prefix lengths at once: the first 64 dimensions, the first 128, and so on. This teaches it to pack the most important information at the start of the vector, like a set of nested matryoshka dolls.

At inference time, you simply keep the first d dimensions and re-normalize the resulting vector. Many modern embedding models are trained this way, but older models aren’t, so MRL-based truncation is not available for many popular embedding models.

PCA works after training, which means it can be used with any model. You take a sample of embeddings, find the directions along which they vary the most, and keep the top d of them as a projection matrix¹. From then on, every document and query gets multiplied by that matrix (and re-normalized) before it goes into the index.

But you get additional operational complexity. You need to store and version the PCA transformation, then apply the same version consistently when adding to and querying the index.

How I ran the experiment

The idea behind the experiment was simple: I would shrink the embeddings with each method, run the benchmarks, and see how much retrieval quality suffers at each size.

I generated all embeddings through OpenRouter and evaluated retrieval on eight BEIR datasets: SciFact, NFCorpus, ArguAna, FiQA, SciDocs, Quora, TREC-COVID, and Webis-Touché 2020. For each dataset, I reduced the embeddings to 512, 256, 128, 64, and 32 dimensions.

NoteDataset details

dataset	task	corpus	queries
SciFact	scientific claim verification	5.2K	300
NFCorpus	medical search	3.6K	323
ArguAna	counterargument retrieval	8.7K	1,406
FiQA-2018	financial question answering	57K	648
SciDocs	citation recommendation	25K	1,000
Quora	duplicate question retrieval	523K	10,000
TREC-COVID	biomedical search (COVID-19)	171K	50
Touché 2020	argument retrieval from web docs	382K	49

There were three questions I wanted to answer.

Q1: which method keeps more retrieval quality as you cut dims?

To answer this, I used two MRL-trained models: OpenAI’s text-embedding-3-small (1,536 dims), and Alibaba’s qwen3-embedding-8b (4,096 dims), which sits at the top of the open-weights MTEB BEIR leaderboard.

I reduced each model’s embeddings in two ways:

1. With truncation, I kept the first d dimensions and re-normalized. This is how you reduce dimensions with MRL.
2. With PCA, I fit the projection on the full-dimension document embeddings of the same dataset I then searched, and kept the top d components. From then on, every document and query embedding gets multiplied by that projection matrix and re-normalized before searching.

Then I ran the benchmarks on the eight datasets, comparing the two methods on both models, to see which one preserves more retrieval quality as the embeddings get smaller.

Q2: is it just the MRL training?

If PCA does well, that might be because MRL training has already organized the embedding space in a convenient way. To test that, I added text-embedding-ada-002, an older 1,536-dimensional model that wasn’t trained using MRL, as a control.

I applied PCA to both models. If PCA retains a similar share of retrieval quality on text-embedding-3-small and text-embedding-ada-002, that suggests its performance does not depend on MRL training.

Q3: does the fitting data matter?

PCA has to be fit on something, which raises two practical questions: how much fitting data do you need, and does it need to come from the corpus you’ll be searching?

For the first question, I fit PCA on random samples of FiQA’s 57K documents (1,000, 5,000, 20,000, and the full corpus) and checked how much the size of the fitting sample changes retrieval quality.

For the second, I compared regular PCA against out-of-domain PCA: a projection fit once on 100,000 MS MARCO passages and reused, unchanged, on every other dataset. This is the lazy version: fit on whatever data you have at the start and never update it.

But first, a sanity check

Before running the experiment, I wanted to make sure the evaluation pipeline was producing sensible results. So I first tried to reproduce the official MTEB scores for the models.

The table below compares NDCG@10 at full dimensions: the official MTEB score vs. the score produced by my evaluation pipeline.

dataset	ada-002		3-small		qwen3-8b
	MTEB	mine	MTEB	mine	MTEB	mine
SciFact	0.7275	0.7277	0.7337	0.7296	0.7846	0.7863
NFCorpus	0.3697	0.3705	0.3833	0.3847	0.4145	0.4150
ArguAna	0.5744	0.5757	0.5549	0.5573	0.7685	0.7689
FiQA	0.4441	0.4440	0.4491	0.4484	0.6457	0.6492
SciDocs	0.1836	0.1837	0.2080	0.2077	0.3274	0.3268
Quora	0.8760	0.8759	0.8883	0.8880	0.8890	0.8901
TREC-COVID	0.6847	0.6884	0.7790	0.7775	0.9499	0.9492
Touché 2020	0.2161	0.2143	0.2428	0.2433	0.3593	0.3596

Official numbers come from the MTEB results repository (text-embedding-3-small, text-embedding-ada-002, qwen3-embedding-8b). My results are close enough to the official numbers that I’m confident the pipeline is working as expected.

Q1: which method keeps more retrieval quality?

PCA matched or outperformed MRL truncation at nearly every dimension across both MRL-trained models.

Each cell below shows how much retrieval quality survives the reduction: NDCG@10 at that dimension divided by NDCG@10 at full dimensions (1,536 for text-embedding-3-small, 4,096 for qwen3-embedding-8b), averaged over the eight datasets.

dims	3-small (MRL)		qwen3-8b (MRL)
	Truncation	PCA	Truncation	PCA
512	98%	97%	99%	98%
256	94%	95%	96%	96%
128	86%	90%	91%	91%
64	71%	82%	83%	84%
32	46%	65%	68%	71%

3-small

qwen3-8b

At 512 dimensions, MRL truncation has a slight advantage. But as the vectors get smaller, PCA catches up and either beats or matches MRL.

On text-embedding-3-small, PCA is clearly better below 256 dims (65% vs 46% retained at 32). On qwen3-embedding-8b the race is much closer (71% vs 68% at 32).

One thing I found interesting is that the MRL training in qwen3-embedding-8b is strong enough that its truncated 32-dim vectors, cut from 4,096 dimensions, retain more quality than 3-small’s cut from 1,536.

Going dataset by dataset, you find some differences:

3-small

qwen3-8b

At the smallest dimensions, PCA wins on seven of the eight datasets for text-embedding-3-small and six of the eight for qwen3-embedding-8b.

Across dimensions, PCA performs better on SciFact, FiQA, and NFCorpus for both models. ArguAna and SciDocs also favor PCA with text-embedding-3-small, although the results are less conclusive with qwen3-embedding-8b. Truncation performs better on Quora for both models. Touché favors truncation on qwen3-embedding-8b but is basically a tie on text-embedding-3-small, and the result for TREC-COVID depends on the dimension.

Q2: is it just the MRL training?

I worried that PCA might only look good on text-embedding-3-small and qwen3-embedding-8b because MRL training had already organized their embedding spaces in some convenient way. If that were true, PCA on text-embedding-ada-002, which never saw MRL training, should be noticeably worse.

For the most part, it isn’t. Down to 128 dimensions, PCA retains nearly the same share of quality on text-embedding-ada-002 as on text-embedding-3-small; below that, a small gap opens:

dims	ada-002 (no MRL)		3-small (MRL)
	Truncation	PCA	Truncation	PCA
512	96%	99%	98%	97%
256	89%	96%	94%	95%
128	83%	89%	86%	90%
64	66%	78%	71%	82%
32	41%	59%	46%	65%

PCA on text-embedding-ada-002 retains 78% at 64 dims and 59% at 32, against 82% and 65% on text-embedding-3-small. It might be that MRL genuinely helps, or simply that text-embedding-3-small is a newer, better model. I can’t separate the two with this experiment. But either way, the hypothesis that PCA only works because of MRL seems unlikely given that even PCA on the non-MRL model beats truncation on the MRL model at those dimensions (59% vs 46% at 32 dims).

Out of curiosity, I also tried truncation on text-embedding-ada-002, even though the model was never trained for it. It held up much better than I imagined, with 83% retained at 128 dimensions.

Q3: does the fitting data matter?

Compared to MRL, PCA adds more operational complexity: fitting and versioning the projection, using it while querying, updating the projection when the index changes. So I wanted to know what happens if you’re lazy: fit PCA once, when you first create the index, and never touch it again.

In-domain fit on a small sample. In this scenario, you fit PCA on the documents you have at the start (1,000, 5,000, 20,000, or all of FiQA’s 57K) and use that projection for everything you index afterwards. At this scale, it doesn’t matter much: the projection fit on 1,000 documents performs almost the same as the one fit on the full corpus, at every dimension, on both models.

3-small

qwen3-8b

Out-of-domain fit. In this scenario, the data you fit on doesn’t match the data you end up searching, either because your corpus drifted over time or because you fit on whatever was available. To simulate the extreme version, I fit PCA once on 100K generic MS MARCO passages and searched every other dataset with that projection.

On text-embedding-3-small, the transferred fit is as good or better on average from 512 through 64 dimensions. Dataset-level gaps are larger in both directions, as the chart shows. On qwen3-embedding-8b it keeps up down to 128 dimensions but falls behind at lower dimensions (56% vs 71% retained at 32 dims).

3-small

qwen3-8b

Good news for lazy people like me. Fit-sample size matters much less than I expected, and an out-of-domain fit holds up at moderate compression. Qwen at 32 dimensions is the clear exception. These patterns may be a consequence of the scale of the experiments or the particular benchmarks I used.

What about quantization?

You can also shrink vectors using a technique called quantization. Instead of reducing the number of dimensions, quantization stores each dimension using fewer bits. With int8, each dimension uses 1 byte (4x smaller than float32); with binary, just the sign bit (32x smaller). For the numbers below, int8 quantizes the document vectors while keeping queries at float32; binary quantizes both documents and queries and ranks by dot product, which is equivalent to Hamming-distance ranking.

Using quantization alone, you get these results for text-embedding-3-small at full 1,536 dimensions:

config	bytes per vector	size vs full float32	quality retained
int8	1,536	25%	100%
binary	192	3.1%	95%

You can push this further by combining quantization with truncation or PCA. The resulting vectors can be dramatically smaller while still preserving a surprising amount of retrieval quality.

For example, combining binary quantization with PCA at 512 dimensions retains 82% of the original performance while reducing the raw vector size to about 1% of the float32 baseline. That seems a bit too optimistic, so it might just be my AI psychosis. Here are the full results.

Limitations

• Few embedding models: I tested three models, and two of them are from OpenAI. Qwen already showed that a stronger MRL implementation narrows PCA’s lead, so other model families could shift the results further.
• Corpus size: I excluded the largest BEIR datasets to keep the budget under control (~$30 spent), so it’s possible PCA falls behind MRL at a scale I didn’t reach.
• Lab conditions: my evaluation runs exact search over raw vectors, which is not how an actual vector database works. A real deployment would use an approximate index, apply its own compression, rescore with full-precision vectors, and often mix in BM25 and a reranker. This could invalidate my results.
• Benchmark contamination: BEIR datasets are widely used to train embedding models, so the absolute scores are probably inflated.

Conclusion

PCA not only held its own against MRL truncation, it won on most dimensions.

On text-embedding-3-small, PCA beat MRL truncation at nearly every dimension, and by a wide margin at aggressive compression: 65% of quality retained at 32 dims against truncation’s 46%. On qwen3-embedding-8b, MRL had a small edge at 512 dimensions and was effectively tied at 256, while PCA led below that.

These results also don’t appear to be an artifact of MRL training. PCA worked similarly well on ada-002, which wasn’t trained for truncation.

It’s also lower-maintenance than I expected. Results barely changed when fitting on a small in-domain sample. An out-of-domain fit also held up at moderate compression.

Not bad for a technique that’s older than the washing machine²!

If you want to look at the data yourself, the full pipeline is in the repo.

Footnotes

1. this page explains it quite well

2. electric washing machine