Semantic Search on Historical Archives: AI Finds Lost Meteorites, Rhinos and Volcano Clues in Dutch East India Company Records

Someone pointed a semantic search pipeline at the Dutch East India Company (VOC) archives and surfaced meteorite fall reports, a shipping record for three rhinoceroses, and candidate volcanic eruptions missing from existing catalogs. The corpus ran to 4.35 million pages and 5.7 million passages, and it was processed overnight on a single GPU. The result matters less for the individual finds than for what it says about how cheap archival mining has become.

What the pipeline actually did

The approach combines semantic search with multi-stage AI filtering. Instead of matching keywords, semantic search ranks passages by meaning, so a description of a fireball falling from the sky can surface even if the word meteorite never appears.

Candidates pulled out of the corpus then pass through several filtering stages, with a small judgment model helping decide which passages are worth keeping. What remained were meteorite fall reports, records of three rhinoceroses being transported, and volcanic eruption candidates not listed in existing catalogs.

The scale is the headline detail: 4.35 million pages, 5.7 million passages, one GPU, one night.

Why keyword search struggles with early modern documents

The VOC operated from the early 1600s to the late 1700s, and its records span several centuries of handwritten and transcribed material. Spelling was not standardized, vocabulary shifted, and clerks described unfamiliar events in ordinary language rather than technical terms.

A natural-phenomenon report from that era is likely to read as a plain description of what a ship's crew saw, not as a labeled event. Exact-match search misses much of that. Embedding-based retrieval is better suited to it, though that is a general property of the technique rather than something specific to this project.

Why the cascade design matters

Running a large language model over 5.7 million passages would be slow and expensive. A staged design is the practical answer: cheap steps discard the obvious non-matches, and only a small remainder gets closer scrutiny.

That is the real news here. The compute that makes a project like this feasible is now within reach of an individual with one GPU, not just a well-funded lab. Previously, a dataset this size would have required a team of readers or a dedicated digitization-and-indexing project.

What this changes for different groups

Historians and archivists. The bottleneck shifts from finding relevant passages to verifying them. Candidates still need expert review, since a model can misread context or flag an unrelated passage. But reviewing a shortlist is a far smaller job than reading everything.

Climate and earth scientists. Historical eruption and meteorite catalogs are built from scattered written sources. Any new candidate, such as the eruption leads found here, can feed into those catalogs once confirmed against other evidence. Colonial-era trading archives are plausible sources of such observations because ships recorded weather and sky events as part of daily logs, though that is an inference about record-keeping, not a claim from this project.

Developers. The pattern is reusable: embed a large corpus, retrieve broadly, then filter in stages with progressively more capable models. It applies to legal archives, old newspapers, and any large digitized collection.

Limits worth keeping in mind

The eruption finds are candidates, not confirmed events. Nothing here replaces cross-checking against other sources, and false positives are expected in any pipeline that retrieves by meaning rather than exact terms.

Quality also depends on the digitized text. Errors from transcription or optical character recognition upstream will carry through to the search results.

There is also a coverage question. Only material that has been digitized and transcribed can be searched this way, so the approach favors well-funded archives like the VOC collection and leaves less-digitized collections out.

Takeaway

The interesting part is not that AI found a rhinoceros shipping record. It is that a single-GPU, overnight job could screen millions of passages from a four-century-old archive and hand researchers a short, reviewable list. Expect more of this on other large collections, with the main open question being how many of the candidates hold up under expert checking.

Reference: https://jessewaites.com/blog/post/i-pointed-ai-at-400-years-of-archives/