Question answering over PDF pages with ColPali

Ask questions about PDFs, charts and tables included, without OCR: every page is indexed as an image, the pages that answer the question are found by late interaction, and Gemini answers from them. I built the web application around the open-source ColPali engine.

When
Feb 2025
Status
Offline The app is not hosted; the demo on this page replays a saved run.
Where
Own project
Built with
Python, ColPali, RAG, FastAPI, PostgreSQL, Svelte
Made as
Individual project

About the project

Most document search first turns a PDF into text and cuts it into chunks. Charts, tables and layout get lost on the way. ColPali skips that step: a vision language model (ColQwen2) reads each page as an image and describes it with one vector for every small patch of the page, about 750 vectors of 128 numbers per page.

A question is turned into vectors the same way, one per word piece. Each of them finds its best-matching patch on a page, and the page's score is the sum of those best matches (late interaction, or MaxSim). The best pages, and only those, go to Gemini with the question: retrieval-augmented generation, so the answer is grounded in the pages it was given, and the heat map shows where on each page the question matched.

I built the application around the open-source ColPali engine: a FastAPI back-end with a PostgreSQL database (SQLAlchemy and Alembic migrations), user accounts with JWT sign-in and bcrypt-hashed passwords, each user's own Gemini key stored encrypted with a per-user Fernet key, PDF upload with indexing progress, query history, timing for every step, and a Svelte front-end. It runs with Docker Compose.

The demo above replays one run of the application's own code on 33 public-domain pages: 19 pages of the NIST AI Risk Management Framework and 14 chart slides of the EIA Annual Energy Outlook 2026. Indexing took 68 seconds on an NVIDIA L4 GPU (about 2 seconds a page); a question is scored against all 33 pages in well under a tenth of a second, while the three Gemini answers take about 30 seconds. The answers are read from the page images by the model, so a value read off a chart can be approximate.

I built the application in February 2025 and recorded this demo in November 2026. To run its code again after almost two years, three settings were changed while it ran, without editing the application: the ColQwen2 model is pinned to its March 2025 version, because the model's repository has since moved to a new format; Gemini 2.0 Flash (experimental), which the application called, has been retired, so Gemini 2.5 Flash gives the answers instead; and the model runs on a GPU. The notebook linked in the demo shows exactly what ran.

Try it yourself

What this is: a replay of one run of my ColPali application on 33 pages from two public documents. What to do: pick a question and follow the four steps: the question becomes vectors, every page is scored, the three best pages are shown with where they matched, and Gemini answers from each page.

    Read the notebook that ran the application's code, with its output (or download it for Jupyter)

    Pick a question

    How the pages were indexed

    Loading the demo…

    How well did it do?

    I checked every answer in the demo against the pages themselves: 14 questions, three pages and three answers each. The questions were written before the run and none were dropped, but I wrote them knowing the documents, so they are easier than questions from real users. Take this as a small, honest check, not a benchmark.

    12 of 14
    questions where the page with the answer was ranked first
    14 of 14
    where it was among the three pages sent to Gemini
    8 of 14
    first answers fully correct; 5 partly right, 1 rightly said the answer was not on its page, none wrong
    5 of 28
    answers from the lower-ranked pages that answered anyway, from the wrong evidence
    Exactly right

    What GDP growth and oil price does the Counterfactual Baseline assume?

    1.7% annual GDP growth; Brent crude at $87 per barrel in 2050.

    Read straight from the table on EIA slide 5, which was ranked first.

    A misread chart

    What share of light-duty vehicles could be battery electric by 2050?

    The data provided only extends to 2045… approximately 48% to 49%.

    The right slide and roughly the right values, but the chart does run to 2050; only its axis labels stop at 2045.

    Answering the wrong question

    Which energy sources are consumed most in 2050?

    The industrial sector is projected to consume the most energy.

    From the second-ranked slide, which shows sectors, not energy sources. The first-ranked slide answered correctly: natural gas and petroleum.

    Retrieval is the strong part. The weak part is the last step: the application sends each page to Gemini on its own with the bare question, so a page without the answer sometimes gets an answer anyway. Sending the top pages together, with an instruction to answer only from them, cite the page and say when the answer is not there, or dropping pages below a score threshold, would fix most of these. Values read off charts stay approximate either way.

    How it is built

    How the ColPali application is builtA Svelte app talks to a FastAPI back end with accounts, documents and questions. PDF pages are indexed by ColQwen2 and stored as files; questions are scored against them and answered by Gemini; PostgreSQL keeps users, keys and history.BrowserFastAPI back endStorageModelsSvelte appupload, ask, see theevidenceAccounts and keysJWT sign-in, encrypted GeminikeysDocumentsupload and indexingQuestionsretrieve, answer, historyPostgreSQLusers, keys, query historyFiles per userPDFs, page images, page vectorsColQwen2the ColPali engine, on a GPUGeminianswers from a page image15243162345

    The back end is one FastAPI application, and every route except sign-in needs the user's token. Users can save their own Gemini key: it is checked against Gemini first, stored encrypted with a key of that user's own (Fernet), and shown back only to its owner. In this version the answers still use the server's key; using each user's key for their own questions is the obvious next step.

    Indexing and asking are deliberately separate. Indexing turns every page into an image and then into about 750 vectors, which takes time, so the app follows its progress; a question only has to be turned into vectors and scored against the stored pages, which takes milliseconds, before the best page goes to Gemini. The demo above replays one run of exactly this flow.

    Get in touch

    Let’s talk if you have a project in mind.