A search engine does three jobs. Crawling discovers pages, indexing organizes them into a searchable catalog, and ranking sorts them by relevance and authority every time someone types a query. That pipeline has ordered the web for over two decades. A fourth step is now being added on top: generative engines such as ChatGPT, Perplexity, and Google's AI Overviews no longer return a list of links. They write the answer.
A search engine is a system that discovers web pages through crawling, stores them in a structured index, and ranks them by relevance and authority in response to a query.
This article walks through each stage, including a little of the math underneath, and then looks at what changes when the engine generates instead of retrieves.
How does crawling work?
Picture the internet as a giant graph, G = (V, E), where V is the set of web pages and E is the set of links between them.
Search engines send out crawlers, programs that move from node to node by following edges. In the simplest model, a crawler behaves like a random walker in graph theory: the probability of moving from page u to page v is P(v|u) = 1/outdeg(u), one divided by the number of outgoing links on u.
Real crawlers are not random. They prioritize pages by freshness, authority, and how often they change. The core task stays the same, though: crawling is the discovery of nodes in an enormous graph that keeps growing.
AI crawlers now walk the same graph. GPTBot, ClaudeBot, and PerplexityBot fetch pages to feed generative answers, and they follow the same robots.txt conventions as their search predecessors.
What happens during indexing?
Once a page is discovered, it has to be stored so it can be found again in milliseconds. Indexing transforms raw HTML into structured data.
The core structure is the inverted index. Instead of mapping each page to the words it contains, it maps each word to the set of pages containing it: I("AI") = {p1, p2, p3, ...}.
A query then becomes set intersection. Search "how engines work" and the system computes I("how") ∩ I("engines") ∩ I("work"), returning only the documents that contain all three terms. It works like a library catalog, except the library holds billions of volumes and answers instantly.
How does ranking decide what you see?
Not all pages that match a query deserve the same position. Ranking blends many signals into one score. A simplified scoring function looks like this: S(d, q) = w1·R(d, q) + w2·A(d) + w3·U(d), where:
- R(d, q) is the relevance of document d to query q
- A(d) is authority, built from backlinks and trust
- U(d) covers user experience signals such as speed and mobile usability
- w1, w2, w3 are weights the engine tunes constantly
The most famous authority measure is PageRank, developed by Google's founders in the late 1990s. In simplified form: PR(P) = (1 - d)/N + d · ∑ PR(Q)/L(Q), summed over every page Q that links to P. Here d is a damping factor around 0.85, N is the total number of pages, and L(Q) is the count of outbound links from Q. The intuition is elegant: a page is important if other important pages link to it.
Modern engines go further. Machine learning models estimate P(relevant | q, d), the probability that document d satisfies query q, and they learn it from billions of past queries, clicks, and satisfaction signals. The question has shifted from "does this page contain the right words" to "does this page answer what the user means".
What changes with generative engines?
The pipeline above ends with a ranked list of links. Generative engines replace that final step with synthesis.
A traditional engine follows: query, retrieve documents, rank results. A generative engine follows: query, retrieve knowledge, then generate a single synthesized answer through model inference. ChatGPT, Perplexity, Gemini, and AI Overviews all work this way.
In mathematical terms, the objective changes. A search engine maximizes P(relevant | q, d) document by document. A generative engine maximizes P(answer | q, K), where K combines retrieved documents with the knowledge already encoded in the model's parameters. The technique that merges the two is retrieval-augmented generation, which we unpack in our guide to RAG.
The practical difference is stark. A search engine points you at ten documents. A generative engine gives you one answer and cites a handful of sources, sometimes none.
How do you get picked when the engine writes the answer?
Every generative answer is the output of a filtering process:
- The engine interprets the question and often rewrites it into several search queries.
- A retrieval layer pulls candidate pages, usually from a conventional search index.
- The model selects the few passages it considers trustworthy and relevant enough to use.
- It writes the answer, citing some of those passages as sources.
Each step eliminates candidates. Being crawlable gets you into step 2. Clear structure, direct definitions, and verifiable claims get you through steps 3 and 4. Files like llms.txt help models find your best pages, and understanding how large language models process text explains why quotable phrasing wins.
Visibility used to mean a position on page one. It increasingly means being the source inside the answer. We built RankWit to measure exactly that: which prompts mention you, which engines cite you, and how both change over time.
FAQ
Do generative engines still use search indexes?
Yes, heavily. When ChatGPT browses, Perplexity answers, or Google generates an AI Overview, a retrieval layer first pulls candidate documents, typically from a web-scale search index. The model then synthesizes an answer from those documents. Classic crawling and indexing remain the foundation; generation is a layer on top rather than a replacement.
Is traditional SEO dead?
No. Generative engines retrieve before they generate, and retrieval still rewards crawlable sites, fast pages, and authoritative content. What changes is the last step: instead of competing for a ranked position, you compete to be quoted inside an answer. Solid SEO remains the entry ticket; GEO decides whether you get cited.
What is the difference between ranking and citation?
Ranking orders whole documents on a results page, and users choose where to click. Citation happens after a model has read several documents: it selects specific passages, synthesizes an answer, and credits a few sources. You can rank well and never be cited, or be cited from a page that ranks modestly, because selection happens at passage level.
What is GEO?
GEO, Generative Engine Optimization, is the practice of making content easy for AI engines to retrieve, understand, and cite. It builds on SEO but optimizes for answers instead of rankings: clear definitions, structured pages, machine-readable signals, and measurement of how often engines like ChatGPT, Perplexity, and Gemini mention your brand.