VereskLearn

Module 4 · Retrieval

Lesson 4.3

Keyword search

Keyword search ranks chunks by the words they share with the question, with rare words counting most.

Lesson 22 of 622 min read

The problem

Why it matters

Without it

Vector search reads meaning. A chunk with an exact product code can rank below a chunk with a similar meaning.

With it

Keyword search counts exact words. A chunk with the rare word "form" and the word "refund" ranks above chunks that share only common words.

How it works

  • The index splits each chunk into words

    A real index stores each word with the chunks that hold it. This example splits each chunk on spaces and scans every chunk for a match. The table shows how many words each of the four FAQ chunks has.

    ChunkWords
    refund the order within 7 days6
    reset the password from sign-in5
    the refund form is on the account page8
    orders ship in 2 business days6
    Four FAQ chunks and their word counts.
    // Split each chunk on spaces and count its words.const chunks = [  "refund the order within 7 days",  "reset the password from sign-in",  "the refund form is on the account page",  "orders ship in 2 business days",];for (const text of chunks) {  console.log(text.split(" ").length, text);}

    Output

    6 refund the order within 7 days
5 reset the password from sign-in
8 the refund form is on the account page
6 orders ship in 2 business days

  • Rare words weigh more

    Each query word gets a weight from the number of chunks that contain it. "Refund" appears in 2 of the 4 chunks, so its weight is 0.69. "Form" appears in only 1 chunk, so its weight is the higher 1.20. The common word "the" appears in 3 chunks and gets only 0.36.

    Weight of each word. A word found in fewer chunks gets a larger weight.
    // N is the number of chunks. n is how many contain the word.const N = 4;const idf = (n: number) => Math.log((N - n + 0.5) / (n + 0.5) + 1);console.log("form", idf(1).toFixed(2));console.log("refund", idf(2).toFixed(2));console.log("the", idf(3).toFixed(2));

    Output

    form 1.20
refund 0.69
the 0.36

  • Length and repeats change the points

    Each word score is divided by a length term, so a longer chunk gets fewer points. The term is 2.5 for a 6-word chunk and 2.875 for an 8-word chunk. The constant k1, set to 1.5, limits how fast repeated words add points.

    6-word chunk

    • Length term: 2.5
    • Score for refund: 0.69

    8-word chunk

    • Length term: 2.875
    • Score for refund: 0.60
    The same hit for "refund" in two chunks of different length.
    // Denominator for one hit, by chunk length in words.const k1 = 1.5;const lengthTerm = (len: number) => 0.25 + 0.125 * len;const part = (len: number) => 1 + k1 * lengthTerm(len);console.log(part(6), part(8));

    Output

    2.5 2.875

  • Ranking the query "refund form"

    Each chunk gets the sum of its scores for the query words. The chunk with both words, "the refund form is on the account page", scores 1.65. The chunk about the refund order scores 0.69 from refund alone. The two chunks with neither word score 0.00.

    Keyword scores for the query "refund form", highest first.

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Expecting a synonym match. "Money back" does not match "refund" because the words differ.
  • Forgetting to lowercase the text. "Refund" and "refund" are different words unless both are lowercased.
  • Using keyword search alone for questions about meaning. Pair it with vector search (lesson 4.4).

Use it when and skip it when

Is it the right tool?

Use it when

  • The question contains a name, a code, or a rare term that must match exactly.
  • The documents use the same wording as the users, such as a product manual.

Skip it when

  • Users write in other words than the documents, such as "money back" for "refund".
  • The content has no words to split, such as images or audio.

Words to know

Terms in this lesson

Term
A word from the question or from a chunk.
Inverse document frequency
A weight that is larger for words that appear in fewer chunks.
BM25
A common keyword scoring rule that uses word weights and a length limit.
Index
A stored structure that makes lookups fast.
Full code