VereskLearn

Module 3 · Preparing documents

Lesson 3.2

Deduplication

Deduplication removes repeated documents, so the same text is not stored and searched twice.

Lesson 15 of 622 min read

The problem

Why it matters

Without it

The same help page is copied into three folders. Search returns three copies of one answer, and the index grows for no gain.

With it

Each document gets a fingerprint. A repeat is found by its fingerprint and skipped before it is stored.

How it works

  • Exact copies have the same bytes

    Read each file as raw bytes. Two exact copies of "faq.txt" have the same bytes, so they match. The file "2024.txt" has one different year, so its bytes differ and it does not match.

    FileTextMatches faq.txt?
    faq.txtRefunds take 5 to 7 business days-
    copy.txtRefunds take 5 to 7 business daysyes
    2024.txtRefunds take 5 to 7 days in 2024no
    Three files, compared with faq.txt.
    const a = "Refunds take 5 to 7 business days";const b = "Refunds take 5 to 7 business days";console.log("Same bytes ->", Buffer.from(a).equals(Buffer.from(b)));

    Output

    Same bytes -> true

  • A fingerprint is a short code

    SHA-256 is a fixed function that turns any text into a 64-character hex code, called a fingerprint. Changing one letter gives a different code. Trimming extra spaces or line breaks first also helps, because those change the bytes.

    Text

    • Refunds take 5 to 7 business days

    Fingerprint

    • 5d8716d1d17ffb0a
    • Full code: 64 hex characters
    One text and its SHA-256 fingerprint.
    import { createHash } from "node:crypto";const code = createHash("sha256")  .update("Refunds take 5 to 7 business days")  .digest("hex");console.log("Length ->", code.length);console.log("Start ->", code.slice(0, 16));

    Output

    Length -> 64
Start -> 5d8716d1d17ffb0a

  • Seen codes make the check

    The store keeps each fingerprint with the name of the file that first had it. When a new file arrives, its fingerprint is looked up. If the code is already stored, the new file is a duplicate of the stored name.

    FingerprintStored fileResult
    5d8716d1faq.txtnew
    5d8716d1faq.txtcopy.txt is a duplicate
    0fa05e8b2024.txtnew
    The store after three files. The first 8 characters of each code are shown.
    import { createHash } from "node:crypto";const hash = (t: string) =>  createHash("sha256").update(t).digest("hex").slice(0, 8);const faq = "Refunds take 5 to 7 business days";const seen = new Map([[hash(faq), "faq.txt"]]);const code = hash(faq);console.log(code, "->", seen.get(code));

    Output

    5d8716d1 -> faq.txt

  • A duplicate is skipped before chunking

    A duplicate is not chunked or embedded again, so the index does not grow. The log records the match, such as "copy.txt is a copy of faq.txt". The check must run before chunking, because each copy would otherwise create its own chunks.

    1. copy.txt
    2. Fingerprint seen before
    3. Skip and log
    4. Not chunked
    A file with a seen fingerprint is logged and skipped.

  • Near duplicates share most words

    Near duplicates differ by a few words, such as the year "2024". An exact fingerprint misses them. Word overlap measures the share of words two texts have in common, divided by all distinct words. Here it is 6 shared words out of 9 distinct words, or 0.67. At scale, MinHash keeps a short signature per document, so full texts are not compared in pairs.

    Word overlap with faq.txt. The copy shares every word.
    const words = (t: string) => new Set(t.toLowerCase().split(" "));const a = words("Refunds take 5 to 7 business days");const b = words("Refunds take 5 to 7 days in 2024");const shared = [...a].filter((w) => b.has(w)).length;const all = new Set([...a, ...b]).size;console.log("Overlap ->", (shared / all).toFixed(2));

    Output

    Overlap -> 0.67

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Hashing text before cleaning it. Extra spaces or line breaks change the fingerprint, so normalize first.
  • Treating near duplicates as exact copies. A changed year gives a different answer, so review near copies.
  • Keeping the wrong copy. Keep one copy, and log the others with the name of the copy kept.

Use it when and skip it when

Is it the right tool?

Use it when

  • The same document arrives from several folders or exports.
  • Search results repeat the same passage again and again.

Skip it when

  • Each document is unique by design, such as one record per customer.
  • The collection is small and checked by hand.

Words to know

Terms in this lesson

Fingerprint
A short code made from a file that changes when the file changes.
Hash
A fixed-size code computed from input of any size.
SHA-256
A common hash function that makes a 64-character hex code.
MinHash
A method that estimates how much two texts overlap, using short signatures.
Full code