VereskLearn

Module 3 · Preparing documents

Lesson 3.3

PII scrubbing

PII scrubbing replaces personal information, such as emails and phone numbers, before the text is stored.

Lesson 16 of 622 min read

The problem

Why it matters

Without it

A support ticket with a customer email is stored and returned in search results. Anyone who can search can read the address.

With it

The email becomes a placeholder, such as [EMAIL], before storage. Search and answers never show the address.

How it works

  • Decide which values count as PII

    PII is personal information that identifies a person. This lesson handles two kinds: email addresses and phone numbers in the 555-123-4567 format. Write the list before any code, so the rules can be reviewed.

    KindExamplePlaceholder
    Emailana.lee@example.com[EMAIL]
    Phone555-123-4567[PHONE]
    The two kinds handled in this lesson.

  • A pattern describes the shape of a value

    A pattern is a regular expression, a short rule for the shape of text. The phone pattern matches three digits, a dash, three digits, a dash, and four digits. The g flag finds every match in the text, not only the first one.

    • ana.lee@example.comemail pattern
    • 555-123-4567phone pattern
    The two values that the patterns find in one ticket.
    const email = /[a-z0-9._]+@[a-z0-9.]+/g;const phone = /[0-9]{3}-[0-9]{3}-[0-9]{4}/g;const ticket = "Ana wrote from ana.lee@example.com, " +  "call 555-123-4567.";console.log("Emails ->", ticket.match(email));console.log("Phones ->", ticket.match(phone));

    Output

    Emails -> [ 'ana.lee@example.com' ]
Phones -> [ '555-123-4567' ]

  • Each match becomes a placeholder

    Each match is replaced with its placeholder, such as [EMAIL]. The sentence keeps its meaning, so a reader can still see that an email was present. These patterns cannot find names, so names need a separate review.

    Before

    • Ana wrote from ana.lee@example.com,
    • call 555-123-4567.

    After

    • Ana wrote from [EMAIL],
    • call [PHONE].
    The ticket before and after scrubbing.
    const pattern = /[a-z0-9._]+@[a-z0-9.]+/g;const ticket = "Ana wrote from ana.lee@example.com, " +  "call 555-123-4567.";console.log(ticket.replace(pattern, "[EMAIL]"));

    Output

    Ana wrote from [EMAIL], call 555-123-4567.

  • Scrub before anything is stored

    Scrub each document before chunking and embedding. A ticket stored before scrubbing keeps "ana.lee@example.com" in every copy of its chunks. Scrubbing first means the address never reaches the store.

    1. Document
    2. Scrub PII
    3. Chunk
    4. Embed and store
    Scrubbing sits before the chunks and vectors are made.

  • Test with known examples

    Keep a small set of texts with known personal values. Run the scrubber on each one and check that every value is replaced. Count the misses, such as 1 phone number left in place out of 20 tests.

    Replaced values in a test set of 20, with one miss.
    const tests = 20;const missed = 1;console.log("Replaced ->", tests - missed, "of", tests);

    Output

    Replaced -> 19 of 20

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Trusting one pattern. A new format, such as "(555) 123 4567", slips through, so test on real samples.
  • Scrubbing after storage. Copies already in the index keep the original value.
  • Forgetting names and street addresses. Simple patterns cannot find them, so add a review step.

Use it when and skip it when

Is it the right tool?

Use it when

  • Documents may contain customer emails, phone numbers, or account numbers.
  • The text will be sent to a model or stored in a shared index.

Skip it when

  • The documents are public manuals with no personal data.
  • You need the exact value for a lookup. Use a separate, protected store instead.

Words to know

Terms in this lesson

PII
Personal information, such as a name, email, or phone number, that identifies a person.
Pattern
A rule that describes the shape of a value, such as a phone number.
Placeholder
A label that stands in for a removed value, such as [EMAIL].
Regular expression
A short rule written in code that matches text by its shape.
Full code