VereskLearn

Module 3 · Preparing documents

Lesson 3.1

Format detection

Format detection reads the first bytes of a file to find its real type, instead of trusting the file name.

Lesson 14 of 622 min read

The problem

Why it matters

Without it

A file named "report.pdf" may really be plain text. The PDF reader then fails, or it returns garbled text.

With it

The first bytes name the real type. The file goes to the right reader, and a mismatch with its name is reported.

How it works

  • Files start with fixed bytes

    Many file types start with a fixed pattern of bytes, called a signature. A byte is a number from 0 to 255, written here in hexadecimal. A PDF file starts with 25 50 44 46, which spells "%PDF" in text.

    FileFirst bytes (hex)Text they show
    manual.pdf25 50 44 46%PDF
    logo.png89 50 4E 47one non-text byte, then PNG
    report.pdf52 65 66 75Refu
    The first four bytes of three uploaded files.
    const head = Buffer.from("%PDF-1.7 fake body").subarray(0, 4);const hex = [...head].map((b) => b.toString(16).toUpperCase());console.log("Bytes ->", hex.join(" "));

    Output

    Bytes -> 25 50 44 46

  • The first bytes are compared with known signatures

    The program compares the first bytes of a file with a short list of known signatures. Only the first few bytes are read, so the check is fast. The PNG signature 89 50 4E 47 does not match the PDF signature, so the two files are told apart.

    1. First bytes
    2. Compare with known list
    3. Matching type
    4. Reader
    The check runs on the first bytes only, not the whole file.
    const signatures = [  { type: "pdf", bytes: [0x25, 0x50, 0x44, 0x46] },  { type: "png", bytes: [0x89, 0x50, 0x4e, 0x47] },];const head = [0x89, 0x50, 0x4e, 0x47, 0x0d];const match = signatures.find((s) =>  s.bytes.every((b, i) => head[i] === b),);console.log("Type ->", match?.type ?? "unknown");

    Output

    Type -> png

  • The name is only a label

    The extension, such as ".pdf", is a label that anyone can set. For "report.pdf", the name says PDF, but the first bytes are plain text with no known signature. The name check then reports that the name and the content do not match. A matching signature does not prove the file is clean, so readers still handle errors.

    File name

    • report.pdf
    • Name says: pdf

    First bytes

    • Plain text: "Refu"
    • Detected: unknown
    The name and the content of report.pdf disagree.
    const name = "report.pdf";const ext = name.split(".").pop();// The byte check found no signature for this file.const detected = "unknown";const note = detected === ext ? "match" : "no name match";console.log(ext, "vs", detected, note);

    Output

    pdf vs unknown no name match

  • Unknown types are not guessed

    A file with no known signature goes to a review list. The program does not guess its type, because a wrong guess can corrupt the text. Plain text has no signature, so it needs a separate check that this lesson skips. Each detected type goes to its own reader, so manual.pdf goes to the PDF reader.

    1. report.pdf
    2. No known signature
    3. Review list
    The path for a file with no known signature.

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Trusting the extension alone. A renamed file then goes to the wrong reader.
  • Reading the whole file to find its type. The first few bytes are enough.
  • Guessing a type for unknown files. Set them aside and review them.

Use it when and skip it when

Is it the right tool?

Use it when

  • Users upload files, and their names cannot be trusted.
  • Your pipeline reads several formats, such as PDF and plain text.

Skip it when

  • Every file comes from one tool that always writes the same format.
  • You accept one format only, and a failed read is shown to the user anyway.

Words to know

Terms in this lesson

Signature
The fixed bytes at the start of a file that name its type.
Extension
The end of a file name, such as ".pdf", used as a label.
Byte
A number from 0 to 255, the basic unit of stored data.
Reader
A program that reads one file type and returns its text.
Full code