RFC 9309, "Robots Exclusion Protocol", September 2022

Errata ID: 7995
Status: Reported
Type: Technical
Publication Format(s) : TEXT
Reported By: Shawn Tice
Date Reported: 2024-06-18

Section 2.2 says:

path-pattern = "/" *UTF8-char-noctl ; valid URI path pattern

It should say:

path-pattern = ("/" / "*") *UTF8-char-noctl ; valid URI path pattern

Notes:

The `path-pattern` rule requires that `/` be the first character, but the Simple Example in section 5.1 has `Disallow: *.gif$`, where the path pattern starts with a `*`. The notes preceding the example explicitly say: The "*" character designates any character, including the otherwise-required forward slash; see Section 2.2.

This seems to indicate that either the formal syntax is wrong or the guidance in section 5.1 is wrong. I assume the formal syntax is wrong.

Report New Errata