l1-xrobots-noindex
The response header X-Robots-Tag: noindex tells crawlers not to index this page. A parser that only looks at<meta name="robots"> would miss it — the check must inspect response headers too.
The response header X-Robots-Tag: noindex tells crawlers not to index this page. A parser that only looks at<meta name="robots"> would miss it — the check must inspect response headers too.