MCQ Hive stays free thanks to ads. Please disable your ad blocker for this site so content and features keep working.
Search Engine Optimization
Technical SEO
Beginner
Q
What does the robots.txt rule 'User-agent: *' followed by 'Disallow:' (with an empty value) mean?
AAll crawlers may crawl everything
BAll crawlers are blocked from the whole site
CIt is invalid syntax
DOnly Googlebot is blocked
Correct Answer:
A. All crawlers may crawl everything
Explanation:
'Disallow: /' blocks the whole site. An empty Disallow allows everything, a common confusion.
ALSO KNOW (related points):
• Are the paths in robots.txt case-sensitive? → Yes; Disallow: /Private does not block /private
• What is the SEO problem with redirect chains (A → B → C → D)? → They waste crawl budget, slow users down, and crawlers may stop following after several hops
