Robots.txt
• Is a file which instructs crawlers (or robots) how to crawl any page or post on your ws.
• We can hide unnecessary pages which increases our crawl budget.
• Usually done by websites with a lot of webpages, not by small blogs.
Type on Notepad:
User-agent:*
Disallow: slug of the page required
Save this notepad file as robots.txt.
Now go to Cpanel>File Manager>Public_HTML > Upload
To hide entire website from crawlers:
Type on Notepad:
User-agent:*
Disallow:/
Save this notepad file as robots.txt.
Now go to Cpanel>File Manager>Public_HTML Upload
OR
Wordpress Dashboard Settings Readings Discourage Crawlers
