Robots.txt validator

Well, I just used this, and the site asked for a link to the site if you found the link useful, so here it is.

A robots.txt file is something you put on your website if you don’t want search engine crawlers to go through the contents of something on your site. You could exclude everything, or just certain folders (maybe yoursite.com/temp), and you can say which bots you do/don’t want to index your site too.

If you want to learn more about this, go here. 

I had a slight shock when googling my name and finding a uni presentation that I’m not particularly proud of (Career Management Studies, anyone? Bleh.)

2 years ago · 0 notes