SlingBlog.com

SEO and Robots.txt

Well, not giving robots.txt some respect finally bit me.

I bought Texas Pitmaster domain about a month and a half ago for a long term project. Traffic will come from all angles, but primarily organic search results. So, I started adding content and socializing the site to help it get indexed faster. 100% of the time my method has worked in the past, until this site. Which really threw me off.

The other sites I started were getting index fine, but this one just would not go into any search engine.

So, I was thinking that I bought a banned domain. I went to the wayback machine to see if I could see an archive of the site to see what the previous owner did to get the domain banned.

Oddly enough, I get an error saying that the robots.txt file was blocking the way back machine from crawling the site.

So, I go out to my FTP — NO ROBOTS.TXT file at all. Really freaking strange.

So, as a last ditch effort I hook up Google Webmaster tools to it and it came back with the same thing. Robots.txt file was saying block all bots and spiders from indexing even though there was no robots.txt file associated with my new site.

Frustrating.

So, I went ahead and created a valid robots.txt file that allows all and I uploaded it yesterday.

Today, I log into GWT and sure enough it picked up the robots.txt file and has started indexing my site.

The Lesson
Apparently Google caches a copy of the robots.txt file for every domain and uses that file until a new one is found. If one is not found at all, it uses the last robots.txt file it found.

So, if you are doing any SEO type projects with your campaigns be sure to create a new robots.txt file just in case Google has a cached one from a previous owner.