How JustAnswer Works:
  • Ask an Expert
    Experts are full of valuable knowledge and are ready to help with any question. Credentials confirmed by a Fortune 500 verification firm.
  • Get a Professional Answer
    Via email, text message, or notification as you wait on our site. Ask follow up questions if you need to.
  • 100% Satisfaction Guarantee
    Rate the answer you receive.
Ask Russell H. Your Own Question
Russell H.
Russell H., Programmer & Coder
Category: Programming
Satisfied Customers: 12220
Experience:  Programmer & Coder
Type Your Programming Question Here...
Russell H. is online now

Website robots.txt question: I have bought a domain which

This answer was rated:

Website robots.txt question:
I have bought a domain which I want to start using to create a website. When I have tried to check the history of it to ensure no bad backlinks etc, it shows it has a robots.txt file. I understand this stops robots from crawling the site.
Will this stop the whole site from being crawled and therefore stop my site from potentially ranking?
If I want search engines to be able to crawl my site, what do I need to do? I have not done anything with the site yet.
Hi, thank you for contacting My name is Russell. I will do my best to provide the right answer to your question.

The disouragement of 'crawling' by a robots.txt file depends upon what the file contains, when you open the robots.txt file.

What does yours contain?

If you want to simply remove the robots.txt file, do so (but keep a backup, unless it is an empty file, under another name such as
and that will allow crawlers to go over the whole site.)
Customer: replied 3 years ago.



I'm not sure as I have purchased the domain but haven't done anything to it yet.


If you are able to check, it is


Are you able to advise?

The file

has this content in it:

User-agent: * Disallow: /

which means basically that all 'agents' (webcrawlers, spiders) are disallowed from the base of the site upward in the site's file structure (in other words all of it pretty much.)

Customer: replied 3 years ago.

1. I assume this is not good for a website to have if I need it to be ranked in the search engines?


2. Is it not standard to have this file?


3. What do I need to do to remove this file?


Thanks in advance.

1. The sub-pages won't be indexed by search engines. Just the base page (at most.)

2. It is not standard. It just is a choice one can have. You can or can not have the file there. Having it there while the site is being built prevents half-formed pages from being indexed, I guess.

3. You need to have administrative access, to build the site by adding or removing files from it. Then you can simply remove the robots.txt file from the site like any other... unless your site hosting provider for some eccentric reason forbids that for whatever reason.

If this answers your question adequately, please rate my Answer - hopefully favorably - so that I may be paid for the work I have done on your question. Thanks.
Customer: replied 3 years ago.

Sorry, just to clear up any confusion, please can you advise how I remove the file. I have not done anything with the domain yet and plan on installing word press. At the moment it is hosted by Hostgator.



Here's how to remove the robots.txt file:

- log in to cpanel.
- in the Files section, click on the File Manager icon.
- select the Web Root directory.
- make sure the Show hidden files (dotfiles) option is checked.
- click the Go button to continue.
- click and highlight the file robots.txt in the file listings.
- from the top toolbar, select Delete.
- click the Delete File(s) button on the popup to delete the file.

If this is an adequate Answer to your question, please rate my Answer - favorably I hope - so that I may be paid for the work I have done on your question.

Russell H. and other Programming Specialists are ready to help you