How JustAnswer Works:
  • Ask an Expert
    Experts are full of valuable knowledge and are ready to help with any question. Credentials confirmed by a Fortune 500 verification firm.
  • Get a Professional Answer
    Via email, text message, or notification as you wait on our site.
    Ask follow up questions if you need to.
  • 100% Satisfaction Guarantee
    Rate the answer you receive.
Ask Russell H. Your Own Question
Russell H.
Russell H., Programmer & Coder
Category: Programming
Satisfied Customers: 10803
Experience:  Programmer & Coder
13568747
Type Your Programming Question Here...
Russell H. is online now

Website robots.txt question: I have bought a domain which

Resolved Question:

Website robots.txt question:
I have bought a domain which I want to start using to create a website. When I have tried to check the history of it to ensure no bad backlinks etc, it shows it has a robots.txt file. I understand this stops robots from crawling the site.
Will this stop the whole site from being crawled and therefore stop my site from potentially ranking?
If I want search engines to be able to crawl my site, what do I need to do? I have not done anything with the site yet.
Submitted: 3 years ago.
Category: Programming
Expert:  Russell H. replied 3 years ago.
Hi, thank you for contacting JustAnswer.com. My name is Russell. I will do my best to provide the right answer to your question.

The disouragement of 'crawling' by a robots.txt file depends upon what the file contains, when you open the robots.txt file.

What does yours contain?

If you want to simply remove the robots.txt file, do so (but keep a backup, unless it is an empty file, under another name such as
backup-robots.txt-bak
and that will allow crawlers to go over the whole site.)
Customer: replied 3 years ago.

Hi


 


I'm not sure as I have purchased the domain but haven't done anything to it yet.


 


If you are able to check, it is uplifthealth.com


 


Are you able to advise?

Expert:  Russell H. replied 3 years ago.
The file

uplifthealth.com/robots.txt

has this content in it:

User-agent: * Disallow: /

which means basically that all 'agents' (webcrawlers, spiders) are disallowed from the base of the site upward in the site's file structure (in other words all of it pretty much.)

Customer: replied 3 years ago.

1. I assume this is not good for a website to have if I need it to be ranked in the search engines?


 


2. Is it not standard to have this file?


 


3. What do I need to do to remove this file?


 


Thanks in advance.

Expert:  Russell H. replied 3 years ago.
1. The sub-pages won't be indexed by search engines. Just the base page (at most.)

2. It is not standard. It just is a choice one can have. You can or can not have the file there. Having it there while the site is being built prevents half-formed pages from being indexed, I guess.

3. You need to have administrative access, to build the site by adding or removing files from it. Then you can simply remove the robots.txt file from the site like any other... unless your site hosting provider for some eccentric reason forbids that for whatever reason.


If this answers your question adequately, please rate my Answer - hopefully favorably - so that I may be paid for the work I have done on your question. Thanks.
Customer: replied 3 years ago.

Sorry, just to clear up any confusion, please can you advise how I remove the file. I have not done anything with the domain yet and plan on installing word press. At the moment it is hosted by Hostgator.


 


Thanks.

Expert:  Russell H. replied 3 years ago.

Here's how to remove the robots.txt file:

- log in to cpanel.
- in the Files section, click on the File Manager icon.
- select the Web Root directory.
- make sure the Show hidden files (dotfiles) option is checked.
- click the Go button to continue.
- click and highlight the file robots.txt in the file listings.
- from the top toolbar, select Delete.
- click the Delete File(s) button on the popup to delete the file.


If this is an adequate Answer to your question, please rate my Answer - favorably I hope - so that I may be paid for the work I have done on your question.

Russell H. and other Programming Specialists are ready to help you

Related Programming Questions