How JustAnswer Works:

  • Ask an Expert
    Experts are full of valuable knowledge and are ready to help with any question. Credentials confirmed by a Fortune 500 verification firm.
  • Get a Professional Answer
    Via email, text message, or notification as you wait on our site.
    Ask follow up questions if you need to.
  • 100% Satisfaction Guarantee
    Rate the answer you receive.

Ask Russell H. Your Own Question

Russell H.
Russell H., Programmer & Coder
Category: Programming
Satisfied Customers: 10478
Experience:  Programmer & Coder
13568747
Type Your Programming Question Here...
Russell H. is online now

Website robots.txt question: I have bought a domain which

Resolved Question:

Website robots.txt question:
I have bought a domain which I want to start using to create a website. When I have tried to check the history of it to ensure no bad backlinks etc, it shows it has a robots.txt file. I understand this stops robots from crawling the site.
Will this stop the whole site from being crawled and therefore stop my site from potentially ranking?
If I want search engines to be able to crawl my site, what do I need to do? I have not done anything with the site yet.
Submitted: 2 years ago.
Category: Programming
Expert:  Russell H. replied 2 years ago.
Hi, thank you for contacting JustAnswer.com. My name is Russell. I will do my best to provide the right answer to your question.

The disouragement of 'crawling' by a robots.txt file depends upon what the file contains, when you open the robots.txt file.

What does yours contain?

If you want to simply remove the robots.txt file, do so (but keep a backup, unless it is an empty file, under another name such as
backup-robots.txt-bak
and that will allow crawlers to go over the whole site.)
Customer: replied 2 years ago.

Hi


 


I'm not sure as I have purchased the domain but haven't done anything to it yet.


 


If you are able to check, it is uplifthealth.com


 


Are you able to advise?

Expert:  Russell H. replied 2 years ago.
The file

uplifthealth.com/robots.txt

has this content in it:

User-agent: * Disallow: /

which means basically that all 'agents' (webcrawlers, spiders) are disallowed from the base of the site upward in the site's file structure (in other words all of it pretty much.)

Customer: replied 2 years ago.

1. I assume this is not good for a website to have if I need it to be ranked in the search engines?


 


2. Is it not standard to have this file?


 


3. What do I need to do to remove this file?


 


Thanks in advance.

Expert:  Russell H. replied 2 years ago.
1. The sub-pages won't be indexed by search engines. Just the base page (at most.)

2. It is not standard. It just is a choice one can have. You can or can not have the file there. Having it there while the site is being built prevents half-formed pages from being indexed, I guess.

3. You need to have administrative access, to build the site by adding or removing files from it. Then you can simply remove the robots.txt file from the site like any other... unless your site hosting provider for some eccentric reason forbids that for whatever reason.


If this answers your question adequately, please rate my Answer - hopefully favorably - so that I may be paid for the work I have done on your question. Thanks.
Customer: replied 2 years ago.

Sorry, just to clear up any confusion, please can you advise how I remove the file. I have not done anything with the domain yet and plan on installing word press. At the moment it is hosted by Hostgator.


 


Thanks.

Expert:  Russell H. replied 2 years ago.

Here's how to remove the robots.txt file:

- log in to cpanel.
- in the Files section, click on the File Manager icon.
- select the Web Root directory.
- make sure the Show hidden files (dotfiles) option is checked.
- click the Go button to continue.
- click and highlight the file robots.txt in the file listings.
- from the top toolbar, select Delete.
- click the Delete File(s) button on the popup to delete the file.


If this is an adequate Answer to your question, please rate my Answer - favorably I hope - so that I may be paid for the work I have done on your question.

Russell H., Programmer & Coder
Category: Programming
Satisfied Customers: 10478
Experience: Programmer & Coder
Russell H. and other Programming Specialists are ready to help you

What Customers are Saying:

 
 
 
  • My Expert answered my question promptly and he resolved the issue totally. This is a great service. I am so glad I found it I will definitely use the service again if needed. One Happy Customer
< Previous | Next >
  • My Expert answered my question promptly and he resolved the issue totally. This is a great service. I am so glad I found it I will definitely use the service again if needed. One Happy Customer
  • Wonderful service, prompt, efficient, and accurate. Couldn't have asked for more. I cannot thank you enough for your help. Mary C.
  • This expert is wonderful. They truly know what they are talking about, and they actually care about you. They really helped put my nerves at ease. Thank you so much!!!! Alex
  • Thank you for all your help. It is nice to know that this service is here for people like myself, who need answers fast and are not sure who to consult. GP
  • I couldn't be more satisfied! This is the site I will always come to when I need a second opinion. Justin
  • Just let me say that this encounter has been entirely professional and most helpful. I liked that I could ask additional questions and get answered in a very short turn around. Esther
  • Wonderful service, prompt, efficient, and accurate. Couldn't have asked for more. I cannot thank you enough for your help. Mary C.
 
 
 

Meet The Experts:

 
 
 
  • ATLPROG

    ATLPROG

    Computer Software Engineer

    Satisfied Customers:

    7260
    MS in IT.Several years of programming experience in Java C++ C C# Python VB Javascript HTML
< Previous | Next >
  • http://ww2.justanswer.co.uk/uploads/SP/spatlanta2010/2011-6-23_12450_photo.64x64.gif ATLPROG's Avatar

    ATLPROG

    Computer Software Engineer

    Satisfied Customers:

    7260
    MS in IT.Several years of programming experience in Java C++ C C# Python VB Javascript HTML
  • http://ww2.justanswer.co.uk/uploads/ComputersGuru/2010-02-13_051118_Photo41.JPG LogicPro's Avatar

    LogicPro

    Computer Software Engineer

    Satisfied Customers:

    5514
    Expert in C, C++, Java, DOT NET, Python, HTML, Javascript, Design.
  • http://ww2.justanswer.co.uk/uploads/lifesaver333/2010-10-17_191349_ls.jpeg lifesaver's Avatar

    lifesaver

    Computer Software Engineer

    Satisfied Customers:

    936
    Several years of intensive programming and application development experience in various platforms.
  • http://ww2.justanswer.co.uk/uploads/EH/ehabtutor/2012-8-2_202016_1.64x64.jpg ehabtutor's Avatar

    ehabtutor

    Computer Software Engineer

    Satisfied Customers:

    766
    Bachelor of computer science, 5+ years experience in software development, software company owner
  • http://ww2.justanswer.co.uk/uploads/eljonis/2010-01-06_130406_eljon2.jpg Eljon's Avatar

    Eljon

    Consultant

    Satisfied Customers:

    558
    Founder of StockCanvas.com
  • http://ww2.justanswer.co.uk/uploads/FA/fael/2011-1-15_104041_profile500x500.64x64.jpg Rafael Martins's Avatar

    Rafael Martins

    Computer Software Engineer

    Satisfied Customers:

    446
    Desktop, Mobile and Web Developer. 7+ years of experience. Creative solutions provider.
  • http://ww2.justanswer.co.uk/uploads/CH/chutz747/2011-11-6_14231_me.64x64.jpg The-PC-Guy's Avatar

    The-PC-Guy

    Software Engineer

    Satisfied Customers:

    325
    Extensive Knowledge in PHP, MYSQL, CSS & Javascript
 
 
 

Related Programming Questions