DISQUS

Alpesh Nakar · 2 years ago

Since, I have /blog, I am assuming that I should have /blog/feed for example. Is that right?

This is my robots.txt http://alpesh.nakars.com/robots.txt

Any suggestions on that? Or is it forum material?

Cheers!
A

Ashwin · 2 years ago

How to I add robot.txt file to my WP blog. People say you add this and that..but they tell how yo add them!

Everton · 2 years ago

@Alpesh

Correct

@Ashwin

Creating a file is simple. Just create a new text file in notepad containing your entries and then upload it to the root directory of your site

Mr.Byte · 2 years ago

Yeah I also have added but I am not sure whether to remove tags page or not. I think at least I should change my posts in tags page to excerpt rather than what it looks like in index page.

Martin · 2 years ago

It took a while longer for my supplemental results to decrease and I heard that it could take a long time for some. Mine dropped from 12600 to 50 in the last days which is really nice.

Ashwin · 2 years ago

Sorry to bother you with noobish questions. But can you tell me where I have to upload. I have my WP in the "/" place only. So shall I upload it with the place where I have some folders like wp-admin, wp-content and things like that or should I upload to a folder before that.

Everton · 2 years ago

@Ashwin

Yes. The directory with your wp-admin etc folders is your root directory.

No question is a newbie question - always feel free to fire off anything on this site. We're all friends here!

Zane · 2 years ago

well, i'll give it go. sounds good, thanks for sharing with us

Ashwin · 2 years ago

Thanks for the help. I've uploaded. Let's wait for the result.

mosey · 2 years ago

Thanks for this tip :) I never realised the significance of robots.txt! I currently have 158 supplementary links so will see if this reduces it. I do already get a fair amount of traffic from google (as the terms are specific to mysite) but maybe this will make it even better?
I'm confused about what robots.txt does ... does this mean that the supplementary pages become main pages?

Dan and Jennifer · 2 years ago

Hey Everton,

Thanks for bringing this up - we've been runing with an old bare-bones robots.txt file which paid no attention to wordpress feeds or other such features.

I wonder if it's the robots.txt file change that increased your traffic, or coincidentally the upcoming pagerank change.

We've noticed a very significant increase in Google traffic the past two weeks, which is very similar in magnitude to the traffic increase right before the last pagerank upgrade.

Have an awesome day!
Dan

Dan and Jennifer · 2 years ago

Ok, I've just spent a whole bunch of time with my new friend from Google Webmaster tools - the robots.txt analysis tool. You paste the content of your new robots.txt and provide him a test URL and he tells you his results based on GoogleBot, AdSenseBot, and the others.

Anyway, here's the new file I came up with, WAY longer thank I'd initially hoped for.

But does cover some important items, like the "tracback" and comment feed for each post as far as GoogleBot is concerned.

Does anyone see ANYTHING wrong with this file?

Thanks!
Dan

-----
# Robots.txt file

# All robots will spider the domain
User-agent: *
Disallow: /Openads/
Disallow: /wp-
Disallow: /feed/
Disallow: /rss/
Disallow: /trackback/
Disallow: /comments/feed/
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /cgi-bin/
Disallow: /styles/
Disallow: /dnld/
Allow: /wp-content/uploads/
Allow: /wp-content/themes/mistylook/img

# GoogleBot
User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /Openads/
Disallow: /wp-
Disallow: /feed/
Disallow: /rss/
Disallow: /trackback/
Disallow: /comments/feed/
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /cgi-bin/
Disallow: /styles/
Disallow: /dnld/
Allow: /wp-content/uploads/
Allow: /wp-content/themes/mistylook/img

# allow adsense bot on entire site
User-agent: Mediapartners-Google
Allow: /*

# allow google image bot to search all images
User-agent: Googlebot-Image
Allow: /*

# allow AdWords PPC bot on entire site
User-agent: Adsbot-Google
Allow: /*

Everton · 2 years ago

I hope you get a few responses Dan as that's a long list. My gut tells me it's too long, but I'm afraid I'm not fully up to speed in this area yet.

Vijay · 2 years ago

I've added a robots.txt. It did help in filtering out google indexing my login form and the like..
I guess it would be useful

sourcer · 2 years ago

I have already added a robots.txt file on my blog and I see more visits from google but I never calculated how much in %

Ajay · 2 years ago

Thanks for the suggestions Everton. I have a basic robots.txt that I use to disallow the bot from certain sections.

How has it affected the overall traffic. i.e. the 16% increase has affected overall site traffic by how much?

Brown Baron · 2 years ago

Finally added mine. Good thing I'm here regularly otherwise I'd forget a lot of things I was planning on doing.

Everton · 2 years ago

@Ajay

the 16% increased overall traffic by around 10%

Eddie · 2 years ago

Martin told me as well about the robots.txt and this article.

Although my site is still young and I have only 31 supplemental results (I was shocked that one of my main articles was among them). So I added the robots.txt a moment ago.

Thank you for that great tip!
Eddie

mosey · 2 years ago

I also run a forum as part of my site and have found quite alot of the forum links in the supplementary search... does this mean I should add /forum to the robots.txt? A little bit confused.

Jen · 2 years ago

Hm, I haven't added a robots.txt file yet, but I think I will now! Thanks for the idea.

Zath · 2 years ago

I've now added a robots.txt file to my site, it's only basic one for wordpress blogs so far, but I'll see what, if any difference that makes to start with.

Glenn · 2 years ago

What does the below command do?

Disallow: /wp-

Does it disallow all directories that start with wp- in the WordPress blog?

I too got bit by Google around 5/1/07 and nealy all of my blog posts went into the Supplemental result decreasing my traffic and sales overall.

Hope it works.

Vikram · 2 years ago

can anyone help me on how to install the Robots.txt file ... Please help me

Please reply to vicky316[at]gmail[dot]com

Everton · 2 years ago

- open notepad
- paste the text you want to use
- save the file as robots.txt
- upload it to your blog's root directory

Zath · 2 years ago

This could be completely unrelated, but about a week after I added the above basic robots.txt file, my search engine traffic seems to have dropped by around 50%

Could this have anything to do with the robots.txt - or just a big co-incidence?

For the time being I've removed the robots.txt to see if my traffic levels go back up.

Glenn · 2 years ago

No change after about 6 days. My blog is in a subdirectory called blog (i.e. www.MySite.com/blog) on my main site. Here is the contents of my robots.txt file.

Anyone have any ideas? Thanks.

User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
User-agent: *
Disallow: /blog/wp-
Disallow: /blog/feed/
Disallow: /blog/trackback/
Disallow: /blog/rss/
Disallow: /blog/comments/feed/
Disallow: /blog/page/
Disallow: /blog/date/
Disallow: /blog/comments/
Disallow: /rsscb

Everton · 2 years ago

@Zath

Nothing in the robots.txt file I posted should decrease your Google traffic as it is pretty basic.

@Glenn

Can't see any problems. Maybe you need to wait a bit longer

Zath · 2 years ago

In that case Everton, I'll put it back, perhaps it was just a co-incidence.

I'll see how it goes in the next couple of weeks. Cheers!

Glenn · 2 years ago

I just discovered my feed is not at www.mysite.com/blog/feed but at www.mysite.com/blog/index.php/feed. So I changed my robots.txt file accordingly. Please Blog guru of the group let me know if it's correct. My site is www.SpyGearCo.com/blog

User-agent: Googlebot
Disallow: /blog/index.php/feed/$
Disallow: /blog/index.php/feed/rss/$
Disallow: /blog/index.php/trackback/$
User-agent: *
Disallow: /blog/index.php/wp-
Disallow: /blog/index.php/feed/
Disallow: /blog/index.php/trackback/
Disallow: /blog/index.php/rss/
Disallow: /blog/index.php/comments/feed/
Disallow: /blog/index.php/page/
Disallow: /blog/index.php/date/
Disallow: /blog/index.php/comments/
Disallow: /rsscb

Glenn · 2 years ago

Anyone have any thoughtst about disallowing the following directories in addition to the ones recommended in this post?

Disallow: /index.php
Disallow: /category/

Here is the post about why these are included?

http://www.earnersblog.com/removing-supplementa...

Ravi · 2 years ago

I have done the same for mine. Waiting for the result

tom · 2 years ago

Hi... Have just built my very first site and desperately trying to get it ranked etc.. haven't got a clue what robots text to use.. the theory I've read above all sounds great but all seems to relate to blogs?? Although I will at some point I guess, I don't yet use blogs..

I'm sure you guys will spot numerous errors with my 1st site so be kind!! main problem at the moment is I have several pages in the main index but a hell of a lot in the supplementary results... including some of my main pages!

I need a basic robots.txt to get the stuff out the supplementary into the main index...

I currently have no robots.txt at all

www.jdelectricalgrantham.co.uk

Help would be very much appreciated.

cheers

Eternalsoul · 2 years ago

Hi Everton
I am a newbie to blogging. I blog from the new blogger platform. Is it possible to add the robots.txt in that and if so, can you please tell me how to do so? As of now, I think I have nil traffic. I would like to build it up and monetise it.
Eternalsoul

Everton · 2 years ago

@Eternalsoul

I've never used blogger, so I wouldn't know.

@tom

adding a robots.txt file should be useful for all sites. What you need to do is go through your site, and think through what pages you want appearing and which you don't. You can also use ‘site:www.YOURDOMAIN.COM -view ***‘ to see what pages are currently appearing the index, to see what things Google is currently picking up that you rather they didn't

Ana Luiza · 2 years ago

hi Everton,
I love this article, I've been giving the robots issue some thought but have been mostly confused till now. I will probably give your robots file a try on my blog soon.

I 've got a few questions - hopefully you'll have the time to see them:

1. I have currently 732 supp pages (on my other blog) and most of them are Share-this pages generated by the plugin Share this. I'm thinking it'd be a good idea to unindex these. What's your opinion, and how do I include this piece of instruction in robots.txt?

2. I've read that unindexing feeds will prevent my blog from being included in Google Blog Search. Do you know if that's true?

3. What kind of Wordpress page has /date/ in the permalink? Is it daily archive pages?

Thank you, and I really appreciate the way you share your discoveries
Ana

Everton · 2 years ago

@Ana

1. Looking at the sahre this url it looks like the structure is: sitename/?p=1923&akst;_action=share-this, so Disallow: /?p=$ shoudl do the trick I think

2.Not sure - anyone else?

3. Yes

Zath · 2 years ago

I ended up putting the robots file back and now the supplemental links has been updated, mine dropped from around 2000 to 250! Whether that will make a big difference to my ranking and traffic, we'll see, but it sure sounds good in terms of supplemental links for what it's worth.

Max · 2 years ago

If I want to block a subdirectory only. e.g. http://www.site.com/news/welcome

What command should I sue:

1. Disallow: /news/welcome/
2. Disallow: /*/welcome/
3. Disallow: /welcome

I know number 1 should be 100% works. But sometimes, I don't want to type the exact path.

As for number 2, I think it works for google and yahoo bot only. I only these 2 which accept "*".

As for number 3, I don't know whether it works or not. As for as I understand, with or without "/" make a big differences. With "/", you are saying to that directory only. Without "/", you are saying anything start from /welcome. Please correct me if I am wrong.

So the subdirectory news/welcome also start from /welcome, so I can't sure whether Disallow: /welcome will block this subdirectory or not.

Everton · 2 years ago

I think you need:

User-agent: Googlebot
Disallow: /*/welcome/$
User-agent: *
Disallow: /news/welcome/

Hoepfully someone else will confirm

Ask Apache · 2 years ago

Bravo! I just took a look at your robots.txt file and I would recommend taking a look at the updated wordpress robots.txt file on askapache.

FreelancersAtLarge.com · 2 years ago

About three weeks ago, I finally got around to uploading the robots.txt file for www.freelancersatlarge.com (it had been way down on the list of prorities).

In three weeks, the number of links in Google's index increased by 5 times!

Webrecsol Optimizer · 11 months ago

hello friend, i rode your article as a robots.txt and you say that by robots file increase the traffic then can u clearly told me how to possible...

patrick · 10 months ago

Do you think this works on a none word press site?