Connected Internet - Latest Comments in Creating A WordPress Robots.txt To Improve SEO

Re: Creating A WordPress Robots.txt To Improve SEO

livereceptions — Thu, 24 Jun 2010 23:31:13 -0000

I like this site, it is nice.
SEO is so important to your site, you also can
do SEO by yourself. see my site
http://www.livereceptions.n...
get free quote.

Re: Creating A WordPress Robots.txt To Improve SEO

learningquranonline — Mon, 28 Dec 2009 04:38:03 -0000

if any body could guide me that well how can i up load this robot text to my word press blog http://learnquranonline1.wo...

Re: Creating A WordPress Robots.txt To Improve SEO

Keresőoptimalizálás Könyv — Sun, 18 Jan 2009 05:10:09 -0000

I think the following site describes perfectly the seo adjustments for a wordpress blog's robots.txt:
http://codex.wordpress.org/...

Re: Creating A WordPress Robots.txt To Improve SEO

George Donnelly — Wed, 16 Apr 2008 11:41:04 -0000

What about to block search results?

Re: Creating A WordPress Robots.txt To Improve SEO

LiveJasmin — Sun, 30 Dec 2007 13:42:49 -0000

should this be ok ?

User-agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/

User-agent: Googlebot-Image
Disallow:
Allow: /*

Re: Creating A WordPress Robots.txt To Improve SEO

AskApache — Tue, 06 Nov 2007 04:04:41 -0000

I finally got my site indexed near perfectly thanks to much reading and viewing my access logs.. Based mostly on this wordpress robots.txt example

Re: Creating A WordPress Robots.txt To Improve SEO

stacey — Wed, 24 Oct 2007 21:11:46 -0000

I fixed the robots.txt. I wasnt putting it in the root directory.

Re: Creating A WordPress Robots.txt To Improve SEO

stacey — Tue, 23 Oct 2007 16:00:37 -0000

Hi Everett,

I installed the KB robots.txt plugin. This what I entered in the robots.txt plugin window.
User-agent: *

Disallow:
But when I do I do www.babygeartoday.com/robot... this is what I get, and in the plugin when it says check the the robots.txt file after I submit I get this also:
# BEGIN XML-SITEMAP-PLUGIN
Sitemap: http://www.babygeartoday.co...
# END XML-SITEMAP-PLUGIN

So uninstalled the google sitemap and analytics, and still the same thing. I was wondering could you help me solve this problem.

Thanks,
Stacey

Re: Creating A WordPress Robots.txt To Improve SEO

dwi kristianto — Sun, 14 Oct 2007 09:47:24 -0000

hi, thanks for posting this.
i use it all unkess :
Disallow: /page/ Disallow: /date/ Disallow: /comments/

Re: Creating A WordPress Robots.txt To Improve SEO

AskApache — Wed, 10 Oct 2007 21:19:31 -0000

Have you read the new robots.txt recently updated article yet? and check out askapache.com/robots.txt

Re: Creating A WordPress Robots.txt To Improve SEO

Everton — Wed, 10 Oct 2007 15:07:43 -0000

Thx Sue for reminding me I need to check my file

Re: Creating A WordPress Robots.txt To Improve SEO

Sue — Wed, 10 Oct 2007 15:00:58 -0000

FWIW: this tester shows a lot of errors:

http://www.searchengineprom...

Sue

Re: Creating A WordPress Robots.txt To Improve SEO

Paula Mooney — Tue, 09 Oct 2007 12:38:13 -0000

Thank you so much for this post.

I found out I had my robots.txt file in the wrong place anyway!

Paula

Re: Creating A WordPress Robots.txt To Improve SEO

Ask Apache — Thu, 09 Aug 2007 09:18:48 -0000

Hey Everton just stopped by to let you know I created a simpler and updated robots.txt file for WordPress, and I used a lot of the recommendations from your visitor comments.

Re: Creating A WordPress Robots.txt To Improve SEO

Ramil — Mon, 06 Aug 2007 00:50:27 -0000

Mike has a good point about duplicate contents :)

Re: Creating A WordPress Robots.txt To Improve SEO

Zath — Thu, 12 Apr 2007 15:20:29 -0000

Another area that I've never looked into before, I'll have to read into this and enlightenment myself, but the idea seems like a sound one to me.

I'm guessing it's best to start with the defaults shown in the linked file on Daily Blog Tips and go from there adding misc pages I have such as pages full of bookmarks etc.

Cheers!

Re: Creating A WordPress Robots.txt To Improve SEO

Daze — Thu, 12 Apr 2007 07:19:59 -0000

Good idea to disallow the bots from viewing the RSS feeds. My feeds usually always appear above the actual content in the SERPs which is good, but not so great for humans! Thanks, I'll give this a go :)

Re: Creating A WordPress Robots.txt To Improve SEO

Everton — Fri, 06 Apr 2007 10:58:43 -0000

if you want your site indexed, why should you exclude something?

Because you don't want the same posts appearing twice and Google thinking it's duplicate content, or junk appearing in results as Google will then downgrade your real pages

Re: Creating A WordPress Robots.txt To Improve SEO

hannes — Fri, 06 Apr 2007 10:20:45 -0000

if you want your site indexed, why should you exclude something?

Re: Creating A WordPress Robots.txt To Improve SEO

Mr. Apache — Mon, 26 Mar 2007 18:42:41 -0000

There is a better article about this on the askapache blog:
WordPress robots.txt optimized for SEO

Re: Creating A WordPress Robots.txt To Improve SEO

IndoDX — Mon, 26 Mar 2007 17:38:22 -0000

Why Disallow all extension? such as .PHP, .JS? can I get explain more about it?

Thanks

Re: Creating A WordPress Robots.txt To Improve SEO

Mike — Mon, 26 Mar 2007 08:02:44 -0000

One of the reasons why you'd want to use a robots.txt file is to prevent Google from indexing content twice and potentially marking it as duplicate content. If it's listed in monthly archives, category folders and on your front page, for example, you might end up with pages in the supplemental index.

Re: Creating A WordPress Robots.txt To Improve SEO

Ibrahim Faour — Mon, 26 Mar 2007 06:54:16 -0000

I don't understand Thilak

he wants them to be redirected towards essential pages

What this has to do with the robots file?!
And why should it be a waste of time to crawl the feeds?

Re: Creating A WordPress Robots.txt To Improve SEO

Thilak — Mon, 26 Mar 2007 00:56:29 -0000

I agree with Daniel, but Everton is right. He doesn't want Googlebots to waste their time spidering unwanted images or feeds. Instead, he wants them to be redirected towards essential pages

Re: Creating A WordPress Robots.txt To Improve SEO

SoKoOLz — Sun, 25 Mar 2007 23:35:59 -0000

omg, this is very useful.
thx u thx u