Robots.txt for Nested Blog

Apr 14, 2011 at 2:45 AM

I have BlogEngine.net installed as a sub-application within my parent application.  I auto-generated a sitemap for my entire site using this online tool and am pointing to this sitemap.xml in my robots.txt.  I'm wondering what I should do about the robots.txt within the /blog directory.  Can I point to multiple sitemaps from my root directory?  If so my robots.txt would look like this:

 

#existing robots.txt

Sitemap: http://www.cheatsheetwarroom.com/sitemap.xml

User-agent: *
Allow: /
User-agent: *
Disallow: /forum/App_Templates

 

#add this?

sitemap: http://www.cheatsheetwarroom.com/blog/sitemap.axd

Disallow: blog/Account/*.*
Disallow: blog/search.aspx
Disallow: blog/error404.aspx
Disallow: blog/archive.aspx

 

Any ideas?

Apr 14, 2011 at 2:05 PM
Edited Apr 14, 2011 at 2:06 PM

Can I point to multiple sitemaps from my root directory?

Yes.

Here is my robot.txt

For BlogEngine.Net in sub-app :

Disallow: /*/Account/*.*

Disallow: /*/search.aspx
Disallow: /*/error404.aspx
Disallow: /*/archive.aspx
Disallow: /*/?tag
Disallow: /*/file.axd
Disallow: /*/?theme

And for my SMF forum :

Disallow: /forum/index.php?action=search
Disallow: /forum/index.php?action=calendar
Disallow: /forum/index.php?action=login
Disallow: /forum/index.php?action=register
Disallow: /forum/index.php?action=profile
Disallow: /forum/index.php?action=stats
Disallow: /forum/index.php?action=activate
Disallow: /forum/index.php?action=help
Disallow: /forum/index.php?action=admin
Disallow: /forum/index.php?action=pm
Disallow: /forum/index.php?action=mlist
Disallow: /forum/index.php?action=notify
Disallow: /forum/index.php?action=post
Disallow: /forum/index.php?action=markasread
Disallow: /forum/index.php?action=sendtopic
Disallow: /forum/index.php?action=printpage
Disallow: /forum/index.php?action=dlattach
Disallow: /forum/index.php?action=reminder
Disallow: /forum/attachments/
Disallow: /forum/avatars/
Disallow: /forum/Packages/
Disallow: /forum/sitemaps/
Disallow: /forum/Smileys/
Disallow: /forum/Sources/
Disallow: /forum/Themes/

Disallow: /*sort=
Disallow: /*.msg
Disallow: *wap2*