Why does blog engine's robot.txt restrict...

Feb 6, 2011 at 5:19 PM
Edited Feb 6, 2011 at 5:19 PM

... archive.aspx ? I dont understand.

Feb 10, 2011 at 4:24 PM

Bump

Feb 10, 2011 at 5:38 PM

Somebody probably didn't want Robots crawling their whole site.  If you don't mind... remove it.

Feb 11, 2011 at 7:35 PM

But is there anything related to security about this?

Feb 15, 2011 at 7:34 PM

Bump

Feb 16, 2011 at 3:18 PM

 

I think it's for not having duplicate content.

Feb 17, 2011 at 5:54 PM

I see, thanks