Blocking using robots.txt?

Nov 29, 2011 at 2:12 PM

Hi guys, just a question for you all.

 

Do you block search engines from indexing "tag" and "category" pages?

 

As in those pages that show all posts that have that "tag"?

 

Thanks

Nov 29, 2011 at 4:33 PM
Edited Nov 29, 2011 at 4:37 PM

Tags of late have lost their credibility in the eyes of search engines due to the misuse by the bloggers . But yes , they are important in terms of post navigational structure and grouping that makes visitors to locate an information easily.

Recently Google has waged a crack down (Panda Update Algorithm) on sites using thin and shallow content , excessive tag uses , empty category pages. So... Yes blocking (or using meta tags noindex,follow) tags , category , archives like pages seems to be the need of the hour.

Nov 29, 2011 at 4:41 PM

Does anyone know where i need to put the robots meta tag if i want google to use "noindex" on my archive, tag, and category pages?

 

Or should i use robots.txt?

 

Thanks

Nov 29, 2011 at 5:06 PM

Go to the

themes/<your selected themes folder>

Open site.master.cs

Add the following codes to Page_Load event.

 

//Adds noindex,follow to all tagged pages

if (rawUrl.Contains("?tag="))

        {        
                HtmlGenericControl meta = new HtmlGenericControl("meta");
                meta.Attributes["name"] = "robots";
                meta.Attributes["content"] = "noindex,follow";
                Page.Header.Controls.Add(meta);
        }
//To Add noindex,follow to archive.aspx page

if (rawUrl.Contains("archive.aspx"))

        {        
                HtmlGenericControl meta = new HtmlGenericControl("meta");
                meta.Attributes["name"] = "robots";
                meta.Attributes["content"] = "noindex,follow";
                Page.Header.Controls.Add(meta);
        }
//Add noindex,follow to category pages

if (rawUrl.Contains("/category/"))

        {        
                HtmlGenericControl meta = new HtmlGenericControl("meta");
                meta.Attributes["name"] = "robots";
                meta.Attributes["content"] = "noindex,follow";
                Page.Header.Controls.Add(meta);
        }
Nov 30, 2011 at 5:11 PM
Edited Nov 30, 2011 at 5:59 PM

Thank you so much!

 

But why do you need to add the "follow" tag as well? I thought if you just add "noindex", google will still follow the links but wont put the page in it's search results.

 

Edit: Also, its request.rawUrl , not just rawUrl

Nov 30, 2011 at 6:26 PM

noindex,follow means

do not index my this page(the page which has noindex,follow meta tag ) but if someone links to my this page pass on (ie follow)  the link juice (page rank/weight) to the pages linked from this page

using noindex,follow is a convention for better clarity and keeping in view about the behavior of other search engines (Bing,Yahoo,etc) about non use of follow.

May 14, 2013 at 3:23 PM
jha_jee wrote:
Go to the themes/<your selected themes folder> Open site.master.cs Add the following codes to Page_Load event.   //Adds noindex,follow to all tagged pages if (rawUrl.Contains("?tag="))         {                         HtmlGenericControl meta = new HtmlGenericControl("meta");                 meta.Attributes["name"] = "robots";                 meta.Attributes["content"] = "noindex,follow";                 Page.Header.Controls.Add(meta);         } //To Add noindex,follow to archive.aspx page if (rawUrl.Contains("archive.aspx"))         {                         HtmlGenericControl meta = new HtmlGenericControl("meta");                 meta.Attributes["name"] = "robots";                 meta.Attributes["content"] = "noindex,follow";                 Page.Header.Controls.Add(meta);         } //Add noindex,follow to category pages if (rawUrl.Contains("/category/"))         {                         HtmlGenericControl meta = new HtmlGenericControl("meta");                 meta.Attributes["name"] = "robots";                 meta.Attributes["content"] = "noindex,follow";                 Page.Header.Controls.Add(meta);         }
I just tried to implement your above code on a blog running 2.0 and when I add the code to site.master.cs the site breaks and loads with some unrecognizable encoding as seen below. I tried also adding the request.rawURL as mentioned by thegatekeeper, but I think it just may be a version compatibility issue. If you could let me know what version of BlogEngine you are running this code successfully on, as well as offer any insight into the encoding issue I am having when trying to implement, I would be forever grateful. (Blog URL: http://www.bestfriendservices.com/blog/)

�I�%&/m�{J�J��t��$ؐ@������iG#)�*��eVe]f@�흼��{���{���;�N'���?\fdl��J�ɞ!���?~|?"��Ey�')=��y6����h����z��<=��~|W>�Z5�u�A:�f��/>����y�(��G�d^ϲe��!z���Ѳ�Y)5��G��A�8�VeU?������/I=����Yd�E��n�գt����/�q� �IU�n �ۻ�/No�������N�|� ��&�} �0��euU��?_O�Y��T˦s�3�0_�/� �ۼV�@ʄJ��TH�����Y>��-��tY-���e^7�U���ŋ:�[�`��+�'U�V�G����]�ݻU��e��F�v{^/gy]�{�#���.���u��su�S��]���Uq�?��' 賏��E�d�� B���w�^�5�[�-���w'eu�qz�Z�Ŕ1?���U1k������)f�]A���(����w V����Q���Jnq���Ƣ����g������"+G�s�%tF���2�������ٲ�~�_�ˬ� �j�8O#��O�fZ+���w'G��4s�Ler�)��O����� ���n�uC�/4y�ˋ�&��7�&J�}�Wj��YQ6iu���>� �$'h�e�_u��ڼ�N�ι+ h���!�3N��j]�F鼺����ܼ=!F��+B�����uXe5�J\�Ȧs��q�f4���-��H���͗��A1�U>-΋�q�7MvAM+�'�IX�P .�(]�4J": �>��_X�a�35��=L��X�zZ��/ZW��U>O��yq!���Z�/=/J~K����tV�����k3Af� �?Y�� :��t�]�i�6�ڢ��Tֶu1Y���Ͽ
Coordinator
May 14, 2013 at 5:38 PM
In the admin -> settings -> advanced uncheck any compressions for encoding error go away (your admin should work fine even with error on the front page).
May 15, 2013 at 5:44 PM

I just got the compression issue resolved, but now it loads with the following error.

Server Error in '/blog' Application.


Compilation Error

Description: An error occurred during the compilation of a resource required to service this request. Please review the following specific error details and modify your source code appropriately.

Compiler Error Message: CS0103: The name 'rawUrl' does not exist in the current context

Source Error:

Line 8: protected void Page_Load(object sender, EventArgs e)

Line 9: {

Line 10: if (rawUrl.Contains("?tag="))

Line 11: {

Line 12: HtmlGenericControl meta = new HtmlGenericControl("meta");


Source File: c:\hshome\bestfriendservic\bestfriendservices.com\blog\themes\bfs\site.master.cs Line: 10

Show Detailed Compiler Output:


Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.272

Oct 2, 2013 at 6:32 PM
arowls,

I just ran into this issue myself. Try this instead:
        if (HttpContext.Current.Request.RawUrl.ToLower().Contains("?tag="))
        {        
            HtmlGenericControl meta = new HtmlGenericControl("meta");
            meta.Attributes["name"] = "robots";
            meta.Attributes["content"] = "noindex,follow";
            Page.Header.Controls.Add(meta);
        }