They use what you call "Google Hacks" or "Google Dorks"
The issues with BlogEngine.net and other Platforms can be fixed :)
For BlogEngine.net I am working on some patches that once done and tested going to submit a pull request that should fix these issues when I get the time.
Hopefully in about 3-5 days hopefully.
I am thinking and currently experimenting with is providing Search Engines with a different "View" than a require user.
Similar what you do for "mobile" users where you have a different theme for "mobile" users so why not the same for Search Engines and Crawlers?
I am testing this new concept out at:
In that case through it is using MVC.
Search Engines get one view
Users get another.
In the case for BlogEngine.Net the Search Engine view can also take out all of the "footprints" or tail signs that the website is using BlogEngine.net
This way none of that information is indexed by the search engines.
Only thing that is indexed is only the actual content of each allowable page and that is it.
SEO wise and saving on bandwidth also have the User view all Meta tags are taken out.
Meta tags are only on the Search Engine View.
Currently Just experimenting to see if this have any negative effects by Google indexing and Page ranking.
BE 3.0 has some of these features I included before and so far seems to be working ok :)
So Next round of fixing should solve hopefully all of the following:
- Search Engine Indexing Security Issues
- Force Search Engines to obey Robot.txt rules
- Have good SEO, and Page Ranking
If you or anyone else see something I am missing please let me know.
If you like to experiment with the test version of this please let me know, will give you link to download once it is available.
Have a great day!