seo,dotnetnuke,robots.txt,googlebot , How to customize DNN robots.txt to allow a module specific sitemap to be crawled by search engines?


How to customize DNN robots.txt to allow a module specific sitemap to be crawled by search engines?

Question:

Tag: seo,dotnetnuke,robots.txt,googlebot

I am using the EasyDNN News module for the blog, news articles, etc. on our DNN website. The core DNN sitemap does not include the articles generated by this module, but the module creates its own sitemap.

For example: domain.com/blog/mid/1005/ctl/sitemap

When I try to submit this sitemap to Google, it says my Robots.txt file is blocking it.

Looking at the Robots.txt file that ships with DNN, I noticed the following lines under the Slurp and Googlebot user-agents:

Disallow: /*/ctl/       # Slurp permits *
Disallow: /*/ctl/       # Googlebot permits *

I'd like to submit the module's sitemap, but I'd like to know why the /ctl is disallowed for these user-agents, and what would the impact be if I just removed these lines from the file? Specifically, as it pertains to Google crawling the site.

As an added reference, I have read the article below about avoiding a duplicate content penalty by disallowing specific urls that contain /ctl such as login, register, terms, etc. I'm wondering if this is why DNN just disallowed any url with /ctl.

http://www.codeproject.com/Articles/18151/DotNetNuke-Search-Engine-Optimization-Part-Remov


Answer:

The proper way to do this would be to use the DNN Sitemap provider, something that is pretty darn easy to do as a module developer.

I don't have a blog post/tutorial on it, but I do have sample code which can be found in

http://dnnsimplearticle.codeplex.com/SourceControl/latest#cs/Providers/Sitemap/Sitemap.cs

This will allow custom modules to add their own information to the DNN Sitemap.

The reason /CTL is disallowed is because the normal way to load the Login/Registration/Profile controls is to do site?ctl=login and that is typically not something that people want to have indexed.

The other option is just edit the robots.txt file.


Related:


SPDY on shared host & SEO Sematics


seo,semantics,shared-hosting,spdy,http2
NodeSPDY on shared host I got a webspace hosted by uberspace and want to use NodeSPDY, but there is a loadbalancer in between which cuts off the tls connection. On uberspace one can request a Port to be opened. With this port it is possible to request resources directly by...

how to create direct link to cms articles


php,mysql,seo,content-management-system
I'm creating a simple CMS with a table called 'articles' that all the articles are stored in it. In my AddArticle.php page on the website I have a form that creates the articles and feeds the table. Now I'm confused with article linking. As far as I know a CMS...

How do I retain search engine accessibility when loading external file into html


javascript,jquery,html,seo,cross-platform
I plan to: $('#main.sidecontent').load('/sidecontent.html'); and insert: <div class="sidecontent" id="main"></div> a div on several html pages since the content of this div is the same and I try to straighten my workflow, basically avoiding editing every single html page when this content div changes. My questions: Is this method recommended for...

Slidershow jquery and convert to css


jquery,css,html5,seo,slider
I downloaded script for slider show and it work without problems but after implemented this slide show i have problems with seo optimalization in HTML5. Because this code using this <div u=""> or <img u=""> and its still write me that i cant use this combination div with tag "u"....

Wordpress - Robotx.txt allows admin login?


wordpress,seo,robots.txt
First, i've searched by robots.txt for Wordpress, but, no one told me where is this file. So, I read that the robots.txt in Wordpress is virtual. Ok, no problem. But, where i find this to edit? My Wordpress is allowing the /author/admin and i don't want this. In dashboard, the...

How seo implemented for distancebetween.com website?


seo,google-search
When I Search for distance between bangalore to mumbai in Google, distancebetween.com comes up in the search results. I mean if I search for distance between any source to destination they have results for that. They have one dynamic page where user can enter source and destination and those inputs...

Convert MVC Website to be Responsive


mobile,model-view-controller,seo
While investigating the Google SEO issue with respect to not having a mobile web site. I've learned recently that lack of proper mobile site would reduce the search ratings, and a different content for a mobile and a desktop sites might hurt your SEO as well. Currently we have a...

Login failed for user DefaultApp pool


c#,asp.net,sql-server,dotnetnuke
I am working on a local DotNetNuke site, and all goes well. But after installing a custom module I created, in which I use Linq 2 Sql. Suddenly I get a module load exception "The login failed. Login failed for user 'IIS APPPOOL\DefaultAppPool'". The site itself is working fine, the...

Schema.org mandatory fields and the time needed until Google shows changes


seo,schema.org,google-rich-snippets
I have implemented Schema.org (using Microdata) inside my product pages and when I check Google Webmaster Tools it is crawled by Googlebot and interpreted successfully. The point is I have not implemented some properties inside Product type like brand. I need to know whether there is some subset of all...

Dotnetnuke URL tracking


sql,dotnetnuke
I have a DNN (5) internal website that I need track URL's. I am unable to use any external analytics (Google or otherwise), so I have to rely on the internal installation as-is. I have also tried a tracking tool, but it does not give me what I need, which...

SEO with single page application


node.js,seo,single-page-application
I built a node.js social networking web site for noders but I get some serious problems to improve my SEO factors. How can I make it SEO friendly? And yes, given that it is a single page app, it is a little harder!

DNN module uninstall how to remove settings from web.config


web-config,settings,dotnetnuke,manifest,uninstall
I have a DNN manifest file with this: <configuration> <nodes> <node path="/configuration/appSettings" action="update" key="key" collision="overwrite"> <add key="LocalCurrencyCode" value="ARS"/> </node> and for uninstall I have this: <uninstall> <configuration> <nodes> <node path="/configuration/appsettings/add[@name='LocalCurrencyCode']" action="remove" /> However, when I uninstall the module the setting is not removed from web.config. Can anyone see what I...

Wordpress - customized pages with blocks - prohibit google seo index of blocks


wordpress,seo,woocommerce,robots.txt,google-sitemap
I'm using Wordpress and WooCommerce for my online shop. With the theme I'm using you can customize the product-category pages by adding "blocks". So if I want to have a text on the top of a product category page I simply create a block page, lets say its called "category-info"....

Robots.txt file in MVC.NET 4


asp.net,asp.net-mvc-4,seo,robots.txt
I have read an article about ignoring the robots from some url in my ASP MVC.NET project. In his article author said that we should add some action in some off controllers like this. In this example he adds the action to the Home Controller: #region -- Robots() Method --...

fullPage.js: Make all slides and sections visible in search engine results


jquery,seo,web-crawler,single-page-application,fullpage.js
I'm using fullpage.js jQuery plugin for a Single page application. I'm using mostly default settings and the plugin works like a charm. When I got to the SEO though I couldn't properly make Google crawl my website on a "per slide" basis. All my slides are loaded at the page...

MixItUp vs PageSpeed Insigths


jquery,seo,pagespeed,mixitup
PageSpeed Insights says “Remove render-blocking scripts” and list jquery.mixitup.min.js :_( But the script is included at the bottom of the page (and minified), and the functions that use MixItUp is also on the bottom of the page! I don’t know what can I do. Any suggestions please? Thanks a lot....

How to avoid the multiple path to same file in php using htaccess?


php,html,apache,.htaccess,seo
My url is www.abc.com/cbn/either/index.php and I want to access this by only current url. When I change the path of the index.php file, i.e. www.abc.com/cbn/index.php Still, I am accessing index.php file which is a bad approach for SEO point of view because now google will index two URLs of the...

My website Images not indexed by Google, Yahoo and Bing [closed]


php,codeigniter,seo
I'm using codeigniter framework. why Search Engine's not indexed my website images ? My website has been made since 2013. My website is : www.shadyab.com. It likes groupon website(Offering daily deals at restaurants, retailers and service providers.). An image url : http://www.shadyab.com/assests/images/upload/kaktoos4.jpg What should I do to tell search engines...

Change the unique generated title names of friendly-id using attribute of another table


ruby-on-rails,ruby,seo,friendly-url,friendly-id
I have a Company Model, and i am using friendly_id like this friendly_id :name, use: :slugged But since there can be many Company with the same name (different branches). I am trying to handle that case by using city attribute from the address of the Company. But the Company address...

SEO and user-friendly URLs for multi-language website


url,seo,multilingual,usability
Let's say I have a website that has 2 languages, one uses Latin and the second one Cyrillic transcriptions in URLs. For example: example.com/link example.com/ссылка My question is which is more user and SEO friendly, if I leave them as is or if I add the language prefix, so they'd...

Best JSON-LD practices: using multiple