seo,dotnetnuke,robots.txt,googlebot , How to customize DNN robots.txt to allow a module specific sitemap to be crawled by search engines?


How to customize DNN robots.txt to allow a module specific sitemap to be crawled by search engines?

Question:

Tag: seo,dotnetnuke,robots.txt,googlebot

I am using the EasyDNN News module for the blog, news articles, etc. on our DNN website. The core DNN sitemap does not include the articles generated by this module, but the module creates its own sitemap.

For example: domain.com/blog/mid/1005/ctl/sitemap

When I try to submit this sitemap to Google, it says my Robots.txt file is blocking it.

Looking at the Robots.txt file that ships with DNN, I noticed the following lines under the Slurp and Googlebot user-agents:

Disallow: /*/ctl/       # Slurp permits *
Disallow: /*/ctl/       # Googlebot permits *

I'd like to submit the module's sitemap, but I'd like to know why the /ctl is disallowed for these user-agents, and what would the impact be if I just removed these lines from the file? Specifically, as it pertains to Google crawling the site.

As an added reference, I have read the article below about avoiding a duplicate content penalty by disallowing specific urls that contain /ctl such as login, register, terms, etc. I'm wondering if this is why DNN just disallowed any url with /ctl.

http://www.codeproject.com/Articles/18151/DotNetNuke-Search-Engine-Optimization-Part-Remov


Answer:

The proper way to do this would be to use the DNN Sitemap provider, something that is pretty darn easy to do as a module developer.

I don't have a blog post/tutorial on it, but I do have sample code which can be found in

http://dnnsimplearticle.codeplex.com/SourceControl/latest#cs/Providers/Sitemap/Sitemap.cs

This will allow custom modules to add their own information to the DNN Sitemap.

The reason /CTL is disallowed is because the normal way to load the Login/Registration/Profile controls is to do site?ctl=login and that is typically not something that people want to have indexed.

The other option is just edit the robots.txt file.


Related:


Server side vs client side website


javascript,html,ajax,html5,seo
We have to build a website and we have to chose where to manage the content, in the server (PHP or JSP) or on the client (JavaScript). This article: http://searchenginewatch.com/sew/how-to/2358775/seo-strategies-for-javascript-heavy-single-page-applications-or-ajax-sites enlighted me a bit but Im still doubting. Good SEO is the most important thing to achieve. Can anyone relate...

Dotnetnuke migration from SQL 2005 to SQL 2012


sql-server-2005,sql-server-2012,dotnetnuke,database-migration
I'm trying to upgrade my DNN v6 from a SQL Server 2005 to a SQL Server 2012. My problem is, after modifying the web.config to match the new appSettings, my website automaticaly runs the install wizard. My IIS was running on a 2003 server and is now on a 2012...

SEO with single page application


node.js,seo,single-page-application
I built a node.js social networking web site for noders but I get some serious problems to improve my SEO factors. How can I make it SEO friendly? And yes, given that it is a single page app, it is a little harder!

Canonical url for google to prevent duplicate meta?


seo,meta-tags,google-webmaster-tools
Today i went to Google Webmaster Tools to check for duplicate meta description. On almost all my news pages, this is true cause my rss feeds links to the news piece with a parameter (?rs=rss) so i can track my traffic from rss feeds. I thought the following snippet would...

MixItUp vs PageSpeed Insigths


jquery,seo,pagespeed,mixitup
PageSpeed Insights says “Remove render-blocking scripts” and list jquery.mixitup.min.js :_( But the script is included at the bottom of the page (and minified), and the functions that use MixItUp is also on the bottom of the page! I don’t know what can I do. Any suggestions please? Thanks a lot....

WooRank Touchscreen Readiness Failing


responsive-design,seo,responsiveness
I'm currently using the WooRank website to improve my sites SEO> However, it keeps failing on "Touchscreen Readiness". It states my buttons should be 40px in height/width which they appear to be.. but it's failing. Any advice? URL: http://www.answer-hut.com...

Does DNN Have anything similar to Views in Drupal or ACF in Wordpress?


dotnetnuke,dotnetnuke-7
The company I work for is switching over completely to DNN. All of our websites were previously built in Drupal and Wordpress. I'm looking for DNN equivalents to Views in Drupal and/or Advanced Custom Fields in Wordpress. I did find this DNN extension, which seems like it will get part...

google analytics code on landing page and cookie law


jquery,google-analytics,seo
New EU cookie law do not allow page to set cookies on first load and until user make any action, scroll is consider as implicit acceptance I'm not sure if ga('set', 'anonymizeIp', true); is enough to allow google analytics to be considered as non profiling cookie how can i activate...

Site name in Google search results for multi-language websites


html,seo,schema.org,google-rich-snippets
Using Schema.org, I can set the name for my website so it’s visible in Google Search: https://developers.google.com/structured-data/site-name Example: <script type="application/ld+json"> { "@context" : "http://schema.org", "@type" : "WebSite", "name" : "Your WebSite Name", "alternateName" : "An alternative name for your WebSite", "url" : "http://www.your-site.com" } </script> What if I have multi-language...

How seo implemented for distancebetween.com website?


seo,google-search
When I Search for distance between bangalore to mumbai in Google, distancebetween.com comes up in the search results. I mean if I search for distance between any source to destination they have results for that. They have one dynamic page where user can enter source and destination and those inputs...

Slidershow jquery and convert to css


jquery,css,html5,seo,slider
I downloaded script for slider show and it work without problems but after implemented this slide show i have problems with seo optimalization in HTML5. Because this code using this <div u=""> or <img u=""> and its still write me that i cant use this combination div with tag "u"....

Heading order in HTML5


html5,seo,semantic-markup
This is a webpage example of my site: <html> <title> artilce header </title> <body> <header> <h1> nme of website</h1></header> <section> <h2> name of section</h2> <article> <h3>article header</h3> </article> </section> </body> </html> I want to know if this order is correct? Or does it maybe have a bad effect on SEO?...

how to create direct link to cms articles


php,mysql,seo,content-management-system
I'm creating a simple CMS with a table called 'articles' that all the articles are stored in it. In my AddArticle.php page on the website I have a form that creates the articles and feeds the table. Now I'm confused with article linking. As far as I know a CMS...

Robots.txt file in MVC.NET 4


asp.net,asp.net-mvc-4,seo,robots.txt
I have read an article about ignoring the robots from some url in my ASP MVC.NET project. In his article author said that we should add some action in some off controllers like this. In this example he adds the action to the Home Controller: #region -- Robots() Method --...

Different addresses for different products


php,seo
I have a table shoes(id,shoename,color,brand,price,imagename,available). I am trying to sell shoes online through my website. Currently what's happening is, catalog.php(a page on my website) shows all the shoes in my table 'shoes'. Here's the code in inside a loop. echo "<div class='shoe-view'>"; echo "<img class='show-view-image' src='scripts/shoes/uploads/".$result["imagename"]."' alt='".$result["imagename"]."'/>"; echo "<form action='viewshoe.php'...

Removing the number of first page in Yii2 Pagination from the URL


.htaccess,pagination,seo,yii2
For SEO purposes I need to remove the first page number from the URL. i.e I have the following: example.com/pages/view/1 and example.com/pages/view the two URLs points to the same contents of the view action. I want to make the pagination free from 1 in the URL. i.e first Page link...

What should be the name of the sitemap file for Google SEO?


seo,sitemap,google-search
I created a sitemap for my website that contains the below code: <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd"> <url> <loc>http://www.example.com/</loc> </url> <url> <loc>http://www.example.com/aboutus.html</loc> </url> <url>...

How to “remove” indexed pages in Google and AngularJS


angularjs,nginx,seo,google-index
I have an AngularJS 1.0.7 web application. Some months ago I sent a sitemap.xml file to Google, so the urls are indexed. Now, I would like to upload a new sitemap file with some updates on those URLs. I mean, the pages are the same but the urls have changed...

Best JSON-LD practices: using multiple