seo,dotnetnuke,robots.txt,googlebot , How to customize DNN robots.txt to allow a module specific sitemap to be crawled by search engines?


How to customize DNN robots.txt to allow a module specific sitemap to be crawled by search engines?

Question:

Tag: seo,dotnetnuke,robots.txt,googlebot

I am using the EasyDNN News module for the blog, news articles, etc. on our DNN website. The core DNN sitemap does not include the articles generated by this module, but the module creates its own sitemap.

For example: domain.com/blog/mid/1005/ctl/sitemap

When I try to submit this sitemap to Google, it says my Robots.txt file is blocking it.

Looking at the Robots.txt file that ships with DNN, I noticed the following lines under the Slurp and Googlebot user-agents:

Disallow: /*/ctl/       # Slurp permits *
Disallow: /*/ctl/       # Googlebot permits *

I'd like to submit the module's sitemap, but I'd like to know why the /ctl is disallowed for these user-agents, and what would the impact be if I just removed these lines from the file? Specifically, as it pertains to Google crawling the site.

As an added reference, I have read the article below about avoiding a duplicate content penalty by disallowing specific urls that contain /ctl such as login, register, terms, etc. I'm wondering if this is why DNN just disallowed any url with /ctl.

http://www.codeproject.com/Articles/18151/DotNetNuke-Search-Engine-Optimization-Part-Remov


Answer:

The proper way to do this would be to use the DNN Sitemap provider, something that is pretty darn easy to do as a module developer.

I don't have a blog post/tutorial on it, but I do have sample code which can be found in

http://dnnsimplearticle.codeplex.com/SourceControl/latest#cs/Providers/Sitemap/Sitemap.cs

This will allow custom modules to add their own information to the DNN Sitemap.

The reason /CTL is disallowed is because the normal way to load the Login/Registration/Profile controls is to do site?ctl=login and that is typically not something that people want to have indexed.

The other option is just edit the robots.txt file.


Related:


How Google “distinguishes” website articles from news? [closed]


html,seo,google-search
When I search keywords from Google, it shows all articles related to these words and It has separate tab called "News", where Google shows related news. How Goolge "Knows" that article from site is about News? I have opened source codes of multiple news websites and they has "itemprops" in...

Different addresses for different products


php,seo
I have a table shoes(id,shoename,color,brand,price,imagename,available). I am trying to sell shoes online through my website. Currently what's happening is, catalog.php(a page on my website) shows all the shoes in my table 'shoes'. Here's the code in inside a loop. echo "<div class='shoe-view'>"; echo "<img class='show-view-image' src='scripts/shoes/uploads/".$result["imagename"]."' alt='".$result["imagename"]."'/>"; echo "<form action='viewshoe.php'...

MixItUp vs PageSpeed Insigths


jquery,seo,pagespeed,mixitup
PageSpeed Insights says “Remove render-blocking scripts” and list jquery.mixitup.min.js :_( But the script is included at the bottom of the page (and minified), and the functions that use MixItUp is also on the bottom of the page! I don’t know what can I do. Any suggestions please? Thanks a lot....

Disallow specific folders in robots.txt with wildcards


seo,search-engine,robots.txt,google-crawlers
Can i hide specific folders from crawlers with wildcards like: User-agent: * Disallow: /system/ Disallow: /v* I want to hide all folders starts with "v" character. It will work this way?...

Site name in Google search results for multi-language websites


html,seo,schema.org,google-rich-snippets
Using Schema.org, I can set the name for my website so it’s visible in Google Search: https://developers.google.com/structured-data/site-name Example: <script type="application/ld+json"> { "@context" : "http://schema.org", "@type" : "WebSite", "name" : "Your WebSite Name", "alternateName" : "An alternative name for your WebSite", "url" : "http://www.your-site.com" } </script> What if I have multi-language...

DNN Can`t add reference


asp.net,dotnetnuke,dotnetnuke-7
I am using Christoc DotNetNuke Module and Theme Development Template to create new module for DNN 7 and try to add reference to my external .dll file. I am able to access my library, I have intellisense, everything seem's to be ok, but when I build my solution, it give's...

google analytics code on landing page and cookie law


jquery,google-analytics,seo
New EU cookie law do not allow page to set cookies on first load and until user make any action, scroll is consider as implicit acceptance I'm not sure if ga('set', 'anonymizeIp', true); is enough to allow google analytics to be considered as non profiling cookie how can i activate...

AngularJS form in develop module of dotnetnuke 7


javascript,angularjs,dotnetnuke
I'm developing a dotnetnuke 7 module with using AngularJS. Here is my code in View.ascx: <script src="http://ajax.googleapis.com/ajax/libs/angularjs/1.3.14/angular.min.js"></script> <h2>Validation Example</h2> <form ng-app="myApp" ng-controller="validateCtrl" name="myForm" novalidate> <p>Username:<br> <input type="text" name="user" ng-model="user" required> <span style="color:red" ng-show="myForm.user.$dirty && myForm.user.$invalid"> <span...

SEO with single page application


node.js,seo,single-page-application
I built a node.js social networking web site for noders but I get some serious problems to improve my SEO factors. How can I make it SEO friendly? And yes, given that it is a single page app, it is a little harder!

DNN module uninstall how to remove settings from web.config


web-config,settings,dotnetnuke,manifest,uninstall
I have a DNN manifest file with this: <configuration> <nodes> <node path="/configuration/appSettings" action="update" key="key" collision="overwrite"> <add key="LocalCurrencyCode" value="ARS"/> </node> and for uninstall I have this: <uninstall> <configuration> <nodes> <node path="/configuration/appsettings/add[@name='LocalCurrencyCode']" action="remove" /> However, when I uninstall the module the setting is not removed from web.config. Can anyone see what I...

My website Images not indexed by Google, Yahoo and Bing [closed]


php,codeigniter,seo
I'm using codeigniter framework. why Search Engine's not indexed my website images ? My website has been made since 2013. My website is : www.shadyab.com. It likes groupon website(Offering daily deals at restaurants, retailers and service providers.). An image url : http://www.shadyab.com/assests/images/upload/kaktoos4.jpg What should I do to tell search engines...

Canonical url for google to prevent duplicate meta?


seo,meta-tags,google-webmaster-tools
Today i went to Google Webmaster Tools to check for duplicate meta description. On almost all my news pages, this is true cause my rss feeds links to the news piece with a parameter (?rs=rss) so i can track my traffic from rss feeds. I thought the following snippet would...

What should be the name of the sitemap file for Google SEO?


seo,sitemap,google-search
I created a sitemap for my website that contains the below code: <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd"> <url> <loc>http://www.example.com/</loc> </url> <url> <loc>http://www.example.com/aboutus.html</loc> </url> <url>...

How to avoid the multiple path to same file in php using htaccess?


php,html,apache,.htaccess,seo
My url is www.abc.com/cbn/either/index.php and I want to access this by only current url. When I change the path of the index.php file, i.e. www.abc.com/cbn/index.php Still, I am accessing index.php file which is a bad approach for SEO point of view because now google will index two URLs of the...

Slidershow jquery and convert to css


jquery,css,html5,seo,slider
I downloaded script for slider show and it work without problems but after implemented this slide show i have problems with seo optimalization in HTML5. Because this code using this <div u=""> or <img u=""> and its still write me that i cant use this combination div with tag "u"....

Multiple modals with galleries vs. a single dynamic one


javascript,dom,seo,image-gallery,bootstrap-modal
Lets say we have a long list of posts on a single page. Each of those posts has a hidden div with multiple img tags inside it. When a user clicks on the post, the images inside the hidden div should be showcased in a modal gallery. Which approach is...

Dotnetnuke migration from SQL 2005 to SQL 2012


sql-server-2005,sql-server-2012,dotnetnuke,database-migration
I'm trying to upgrade my DNN v6 from a SQL Server 2005 to a SQL Server 2012. My problem is, after modifying the web.config to match the new appSettings, my website automaticaly runs the install wizard. My IIS was running on a 2003 server and is now on a 2012...

disqus SEO google crawler doesn't load comments


seo,disqus
I see in google webmaster We were unable to load Disqus. If you are a moderator please see our troubleshooting guide. instead of comments. But i read in the Internet, disqus comments are readable by google Crawler. As i understand to show "We were unable to load..." google had to...

Does DNN Have anything similar to Views in Drupal or ACF in Wordpress?


dotnetnuke,dotnetnuke-7
The company I work for is switching over completely to DNN. All of our websites were previously built in Drupal and Wordpress. I'm looking for DNN equivalents to Views in Drupal and/or Advanced Custom Fields in Wordpress. I did find this DNN extension, which seems like it will get part...

Change the unique generated title names of friendly-id using attribute of another table


ruby-on-rails,ruby,seo,friendly-url,friendly-id
I have a Company Model, and i am using friendly_id like this friendly_id :name, use: :slugged But since there can be many Company with the same name (different branches). I am trying to handle that case by using city attribute from the address of the Company. But the Company address...

si​tem​ap-​tax​-po​st_​tag​.xm​l not found - webmaster tools


wordpress,seo
I'm a newbie in webmaster tools. I get 3 errors in webmaster tools: 1.2: We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit. *General HTTP error: 404 not found Sitemap:...

SEO and tags with JavaScript functionality


javascript,html,twitter-bootstrap,seo
Since we are diving into SEO guidelines the past weeks we came across a question for which we didn't find a satisfying answer. (We simply didn't agree on this topic). We would like more opinions on this. Since many projects use jQuery and Bootstrap lately, anchor tags often get used...

DNN - How to Transfer Production Version Code in Local with enabling debug


dotnetnuke
I have downloaded DNN from production and copied in local server. and followed below steps to make it running in local. 1. Copy DNN source to local 2. Created Virtual Directory in IIS 7 and created application as localhost/sitename 2. Changed Portal Alias in database to localhost/sitename 3. Changed Web.Config...

Grails produce seo friendly URLS


grails,seo
I'm very new to grails and I have some questions about creating views with SEO friendly URLs. Lets say I have a page I'd like to call used-products or https://www.sampledomain.com/used-products, how would I go about creating a view and have it resolve for used-products? Another example would be something like...

Server side vs client side website


javascript,html,ajax,html5,seo
We have to build a website and we have to chose where to manage the content, in the server (PHP or JSP) or on the client (JavaScript). This article: http://searchenginewatch.com/sew/how-to/2358775/seo-strategies-for-javascript-heavy-single-page-applications-or-ajax-sites enlighted me a bit but Im still doubting. Good SEO is the most important thing to achieve. Can anyone relate...

Dotnetnuke URL tracking


sql,dotnetnuke
I have a DNN (5) internal website that I need track URL's. I am unable to use any external analytics (Google or otherwise), so I have to rely on the internal installation as-is. I have also tried a tracking tool, but it does not give me what I need, which...

Login failed for user DefaultApp pool


c#,asp.net,sql-server,dotnetnuke
I am working on a local DotNetNuke site, and all goes well. But after installing a custom module I created, in which I use Linq 2 Sql. Suddenly I get a module load exception "The login failed. Login failed for user 'IIS APPPOOL\DefaultAppPool'". The site itself is working fine, the...

How can I get better google indexing results?


seo,google-search,pagerank
I have just launched a new domain, www.nextlevelsmf.com and it's not showing for some keywords I would like it to. Can anyone give me some advice to help it rank better please? I'd like it to show on the first 2 pages for: Managed SMF hosting SMF host/hosting Managed SMF...

Removing the number of first page in Yii2 Pagination from the URL


.htaccess,pagination,seo,yii2
For SEO purposes I need to remove the first page number from the URL. i.e I have the following: example.com/pages/view/1 and example.com/pages/view the two URLs points to the same contents of the view action. I want to make the pagination free from 1 in the URL. i.e first Page link...

Schema.org mandatory fields and the time needed until Google shows changes


seo,schema.org,google-rich-snippets
I have implemented Schema.org (using Microdata) inside my product pages and when I check Google Webmaster Tools it is crawled by Googlebot and interpreted successfully. The point is I have not implemented some properties inside Product type like brand. I need to know whether there is some subset of all...

How do I strip out ?_escaped_fragment_= using .htaccess


ajax,.htaccess,mod-rewrite,seo
Google discovered that I'm allowing end users to navigate my content using ajax loading, and is loading my pages as a user client rather than requesting them as new page loads. So instead of trying to index www.mysite.com/page, it's requesting www.mysite.com/?_escaped_fragment_=/page Which is not at all what I want it...

What is more important for images - alt tag or name


seo
I am making a blog with huge ammount of images, and one way to do it, is by using Flickr Gallery plugin, which provides a functional gallery or your albumbs, but the links aren't looking good (www.......5129512891.jpg), but they do have the proper alt tags (Red Carpet From Turkey). So...

Auto logout in DNN, once browser is closed


dotnetnuke
I am developing an application using DNN. One of the requirements are to automatically logout a User if he closes his browser clicking on the Close button. I have tried many ways but its not working? Any suggestions, how this can be implemented.

WooRank Touchscreen Readiness Failing


responsive-design,seo,responsiveness
I'm currently using the WooRank website to improve my sites SEO> However, it keeps failing on "Touchscreen Readiness". It states my buttons should be 40px in height/width which they appear to be.. but it's failing. Any advice? URL: http://www.answer-hut.com...

Convert MVC Website to be Responsive


mobile,model-view-controller,seo
While investigating the Google SEO issue with respect to not having a mobile web site. I've learned recently that lack of proper mobile site would reduce the search ratings, and a different content for a mobile and a desktop sites might hurt your SEO as well. Currently we have a...

fullPage.js: Make all slides and sections visible in search engine results


jquery,seo,web-crawler,single-page-application,fullpage.js
I'm using fullpage.js jQuery plugin for a Single page application. I'm using mostly default settings and the plugin works like a charm. When I got to the SEO though I couldn't properly make Google crawl my website on a "per slide" basis. All my slides are loaded at the page...

SPDY on shared host & SEO Sematics


seo,semantics,shared-hosting,spdy,http2
NodeSPDY on shared host I got a webspace hosted by uberspace and want to use NodeSPDY, but there is a loadbalancer in between which cuts off the tls connection. On uberspace one can request a Port to be opened. With this port it is possible to request resources directly by...

Wordpress - Robotx.txt allows admin login?


wordpress,seo,robots.txt
First, i've searched by robots.txt for Wordpress, but, no one told me where is this file. So, I read that the robots.txt in Wordpress is virtual. Ok, no problem. But, where i find this to edit? My Wordpress is allowing the /author/admin and i don't want this. In dashboard, the...

Best JSON-LD practices: using multiple