seo,sitemap,robots.txt , Use `robots.txt` in a multilingual site


Use `robots.txt` in a multilingual site

Question:

Tag: seo,sitemap,robots.txt

Having to manage a multilingual site, where users are redirected to a local version of site, like

myBurger.com/en // for users from US, UK, etc..
myBurger.com/fr // for users from France, Swiss, etc...

How should be organized the robots.txt file in pair with the sitemap?

myBurger.com/robots.txt // with - Sitemap: http://myBurger.com/??/sitemap
OR
myBurger.com/en/robots.txt  // with - Sitemap: http://myBurger.com/en/sitemap
myBurger.com/fr/robots.txt  // with - Sitemap: http://myBurger.com/fr/sitemap

kwnowing that en and fr sites are in fact independent entities not sharing common content, even if similar appearance.


Answer:

You need to put one robots.txt at the top level.

The robots.txt file must be in the top-level directory of the host, accessible though the appropriate protocol and port number.

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


Related:


Schema.org mandatory fields and the time needed until Google shows changes


seo,schema.org,google-rich-snippets
I have implemented Schema.org (using Microdata) inside my product pages and when I check Google Webmaster Tools it is crawled by Googlebot and interpreted successfully. The point is I have not implemented some properties inside Product type like brand. I need to know whether there is some subset of all...

How do I retain search engine accessibility when loading external file into html


javascript,jquery,html,seo,cross-platform
I plan to: $('#main.sidecontent').load('/sidecontent.html'); and insert: <div class="sidecontent" id="main"></div> a div on several html pages since the content of this div is the same and I try to straighten my workflow, basically avoiding editing every single html page when this content div changes. My questions: Is this method recommended for...

How to avoid the multiple path to same file in php using htaccess?


php,html,apache,.htaccess,seo
My url is www.abc.com/cbn/either/index.php and I want to access this by only current url. When I change the path of the index.php file, i.e. www.abc.com/cbn/index.php Still, I am accessing index.php file which is a bad approach for SEO point of view because now google will index two URLs of the...

SEO and user-friendly URLs for multi-language website


url,seo,multilingual,usability
Let's say I have a website that has 2 languages, one uses Latin and the second one Cyrillic transcriptions in URLs. For example: example.com/link example.com/ссылка My question is which is more user and SEO friendly, if I leave them as is or if I add the language prefix, so they'd...

SEO and tags with JavaScript functionality


javascript,html,twitter-bootstrap,seo
Since we are diving into SEO guidelines the past weeks we came across a question for which we didn't find a satisfying answer. (We simply didn't agree on this topic). We would like more opinions on this. Since many projects use jQuery and Bootstrap lately, anchor tags often get used...

My website Images not indexed by Google, Yahoo and Bing [closed]


php,codeigniter,seo
I'm using codeigniter framework. why Search Engine's not indexed my website images ? My website has been made since 2013. My website is : www.shadyab.com. It likes groupon website(Offering daily deals at restaurants, retailers and service providers.). An image url : http://www.shadyab.com/assests/images/upload/kaktoos4.jpg What should I do to tell search engines...

how to create direct link to cms articles


php,mysql,seo,content-management-system
I'm creating a simple CMS with a table called 'articles' that all the articles are stored in it. In my AddArticle.php page on the website I have a form that creates the articles and feeds the table. Now I'm confused with article linking. As far as I know a CMS...

Slidershow jquery and convert to css


jquery,css,html5,seo,slider
I downloaded script for slider show and it work without problems but after implemented this slide show i have problems with seo optimalization in HTML5. Because this code using this <div u=""> or <img u=""> and its still write me that i cant use this combination div with tag "u"....

google analytics code on landing page and cookie law


jquery,google-analytics,seo
New EU cookie law do not allow page to set cookies on first load and until user make any action, scroll is consider as implicit acceptance I'm not sure if ga('set', 'anonymizeIp', true); is enough to allow google analytics to be considered as non profiling cookie how can i activate...

Google snippet tree structure


sitemap,rich-snippets,google-rich-snippets
How can I get a snippet like in the picture below in google page results? I submitted a sitemap.xml in google webmaster tools 3 months ago, but there is no change until now? Do you know how I should proceed to get this Result? Or the name (keyword) of this...

TYPO3 Extbase build own Sitemap


typo3,sitemap,extbase
There are a lot of sitemap Generators for TYPO3 in the TER available. But none of them can handle Sites created by Extbase, which are not shown in the TYPO3 page tree. Edit Thanks to biesior, detailed informations: Unconventional I switch in TypoScript the GET Parameter for my Extbase extension[globalVar...

Server side vs client side website


javascript,html,ajax,html5,seo
We have to build a website and we have to chose where to manage the content, in the server (PHP or JSP) or on the client (JavaScript). This article: http://searchenginewatch.com/sew/how-to/2358775/seo-strategies-for-javascript-heavy-single-page-applications-or-ajax-sites enlighted me a bit but Im still doubting. Good SEO is the most important thing to achieve. Can anyone relate...

Multiple modals with galleries vs. a single dynamic one


javascript,dom,seo,image-gallery,bootstrap-modal
Lets say we have a long list of posts on a single page. Each of those posts has a hidden div with multiple img tags inside it. When a user clicks on the post, the images inside the hidden div should be showcased in a modal gallery. Which approach is...

Convert MVC Website to be Responsive


mobile,model-view-controller,seo
While investigating the Google SEO issue with respect to not having a mobile web site. I've learned recently that lack of proper mobile site would reduce the search ratings, and a different content for a mobile and a desktop sites might hurt your SEO as well. Currently we have a...

How seo implemented for distancebetween.com website?


seo,google-search
When I Search for distance between bangalore to mumbai in Google, distancebetween.com comes up in the search results. I mean if I search for distance between any source to destination they have results for that. They have one dynamic page where user can enter source and destination and those inputs...

How do I strip out ?_escaped_fragment_= using .htaccess


ajax,.htaccess,mod-rewrite,seo
Google discovered that I'm allowing end users to navigate my content using ajax loading, and is loading my pages as a user client rather than requesting them as new page loads. So instead of trying to index www.mysite.com/page, it's requesting www.mysite.com/?_escaped_fragment_=/page Which is not at all what I want it...

Change the unique generated title names of friendly-id using attribute of another table


ruby-on-rails,ruby,seo,friendly-url,friendly-id
I have a Company Model, and i am using friendly_id like this friendly_id :name, use: :slugged But since there can be many Company with the same name (different branches). I am trying to handle that case by using city attribute from the address of the Company. But the Company address...

Grails produce seo friendly URLS


grails,seo
I'm very new to grails and I have some questions about creating views with SEO friendly URLs. Lets say I have a page I'd like to call used-products or https://www.sampledomain.com/used-products, how would I go about creating a view and have it resolve for used-products? Another example would be something like...

SPDY on shared host & SEO Sematics


seo,semantics,shared-hosting,spdy,http2
NodeSPDY on shared host I got a webspace hosted by uberspace and want to use NodeSPDY, but there is a loadbalancer in between which cuts off the tls connection. On uberspace one can request a Port to be opened. With this port it is possible to request resources directly by...

Wordpress - Robotx.txt allows admin login?


wordpress,seo,robots.txt
First, i've searched by robots.txt for Wordpress, but, no one told me where is this file. So, I read that the robots.txt in Wordpress is virtual. Ok, no problem. But, where i find this to edit? My Wordpress is allowing the /author/admin and i don't want this. In dashboard, the...

Wordpress - customized pages with blocks - prohibit google seo index of blocks


wordpress,seo,woocommerce,robots.txt,google-sitemap
I'm using Wordpress and WooCommerce for my online shop. With the theme I'm using you can customize the product-category pages by adding "blocks". So if I want to have a text on the top of a product category page I simply create a block page, lets say its called "category-info"....

Site name in Google search results for multi-language websites


html,seo,schema.org,google-rich-snippets
Using Schema.org, I can set the name for my website so it’s visible in Google Search: https://developers.google.com/structured-data/site-name Example: <script type="application/ld+json"> { "@context" : "http://schema.org", "@type" : "WebSite", "name" : "Your WebSite Name", "alternateName" : "An alternative name for your WebSite", "url" : "http://www.your-site.com" } </script> What if I have multi-language...

MixItUp vs PageSpeed Insigths


jquery,seo,pagespeed,mixitup
PageSpeed Insights says “Remove render-blocking scripts” and list jquery.mixitup.min.js :_( But the script is included at the bottom of the page (and minified), and the functions that use MixItUp is also on the bottom of the page! I don’t know what can I do. Any suggestions please? Thanks a lot....

How can I get better google indexing results?


seo,google-search,pagerank
I have just launched a new domain, www.nextlevelsmf.com and it's not showing for some keywords I would like it to. Can anyone give me some advice to help it rank better please? I'd like it to show on the first 2 pages for: Managed SMF hosting SMF host/hosting Managed SMF...

Best JSON-LD practices: using multiple