seo,sitemap,robots.txt , Use `robots.txt` in a multilingual site


Use `robots.txt` in a multilingual site

Question:

Tag: seo,sitemap,robots.txt

Having to manage a multilingual site, where users are redirected to a local version of site, like

myBurger.com/en // for users from US, UK, etc..
myBurger.com/fr // for users from France, Swiss, etc...

How should be organized the robots.txt file in pair with the sitemap?

myBurger.com/robots.txt // with - Sitemap: http://myBurger.com/??/sitemap
OR
myBurger.com/en/robots.txt  // with - Sitemap: http://myBurger.com/en/sitemap
myBurger.com/fr/robots.txt  // with - Sitemap: http://myBurger.com/fr/sitemap

kwnowing that en and fr sites are in fact independent entities not sharing common content, even if similar appearance.


Answer:

You need to put one robots.txt at the top level.

The robots.txt file must be in the top-level directory of the host, accessible though the appropriate protocol and port number.

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


Related:


Code Igniter SEO friendly URL rewriting


php,.htaccess,codeigniter,url-rewriting,seo
I'm facing issues while rewriting URLs in codeigniter framework. Here is my URL http://localhost/article/lg-watch-urbane-vs-moto-360-a-detailed-real-world-comparison/19 And I've a controller like this : class Article extends CI_Controller { function __construct() { $this->set_page_name( "View Article" ); } function index($perma_link="",$id="") { $this->response['article'] = $this->restutil->get("portal/articles/get/".$id,array('id'=>$id))->article; $this->render_view('articles/view'); } } And my .htaccess is like this:...

Grails produce seo friendly URLS


grails,seo
I'm very new to grails and I have some questions about creating views with SEO friendly URLs. Lets say I have a page I'd like to call used-products or https://www.sampledomain.com/used-products, how would I go about creating a view and have it resolve for used-products? Another example would be something like...

Server side vs client side website


javascript,html,ajax,html5,seo
We have to build a website and we have to chose where to manage the content, in the server (PHP or JSP) or on the client (JavaScript). This article: http://searchenginewatch.com/sew/how-to/2358775/seo-strategies-for-javascript-heavy-single-page-applications-or-ajax-sites enlighted me a bit but Im still doubting. Good SEO is the most important thing to achieve. Can anyone relate...

Schema.org mandatory fields and the time needed until Google shows changes


seo,schema.org,google-rich-snippets
I have implemented Schema.org (using Microdata) inside my product pages and when I check Google Webmaster Tools it is crawled by Googlebot and interpreted successfully. The point is I have not implemented some properties inside Product type like brand. I need to know whether there is some subset of all...

Block “cloner” servers rendering content from our server


apache,seo,clone,cracking
I have a website of mine (freeofficefinder.com) that is being cloned (see here: thelawyerserviceratings.org) There are actually over 25 websites that are currently cloning our website. Obviously this is hurting our SEO ranking greatly due to "duplicate content". Is there something that I could add to the Apache config file...

What is more important for images - alt tag or name


seo
I am making a blog with huge ammount of images, and one way to do it, is by using Flickr Gallery plugin, which provides a functional gallery or your albumbs, but the links aren't looking good (www.......5129512891.jpg), but they do have the proper alt tags (Red Carpet From Turkey). So...

How Google “distinguishes” website articles from news? [closed]


html,seo,google-search
When I search keywords from Google, it shows all articles related to these words and It has separate tab called "News", where Google shows related news. How Goolge "Knows" that article from site is about News? I have opened source codes of multiple news websites and they has "itemprops" in...

Wordpress - Robotx.txt allows admin login?


wordpress,seo,robots.txt
First, i've searched by robots.txt for Wordpress, but, no one told me where is this file. So, I read that the robots.txt in Wordpress is virtual. Ok, no problem. But, where i find this to edit? My Wordpress is allowing the /author/admin and i don't want this. In dashboard, the...

Presta Sitemap Bundle Install Error


php,xml,symfony2,composer-php,sitemap
I have followed the presta sitemap bundle documentation and I am still having issues. I have this line in my composer.json file: "presta/sitemap-bundle": "dev-master" But I get this error: A typo in the package name The package is not available in a stable-enough version according to your minimum-stability setting On...

Does document format change behaviour of Google bot in terms of SEO? Like /path and /path.htm .html .php [closed]


seo,googlebot
I was told to delete ".html" from web application links in order to be better seen by Google bot. So example.com/path/to/resource.html should be example.com/path/to/resource. I didn't find any document saying that format of the resource, placed in urls, has any influence as such. No matter if it's php, htm, html...

SEO and tags with JavaScript functionality


javascript,html,twitter-bootstrap,seo
Since we are diving into SEO guidelines the past weeks we came across a question for which we didn't find a satisfying answer. (We simply didn't agree on this topic). We would like more opinions on this. Since many projects use jQuery and Bootstrap lately, anchor tags often get used...

Best JSON-LD practices: using multiple