seo,sitemap,robots.txt , Use `robots.txt` in a multilingual site


Use `robots.txt` in a multilingual site

Question:

Tag: seo,sitemap,robots.txt

Having to manage a multilingual site, where users are redirected to a local version of site, like

myBurger.com/en // for users from US, UK, etc..
myBurger.com/fr // for users from France, Swiss, etc...

How should be organized the robots.txt file in pair with the sitemap?

myBurger.com/robots.txt // with - Sitemap: http://myBurger.com/??/sitemap
OR
myBurger.com/en/robots.txt  // with - Sitemap: http://myBurger.com/en/sitemap
myBurger.com/fr/robots.txt  // with - Sitemap: http://myBurger.com/fr/sitemap

kwnowing that en and fr sites are in fact independent entities not sharing common content, even if similar appearance.


Answer:

You need to put one robots.txt at the top level.

The robots.txt file must be in the top-level directory of the host, accessible though the appropriate protocol and port number.

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


Related:


Presta Sitemap Bundle Install Error


php,xml,symfony2,composer-php,sitemap
I have followed the presta sitemap bundle documentation and I am still having issues. I have this line in my composer.json file: "presta/sitemap-bundle": "dev-master" But I get this error: A typo in the package name The package is not available in a stable-enough version according to your minimum-stability setting On...

How to avoid the multiple path to same file in php using htaccess?


php,html,apache,.htaccess,seo
My url is www.abc.com/cbn/either/index.php and I want to access this by only current url. When I change the path of the index.php file, i.e. www.abc.com/cbn/index.php Still, I am accessing index.php file which is a bad approach for SEO point of view because now google will index two URLs of the...

Site name in Google search results for multi-language websites


html,seo,schema.org,google-rich-snippets
Using Schema.org, I can set the name for my website so it’s visible in Google Search: https://developers.google.com/structured-data/site-name Example: <script type="application/ld+json"> { "@context" : "http://schema.org", "@type" : "WebSite", "name" : "Your WebSite Name", "alternateName" : "An alternative name for your WebSite", "url" : "http://www.your-site.com" } </script> What if I have multi-language...

How to “remove” indexed pages in Google and AngularJS


angularjs,nginx,seo,google-index
I have an AngularJS 1.0.7 web application. Some months ago I sent a sitemap.xml file to Google, so the urls are indexed. Now, I would like to upload a new sitemap file with some updates on those URLs. I mean, the pages are the same but the urls have changed...

si​tem​ap-​tax​-po​st_​tag​.xm​l not found - webmaster tools


wordpress,seo
I'm a newbie in webmaster tools. I get 3 errors in webmaster tools: 1.2: We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit. *General HTTP error: 404 not found Sitemap:...

Best JSON-LD practices: using multiple