seo,sitemap,robots.txt , Use `robots.txt` in a multilingual site


Use `robots.txt` in a multilingual site

Question:

Tag: seo,sitemap,robots.txt

Having to manage a multilingual site, where users are redirected to a local version of site, like

myBurger.com/en // for users from US, UK, etc..
myBurger.com/fr // for users from France, Swiss, etc...

How should be organized the robots.txt file in pair with the sitemap?

myBurger.com/robots.txt // with - Sitemap: http://myBurger.com/??/sitemap
OR
myBurger.com/en/robots.txt  // with - Sitemap: http://myBurger.com/en/sitemap
myBurger.com/fr/robots.txt  // with - Sitemap: http://myBurger.com/fr/sitemap

kwnowing that en and fr sites are in fact independent entities not sharing common content, even if similar appearance.


Answer:

You need to put one robots.txt at the top level.

The robots.txt file must be in the top-level directory of the host, accessible though the appropriate protocol and port number.

https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt


Related:


Multiple modals with galleries vs. a single dynamic one


javascript,dom,seo,image-gallery,bootstrap-modal
Lets say we have a long list of posts on a single page. Each of those posts has a hidden div with multiple img tags inside it. When a user clicks on the post, the images inside the hidden div should be showcased in a modal gallery. Which approach is...

Wordpress - customized pages with blocks - prohibit google seo index of blocks


wordpress,seo,woocommerce,robots.txt,google-sitemap
I'm using Wordpress and WooCommerce for my online shop. With the theme I'm using you can customize the product-category pages by adding "blocks". So if I want to have a text on the top of a product category page I simply create a block page, lets say its called "category-info"....

TYPO3 Extbase build own Sitemap


typo3,sitemap,extbase
There are a lot of sitemap Generators for TYPO3 in the TER available. But none of them can handle Sites created by Extbase, which are not shown in the TYPO3 page tree. Edit Thanks to biesior, detailed informations: Unconventional I switch in TypoScript the GET Parameter for my Extbase extension[globalVar...

Change the unique generated title names of friendly-id using attribute of another table


ruby-on-rails,ruby,seo,friendly-url,friendly-id
I have a Company Model, and i am using friendly_id like this friendly_id :name, use: :slugged But since there can be many Company with the same name (different branches). I am trying to handle that case by using city attribute from the address of the Company. But the Company address...

proper use link hreflang


html,seo,multiple-languages
I use below link tag for multiple language, have two question should I add <link href="http://domain.com/" rel="canonical"> in url http://domain.com/?hl=en or http://domain.com/?hl=es? should I add <html lang='en'> in url http://domain.com/?hl=en and <html lang='es'> in http://domain.com/?hl=es? <head> <!-- … --> <link href="http://domain.com/" hreflang="x-default" rel="alternate"> <link href="http://domain.com/?hl=en" hreflang="en" rel="alternate"> <link href="http://domain.com/?hl=es"...

How Google “distinguishes” website articles from news? [closed]


html,seo,google-search
When I search keywords from Google, it shows all articles related to these words and It has separate tab called "News", where Google shows related news. How Goolge "Knows" that article from site is about News? I have opened source codes of multiple news websites and they has "itemprops" in...

Slidershow jquery and convert to css


jquery,css,html5,seo,slider
I downloaded script for slider show and it work without problems but after implemented this slide show i have problems with seo optimalization in HTML5. Because this code using this <div u=""> or <img u=""> and its still write me that i cant use this combination div with tag "u"....

Removing the number of first page in Yii2 Pagination from the URL


.htaccess,pagination,seo,yii2
For SEO purposes I need to remove the first page number from the URL. i.e I have the following: example.com/pages/view/1 and example.com/pages/view the two URLs points to the same contents of the view action. I want to make the pagination free from 1 in the URL. i.e first Page link...

Schema.org mandatory fields and the time needed until Google shows changes


seo,schema.org,google-rich-snippets
I have implemented Schema.org (using Microdata) inside my product pages and when I check Google Webmaster Tools it is crawled by Googlebot and interpreted successfully. The point is I have not implemented some properties inside Product type like brand. I need to know whether there is some subset of all...

Best JSON-LD practices: using multiple