sitemap,google-search,robots.txt , Robotstxt Google Searchresults


Robotstxt Google Searchresults

Question:

Tag: sitemap,google-search,robots.txt

My search results index all the sites on Google. I only want to index the main page and not the /nl/, /en/ and /fr/ How can I prevent this in my robots.txt?

I used

Disallow: /nl/
Disallow: /fr/
Disallow: /en/

But what with my sitemap with the URL www.eikhoeve.be/nl/sitemap.xml how can I allow this one for my Google Webmaster Tools?

My website


Answer:

You can use the Allow keyword to give access to a URL in a Disallowed directory.

Allow: /nl/sitemap.xml
Disallow: /nl/

Related:


Robotstxt Google Searchresults


sitemap,google-search,robots.txt
My search results index all the sites on Google. I only want to index the main page and not the /nl/, /en/ and /fr/ How can I prevent this in my robots.txt? I used Disallow: /nl/ Disallow: /fr/ Disallow: /en/ But what with my sitemap with the URL www.eikhoeve.be/nl/sitemap.xml how...

MvcSiteMapProvider is using ascx


asp.net,asp.net-mvc,asp.net-mvc-5,sitemap,mvcsitemapprovider
I have donwloaded the Nuget package, MvcSiteMapProvider. In one project it only downloaded the razor views. In my other, I have both the Razor and ascx files. Also in that project it is default to using the ascx. I want it to use the razor pages instead. Does anyone know...

google sitemap for tx_news records with dd_googlesitemap_dmf (or alternative)


typo3,sitemap,typo3-6.2.x
I try to let typo3 generate a sitemap for all the news records. For that I tried the dd_googlesitemap_dmf extention. The dd_googlesitemap works (it creates a sitemap for all the typo3 pages - but not for extensions). I filled in the basic infos into the configuration and called the url...

How to increse number of sitemapindex


xml,seo,sitemap,googlebot
I'm intereset if i can have many sitemapindex like this: <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <sitemap> <loc>https://domain.com/sitemap/destinatieTag.xml</loc> <lastmod>2015-02-01T05:00:34+02:00</lastmod> </sitemap> </sitemapindex> I mean 1 sitemapindex to refer athor sitemapindex , or what is maxium limit for a sitemap? Example if destinatieTag.xml is athor <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <sitemap>...

Does JSON-LD have to be embedded?


html,google-search,json-ld
We are currently using the Microdata format to expose data to search engines and we are looking at exposing more info to be able to support some more advanced Google Search features. As I'm working my way through the fields I'm finding I need information that we currently load asynchronously...

Is it possible to search by microdata/structured data in a standard Google search?


google-search,microdata,google-custom-search,structured-data
I have been using a Google Custom Search Engine to return useful results using standard X-Ray techniques and the more:pagemap:: techniques as outlined here https://developers.google.com/custom-search/docs/structured_search Does anyone know how, on a standard google search, I could apply the same idea to query the Structured Data on a page when x-ray...

How do I set main URL a priority of 1.0 and subdirectory urls priority of 0.5 in sitemap.xml


php,xml,sitemap,priority,sitemap.xml
How can I make the root index in the sitemap a priority of 1.0 and the rest of the sub directory files a priority of 0.5? I'm using this php code to generate my sitemap on the fly when page is accessed. As it is right now, it gives a...

TYPO3 Extbase build own Sitemap


typo3,sitemap,extbase
There are a lot of sitemap Generators for TYPO3 in the TER available. But none of them can handle Sites created by Extbase, which are not shown in the TYPO3 page tree. Edit Thanks to biesior, detailed informations: Unconventional I switch in TypoScript the GET Parameter for my Extbase extension[globalVar...

How to get Google to re-index a page after removing noindex metatag?


web-crawler,sitemap,meta-tags,google-webmaster-tools,noindex
By accident, I had put <meta name="robots" content="noindex"> into lots of pages on my domain. I have now removed this meta-tag, but how can I get these pages to be re-indexed by Google? Any tip? I have tried re-submitting my sitemap.xml in Webmaster Tools, but I'm not sure if it...

Why do I get the search bar in a separate line?


html,css,twitter-bootstrap,google-search
I am trying to add a Google search bar in the navbar, but i do not why it comes in a new line as shown below: This is the html that should brings the navbar and its components all in the same column. <div class='col-lg-12 col-md-12 col-sm-12 col-xs-12'> <nav class="navbar...

Add SimpleXMLElement Object to Array


php,arrays,object,simplexml,sitemap
I have and object array as displayed below. Im having trouble adding to this object array as i keep getting an error. Here is how it is at the moment: SimpleXMLElement Object ( [url] => Array ( [0] => SimpleXMLElement Object ( [loc] => http://jbsoftware.co.uk/ [lastmod] => 2015-02-02 [changefreq] =>...

escaped_fragment in sitemap or shebang?


javascript,ajax,seo,sitemap,shebang
My site is a Single Page Application and uses alot of javascript. I've got my server setup to generate the site the user sees with javascript so Google sees the same content. Google knows about this via the fragment meta tag and converts links with the shebang to escaped_fragment to...

Is a product microsite a bad idea for google [closed]


seo,google-search,backlink
I am a bit confused to if a product microsite/brochure site is a bad idea. Since back links are not a good thing. I've read a fair few articles but couldn't get to any conclusion. Basically I have this client how sells online hundreds of products on their main website....

How can I get better google indexing results?


seo,google-search,pagerank
I have just launched a new domain, www.nextlevelsmf.com and it's not showing for some keywords I would like it to. Can anyone give me some advice to help it rank better please? I'd like it to show on the first 2 pages for: Managed SMF hosting SMF host/hosting Managed SMF...

google search results keyboard shortcuts tab navigator


google-chrome,google-search
Some will think is not related to programming but I think it is, because most of the time I am encounter programming issues I search in google to find solutions, or ways to do what I plan to do before I am writing it from scratch, lets face it we...

Presta Sitemap Bundle Install Error


php,xml,symfony2,composer-php,sitemap
I have followed the presta sitemap bundle documentation and I am still having issues. I have this line in my composer.json file: "presta/sitemap-bundle": "dev-master" But I get this error: A typo in the package name The package is not available in a stable-enough version according to your minimum-stability setting On...

404 after attempting to implement Django(1.5.4) Sitemaps


django,sitemap
I've gone through the process, which is seemingly fairly straightforward of setting up an sitemap.xml file.. I'm getting 404 page when I go to www.example.com/sitemap.xml 'page not found' and a message (I've still got debug settings turned on..) along the lines of: " Django tried these URL patterns, in this...

'Distinct' entries only to create a sitemap.xml file using Django (1.5.4)


python,django,sitemap
I'm trying to output a sitemap.xml file within Django and have been successfully doing so as per the following. <url> <loc>http://example.com/day/AAA</loc> </url> <url> <loc>http://example.com/day/AAA</loc> </url> <url> <loc>http://example.com/day/BBB</loc> </url> ...etc However, as you will see from my attached sitemaps file, I am trying to only return the distinct 'name' field as...

delete a node from sitemap xml


c#,sitemap,mvcsitemap,google-sitemap
I have sitemap format as below. I want to delete a complete node that I find loc. For example: Where a node has <loc>with a value of http://www.my.com/en/flight1. I want to delete the <url> node and his child I want to delete loc than lastmod than priority and than changefreq...

What should be the name of the sitemap file for Google SEO?


seo,sitemap,google-search
I created a sitemap for my website that contains the below code: <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd"> <url> <loc>http://www.example.com/</loc> </url> <url> <loc>http://www.example.com/aboutus.html</loc> </url> <url>...

Create SiteMap in ASP.NET MVC


asp.net-mvc,routing,sitemap,mvcsitemap
I Want Create SiteMap For ASP.NET MVC Site. I Write This Code [XmlRoot("urlset", Namespace = "http://www.sitemaps.org/schemas/sitemap/0.9")] public class Sitemap { private ArrayList _map; public Sitemap() { _map = new ArrayList(); } [XmlElement("url")] public Location[] Locations { get { Location[] items = new Location[_map.Count]; _map.CopyTo(items); return items; } set { if...

Change “Opportunities” button name in navigation pane using Site Map


dynamics-crm,sitemap,dynamics-crm-2015
Is it possible that using the SiteMap to change the name of the Opportunity button that appears in the Sales navigation pane to any custom name instead of "Opportunities" ? Or this is a fixed name based on the name of the entity ? ...

Using Google Custom Search engine with a little privacy


javascript,html,search-engine,google-search
I would like to use a Google Custom Search Engine on my website. With Google's default implementation, you have to put Javascript on each page that has the search box. For privacy reasons, I would like to load that Javascript only for those users who actually use the search engine....

Redirects required for pages no longer in sitemap


.htaccess,redirect,seo,sitemap,google-webmaster-tools
I have a relatively new site that has just started to pick up a bit of traction in the SERP's. My problem is that I have published it and had it indexed with PHP URL extensions, as follows: www.example.com/page.php www.example.com/product.php And so on. Obviously it is a fairly easy matter...

How to retrieve Single result from google?


android,eclipse,google-search,google-search-platform
I am developing the android application in which user gives the key to search that key has to be search in google and read the output of google (like google voice search). The google gives the multiple results as per the key but i need only first result among the...

Use `robots.txt` in a multilingual site


seo,sitemap,robots.txt
Having to manage a multilingual site, where users are redirected to a local version of site, like myBurger.com/en // for users from US, UK, etc.. myBurger.com/fr // for users from France, Swiss, etc... How should be organized the robots.txt file in pair with the sitemap? myBurger.com/robots.txt // with - Sitemap:...

How to evidence a particular page on Google SERP?


seo,sitemap,googlebot,google-sitemap
I noticed that some results of searching on Google are not a single url but a single url with a two-column list of what I call 'important links' of this website. For example: If you open Google and search for "amazon.it", without the double quote, you got this: As you...

Google Un-index / remove page from search results


html,google-search
I hope this is the right place to ask my question: For example I had a webpage: http://something.com/extra/index.html Then from /extra/index.html I have created a new page (for example http://extra.com) and completely removed the extra folder from something.com But now, when I try to search in Google "something extra" it...

XML Sitemaps: best practice when listing 'alternatives'?


xml,sitemap,google-sitemap
I have put together a XML sitemap for Google that attempts to explain that my website's five main pages each exist in one of eight languages, so the other languages have more or less identical content, just not in English. Here's a representative snippet of what I'm doing: <url> <loc>http://www.mywebsite.com/lang/french/</loc>...

make a google search programmatically and process results


php,node.js,google-search,google-image-search
I want to make a search on google, using php or node.js... I not yet decided that, it depends about what answer for this question is easier to implement (the rest of what I want to do is easy in both languages). After make this consultation I want to process...

Regex for URL to sites


regex,sitemap,regex-negation,regex-greedy,regedit
I have two URLs with the patterns: 1.http://localhost:9001/f/ 2.http://localhost:9001/flight/ I have a site filter which redirects to the respective sites if the regex matches. I tried the following regex patterns for the 2 URLs above: http?://localhost[^/]/f[^flight]/.* http?://localhost[^/]/flight/.* Both URLS are getting redirected to the first site, as both URLs are...

Microsoft Azure SqlSiteMapProvider Issue


sql,azure,sitemap,provider,sqlsitemapprovider
I'm trying to migrate my company website to a Azure Server but I'm getting problems with Sql SiteMap Provider. I'm always getting this error: Parser Error Message: Could not load type 'SqlSiteMapProvider'. Hope someone could help me how to solve this problem....

SERP is just showing the main page but none of the subpages


javascript,html,ajax,seo,google-search
Have a look on this search where you can see that just my main page is indexed. But why does Google/Search engines not take arda-maps.org/about/ and the other subpages? Is my deep linking done in a wrong way? Do the search engines need more time? If they do need more...

ASP.NET sitemap , how important?


c#,asp.net,webforms,sitemap
I am finished with my website, or so I thought. I don't have a sitemap and by some miracle I've missed the entire concept of a sitemap, didn't even know it was a thing. Big shoutout to my CS teachers I guess. I've been reading up on it and it...

Is it necesary to generate sitemaps for old indexed urls?


seo,search-engine,sitemap
I have a web site with content from 2001 and I need to remake the sitemap. Question arises: if the old urls have already been indexed do I need to add them again (the same urls) to the sitemap even if not haven't changed? for example: the sitemap have this...

How to get the HTML source of a google search page?


html,vb.net,google-search
I'm trying to develop my coding skills, I was trying to design a sort of custom client for viewing google searches. I don't want to use the google API, mainly because I feel like this will cut out a lot of the interesting programming. I tried downloading the webpages, with...

Where can I find a corpus of search engine queries?


nlp,search-engine,google-search,bing
I'm interested in training a question-answering system on top of user-generated search queries but so far it looks like such data is not made available. Are there some research centers or industry labs that have compiled corpora of search-engine queries?

XML Sitemap Parsing Error With PHP


php,xml,parsing,sitemap
I'm using php to create a sitemap xml file for google submission but I'm getting an error in my code, which is: <?php $get_posts_sql = "SELECT * FROM posts ORDER BY added DESC"; $get_posts_res = mysqli_query($con, $get_posts_sql); while($post = mysqli_fetch_assoc($get_posts_res)){ $post_id = $post["id"]; $post_title = $post["title"]; $post_added = $post["added"]; $post_date...

cannot create google sitemap for Magento 2nd storefront


xml,magento,sitemap
Wondering if anyone can shed some light on this problem... I have my magento install at showcarsign.com and i'm also running a 2nd storefront with the install at showcarboards.com. I can generate a sitemap for showcarsign.com no problem (showcarsign.com/sitemap/sitemap.xml). But when I go to create the sitemap for showcarboards.com I...

Search a site for a link with Google?


google-search
Can you use Google to search a site for a specific link? I dont know if it would be a relative or absolute link but its an internal link to that domain. Another solution to my issue (but not quite as good) would be to test if a page is...

How Google “distinguishes” website articles from news? [closed]


html,seo,google-search
When I search keywords from Google, it shows all articles related to these words and It has separate tab called "News", where Google shows related news. How Goolge "Knows" that article from site is about News? I have opened source codes of multiple news websites and they has "itemprops" in...

How seo implemented for distancebetween.com website?


seo,google-search
When I Search for distance between bangalore to mumbai in Google, distancebetween.com comes up in the search results. I mean if I search for distance between any source to destination they have results for that. They have one dynamic page where user can enter source and destination and those inputs...

Google snippet tree structure


sitemap,rich-snippets,google-rich-snippets
How can I get a snippet like in the picture below in google page results? I submitted a sitemap.xml in google webmaster tools 3 months ago, but there is no change until now? Do you know how I should proceed to get this Result? Or the name (keyword) of this...