sitemap,google-search,robots.txt , Robotstxt Google Searchresults

Robotstxt Google Searchresults


Tag: sitemap,google-search,robots.txt

My search results index all the sites on Google. I only want to index the main page and not the /nl/, /en/ and /fr/ How can I prevent this in my robots.txt?

I used

Disallow: /nl/
Disallow: /fr/
Disallow: /en/

But what with my sitemap with the URL how can I allow this one for my Google Webmaster Tools?

My website


You can use the Allow keyword to give access to a URL in a Disallowed directory.

Allow: /nl/sitemap.xml
Disallow: /nl/


Google snippet tree structure

How can I get a snippet like in the picture below in google page results? I submitted a sitemap.xml in google webmaster tools 3 months ago, but there is no change until now? Do you know how I should proceed to get this Result? Or the name (keyword) of this...

Where can I find a corpus of search engine queries?

I'm interested in training a question-answering system on top of user-generated search queries but so far it looks like such data is not made available. Are there some research centers or industry labs that have compiled corpora of search-engine queries?

How can I get better google indexing results?

I have just launched a new domain, and it's not showing for some keywords I would like it to. Can anyone give me some advice to help it rank better please? I'd like it to show on the first 2 pages for: Managed SMF hosting SMF host/hosting Managed SMF...

Search a site for a link with Google?

Can you use Google to search a site for a specific link? I dont know if it would be a relative or absolute link but its an internal link to that domain. Another solution to my issue (but not quite as good) would be to test if a page is...

Is it possible to search by microdata/structured data in a standard Google search?

I have been using a Google Custom Search Engine to return useful results using standard X-Ray techniques and the more:pagemap:: techniques as outlined here Does anyone know how, on a standard google search, I could apply the same idea to query the Structured Data on a page when x-ray...

Use `robots.txt` in a multilingual site

Having to manage a multilingual site, where users are redirected to a local version of site, like // for users from US, UK, etc.. // for users from France, Swiss, etc... How should be organized the robots.txt file in pair with the sitemap? // with - Sitemap:...

How do I set main URL a priority of 1.0 and subdirectory urls priority of 0.5 in sitemap.xml

How can I make the root index in the sitemap a priority of 1.0 and the rest of the sub directory files a priority of 0.5? I'm using this php code to generate my sitemap on the fly when page is accessed. As it is right now, it gives a...

Does JSON-LD have to be embedded?

We are currently using the Microdata format to expose data to search engines and we are looking at exposing more info to be able to support some more advanced Google Search features. As I'm working my way through the fields I'm finding I need information that we currently load asynchronously...

make a google search programmatically and process results

I want to make a search on google, using php or node.js... I not yet decided that, it depends about what answer for this question is easier to implement (the rest of what I want to do is easy in both languages). After make this consultation I want to process...

Robotstxt Google Searchresults

My search results index all the sites on Google. I only want to index the main page and not the /nl/, /en/ and /fr/ How can I prevent this in my robots.txt? I used Disallow: /nl/ Disallow: /fr/ Disallow: /en/ But what with my sitemap with the URL how...

How to get Google to re-index a page after removing noindex metatag?

By accident, I had put <meta name="robots" content="noindex"> into lots of pages on my domain. I have now removed this meta-tag, but how can I get these pages to be re-indexed by Google? Any tip? I have tried re-submitting my sitemap.xml in Webmaster Tools, but I'm not sure if it...

Create SiteMap in ASP.NET MVC,routing,sitemap,mvcsitemap
I Want Create SiteMap For ASP.NET MVC Site. I Write This Code [XmlRoot("urlset", Namespace = "")] public class Sitemap { private ArrayList _map; public Sitemap() { _map = new ArrayList(); } [XmlElement("url")] public Location[] Locations { get { Location[] items = new Location[_map.Count]; _map.CopyTo(items); return items; } set { if...

google sitemap for tx_news records with dd_googlesitemap_dmf (or alternative)

I try to let typo3 generate a sitemap for all the news records. For that I tried the dd_googlesitemap_dmf extention. The dd_googlesitemap works (it creates a sitemap for all the typo3 pages - but not for extensions). I filled in the basic infos into the configuration and called the url...

Why do I get the search bar in a separate line?

I am trying to add a Google search bar in the navbar, but i do not why it comes in a new line as shown below: This is the html that should brings the navbar and its components all in the same column. <div class='col-lg-12 col-md-12 col-sm-12 col-xs-12'> <nav class="navbar...

XML Sitemaps: best practice when listing 'alternatives'?

I have put together a XML sitemap for Google that attempts to explain that my website's five main pages each exist in one of eight languages, so the other languages have more or less identical content, just not in English. Here's a representative snippet of what I'm doing: <url> <loc></loc>...

What should be the name of the sitemap file for Google SEO?

I created a sitemap for my website that contains the below code: <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="" xmlns:xsi="" xsi:schemaLocation=""> <url> <loc></loc> </url> <url> <loc></loc> </url> <url>...

'Distinct' entries only to create a sitemap.xml file using Django (1.5.4)

I'm trying to output a sitemap.xml file within Django and have been successfully doing so as per the following. <url> <loc></loc> </url> <url> <loc></loc> </url> <url> <loc></loc> </url> ...etc However, as you will see from my attached sitemaps file, I am trying to only return the distinct 'name' field as...

How Google “distinguishes” website articles from news? [closed]

When I search keywords from Google, it shows all articles related to these words and It has separate tab called "News", where Google shows related news. How Goolge "Knows" that article from site is about News? I have opened source codes of multiple news websites and they has "itemprops" in...

404 after attempting to implement Django(1.5.4) Sitemaps

I've gone through the process, which is seemingly fairly straightforward of setting up an sitemap.xml file.. I'm getting 404 page when I go to 'page not found' and a message (I've still got debug settings turned on..) along the lines of: " Django tried these URL patterns, in this...

Presta Sitemap Bundle Install Error

I have followed the presta sitemap bundle documentation and I am still having issues. I have this line in my composer.json file: "presta/sitemap-bundle": "dev-master" But I get this error: A typo in the package name The package is not available in a stable-enough version according to your minimum-stability setting On...

Redirects required for pages no longer in sitemap

I have a relatively new site that has just started to pick up a bit of traction in the SERP's. My problem is that I have published it and had it indexed with PHP URL extensions, as follows: And so on. Obviously it is a fairly easy matter...

Change “Opportunities” button name in navigation pane using Site Map

Is it possible that using the SiteMap to change the name of the Opportunity button that appears in the Sales navigation pane to any custom name instead of "Opportunities" ? Or this is a fixed name based on the name of the entity ? ...

escaped_fragment in sitemap or shebang?

My site is a Single Page Application and uses alot of javascript. I've got my server setup to generate the site the user sees with javascript so Google sees the same content. Google knows about this via the fragment meta tag and converts links with the shebang to escaped_fragment to...

MvcSiteMapProvider is using ascx,,,sitemap,mvcsitemapprovider
I have donwloaded the Nuget package, MvcSiteMapProvider. In one project it only downloaded the razor views. In my other, I have both the Razor and ascx files. Also in that project it is default to using the ascx. I want it to use the razor pages instead. Does anyone know...

Microsoft Azure SqlSiteMapProvider Issue

I'm trying to migrate my company website to a Azure Server but I'm getting problems with Sql SiteMap Provider. I'm always getting this error: Parser Error Message: Could not load type 'SqlSiteMapProvider'. Hope someone could help me how to solve this problem....

How to evidence a particular page on Google SERP?

I noticed that some results of searching on Google are not a single url but a single url with a two-column list of what I call 'important links' of this website. For example: If you open Google and search for "", without the double quote, you got this: As you...

SERP is just showing the main page but none of the subpages

Have a look on this search where you can see that just my main page is indexed. But why does Google/Search engines not take and the other subpages? Is my deep linking done in a wrong way? Do the search engines need more time? If they do need more...

delete a node from sitemap xml

I have sitemap format as below. I want to delete a complete node that I find loc. For example: Where a node has <loc>with a value of I want to delete the <url> node and his child I want to delete loc than lastmod than priority and than changefreq...

How seo implemented for website?

When I Search for distance between bangalore to mumbai in Google, comes up in the search results. I mean if I search for distance between any source to destination they have results for that. They have one dynamic page where user can enter source and destination and those inputs...

ASP.NET sitemap , how important?

I am finished with my website, or so I thought. I don't have a sitemap and by some miracle I've missed the entire concept of a sitemap, didn't even know it was a thing. Big shoutout to my CS teachers I guess. I've been reading up on it and it...

Is a product microsite a bad idea for google [closed]

I am a bit confused to if a product microsite/brochure site is a bad idea. Since back links are not a good thing. I've read a fair few articles but couldn't get to any conclusion. Basically I have this client how sells online hundreds of products on their main website....

Add SimpleXMLElement Object to Array

I have and object array as displayed below. Im having trouble adding to this object array as i keep getting an error. Here is how it is at the moment: SimpleXMLElement Object ( [url] => Array ( [0] => SimpleXMLElement Object ( [loc] => [lastmod] => 2015-02-02 [changefreq] =>...

google search results keyboard shortcuts tab navigator

Some will think is not related to programming but I think it is, because most of the time I am encounter programming issues I search in google to find solutions, or ways to do what I plan to do before I am writing it from scratch, lets face it we...

cannot create google sitemap for Magento 2nd storefront

Wondering if anyone can shed some light on this problem... I have my magento install at and i'm also running a 2nd storefront with the install at I can generate a sitemap for no problem ( But when I go to create the sitemap for I...

How to get the HTML source of a google search page?

I'm trying to develop my coding skills, I was trying to design a sort of custom client for viewing google searches. I don't want to use the google API, mainly because I feel like this will cut out a lot of the interesting programming. I tried downloading the webpages, with...

How to increse number of sitemapindex

I'm intereset if i can have many sitemapindex like this: <sitemapindex xmlns=""> <sitemap> <loc></loc> <lastmod>2015-02-01T05:00:34+02:00</lastmod> </sitemap> </sitemapindex> I mean 1 sitemapindex to refer athor sitemapindex , or what is maxium limit for a sitemap? Example if destinatieTag.xml is athor <sitemapindex xmlns=""> <sitemap>...

Regex for URL to sites

I have two URLs with the patterns: 1.http://localhost:9001/f/ 2.http://localhost:9001/flight/ I have a site filter which redirects to the respective sites if the regex matches. I tried the following regex patterns for the 2 URLs above: http?://localhost[^/]/f[^flight]/.* http?://localhost[^/]/flight/.* Both URLS are getting redirected to the first site, as both URLs are...

Using Google Custom Search engine with a little privacy

I would like to use a Google Custom Search Engine on my website. With Google's default implementation, you have to put Javascript on each page that has the search box. For privacy reasons, I would like to load that Javascript only for those users who actually use the search engine....

How to retrieve Single result from google?

I am developing the android application in which user gives the key to search that key has to be search in google and read the output of google (like google voice search). The google gives the multiple results as per the key but i need only first result among the...

Google Un-index / remove page from search results

I hope this is the right place to ask my question: For example I had a webpage: Then from /extra/index.html I have created a new page (for example and completely removed the extra folder from But now, when I try to search in Google "something extra" it...

TYPO3 Extbase build own Sitemap

There are a lot of sitemap Generators for TYPO3 in the TER available. But none of them can handle Sites created by Extbase, which are not shown in the TYPO3 page tree. Edit Thanks to biesior, detailed informations: Unconventional I switch in TypoScript the GET Parameter for my Extbase extension[globalVar...

Is it necesary to generate sitemaps for old indexed urls?

I have a web site with content from 2001 and I need to remake the sitemap. Question arises: if the old urls have already been indexed do I need to add them again (the same urls) to the sitemap even if not haven't changed? for example: the sitemap have this...

XML Sitemap Parsing Error With PHP

I'm using php to create a sitemap xml file for google submission but I'm getting an error in my code, which is: <?php $get_posts_sql = "SELECT * FROM posts ORDER BY added DESC"; $get_posts_res = mysqli_query($con, $get_posts_sql); while($post = mysqli_fetch_assoc($get_posts_res)){ $post_id = $post["id"]; $post_title = $post["title"]; $post_added = $post["added"]; $post_date...