seo,search-engine,robots.txt,google-crawlers , Disallow specific folders in robots.txt with wildcards


Disallow specific folders in robots.txt with wildcards

Question:

Tag: seo,search-engine,robots.txt,google-crawlers

Can i hide specific folders from crawlers with wildcards like:

 User-agent: *
 Disallow: /system/
 Disallow: /v*

I want to hide all folders starts with "v" character. It will work this way?


Answer:

You don't need wildcards at all for this. Your example will work, but it would work just as well without the wildcard. Trailing wildcards do not do anything useful.

For example, this:

Disallow: /x

means: "Block any path that starts with '/x', followed by zero or more characters."

And this:

Disallow: /x*

means: "Block any path that starts with '/x', followed by zero or more characters, followed by zero or more characters."

This is redundant, and it blocks all of the same things the first one blocks. The only practical difference is that the second version will fail to work on crawlers that don't support wildcards.


Related:


Can anyone help me make the search bar work as I now have the JS prompt? [on hold]


javascript,html5,search,youtube-api,search-engine
I have created a small program that pulls from the YouTube API which allows you to search for a random video for whatever title you enter when prompted. My goal is to have this work like a search engine. I would like to make my search bar the input instead...

Grails produce seo friendly URLS


grails,seo
I'm very new to grails and I have some questions about creating views with SEO friendly URLs. Lets say I have a page I'd like to call used-products or https://www.sampledomain.com/used-products, how would I go about creating a view and have it resolve for used-products? Another example would be something like...

Wordpress - Robotx.txt allows admin login?


wordpress,seo,robots.txt
First, i've searched by robots.txt for Wordpress, but, no one told me where is this file. So, I read that the robots.txt in Wordpress is virtual. Ok, no problem. But, where i find this to edit? My Wordpress is allowing the /author/admin and i don't want this. In dashboard, the...

What is more important for images - alt tag or name


seo
I am making a blog with huge ammount of images, and one way to do it, is by using Flickr Gallery plugin, which provides a functional gallery or your albumbs, but the links aren't looking good (www.......5129512891.jpg), but they do have the proper alt tags (Red Carpet From Turkey). So...

Search box/field design with multiple search locations


python,search,design,search-engine,pyramid
Not sure if this question is better suited for a different StackExchange site but, here goes: I have a search page that searches a number of different type of things. All (at the moment) requiring a different input field for each type of search. For example, one might search for...

Where can I find a corpus of search engine queries?


nlp,search-engine,google-search,bing
I'm interested in training a question-answering system on top of user-generated search queries but so far it looks like such data is not made available. Are there some research centers or industry labs that have compiled corpora of search-engine queries?

Elasticsearch two sets of terms against two fields


elasticsearch,search-engine
I'm trying to use Elasticsearch to return docs that have different terms in two fields. Not knowing how to write this it would be something like this: query: field1: "term set #1" field2: "very different term set #2" Ideally the term sets would be arrays of strings. I'd like all...

Wordpress - customized pages with blocks - prohibit google seo index of blocks


wordpress,seo,woocommerce,robots.txt,google-sitemap
I'm using Wordpress and WooCommerce for my online shop. With the theme I'm using you can customize the product-category pages by adding "blocks". So if I want to have a text on the top of a product category page I simply create a block page, lets say its called "category-info"....

si​tem​ap-​tax​-po​st_​tag​.xm​l not found - webmaster tools


wordpress,seo
I'm a newbie in webmaster tools. I get 3 errors in webmaster tools: 1.2: We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit. *General HTTP error: 404 not found Sitemap:...

How do I strip out ?_escaped_fragment_= using .htaccess


ajax,.htaccess,mod-rewrite,seo
Google discovered that I'm allowing end users to navigate my content using ajax loading, and is loading my pages as a user client rather than requesting them as new page loads. So instead of trying to index www.mysite.com/page, it's requesting www.mysite.com/?_escaped_fragment_=/page Which is not at all what I want it...

fullPage.js: Make all slides and sections visible in search engine results


jquery,seo,web-crawler,single-page-application,fullpage.js
I'm using fullpage.js jQuery plugin for a Single page application. I'm using mostly default settings and the plugin works like a charm. When I got to the SEO though I couldn't properly make Google crawl my website on a "per slide" basis. All my slides are loaded at the page...

Site name in Google search results for multi-language websites


html,seo,schema.org,google-rich-snippets
Using Schema.org, I can set the name for my website so it’s visible in Google Search: https://developers.google.com/structured-data/site-name Example: <script type="application/ld+json"> { "@context" : "http://schema.org", "@type" : "WebSite", "name" : "Your WebSite Name", "alternateName" : "An alternative name for your WebSite", "url" : "http://www.your-site.com" } </script> What if I have multi-language...

Slidershow jquery and convert to css


jquery,css,html5,seo,slider
I downloaded script for slider show and it work without problems but after implemented this slide show i have problems with seo optimalization in HTML5. Because this code using this <div u=""> or <img u=""> and its still write me that i cant use this combination div with tag "u"....

My website Images not indexed by Google, Yahoo and Bing [closed]


php,codeigniter,seo
I'm using codeigniter framework. why Search Engine's not indexed my website images ? My website has been made since 2013. My website is : www.shadyab.com. It likes groupon website(Offering daily deals at restaurants, retailers and service providers.). An image url : http://www.shadyab.com/assests/images/upload/kaktoos4.jpg What should I do to tell search engines...

Heading order in HTML5


html5,seo,semantic-markup
This is a webpage example of my site: <html> <title> artilce header </title> <body> <header> <h1> nme of website</h1></header> <section> <h2> name of section</h2> <article> <h3>article header</h3> </article> </section> </body> </html> I want to know if this order is correct? Or does it maybe have a bad effect on SEO?...

MixItUp vs PageSpeed Insigths


jquery,seo,pagespeed,mixitup
PageSpeed Insights says “Remove render-blocking scripts” and list jquery.mixitup.min.js :_( But the script is included at the bottom of the page (and minified), and the functions that use MixItUp is also on the bottom of the page! I don’t know what can I do. Any suggestions please? Thanks a lot....

ERROR: index 'products': too many string attributes (current index format allows up to 4 GB)


database,indexing,full-text-search,search-engine,sphinx
Got this when tried to index table in database with 25GB of data. Sphinx contains index declaration with following fields: sql_field_string = field_indexer #some keywords sql_field_string = product_name sql_field_string = description sql_attr_float = price sql_field_string = product_url sql_field_string = image_url sql_field_string = sku sql_attr_uint = merchant_id sql_attr_uint = network_id All...

How seo implemented for distancebetween.com website?


seo,google-search
When I Search for distance between bangalore to mumbai in Google, distancebetween.com comes up in the search results. I mean if I search for distance between any source to destination they have results for that. They have one dynamic page where user can enter source and destination and those inputs...

Block “cloner” servers rendering content from our server


apache,seo,clone,cracking
I have a website of mine (freeofficefinder.com) that is being cloned (see here: thelawyerserviceratings.org) There are actually over 25 websites that are currently cloning our website. Obviously this is hurting our SEO ranking greatly due to "duplicate content". Is there something that I could add to the Apache config file...

SEO and tags with JavaScript functionality


javascript,html,twitter-bootstrap,seo
Since we are diving into SEO guidelines the past weeks we came across a question for which we didn't find a satisfying answer. (We simply didn't agree on this topic). We would like more opinions on this. Since many projects use jQuery and Bootstrap lately, anchor tags often get used...

How can I tell if prerender.io is running correctly on modulus.io?


meteor,seo,prerender,modulus.io
UPDATE I am able to install prerender on the modulus server now. BUT there is a problem with where to place the prerender token: app.use(require('prerender-node').set('prerenderToken', 'YOUR_TOKEN')); Where in the .demeteorized node app does this line go? I am running a meteor app on modulus.io I have installed the https://github.com/prerender/prerender-node package....

proper use link hreflang


html,seo,multiple-languages
I use below link tag for multiple language, have two question should I add <link href="http://domain.com/" rel="canonical"> in url http://domain.com/?hl=en or http://domain.com/?hl=es? should I add <html lang='en'> in url http://domain.com/?hl=en and <html lang='es'> in http://domain.com/?hl=es? <head> <!-- … --> <link href="http://domain.com/" hreflang="x-default" rel="alternate"> <link href="http://domain.com/?hl=en" hreflang="en" rel="alternate"> <link href="http://domain.com/?hl=es"...

Server side vs client side website


javascript,html,ajax,html5,seo
We have to build a website and we have to chose where to manage the content, in the server (PHP or JSP) or on the client (JavaScript). This article: http://searchenginewatch.com/sew/how-to/2358775/seo-strategies-for-javascript-heavy-single-page-applications-or-ajax-sites enlighted me a bit but Im still doubting. Good SEO is the most important thing to achieve. Can anyone relate...

Different addresses for different products


php,seo
I have a table shoes(id,shoename,color,brand,price,imagename,available). I am trying to sell shoes online through my website. Currently what's happening is, catalog.php(a page on my website) shows all the shoes in my table 'shoes'. Here's the code in inside a loop. echo "<div class='shoe-view'>"; echo "<img class='show-view-image' src='scripts/shoes/uploads/".$result["imagename"]."' alt='".$result["imagename"]."'/>"; echo "<form action='viewshoe.php'...

Robots.txt file in MVC.NET 4


asp.net,asp.net-mvc-4,seo,robots.txt
I have read an article about ignoring the robots from some url in my ASP MVC.NET project. In his article author said that we should add some action in some off controllers like this. In this example he adds the action to the Home Controller: #region -- Robots() Method --...

google analytics code on landing page and cookie law


jquery,google-analytics,seo
New EU cookie law do not allow page to set cookies on first load and until user make any action, scroll is consider as implicit acceptance I'm not sure if ga('set', 'anonymizeIp', true); is enough to allow google analytics to be considered as non profiling cookie how can i activate...

Removing the number of first page in Yii2 Pagination from the URL


.htaccess,pagination,seo,yii2
For SEO purposes I need to remove the first page number from the URL. i.e I have the following: example.com/pages/view/1 and example.com/pages/view the two URLs points to the same contents of the view action. I want to make the pagination free from 1 in the URL. i.e first Page link...

Schema.org mandatory fields and the time needed until Google shows changes


seo,schema.org,google-rich-snippets
I have implemented Schema.org (using Microdata) inside my product pages and when I check Google Webmaster Tools it is crawled by Googlebot and interpreted successfully. The point is I have not implemented some properties inside Product type like brand. I need to know whether there is some subset of all...

Canonical url for google to prevent duplicate meta?


seo,meta-tags,google-webmaster-tools
Today i went to Google Webmaster Tools to check for duplicate meta description. On almost all my news pages, this is true cause my rss feeds links to the news piece with a parameter (?rs=rss) so i can track my traffic from rss feeds. I thought the following snippet would...

Change the unique generated title names of friendly-id using attribute of another table


ruby-on-rails,ruby,seo,friendly-url,friendly-id
I have a Company Model, and i am using friendly_id like this friendly_id :name, use: :slugged But since there can be many Company with the same name (different branches). I am trying to handle that case by using city attribute from the address of the Company. But the Company address...

How Google “distinguishes” website articles from news? [closed]


html,seo,google-search
When I search keywords from Google, it shows all articles related to these words and It has separate tab called "News", where Google shows related news. How Goolge "Knows" that article from site is about News? I have opened source codes of multiple news websites and they has "itemprops" in...

Multiple modals with galleries vs. a single dynamic one


javascript,dom,seo,image-gallery,bootstrap-modal
Lets say we have a long list of posts on a single page. Each of those posts has a hidden div with multiple img tags inside it. When a user clicks on the post, the images inside the hidden div should be showcased in a modal gallery. Which approach is...

disqus SEO google crawler doesn't load comments


seo,disqus
I see in google webmaster We were unable to load Disqus. If you are a moderator please see our troubleshooting guide. instead of comments. But i read in the Internet, disqus comments are readable by google Crawler. As i understand to show "We were unable to load..." google had to...

Best JSON-LD practices: using multiple