url,robots.txt,googlebot , EHow to Disallow few list of URL crawled by google crawler using robots.txt

EHow to Disallow few list of URL crawled by google crawler using robots.txt


Tag: url,robots.txt,googlebot

I have couple of pages and URL which I do not want to be crawled by Google crawler.

I know it can be done via robots.txt. I search Google and found this way we need to arrange the whole things in robots.txt for disallow crawler but I am not sure does it right or not.

User-Agent: *
Disallow: /music?
Disallow: /widgets/radio?

Disallow: /affiliate/
Disallow: /affiliate_redirect.php
Disallow: /affiliate_sendto.php
Disallow: /affiliatelink.php
Disallow: /campaignlink.php
Disallow: /delivery.php

Disallow: /music/+noredirect/
Disallow: /user/*/library/music/
Disallow: /*/+news/*/visit
Disallow: /*/+wiki/diff

# AJAX content
Disallow: /search/autocomplete
Disallow: /template
Disallow: /ajax
Disallow: /user/*/tasteomatic

Can I give the URL like this way? I mean, can I specify full URL as disallow?

Disallow: http://www.bba-reman.com/admin/feedback.htm


my current robots.txt entries looks like below

User-Agent: *
Disallow: /CheckLogin
Disallow: /DTC.pdf
Disallow: /catalogue/bmw.htm
Disallow: /auto-mine/bmw/index.htm
Disallow: /forums/parent.Jmp('i100')
Disallow: /forums/parent.Jmp('i040')
Disallow: /forums/CodeDescriptions.html
Disallow: /forums/parent.Jmp('i050')
Disallow: /forums/parent.Scl('000','24601')
Disallow: /forums/parent.Jmp('i030')
Disallow: /catalogue/peugeot.htm

is it ok.....just tell me. thanks


The value of the Disallow field is always the beginning of the URL path.

So if your robots.txt is accessible from http://example.com/robots.txt, and it contains this line

Disallow: http://example.com/admin/feedback.htm

then URLs like these would be disallowed:

So if you want to disallow the URL http://example.com/admin/feedback.htm, you have to use

Disallow: /admin/feedback.htm

which would block URLs like these:


Trying to pass variable in url

I'm echoing this form: echo "<form action='/leaguemaster/fichaTorneio.php?id=" . $_GET['torneioid'] . "'>"; When I submit the form and it links to the page from the action, the variable I'm passing doesn't reach the page... /leaguemaster/fichaTorneio.php? I've done this in other pages and it worked, I don't know what's going on here....

Using URL vs Filepath to include css?

Which one is better ? Using URL to include css ? <link rel="stylesheet" href="<?php echo BASE_URL; ?>/css/style.css"> Or, using filesystem ? <link rel="stylesheet" href="<?php echo BASE_PATH; ?>/css/style.css"> BASE_URL containes the base url of the app, and BASE_PATH contains the file path to the base of the app....

Coinbase Oauth2 - token request URL - “404 Not found”

First steps of the Coinbase Oauth Authorization seem to work fine. I request the customer code via the following URL: "https://www.coinbase.com/oauth/authorize?response_type=code&client_id=XXXXXXXXXXXXXXXXXXXX&redirect_uri=urn:ietf:wg:oauth:2.0:oob&scope=user+balance" I get back the code via URL.. Then trying to request the token with given CODE and CLIENT SECRET and CLIENT ID:...

Punycode for Unicode query parameter

I am trying encode some Unicode URLs with Punycode. These URLs have a query parameter that contains non-ASCII characters, for example: https://en.wiktionary.org/w/index.php?title=Clœlia&printable=yes The problem is, when I try to do it in Java, the resulting URL is wrong: String link = "https://en.wiktionary.org/w/index.php?title=Clœlia&printable=yes"; link = IDN.toASCII(link); // -> link = http://en.wiktionary.org/w/index.xn--php?title=cllia&printable=yes-hgf...

Changing URL as long as you are scrolling the webpage

I was seeing the NASA's press release / image gallery, (you can see it here) and it has come to my attention that as I'm scrolling down the webpage and seeing different posts, the URL in the address bar was changing (I was using chrome, but tried in Internet Explorer...

Why/how does the browser decide ☃.net goes to xn--n3h.net

If we type into firefox or chrome http://☃.net/ It takes us to http://xn--n3h.net/ Which is a mirror of unicodesnowmanforyou.com What I don't understand is by what rules the unicode snowman can decode to xn--n3h, it doesn't look anything like utf-8 or urlencoding. I think I found a hint while mucking...

Force WWW when URL contains path using .htaccess

I'm having a problem with my URL and my sessions. I wish to have ALL website pages be forced to use www. As it looks like now, the website looks like this: www.example.com into www.example.com example.com into www.example.com www.example.com/example/ into www.example.com/example/ example.com/example into example.com/example (this is what's wrong) This is...

Switching language with href

I have three languages running on my website and would like to have working buttons to switch languages on the bottom of my page. The problem is, that the homepage (index.php) requires the suffix ?lang=X while the other pages, (index.php?contact, index.php?about etc) require &lang=X in the URL. I've set up...

FullCalendar event directing to wrong url

So I have set up FullCalendar to my site. I used this tutorial. Everything works fine except the urls I add to events on my calendar. So when I've added a URL to an event it should direct me to that url when I press on the event. For example,...

Servlet path is not included in

I have a simple Java Webapp (eg. test) containing two different SpringMVC application. My web.xml maps them as: <servlet-mapping> <servlet-name>web</servlet-name> <url-pattern>/web/*</url-pattern> </servlet-mapping> <servlet-mapping> <servlet-name>restful</servlet-name> <url-pattern>/restful/*</url-pattern> </servlet-mapping> Inside the web part I'm using "classic" libraries, such as JSTL core. I don't understand how to avoid JSTL c:url tag ignoring the URL...

Combining bookmarklets to create a toggle between two URL

This is basically a variant of Combining Bookmarklets to create a toggle between HTTP and HTTPS? I would like to combine the following two bookmarklets into one: javascript:q=(document.location.href);location=location.href.replace('dp','gp/aw/d') javascript:q=(document.location.href);location=location.href.replace('gp/aw/d','dp') i.e. basically switch between two directories on a server. This does not work (or only for one of the cases, i.e....

Getting HTTP 302 when downloading file in Java using Apache Commons

I am using the following method to download a file off the internet: try { URL url = new URL("http://search.maven.org/remotecontent?filepath=com/cedarsoftware/json-io/4.0.0/json-io-4.0.0.jar"); FileUtils.copyURLToFile(url, new File(jsonerFolder.getParent() + "\\mods\\json-io-4.0.0.jar")); } catch (Exception e1) { logger.error("Downloading json-io failed with an exception"); e1.printStackTrace(); } But the downloaded file is not a jar, rather, it is an...

index.php url need to hide from homepage of php website

The website www.indowesterncaterers.com/ has menu in header.php <ul> <li class="<?php if(basename($_SERVER['SCRIPT_NAME']) == 'index.php'){echo 'active'; }else { echo ''; } ?>"><a href="index.php">Home</a></li> <li class="<?php if(basename($_SERVER['SCRIPT_NAME']) == 'about-us.php'){echo 'active'; }else { echo ''; } ?>"><a href="about-us.php">About US</a></li> <li class="<?php if(basename($_SERVER['SCRIPT_NAME']) == 'plan-your-wedding.php'){echo 'active'; }else {...

Get Parameter from URL using PHP

I have a url: domain.com/?city=newyork&postback=test I am currently successfully passing the postback parameter using the PHP below. However, I can not figure out how to also pass the city parameter. <?php session_start(); if(isset($_SESSION['postback'])) { if($_GET['postback'] == "") { header ("Location: qualify-step2.php?postback=".$_SESSION['postback']); } } ?> Can someone please help me edit...

What is the “unique” keyword in robots.txt?

I have the following code in robots.txt to allow crawling from everyone for now User-agent: * Disallow: Before I changed this the layout of the file was this below. I've been looking for details about unique and I can't find it. Anyone see this before and what is "unique" doing...

How do I load an HTTP URL with App Transport Security enabled in iOS 9?

So, the new beta SDK of iOS released last night has "App Transport Security" which encourages developers to use https instead of http. In principle, this is a great idea, and I already use https in our staging/production environments. However, I don't have https set up in my local development...

Rails view URL to another website

In my view I display a url to another website like so: <a href="url"><%= @article.url %></a> However, when I click it, my website crashes(it probably interprets it as a url to another page within the website). How can I change this code such that the url takes me to another...

Wordpress - customized pages with blocks - prohibit google seo index of blocks

I'm using Wordpress and WooCommerce for my online shop. With the theme I'm using you can customize the product-category pages by adding "blocks". So if I want to have a text on the top of a product category page I simply create a block page, lets say its called "category-info"....

How can I detect a URL really exists despite a 301 or 302 redirect?

I am looking to write a function in python to confirm the existence of files that might exist on a remote server. Some of the downloads I am checking, however, are behind 301 or 302 redirects. I have tried using the requests module as follows: def exists(path): r = requests.head(path)...

where to put robots.txt for a CodeIgniter

Where to place the robots.txt file in codeigniter. I dont no how to put where folder. User-agent: * Disallow: / ...

Fetching URL vars into form and submitting to other page

I have this url: torneioJogo.php?equipaleft=24&equiparight=25&torneioid=8 The following code is in this url... <form method='GET' action='torneioJogoSub.php?'> //THESE INPUTS DOWN HERE <input type="hidden" name="equipa1" value="<?php $_GET['equipaleft'] ?>"> <input type="hidden" name="equipa2" value="<?php $_GET['equiparight'] ?>"> <input type="hidden" name="torneioid" value="<?php $_GET['torneioid'] ?>"> <div id="pontos"> <input type="submit" name="win1"...

Open website url links with parameter by using a batch file

I am trying to open some website url links with a parameter in my browser. I made a batch file using this code : @echo off start "images" "www.images.com/1000/image.jpg" start "images" "www.images.com/1001/image.jpg" start "images" "www.images.com/1002/image.jpg" ... ... start "images" "www.images.com/5000/image.jpg" All I want is to create a For loop that...

How to generate a new url for each site

I'm currently making a website with the system of replacing content of the main div in the "body" part. Now I figured out that I messed up a bit because every content will be placed in the same url. Do you have a idea of fixing this? HTML/PHP(index.php): <html> <head>...

What is Fragment URLs and why to use it

I am new in PHP Development. Today I came across the interesting topic URL fragmentation specifically the '#' part of the URLs. I searched about that It says it's like www.example.com\foo.html#bar. But I don't understand why this "#bar" is needed. and how to read it by PHP?...

how is youtube url constructed?

I am wondering how the youtube url is constructed? for example https://www.youtube.com/watch?v=L7Jd910Bw84 how is the value L7Jd910Bw8 created, is it a hash value ? is there any PHP function to use for creating this kind of urls? how is it possible to make sure that the two diffrent objects dont...

What is the semantic HTML tag to display for URLs that are not links?

I have a search engine plugin that outputs results in a basic structure: <a href="page url"> <div class="searchResult"> <h3>Page title</h3> <p>Page url</p> <p>Page <meta> description</p> </div> </a> The plugin is designed to integrate into an existing website so it has no default styling. However, websites are supposed to be usable...

Convert a column of text URLs into active hyperlinks in Shiny

I am creating a user interface for a pathway enrichment program. The results are shown in a table as shown below. Below is a snippet showing that I am using DT::renderDataTable and DT::datatable to output the table in a tab. spia_out() is just a reactive function that runs the pathway...

why i am not able to read Html Content from a website in a file?

I have made a java program where in i can use any website to read its Html Content using Scanner class and Varargs.I am not able to get the output while i am using Scanner class and VarArgs. Below is the following Code. import java.io.FileWriter; import java.io.IOException; import java.io.InputStreamReader; import...

C# mvc4 - direct URL to controller

I have an external URL that points directly to a function in my controller but when I click on it for the first time, instead of reaching the page, I get to the home page. However, on the second click (and the following ones), I can reach the desired page....

Django: html without CSS and the right text

First of all, this website that I'm trying to build is my first, so take it easy. Thanks. Anyway, I have my home page, home.html, that extends from base.html, and joke.html, that also extends base.html. The home page works just fine, but not the joke page. Here are some parts...

Rails less url path change

Developing a Rails application with the less-rails gem I found something unusual : // app/assets/common/css/desktop/typo.less @font-face{ font-family:'SomeFont'; src:url("fonts/db92e416-da16-4ae2-a4c9-378dc24b7952.eot?#iefix"); // ... } The requested font is app/assets/common/css/fonts/db92e416-da16-4ae2-a4c9-378dc24b7952.eot This font is compiled with less and the results is : @font-face { font-family: 'SomeFont'; src: url("desktop/fonts/db92e416-da16-4ae2-a4c9-378dc24b7952.eot?#iefix"); //... } Do you know why is...

htaccess rewrite rules conflict

www.example.com/category/sub_category/Having problem with the rewririte rules in .htaccess file. my current .htaccess file looks like this. RewriteRule ^([0-9a-zA-Z_-]+)/([0-9]+)$ products.php?cat=$1&id=$2 [NC,L] RewriteRule ^([^/]*)/([0-9a-zA-Z_-]+)/([0-9]+)$ product_categories.php?cat=$2&id=$3 [NC,L] RewriteRule ^(.*)/(.*)/([0-9a-zA-Z_-]+)/([0-9]+)$ product_details.php?cat=$3&id=$4 [NC,L] RewriteRule ^([0-9]+)$ gallery.php?id=$1 [NC,L] I'm trying create urls like the following. www.example.com/product_name/1...

Java - Getting the URL's parameter

I have a URL something like this I'm using play framework HTTP Routes: GET /students controllers.Application.Get() POST /students controllers.Application.Post() PUT /students/{studentNo} controllers.Application.Put() How will I get the {studentNo} value? For example I'm using a postman(rest client) and I typed something in the url http://localhost:9000/students/2012111222 , how will I get the...

Iframe not passing url parameters

I am loading a web page inside an iframe. If I load the page without the iframe I using the same url it works as expected. However if the iframe loads the url it doesn't work. I have tried googling it and all I can seem to find is how...

Hiding elements based on URL

I'm trying to hide an element that is created dynamically, below is the code I have tried but for some reason the menu still appears after the JavaScript has run. It works in an html page, but not in asp.net. <script type="text/javascript"> if (window.location.href.indexOf("http://www.test.com/xzy") != -1) { $("#Menu").hide(); } </script>...

What is wrong with this Google Elevation API URL?

I am making an application using mainly Python to find the most fuel efficient route using primarily Google Maps data. I have been using the Encoded Polyline Algorithm to get elevation data, as well as a small Python module for the conversions. Normally, the conversion from LatLngs into polyline works...

PHP - url path as parameters

My question might be silly, but I have no idea how to do it. I'm using PHP for creating websites. To change content of the page there is script which includes different file based on url parameter, eg. http://example.com/index.php?page=news This loads some news page. When I want to load some...

Get special character from Query string in classic asp

I have an URL like ViewProfile.asp?mem_id=phani#1&page=1 I am trying to get the value of mem_id by using request.Querystring("mem_id"), I am getting only phani. The part #1 of it is not coming to the variable. I have used HttpUtility.UrlDecode() and Server.UrlDecode(). I am not sure whether these are there. Can someone...

How to build custom URL by adding ?¶ms in Angular?

I'm trying to build a custom URL and update the URL as the user selects items in my dashboard. For example, after clicking a few items the URL could/should look like: #/dashboard?&portfolio=GOOG&ticker1=GOOG Currently the URL only ends in /dashboard (console.log($location.path());) How would I update the $location.path() to add params like...

Htaccess rewrite URL with virtual directory and 2 variables

its my URL... result.php?team=arsenal&player=ospina I want to like this mysite.com/virtualdirectory/arsenal/ospina.html I tried this code.. cant work.. Not Found RewriteRule ^(.*)/(.*)/(.*).html$ result.php?team=$2&player=$3 [L] The requested URL /subfolder/arsenal/ospina.html was not found on this server. Apache/2.2.8 (Win32) PHP/5.2.6 Server at localhost Port 80 Thanks for helps, Best regards!!...

CakePHP 1.3 - Extract URL GET parameters

I have CakePHP 1.3.2 application working with similar URLs: /controller/action/name1:value1/name2:value2.html I'm trying to extract the values after the : sign specified by the names before that. I can see the whole URL with no problem using $this->params['url'], however, I can't get the value itself using neither $this->params['name1'], nor $this->params['url']['name1']. Any...

Build urls with parameters

I'm trying to generate a URL similar to https://website.com/oltp-­‐web/processTransaction?REQUEST_TYPE=2&MID=5 I'm using PHP's http_build_query function but it's not generating a proper URL. Code sample: <?php $parameters =array( 'https://pguat.paytm.com/oltp-­‐web/processTransaction?', 'REQUEST_TYPE'=>'2', 'MID'=>'5'); $url = http_build_query($parameters); echo $url; ?> ...

Hiding param of struts.xml values in Struts 2

Is there any way to hide the param values in struts.xml as they are visible on the URL when I pass them from one action to another? <action name="BackToManagerIndex" class="Action.LoginAction" method="ManagerList"> <result type="redirectAction"> <param name="actionName">login</param> <param name="namespace">../Manager</param> <param name="User_Id">%{User_Id} </param> <param name="Password">%{Password}</param> </result> Here when ever the BackToManagerIndex action is...

Android BufferedReader does not read the whole response

I have a problem with reading a ULR response. On Android it only reads around the half of the response. If I use the same code in a normal Java project everything works fine. try { String _output = null; URL url = new URL("http://example.com"); BufferedReader buffer = new BufferedReader(new...

Encrypted Querystring in URL getting changed to lowercase in Outlook

I am providing a CANCEL button in a registration email, so that the user can click the link and cancel their registration. This works fine except Outlook is converting links to lowercase. So when the user clicks the link, I can't decrypt the URL because the encrypted querystring is now...

Upload info that appear when share web link on Facebook

I have made a website. I did it with wordpress, making a custom theme. All works fine, but I have a problem. When I share any url on facebook, the info is not showed correctly. Only shows correctly if I go to open graph stories and I fetch the newest...

GitHub API returned Url meaning

Take this very simple API as example, https://api.github.com/users/zhuhuihuihui In the JSON formatted response, you'll find something like following, "following_url":"https://api.github.com/users/zhuhuihuihui/following{/other_user}", "gists_url": "https://api.github.com/users/zhuhuihuihui/gists{/gist_id}", "starred_url": "https://api.github.com/users/zhuhuihuihui/starred{/owner}{/repo}", Ok, I know like the following_url, starred_url is something like a sub API which you can call to fetch who that user are following, or what repos...

Clean Urls with regular expression

i have Data in thousand lines like http://xxxx.com/xxx-xxx-xxx-xxxx/ 60% 2 Weekly 2014-01-01 00:00 want to remove everything after / in every url (output should be in clean url like below) http://xxxx.com/xxx-xxx-xxx-xxxx/ Thanks ...

CodeIgniter URL parsing incorrectly

I have a code igniter site which I have enabled some special characters so that characters such as £ can be used in the URL: EG: http://mysite.co.uk/products/details/Product-name-£ When I click on the URL, it tries to take me to the product page and then redirects me back to the homepage....

Run tomcat application from any subdomain, subfolder or ROOT

I'm new to java web development so bear with me. I have created a application which runs on localhost:8080. The application uses a simpel MVC setup (controller servlet forwards request to jsp page). If I deploy my application to a remote server in a subfolder (www.myserver.com/myapplication) the application does not...