php,node.js,google-search,google-image-search , make a google search programmatically and process results


make a google search programmatically and process results

Question:

Tag: php,node.js,google-search,google-image-search

I want to make a search on google, using php or node.js... I not yet decided that, it depends about what answer for this question is easier to implement (the rest of what I want to do is easy in both languages).

After make this consultation I want to process the result, get the links, the number of results (only with the number of results could be great)...

The searching is for a url image.

Any suggestion??


Answer:

Google has implemented lots of safeguards to ensure that it's search engine can't be scraped. However, Google must still work, that's the whole point. So the best way to do google scraping I've found so far is to control a real web browser.

There's Selenium if you want to go that route. However, I prefer my programs to be self-contained than needing to depend on an installed web browser (I run most of my programs on headless servers). So I prefer using phantomjs which is a full webkit based browser (like Safari and Konqueror) driven javascript.

Phantomjs scripts tend to be verbose however so most people use it with a wrapper such as casperjs, node-horseman or nightmarejs (there are lots more, search npm).

Here's an example of google scraping from node-horseman web page:

var Horseman = require('node-horseman');
var horseman = new Horseman();

var numLinks = horseman
  .open('http://www.google.com')
  .type('input[name="q"]', 'github')
  .click("button:contains('Google Search')")
  .waitForNextPage()
  .count("li.g");

console.log("Number of links: " + numLinks);

horseman.close();

If you know how to inspect a page with the developer tools, you'll know how to write a scraper using phantomjs.


One word of warning. Don't download google search too frequently otherwise google will probably detect your script as a bot and temporarily ban you. Make sure you wait an appropriate amount of time between searches.


Related:


Wordpress log out using URL and redirect to specify page


javascript,php,wordpress
I have one Wordpress installation . i need to log out the user without any indication user is coming from particular URL.Is it possible? My code: <?php if($_GET['logout'] == 1) { $redirect_to = current_page_url(); ?> <script> window.location.href="<?php echo wp_logout_url( $redirect_to ); ?>"; </script> <?php } ?> I am using above...

How to search images by name inside a folder?


php,mysql,image
I have a MySQL table with a column "rounds" and each "rounds" has his own photos. Exemple round1 has photos from start=380 end=385. This means it has 6 photos and the name of the photos contains 380,381,382,383,384 or 385 inside. I use this PHP code to display the photos from...

array and function php


php,arrays
I'm just a beginner in PHP coding. I've been reading through a tutorial, but having some trouble with basic PHP concepts. If you could help me, I'd be much obliged. I'm having trouble understanding why the following code doesn't work. <?php function sum($x, $y) { $z = $x + $y;...

Dynamically select from a dynamically generated dropdown


php,html,select,drop-down-menu
I have a dynamically generated dropdown list - list of course identifiers and names. On the basis of a variable, “assigned_course_id”, I would like to preselect the appropriate value from the dropdown list. My best attempt is as follows. Thanks in advance for your assistance. <select name="course_id" id="course_id"> <?php $assigned_course_id...

Include both local and server at the same time


php
I have an include statement in my program: include_once("C:/apache2.2/htdocs/AdminTool/includes/functions.php"); //include_once("/var/www/AdminTool/includes/functions.php"); I only use one at a time. In the code above I am using it for my localhost. But if I will run it on server, I have to comment the local one. Because it will cause error. Is there...

Laravel 4.2 Sending email error


php,email,laravel,laravel-4
Hello everyone I have an error in laravel when I am sending an email. I have a form with a select tag and when I select the user and click submit I need to send him a mail after I select it. Here is my Controller method: public function store()...

Error connecting to MSSQL using PHP


php,sql-server,pdo,odbc,sqlsrv
I am receiving an error as below: PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[08001]: [Microsoft][ODBC Driver 11 for SQL Server]SQL Server Network Interfaces: Connection string is not valid [87]. ' My codes are as follow: $this->link = new PDO( "sqlsrv:server=$this->serverName:$this->port;Database=$this->db", "$this->uid", "$this->pwd" ); I wish someone can enlighten...

PHP / MySQLi: How to prevent SQL injection on INSERT (code partially working)


php,mysql,mysqli,sql-injection,sql-insert
I am new to PHP and hope someone can help me with this. I would like to store two values (an email and a password) in a MySQL db using PHP. The input is passed to the PHP page via Ajax in jQuery (through an onclick event on a website)....

Why am getting this error?: Unknown column 'firstname' in 'field list'


php,database,mysqli
if(isset($_POST["submit"])) { // Details for inserting into the database $id = htmlentities($_POST["id"]); $firstname = htmlspecialchars($_POST["firstname"]); $lastname = htmlspecialchars($_POST["lastname"]); $username = htmlspecialchars($_POST["username"]); $password = htmlspecialchars($_POST["password"]); // Dealing with inserting $query = "INSERT INTO `myDatabaseForAll`.`users` (`id`, `firstname`, `lastname`, `username`, `password`) VALUES (NULL, $firstname, $lastname,$username,$password)"; $result = mysqli_query($connection,$query);...

How do I display my mysql table column headers in my php/html output?


php,html,mysql,table,data
2 Questions... Scenario: I would like to query my database table via a form and then display the results that occur(if there are results) and my current situation is that it does work but it clears the form completely and leaves my to an empty page with just the results...

nodejs head request isn't triggering events


node.js,http
Here's my code: var http = require('http'); var req = http.request( { host: 'example.com', method: 'HEAD', path: '/' }, function(res){ res.on('end',function(){ console.log('Ended'); }); res.on('finish',function(){ console.log('Finished'); }); res.on('close',function(){ console.log('Closed'); }); } ); req.end(); Strange thing about it is that it doesn't print anything. Takes longer than expected and exits. Is this...

Time format conversion with PHP


php,time
If have the duration for a recipe in the format 1H10M (1 hour, 10 minutes) or 20M (20 minutes). I want to use the format as described in the parentheses. I've tried using strtotime() without luck. Any help would be greatly appreciated....

Composer dump-autoload gives preg_match error


php,composer-php,autoload
I have Composer in my PHP project installed, and want to use the autoloader. On this page I read how the composer.json file should look like and that I should run the command dump-autoload. My composer.json file looks as follows { "require-dev":{ "phpunit/phpunit":"4.5.*", "autoload":{ "psr-0":{ "Yii\\":"yii-1.1.14.f0fee9/" } } } }...

I'd like to count the documents with the matching “name” property AND group them by “name” at the same time


node.js,mongoose,group-by
Let's say I have a User collection: schema = mongoose.Schema({ firstName: { type: String, required: true }, ... ... }); module.exports = mongoose.model("User", schema); I would like to write a mongoose query that would count how many users go by the name Mike, Andy, Jerry... In other words, I would...

How to modify CodeIgniter calendar to handle multiple events per day?


php,codeigniter,calendar
I am creating a calendar without jquery calendar plugin. I could retrieve the data which is in the database to the calendar. But when there are multiple data per day only one data comes to the calendar view. I want it to be like this when there is multiple data....

How to secure configuration file containing database username and password


php,security
Issue In order to connect my PHP code with MySQL database I use PDO way, creating variable, assigning it with new PDO object where arguments contain settings such as server, database, login and password. So in resulting code it could look like this: $DAcess=new PDO("mysql:host=server;dbname=database","login","password"); I don't feel comfortable having...

Symfony 2 unable to pass entity repository to form


php,forms,symfony2,runtime-error
I have a form with a drop down and a set of checkboxes, i've used entity form field type to get the values via DB. it works with one of the entity but not with the other. i have this code seperately inside AddBadgesType there is NO AddBadges entity <?php...

Redis: Delete user token by email ( find Key by Value )


node.js,express,redis
I have followed tutorial on how to create token-based authentication with node from this tutorial http://www.kdelemme.com/2014/08/16/token-based-authentication-with-nodejs-redis/ I got it all worked out, but I got 1 problem. The way I store token is : KEY = TOKEN VALUE = UserData (Username, email, etc.) To protect multiple devices login, I would...

How to get my node.js mocha test running?


javascript,node.js,mocha,supertest
I have developed a service in node.js and looking to create my first ever mocha test for this in a seperate file test.js, so I can run the test like this: mocha test I could not figure out how to get the reference to my app, routes.js: var _ =...

access the json encoded object returned by php in jquery


php,jquery,ajax,json
I want to post some data to php function by ajax, then get the encoded json object that the php function will return, then I want to get the information (keys and values) from this object, but I don't know how, here is my code: $.ajax({ url: "functions.php", dataType: "JSON",...

RecursiveIteratorIterator to fetch subdirectories


php
Currently I am working with directories through php. I am able to list subdirectories for any given directory. However, the results are not 100% what I am looking for. The below code returns subdirectories but in addition it also returns the main directory in the array. How can I only...

Mixing


php
This question already has an answer here: <? ?> tags not working in php 5.3.1 5 answers I had to made some changes to an old PHP-Project (with very poor code quality...) which runs on an PHP 5.3 server. So I've downloaded the project and tried to run it...

Php Mysql Query not working properly


php,mysql
I have a table name tblnetworkstatus and I have 11 columns Id issue_name affected_server affected_service issue_type priority duration status start_date end_date description I am getting id in affected_server and affected_service which I am storing in my DB, now I have three situations Either both affected_server and affected_service has been selected...

PHP Regular Expressions Counting starting consonants in a string


php,regex
I need to find out how many starting consonants a word has. The number is used later in the program. The code below does work, I am wondering if it is possible to do this with a regular expression. $mystring ="SomeStringExample"; $mystring2 =("bcdfghjklmnpqrstvwxyzABCDFGHJKLMNPQRSTWVXYZ"); $var = strspn($mystring, $mystring2); Using a regular...

What type of database is the best for storing array or object like data [on hold]


database,node.js,sockets
I'm just curious what the best method would be if I'm trying to have a bot running on my Node server that I could play Blackjack against. But for multiple connected clients via sockets, each connected socket will have their own bot to play against but I need some way...

php redirection working in chorme but not on firefox


php,google-chrome,mozilla
Below is my php code that caries out the redirection Code Snippet :- echo "<form action='exp_yogesh.php?id=$id' method='post'>"; echo "<td> <input type='image' name='putonline' value='$id' src='images/on_button.png' no-repeat; border:none;' alt='submit'> </td> "; echo "<td> <input type='image' name='putoffline' value='$id' src='images/off_botton.png' no-repeat; border:none;' alt='submit'> </td> "; echo "</form>"; Here's the exp_yogesh.php file <?php include 'includes/connection.php';...

Sockets make no sense?


javascript,node.js,sockets
I'm using the 'ws' library for Node.js. I can write code that sends data from my server to my client, posting a date and time update, and closes the socket when I click a button; var wss = new WebSocketServer({server: server}); console.log("WebSocket server created"); wss.on('connection', function(socket) { // SEND DATE...

Node Server - Source Code accessible


node.js,express
I noticed that when i navigate to localhost:8080/server.js (where my server.js is the server-expressjs obviously) the code of my server is shown in the browser! Even, if i upload the application to openshift, i get the same result (you can test it): http://tickets-shkobba125.rhcloud.com/ http://tickets-shkobba125.rhcloud.com/server.js Is this a security issue? How...

php include capitalization on files


php
This example works fine on my localhost (both files are included), but on my server only the second one is: <?php include('Test.php'); echo '<br/>'; include('test.php'); ?> The only difference is the caps on the second include, so I was trying to figure out how to make the caps not matter....

how to multiply two column names using codeigniter validation rule


php,codeigniter,validation
I have three columns.The product of two columns get into third column name income_amount using codeigniter validation rule.the first column is crop_quantity and the second is per_rate controller $this->form_validation->set_rules('crop_quantity', 'Crop Quantity', 'required|numeric'); $this->form_validation->set_rules('per_rate', 'Per Rate', 'required|numeric|callback_get_product'); $this->form_validation->set_rules('income_amount', 'Income Amount', 'required|numeric');...

Trying to rewrite mysql_* to pdo


php,mysql,pdo
First of all I should say that I started to learn PDO and trying to rewrite some old mine codes. Here is one that I found on tutorial for a jquery/ajax/pdo/mysql comment system which is working. This is the mysql_* part which I'm trying to rewrite -> file submit.php which...

Click on link next link should be display on same page


javascript,php,jquery,html,css3
I have a single page website and need to link the navigation to IDs in the page. I have three links: "About us", "Our Project", "contact". So if user clicks on "About ", the About section will be displayed, same with other links. Inside Our project there is Two buttons...

MySQL Query returning strange values


php,mysql
The query is supposed to do the following: Obtain the question and associated answers based on the identification number of the question. (In this case the identification number is called account_id. Order them so that each question (and it's appropriate answers) are lined up in order. The query: SELECT *...

What are correct permissions for Linux Apache2 PHP 5.3 log file?


php,linux,apache,logging,permissions
I discovered the reason why I was not getting entries into my php_errors.log file related to permissions. Right now, I have set it to 666 (rw-rw-rw-) but surely this is a security weakness? Thus, my question. php.ini file: error_log /var/log/httpd/php_errors.log log_errors On # ls -ld /var/log /var/log/httpd /var/log/httpd/php_errors.log drwxr-xr-x 6...

Socket.IO message doesn't update Angular variable


javascript,angularjs,node.js,sockets
I have a socket.io client-server setup with AngularJS running on the client. // Server.js var io = require('socket.io')(server); io.on('connection', function (socket) { socket.on('message', function (msg) { //console.log(msg); console.log(msg); io.emit('message', msg); }); }); As observed, it essentially emits a message events with the data stored in the variable msg. And then...

node.js winston logger no colors with nohup


node.js,logging,nohup,winston
We are using winston logger in our project with the following transport settings: file: { filename: __base + '/log/server.log', colorize : true, timestamp : true, json : false, prettyPrint : true } If the application is started with nohup, log file is not colorized. It works only without nohup. nohup...

how to escape php code in echo with javascript


javascript,php
how can i escape this : var tab_mois_nb_match = <?php ".json_encode($tab_mois_nb_match)." ;?> ; I have an code error, but the arrays are generate with sussess in the conle.log, it's totally crazy. foreach($tab_bases2 as $key => $univers){ $tab_nb_match_par_user = users_nb_match($univers); $tab_mois_nb_match = mois_nb_match($univers); echo "<div id='".$univers."' ></div>"; echo "<script type='text/JavaScript'> var...

Unable to configure Symfony (3rd party) bundle


php,symfony2,rss
I am faily new to Symfony and I am trying to setup a third party bundle that reads RSS feeds and then insert them into database. The third party bundle I am trying to use is called rss-atom-bundle After reading the instructions I can get the RSS feeds however I...

compare today's date with unix timestamp value in database


php,mysql
In database I am storing date value unix timestamp value for e.g.'1434952110' using time() now I am trying to compare todays value with this value as below $jobpostdate = date("Y-m-d");//today's date passing in database to compare query $sql = "SELECT jsp_title, jsp_subtitle, jsp_desc, jsp_location, jsp_date "; $sql .= "FROM js_projects...

How to register global variable for my Laravel application?


php,laravel,laravel-5
I have started with Laravel a few days ago, and today I just installed the vespakoen/menu that seems to be very nice, and probably will work for what I need it. Currently I have installed Laravel 5.1 on my system. The problem I currently have, is where to register my...

Laravel Interfaces


php,laravel,interface,namespaces
I used the following tutorial to get an idea about interfaces: http://vegibit.com/what-is-a-laravel-interface/ But I wanted to change the directory of where I am putting my interfaces to "App/Models/Interfaces". And so I did. But now I cannot get it to work anymore. Here is my code: Routes.php App::bind('CarInterface', 'Subaru'); Route::get('subaru', function()...

Pull information from SQL database and getting login errors


php,sql,database
I am creating a very small, simple CRM for a company, they require the function to be able to view the last 25 orders via the dashboard. The orders are added via a Order-add form within the CRM. When adding the following code to the CRM I get an error:...

Cant submit form


javascript,php
Basically I've got a form with 5 radio buttons. No submit button. I want the form to run when the user clicks on any of the radio buttons, so this is what I did. <input id="5" type="radio" name="star" onchange="this.form.submit();" <?php if ($row["star"] =="5") echo "checked";?> value="5"/> a querystring is required...

When I click to the next page on pagination,it goes to 404 error in codeigniter


php,codeigniter,pagination
In my News.phpcontroller,i'm having the below code: public function index() { $data['title'] = 'Database Details'; $config = array(); $config['base_url'] = base_url("news/index"); $config['total_rows'] = $this->news_model->record_count(); $config['per_page'] = 5; $config['uri_segment'] = 3; //$config['use_page_numbers'] = TRUE; // $config['page_query_string'] = TRUE; $choice = $config["total_rows"] / $config["per_page"]; $config["num_links"] = round($choice); $this->pagination->initialize($config); $page =...

How can I replace the white rectangle within an image using ImageMagick?


php,image-processing,imagemagick
Overview: The first picture is my original image. Here I want to replace the white rectangle shown with another image. My approach: I have created a mask image using floodfill and it looks as: Problem: Now I would like to get the distance or co-ordinates of the rectangle in the...

Emitting and receiving socket io within the same file


node.js,express,socket.io
I want to do something where I can set up a route to a link like /visit using app.get('/visit', function(req, res){}) etc. And inside that, I want to emit a message like socket.emit("event", "a message") and then inside my io.on("connection") function, be able to listen for event and return the...

mysql_real_escape_string creates \ in server only not in local


php,sql
When I use mysql_real_escape_string in my localhost and I output the result of html in a table I have no problem. But when I use it on my server it outputs even the \ This is how I use it: $_GETVARS['txtEmpNum'] = mysql_real_escape_string($_GETVARS['txtEmpNum']); $_GETVARS['txtLName'] = mysql_real_escape_string($_GETVARS['txtLName']); $_GETVARS['txtFName'] = mysql_real_escape_string($_GETVARS['txtFName']); $varSQL...

Codeigniter PHP Mailer, Sender Info


php,email,codeigniter-2,phpmailer,contact-form
I'm using Codeigniter PHP Mailer which is hosted here: https://github.com/ivantcholakov/codeigniter-phpmailer/ and with gmail smtp it works flawless. However,using it for a standard contact form, when visitors use that contact form and send us an email, they basically send mails from our mail address to our another mail addess. When they...

$http.get returns actual php script instead of running it (yeoman, grunt)


php,angularjs,pdo,gruntjs
I'm building a "simple" AngularJS app with an articles newsfeed. My articles are stored in a mysql database, and I extract them using php PDO. I used to do this using AJAX with a simple LAMP configuration (php5, mysql, apache2), and everything worked as intended. Now I'm trying to rebuild...

How to pass a value from a page to another page in PHP


php
Im trying to pass a variable value from a page to another page. Im using to go to the next page and I use Session to get the value, the link variable does working but the price variable doesn't work Here's the code: index.php <?php require('connect.php'); if(!isset($_SESSION)){ session_start(); } $query...