proxy,yql,head , No head section when requesting page via YQL API


No head section when requesting page via YQL API

Question:

Tag: proxy,yql,head

I'm trying to get the full page via yahoo YQL API, in order to access metadata fields defined in the target page head section (specifically open graph metadata).

My example URL:

http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%3D%22http://www.google.com%22&format=xml

But it does not contain any metadata parameters...is there another way to call this Yahoo API to include the head section or perhaps a similar service that does include the head section?


Answer:

You cannot use YQL to parse google.com directly, use the Community Tables

Here's an example to get the meta tags of http://php.net/explode via YQL.

SELECT * FROM html WHERE url = 'http://php.net/explode' AND xpath='descendant-or-self::meta'

Partial Query Output:

    <results>
        <meta charset="utf-8"/>
        <meta content="width=device-width, initial-scale=1.0" name="viewport"/>
    </results>

Related:


I dont know how to add Proxy to my Phantomjs script


python,proxy,phantomjs,casperjs
I've been having issues accessing websites and just realized that since the websites are controlled by my company I need to add a step in my script that shows the webpage I have my companies proxy IP or Im on their server. I hope that makes sense. For example if...

DocumentDB Java SDK behind a proxy


java,proxy,azure-documentdb
I'm trying to create a simple web application from the todo tutorial. As I'm behind a corporate proxy, I used the following common java system properties : -Dhttp.proxyHost=myproxy Dhttp.proxyPort=3128 -Dhttps.proxyHost=myproxy -Dhttps.proxyPort=3128 I also try the following parameter: -Djava.net.useSystemProxies=true I still have a client exception java.lang.IllegalStateException: Http client execution failed. at...

I need to do a redirect on a proxied path in Nginx


nginx,proxy,url-redirection
I'm working with a poorly architected app and I have a need to proxy / and redirect it to a sub-directory (unless someone can comeup with a better solution). Here's the layout of the paths the app uses: var securePaths = [ "/someaction", "/anotheraction", "/athirdaction", "/client/interfaceishere/index.html" //path to client interface...

Request from different Country IP


python,proxy,ip
I would like to view the movie content from a different country on the Playstation store. However, the playstation store blocks the IP based on location of the request, so a movie in Canada cannot be viewed from the US: https://store.playstation.com/#!/en-ca/movies/the-house/cid=UV0130-NPVA92773_CN-0000000000236063 Is there a way to do something like the...

Not able to access Server-Sent-Events over Mobile 3g Network


nginx,https,proxy,server-sent-events
I am having an issue with Server Sent events. My endpoint is not available on mobile 3G network. One observation I have is that a https endpoint like the one below which is available on my mobile network. https://s-dal5-nss-32.firebaseio.com/s1.json?ns=iot-switch&sse=true But the same endpoint when proxy passed using an nginx and...

Persistent session in glassfish behind nginx proxy


java,session,nginx,proxy,glassfish
I am running a glassfish webserver behind a nginx proxy (which resolves domains). I want to store @SessionScoped data on the Webserver. My problem is, that nginx opens a new session everytime it connects to glassfish and that session is in no way connected to the client. The client itself...

Using phantomjs print proxy it used to access website


python,proxy,phantomjs
can someone point me in the right direction. Just need some documentation. I manually input a proxy, but I think it might be by passing it. I want to test my script to see if its actually going through my proxy with phantom. It looks like I successfully went through...

Proxying casperjs / phantomjs through Charles


ssl,proxy,phantomjs,casperjs,charles
I'd like to use casperjs / phantomjs to autofill some web forms. In order to debug my casperjs scripts, I'd like to use the (wonderful) Charles Web Proxy as a man-in-the-middle SSL Proxy, but haven't figured out how to make that work in casperjs / phantomjs. What works I've enabled...

How to connect to a SOCKS proxy in a java applet


java,proxy,socks
I've made a Java applet that accesses a webpage. I need it to connect to a website via a SOCKS proxy I've already tried putting this code where my program initialises: System.setProperty("socksProxyHost", "66.85.144.228"); System.setProperty("socksProxyPort", "1080"); But nothing seems to happen and it just uses my normal IP address?...

How to configure AppJS to work with node-http-proxy server


node.js,proxy,server,appjs
I'm trying to get node-http-proxy working with AppJS. Unfortanly it crashes on launch of the app. What I did: Download & extracted AppJS from http://appjs.com/; Installed node-http-proxy with npm install http-proxy; Edited the app.js window.on(create) function: window.on('create', function(){ console.log("Window Created"); window.frame.show(); window.frame.center(); window.frame.setMenuBar(menubar); var http = require('http'), httpProxy = require('http-proxy');...

How to configure proxy using Squid


proxy,squid
I have a connection and I want change the connection behavior of the process to connect through that proxy. I would also like it to use a new IP in the event that it detects that it has been throttled. The new connection process that I want is : connect...

WSO2 DOCTYPE not allowed calling REST Api


proxy,wso2,wso2esb,doctype
i m working with this proxy: <?xml version="1.0" encoding="UTF-8"?> <proxy xmlns="http://ws.apache.org/ns/synapse" name="PubMedDataProxy" transports="https,http" statistics="disable" trace="disable" startOnLoad="true"> <target> <inSequence> <call> <endpoint> <http method="get" uri-template="http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=omim&amp;Mdb=pubmed&amp;id=230900"/> </endpoint> </call> <log level="custom"> <property name="MyLOg"...

Reverse Proxy in Python using WSGI


python,proxy,wsgi,reverse-proxy
app, built with Flask. I want to have the following routing : "/" -> my Flask app "/foo/" -> Reverse proxy toward http://bar/ So far, I don't have any reverse proxy, so my application looks like : import app [...] if __name__ == '__main__': app.app.secret_key = 'XXX' app.app.run(debug=True, use_reloader=False) I...

Redirect Static Request Express Nodejs


javascript,node.js,express,proxy
I am making a game using websockets. I own 3 Nodejs servers: game1.website.com, game2.website.com, website.com. I want game1 and game2 to only handle game logic (websocket communication) while website.com handles all http requests. Players go to game1.website.com and connect with game1.website.com server with websocket. If players request static files (images,...

Is it possible to make two SOCKSv4a session connections over one TCP connection?


sockets,tcp,proxy,protocols,socks
I'm trying to create multiple tunnels out of a single TCP connection to a SOCKSv4a proxy server in order to keep from closing and reopening a bunch of sockets to the SOCKS proxy. Something like "Connection: keep-alive". How can I do that? Does the protocol allow it?...

Node JS TCP Proxy: Set up HTTP Tunnel using Node JS


node.js,networking,tcp,proxy,http-tunneling
I'm trying to implement a TCP proxy in Node JS. I only have some experience with Javascript so I met a lot of problems along the way. I've done a lot of searching for this one but had no luck. I want to implement a two-hop TCP connection for HTTPS...

how do you download files via ant though proxy [duplicate]


java,maven,ant,proxy
This question already has an answer here: Ant GET task and proxy 1 answer I am using ant which uses an build.xml file. This is the command: java version : 1.8.0 ant version: 1.7.1 CLASSPATH=/app/hbase-0.94.27/lib/hadoop-core-1.0.4.jar CFLAGS=-m64 CXXFLAGS=-m64 ant compile-native tar it hangs here: Buildfile: build.xml ivy-download: [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar...

How to check Proxy headers to check anonymity?


python,http,proxy,anonymity
I'm trying to determine high anonymity proxies. Also called private/elite proxies. From a forum I've read this: High anonymity Servers don't send HTTP_X_FORWARDED_FOR, HTTP_VIA and HTTP_PROXY_CONNECTION variables. Host doesn't even know you are using proxy server and of course it doesn't know your IP address. A highly anonymous proxy will...

Ember CLI with Multiple Proxies


proxy,ember-cli
The Problem: I have an Ember CLI Application that will consume multiple APIs, which I need to proxy to in development mode. Background: I have a legacy api which exposes services at /api running on my local development machine at localhost:3000 I have a new api which exposes services at...

How to pass nginx proxy url for socket


node.js,sockets,nginx,proxy,socket.io
I am using socket in my application,i want to pass socket connection url from nginx as proxy url.I am doing it like that My socket code var socket = io.connect('/explorer/socket',{ 'reconnect': true, 'reconnection delay': 500 }); My nginx conf location /explorer/socket { proxy_pass http://xxx.xxx.xx.xxx:3000; } but it is not working,it...

HTTP to HTTPS mapping using proxy servers


apache,ssl,https,proxy,squid
I have a java application which is trying to call a HTTPS endpoint which is setup in my internal network. Also this request go through a corporate proxy. Having said that, I don't want to implement a HTTPS client at my application level. Instead I will just trigger a plain...

How to add custom cookies in browsermob proxy on java


java,cookies,selenium-webdriver,proxy,browsermob
I am writing automation test and to capture the network call made in background, I am using browsermob-proxy. In browsermob-proxy, I want to set cookie before making requests. How can i do it? Below is my code:- String strFilePath = "data.har"; // start the proxy ProxyServer server = new ProxyServer(4444);...

NGINX Serve all static files and forward others to proxy without / giving 403?


node.js,nginx,proxy
I'm trying to configure an nginx server to act as a proxy to a node application. What I would like is to have any static files served by nginx and all other paths forwarded on to the Node.js application (inluding the / path). I've seen the following question/answers: How to...

Proxy awareness with pip


python,proxy,pip
I have tried to set up a python development environment at my work place to create some basic applications. I can't seem to get pip to work, for example I type in pip install numpy and get the following error message: Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by...

Can't install Composer on Ubuntu behind proxy


php,ubuntu,proxy,composer-php
I'm trying to install Composer, in order to use Laravel, but I'm behind the company proxy. The proxy is already configured in the system, so wget --proxy-user=<my_user_name> --proxy-password=<my_password> https://getcomposer.org/installer works (curl doesn't!), and I get the 270kB "installer" file. Next, I'm trying to run php installer as the manual says,...

Nginx proxy_pass doesn't work on port 80


node.js,nginx,meteor,proxy
I am using a DigitalOcean VPS hosting a meteor app. I don't have a domain name yet, so just use the plain IP address. When I set below config and use myipaddress:3000 and myipaddress:8080, both of them worked well; but if I change the 8080 to 80, only myipaddress:3000 works....

Cannot find .bowerrc file


php,ubuntu,proxy,bower
I am trying to install Learning Locker on my Ubuntu 14.04 which runs apache2 and php5, I have installed all the dependencies listed here. How ever when I run composer (actually its composer.phar, I dont know why) it shows that its downloading but bower shows ECONNRESET errors all along. So...

ENOTFOUND error with express.js (nodejs)


javascript,node.js,express,proxy,reverse-proxy
whenever I try to do some redirecting to another web page (e.g. google.com) with my nodejs express server it will fail returning the following error: throw er; // Unhandled stream error in pipe. ^ Error: getaddrinfo ENOTFOUND google.com at errnoException (dns.js:44:10) at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:94:26) I tried many differend...

How to set up a reverse proxy in nodejs for multiple targets?


javascript,node.js,proxy,reverse-proxy
I am using nodejs, expressjs and request to create a reverse proxy. What I want is that when a request is sent to http://localhost:PORT/NAME/ that the corresponding page is loaded. For example, if I enter http://localhost:1234/google/ the google website would be loaded and if I enter http://localhost:1234/stack/ the stackoverflow page...

Web Service Client - Construction Failing Due To Timeout (Proxy Issue)


java,web-services,soap,https,proxy
I have a webservice in Java 1.6 that extends javax.xml.ws.Service. The WSDL URL is located at an HTTPS endpoint and I am behind a corporate proxy (NTLM I believe). I have the proxy host, port, username, and password. I have verified that I can access the WSDL using curl if...

Create proxy according URL path


javascript,node.js,proxy,supertest,node-http-proxy
I need to create proxy according to some URL which is coming from the browser, since I fairly new to this topic Im not sure how to test it...:( I need some way to test it and see that this is actually working I use this following code from this...

Why is my real IP address still visible even when using a proxy?


python,google-chrome,selenium,proxy,selenium-chromedriver
I am trying to understand how it is that our real IP address is discovered even after going through a private exclusive (I'm the only user) high security proxy. PROXY_IP = "<private proxy IP>:<port>" options = webdriver.ChromeOptions() options.add_argument("--proxy-server=%s" % PROXY_IP) driver = webdriver.Chrome(executable_path=".\\driver\\chromedriver.exe", chrome_options=options) driver.get("http://www.stayinvisible.com/") This website displays a bunch...

what is “root:'data' ” extjs store config


javascript,extjs,proxy,treeview
I am very new to ExtJs and I have the following code for a treeView,,which has a default root in the beginning and then an Ajax call to a servlet .But i don't understand the 'root' field with value 'data' .Is data an alias or xType .Here is the code:...

My workplace's proxy is preventing me from using Maven


java,spring,maven,spring-mvc,proxy
I work in the public sector so my workplace controls internet access pretty strictly. I've tried configuring the settings.xml file that M2Eclipse reads to no avail. It may be simple user error in my case, but I'd like to explore other options anyways just in case that it isn't. I...

How Can I Run a Local PHP Simple HTML DOM Parser with a Proxy?


php,curl,proxy,simple-html-dom,proxylocal
I have a PHP Simple HTML DOM Parser locally in MAMP that pulls information and works well with the Japan version of a website, since I'm located in Japan. However, I would like to pull information from the UK version of the site. What is the simplest way to do...

How to specify proxy authentication credentials in Apache httpclient in addition to the target site credentials?


java,authentication,proxy,apache-httpclient-4.x
I need to connect to a remote web site via proxy using Apache httpclient. Both the proxy and the web site are using basic authentication with different logins/passwords. I have the below code that connects to the remote site via proxy. But I can't figure out how to add proxy...

How can I make a special URL like "http://appname/


networking,nginx,proxy,reverse-proxy
I know some applications run on companies' internal networks with URLS like "https:/statsd/", so I was wondering how to go about doing this. I was thinking maybe I would use a reverse proxy like Nginx to proxy all urls that match a particular regex to the correct server, but I...

Node JS TCP Proxy: Reuse socket in callback function


node.js,networking,tcp,proxy,event-handling
I'm trying to implement a TCP proxy in Node JS. I only have some experience with Javascript so I met a lot of problems along the way. I've done a lot of searching for this one but had no luck. The problem occurs when browser sends a CONNECT request for...

npm ERR! Windows_NT 6.3.9600, unable to install cordova in windows8


node.js,cordova,proxy,netbeans-8
I tried to install cordova on my windows8 machine by using the below command in command prompt npm install -g cordova but i'm not able to install because of the following error can any one help me out please. Error: npm ERR! Windows_NT 6.3.9600 npm ERR! argv "C:\Program Files\nodejs\\node.exe" "C:\Program...

Powershell Proxy setting


powershell,proxy,powershell-v2.0
With this cmdlet can I check my proxy setting in powershell: $regKey="HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" $proxyServer = Get-ItemProperty -Path $regKey $proxyServer | fl *proxy* Or with this cmdlet: Get-ItemProperty Registry::HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings' | Select-Object *Proxy* My question is, how can I get the proxy settings for another network user?...

Parse Returned Text into JSON Object in Swift?


json,swift,yql
Solved: I was able to solve the problem in question with this: if let returnedData = NSJSONSerialization.JSONObjectWithData(data, options: nil, error: nil) as? NSString? { let dataString = NSString(data: data!, encoding: NSUTF8StringEncoding)! let trimmedDataString = ((dataString.stringByReplacingOccurrencesOfString("YAHOO.Finance.SymbolSuggest.ssCallback", withString: "") as NSString).stringByReplacingOccurrencesOfString("(", withString: "") as NSString).stringByReplacingOccurrencesOfString(")", withString: "") let dataFromDataString =...

HTTP Proxy Servlet Error


java,maven,tomcat,servlets,proxy
Let me start off by saying this was working yesterday, so I'm totally confused as to why it stopped working all of a sudden. And Java / Plugins are really not my specialty. I'm working on an application that makes POST and GET calls to a server on another domain....

ZMQ: No subscription message on XPUB socket for multiple subscribers (Last Value Caching pattern)


python,sockets,proxy,zeromq,publish-subscribe
I implemented the Last Value Caching (LVC) example of ZMQ (http://zguide.zeromq.org/php:chapter5#Last-Value-Caching), but can't get a 2nd subscriber to register at the backend. The first time a subscriber comes on board, the event[0] == b'\x01' condition is met and the cached value is sent, but the second subscriber (same topic) doesn't...

How to add http proxy for apachebench(ab)


proxy,apachebench
I want to grab http packets which ab sends or receives. So I add a http_proxy in shell command: $ export http_proxy=127.0.0.1:8888 Then I execute ab command: $ ab -c 1 -n 1 http://localhost/ Finally, my proxy(127.0.0.1:8888) failed to get http packets from ab. Is there any way to let...

Apache2 mod_auth_form too many redirects under Location


apache,proxy,virtualhost,http-basic-authentication,mod-auth-form
I am trying to build a Proxy HTTP Authorisation page using mod_auth_form My goal is to have a single Auth page in the DocumentRoot directory, then once the user is connected, just Proxy all routes to the "real" application, running on localhost with another port. I set up my vhost...

Can the HTTP method “PATCH” be safely used across proxies etc.?


http,caching,filter,proxy,http-patch
Suppose my server exposes an HTTP-based API that uses the PATCH method introduced by RFC 5789. Is it possible that clients (browsers or otherwise) behind corporate firewalls, proxies, caches, parental controls filters and the like will encounter any problems using this method? If so, how likely is this? Given that...

Camel as an HTTP Proxy to a REST service - How to route paths and parameters?


rest,servlets,proxy,apache-camel
I am trying to use Apache Camel as an HTTP proxy to a REST service, performing some auth in the middle, and have a route which looks like this (auth removed for testing purposes): from("servlet:apiwrapper?matchOnUriPrefix=true") .to("http://HOST/BASEPATH?bridgeEndpoint=true&amp;throwExceptionOnFailure=false"); When accessing the Camel Servlet path like so: http://CAMELHOST/apiwrapper/node http://CAMELHOST/apiwrapper/node/stuff/blah?etc=t And so on... I...

actionHero.js Set Http Header


proxy,http-headers,actionhero
I want to know if there is a way to Set the Http.Header in actionHero.js: in our server we are working with actionHero.js and Golang I need to send on the "Header" two domain name "client domain & mine(actionHero Service)" to the Golang service. or if there is any other...

Domino Xpages with Reverse Proxy


proxy,xpages,lotus-domino
I have a problem with Domino Web Server with an XPages Application which is placed behind some reverse proxy. The problem is, the proxy forwards all requests from the URL like h2tps://organization/test_server/ to the Domino Web Server. This makes all links in the application brocken. And I don't know how...

Apache proxy - HTTP to HTTPS


windows,apache,proxy,reverse-proxy,mod-proxy
I work with application which comunicate with server over HTTP, but this server runs on HTTPS in my case and there is also basic authentication. There is no way to change this settings. My idea is configure proxy on Apache server and then website wil be accessible at http://localhost/ and...