FAQ Database Discussion Community


Sending data from a client to a server by means of a http post

php,python,post,python-requests
I'm trying to send data from the client to the server. The client runs a simple python script that uses the 'request' library. The server side consists of another simple php script using the $_POST. I need the webpage to update depending on the data that is given through the...

What is the python 'requests' api equivalent to my shell scripts curl command?

python,bash,curl,python-requests
I have this curl command I use a shell script: curl -X POST -H 'Content-Type: application/json' \ --cert /etc/puppetlabs/puppet/ssl/certs/${PEFQDN}.pem \ --key /etc/puppetlabs/puppet/ssl/private_keys/${PEFQDN}.pem \ --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem \ --data "{ \"name\": \"$GROUP\", \"parent\": \"00000000-0000-4000-8000-000000000000\", \"environment\": \"production\", \"classes\": { \"$CLASS\": {}} }" \ https://${PEFQDN}:4433/classifier-api/v1/groups | python -m json.tool I want to get away...

python-requests does not grab JSESSIONID and SessionData cookies

python,django,session,cookies,python-requests
I want to scrape a pdf file from http://www.jstor.org/stable/pdf/10.1086/512825.pdf but it wants me to accept Terms and Conditions. While downloading from browser I found out that JSTOR saves my acceptance in 2 cookies with names JSESSIONID and SessionData but python-requests does not grab these two cookie( It grab two other...

Google App Engine Python Persistent Session Shared by Multiple Users

python-2.7,google-app-engine,session-cookies,python-requests
What am I trying to do (Background info)? I am tasked with creating a google app engine application (Python 2.7) that posts form data to another server. The authentication for the server I am trying to connect to uses user:password in exchange for a cookie. I am able to post...

How can I set a single proxy for a requests session object?

python,python-requests
I'm using the Python requests package to send http requests. I want to add a single proxy to the requests session object. eg. session = requests.Session() session.proxies = {...} # Here I want to add a single proxy Currently I am looping through a bunch of proxies, and at each...

Extracting data from webpage using lxml XPath in Python

python,xpath,web-crawler,lxml,python-requests
I am having some unknown trouble when using xpath to retrieve text from an HTML page from lxml library. The page url is www.mangapanda.com/one-piece/1/1 I want to extract the selected chapter name text from the drop down select tag. Now I just want the first option so the XPath to...

Alternative to using mechanize to fetch data?

python,post,python-requests,developer-tools,mechanize-python
Currently, I am using mechanize to fill up a form and send a POST request, then I am doing a regex search on the response to get the data ( a floating point number). Is there any way I can do this by just sending a POST request? I know...

Django Rest Framework: empty request.data

python,django,django-rest-framework,python-requests
I have the following code for a view of DRF: from rest_framework import viewsets class MyViewSet(viewsets.ViewSet): def update(self, request, pk = None): print pk print request.data I call the URL via python-requests in the following way: import requests payload = {"foo":"bar"} headers = {'Content-type': 'application/json'} r = requests.put("https://.../myPk", data= payload,...

Is there a way to download the source from pypi using a script?

python,python-requests,pypi
Following the links (Eg: https://pypi.python.org/packages/source/p/py-web-search/py-web-search-0.2.1.tar.gz#md5=4e2f7363acdc1e7c08629edfa7996a5a ) provided on pypi from a browser will allow us to download the source code. Is there a way to do this from a script? So far I have this: import requests s = requests.get('https://pypi.python.org/packages/source/p/py-web-search/py-web-search-0.2.1.tar.gz#md5=4e2f7363acdc1e7c08629edfa7996a5a') with open('pws.tar.gz', 'w') as fp: fp.write(s.text) Note: Opening the file...

Python3 requests module or urllib.request module both retrieving incomplete json

python,json,python-requests
I'm doing some scraping and looking at pages like this one (https://www.endomondo.com/rest/v1/users/20261627/workouts/526622897), but I have not been able to fully retrieve the JSON content.I have tried using both of the following sets of code, but each returns an incomplete JSON object: url = 'https://www.endomondo.com/rest/v1/users/%s/workouts/%s'%(string_use_user, string_use_workout) print(url) response = urlopen(url) try:...

python requests with redirection

python,authentication,redirect,curl,python-requests
Trying to authenticate on http://72.ru site, noticed that there were a redirect to https://loginka.ru/auth/. Found that there were 302 POST with plain credentials in data form. Copying headers from Chrome can reproduce that in cURL, but still can't reach in requests module. Warning: page is full of russian letters, registration...

python requests 400 error

python,python-requests,amazon-mws
Hi I am using python requests module. My code is: import requests payload = {'AWSAccessKeyId':'AKIAJHSXAECVML4XJT7NvVLAQ', 'Action':'ListOrders', 'CreatedAfter':'2015-05-31T18%3A30%3A00Z', 'MarketplaceId':'A21TJRUUN4KGV', 'SellerId':'A3AZITEDAVC4SLU02M7', 'SignatureMethod':'HmacSHA256', 'SignatureVersion':2, 'Timestamp':'2015-06-03T10%3A13%3A48Z', 'Version':'2013-09-01'} response = requests.post('https://mws.amazonservices.in/Orders/2013-09-01', params=payload) print response.status_code These are Fake credentials I get this : 400...

Convert curl into requests [closed]

python,python-requests
How would I convert the following to a requests call? curl -H "Authorization: Bearer ya29.token" https://play.googleapis.com/v1/accounts/1234/avails/ So far I was trying: import requests headers = { 'Authroization': 'Beader ya29.token' } url 'https://play.googleapis.com/v1/accounts/1234/avails/' requests.get(url, headers=headers) But this gives me an error. What would be the correct conversion?...

How to perform an HTTP/XML authentication with requests

python,xml,authentication,cookies,python-requests
I am trying to authenticate to Docushare with Python 3.4 using requests 2.7. I am relatively new to Python and to the requests module but I've done a lot of reading and am not able to make any more progress. My code doesn't give any errors and I receive a...

Python append json to json file in a while loop

python,json,python-requests,github-api,writefile
I'm trying to get all users information from GitHub API using Python Requests library. Here is my code: import requests import json url = 'https://api.github.com/users' token = "my_token" headers = {'Authorization': 'token %s' % token} r = requests.get(url, headers=headers) users = r.json() with open('users.json', 'w') as outfile: json.dump(users, outfile) I...

compare python requests with curl

python,curl,python-requests
I am interfacing with an API using requests and requests_oauthlib. I successfully authenticate and access all the GET methods of the API, but get error 500 with POST methods. For example: r = oauth.post("https://api.timelyapp.com/1.0/1/clients", data={"client":{"name":"newclient", "color":"c697c0" }}, allow_redirects=False, headers={"Content-Type": "application/json"}) The issue is that I tested the same exact call...

python requests handle error 302?

python,redirect,python-requests
I am trying to make a http request using requests library to the redirect url (in response headers-Location). When using Chrome inspection, I can see the response status is 302. However, in python, requests always returns a 200 status. I added the allow_redirects=False, but the status is still always 200....

using requests to login to a website that has javascript login form

javascript,python,ajax,post,python-requests
Let me preface by saying I have very little programming experience. I've learned a bunch in the last few days trying to write this program. I am running Python 2.7 on Windows 7 using PyCharm, requests, Beautiful Soup, and lxml. I am trying to scrape data from a website that...

NoneType Error when using Beautiful Soup object inside function

python,beautifulsoup,python-requests
Why is it that the penultimate line of this snippet completes successfully, but the last one gives the error: TypeError: 'NoneType' object is not callable? What is different inside the scope of the function, and how can it be fixed? import requests from bs4 import BeautifulSoup def findDiv(soup): print soup.body.FindAll("div")...

Retrieve access token for Yahoo API using OAuth 2.0 and Python requests

python,api,oauth,python-requests,yahoo-api
I am trying to retrieve the access token for the Yahoo API, using the explicit grant flow as described in this document: https://developer.yahoo.com/oauth2/guide/flows_authcode Everything is fine until Step 4: Exchange authorization code for Access Token I wrote the following python script to retrieve the code: import urllib2 import requests import...

Python requests base64 image

python,base64,python-requests
I am using requests to get the image from remote URL. Since the images will always be 16x16, I want to convert them to base64, so that I can embed them later to use in HTML img tag. import requests import base64 response = requests.get(url).content print(response) b = base64.b64encode(response) src...

How to get original url from requests

python,python-requests
In doing the following: >>> res=requests.get('http://www.hulu.com/return-of-the-one-armed-swordsman') >>> res.url u'http://www.hulu.com/watch/800769' How would I get the original url that was called from res. That is, how would I get the res object to return http://www.hulu.com/return-of-the-one-armed-swordsman?...

Python requests module url encode

python,python-requests
I'm trying to send json as a parameter thru a get method for an api, I found that the url to which it is hitting is little bit different from the original url. Some ":%20" text is inserted in between the url. Not sure why this difference is coming, Can...

PoolParty Login using python

python,python-3.x,python-requests
I have Python 3.4 installed on my system. How do I login to the website http://termmanagement.poolparty.biz/PoolParty/!/auth/login using a python script? Code 1: import requests login_page = "http://termmanagement.poolparty.biz/PoolParty/!/auth/login" sess = requests.session() sess.post(login_page, {"j_username": "User", "j_password": "Pass"}) Code 2: import requests, json login_url='http://termmanagement.poolparty.biz/PoolParty/!/auth/login' r = requests.post(login_url, auth=('myuser','mypass')) print(r.status_code) print(r.text) Both the...

Pyinstaller error ImportError: No module named 'requests.packages.chardet.sys

python,python-2.7,python-requests,pyinstaller
I can't seem to find the root cause of this. I don't know if it's pyinstaller, a pip problem, the requests module, or something else as nothing can be eliminated conclusively. I wrote a script in python that properly configures a new hardware sonicwall for our enterprise network when we...

How to handle errors in GRequests?

python,python-requests,grequests
I have this code #!/usr/bin/python import grequests urls = [ 'http://google.com', 'http://doesnotexists.tld' ] def do_something(response, **kwargs): print response.text async_list = [] for u in urls: action_item = grequests.get(u, timeout=10, hooks = {'response' : do_something}) async_list.append(action_item) grequests.map(async_list,size=10) How do I handle errors without getting usual Python error messages? For example for...

Convert urllib download to using requests

python,python-requests
How would I do the following using requests ? urllib.urlretrieve('ftp://USER:[email protected]/upload/file.csv', filepath) If I try doing a requests.get(), I get an error that says requests.exceptions.InvalidSchema: No connection adapters were found for 'ftp://....

Calling the mixpanel API never returns a response

python,python-requests,mixpanel
When I use the following Python code to call the Mixpanel API, I never get a response. import requests requests.get("https://data.mixpanel.com/") But when I try in the browser it works fine. (I get the following response: {error: "Not Found"} which is expected.) Why is this? Or what can I do to...

Fast way to check if image on remote URL exists in python

python,django,python-requests
I am using python-requests library to do my requests. On the home page of the website, I get a bunch of images and show them to the user. Sometimes those images get deleted and I get a broken image url. So I want to check whether images exist. Here is...

Python, trouble getting embedded video url

python,web-scraping,beautifulsoup,python-requests
Ok, I have been scratching my head on this for way too long. I am trying to retrieve the url for an embedded video on a web page using Beautiful Soup and requests modules in Python 2.7.6. I inspect the html in chrome and I can see the url to...

Python: ValueError: could not convert string to float:

python-2.7,type-conversion,python-requests
I get a Typecast - Error, when I try to convert unicode-strings, parsed from a website. Here are the relevant codebits: # special IPython command to prepare the notebook for matplotlib %matplotlib inline from fnmatch import fnmatch import numpy as np import pandas as pd import matplotlib.pyplot as plt import...

Printing out errors with Requests library [duplicate]

python,python-requests
This question already has an answer here: Correct way to try/except using Python requests module? 1 answer I've been reading through http://www.mobify.com/blog/http-requests-are-hard/ , which discusses various types of errors which can be encountered with a request. The article focuses on catching each one. I would like to simply print...

Is there a Python HTTP client that lets you set the size in Content-Disposition for posting files?

python,http,python-requests
I am POSTing files from Python to a vendor's API, and the vendor's API is complaining that the content is missing size in the Content-Disposition header. The example they give is like: Content-Disposition: form-data; filename=filename; name=name; size=1234 Is there a Python HTTP client that will let me include the size...

Google OAuth error Required parameter is missing: grant_type

python,flask,google-oauth,python-requests
I'm trying this code: import requests url_token = 'https://accounts.google.com/o/oauth2/token' params = { 'code': request.values.get('code', '', type=str), 'client_id': app.config['SOCIAL_GOOGLE']['consumer_key'], 'client_secret': app.config['SOCIAL_GOOGLE']['consumer_secret'], 'redirect_uri': urllib.quote_plus("https://example.com/google/code"), 'grant_type': 'authorization_code' } out = requests.post(url_token, params=params) using Python-Request in Flask. But I get error response: ERROR - out={ "error" : "invalid_request",...

How to reach a forwarded webpage using Python Requests?

python,web-scraping,python-requests
How can I reach the following webpage using Python Requests? https://www.fidelity.com.hk/investor/en/fund-prices-performance/fund-price-details/factsheet-historical-nav-dividends.page?FundId=10306 The page is forwarded until I click the 2 "Accept" buttons. This is what I do: import requests s = requests.Session() r = s.post("https://www.fidelity.com.hk/investor/en/important-notice.page?submit=true&componentID=1298599783876") r =...

Extracting strings from HTML with Python wont work with regex or BeautifulSoup

python,regex,parsing,beautifulsoup,python-requests
Im using Python 2.7, BeautifulSoup4, regex, and requests on windows 7. I've scraped some code from a website and I am having problems parsing and extracting the bits I want and storing them in a dictionary. What I'm after is text that is presented as follows in the code: @CAD_DTA\">I...

Request call fails on Task queue with Connection aborted, Permission denied

python,google-app-engine,python-requests
When I am making a direct call to the OpenCalais API, it all seems to be fine and I get a 200 request. However the moment I put the same code on a GAE Task queue, something strange happens: response = requests.post('https://api.thomsonreuters.com/permid/calais/', data=news.body.encode('utf8'), timeout=60, headers={'x-ag-access-token': 'xxx', 'content-type': 'TEXT/RAW', 'outputFormat': 'Application/JSON',...

Python 2.7 download images

python,python-2.7,python-requests
I'm using python 2.7 and pycharm is my editor. What i'm trying to do is have python go to a site and download an image from that site and save it to my directory. Currently I have no errors but i don't think its downloading because the file is not...

Python requests_toolbelt MultipartEncoder filename

python,python-requests
Using requests_toolbelt to upload large files in a Multipart form, I have constructed a method below which succeeds in uploading the file, however I cannot access the posted filename. How do I access the filename on the server? # client-side file = open('/Volumes/Extra/test/my_video.mpg', 'rb') payload = MultipartEncoder({file.name: file}) r =...

Requests and Requests-toolbelt on Windows throws UnicodeDecodeError

python,python-requests
This code works well between unix servers, but when the client is Windows, the server consistently throws a UnicodeDecodeError, as per the stack trace below. I cannot figure out why requests is trying to decode a file object. It seems like temp_file on the Windows client takes on a different...

Using Python requests.get to parse html code that does not load at once

python,html,web-scraping,python-requests
I am trying to write a Python script that will periodically check a website to see if an item is available. I have used requests.get, lxml.html, and xpath successfully in the past to automate website searches. In the case of this particular URL (http://www.anthropologie.com/anthro/product/4120200892474.jsp?cm_vc=SEARCH_RESULTS#/) and others on the same website,...

How to send mix of GET and POST in python

python,post,get,request,python-requests
I'm trying to send a mix of GET and POST to a URL using requests module. Is there a way to do this? What I 've tried is the following: import requests payload = {'test': 'test'} r = requests.post("http://httpbin.org/post?key1=val1&key2=val2",params=payload) print r.text but when I see what is actually being sent...

Python Requests to access flask route with url variable

python,flask,python-requests
I have been trying to use requests to access a flask web service I have created. The problem I am encountering though is how to access routes which have url variables in them. for example if I have route: @app.route('/v1/game/<gameid>', methods = ['POST']) def testing(gameid): return jsonify({'status':'success'}) I have not...

Single session multiple post/get in python requests

python,web-crawler,python-requests
I am trying to write a crawler to automatically download some files using python requests module. However, I met a problem. I initialized a new requests session, then I used post method to login into the website, after that as long as I try to use post/get method (a simplified...

Python Requests image upload HTTP POST

python,python-requests
I am having some problem with a POST request that is driving me crazy. I am trying to upload an image using a POST using python requests by replicating the original request First of all, this is the original request I can see with Firebug that the request header is:...

Error downloading a file using python package requests with https

python,python-2.7,python-requests
I'm a begginer in Python and have been struggling with downloading a file using the package requests. My code works for an url with http, but not for https. The code is the following: import requests url = 'https://firms.modaps.eosdis.nasa.gov/active_fire/shapes/zips/Central_America_24h.zip' print "downloading with requests" r = requests.get(url) with open("Central_America_24h.zip", "wb") as...

Automate Streaming API with Python?

python,json,api,python-requests
How to consume response chunk-ed data from Streaming API using Python. Tried with "request" module but after sending request to API, python script hangs with no response written over console....

python requests SSLError

python,ssl,python-requests
While trying to use the SnapWrap library I've come across this error: File "/usr/local/lib/python2.7/dist-packages/requests-2.7.0-py2.7.egg/requests/adapters.py", line 431, in send raise SSLError(e, request=request) requests.exceptions.SSLError: hostname 'android.clients.google.com' doesn't match either of '*.google.com', '*.android.com', '*.appengine.google.com', '*.cloud.google.com', '*.google-analytics.com', '*.google.ca', '*.google.cl', '*.google.co.in', '*.google.co.jp', '*.google.co.uk', '*.google.com.ar', '*.google.com.au',...

Python requests returns: CSRF verification failed. Request aborted

python,python-requests
I want to login to my account as follow: >> session=requests.Session() >> session.get('http://mywebsite.com') >> csrftoken=session.cookies['ThisIsMyToken'] But the following error occurs: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/dist-packages/requests/cookies.py", line 276, in __getitem__ return self._find_no_duplicates(name) File "/usr/lib/python2.7/dist-packages/requests/cookies.py", line 331, in _find_no_duplicates raise KeyError('name=%r, domain=%r,...

Ignoring exceptions for a specific amount of time

python,python-2.7,loops,while-loop,python-requests
Trying to make the try to run in a loop, since I am booting a machine containing the webserver and I want to make it run and not just go direct to the except and stop the script. I have made a while for the http-status code, but that does...

Python requests http post data with cookies

python,cookies,http-post,python-requests
I have written an script in order to login to my account: import requests session=requests.Session() post_data={'login':'login', 'username':'user', 'password':'password'} cookies=session.cookies post_response=session.post(url='https://www.mywebsite.com', cookies=cookies, data=post_data) print post_response.content When I run that the web site returns me an error: It seems your browser doesn't allow cookies, please enable cookies first....

How to retrieve redirect url given in window.location

python,beautifulsoup,web-crawler,python-requests,url-redirection
I am trying to make a crawler using Python. I am making use of beautifulsoup and requests library and need the set of URLs for a given website. However, in a certain part, there is redirect and when I print the response.text i.e the page content I get the following...

Specific string in python requests.get gives ValueError

python,python-3.x,python-requests
Trying to get this specific page... request = requests.get('http://market.yandex.ru/catalog/90555/list') ...gives me a strange error: ValueError Traceback (most recent call last) C:\Python34\lib\site-packages\requests\packages\urllib3\response.py in read_chunked(self, amt) 406 try: --> 407 self.chunk_left = int(line, 16) 408 except ValueError: ValueError: invalid literal for int() with base 16: '' I figured out that some part...

Cannot import requests.packages.urllib3.util 'Retry'

python,python-requests,urllib3
I am using Python 2.7 64 bit on Windows 8. I have Requests version 2.3 installed. I am trying to run this import statement as part of bringing in number of retries within my code: from requests.packages.urllib3.util import Retry I have urllib3 installed also (I've just installed it now via...

Iterate URL's with Python Requests

python,python-requests
I am trying to iterate over a list of URI's that I'm pulling from a CSV. I would appear that requests cannot use a variable in the URL string, but I wanted to check and see if anyone had any thoughts on how to make something like this work. with...

Python URLs in file Requests

python,python-requests
I have a problem with my Python script in which I want to scrape the same content from every website. I have a file with a lot of URLs and I want Python to go over them to place them into the requests.get(url) object. After that I write the output...

Why python print is delayed?

python,python-3.x,web-crawler,python-requests
I am trying to download file using requests, and print a dot every time retrieve 100k size of file, but all the dots is printed out at the end. See code. with open(file_name,'wb') as file: print("begin downloading, please wait...") respond_file = requests.get(file_url,stream=True) size = len(respond_file.content)//1000000 #the next line will not...

Python and pushbullet api: send a file

python,python-requests,pushbullet
I'm trying to send a file using Pushbullet following their API docs. This is my function: def push_file(AccessToken, file_name): f = open(file_name, 'rb') file_type = mimetypes.guess_type(file_name)[0] print("Uploading {0}...".format(file_name)) try: data = { 'file_name': file_name, 'file_type' : file_type } resp = requests.post(UPLOAD_REQUEST_URL, data=data, auth=(AccessToken, '')).json() if resp.get('error') != None: print("Error: {0}".format(resp.get('error')['message']))...

Python change Accept-Language using requests

python,python-requests,http-accept-language
I'm new to python and trying to get some infos from IMDb using requests library. My code is capturing all data (e.g., movie titles) in my native language, but i would like to get them in english. How can i change the accept-language in requests to do that?

Python requests module not using post method

python,forms,http,python-requests,python-3.3
I am trying to post form data to a URL. I am not getting the expected response and curious about some information I am getting from requests module (2.6.2). The following is the post method: >>> response = requests.post(url, data={'uname':user, 'pwd':password,'phrase':'','submit':True}) As you can see I am using the post()...

REQUESTS: Return file object from url (as with open('','rb') )

python,file,download,python-requests,pypdf
I want to download a file straight into memory using requests in order to pass it directly to PyPDF2 reader avoiding writing it to disk, but I can't figure out how to pass it as a file object. Here's what I've tried: import requests as req from PyPDF2 import PdfFileReader...

Converting to json re-orders objects in python

python,json,python-requests,steam,steam-web-api
I'm trying to get a response from http://steamcommunity.com/profiles/76561198081591043/inventory/json/730/2 in python using requests. import requests url = "http://steamcommunity.com/profiles/76561198081591043/inventory/json/730/2" r = requests.get(url) print r.text print r.json() r.text and r.json() return the objects ordered differently. In 'rgInventory' for instance the first 3 "ids": in .text end in 925, 658, 891 but in .json()...

How to pass a nested dictionary to Flask's GET request handler

python,flask,python-requests
I am trying to pass a nested dictionary as a parameter to a GET request, which is handled by a Flask worker. The whole setup is Nginx+Gunicorn+Flask. On the client, I am doing the following: import requests def find_cabin(): party = {'People' : [{'Age': 44, 'Gender': 'F', 'Habits': 'Smoking,Drinking'}, {'Age':...

Customizing GET Response Using Python Requests Module

python,oauth,get,python-requests
I'm a newbie to Python here, so bear with me... Trying to experiment making a simple oAuth call to Instagram API. After you register your application, you get your client ID, client secret, etc, the first step in the oAuth process is to direct the user to this authorization URL:...

Reusing connections in Django with Python Requests

python,django,python-requests,connection-pooling,keep-alive
What's the correct way of reusing Python Requests connections in Django across multiple HTTP requests. That's what I'm doing currently: import requests def do_request(data): return requests.get('http://foo.bar/', data=data, timeout=4) def my_view1(request) req = do_request({x: 1}) ... def my_view2(request) req = do_request({y: 2}) ... So, I have one function that makes the...

Doing http request from behind a Tor relay

python-3.x,python-requests,tor
I am trying to do a http request form behind a tor relay. I have a little script that logs in and gets data from a page like this: import requests def requestAccountData(accountName, accountPass): print("requesting data") credentials = {'loginname': accountName, 'loginpassword': accountPass} s = requests.Session() s.post('https://example.com/account/', data=credentials) button = {...

How to install requests module in Python 3.4, instead of 2.7

python-3.x,pip,python-requests
I have both Python 2.7 and 3.4 installed on my Ubuntu 14.04 machine. I want to install the 'requests' module so it is accessible from Py3.4. When I issued pip install requests on my terminal cmd line I got back: "Requirement already satisfied (use --upgrade to upgrade): requests in /usr/lib/python2.7/dist-packages"...

How to override the .get() method in requests?

python,override,python-requests
I would like the .get() method in requests to do extra operations beside the GET itself: print out "hello world" (in reality this will be logging) wait 5 seconds before issuing the actual GET (in reality this will be a more complex wait-and-retry operation) Right now my simplistic solution is...

where is the urllib3 connection pool bound?

python-requests,connection-pooling,urllib3
If a new requests session is instantiated in different methods like so ... myutil.py import requests def method1(): s1 = requests.Session() def method2(): s1 = requests.Session() Where is the requests.packages.urllib3.poolmanager.PoolManager bound? Is it bound globally so that the connection pool is shared between s1 and s2, or is it bound...

zsh: no matches found: requests[security]

python,security,python-2.7,ssl,python-requests
I am trying to run a python urllib2 script and getting this error: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. After googling the error the solution, on stack overflow...

While loop with else

python,python-2.7,while-loop,python-requests
I am working with a while loop which I am trying to run for 5 minutes and check if it gets the http response code 200. The problem is if it doesn't get 200 it will be keep running, but it doesn't go to the else when the site should...

Python Requests Get XML

python,xml,python-requests
If I go to http://boxinsider.cratejoy.com/feed/ I can see the XML just fine. But when I try to access it using python requests, I get a 403 error. blog_url = 'http://boxinsider.cratejoy.com/feed/' headers = {'Accepts': 'text/html,application/xml'} blog_request = requests.get(blog_url, timeout=10, headers=headers) Any ideas on why?...

Google API Quota on Google API place autocomplete

google-maps,google-api,python-requests,google-places-api
I working on an application I would like to use Google API place autocomplete. It looks like every Google API has a limited use of,2 500 requests per day, even though this special API isn't mentioned in the following page API Google Maps for Work. My question is : How...

Unable to retain login credentials across pages while using requests

python,python-3.x,python-requests
I am pretty new to using urllib and requests module in python. I am trying to access a wikipage in my company's website which requires me to provide my login credentials through a pop up window when I try to access it through a browser. I was able to write...

JSON list/dictionary parsing from API

python,json,python-3.x,python-requests,simplejson
I have developed a small library and am interested in making it easier on users to retrieve data from the JSON lists/dictionaries that are returned. I have created functions that handle the calls using requests. Now suppose I call this function and pass in a few parameters: precip = precipitation_obs(stid='kfnl',...

Caching python web requests for offline use

python-2.7,caching,python-requests
Is there a way to "cache" the requests I make with the python "requests" module in a way that even if I go offline the module still returns the webpage as if I was online? How can I achieve something like this? Currently I only found caching libraries that just...

Using Python's Requests library, cannot get API call to properly format

python,json,api,rest,python-requests
For reference, I'm a novice trying to learn Python and working with British Airway's Flight Offer Basic API, who's documentation can be found here: https://developer.ba.com/docs/api/Flight_Offer_Basic Using the Requests library documentation from docs.python-requests, I'm trying to build a matching API call that actually works, but I'm getting a 403 when not...

Does requests_cache automatically update cache on update of info?

python,caching,python-requests
I have a very unreliable API that I request using Python. I have been thinking about using requests_cache and setting expire_after to be 999999999999 like I have seen other people do. The only problem is, I do not know if when the API works again, that if the data is...

python - requests timeout for the whole request?

python,python-requests
I am crawling many sites for data, but some links are freezing my script permanently. This shouldn't happen, since I used a timeout like this : page = requests.get(url,timeout=4) I want a timeout for the whole request. So when the request take 4 seconds it will stop trying. I searched...

Python Requests - Use navigate site by servers IP

python,python-requests
I want to crawl a site, however cloudflare was getting in the way. I was able to get the servers IP, so cloudflare won't bother me. How can I utilize this in the requests library? For example, I want to go directly to www.example.com/foo.php, but in requests it will resolve...

Get list of all paginated URL's from links in txt file in python requests

python,list,function,python-requests
Hi Guys Define a Function to Get list of all paginated URLs at bottom from links in txt file in python. Here is an example of what i need done. Input link http://www.apartmentguide.com/apartments/Alabama/Hartselle/ Desired Output www.apartmentguide.com/apartments/Alabama/Hartselle/?page=2 www.apartmentguide.com/apartments/Alabama/Hartselle/?page=3 www.apartmentguide.com/apartments/Alabama/Hartselle/?page=4 www.apartmentguide.com/apartments/Alabama/Hartselle/?page=5 www.apartmentguide.com/apartments/Alabama/Hartselle/?page=6...

Is there a better way to retrieve webpage sizes with Python?

python,urllib2,python-requests,urllib
I'd like a sanity check on this Python script. My goal is to input a list of urls and get a byte size, giving me an indicator if the url is good or bad. import urllib2 import shutil urls = (LIST OF URLS) def getUrl(urls): for url in urls: file_name...

Log in to website behind CloudFlare using Python Requests

python,python-requests
Trying to post a login to a website but cant seem to get it working. Here is the returned error: <html> <head><title>400 Bad Request</title></head> <body bgcolor="white"> <center><h1>400 Bad Request</h1></center> <hr><center>cloudflare-nginx</center> </body> Here is form that I'm trying to post the requests to login to: <form name="loginform" id="loginform" method="post" action="login.php" onsubmit="return...

Requests and Requests_Toolbelt filesize limit

python,python-requests,pyramid
Using request_toolbelt's MultipartEncoder, I am able to upload files to my Pyramid server, but only up to a certain size. Once the file are beyond a certain size (not sure of the exact size), requests throws an ConnectionError. My Pyramid server shows absolutely no sign of ever getting the request....

Requests library GET method redirects all responses with a location in the header or only responses with status code of 300s?

python,python-requests
I know that the requests library redirects automatically GET requests that have a status code of 300s. But sometimes, programmers just send the location in the headers, but don't set the status for Redirection (300s). So, I was wondering if requests redirects all responses with a location in the header,...

How to covert curl -F command into python code with requests?

python,python-requests
I am developing with apis however I faced with A problem that I have never met. curl -F "[email protected]_NAME" 'xxxx url' How do i covert it into python code with requests?...

Pause the run method of a QThread to test it

python,pyqt,python-requests,qthread
I create a Worker object like this: from PyQt4 import QtSql, QtCore from requests_futures.sessions import FuturesSession class Worker(QtCore.QThread): def __init__(self): QtCore.QThread.__init__(self) self.exiting = False self.list_urls = # Urls coming from somewhere # List to store the urls of the pages to request self.list_futures_urls = [] def __del__(self): """Method to destroy...

Converting a basic CURL to Python's request

python,curl,python-requests
The following curl works fine: curl -d '{"date":"2015-05-27T03:48:29.002Z"}' http://blabla.com but I can't get it to work with requests: payload = {'date' : '2015-05-27T03:48:29.002Z'} r = requests.get(url, params=payload, headers=h) what am I doing wrong?...

How to specify form field and file type with Requests

python,python-3.x,python-requests
I want to upload an image using requests module (Python 3). Sadly, the server answers my request with an error, saying that I should only upload files of type jpg, png or gif. I guess I should fill every field of the form, but I want to figure out how...

Grequests with exponential backoff

python,python-requests,grequests
The following is a pattern I have for doing simultaneous requests: rs = (grequests.get(url) for url in urls) res_items = grequests.map(rs) for num, res in enumerate(res_items): json_data = json.loads(res.text) However, this crashes with the error ConnectionError: HTTPConnectionPool(host='apicache.vudu.com', port=80): Max retries exceeded with url: about every 5,000 requests. What would be...

Automating filename generation from url text

python,filenames,python-requests
I am parsing some content from the web and then saving it to a file. So far I manually create the filename. Here's my code: import requests url = "http://rads.stackoverflow.com/amzn/click/1593271840" html = requests.get(url).text.encode('utf-8') with open("html_output_test.html", "wb") as file: file.write(html) How could I automate the process of creating and saving the...

How do I skip a requests response with no valid json?

json,python-2.7,python-requests
I am using requests as part of a geocoding process to loop through lots of text and return the coordinates like so: for t in text: payload = {"q":t} located = requests.get("http://localhost:8999/CLIFF-2.1.1/parse/text", params=payload) l = located.json() for i in l['results']['places']['focus']: print i['name'] print i['lat'] print i['lon'] For the most part...

GAE BlobstoreUploadHandler doesn't recognize files

python,google-app-engine,python-requests,blobstore
When I send a request from a client like this: with open(config.datafile, "rb") as f: r = requests.post(config.endpoint, files={"file": f}) print r.text to my GAE app like this: class FileUploadHandler(blobstore_handlers.BlobstoreUploadHandler): def post(self): self.response.out.write(self.get_uploads()) I would expect to get something like this: { "file": "line 1\nline 2\nline 3\nline 4\nline 5\nline 6\nline...

Python 2.7 writing strings elements (character) to a binary file

json,python-2.7,python-requests
I am using Python 2.7 to access an API that returns JSON with a single key="ringtone_file" and an associated value that is an mp3 file encoded for transport via HTTP. I created a bogus mp3 file consisting of 256 bytes in order from 0x00 through 0xff and the returned file...

python lxml.html.parse not reading url

python,lxml,python-requests
Why is html.parse(url) failing, when using requests then html.fromstring works and html.parse(url2) works? lxml 3.4.2 Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)] on win32 Type "copyright", "credits" or "license()" for more information. >>> import requests >>> from lxml import html >>> url = 'http://www.oddschecker.com' >>>...

python requests cookies date format

python,date,python-3.x,cookies,python-requests
I get cookies from site with the help of requests. When I print cookies keys I get the following: [Cookie(version=0, name='Session', value='4safsadfsdfsdfsd06206ba8cea7c2', port=None, port_specified=False, domain='.site.info', domain_specified=True, domain_initial_dot=True, path='/', path_specified=True, secure=False, expires=1433352401, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False), Cookie(version=0, name='UserID', value='555777', port=None, port_specified=False, domain='.site.info', domain_specified=True,...

Python Requests cookies of a specific request within the session

python,http,python-3.x,cookies,python-requests
I'm trying to find a way to extract and optionally delete cookies sent by a specific request, instead of checking the session cookie jar. The Requests library stores cookies of all the requests, and I'm having trouble accessing cookies sent by a request the program is currently working at. Example...

How to get Python Requests to Trust a Self Signed SSL Certificate?

python,ssl,python-requests
import requests data = {'foo':'bar'} url = 'https://foo.com/bar' r = requests.post(url, data=data) If the URL uses a self signed certificate, this fails with requests.exceptions.SSLError: [Errno 1] _ssl.c:507: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed I know that I can pass False to the verify parameter, like this: r = requests.post(url, data=data, verify=False) However,...

Requests dict from cookiejar issue with escaped chars

python,cookies,python-requests
I'm running into some issues getting a cookie into a dictionary with python. It seems to be all escaped somehow even after running the command provided by requests. resp = requests.get(geturl, cookies=cookies) cookies = requests.utils.dict_from_cookiejar(resp.cookies) and this is what cookies looks like {'P-fa9d887b1fe1a997d543493080644610':...

How to grab/generate a cookie for a site

python,cookies,python-requests
When I go to Amazon.com in my browser, it sets a Cookie and then sends that as a header: Connection:keep-alive Cookie:x-wl-uid=1dxBAkbJJ9ys0awDahH9dLcLJmDR8XcztYvpl8p8Ojo8MNjbEo/ZBskQ8W/JSw7clLSM90d2OlFs=; session-token=/NCn9s4+YfWEyVyK0KIolKDyY2xx9L17aG/ZUZDY+5vH88wsk5n0+FcLDN9f/Y+xE/rNPKjev1iJrjSOgp26OVt5+EJynFZ6C/USFQkh3cXgzQkxXossA5Yxo9kD9S3yhpZbkZfGu1F63HNS1KQL/iIvchs3fvpMlDqmYXtiOn0H05ExWvcNdnio1ys8qU0W5LVosvUC45CuYQpi3n+7qi86AoHGjWHS/cm534xPcehsB+Xn0ndrWFNjVHSwNKR9; skin=noskin; ubid-acbjp=376-3466160-3751758;...