FAQ Database Discussion Community


How to POST - API in thingspeak.comusing urllib in Python

python,api,post,urllib
I am working on a project where I have to get data from a website and then post it to a different website. At the moment I am still new, so I am using a thingspeak.com account to experiment on posting, and I am also following the same example used...

How to convert an urlopen into a string in python

python,string,urllib
Most certainly my question is not asked properly . Anyway, I am looking for a way to convert some data I ve extracted from the web to a list or a string (string would be better tho). At the moment, this is what I wrote : import urllib as web...

Image not Found using PIL and Urllib to retrieve URL

python,python-2.7,tkinter,python-imaging-library,urllib
I keep receiving the following error no matter what image url I try to use: line 76, in <module> radar = Label(root, image = im) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk/Tkinter.py", line 2556, in __init__ Widget.__init__(self, master, 'label', cnf, kw) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk/Tkinter.py", line 2055, in __init__ (widgetName, self._w) + extra + self._options(cnf)) TclError: image...

'module' has no attribute 'urlencode'

python,python-3.x,urllib
When i try to follow python wiki page example related to URL encoding: >>> import urllib >>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) >>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params) >>> print f.read() An error is raised on the second line: Traceback (most recent call last): File "<stdin>", line 1,...

Python avoid item = None in a request of multiple items

python,function,request,urllib,nonetype
I'm doing a loop for multiple requests on a web database, each time it takes one geneId to request it to the database. If the geneId is referenced I can use the data I get for another request on a second database. But if the geneId is not referenced, it...

python3 URLError unknown url type http

python,python-3.x,urllib
I am trying to use urllib.request.urlretrieve along with the multiprocessing module to download some files and do some processing on them. However, each time I try to run my program, it gives me the error: multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.4/multiprocessing/pool.py", line 119, in worker result =...

Convert multichar %xx escapes to unicode

python,unicode,urllib
In the middle of writing this I got this to work. Here it is anyway in case it's useful or the solution is less than optimal. I have a unicode string u'http://en.wikipedia.org/wiki/Espa%C3%B1ol' from which I'd like to have u'http://en.wikipedia.org/wiki/Español'. My attempt using urllib.unquote gives me u'http://en.wikipedia.org/wiki/Espa\xc3\xb1ol'....

IOError when Using Urllib to Download Pics

python,urllib
Can anyone help me on the issue of downloading multiple files? For a while, it will stop me with IOError and told me connection attempt failed. I tried to use time.sleep function to sleep for random seconds but it doesn't help. And when I re-run the code, it starts to...

Get File Upload Time From Server

python,urllib2,urllib
Is there a way, using urllib2 or something else, to check the time a file was uploaded to a URL? Or even the time the file on the server side was last modified? At the moment I'm manually using urllib2.urlopen() to read data from a url address. The arguments for...

Login on a site using urllib

python,urllib
I'm trying to get information from this site http://cheese.formice.com/maps/@5865339 , but when i request using urllib.urlopen, its says that i need to login, i was using this code: import urllib data = { 'login':'Cfmaccount', 'password':'tfmdev321', 'submit':'Login', } url = 'http://cheese.formice.com/login' data = urllib.urlencode(data) response = urllib.urlopen(url, data) What i'm doing...

URL Readable by urllib in Python 2 but not in Python 3

python,python-3.x,urllib
I can read a specific web page in Python2 quite easily: >>> import urllib >>> urllib.urlopen("http://www.pluralsight.com/authors") <addinfourl at 4566566312 whose fp = <socket._fileobject object at 0x10fd18a50>> When I try to read the same URL using Python3, however, I get an exception: >>> from urllib.request import urlopen >>> urlopen("http://www.pluralsight.com/authors") Traceback (most...

Python urllib response returns file with added blank lines

python,xml,httprequest,response,urllib
While testing services I created a function in python that call http request on access point URL. Response is saved to the file in a given path. import urllib.request import urllib.response import urllib.parse def get_response(service_access_point, request_parameters, response_file_path): req = urllib.request.urlopen(service_access_point.format(request_parameters)) res = req.read().decode('utf-8') response_file = open(response_file_path, 'w') response_file.write(res) response_file.flush() response_file.close()...

Handling a url which fails to open, error handling using urllib

python,python-3.x,urllib
I would like some help on how to handle an url which fails to open, currently the whole program gets interrupted when it fails to open the url ( tree = ET.parse(opener.open(input_url)) )... If the opening of an url fails on my first function call (motgift) I would like it...

trying to regex in python

python,regex,python-3.x,beautifulsoup,urllib
Can anyone please help me understand this code snippet, from http://garethrees.org/2007/05/07/python-challenge/ Level2 >>> import urllib >>> def get_challenge(s): ... return urllib.urlopen('http://www.pythonchallenge.com/pc/' + s).read() ... >>> src = get_challenge('def/ocr.html') >>> import re >>> text = re.compile('<!--((?:[^-]+|-[^-]|--[^>])*)-->', re.S).findall(src)[-1] >>> counts = {} >>> for c in text: counts[c] = counts.get(c, 0) +...

Python 3: AttributeError: 'module' object has no attribute '__path__' using urllib in terminal

python,urllib
My code is runnning perfectly in PyCharm, but I have error messages while trying to open it in terminal. What's wrong with my code, or where I made mistakes? import urllib.request with urllib.request.urlopen('http://python.org/') as response: html = response.read() print(html) Output from terminal: λ python Desktop\url1.py Traceback (most recent call last):...

Python percent encoding only certain characters in a URL

python,python-3.x,urllib2,urllib,percent-encoding
I have to percent encode only # character if it appears in a given url. I know that we can encode a URL using urllib.quote. It takes a safe keyword to set a particular character to be safe for the URL. I am looking for something like: a = 'http://localhost:8001/3.0/lists/list_1.localhost.org/roster/owner#iammdkdkf'...

Python: urllib.request.urlretrieve saves an empty file. Writes in it “Supplied id parameter is empty.”

python,url,download,urllib
This is the type of file I want to download and save: http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=nuccore&id=562868704,585641505&rettype=fasta&retmode=text ...and this is the test code: import urllib.request import xml.etree.ElementTree as ET mystring = ' ' link = urllib.request.urlopen('http://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=protein&db=nuccore&linkname=protein_nuccore_mrna&id=13591999,149050462') tree = ET.parse(link) root = tree.getroot() for branch in root.iter('Link'):...

Inserting variable values into URL (or any string)

python,json,urllib
I am currently using urllib and json to perform a JSON request against a REST API as below: import urllib, json url = "http://www.cvedetails.com/json-feed.php?numrows=5&vendor_id=26&product_id=0&version_id=0&hasexp=1&opec=1&opov=1&opcsrf=1&opfileinc=1&opgpriv=0&opsqli=1&opxss=0&opdirt=0&opmemc=0&ophttprs=0&opbyp=0&opginf=0&opdos=0&orderby=0&cvssscoremin=0" response = urllib.urlopen(url) data = json.loads(response.read()) for index, entry in...

Getting errors while using Python urllib

python,urllib
I'm having trouble using Python's urllib. Here is the code I have tried: import urllib s = urllib.urlopen("https://www.mci.ir/web/guest/login") And here is the error I am seeing: Traceback (most recent call last): File "<pyshell#3>", line 1, in <module> s = urllib.urlopen("https://www.mci.ir/web/guest/login") File "C:\Python27\lib\urllib.py", line 86, in urlopen return opener.open(url) File "C:\Python27\lib\urllib.py",...

Is there a better way to retrieve webpage sizes with Python?

python,urllib2,python-requests,urllib
I'd like a sanity check on this Python script. My goal is to input a list of urls and get a byte size, giving me an indicator if the url is good or bad. import urllib2 import shutil urls = (LIST OF URLS) def getUrl(urls): for url in urls: file_name...

Why doesn't Url Decode convert + to space?

python,python-2.7,urllib
Why are the + not converted to spaces: >>> import urllib >>> url = 'Q=Who+am+I%3F' >>> urllib.unquote(url) 'Q=Who+am+I?' >>> ...

Strange Output from Python urllib2

python,html,python-2.7,urllib2,urllib
I would like to read to source code of a webpage using urllib2; however, I'm seeing a strange output that I've not seen before. Here's the code (Python 2.7, Linux): import urllib2 open_url = urllib2.urlopen("http://www.elegantthemes.com/gallery/") site_html = open_url.read() site_html[50:] Which gives the output: '\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\xe5\\ms\xdb\xb6\xb2\xfel\xcf\xe4?\xc0<S[\x9a\x8a\xa4^\xe28u,\xa5\x8e\x93\xf4\xa4\x93&\x99:9\xbdw\x9a\x8e\x07"' Does anyone know why it's showing...

python 3.4.2 urlib no attribute 'pathname2url'

python,urllib
Why is the next code not working? I cant find why. import mimetypes import glob, urllib for file in glob.glob("C:\\Users\\joey\\Desktop\\school\\ICOMMH"): url = urllib.pathname2url(file) print(file, mimetypes.guess_type(url)) The error message I get is: AttributeError: 'module' object has no attribute 'pathname2url' I am trying display all file typs of a directory. Is this...

Resize url image using urllib and pil

python,python-2.7,io,python-imaging-library,urllib
I am looking to retrieve an image from a url, resize it (to fit appropriately in the GUI), and then display the image in the GUI. I believe I have the basic premise down...or at least I know how to retrieve and display an image. I have provided some sample...

“getaddr info failed” Error When Attempting POST

python,python-2.7,ssl,urllib,httplib
I am using urllib and httplib to do an HTTPS POST to a web page, using a PEM cert file (P:\Bailey\certs\usercert.pem) and key file (P:\Bailey\certs\userkey.pem). When I ran it I got the following error: "I/O error (10109): getaddrinfo failed" Here is the Python code: from httplib import * from urllib...

Python 3 urllib produces TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str

python,python-2.7,python-3.x,urllib2,urllib
I am trying to convert working Python 2.7 code into Python 3 code and I am receiving a type error from the urllib request module. I used the inbuilt 2to3 Python tool to convert the below working urllib and urllib2 Python 2.7 code: import urllib2 import urllib url = "https://www.customdomain.com"...

urllib keeps freezing while trying to pull HTML data from a website - is my code correct?

python,osx,parsing,urllib,python-3.4
I'm trying to build a simple Python script algorithm on Mac OS X that has four parts to it. go to a defined website and grab all the HTML using urllib parse the HTML data to find a table of numbers (using beautifulsoup) with those numbers do a simple calculation...

How do I make a BasicAuth REST query over SSL with python3

python-3.x,afnetworking,urllib
I have a tornado server that acts as a REST interface to an iPhone app. I want to do some queries against said server using a python program. On the iOS side, I'm using the AFNetworking library. But I don't know what the analogs are on the python client side....

Python urllib requesting multiple times

python,urllib
I've written the following python code which connects to a URL to get data using json. However, when my server logs these requests, they are being made twice each time. I am assuming it has something to do with the fact that it's first using try and then re-requesting the...

python save image from url

python,image,download,urllib
I got a problem when I am using python to save an image from url either by urllib2 request or urllib.urlretrieve. That is the url of the image is valid. I could download it manually using the explorer. However, when I use python to download the image, the file cannot...

urllib.request.urlopen(url) how to use this function with ip address?

network-programming,urllib2,urllib,urlopen,urllib3
I'm working on Python3 with testing page load times so I created a local apache server for compare but the problem is I use urllib.request.urlopen(url) function which doesn't allow me to use my own ip address. Is there anything that helps me to get page with only ip address. Here's...

Python scp copy images from image_urls to server

python,python-imaging-library,urllib,scp
I have written one function which recieves a url and copy it to all server. Server remote path is stored in db. def copy_image_to_server(image_url): server_list = ServerData.objects.values_list('remote_path', flat=True).filter(active=1) file = cStringIO.StringIO(urllib.urlopen(image_url).read()) image_file = Image.open(file) image_file.seek(0) for remote_path in server_list: os.system("scp -i ~/.ssh/haptik %s %s " % (image_file, remote_path)) I am...

Proper way to fix a url without http://

python,url,urllib2,urllib
I'm trying to open a list of urls of this format, using in Urllib2: google.com facebook.com youtube.com yahoo.com baidu.com Using this method: urllib2.urlopen(url): And getting this error: File "fetcher.py", line 98, in fetch_urls_and_save response = urllib2.urlopen(url) File "urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "urllib2.py", line 423,...

using python urllib and beautiful soup to extract information from html site

python,beautifulsoup,urllib
I am trying to extract some information from this website i.e. the line which says: Scale(Virgo + GA + Shapley): 29 pc/arcsec = 0.029 kpc/arcsec = 1.72 kpc/arcmin = 0.10 Mpc/degree but everything after the : is variable depending on galtype. I have written a code which used beautifulsoup and...

Openstack token, I can get token with curl but can't with urllib

curl,urllib,openstack
[email protected]_PC:~$ curl -D- -H 'X-Storage-User:Lucy' [email protected]_PC:~$ curl -D- -H 'X-Storage-User:Lucy' http://192.168.119.89:8080/auth/v1.0 HTTP/1.1 200 OK X-Storage-Url: http://192.168.119.89:8080/v1/AUTH_Lucy X-Auth-Token: AUTH_tk713d067336d24348bcea1ab220965785 Content-Type: text/html; charset=UTF-8 X-Storage-Token: AUTH_tk713d067336d24348bcea1ab220965785 X-Trans-Id: tx8a03bc278f6d4ab899a25-00557554df Content-Length: 0 Date: Mon, 08 Jun 2015 08:39:59 GMT I can get the X-Storage-token with curl. import urllib,httplib,json from urlparse...

xlrd throws TypeError: embedded NUL character when trying to open an `.xls` file from web in Python 3.4

python,python-3.x,typeerror,urllib,xlrd
I am trying to open an excel file from the web and extract one of the columns. However, when I try to open the file with xlrd I get an error. The code I am trying is: from urllib.request import urlopen import xlrd DJIA_URL = 'http://www.djaverages.com/?go=export-components&symbol=DJI' xlfile = urlopen(DJIA_URL).read() xlbook...

Python 3.4 urllib.request error (http 403)

python,python-3.x,urllib
I'm trying to open and parse a html page. In python 2.7.8 I have no problem: import urllib url = "https://ipdb.at/ip/66.196.116.112" html = urllib.urlopen(url).read() and everything is fine. However I want to move to python 3.4 and there I get HTTP error 403 (Forbidden). My code: import urllib.request html =...

download files from the internet using urllib issue

python,urllib
I am trying to download data files from a website using urllib. My code is import urllib url_common = 'http://apps.waterconnect.sa.gov.au/SiteInfo/Data/Site_Data/' site_list=['4260514','4260512','4260519'] parameter_list=['ecrec','ecday','flowrec','flowcday'] for site in site_list: for parameter in parameter_list: try: url = url_common+'A'+site+'/'+'a'+site+'_'+parameter+'.zip' urllib.urlretrieve(url,'A'+site+'_'+parameter+'.zip') except ValueError: break My issue is some sites do not have all the parameter files....

Python 3 urllib.request byte indicator b' causing issues

python-3.x,urllib
I have a variable storing random characters which may be special chars that came from decoding base64: variable = qb'l*,707$}p*yk And i append that onto the end of a URL I already know to navigate a specific page www.website.com/1/2/3/index.php?addafterthis= So I try to open like this: opener = urllib.request.build_opener() opener.open('www.website.com/1/2/3/index.php?addafterthis=',str.encode(string))...

python 3 - urllib exception http.client.BadStatusLine

exception,python-3.x,urllib
I cant for the life of me figure out why I can't catch this exception. Looking here as a guide https://docs.python.org/3/howto/urllib2.html def get_team_names(get_team_id_url, team_id): print(get_team_id_url + team_id) try: response = urllib.request.urlopen(get_team_id_url + team_id) except urllib.error.HTTPError as e: print(e.code) print(e.read()) except urllib.error.URLError as e: print(e.code) print(e.read()) exception: Traceback (most recent call...

Python XML parsing, lxml, urllib.request

python,xml,lxml,urllib
I am a little bit stuck trying to parse a XML file retrieved from url, my goal is to get this xml file into a well structured object to easily retrieve its data. My current code results in the following error: >>> tree = etree.parse(data) Traceback (most recent call last):...

Is there way to ignore 302 Moved Temporarily redirection or find what it is caused by?

python,redirect,python-3.x,urllib
I am Writing some parsing script and need to access to many web pages like this one. Whenever I try to get this page with urlopen and then read(), I get redirected to this page. When I launch the same links from google chrome redirect happens but really rarely, most...

URLOpen Error while combining url with word from wordlist

python,urllib,host
Hey guys im making a Python Webcrawler at the Moment. So i have a link, which last chars are: "search?q=" and after that im using my wordlist which i have loaded before into a list. But when i try to open that with : urllib2.urlopen(url) it throws me an Error...

Python - bypass Malwarebytes blocked websites with urllib?

python,block,urllib
Here's a part my code that gets source code from websites: #Header dictionary headers = {} #Header for websites headers['User-Agent'] = "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17" #Urllib code req = urllib.request.Request(url, headers=headers) resp = urllib.request.urlopen(req) respData = resp.read() This works, however my Malwarebytes blocks some...