python-2.7,selenium,web-crawler , “TypeError: 'Rule' object is not iterable” webscraping an .aspx page in python


“TypeError: 'Rule' object is not iterable” webscraping an .aspx page in python

Question:

Tag: python-2.7,selenium,web-crawler

I am using the following code to scrape this website (http://profiles.ehs.state.ma.us/Profiles/Pages/ChooseAPhysician.aspx?Page=1) ; however, obtain the following TypeError:

"File "C:\Users\Anaconda2\lib\site-packages\scrapy\contrib\spiders\crawl.py", line 83, in _compile_rules self._rules = [copy.copy(r) for r in self.rules] TypeError: 'Rule' object is not iterable"

I don't have any code written on line 83, thus, wondering if anyone has ideas on how to resolve the issue? I'm using Python 2.7 in Windows.

Thanks!

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import scrapy
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.selector import HtmlXPathSelector


class MdiMassSpider(CrawlSpider):
    name = "MdiMass"
    allowed_domains = ["http://profiles.ehs.state.ma.us/Profiles/Pages/FindAPhysician.aspx"]
    start_urls = ["http://profiles.ehs.state.ma.us/Profiles/Pages/ChooseAPhysician.aspx?Page=1"]
    driver = webdriver.Chrome()

    rules = (Rule(LinkExtractor(allow=(".*http://profiles.ehs.state.ma.us/Profiles/Pages/PhysicianProfile.aspx?PhysicianID=.*,"))),)
    # all pages to scrape follow the same: http://profiles.ehs.state.ma.us/Profiles/Pages/PhysicianProfile.aspx?PhysicianID=1311
    #PhysicianID=XXXX

    def __init__(self):
        CrawlSpider.__init__(self)
        self.driver = webdriver.Chrome()

    #def __del__(self):
    #   self.selenium.quit()
    #   print self.verificationErrors

    def parse(self, response):
        self.driver.get('http://profiles.ehs.state.ma.us/Profiles/Pages/ChooseAPhysician.aspx?Page=1')


    def parse(self):
        select = Select(driver.find_element_by_xpath("//select[@id=\"ctl00_ContentPlaceHolder1_cmbDistance\"]"))
        print select.options
        print [o.text for o in select.options]
        select.select_by_visible_text("15")

        zip = driver.find_element_by_xpath("//*[@id=\"ctl00_ContentPlaceHolder1_txtZip\"]")
        zip.send_keys("02109")

        prim_care_chekbox = driver.find_element_by_xpath("//*[@id=\"ctl00_ContentPlaceHolder1_SpecialtyGroupsCheckbox_6\"]")
        prim_care_chekbox.click()

        find_phy_button = driver.find_element_by_xpath("//*[@id=\"ctl00_ContentPlaceHolder1_btnSearch\"]")
        find_phy_button.click()

        for sel in response.xpath("//*[@id=\"PhysicianSearchResultGrid\"]/tbody/tr[2]/td[1]/a"):
            item = MdiMassItem()
            item["phy_name"] = sel.xpaths("//*[@id=\"content\"]/center/p[1]").extract() 
            item["lic_status"] = driver.find_elements_by_xpaths("//*[@id=\"content\"]/center/table[2]/tbody/tr[3]/td/table/tbody/tr/td[1]/table/tbody/tr[2]/td[2]/a[1]").extract()
            item["lic_issue_date"] = driver.find.elements_by_xpaths("//*[@id=\"content\"]/center/table[2]/tbody/tr[3]/td/table/tbody/tr/td[1]/table/tbody/tr[3]/td[2]").extract()
            item["prim_worksetting"] = driver.find.elements_by_xpaths("//*[@id=\"content\"]/center/table[2]/tbody/tr[3]/td/table/tbody/tr/td[1]/table/tbody/tr[5]/td[2]").extract()
            item["npi"] = driver.find_elements_by_xpaths("//*[@id=\"content\"]/center/table[2]/tbody/tr[3]/td/table/tbody/tr/td[2]/table/tbody/tr[6]/td[2]").extract()
            item["Med_sch_grad_date"] = driver.find_elements_by_xpaths("//*[@id=\"content\"]/center/table[3]/tbody/tr[3]/td/table/tbody/tr[2]/td[2]").extract()
            item["Area_of_speciality"] = driver.find_elements_by_xpaths("//*[@id=\"content\"]/center/table[4]/tbody/tr[3]/td/table/tbody/tr/td[2]").extract()
            item["link"] =  driver.find_element_by_xpath("//*[@id=\"PhysicianSearchResultGrid\"]/tbody/tr[2]/td[1]/a").extract()
            print item

Edited Code:

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import scrapy
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.selector import HtmlXPathSelector


class MdiMassSpider(CrawlSpider):
    name = "MdiMass"
    allowed_domains = ["http://profiles.ehs.state.ma.us/Profiles/Pages/FindAPhysician.aspx"]
    start_urls = ["http://profiles.ehs.state.ma.us/Profiles/Pages/ChooseAPhysician.aspx?Page=1"]

    rules = (Rule(LinkExtractor(allow=(".*http://profiles.ehs.state.ma.us/Profiles/Pages/PhysicianProfile.aspx?PhysicianID=.*"))),)
    # all pages to scrape follow the same: http://profiles.ehs.state.ma.us/Profiles/Pages/PhysicianProfile.aspx?PhysicianID=1311
    #PhysicianID=XXXX

    def __init__(self):
        CrawlSpider.__init__(self)
        self.driver = webdriver.Chrome()
        self.driver.get('http://profiles.ehs.state.ma.us/Profiles/Pages/ChooseAPhysician.aspx?Page=1')

    def parse(self, response):
        driver = self.driver
        select = Select(self.driver.find_element_by_xpath("//select[@id=\"ctl00_ContentPlaceHolder1_cmbDistance\"]"))
        print select.options
        print [o.text for o in select.options]
        select.select_by_visible_text("15")

        zip = self.driver.find_element_by_xpath("//*[@id=\"ctl00_ContentPlaceHolder1_txtZip\"]")
        zip.send_keys("02109")

        prim_care_chekbox = self.driver.find_element_by_xpath("//*[@id=\"ctl00_ContentPlaceHolder1_SpecialtyGroupsCheckbox_6\"]")
        prim_care_chekbox.click()

        find_phy_button = self.driver.find_element_by_xpath("//*[@id=\"ctl00_ContentPlaceHolder1_btnSearch\"]")
        find_phy_button.click()

        for sel in response.xpath("//*[@id=\"PhysicianSearchResultGrid\"]/tbody/tr[2]/td[1]/a"):
            item = MdiMassItem()
            item["phy_name"] = sel.xpaths("//*[@id=\"content\"]/center/p[1]").extract() 
            item["lic_status"] = driver.find_elements_by_xpaths("//*[@id=\"content\"]/center/table[2]/tbody/tr[3]/td/table/tbody/tr/td[1]/table/tbody/tr[2]/td[2]/a[1]").extract()
            item["lic_issue_date"] = driver.find.elements_by_xpaths("//*[@id=\"content\"]/center/table[2]/tbody/tr[3]/td/table/tbody/tr/td[1]/table/tbody/tr[3]/td[2]").extract()
            item["prim_worksetting"] = driver.find.elements_by_xpaths("//*[@id=\"content\"]/center/table[2]/tbody/tr[3]/td/table/tbody/tr/td[1]/table/tbody/tr[5]/td[2]").extract()
            item["npi"] = driver.find_elements_by_xpaths("//*[@id=\"content\"]/center/table[2]/tbody/tr[3]/td/table/tbody/tr/td[2]/table/tbody/tr[6]/td[2]").extract()
            item["Med_sch_grad_date"] = driver.find_elements_by_xpaths("//*[@id=\"content\"]/center/table[3]/tbody/tr[3]/td/table/tbody/tr[2]/td[2]").extract()
            item["Area_of_speciality"] = driver.find_elements_by_xpaths("//*[@id=\"content\"]/center/table[4]/tbody/tr[3]/td/table/tbody/tr/td[2]").extract()
            item["link"] =  driver.find_element_by_xpath("//*[@id=\"PhysicianSearchResultGrid\"]/tbody/tr[2]/td[1]/a").extract()
            print item

Python Log Message:

C:\Users\Anaconda2\MdiMass>scrapy crawl MdiMass -o items.csv
2015-02-26 01:11:47-0500 [scrapy] INFO: Scrapy 0.24.4 started (bot: MdiMass)
2015-02-26 01:11:47-0500 [scrapy] INFO: Optional features available: ssl, http11, boto
2015-02-26 01:11:47-0500 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'MdiMass.spiders', 'FEED_FORMAT': 'csv', 'SPIDER_M
ODULES': ['MdiMass.spiders'], 'FEED_URI': 'items.csv', 'BOT_NAME': 'MdiMass'}
2015-02-26 01:11:47-0500 [scrapy] INFO: Enabled extensions: FeedExporter, LogStats, TelnetConsole, CloseSpider, WebService, CoreStat
s, SpiderState
2015-02-26 01:11:49-0500 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMidd
leware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMidd
leware, ChunkedTransferMiddleware, DownloaderStats
2015-02-26 01:11:49-0500 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLe
ngthMiddleware, DepthMiddleware
C:\Users\Anaconda2\MdiMass
2015-02-26 01:11:49-0500 [scrapy] INFO: Enabled item pipelines: CsvWriterPipeline
2015-02-26 01:11:49-0500 [MdiMass] INFO: Spider opened
2015-02-26 01:11:49-0500 [MdiMass] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-02-26 01:11:49-0500 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-02-26 01:11:49-0500 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
2015-02-26 01:11:49-0500 [MdiMass] DEBUG: Redirecting (302) to <GET http://profiles.ehs.state.ma.us/Profiles/Pages/FindAPhysician.as
px> from <GET http://profiles.ehs.state.ma.us/Profiles/Pages/ChooseAPhysician.aspx?Page=1>
2015-02-26 01:11:49-0500 [MdiMass] DEBUG: Crawled (200) <GET http://profiles.ehs.state.ma.us/Profiles/Pages/FindAPhysician.aspx> (re
ferer: None)
[<selenium.webdriver.remote.webelement.WebElement object at 0x0493B210>, <selenium.webdriver.remote.webelement.WebElement object at
0x0493B0B0>, <selenium.webdriver.remote.webelement.WebElement object at 0x0493B1F0>, <selenium.webdriver.remote.webelement.WebElemen
t object at 0x0493B110>, <selenium.webdriver.remote.webelement.WebElement object at 0x0493B150>, <selenium.webdriver.remote.webeleme
nt.WebElement object at 0x0493B170>]
[u'', u'5', u'10', u'15', u'30', u'50']
2015-02-26 01:11:50-0500 [MdiMass] INFO: Closing spider (finished)
2015-02-26 01:11:50-0500 [MdiMass] INFO: Dumping Scrapy stats:
        {'downloader/request_bytes': 575,
         'downloader/request_count': 2,
         'downloader/request_method_count/GET': 2,
         'downloader/response_bytes': 19312,
         'downloader/response_count': 2,
         'downloader/response_status_count/200': 1,
         'downloader/response_status_count/302': 1,
         'finish_reason': 'finished',
         'finish_time': datetime.datetime(2015, 2, 26, 6, 11, 50, 150000),
         'log_count/DEBUG': 4,
         'log_count/INFO': 7,
         'response_received_count': 1,
         'scheduler/dequeued': 2,
         'scheduler/dequeued/memory': 2,
         'scheduler/enqueued': 2,
         'scheduler/enqueued/memory': 2,
         'start_time': datetime.datetime(2015, 2, 26, 6, 11, 49, 134000)}
2015-02-26 01:11:50-0500 [MdiMass] INFO: Spider closed (finished)

Answer:

The problem is in the following line.

rules = (Rule(LinkExtractor (allow=(" 
.*http://profiles.ehs.state.ma.us/Profiles/Pages/PhysicianProfile.aspx?PhysicianID=.*,"
))))

You miss positioned a comma. The correct code is:

rules = (Rule(LinkExtractor(allow=("
.*http://profiles.ehs.state.ma.us/Profiles/Pages/PhysicianProfile.aspx?PhysicianID=.*"
))),)

By this correction you make the rule iterable.

Good definition of iterators here: (Build a Basic Python Iterator)

Iterator objects in python conform to the iterator protocol, which basically means they provide two methods: __iter__() and next().

The __iter__ returns the iterator object and is implicitly called at the start of loops.

The next() method returns the next value and is implicitly called at each loop increment.
next() raises a StopIteration exception when there are no more value to return, which is implicitly captured by looping constructs to stop iterating.


Related:


Java Selenium - Can't seem to select an element


java,selenium,xpath,hidden
I'm trying to select an e-mail href to grab the text, but no matter what selections I use to try to select the e-mail, my selection doesn't seem to work and I am curious what other ways might exist to fix the problem. I have used absolute and relative xpath,...

Find element by class name


python,parsing,selenium,selenium-webdriver,css-selectors
I'm trying to find one tag using we.find_element_by_css_selector('p.p1.transfer strong.ng-binding').text The problem is that there is sometimes a 'strong' tag before the tag I'm searching for which is very similar but it's class is: class="ng-binding ng-hide" instead of class="ng-binding". But when I try to find it it finds the first tag....

How to skip a function


function,python-2.7
I am learning python and trying to make a little game. So my question is can you define a function but skip it and use it later. EX. def func() print"1,2,3,4" func() def func2() print "counting" func() func2() How would I skip func but still be able to print it...

Upload file - Protractor


javascript,selenium,testing,protractor,end-to-end
I fight with protractor because for some tests I need to UPLOAD file. My HTML looks like: <div class="panel-footer"> <ul class="list-unstyled"> <!-- ngRepeat: file in imagesToUpload --> </ul> <button class="btn btn-sm btn-success pull-right ng-binding ng-hide" ng-show="imagesToUpload.length" ng-click="uploadImages()">Nahrát na server</button> <button class="btn btn-sm btn-primary ng-binding" ng-file-select="onImageSelect($files)" data-multiple="true" style="overflow: hidden;">Vybrat soubory<input...

Checking element existence on the page with WebDriver findElements().isEmpty method


java,selenium,selenium-webdriver
I wish to check if some element exists on the page or not. I already saw this WebDriver: check if an element exists? issue but I'm wondering why not simply apply findElements().isEmpty method? I thought it would do the same work. UPD Now I see findElements().isEmpty works perfect so I'm...

How do I copy a row from one pandas dataframe to another pandas dataframe?


python,python-2.7,pandas,dataframes
I have a dataframe of data that I am trying to append to another dataframe. I have tried various ways with .append() and there has been no successful way. When I print the data from iterrows. I provide 2 possible ways I tried to solve the issue below, one creates...

webdriver C# - click this element with XPath position?


c#,selenium,xpath,webdriver
I have an element whit this XPath Position: //td[4]/a I try this but does not work: driver.FindElement(By.XPath(".//*[@position='//td[4]/a']")).Click(); Using C# Webdriver and not java, please....

Count function counting only last line of my list


python,python-2.7
Count function counting only last line of my list N = int(raw_input()) cnt = [] for i in range(N): string = raw_input() for j in range(1,len(string)): if string[j] =='K': cnt.append('R') elif string[j] =='R': cnt.append('R') if string[0] == 'k': cnt.append('k') elif string[0] == 'R': cnt.append('R') print cnt.count('R') if I am giving...

I need to make sure that only certain characters are in a list?


python,regex,string,list,python-2.7
I have this to get input and put it in a list: def start(): move_order=[raw_input("Enter your moves: ").split()] And I only want the characters A, D, S, C, H (it's for a game >_>) to be allowed. I've tried using the regular expressions stuff: if re.match('[ADSCH]+', [move_order]) is False: print...

Junit parameterized testing ,a different thought


java,selenium,junit
Here, I have a custom independent component which accept input as a jar file and identify the junit test classes from it and execute them using the code given below. Result result = JUnitCore.runClasses(cls);//cls is Class<?>instance. Next thing I required is a provision to inject parameters (Assuming that input jar...

How to write xpath in selenium webdriver for below HTML expressions?


selenium
I wrote xpath for below HTML code i.e. displayed below 1. //a[@text()='Life Insurance'] 2. //span[@text()='Apply now'] But I got element not found exception. If I used Absolute xpath processor then It's working and I wrote own xpath then it thrown exception. Please tell me how to write it. Below are...

Strange Behavior: Floating Point Error after Appending to List


python,python-2.7,behavior
I am writing a simple function to step through a range with floating step size. To keep the output neat, I wrote a function, correct, that corrects the floating point error that is common after an arithmetic operation. That is to say: correct(0.3999999999) outputs 0.4, correct(0.1000000001) outputs 0.1, etc. Here's...

Python code not executing in order? MySQLdb UPDATE commits in unexpected order


python,python-2.7,threadpool,mysql-python
I've got a Python 2.7 script I'm working on that retrieves rows from a MySQL table, loops through the data to process it, then is supposed to do the following things in this order: UPDATE the table rows we just got previously to set a locked value in each row...

List of tuples from (a, all b) to (b, all a)


python,list,python-2.7,tuples
I am starting with a list of tuples (a,all b). I want to end with a list of tuples (b,all a). For example: FROM (a1,[b1,b2,b3]) (a2,[b2]) (a3,[b1,b2]) TO (b1,[a1,a3]) (b2[a1,a2,a3]) (b3,[a1] How do I do this using Python 2? Thank you for your help....

Keep strings that occur N times or more


python,python-2.7,counter
I have a list that is mylist = ['a', 'a', 'a', 'b', 'b', 'c', 'c', 'd'] And I used Counter from collections on this list to get the result: from collection import Counter counts = Counter(mylist) #Counter({'a': 3, 'c': 2, 'b': 2, 'd': 1}) Now I want to subset this...

multiple iteration of the same list


python,python-2.7,python-3.x,numpy,shapely
I have one list of data as follows: from shapely.geometry import box data = [box(1,2,3,4), box(4,5,6,7), box(1,2,3,4)] sublists = [A,B,C] The list 'data' has following sub-lists: A = box(1,2,3,4) B = box(4,5,6,7) C = box(1,2,3,4) I have to check if sub-lists intersect. If intersect they should put in one tuple;...

How to extract efficientely content from an xml with python?


python,xml,python-2.7,pandas,lxml
I have the following xml: <?xml version="1.0" encoding="UTF-8" standalone="no"?><author id="user23"> <document><![CDATA["@username: That boner came at the wrong time ???? http://t.co/5X34233gDyCaCjR" HELP I'M DYING ]]></document> <document><![CDATA[Ugh ]]></document> <document><![CDATA[YES !!!! WE GO FOR IT. http://t.co/fiI23324E83b0Rt ]]></document> <document><![CDATA[@username Shout out to me???? ]]></document> </author> What is the most efficient...

Parse text from a .txt file using csv module


python,python-2.7,parsing,csv
I have an email that comes in everyday and the format of the email is always the same except some of the data is different. I wrote a VBA Macro that exports the email to a text file. Now that it is a text file I want to parse the...

Import on class instanciation


python,python-2.7
I'm creating a module with several classes in it. My problem is that some of these classes need to import very specific modules that needs to be manually compiled or need specific hardware to work. There is no interest in importing every specific module up front, and as some modules...

Parsing Google Custom Search API for Elasticsearch Documents


json,python-2.7,elasticsearch,google-search-api
After retrieving results from the Google Custom Search API and writing it to JSON, I want to parse that JSON to make valid Elasticsearch documents. You can configure a parent - child relationship for nested results. However, this relationship seems to not be inferred by the data structure itself. I've...

Use NamedTemporaryFile to read from stdout via subprocess on Linux


python-2.7,subprocess,temporary-files
import subprocess import tempfile fd = tempfile.NamedTemporaryFile() print(fd) print(fd.name) p = subprocess.Popen("date", stdout=fd).communicate() print(p[0]) fd.close() This returns: <open file '<fdopen>', mode 'w' at 0x7fc27eb1e810> /tmp/tmp8kX9C1 None Instead, I would like it to return something like: Tue Jun 23 10:23:15 CEST 2015 I tried adding mode="w", as well as delete=False, but...

Identify that a string could be a datetime object


python,regex,algorithm,python-2.7,datetime
If I knew the format in which a string represents date-time information, then I can easily use datetime.datetime.strptime(s, fmt). However, without knowing the format of the string beforehand, would it be possible to determine whether a given string contains something that could be parsed as a datetime object with the...

WebDriver can't get dropdown menu element (Java)


java,selenium,webdriver,junit4
I write a script on Java for Selenium WebDriver, and I have a problem with selected from dropdown menu. Here's my locator: new Select(driver.findElement(By.id("FormElement_select_68_input_input"))).selectByVisibleText("Image"); Here's an error: http://prntscr.com/7jul03 Here's HTML code: http://prntscr.com/7jvou6 Need to select "Image" from this menu, but have an error. Before I had the error like this,...

Selenium catch popup on close browser


java,selenium,browser
I'm trying to test that when I close my window a popup shows with a warning message. I've tried both driver.close() and driver.quit() after making sure I'm on the proper window but this just terminates the process since my popup doesn't show. I could test it by using the awt...

CSS Selector-How to locate Parent Element


selenium,selenium-webdriver,css-selectors,webdriver
Is there any way to locate parent element in CSS Selector? i am using below code but i am not getting the parent element. WebElement we=dr.findElement(By.cssSelector("div[id='gf-BIG']:parent")); I know there is a way in Xapth but please let me know that how we can locate parent in CSS Selector. ...

lookbehind for start of string or a character


regex,python-2.7,regex-lookarounds
The command re.compile(ur"(?<=,| |^)(?:next to|near|beside|opp).+?(?=,|$)", re.IGNORECASE) throws a sre_constants.error: look-behind requires fixed-width pattern error in my program but regex101 shows it to be fine. What I'm trying to do here is to match landmarks from addresses (each address is in a separate string) like: "Opp foobar, foocity" --> Must match...

Is there a way to say for every x values, do this?


python,python-2.7
Before I start, I am new to Python, so any low-level description would be incredibly helpful! I have a list of lets say 60 values (representing one hour, from 8:00-9:00) and I want to run average, maximums, minimums, and standard deviation for each set of 15. (I already have the...

close and open browser -selenium C# Visual studio


c#,selenium,browser,close
I want to close and then open the browser without close the driver of selenium, using C# I don't want to: driver.Quit(); How Can I close and the open again the browser Chrome?...

Pass function call as a function argument


python,python-2.7,python-3.x
Code: def function1(a,b): return a-1,b-1 def function2(c,d): return c+1,d+1 print function1(function2(1,2)) Error: Traceback (most recent call last): File "C:\Users\sony\Desktop\Python\scripts\twitter_get_data.py", line 6, in <module> print function1(function2(1,2)) TypeError: function1() takes exactly 2 arguments (1 given) [Finished in 0.1s with exit code 1] Why the above error? ...

What is the difference between <> and == in python?


python-2.7
In [142]: (MON,TUE,WED,THR,FRI,SAT,SUN)<>range(7) Out[142]: True In [143]: (MON,TUE,WED,THR,FRI,SAT,SUN)==range(7) Out[143]: False ...

Why does `for lst in lst:` work? [duplicate]


python,python-2.7
This question already has an answer here: Why can I use the same name for iterator and sequence in a Python for loop? 6 answers The following code lst = ['foo', 'bar', 'baz'] for lst in lst: print lst gives me this output foo bar baz I would expect...

Anaconda site-packages


python,python-2.7,pip,anaconda
After installing a package in an anaconda environment, I'll like to make some changes to the code in that package. Where can I find the site-packages directory containing the installed packages? I do not find a directory /Users/username/anaconda/lib/python2.7/site-packages...

How to Find class inside in
, using Selenium WebDriver


selenium,webdriver
here is the code: <div class="padding-tlb"> ....Some code.... </div> <article class="non-article-login"> <div class="one-fourth-percent columns-percent">&nbsp;</div> <div class="one-half-percent columns-percent"> <div class="one-fourth-percent columns-percent"> User ID </div> </div> </div> </article> I want to check if exist the class "one-fourth-percent columns-percent". WebElement test = driver.findElement(By.className("one-fourth-percent columns-percent")); But...

how to fetch a column in browse_record_list in orm browse method in openERP


python-2.7,orm,openerp-7
I'm beginner in openERP. I'm trying to get a column in a table. While using ORM browse method and iterating that object i got the result in browse_record_list as browse_record(table.name,21). I want to fetch that particular id 21 alone through that browse method but instead im getting same browse_record as...

How to remove structure with python from this case?
python,python-2.7
How to remove "table" from HTML using python? I had case like this: paragraph = ''' <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit. Quidem molestiae consequuntur officiis corporis sint.<br /><br /> <table> <tr> <td> text title </td> <td> text title 2 </td> </tr> </table> <p> lorem ipsum</p> ''' how...

Syntax Error (FROM) in Python, I do not want to use it as function but rather use it as to print something


python,python-2.7
I am trying to print out usernames from Instagram. When I type in print i.from.username, there will be syntax error because Python thinks that I am using from function, which i actually not. for i in a: print i.from.username Is there anyway to troubleshoot it? I tried using making a...

python - how to properly evaluate a value from a system command


python,python-2.7
I'm just learning python (using 2.7.8) and i'm trying to figure out what is the best way to evaluate the output of a system command. I've read to use subprocess. For example, I need to run this IF statment and evaluate for anything > 0, then process it. Example of...

Python Popen - wait vs communicate vs CalledProcessError


python,python-2.7,error-handling,popen
Continuing from my previous question I see that to get the error code of a process I spawned via Popen in python I have to call either wait() or communicate() (which can be used to access the Popen stdout and stderr attributes): app7z = '/path/to/7z.exe' command = [app7z, 'a', dstFile.temp,...

Pandas Dataframe Complex Calculation


python,python-2.7,pandas,dataframes
I have the following dataframe,df: Year totalPubs ActualCitations 0 1994 71 191.002034 1 1995 77 2763.911781 2 1996 69 2022.374474 3 1997 78 3393.094951 I want to write code that would do the following: Citations of currentyear / Sum of totalPubs of the two previous years I want something to...

How can I resolve my variable's unexpected output?


django,python-2.7
I have a variable in django named optional_message. If I debug the variable then it says Swenskt but when I try to print the variable on my page the following comes out: (u'Swenskt',) and the variable can't be tested for its length etc. What should I do if I only...

Slicing a Python OrderedDict


python-2.7,slice,ordereddictionary
In my code I frequently need to take a subset range of keys+values from a Python OrderedDict (from collections package). Slicing doesn't work (throws TypeError: unhashable type) and the alternative, iterating, is cumbersome: from collections import OrderedDict o = OrderedDict([('a', 1), ('b', 2), ('c', 3), ('d', 4)]) # want to...

from x(defined in program) import y(defined in program) python


python,python-2.7,import,filenames
I need some assitance since I really have no idea how I can fix this: x="test" y="test2" When I try to import y from x , it says that there is no file with the name "x" (from x import y) Is there any way to import test2 from test...

How to check for multiple attributes in a list


python,python-2.7
I am making a TBRPG game using Python 2.7, and i'm currently making a quest system. I wanted to make a function that checks all of the quests in a list, in this case (quests), and tells you if any of of the quests in the list have the same...

Unable to select text boxes in selenium webdriver through XPath?


selenium,xpath
My HTML <table border="2" style="background:gray"> <tbody> <td class="std"> <input id="ActQ1Revenue" type="text" name="amount" disabled="disabled" maxlength="20" style="background:wheat"> </td> </tr> <tr> <td class="atd"> <input id="ActAprRevenue" type="text" name="amount" maxlength="20"> </td> </tr> <tr> <td class="atd"> <input id="ActMayRevenue" type="text" name="amount" maxlength="20"> </td> </tbody> </table> My XPath...

Python split by comma delimiter and strip


python,python-2.7
I am trying to open a file in python in read mode then write striped and split data to an output file. I am unsure how to split and strip the same data. Do I need to create a different line? Do I need to write the data out first?...

Python initialize strings as variable


python,python-2.7
I have a calculation schema as string calc = "((k+m+46)/2)" and some strings containing variable like m = 2 k = m*2 all only strings. Now I want to initialize them into Python. my goal is it to calculate with the calculating schema the varible values. calc should return 26...

XPath for child element


selenium,xpath
What is the Xpath to select the first element of this jquery selector: $('.A .B:eq(1)') ? HTML Sample <div class="x A z"> <div class="y"> <div class="r B z"></div> <---- that is that I need to select <div class="r B z"></div > ... So far I've tried this: (//div[(@class='A') and div[contains(concat('...

Stopping list selection in Python 2.7


python,list,python-2.7
Imagine that I have an order list of tuples: s = [(0,-1), (1,0), (2,-1), (3,0), (4,0), (5,-1), (6,0), (7,-1)] Given a parameter X, I want to select all the tuples that have a first element equal or greater than X up to but not including the first tuple that has...

Webdriver C# -Click second linktext not first


c#,selenium,webdriver
By this line I click on the first text, but how can I click the second exact text? driver.FindElement(By.PartialLinkText("Play")).Click(); ...

PySerial client unable to write data


python,python-2.7,pyserial
I'm trying to write a python program which can communicate over a serial interface using PySerial module as follows: import serial if __name__ == '__main__': port = "/dev/tnt0" ser = serial.Serial(port, 38400) print ser.name print ser.isOpen() x = ser.write('hello') ser.close() print "Done!" But if I execute the above I get...