selenium,web-crawler,mechanize , Selenium interpret javascript on mac?


Selenium interpret javascript on mac?

Question:

Tag: selenium,web-crawler,mechanize

I'm trying to make a web crawler that click on ads (yes, i know), it's very sophisticated, but, I realise that Google Ads aren't showed when javascript is disabled. Today, i use Mechanize, and it doesn't "accept" javasript. I heard selenium use another system to crawl the net. The only thing I want to do is access my page, and click on the ad (generated by javascript). Can Selenium do it ?


Answer:

Selenium is a browser automation tool. You can basically automate everything you can do in your browser. Start with going through the Getting Started section of the documentation.

Example:

from selenium import webdriver

driver = webdriver.Firefox()
driver.get("http://www.python.org")

print driver.title

driver.close()

Besides automating common browsers, like Chrome, Firefox, Safari or Internet Explorer, you can also use PhantomJS headless browser.


Related:


How do I tell Selenium that an Angular controller has “loaded”


javascript,angularjs,selenium
I've got some UI tests that are attempting to test that a click on an element makes something else appear. There's an existing check for all tests that looks to see if the DOM is Ready, however there's a small amount of time between that even firing and the app.controller()...

Execute javascript within webpage with selenium WebDriver


java,javascript,html,selenium,webdriver
I am trying to execute a javascript method on a webpage with the selenium WebDriver. I am able to execute my own script on the page, but not call the method from the page's HTML. Here is the HTML function Search(){ if(check(document.forms[0],"searchResults")){ var fieldCode = ""; var fieldText = "";...

Value attribute of a text box retains old value even after new value has been keyed in


java,selenium,selenium-webdriver,webdriver
This is the HTML I have for a text box on a form: <input type="text" onblur="javascript:itemLostFocus('18A3461989AC0244E050A20AB17C4671');" onkeydown="javascript:itemChanging('18A3461989AC0244E050A20AB17C4671');" value="972" name="18A3461989AC0244E050A20AB17C4671" maxlength="3" size="3"/> What I do next is update the text box input to 123 from 972. Now if I check the value attribute of the same text box using the getAttribute("value")...

Python AttributeError in Testing Goat


python,unit-testing,selenium
I've just started working through Test-Driven Development with Python and don't understand the AttributeError I'm getting as it's different from the one in the book. The code to run a selenium test is: from selenium import webdriver import unittest class new_visitor_test(unittest.TestCase): def set_up(self): self.browser = webdriver.Firefox() def tear_down(self): self.browser.quit() def...

Apache Nutch REST api


api,rest,web-crawler,nutch
I'm trying to launch a crawl via the rest api. A crawl starts with injecting urls. Using a chrome developer tool "Advanced Rest Client" I'm trying to build this POST payload up but the response I get is a 400 Bad Request. POST - http://localhost:8081/job/create Payload { "crawl-id":"crawl-01", "type":"INJECT", "config-id":"default",...

Return html code of dynamic page using selenium


python,python-2.7,selenium,selenium-webdriver,web-scraping
I'm trying to crawl this website, problem is it's dynamically loaded. Basically I want what I can see from the browser console, not what I see when I right click > show sources. I've tried some selenium examples but I can't get what I need. The code below uses selenium...

selenium webdriver - xpath locator not working if element's text contains Unicode Characters


selenium,xpath,unicode,selenium-webdriver,webdriver
I'm trying to select an option contained inside a menu. It's not a select menu, but it's styled to appear as such. Anyway, if the text contained inside the menu is in English, I can select it ok. Trouble is, the text I need to select is french so it...

Selenium can't seem to find and click a button on website? - Python


python,selenium,selenium-webdriver,find,element
I am trying to make software that automaticly collects data for me from fiverr.com website. It searches for specific term, and then it should press LOAD MORE button at the end of page, and keep clicking it until it loads all the 'gigs'. Anyway, I have good idea of how...

Upload file - Protractor


javascript,selenium,testing,protractor,end-to-end
I fight with protractor because for some tests I need to UPLOAD file. My HTML looks like: <div class="panel-footer"> <ul class="list-unstyled"> <!-- ngRepeat: file in imagesToUpload --> </ul> <button class="btn btn-sm btn-success pull-right ng-binding ng-hide" ng-show="imagesToUpload.length" ng-click="uploadImages()">Nahrát na server</button> <button class="btn btn-sm btn-primary ng-binding" ng-file-select="onImageSelect($files)" data-multiple="true" style="overflow: hidden;">Vybrat soubory<input...

WebDriver can't get dropdown menu element (Java)


java,selenium,webdriver,junit4
I write a script on Java for Selenium WebDriver, and I have a problem with selected from dropdown menu. Here's my locator: new Select(driver.findElement(By.id("FormElement_select_68_input_input"))).selectByVisibleText("Image"); Here's an error: http://prntscr.com/7jul03 Here's HTML code: http://prntscr.com/7jvou6 Need to select "Image" from this menu, but have an error. Before I had the error like this,...

Checking element existence on the page with WebDriver findElements().isEmpty method


java,selenium,selenium-webdriver
I wish to check if some element exists on the page or not. I already saw this WebDriver: check if an element exists? issue but I'm wondering why not simply apply findElements().isEmpty method? I thought it would do the same work. UPD Now I see findElements().isEmpty works perfect so I'm...

When starting Nightwatch with Grunt, the website server is not started


javascript,node.js,selenium,gruntjs,nightwatch.js
I am using Nightwatch.js to run system tests for a website. I want to automate the tests by running them via grunt. My Gruntfile contains these lines: ... var nightwatch = require('nightwatch'); nightwatch.initGrunt(grunt); ... nightwatch: { options: { standalone: true, test_settings: { "default": { "launch_url": "http://localhost", "selenium_port": 4444, "selenium_host": "localhost",...

Not able to click ExtJS Dropdown button and select list elements - Selenium Webdriver Java


java,selenium,testing,extjs,selenium-webdriver
I am having difficulty in clicking a drop down and selecting option from the list. Below screenshot of the application. Our application uses heavy ExtJS and below is the HTML code for dropdown button and textbox. <div id="combobox-1115-bodyEl" class="x-form-item-body x-form-trigger-wrap-focus" role="presentation" style="width: 325px;"> <div id="ext-gen1273" class="x-hide-display x-form-data-hidden" role="presentation"></div> <input id="ext-gen1272"...

How to get selected option using Selenium WebDriver with Python?


python,selenium,selenium-webdriver,selecteditem,selected
How to get selected option using Selenium WebDriver with Python: Someone have a solution for a getFirstSelectedOption? I'm using this to get the select element: try: FCSelect = driver.find_element_by_id('FCenter') self.TestEventLog = self.TestEventLog + "<br>Verify Form Elements: F Center Select found" except NoSuchElementException: self.TestEventLog = self.TestEventLog + "<br>Error: Select FCenter element...

How to parse Selenium driver elements?


python,parsing,selenium,selenium-webdriver,web-scraping
I'm new in Selenium with Python. I'm trying to scrape some data but I can't figure out how to parse outputs from commands like this: driver.find_elements_by_css_selector("div.flightbox") I was trying to google some tutorial but I've found nothing for Python. Could you give me a hint?...

Webdriver C# -Click second linktext not first


c#,selenium,webdriver
By this line I click on the first text, but how can I click the second exact text? driver.FindElement(By.PartialLinkText("Play")).Click(); ...

Python Selenium - Login error


python,selenium
I'm trying to input login details into GMAIL with the below coode from selenium import webdriver import getpass chromedriver = 'C:\Python34\Scripts\chromedriver' driver = webdriver.Chrome(chromedriver) driver.get('http://www.google.com/adwords/') signin = driver.find_element_by_class_name('ignore-channel') signin.click() email = input('Enter your Email ID : ') password = getpass.getpass('Password :') email = driver.find_element_by_id('Email') email.send_keys(email) passwd = driver.find_element_by_id('Passwd') passwd.send_keys(password) submit...

XPath for child element


selenium,xpath
What is the Xpath to select the first element of this jquery selector: $('.A .B:eq(1)') ? HTML Sample <div class="x A z"> <div class="y"> <div class="r B z"></div> <---- that is that I need to select <div class="r B z"></div > ... So far I've tried this: (//div[(@class='A') and div[contains(concat('...

How to Find class inside in
, using Selenium WebDriver


selenium,webdriver
here is the code: <div class="padding-tlb"> ....Some code.... </div> <article class="non-article-login"> <div class="one-fourth-percent columns-percent">&nbsp;</div> <div class="one-half-percent columns-percent"> <div class="one-fourth-percent columns-percent"> User ID </div> </div> </div> </article> I want to check if exist the class "one-fourth-percent columns-percent". WebElement test = driver.findElement(By.className("one-fourth-percent columns-percent")); But...

Selecting elements by ng-model attribute with selenium-webdriver


ruby,angularjs,selenium,selenium-webdriver
I am using Ruby and the gem selenium-webdriver I am working with an angular form that I have no control over and would like to know how to find elements via their ng-model attributes. Here is an example select element: <select class="medium-3 small-12 columns ng-pristine ng-valid ng-touched" ng-model="ctrl.type" ng-options="type.name for...

Is it possible to save an adobe pdf file using selenium web driver and one click build Jenkins


java,pdf,selenium,jenkins
Here's my dilemma. I'm creating some automation tests using selenium web driver coding in Java. The main output of the software is to load a PDF in adobe acrobat. I'm also using Jenkins which is why my problem has come about as I wont be able to physically see the...

Selenium C# Element Not Found Taking a Long Time


c#,unit-testing,selenium,selenium-webdriver
I am using Selenium for C# to test a page. Is there a way to quickly fail if the element is not found on the page? I am experiencing if the HTML element is not found on the page the Selenium test a very long time and then eventually it...

Getting value from div which contains exact string using Selenium IDE


selenium,selenium-ide
I am trying to get a value from a nested div in a web page using Selenium IDE where class names are repeated but contain unique strings, such as the structure below, but am having no luck. <div class='ClassName'> <div class="col_Num">1 </div> <div class="col_Val">5.00 </div> </div> <div class='ClassName'> <div class="col_Num">2...

Java Selenium - Can't seem to select an element


java,selenium,xpath,hidden
I'm trying to select an e-mail href to grab the text, but no matter what selections I use to try to select the e-mail, my selection doesn't seem to work and I am curious what other ways might exist to fix the problem. I have used absolute and relative xpath,...

Selenium catch popup on close browser


java,selenium,browser
I'm trying to test that when I close my window a popup shows with a warning message. I've tried both driver.close() and driver.quit() after making sure I'm on the proper window but this just terminates the process since my popup doesn't show. I could test it by using the awt...

Selenium WebDriver: unable to locate element inside iframe using TinyMCE editor


java,selenium,iframe,selenium-webdriver
Maybe my question is similar to this Selenium WebDriver NoSuchElementException: Unable to locate element however I still can't find required element. The element itself looks: body id="tinymce" class="mceContentBody" contenteditable="true" onload="window.parent.tinyMCE.get('compose_295_composeEditor').onLoad.dispatch();" dir="ltr" style="overflow: auto;"> the iframe containing this element is: body id="tinymce" class="mceContentBody" contenteditable="true"...

Junit parameterized testing ,a different thought


java,selenium,junit
Here, I have a custom independent component which accept input as a jar file and identify the junit test classes from it and execute them using the code given below. Result result = JUnitCore.runClasses(cls);//cls is Class<?>instance. Next thing I required is a provision to inject parameters (Assuming that input jar...

Check if element exists in fetched URL [closed]


javascript,jquery,python,web-crawler,window.open
I have a page with, say, 30 URLS, I need to click on each and check if an element exists. Currently, this means: $('area').each(function(){ $(this).attr('target','_blank'); var _href = $(this).attr("href"); var appID = (window.location.href).split('?')[1]; $(this).attr("href", _href + '?' + appID); $(this).trigger('click'); }); Which opens 30 new tabs, and I manually go...

How to map browser dialog using capybara/selenium


ruby,selenium,capybara
What is the best way to handle browser dialogs (with ok/cancel buttons) using capybara/selenium? The way I found easiest is by running the following: dialog = page.driver.browser.switch_to.alert dialog.accept Any thoughts for a better approach? Whoever downed this post please give a valid reason... it's not helpful to down vote with...

Multi user file property Selenium


java,selenium,junit,webdriver,automated-tests
I made autotest on Selenium. This test I will start on JMeter for load testing for 10, 20, 50+ users. What I should do. I create a property file (configuration file) and put there URL, Login, Password. So I made a cycle and put there this code where I'll start...

close and open browser -selenium C# Visual studio


c#,selenium,browser,close
I want to close and then open the browser without close the driver of selenium, using C# I don't want to: driver.Quit(); How Can I close and the open again the browser Chrome?...

webdriver C# - click this element with XPath position?


c#,selenium,xpath,webdriver
I have an element whit this XPath Position: //td[4]/a I try this but does not work: driver.FindElement(By.XPath(".//*[@position='//td[4]/a']")).Click(); Using C# Webdriver and not java, please....

Find element by class name


python,parsing,selenium,selenium-webdriver,css-selectors
I'm trying to find one tag using we.find_element_by_css_selector('p.p1.transfer strong.ng-binding').text The problem is that there is sometimes a 'strong' tag before the tag I'm searching for which is very similar but it's class is: class="ng-binding ng-hide" instead of class="ng-binding". But when I try to find it it finds the first tag....

How to select a text from the autocomplete textbox using selenium


java,selenium,selenium-webdriver,browser-automation
I need to enter some text in a autocomplete textbox. Then I will select a option from that autocomplete option and need to click it. I have tried with the following code: public static void main(String[] args) throws InterruptedException { // TODO Auto-generated method stub String textToSelect = "headlines today";...

Unable to select text boxes in selenium webdriver through XPath?


selenium,xpath
My HTML <table border="2" style="background:gray"> <tbody> <td class="std"> <input id="ActQ1Revenue" type="text" name="amount" disabled="disabled" maxlength="20" style="background:wheat"> </td> </tr> <tr> <td class="atd"> <input id="ActAprRevenue" type="text" name="amount" maxlength="20"> </td> </tr> <tr> <td class="atd"> <input id="ActMayRevenue" type="text" name="amount" maxlength="20"> </td> </tbody> </table> My XPath...

Protractor, Done and Expect, why do we need wait?


selenium,selenium-webdriver,jasmine,protractor
I thought Done would make things run synchronously, meaning after I click a link, the click call back would happen after the click, apparently not as this doesn't work. browser.ignoreSynchronization = true; var firstURL = "https://www.google.com/?gws_rd=ssl"; describe("test", function () { browser.get("http://www.google.com"); it("Should be on google url", function () { expect(browser.getCurrentUrl()).toBe(firstURL);...

How to write xpath in selenium webdriver for below HTML expressions?


selenium
I wrote xpath for below HTML code i.e. displayed below 1. //a[@text()='Life Insurance'] 2. //span[@text()='Apply now'] But I got element not found exception. If I used Absolute xpath processor then It's working and I wrote own xpath then it thrown exception. Please tell me how to write it. Below are...

Not able to locate the dropdown value using selenium webdriver


java,selenium,selenium-webdriver
I want to select the value from dropdown. Currently I am able to click on dropdown but not able to select the value from the dropdown. Below are the code which I am using for selecting the value from the dropdown. temp.click(); driver.manage().timeouts().implicitlyWait(1, TimeUnit.SECONDS); Select clickThis = new Select(temp); try{...

Is there a different way to use methods of another class? [on hold]


java,selenium
I fixed it. I extended the AbstractPage where the methods are. At first I would get a Test class must have a singular no-Argument constructor. Then I realized that in the @Before the driver variable and startup variable made problems after I removed those and just called the open chromedriver()...

Should I use ids to locate elements?


angularjs,selenium,protractor,end-to-end,e2e-testing
Started with Angular and Protractor. It just feels wrong to write some heavy css selectors which will break instant when you change something. Using ID's would make testing way easier. I'm not using any id attribute for styling yet. Are there any drawbacks using ids for testing I haven't considered?...

Python Selenium selecting div class


python,selenium
So I'm writing a script using Selenium but I got stuck at finding a div class. The structure on the website is like this: <div class="field first_name"> <input type="text" placeholder="First Name"></input> </div>` Already tried something like driver.find_element_by_xpath("//field[@first_name]") But it didn't work for some reason, maybe I did it wrong or...

Distinguishing between HTML and non-HTML pages in Scrapy


python,html,web-crawler,scrapy,scrapy-spider
I am building a Spider in Scrapy that follows all the links it can find, and sends the url to a pipeline. At the moment, this is my code: from scrapy import Spider from scrapy.http import Request from scrapy.http import TextResponse from scrapy.selector import Selector from scrapyTest.items import TestItem import...

T_STRING error in my php code [duplicate]


php,web-crawler
This question already has an answer here: PHP Parse/Syntax Errors; and How to solve them? 10 answers I have this PHP that is supposed to crawl End Clothing website for product IDs When I run it its gives me this error Parse error: syntax error, unexpected 'i' (T_STRING), expecting...

Heritrix not finding CSS files in conditional comment blocks


java,web-crawler,heritrix
The Problem/evidence Heritrix is not detecting the presence of files in conditional comments that open & close in one string, such as this: <!--[if (gt IE 8)|!(IE)]><!--> <link rel="stylesheet" href="/css/mod.css" /> <!--<![endif]--> However standard conditional blocks like this work fine: <!--[if lte IE 9]> <script src="/js/ltei9.js"></script> <![endif]--> I've identified the...

Selenium Web Driver : How to map html elements to Java Object.


selenium,webdriver,mapping
As part of Selenium Web-driver learning I came across a scenario. Please let me know the professional approach to proceed. I am testing a eCommerce application where while I click on Mobile link all mobile phones are getting displayed.I want to check whether they are sorted based on name and...

Adding a public class to click using xpath with Java Selenium


java,selenium,xpath
I am trying to create a public class to click an item on a webpage with selenium by just passing it the xpath and driver I'm using. I want to be able to just do: ClickByXpath(driver, "/html/body/div/div[3]/form/div[2]/div[2]/div[1]/div[1]/div[3]/div/div[3]/div/input[1]"); Here's the code I'm using, but it's complaining that the method xpath string...

Scrapy not entering parse method


python,selenium,web-scraping,web-crawler,scrapy
I don't understand why this code is not entering the parse method. It is pretty similar to the basic spider examples from the doc: http://doc.scrapy.org/en/latest/topics/spiders.html And I'm pretty sure this worked earlier in the day... Not sure if I modified something or not.. from selenium.webdriver.support.wait import WebDriverWait from selenium.webdriver.common.by import...

CSS Selector-How to locate Parent Element


selenium,selenium-webdriver,css-selectors,webdriver
Is there any way to locate parent element in CSS Selector? i am using below code but i am not getting the parent element. WebElement we=dr.findElement(By.cssSelector("div[id='gf-BIG']:parent")); I know there is a way in Xapth but please let me know that how we can locate parent in CSS Selector. ...

selenium exceptions.NoSuchElementException:


python,selenium,webdriver
The html code snippet as: <thead id="addDet:cust:th" class="emp"> <tr id="addDet:cust:ch" class="emp"> <th class="emp" scope="col" id="addDet:cust:ch:j_idt484">S. No.</th> <th class="emp" scope="col" id="addDet:cust:ch:j_idt487">Name <font color="#FA5882">*</font></th> I tried to fill the field as: driver.find_element_by_id("addDet:cust:j_idt487").send_keys("XX") But I am getting the error as: selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element:...