python,html,web-scraping,beautifulsoup,html-parsing , BeautifulSoup is not getting all data, only some

BeautifulSoup is not getting all data, only some


Tag: python,html,web-scraping,beautifulsoup,html-parsing
import requests
from bs4 import BeautifulSoup

def trade_spider(max_pages):
page = 0
while page <= max_pages:
    url = '' + str(page * 100)
    source_code = requests.get(url)
    plain_text = source_code.text
    soup = BeautifulSoup(plain_text)
    for link in soup.findAll('a', {'class':'hdrlnk'}):
        href = '' + link.get('href')
        title = link.string
        print title
        #print href
    page += 1

def get_single_item_data(item_url):
    source_code = requests.get(item_url)
    plain_text = source_code.text
    soup = BeautifulSoup(plain_text)
    for item_name in soup.findAll('section', {'id':'postingbody'}):
        print item_name.string


I am trying to crawl craigslist (for practice), in particular. I have it right now set to print the title of the entry and the description of the entry. The issue is that although the title correctly prints for every object listed, the description is listed as "None" for most of them, even though there is clearly a description. Any help would be appreciated. Thanks.


Instead of getting the .string, get the text of the posting body (worked for me):


As a side note, your script has a blocking "nature", you may speed things up dramatically by switching to Scrapy web-scraping framework.


Python - Opening and changing large text files

I have a ~600MB Roblox type .mesh file, which reads like a text file in any text editor. I have the following code below: mesh = open("file.mesh", "r").read() mesh = mesh.replace("[", "{").replace("]", "}").replace("}{", "},{") mesh = "{"+mesh+"}" f = open("p2t.txt", "w") f.write(mesh) It returns: Traceback (most recent call last): File...

Centering navbar pills vertically within the navbar using flexbox

I am terrible at CSS so I am having trouble centering my <li> (navbar pills) vertically for my navbar. This navbar is from twitter bootstrap Here is the HTML for my navbar: <div class="container"> <nav class="navbar navbar-default navbar-fixed-top"> <ul id="nav_pills" class="nav nav-pills" role="tablist"> <li role="presentation"> <a href="/">About</a> </li> <li role="presentation">...

How to set DIV's width based on CSS indexes

I have 6 floated DIVs in two rows. I use PocketGrid as CSS framework (in particular its "automatic rows" feature) and I would like to size each one using CSS indexes and not specific class name. I tried it in this jsFiddle but even if first 3 blocks (first row)...

Dynamically select from a dynamically generated dropdown

I have a dynamically generated dropdown list - list of course identifiers and names. On the basis of a variable, “assigned_course_id”, I would like to preselect the appropriate value from the dropdown list. My best attempt is as follows. Thanks in advance for your assistance. <select name="course_id" id="course_id"> <?php $assigned_course_id...

CSS - Linear Gradient Background Color no-repeat is not working for if it has multiple tds

I want to apply css3 gradient (left and right transparent) background color for a TR element like below. Below code is working fine if it has only one TD element, where as more than one td, background gradient started repeating for each td... Am I missing something? Online Demo CSS...

How to make background body overlay when use twitter-bootstrap popover?

Currently I used this twitter-bootstrap popover: <button type="button" class="btn btn-default" style="margin-top:11px; padding-bottom:4px; padding-top:4px;" data-container="body" data-toggle="popover" data-trigger="focus" data-placement="left" data-content="Vivamus sagittis lacus vel augue laoreet rutrum faucibus."> Upgrade The output: How I can make the background overlay like bootstrap modal? My expected output I use this jquery to popover $(function(){ $('[data-toggle="popover"]').popover({ placement...

Setting radio button checked true on button click

I have a table cell that lights up when selected. There will be many buttons on the page but only one choice can be made at a time. How can I make them mutually exclusive using hidden input radio tags? (in order to handle the group later) <td class="choice-option"> <input...

HTML CSS Two 2-column tables side by side with same height and width

I'm quering a MySQL database and use a script to put results into an automatically generated HTML-file. I have two tables with 2 columns and 4 rows each that need to be put side-by-side. I have put my html & css in fiddle. My HTML code: <!-- gene description...

Find the tf-idf score of specific words in documents using sklearn

I have code that runs basic TF-IDF vectorizer on a collection of documents, returning a sparse matrix of D X F where D is the number of documents and F is the number of terms. No problem. But how do I find the TF-IDF score of a specific term in...

show div only when printing

Let's say that I have <div id="printOnly"> <b>Title</b> <p> Printing content </p> </div> Is it possible to hide this div when page rendering and to show only when printing this div?...

Adding horizontal space between 2 buttons using javascript

I am dynamically creating submit buttons using javascript. I wanted to know how to introduce horizontal spacing between every other button in the process. My code is var allFields = commonButtons.concat(extraFields); for (var i = 0; i < allFields.length; i++) { var node = document.createElement("input"); var btName = document.getElementById("submit-buttons"); node.setAttribute('data-value',...

bootstrap - dynamically changing jumbotron background image

I'm trying to change the background image of the jumbotron div with jQuery when a new tab is clicked, like this: $(".about").click(function(){ $('.active').toggleClass('active'); $('.about').toggleClass('active'); $('.jumbotron').fadeOut(500); //change background image $('.jumbotron').fadeIn(500); }); I tried using $('.jumbotron').css('background-image',url('/path/to/new/image')); but when I include that line, the jumbotron background image doesn't display at all even before...

copy last column of one table to another table based on some condition

I have two tables. I want to take last column of table3 and put in to table4 eg:- table3 Names Process_id total construction 1 1111 construction_1 1 0000 engineering 1 2222 permitting 1 3333 eg:- table4 Names Process_id construction 1005 engineering 1005 permitting 1005 final result of table4 output will...

Website showing differently in windows xp and mobile

I made a website for one of my clients, and I change the background and font color. Now on windows xp and smartphones the background changed color to the default color and the fonts remained the same. The website is I don't know what to do to make it...

Sum of two variables in RobotFramework

I have two variables: ${calculatedTotalPrice} = 42,42 ${productPrice1} = 43,15 I executed ${calculatedTotalPrice} Evaluate ${calculatedTotalPrice}+${productPrice1} I got 42,85,15 How can I resolve it?...

Target next instance of an element/div

I have the following HTML structure and JavaScript file: .html <li> <button class="show-more"></button> some more elements <div class="hidden"></div> </li> JavaScript $( ".show-more" ).click(function() { event.preventDefault(); $( this ).next().slideToggle( "fast", function() { }); }); I need the click event to toggle the next first instance of .hidden, however the click event...

trying to understand LSH through the sample python code

the concise python code i study for is here Question A @ line 8 i do not really understand the syntax meaning for "res = res << 1" for the purpose of "get_signature" Question B @ line 49 (SOLVED BY myself through another Q&A) "xor = r1^r2" does not really...

How do variables inside python modules work?

I am coming from a Java background with Static variables, and I am trying to create a list of commonly used strings in my python application. I understand there are no static variables in python so I have written a module as follows: import os APP_NAME = 'Window Logger' APP_DATA_FOLDER_PATH...

Strange Behavior: Floating Point Error after Appending to List

I am writing a simple function to step through a range with floating step size. To keep the output neat, I wrote a function, correct, that corrects the floating point error that is common after an arithmetic operation. That is to say: correct(0.3999999999) outputs 0.4, correct(0.1000000001) outputs 0.1, etc. Here's...

Get elements containing text from array

I want to search all the elements containing any string in the array. For example I have following list of items <ul> <li>cricket bat</li> <li>tennis ball</li> <li>golf ball</li> <li>hockey stick</li> </ul> and this array var arr = ['bat', 'ball']; It should select all the elements having text bat and ball....

Automatically calling server side class without

I have an anchor which calls a server side class when clicked, but I want to modify it so that the class is called as soon as the page loads, without having to click an anchor. <a href="#" class="_repLikeMore" data-id="1234" data-type="pid" data-app="forums"> ...

Div with the form of a pencil [duplicate]

This question already has an answer here: CSS triangle custom border color 2 answers How can i create a div with the form of a pencil, just like this: It seems a basic thing but im trying to do it since some time ago and still couldnt do it....

How to remove unmatched row in html table using jquery

I have two tables. I want to delete unmatching rows. Compare with first column in Table1 and compare with first column in Table2. Table1 111 aaa 222 bbb 333 ccc Table2 333 xxx 444 zzz 111 vvv result of Table2 333 xxx 111 vvv I tried some thing here Please...

Top header 100% of screen, but body only 70%?

Pretty new to CSS and just having quite a bit of trouble, I've tried everything, searched here, but can't seem to make it work. Right now my header/body are both 70% of the screen. However I want my top header (.mainheader) to be 100% of the screen, but have the...

SQLAlchemy. 2 different relationships for 1 column

I have a simple many-to-many relationship with associated table: with following data: matches: users: users_mathces: ONE user can play MANY matches and ONE match can involve up to TWO users I want to realize proper relationships in both "Match" and "User" classes users_matches_table = Table('users_matches', Base.metadata, Column('match_id', Integer, ForeignKey('', onupdate="CASCADE",...

How to change the IP address of Amazon EC2 instance using boto library

How can I assign a new IP address (or Elastic IP) to an already existing AWS EC2 instance using boto library.

submitting form then showing loading image by javascript

I am submitting a form and loading loading gif in the same div by replacing html of form by html of loading image. I am first submitting the form then loading gif, because I have to replace the content of div(in which form exist) with loading image. Logs 1,2,3 are...

Calling function and passing arguments multiple times

I want to call the function multiple time and use it's returned argument everytime when it's called. For example: def myfunction(first, second, third): return (first+1,second+1,third+1) 1st call: myfunction(1,2,3) 2nd call is going to be pass returned variables: myfunction(2,3,4) and loop it until defined times. How can I do such loop?...

How to remove all the borders of a selectbox?

How can I remove the all the borders of the selectbox using css or Jquery ? My code, <select id="doctor_ch"> <option value="1" selected>One</option> <option value="2">Two</option> </select> CSS #doctor_ch{ background-color: #88AFF2; color:#fff; margin-top: 15px; } When I use this code it only changes the arrow style. I want to remove the...

Detect when the jQuery UI slider is being moved?

I have this custom video time UI slider to change the time of the YouTube video when scrubbed. My problem is that when the video is trying to load when the user is moving the slider, it causes the handle to jerk around and flip around. What I'm trying to...

In sklearn, does a fitted pipeline reapply every transform?

Apologies if this is obvious but I couldn't find a clear answer to this: Say I've used a pretty typical pipeline: feat_sel = RandomizedLogisticRegression() clf = RandomForestClassifier() pl = Pipeline([ ('preprocessing', preprocessing.StandardScaler()), ('feature_selection', feat_sel), ('classification', clf)]),y) Now when I apply pl on a new set, pl.predict(X_classify); is RandomizedLogisticRegression going...

Count function counting only last line of my list

Count function counting only last line of my list N = int(raw_input()) cnt = [] for i in range(N): string = raw_input() for j in range(1,len(string)): if string[j] =='K': cnt.append('R') elif string[j] =='R': cnt.append('R') if string[0] == 'k': cnt.append('k') elif string[0] == 'R': cnt.append('R') print cnt.count('R') if I am giving...

Using counter on array for one value while keeping index of other values

After reading the answers on this question How to count the frequency of the elements in a list? I was wondering how to count the frequency of something, and at the same time retreive some extra information, through something like an index. For example a = ['fruit','Item#001'] b = ['fruit','Item#002']...

Python recursive function not recursing

I'm trying to solve a puzzle, which is to reverse engineer this code, to get a list of possible passwords, and from those there should be one that 'stands out', and should work function checkPass(password) { var total = 0; var charlist = "abcdefghijklmnopqrstuvwxyz"; for (var i = 0; i...

why i don't get return value javascript

When i debug my code i can see i have value but i don't get value to createCheckBoxPlatform FN function createCheckBoxPlatform(data) { var platform = ""; $.each(data, function (i, item) { platform += '<label><input type="checkbox" name="' + item.PlatformName + ' value="' + item.PlatformSK + '">' + item.PlatformName + '</label>' +...

HTML elements in Angular bindings expression

Is that possible to insert HTML elements in an Angular expression ? Let's take a few example. I would like to do something like this: <table> <tr ng-repeat="employee in employees"> <td>{{employee.firstname ? employee.firstname : '<p style="color:red">No name</p>'}}</td> <td>{{employee.job}}</td> </tr> </table> In our controller, we have: $scope.employees = [{firstname:'Bob', job:'Developer'}, {firstname:'Paul',...

CSS :hover that shows more than one image

I have an icon on my website. I want to change the icon to 4 different images when ever I hover over the first one. So I know how to switch between the regilar image to yellow0.png, but how do I continue to the next one (after half a second...

Sort when values are None or empty strings python

I have a list with dictionaries in which I sort them on different values. I'm doing it with these lines of code: def orderBy(self, col, dir, objlist): if dir == 'asc': sorted_objects = sorted(objlist, key=lambda k: k[col]) else: sorted_objects = sorted(objlist, key=lambda k: k[col], reverse=True) return sorted_objects Now the problem...

Identify that a string could be a datetime object

If I knew the format in which a string represents date-time information, then I can easily use datetime.datetime.strptime(s, fmt). However, without knowing the format of the string beforehand, would it be possible to determine whether a given string contains something that could be parsed as a datetime object with the...

Python: histogram/ binning data from 2 arrays.

I have two arrays of data: one is a radius values and the other is a corresponding intensity reading at that intensity: e.g. a small section of the data. First column is radius and the second is the intensities. 29.77036614 0.04464427 29.70281027 0.07771409 29.63523525 0.09424901 29.3639355 1.322793 29.29596385 2.321502 29.22783249...

show/hide an overflow div on anchor

I'm trying to make a div appear (if not already visible) and be scrolled to a specific anchor. I found this answer and try to use it but it looks like it doesn't work well... My code : As you can see, when you click on button it...

How does the class_weight parameter in scikit-learn work?

I am having a lot of trouble understanding how the class_weight parameter in scikit-learn's Logistic Regression operates. The Situation I want to use logistic regression to do binary classification on a very unbalanced data set. The classes are labelled 0 (negative) and 1 (positive) and the observed data is in...

How to remove structure with python from this case?
How to remove "table" from HTML using python? I had case like this: paragraph = ''' <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit. Quidem molestiae consequuntur officiis corporis sint.<br /><br /> <table> <tr> <td> text title </td> <td> text title 2 </td> </tr> </table> <p> lorem ipsum</p> ''' how...

Click on link next link should be display on same page

I have a single page website and need to link the navigation to IDs in the page. I have three links: "About us", "Our Project", "contact". So if user clicks on "About ", the About section will be displayed, same with other links. Inside Our project there is Two buttons...

Background-image style with JS not working in ie9

I'm working on a site and im having some issues with my slider in IE9. What I've done is that I've made a div that gets the background image changed every few seconds depending on the img tags within. You can see it in function here: It works wonders...

Dynamically resize side-by-side images with different dimensions to the same height

I have two images side-by-side within a block-level container with arbitrarily different dimensions (as in, they could be any two images) that I want to dynamically adjust the width of so that the overall height of the two images is the same. I don't think this can be done in...

How do I display my mysql table column headers in my php/html output?

2 Questions... Scenario: I would like to query my database table via a form and then display the results that occur(if there are results) and my current situation is that it does work but it clears the form completely and leaves my to an empty page with just the results...

How do I read this list and parse it?

I'm using requests and the output I get from the sites API is a list, I've been stuck trying to parse it to get the data from it. I use r = requests.get(urlas, params=params) r.json() to get the data I want. Here is a snippet of the list [{'relation_type': None,...

tag in HAML

How do I create a I-tag in haml? Example: <i class="fa fa-search"></i> ...

change css dynamically by selecting dropdown list item

I have the following Dropdownlist with yes or no options-:` <asp:DropDownList ID="ddlchangecss" runat="server"> <asp:ListItem Text="yes">yes</asp:ListItem> <asp:ListItem Text="no"></asp:ListItem> </asp:DropDownList> i want to add a css dynamically like that-: .imgFullWidth { width: 100%; height: auto; float: left; margin: 0px 0px 20px 0px; } when I select yes option in Dropdownlist. How can...