ruby,web-crawler,nokogiri,net-http,mechanize-ruby , Net/HTTPS not getting all the content

## Question:

Tag: ruby,web-crawler,nokogiri,net-http,mechanize-ruby

I need to login into Jenkins through a crawler to collect some data, but Net/HTTPS gets an incomplete page in comparison to Jenkins' source, here are both sources:

Net/HTTPS' HTML

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html>

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<script>
</script>

<body style="background-color:white; color:white;">Authentication required</body>

</html>

Nokogiri's XML

=> #
<Nokogiri::HTML::Document:0x1a11444 name="document" children=[#<Nokogiri::XML::DTD:0x1a109b8 name="html">, #
<Nokogiri::XML::Element:0x1a101ac name="html" children=[#<Nokogiri::XML::Element:0x2047ee4 name="head" children=[#<Nokogiri::XML::Element:0x2047d04 name="meta" attributes=[#<Nokogiri::XML::Attr:0x2047ca0 name="http-equiv" value="refresh">, #
<Nokogiri::XML::Element:0x20471ec name="body" attributes=[#<Nokogiri::XML::Attr:0x2047188 name="style" value="background-color:white; color:white;">] children=[#
<Nokogiri::XML::Text:0x2046d50 "Authentication required">]>]>]>

Jenkins' source

<!DOCTYPE html>
<html>

<title>Jenkins</title>
<link rel="shortcut icon" type="image/vnd.microsoft.icon" href="/static/98ff49d3/favicon.ico" />
<script>
var isRunAsTest = false;
var rootURL = "";
var resURL = "/static/98ff49d3";
</script>
<script src="/static/98ff49d3/scripts/prototype.js" type="text/javascript"></script>
<script src="/static/98ff49d3/scripts/behavior.js" type="text/javascript"></script>
<script src="/static/98ff49d3/scripts/yui/yahoo/yahoo-min.js"></script>
<script src="/static/98ff49d3/scripts/yui/dom/dom-min.js"></script>
<script src="/static/98ff49d3/scripts/yui/event/event-min.js"></script>
<script src="/static/98ff49d3/scripts/yui/animation/animation-min.js"></script>
<script src="/static/98ff49d3/scripts/yui/dragdrop/dragdrop-min.js"></script>
<script src="/static/98ff49d3/scripts/yui/container/container-min.js"></script>
<script src="/static/98ff49d3/scripts/yui/connection/connection-min.js"></script>
<script src="/static/98ff49d3/scripts/yui/datasource/datasource-min.js"></script>
<script src="/static/98ff49d3/scripts/yui/autocomplete/autocomplete-min.js"></script>
<script src="/static/98ff49d3/scripts/yui/element/element-min.js"></script>
<script src="/static/98ff49d3/scripts/yui/button/button-min.js"></script>
<script src="/static/98ff49d3/scripts/yui/storage/storage-min.js"></script>
<script src="/static/98ff49d3/scripts/hudson-behavior.js" type="text/javascript"></script>
<script src="/static/98ff49d3/scripts/sortable.js" type="text/javascript"></script>
<script>
crumb.init("", "");
</script>
<meta name="ROBOTS" content="INDEX,NOFOLLOW" />

<div class="logo">
<img id="jenkins-name-icon" height="34" alt="title" width="139" src="/static/98ff49d3/images/title.png" />
</a>
</div>
|
</div>
<div class="searchbox hidden-xs">
<form style="position:relative;" name="search" action="/search/" class="no-json" method="get">
<div id="search-box-minWidth"></div>
<div id="search-box-sizer"></div>
<div id="searchform">
<input id="search-box" placeholder="search" name="q" class="has-default-text" />
<a href="http://wiki.jenkins-ci.org/display/JENKINS/Search+Box">
<img style="width: 16px; height: 16px; " class="icon-help icon-sm" src="/static/98ff49d3/images/16x16/help.png" />
</a>
<div id="search-box-completion"></div>
<script>
createSearchBox("/search/");
</script>
</div>
</form>
</div>
</div>
<tr id="top-nav">
<td id="left-top-nav" colspan="2">
<div class="top-sticker noedge">
<div class="top-sticker-inner">
<div id="right-top-nav"></div>
</li>
<li class="children" href="/"></li>
</ul>
</div>
</div>
</td>
</tr>
</div>
</div>
<div id="page-body">
<div class="row">
<div id="side-panel">
<div id="side-panel-content"></div>
</div>
<div id="main-panel">
<div id="main-panel-content">
<a name="skip2content"></a>
<div style="margin: 2em;">
<table>
<tr>
<td>User:</td>
<td>
</td>
</tr>
<tr>
<td>
</td>
</tr>
<tr>
<td align="right">
<input id="remember_me" type="checkbox" name="remember_me" />
</td>
<td>
<label for="remember_me">Remember me on this computer</label>
</td>
</tr>
</table>
<input name="from" value="/" type="hidden" />
<script>

## Seeding fails validation for nested tables (validates_presence_of)

ruby-on-rails,ruby,validation,ruby-on-rails-4,associations
An Organization model has a 1:many association with a User model. I have the following validation in my User model file: belongs_to :organization validates_presence_of :organization_id, :unless => 'usertype==1' If usertype is 1, it means the user will have no organization associated to it. For a different usertype the presence of...

## Heroku rake db:migrate failing - uninitialized constant

ruby-on-rails,ruby,heroku
My app is working fine locally and my push to Heroku was successful. But, when I run heroku run rake db:migrate, I get the following error: NameError: uninitialized constant AddWeightToExercises Here is the failed migration: class AddWeightToExercise < ActiveRecord::Migration def change add_column :exercises, :weight, :float end end edit: Thanks for...

## Dividing by half in ruby to create an effective calculator

ruby,calculator
For the past while I've been working on a calculator, but have run into problems when needing to divide by a half. I'll add the offending bit of code along with a loop to keep it open below. on = true while on == true do half = 1.0 /...

## regex to pull in number with decimal or comma

ruby,regex
This is my line of code: col_value = line_item[column].scan(/\d+./).join().to_i When I enter 30,000 into the textfield, col_value is 30. I want it to bring in any number: 30,000 30.5 30.55 30000 Any of these are valid... Is there a problem with the scan and or join which would cause it...

## rails - NameError (undefined local variable or method while using has_many :through

ruby-on-rails,ruby,ruby-on-rails-4
My rails app gives following error: NameError (undefined local variable or method 'fac_allocs' for #): app/models/room.rb:4:in '' app/models/room.rb:1:in '' app/controllers/rooms_controller.rb:3:in 'index' room.rb file class Room < ActiveRecord::Base has_many :bookings has_many :fac_allocs has_many :facs, :through => fac_allocs end ...

## Ruby: How to copy the multidimensional array in new array?

ruby-on-rails,arrays,ruby,multidimensional-array
seating_arrangement [ [:first, :second, :none], [:first, :none, :second], [:second, :second, :first], ] I need to copy this array into new array. I tried to do it by following code: class Simulator @@current_state def initialize(seating_arrangement) @@current_state = seating_arrangement.dup end But whenever I am making any changes to seating_arrangement current_state changes automatically....

## Rails less url path change

ruby-on-rails,ruby,url,path,less
Developing a Rails application with the less-rails gem I found something unusual : // app/assets/common/css/desktop/typo.less @font-face{ font-family:'SomeFont'; src:url("fonts/db92e416-da16-4ae2-a4c9-378dc24b7952.eot?#iefix"); // ... } The requested font is app/assets/common/css/fonts/db92e416-da16-4ae2-a4c9-378dc24b7952.eot This font is compiled with less and the results is : @font-face { font-family: 'SomeFont'; src: url("desktop/fonts/db92e416-da16-4ae2-a4c9-378dc24b7952.eot?#iefix"); //... } Do you know why is...

## How to pivot array into another array in Ruby

arrays,ruby,csv
I have a multidimensional array like this one : myArray = [["Alaska","Rain","3"],["Alaska","Snow","4"],["Alabama","Snow","2"],["Alabama","Hail","1"]] I would like to end up with CSV output like this. State,Snow,Rain,Hail Alaska,4,3,nil Alabama,2,nil,1 I know that to get this outputted to CSV the way I want it I have to have output array like this: outputArray =[["State","Snow","Rain","Hail"],["Alaska",4,3,nil],["Alabama",2,nil,1]]...

## Get value from string representing local variable [duplicate]

ruby,local-variables
This question already has an answer here: Is there a 'variable_get' method? If not, how can I create my own? 2 answers I have a local variable name as a string, and need to get its value. variable = 22 "variable".to_variable? How can I get the value 22 form...

## Loop until i get correct user

ruby,redis
I have users stored in Redis and want to be able to call only certain subsets from a set, if i don't get the correct user back i want to put it back in the set and then try again until i get one of the desired users @redis =...

## .split and regular expression in Ruby

ruby,regex
I want to split my string by (',') but ignore ',' if they are inside quotes. For example " 2,2,4,'hello', 'world', 'hi, there' " I want to split it so 'hi, there' will not be split into two different array elements in ruby. How can I do that? Probably use...

## Undefined local variable post

ruby-on-rails,ruby,variables,undefined,local
I write this code: <% @user.posts.each do |post| %> <%= render 'post', locals: { post: post, user: @user} %> <% end %> Then in _post.htm.erb I write follow code: <div class="post-title"> <img src="<%= @user.avatar.url%>" class="img-rounded post-image"> <h4 id="post-name"><%= @user.first_name + ' ' [email protected]_name %> </h4> <div id="post-date"><%= post.created_at.strftime('%d %b -...

## Call method to generate arguments in ruby works in 1.8.7 but not 1.9.3

ruby-on-rails,ruby,ruby-1.9.3
This is something that I had working in ruby 1.8.7, but no longer works in 1.9.3, and I am not sure what changes make this fail. Previously, I had something like this myFunction(submitArgs()) where submitArgs was a helper method that could be called with some options def submitArgs(args={}) #Some logic/manipulations...

## How could I padding spaces to a fix length

ruby
I need all strings' length with 5 Original [477, 4770,] Expected ["477 ", "4770 "] How could I do it with Ruby ?...

## Ruby- get a xml node value

ruby,xml
can someone help me in extracting the node value for the element "Name". Type 1: I am able to extract the "name" value for the below xml by using the below code <Element> <Details> <ID>20367</ID> <Name>Ram</Name> <Name>Sam</Name> </Details> </Element> doc = Nokogiri::XML(response.body) values = doc.xpath('//Name').map{ |node| node.text}.join ',' puts values...

## Rails Association Guidance [on hold]

ruby-on-rails,ruby,ruby-on-rails-4,ruby-on-rails-3.2
I am new to rails 4. I have gone through lots of tutorials and trying to solve below scenario. But still no success. Can anybody point me in the right direction. How to handle associations for below scenario. Scenario: 1. Patient can have many surgeries. 2. Surgery has two types...

## Ruby boolean logic: some amount of variables are true

ruby
Let say I have 3 variables: a, b, c. How can I check that just zero or one of them is true?...

## Same enum values for multiple columns

ruby-on-rails,ruby,enums
I need to do something like this: class PlanetEdge < ActiveRecord::Base enum :first_planet [ :earth, :mars, :jupiter] enum :second_planet [ :earth, :mars, :jupiter] end Where my table is a table of edges but each vertex is an integer. However, it seems the abvove is not possible in rails. What might...

## How to flatten a structure of embedded Set and Hash

ruby,recursion
I would like to convert an embedding structure into a flat one. An embedding structure is a set of 0 or more objects, such as: a string or a hash having some string as key and some other embedding structure as value. A flat structure is a set of arrays...

ruby-on-rails,ruby,rest,activerecord,one-to-many
I'm creating a rails application that is a backend for a mobile application. The backend is implemented with a RESTful web API. Currently I am trying to add gamification to the platform through the use of badges that can be earned by the user. Right now the badges are tied...

## RSpec test for rake task

I have created a custom rake task that deletes all items that are >= 7 days old. I am trying to write a RSpec test for this new task but it seems like my task isn't really running in the test. I have tested the task manually and it works...

## On rendering from controller, current_page method does not seem to work

ruby-on-rails,ruby,ruby-on-rails-4,model-view-controller
I have a navigation bar included in application.html.erb. Because for some pages, such as the signup page, I need to place additional code inside the navigation bar, I have excluded those pages for showing the navigation bar through application.html.erb and instead included it in their respective view pages. See code...

## edit first line of multiple files in place with a ruby one-liner

ruby
I can edit multiple files in place using the -p -i -e command line options: ruby -pi -e 'sub(/a/,"b")' file* How can I edit only the first line of each file? This works for one file only: ruby -pi -e 'sub(/^/,"New line goes at top\n") if \$. == 1' file1...

## In Ruby how to put multiple lines in one guard clause?

ruby-on-rails,ruby
I have the following line of code : if params[:"available_#{district.id}"] == 'true' @deliverycharge = @product.deliverycharges.create!(districtrate_id: district.id) delivery_custom_price(district) end Rubocop highlight it and asks me to use a guard clause for it. How can I do it? EDIT : Rubocop highlighted the first line and gave this message Use a guard...

## Using Ruby Pathname to access relative directory

ruby,path,pathname
Given I have a relative path pointing to a directory how can I use it with Ruby's Pathname or File library to get the directory itself? p = Pathname.new('dir/') p.dirname => . p.directory? => false I have tried './dir/', 'dir/', 'dir'. What I want is p.dirname to return 'dir'. I...

## Rails - link_to path based on object's name + refactoring multiple custom actions

ruby-on-rails,ruby,refactoring
I'm looking to simplify the link_to path based on thr object's name and also am looking into refactoring multiple custom actions. I've managed to get this working below. <% ServiceMenu.all.each do |menu| %> <tr class=" <%= cycle('odd', 'even') %>"> <td><%= link_to menu.name, ("tech/""#{menu.name.parameterize}") %></td> </tr> <% end %> I feel...

## Saying there are 0 arguments when I have 2? Trying to open a CSV file to write into

ruby,file,csv,dir
I'm trying to read from a CSV file and codify people into groups using an equation. I append the name of their group they fall into to the end of the array that their row creates. Then I write it to a new file so I don't overwrite the original...

## Get X days out of an Array

ruby,ruby-on-rails-4
I have an array filled with Datetime objects: [Mon, 22 Jun 2015, Tue, 23 Jun 2015, Wed, 24 Jun 2015, Thu, 25 Jun 2015, Fri, 26 Jun 2015, Sat, 27 Jun 2015, Sun, 28 Jun 2015] I know how to select what I want from the array ex: week.select{|x|x.monday? ||...

## Can't map a range of dates in Ruby/Rails

ruby-on-rails,ruby
I'm trying to map a range of dates and pass them to my view as an array, as follows: from, to = Date.parse("2014-01-01"), Date.yesterday date_range = (from..to) @mapped_dates = date_range.map {|date| date.strftime("%b %e")} I reference them in some JS in my view as follows: dateLabels = <%= raw @mapped_dates.to_json %>;...

## Ruby gsub group parameters do not work when preceded by escaped slashes

ruby,regex
I am trying to perform a trivial substitution, that in any other language I have come across, work as per the documentation. However, my substitution fails for some reason. The documentation examples list: "hello".gsub(/[aeiou]/, '*') #=> "h*ll*" "hello".gsub(/([aeiou])/, '<\1>') #=> "h<e>ll<o>" "hello".gsub(/./) {|s| s.ord.to_s + ' '} #=> "104 101...

## Error when trying to install app with mysql2 gem

mysql,ruby-on-rails,ruby,ruby-on-rails-4
Im trying to install an open source rails 3.2.21 application that uses the mysql2 gem, but when i try and run the bundle commant I get the following error: Fetching: mysql2-0.3.18.gem (100%) Building native extensions. This could take a while... p ERROR: Error installing mysql2: ERROR: Failed to build gem...

## Allowing some enabled and disabled option on collection_select

ruby-on-rails,ruby
I am trying to populate a dropdown box on a view that has all the states. This works just fine: <%= f.collection_select :state_id, @states, :id, :name %> Now, I need to make the following: Some states are going to be disabled for choosing, but they still have to appear on...

## Keep leading zeroes when converting string to integer

ruby
For no particular reason, I am trying to add a #reverse method to the Integer class: class Integer def reverse self.to_s.reverse.to_i end end puts 1337.reverse # => 7331 puts 1000.reverse # => 1 This works fine except for numbers ending in a 0, as shown when 1000.reverse returns 1 rather...

## is there an equivalent of the ruby any method in javascript?

javascript,arrays,ruby,iteration
Is there an equivalent of ruby's any method for arrays but in javascript? I'm looking for something like this: arr = ['foo','bar','fizz', 'buzz'] arr.any? { |w| w.include? 'z' } #=> true I can get a similar effect with javascript's forEach method but it requires iterating through the entire array rather...

## Map with accumulator on an array

ruby,inject
I'm looking to create a method for Enumerable that does map and inject at the same time. For example, calling it map_with_accumulator, [1,2,3,4].map_with_accumulator(:+) # => [1, 3, 6, 10] or for strings ['a','b','c','d'].map_with_accumulator {|acc,el| acc + '_' + el} # => ['a','a_b','a_b_c','a_b_c_d'] I fail to get a solution working. I...

## Stack level too deep because recursion

I have a model named Tweet. The columns of the Tweet model are: -id -content -user_id -picture -group -original_tweet_id Every tweet can have one or multiple retweets. The relation happens with the help of original_tweet_id. All the tweets have original_tweet_id nil , whilst the retweets contain the id of the...

## Rails shared controller actions

ruby-on-rails,ruby,ruby-on-rails-4
I am having trouble building a controller concern. I would like the concern to extend the classes available actions. Given I have the controller 'SamplesController' class SamplesController < ApplicationController include Searchable perform_search_on(Sample, handle: [ClothingType, Company, Collection, Color]) end I include the module 'Searchable' module Searchable extend ActiveSupport::Concern module ClassMethods def...

## Ruby access words in string

ruby
I don't understand the best method to access a certain word by it's number in a string. I tried using [] to access a word but instead it returns letter. puts s # => I went for a walk puts s[3] # => w ...

## Get the actual value of a boolean attribute

ruby,page-object-gem,rspec3,rspec-expectations
I have the span: <span disabled="disabled">Edit Member</span> When I try to get the value of the disabled attribute: page.in_iframe(:id => 'MembersAreaFrame') do |frame| expect(page.span_element(:xpath => "//span[text()='Edit Member']", :frame => frame).attribute('disabled')).to eq("disabled") end I get: expected: "disabled" got: "true" How do I get the value of specified attribute instead of a...

## What is Rack::Utils.multipart_part_limit within Rails and what function does it perform?

ruby-on-rails,ruby,rack,multipart
Rack::Utils.multipart_part_limit is set to 128 by default. What purpose does the value have and what effect does it have within the Rails system?...

## How to get return value from a forked / spawned process in Ruby?

ruby,process,output,fork,spawn
My simple test program: pid = Process.spawn("sleep 10;date") How can I place the output (eg stdout) of the "date" command in a variable when it is available? I don't want to use a file for the data exchange....

## String#scan not capturing all occurences

ruby,regex
I'm facing a very strange behaviour with ruby String#scan method return. I have this code below and I can't find out why "scan" doesn't return 2 elements. str = "10011011001" regexp = "0110" p str.scan(/(#{regexp})/) ==> [["0110"]] String "str" clearly contains 2 occurences of pattern "0110". I want to fetch...