FAQ Database Discussion Community


Identifying consecutive NaN's with pandas

python,pandas,nan
I am reading in a bunch of CSV files (measurement data for water levels over time) to do various analysis and visualizations on them. Due to various reasons beyond my control, these time series often have missing data, so I do two things: I count them in total with Rlength=len(RainD)...

When/Why does Oracle adds NaN to a row in a database table

sql,oracle,nan
I know that NaN stands for Not a Number. But, I have trouble understanding when and why Oracle adds this to a row. Is it when it encounters a value less than 0 like a negative number or when its a garbage value....

JavaScript function is returning NaN

javascript,function,module,nan
I have a function and it is supposed to return a number but instead it is returning NaN. Here is the code function priceCalc(cust, transit, passType){ var price = 0; if (passType === "monthly"){ if (cust === "student" || cust === "elderly"){ price = transit.monthly / 2; } else if...

Remove chosen lines in an np.array

python,arrays,numpy,nan,genfromtxt
I have a selection of values coming from an experiment and I want to drop some of the lines with respect to other lines. Meaning: I measure a field, a polarization and an error of the polarization. Now the machine doing this measurement sometimes does not write values in some...

REPLACE Nan in SSRS

reporting-services,nan
I have the following issue in SSRS when I'm doing the following calculations. For the columns under Total Number - I'm doing the following - =SUM(IIF(Fields!CallbackStatus.Value="Completed",1,0)) and =SUM(IIF(Fields!CallbackStatus.Value="Outstanding",1,0)) In this particular example - the Total number of Callbacks is 0. Why by using the above expression would the below not...

Return Null in a query, instead of NaN

sql,postgresql,replace,nan
in PostgreSQL, I would like a query to return Null, or empty value, instead of NaN (this NaN was inserted by python's pandas to fill an empty value). Exemple: Select name, age From "People" I would like to get: John 24 Emily Laura 50 Instead of: John 24 Emily NaN...

Set values in numpy array to NaN by index

python,arrays,numpy,nan
I want to set specific values in a numpy array to NaN (to exclude them from a row-wise mean calculation). I tried import numpy x = numpy.array([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 0, 0, 0, 0, 0, 0, 0, 1, 0]]) cutoff = [5, 7]...

numpy: difference between NaN and masked array

python,numpy,nan
In numpy there are two ways to mark missing values: I can either use a NaN or a masked array. I understand that using NaNs is (potentially) faster while masked array offers more functionality (which?). I guess my question is if/ when should I use one over the other? What...

Operations that interpret non-signaling NaNs as missing data points in Eigen

nan,eigen,convolution
In numpy, there are functions like nanmean() that treat quiet NaNs as missing data, i.e. sum only over finite elements and divide by the number of finite elements. This produces quite clean and readable code. Are there counterparts of such functions in Eigen? Currently I need to implement a masked...

Dynamic float in form changes to NaN when form submitted

jquery,nan,dynamic-values
I have a form where a user can select services they want with different fees involved. When they select a service I am adding up the total payable at the bottom using jQuery which all works fine, and its only there to show the user how much they have to...

Python new column based on NaN in other columns

python,pandas,boolean,nan,nonetype
I'm quite new to Python and this is my first ever question so please be gentle with me! I have tried out answers to other similar questions but am still quite stuck. I am using Pandas and I have a dataframe which is a merge from multiple different SQL tables...

Elegant way to create empty pandas DataFrame with NaN of type float

python,numpy,pandas,nan
I want to create a Pandas DataFrame filled with NaNs. During my research I found an answer: import pandas as pd df = pd.DataFrame(index=range(0,4),columns=['A']) This code results in a DataFrame filled with NaNs of type "object". So they cannot be used later on for example with the interpolate() method. Therefore,...

Eigen c++ NaNs disappear when I try to print them

c++,nan,eigen,ofstream
I have written a code using Eigen in c++ which returns some NaNs and the whole thing blows up. When I try to print out variables (with ofstream) and trace back to where the NaN originated, suddenly the NaNs don't appear and the calculation completes successfully. Does anybody have any...

Write a user defined fillna function in pandas dataframe to fill np.nan different values with conditions

python,pandas,dataframes,user-defined-functions,nan
Considering the following pandas dataframe: import pandas as pd change = [0.475, 0.625, 0.1, 0.2, -0.1, -0.75, 0.1, -0.1, 0.2, -0.2] position = [1.0, 1.0, nan, nan, nan, -1.0, nan, nan, nan, nan] date = ['20150101', '20150102', '20150103', '20150104', '20150105', '20150106', '20150107', '20150108', '20150109', '20150110'] pd.DataFrame({'date': date, 'position': position, 'change':...

R: Handling NaN values in t.test

r,nan
I have many vectors on which I should perform one t-test for each. each of these vectors contain data relative to two groups, but some values are missing. Fo eaxmple, one of such vectors may be: A <- c(1, NaN, 1.2, 1.5, 5.6, 5.1, 4.8, 4.7) where the factors are...

Count number of Double.NaN values in scala Array

arrays,scala,nan
For example: scala> val my_array = Array(4,5,Double.NaN,6,5,6, Double.NaN) my_array: Array[Double] = Array(4.0, 5.0, NaN, 6.0, 5.0, 6.0, NaN) scala> my_array.count(_ == Double.NaN) res13: Int = 0 I understand that two Double.NaN are not equal to each other scala> Double.NaN == Double.NaN res14: Boolean = false and therefore, I get the...

Replace -inf, NaN and NA values with zero in a dataset in R

r,time-series,nan,zoo
I am trying to run some trading strategies in R. I have downloaded some stock prices and calculated returns. The new return dataset has a number of -inf, NaN, and NA values. I am reproducing a row of the dataset (log_ret). Its a zoo dataset. library(zoo) log_ret <- structure( c(0.234,-0.012,-Inf,NaN,0.454,Inf),...

ROS - Getting nan values with navsat_transform_node from robot_localization package

localization,transform,nan,ros
I need to fuse gps, imu and odometry data, so I started to test robot_localization package. I’m publishing valid mock messages sensor_msgs/Imu, and nav_msgs/Odometry for the inputs of ekf_localization_node, then I’m feeding the input of navsat_transform_node with the odometry message from the output of ekf_localization_node and a mock sensor_msgs/NavSatFix message....

Filter image that contains NaNs in Matlab?

matlab,filtering,nan
I have a 2d array (doubles) representing some data, and it has a bunch of NaNs in it. The contour plot of the data looks like this: All of the white spaces are NaNs, the gray diamond is there for reference, and the filled contour shows the shape of my...

What is the “char-sequence” argument to NaN generating functions for?

c,standards,nan,c99
Aside from the NAN macro, C99 has two ways to generate a NaN value for a floating point number, the nanf(const char *tagp) function and strtof("NAN(char-sequence)"). Both of these methods of generating a NaN take an optional string argument (*tagp in nanf() and the char-sequence in the strtof method). What...

How to logically replace a NaN value with previous non-NaN value in Matlab?

matlab,nan
I have a matrix data, that I am saving into data_Buffer. All values <=-999 in data are replaced by NaN and stored in data_Buffer. This is my code: fid = fopen('D:\Data\'); data = fread(fid,'*int16');% 21 x 200 matrix fclose(fid); data(data <= -999) = nan; s = size(data,2); data_Buffer(:,k:k+s-1) = data;%...

Calculate dataframe mean by skipping certain values in Python / Pandas

python,pandas,nan,mean
I need to calculate the mean of the first column of the dataframe and I can do that using the mean() method. The problem: Sometimes, there are -9999 values in the data denoting missing observations. I know that NaN values are inherently skipped when calculating the mean in Pandas, but...

my DataFrame has NaN values but should not

python,pandas,nan
I cannot seem to access the first row of data (non-index) that I have, all others are fine: df = pd.read_csv('stock_conf_GT_50.csv') df.head() Data looks fine here: close eqId date IntDate expiry delta ivMid conf 0 37.380005 7 2008-01-02 39447 1 50 0.3850 0.8663 1 37.380005 7 2008-01-02 39447 1 90...

Python NaN JSON encoder

python,json,numpy,nan
The default behavior for the JSON encoder is to convert NaNs to 'NaN', e.g. json.dumps(np.NaN) results in 'NaN'. How can I change this 'NaN' value to 'null'? I have tried to subclass the JSONEncoder and implement the default() method as follows: from json import JSONEncoder, dumps import numpy as np...

why does isNaN always return true

javascript,nan,parseint
Having some trouble trying to validate strings that are numbers in JS. isNaN in particular seems to be misbehaving. <!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <title>No Quirks!</title> </head> <body> <h1>JavaScript Test</h1> <button onclick="myNanIsTrue()">Is it a number?</button> <p id="writeToHTML"></p> <script type="text/javascript"> function myNanIsTrue() { var g = parseInt("40000000") + "<br...

python: eliminate positions of nan in multiple lists

python,list,nan,idl-programming-language
I am coming from IDL and trying to find and eliminate the NaNs in two lists of data. lets say there is a NaN in position 5 for list A but not list B. I need position 5 to be removed in both lists. Like so... A = [1, NaN,...

Array returning NaN

javascript,arrays,nan
I'm having an issue assigning values to an array and when I later access that array slot, it returns NaN. First, I declare the arrays like: var oldTherms = []; var newTherms = []; var oldInputTherms; var newInputTherms; I'm using a data set of variables to calculate a value to...

Assigning using += gives NaN in javascript

javascript,nan,compound-assignment
Assignment a number to an attribute using the += operator gives me NaN in JavaScript. This code works as expected: > var result = {}; undefined > result['value'] = 10; 10 > result['value'] += 10; 20 But here we get NaN: > var test = {}; undefined > test['value'] +=...

df.fillna(0) command won't replace NaN values with 0

python-2.7,pandas,dataframes,nan
I'm trying to replace the NaN values generated in the code below to 0. I don't understand what the below won't work. It still keeps the NaN values. df_pubs=pd.read_sql("select Conference, Year, count(*) as totalPubs from publications where year>=1991 group by conference, year", db) df_pubs['Conference'] = df_pubs['Conference'].str.encode('utf-8') df_pubs = df_pubs.pivot(index='Conference', columns='Year',...

NaN error for gaussian random distribution

java,random,distribution,nan,gaussian
I get random NaN between my outputs. Random Temp:61.816288952756864 'F Random Temp:NaN 'F Random Temp:NaN 'F Random Temp:63.674960517071916 'F Random Temp:NaN 'F Random Temp:62.581425292320894 'F Random Temp:58.21928355494033 'F Random Temp:60.00714587886456 'F Random Temp:NaN 'F Random Temp:60.62983167247955 'F Random Temp:62.74961457200155 'F Random Temp:58.50731966210792 'F Random Temp:NaN 'F Random Temp:55.20026299393227 'F Here...

cut vector according to NaN values

matlab,nan,partition
data_test is a vector that is populated by numbers with some NaN. data_test = [NaN, 2, 3, 4, NaN,NaN,NaN, 12 ,44, 34, NaN,5,NaN]; I would like to cut data_test according to the NaNs and create a cell array containing the pieces of data_set in between NaNs. data_cell{1}=[2 3 4]; data_cell{2}=[12...

Getting returned “[object Object] NaN” - Javascript

javascript,canvas,nan,parseint
As the title says, when i execute my code, in the console i get returned "[object Object] NaN" (with quote marks in console) As i am always not so sure what code to send out as their might be other factors involved, I am going to post all my code...

JavaScript - Why does the increment operator modify immutable strings?

javascript,string,immutability,nan
In JavaScript, strings are immutable. That means any operation on them returns a new object. Methods like trim, replace and slice don't modify the existing string. However, I was playing around in jsconsole and found an exception string = "string"; string + 37; => "string37" string; => "string" The original...

Python/Pandas: counting the number of NaN in each row

pandas,count,row,dataframes,nan
I've got a dataset with a big number of rows. Some of the values are NaN, like this: In [91]: df Out[91]: 1 3 1 1 1 1 3 1 1 1 2 3 1 1 1 1 1 NaN NaN NaN 1 3 1 1 1 1 1 1...

gettin NaN instead of a value in my array result

javascript,arrays,nan
I'm creating a function to do some calculus and the result will be displayed on the navigator console using console.log ,but I'm facing a problem with the Array result which succeed on adding the first result to my main Array using arr.push(result) but the rest of the occurrence are displayed...

Making pandas fillna persistent

python,pandas,nan
I am working with some csv datasets that do contain missing data, which is marked as Lücke. I import the datasets as follows: MyData=pd.read_csv('filename.csv',sep=';',skiprows=19,index_col='Date',dayfirst=True,parse_dates=True, na_values='L\xfccke') Which works as expected, turning the Lücke into the more familiar NaN. Looking at the tail of it, where I know that there is a...

Risks re changing NaN responses to zero in Python DataFrame

python,numpy,pandas,nan,data-cleaning
I have a large-ish survey dataset to clean (300 columns, 30000 rows) and the columns are mixed. I'm using Python with pandas and numpy. Am very much in the learner wheels stage using Python. Some of the columns had Y or N answers to questions (and these are filled "Y"...

Aggregate NAs in R

r,aggregate,nan,na
I'm having trouble handling NAs while calculating aggregated means. Please see the following code: tab=data.frame(a=c(1:3,1:3), b=c(1,2,NA,3,NA,NA)) tab a b 1 1 1 2 2 2 3 3 NA 4 1 3 5 2 NA 6 3 NA attach(tab) aggregate(b, by=list(a), data=tab, FUN=mean, na.rm=TRUE) Group.1 x 1 1 2 2 2...

How to column stack arrays ignoring nan in Python?

python,arrays,numpy,pandas,nan
I have data of the form in a text file. Text file entry #x y z 1 1 1 2 4 3 9 4 16 5 25 6 36 7 49 8 64 512 9 81 729 10 100 1000 11 121 12 144 1728 13 169 14 196 15...

Using empty boost::accumulators

c++,boost,nan,boost-accumulators
How to check an empty boost::accumulators acc or not? For example: if (acc.isEmpty())//I don't know what function here return 0; else return boost::accumulators::mean(acc). Because if it's empty, i get NaN for boost::accumulators::mean(acc)....

Javascript value returning not a number (NaN)

javascript,variables,nan
I have a section of code as follows: var percentageTemp =((parseInt(score)/parseInt(guesses))*100); var percentage= percentageTemp.toFixed(2); score and guesses are both set to 0 at the start of the code, and are altered as the user plays the game. The code works as it should, exept if I leave score variable 0,...

SpreadsheetGear SetArray of double when double.NaN is present

excel,nan,spreadsheetgear
I have a simplified test scenario created where I have a spreadsheet with two cells (C2/C3) having an array formula: {=NaNTest()} My simplified CustomFunction is as follows: public class NaNTest : CustomFunctions.Function { public NaNTest() : this( "NaNTest" ) { } public NaNTest( string name ) : base( name, CustomFunctions.Volatility.Invariant,...

JS Script Returns NaN

javascript,jquery,variables,var,nan
I am having trouble with NaN - Not a Number. I have looked at multiple web resources and couple of questions here to try and find a solution but none of them worked. Any Ideas: var cur_lv = 1; var newscount = document.getElementById("newscount").value; var btn_up = document.getElementById("btn_up"); var btn_down =...

Check NAN values java

java,eclipse,nan
It maybe silly question but I need to check if the value of a cell is NAN which means (1/zero) and then convert it to zero I tried the following code for(i=0;i<5;i++){ do some calculation to get cell[i] if(cell[i]==NAN)cells[i]=0;} But this doesn't work because java can't recoginize the variable NAN...

Read Json with NaN into Python and Pandas

python,json,pandas,nan
I understand that NaN is not allowed in JSON files. I usually use import pandas as pd pd.read_json('file.json') to read in JSON into python. Looking through the documentation, I do not see an option to handle that value. I have a JSON file, data.json, that looks like [{"city": "Los Angeles","job":"chef","age":30},...

jquery: Summing values after dynamic delete of content gets NaN

javascript,jquery,nan
here is my js: function sumAllFields() { var priceSum = 0; $(".price").each(function () { var o = $(this).parent().parent().index(); priceSum += Number($("#area" + o).text()) * $("#price" + o).val(); }) $("#sumPrice, #printSumPrice").html(priceSum.toFixed(2)); } And here is relevant html: <tbody id="tableBody"> <tr class="tableRow" id="tableRow0"> <td><input class="idNumber" id="idNumber0" type="number"></td> <td><input class="description" id="description0" type="text"></td> <td...

jQuery getting back NaN and [object Object] from variables

javascript,jquery,forms,function,nan
I have some problems getting my script to work. My script calculates the number of orders and the weight of each order. My form is actually created by javascript when pressed the "New Line" button. I am trying to make it so whenever 3 values (number of items, number of...

How to withdraw rows and columns that are (almost) entirely made up of NaNs?

r,matrix,nan
> matrix(c(c(0, 3.75882e-06, 3.71645e-05, 2.16088e-06, 1.357e-06, 1.19274e-06, NaN, 1.14748e-06, 9.3314e-07), c(3.75882e-06, 0, 3.94165e-05, 3.58464e-06, 3.60392e-06, 3.43881e-06, NaN, 3.39315e-06, 3.17616e-06), c(3.71645e-05, 3.94165e-05, 0, 3.78173e-05, 3.70121e-05, 3.68449e-05, NaN, 3.6798e-05, 3.65591e-05), c(2.16088e-06, 3.58464e-06, 3.78173e-05, 0, 2.00581e-06, 1.84085e-06, NaN, 1.79527e-06, 1.57976e-06), c(1.357e-06, 3.60392e-06, 3.70121e-05, 2.00581e-06, 0, 1.03709e-06, NaN, 9.91615e-07, 7.77135e-07), c(1.19274e-06, 3.43881e-06,...

Getting NaN Instead of Number

javascript,numbers,nan
I am having a small problem with the code below, instead of a number I am getting NaN. What's wrong? for (var i = 0; i<100; i++) { x = i ^ i result = x + result; } var y = result - (result ^ result); console.log(y); ...

Why is an empty DataFrame of dtype=str filled of “n”?

python,string,pandas,nan
I cannot understand why In [2]: pandas.Series(index=range(2), dtype=str) Out[2]: 0 NaN 1 NaN dtype: object but In [3]: pandas.DataFrame(index=range(2), columns=[0], dtype=str) Out[3]: 0 0 n 1 n Why strings with just the letter "n"? Why this difference between Series and DataFrame? And where is this documented?!...

JavaFX LineChart appears blank at high value

java,user-interface,charts,javafx,nan
Just encountered an issue with javafx.scene.chart.LineChart. When populating the chart data with Double values above 5E13 or so the series happen to disapear (see screenshot). Before After Just in case: I'm adding data with the folowing code chart.getData().clear(); chart.getData().add(new XYChart.Series<>("Simulation name", sim.getDataAsList())); It appeared to be that upperBound property of...

Hadoop context.write() output prints NaN with a Text object

java,hadoop,mapreduce,output,nan
This is my first time using Hadoop, and I'm having trouble writing to my output file. When I print the value with a System.out it displays fine, but using context.write(key, value) the value prints as NaN. Example: System.out.println(stockName.toString() + " " + result.toString()); outputs correctly to userlogs: AAPL.csv 0.076543 But...