FAQ Database Discussion Community

## Common elements between two lists of lists (intersection of nested lists) [duplicate]

python,python-2.7,numpy,nested-lists
This question already has an answer here: Get intersecting rows across two 2D numpy arrays 3 answers I have two large lists of points in 2D and I want to find their common sublists, if they have some. Both of the lists are quite large and efficiency is an...

## Python 2D Gaussian Fit with NaN Values in Data

python,numpy,scipy,gaussian
I'm very new to Python but I'm trying to produce a 2D Gaussian fit for some data. Specifically, stellar fluxes linked to certain positions in a coordinate system/grid. However not all of the positions in my grid have corresponding flux values. I don't really want to set these values to...

## Identifying the nearest grid point

python,python-2.7,numpy
I have three arrays lat=[15,15.25,15.75,16,....30] long=[91,91.25,91.75,92....102] data= array([[ 0. , 0. , 0. , ..., 0. , 0. , 0. ], [ 0. , 0. , 0. , ..., 0. , 0. , 0. ], [ 0. , 0. , 0. , ..., 0. , 0. , 0. ], ...,...

## python matrix operation with numpy

python,numpy
I have a python array such as: [[1], [2], [3], [4] ] I want to make it to: [ [1 0 0 0], [2 0 0 0 ], [3 0 0 0], [4 0 0 0] ] What is the python way to do this? Suppose I use numpy....

## Array stacking/ concatenation error in python

python,arrays,numpy,concatenation
I am trying to concatenate two arrays: a and b, where a.shape (1460,10) b.shape (1460,) I tried using hstack and concatenate as: np.hstack((a,b)) c=np.concatenate(a,b,0) I am stuck with the error ValueError: all the input arrays must have same number of dimensions Please guide me for concatenation and generating array c...

## Python RuntimeWarning overflow encountered in double_scalars

python,c,numpy,overflow,warnings
I'm trying to port a "C" code to Python, this is what I coded: scale = 1.0 / (rows * cols) RemoveConstantBiasFrom(rarray, scale) zarray = rarray[:] zarray = DCT(zarray, rows, cols) zarray = zarray.flatten() beta = np.dot(rarray, zarray) if iloop == 0: parray = zarray[:] else: btemp = beta /...

## Parse multicolumn string using python

python,regex,numpy,cheminformatics
I'm trying to extract data from the text output of a cheminformatics program called NWChem, I've already extraced the part of the output that I'm interested in(the vibrational modes), here is the string that I have extracted: s = ''' 1 2 3 4 5 6 P.Frequency -0.00 0.00 0.00...

## Filling missing values pandas dataframe

python,numpy,pandas,dataframes
I'm trying to fill missing datavalues in a pandas dataframe based on date column. df.head() col1 col2 col3 date 2014-06-20 3 752 4028 2014-06-21 4 752 4028 2014-06-22 32 752 4028 2014-06-25 44 882 4548 2014-06-26 32 882 4548 I tried the following idx = pd.date_range(df.index[0], df.index[-1]) df = df.reindex(idx).reset_index()...

## Correlate a single time series with a large number of time series

python,numpy,correlation
I have a large number (M) of time series, each with N time points, stored in an MxN matrix. Then I also have a separate time series with N time points that I would like to correlate with all the time series in the matrix. An easy solution is to...

## Saving dictionary of numpy arrays

python,numpy
So I have a DB with a couple of years worth of site data. I am now attempting to use that data for analytics - plotting and sorting of advertising costs by keyword, etc. One of the data grabs from the DB takes minutes to complete. While I could spend...

## Simpler code to combine multiple lists into one array? I actually wish “lists” are still “lists” in that array

python,numpy
I have a question similar to this and that, but the solutions to those is that all lists are "united" with lack of "differentiation". My Python code is like following: y = np.empty([1,len(test)]) count = -1 for feature in test : N = engine.neighbours(np.asarray(feature)) if len(N) != 4096: print "error"...

## what is the best method to extract highly correlated vaiables within the given threshold

python,numpy,pandas,scipy
I have one data frame and pairwise correlation were calculated >>> df1 = pd.read_csv("/home/zebrafish/Desktop/stack.csv") >>> df1.corr() GA PN PC MBP GR AP GA 1.000000 0.070541 0.259937 -0.452661 0.115722 0.268014 PN 0.070541 1.000000 0.512536 0.447831 -0.042238 0.263601 PC 0.259937 0.512536 1.000000 0.331354 -0.254312 0.958877 MBP -0.452661 0.447831 0.331354 1.000000 -0.467683 0.229870...

## How to fit datasets so that they share some (but not all) parameter values

python,numpy,scipy,curve-fitting,data-fitting
Say I want to fit two arrays x_data_one and y_data_one with an exponential function. In order to do that I might use the following code (in which x_data_one and y_data_one are given dummy definitions): import numpy as np from scipy.optimize import curve_fit def power_law(x, a, b, c): return a *...

## Convert numpy matrix to R matrix and keep it that way

python,r,numpy,rpy2
I'd like to convert a numpy matrix to R an matrix. I'm aware that you to do this: from rpy2 import robjects as ro ro.conversion.py2ri = ro.numpy2ri ro.numpy2ri.activate() And then build an R matrix: mat_r = ro.r.matrix(mat_py) But the problem is, whenever I refer to the new matrix in python...

## How can I convert a list to a numpy array for filtering elements?

python,arrays,numpy,where
I have a list of float numbers and I would like to convert it to numpy array so I can use numpy.where() to get indices of elements that are bigger than 0.0 (not zero) I tried this, but with no luck: import numpy as np arr = np.asarray(enumerate(grade_list)) g_indices =...

## numpy, same calculation different result

python,numpy
I want to calculate this fomula. I think the result is A. So I write a python code using numpy. But depending on the computation sequence, result is not A. What brought this on? import numpy as np from numpy import * from numpy.random import * import decimal #generate matrix...

## Multiply high order matrices with numpy

python,numpy,matrix,scipy
I created this toy problem that reflects my much bigger problem: import numpy as np ind = np.ones((3,2,4)) # shape=(3L, 2L, 4L) dist = np.array([[0.1,0.3],[1,2],[0,1]]) # shape=(3L, 2L) ans = np.array([np.dot(dist[i],ind[i]) for i in xrange(dist.shape[0])]) # shape=(3L, 4L) print ans """ prints: [[ 0.4 0.4 0.4 0.4] [ 3. 3....

## Optional parameter to theano function

python,numpy,theano
I have a function f in theano which takes two parameters, one of them optional. When I call the function with the optional parameter being None the check inside f fails. This script reproduces the error: import theano import theano.tensor as T import numpy as np # function setup def...

## How to index List/ numpy array in order to plot the data with matplotlib

python,numpy,matplotlib
I have a function f(x,t) = cos(t)*t + x and i want to display the change of the result over the width x and time t at discretised time steps t_i and discretised width steps x_j. Now I am a while here on SX and feel really embarrassed to only...

## Memory-efficient Benjamini-Hochberg FDR correction using numpy/h5py

python,numpy,statistics,hdf5,h5py
I am trying to calculate a set of FDR-corrected p-values using Benjamini & Hochberg's method. However, the vector I am trying to run this on contains over 10 billion values. Given the amount of data the normal method from statsmodel's multicomp module quickly runs out of memory. Looking at the...

## How do I make a decaying oscilating function in python?

python,numpy,matplotlib,graph,physics
I have a code in python to represent the energy decay in a damped oscilator, it reads like this: def E(wt, Q): return (np.e**(-x/Q))*(1-(1/2*Q)*np.sin(2*x)) x = np.linspace(0,20,1000) y0 = E(x,2) y1 = E(x,4) y2 = E(x,8) y3 = E(x,16) plt.plot(x, y0, 'p', label=r'\$Q=2\$') plt.plot(x, y1, 'r', label=r'\$Q=4\$') plt.plot(x, y2, 'g',...

## Linear programming with scipy.optimize.linprog

python,numpy,scipy
I've just check the simple linear programming problem with scipy.optimize.linprog: 1*x[1] + 2x[2] -> max 1*x[1] + 0*x[2] <= 5 0*x[1] + 1*x[2] <= 5 1*x[1] + 0*x[2] >= 1 0*x[1] + 1*x[2] >= 1 1*x[1] + 1*x[2] <= 6 And got the very strange result, I expected that x[1]...

## Plotting two different arrays of different lengths

python,numpy,matplotlib,plot
I have two arrays. One is the raw signal of length (1000, ) and the other one is the smooth signal of length (100,). I want to visually represent how the smooth signal represents the raw signal. Since these arrays are of different length, I am not able to plot...

## Converting list to array with NumPy asarray method

python,csv,python-3.x,numpy
I try get mean from csv line. I get data from csv in string list, further i convert it to array with numpy. Its work perfect when i try plot some graphics. But when i calculate mean i get some errors with my data. If i use NumPy i get:...

## Removing repeated sub-lists from a list

python,numpy
I have a list as follows: l = [['A', 'C', 'D'], ['B', 'E'], ['A', 'C', 'D'], ['A', 'C', 'D'], ['B', 'E'], ['F']] The result should be: [['A', 'C', 'D'], ['B', 'E'], ['F']] The order of elements is also not important. I tried as: print list(set(l)) Does numpy has better way...

## Insert a numpy array into another without having to worry about length

python,numpy
When doing: import numpy A = numpy.array([1,2,3,4,5,6,7,8,9,10]) B = numpy.array([1,2,3,4,5,6]) A[7:7+len(B)] = B # A[7:7+len(B)] has in fact length 3 ! we get this typical error: ValueError: could not broadcast input array from shape (6) into shape (3) This is 100% normal because A[7:7+len(B)] has length 3, and not a...

## Why is the output of my function in binary?

python,numpy,matplotlib
I wrote a differentiable Heaviside function and vectorised it. However, the output seems to be odd and binary. The code is as follows: import numpy as np import matplotlib.pyplot as plt def heaviside(x, epis): if (x>= epis): y=1 elif (x< -epis): y=0; else: y = 0.5 + x/(2*epis) + np.sin(np.pi*x/epis)/(2*np.pi);...

## Sending live video frame over network in python opencv

python,opencv,numpy
I'm trying to send live video frame that I catch with my camera to a server and process them. I'm usig opencv for image processing and python for the language. Here is my code client_cv.py import cv2 import numpy as np import socket import sys import pickle cap=cv2.VideoCapture(0) clientsocket=socket.socket(socket.AF_INET,socket.SOCK_STREAM) clientsocket.connect(('localhost',8089))...

## mean() of column in pandas DataFrame returning inf: how can I solve this?

python,numpy,pandas,precision
I'm trying to implement some machine learning algorithms, but I'm having some difficulties putting the data together. In the example below, I load a example data-set from UCI, remove lines with missing data (thanks to the help from a previous question), and now I would like to try to normalize...

## represent an index inside a list as x,y in python

python,list,numpy,multidimensional-array
I have a list which contains 1000 integers. The 1000 integers represent 20X50 elements of dimensional array which I read from a file into the list. I need to walk through the list with an indicator in order to find close elements to each other. I want that my indicator...

## Index 3D aray by 2D array

python,numpy
I have a 3D color image im (shape 512 512 3), and a 2D array mask(512 512). I want to annotate this color image by the mask: im = im[mask>threshold] + im[mask<threshold] * 0.2 + (255,0,0) * [mask<threshold]. How do I write this in Python efficiently?...

## Disabling fp-model strict when using ifort with f2py

python,numpy,optimization,fortran,f2py
I'm using f2py to offload performance critical sections of my python code to fortran. I have two fortran compilers, gfortran and ifort. Since the code I run will be on intel processors, ifort is usually faster. But for the most critical part of my code, I observe the opposite, with...

## create ndarray out of c++ pointer

python,c++,pointers,numpy,cython
I created a module in c++ and need to use the results in python. Already wrote a wrapper and it is working with this code a = np.empty([r, hn]) for i in xrange(r): for j in xrange(hn): a[i,j]=self.thisptr.H[i*hn+j] return a The code is working, but I think there should be...

## Subsetting a 2D numpy array

python,numpy,multidimensional-array,subsetting
I have looked into documentations and also other question in here, but seems I have not got the hang of subsetting in numpy arrays yet. I have a numpy array, and for the sake of argument, let it be defined as follows: import numpy as np a = np.arange(100) a.shape...

## What's the fastest way to compare datetime in pandas?

python,python-3.x,numpy,pandas,datetime64
I have two big csv files with different number of rows which I am importing as follows: tdata = pd.read_csv(tfilepath, sep=',', parse_dates=['date_1']) print(tdata.iloc[:, [0,3]]) TBA date_1 0 0 2010-01-04 1 9 2010-01-05 2 0 2010-01-06 3 8 2010-01-07 4 0 2010-01-08 5 0 2010-01-09 pdata = pd.read_csv(pfilepath, sep=',', parse_dates=['date_2']) print(pdata.iloc[:,...

## What are the loc and scale parameters in scipy.stats.maxwell?

python,numpy,scipy,distribution
The maxwell-boltzmann distribution is given by . The scipy.stats.maxwell distribution uses loc and scale parameters to define this distribution. How are the parameters in the two definitions connected? I also would appreciate if someone could tell in general how to determine the relation between parameters in scipy.stats and their usual...

## Which argument is required for np.savetxt to output float data?

python,numpy
I have a little piece of code which is running well, however I am struggling to determine a way of outputting it into a .txt file. Here is the code: with open("Coord") as f: line=f.readline() for line in f: coords=map(float,line.split(" ")) if poly.contains(Point(coords[0],coords[1])): print line The print command works and...

## Can't use scipy stats function on nested list

python,numpy,statistics,scipy,nested-lists
I've been trying to scipy.mstats.zscore a dataset that is intentionally organized into a nested list, and it gives: TypeError: unsupported operand type(s) for /: 'list' and 'long' which probably suggests that scipy.stats doesn't work for nested lists. What can I do about it? Does a for loop affect the nature...

## How to set first column to a constant value of an empty np.zeros numPy matrix?

python,numpy,matrix,modeling
I'm working on setting some boundary conditions for a water table model, and I am able to set the entire first row to a constant value, but not the entire first column. I am using np.zeros((11,1001)) to make an empty matrix. Does anyone know why I am successful at defining...

## Getting index of a certain element in a 2D numpy array

python,numpy,multidimensional-array,indexing,dynamic-arrays
I'm new to python so I need some help with this: I have 2D array of numbers that represent density of a circle of material in a space and I want to locate the centre. So I want to get the index of the numbers representing the diameter and then...

## Error in creating LMDB database file in Python for Caffe

python,numpy,anaconda,caffe,lmdb
I'm trying to create an LMDB data base file in Python to be used with Caffe according to this tutorial. The commands import numpy as np and import caffe run perfectly fine. However, when I try to run import lmdb and import deepdish as dd, I'm getting the following errors:...

## Looping in c vs. python

python,c,arrays,numpy
I have wrapped a C function calculate_something in Python. The function takes 3 floats, does some calculations, and returns 2 floats. I want to use this function on 3 arrays of floats. That is, for each index in the arrays, call calculate_something on the corresponding floats and store the outputs...

## Numerical issue with np.exp()

python,numpy,floating-point
I have the following code x = -10 for i in range(2,10): print i, " | ",np.exp(-x**i) with the following output: 2 | 3.72007597602e-44 3 | inf 4 | 0.0 5 | inf 6 | 0.0 7 | inf 8 | 0.0 9 | inf Why is the results ~0...

## How to produce multiple output files with a single input file using the command 'np.random.normal'?

python,loops,numpy,random,normal-distribution
I have a file with 44,586 lines of data. It is read in using pylab: data = pl.loadtxt("20100101.txt") density = data[:,0] I need to run something like... densities = np.random.normal(density, 30, 1) np.savetxt('1.txt', np.vstack((densities.ravel())).T) ...and create a new file named 1.txt which has all 44,586 lines of my data randomised...

## np.argmax on multidimensional arrays, keeping some indexes fixed

numpy,multidimensional-array,indexing,argmax
I have a collection of 2D narrays, depending on two integer indexes, say p1 and p2, with each matrix of the same shape. Then I need to find, for each pair (p1,p2), the maximum value of the matrix and the indexes of these maxima. A trivial, albeit slow, way to...

## merging 2D slices into 3D volume in python

python,arrays,numpy
Seems to be a rather simple question but actually not that easy with no experience in image handling/processing. I do not just want to stack 2D-arrays on top of each other but create a 3D volume by merging multiple binary 2D slices which are separated by empty space. Example: Define...

## How to use numpy.random.choice in a list of tuples?

python,numpy,random
I need to do a random choice with a given probability for selecting a tuple from a list. EDIT: The probabiliy for each tuple is in probabilit list I do not know forget the parameter replacement, by default is none The same problem using an array instead a list The...

## Splitting columns of a numpy array easily

python,numpy
How can I split an array's columns into three arrays x, y, z without manually writing each of the [:,0],[:,1],[:,2] separately? import numpy as np data = np.array([[1,2,3],[4,5,6],[7,8,9]]) print data [[1 2 3] [4 5 6] [7 8 9]] x, y, z = data[:,0], data[:,1], data[:,2] ## Help me here!...

## numpy.repeat() to create block-diagonal indices?

python,numpy,scipy
I am trying to figure out how to speed up the following Python code. Basically, the code builds the matrix of outter products of a matrix C and stores it as block diagonal sparse matrix. I use numpy.repeat() to build indices into the block diagonal. Profiling the code revealed that...

## Numpy and dot products of multiple vector pairs: how can it be done?

python,numpy,matrix,scipy
I want to get dot product of N vector pairs (a_vec[i, :], b_vec[i, :]). a_vec has shape [N, 3], bvec has the same shape (N 3D vectors). I know that it can be easily done in cycle via numpy.dot function. But cannot it be done somehow simpler and faster?...

## Numpy sum function returns 1.67772e+07

python,arrays,numpy,floating-point,floating-point-precision
I have two big (432*136*136*46) 'numpy.ndarray' H1 and H2 which encompass altitude values corresponding to two simulations. I want to generate an array with 1 when H1 and H2 have the same altitude and 0 when they don't. Then, I want to know how many elements I selected, so I...

## Easy way to collapse trailing dimensions of numpy array?

python,arrays,numpy,multidimensional-array
In Matlab, I can do the following: X = randn(25,25,25); size(X(:,:)) ans = 25 625 I often find myself wanting to quickly collapse the trailing dimensions of an array, and do not know how to do this in numpy. I know I can do this: In [22]: x = np.random.randn(25,25,25)...

## Rebin data and update imshow plot

python,numpy,matplotlib,draw,imshow
I have a large data set I want to be able to "zoom" in on. What I really want is for the data to be rebinned based on the selection and then update the data in the graph. So the graph will show different limits but maintain the same resolution....

## Read One Input File and plot multiple

python,numpy,matplotlib,graph,plot
I am trying to read one input file of below format. Where Col[1] is x axis and Col[2] is y axis and col[3] is some name. I need to plot multiple line graphs for separate names of col[3]. Eg: Name sd with x,y values will have one line graph and...

## array of minimum euclidian distances between all points in array

python,numpy,scipy,distance
I have this numpy array with points, something like [(x1,y1), (x2,y2), (x3,y3), (x4,y4), (x5,y5)] What I would like to do, is to get an array of all minimum distances. So for point 1 (x1, y1), I want the distance of the point closest to it, same for point 2 (x2,y2),...

## Obtaining Legendre polynomial form once Legendre coefficients are determined

python,numpy
I have obtained the coefficients for the Legendre polynomial that best fits my data. Now I am needing to determine the value of that polynomial at each time-step of my data. I need to do this so that I can subtract the fit from my data. I have looked at...

## How avoid error “TypeError: invalid data type for einsum” in Python

python,python-2.7,numpy,pandas,machine-learning
I try to load CSV file to numpy-array and use the array in LogisticRegression etc. Now, I am struggling with error is shown below: import numpy as np import pandas as pd from sklearn import preprocessing from sklearn.linear_model import LogisticRegression dataset = pd.read_csv('../Bookie_test.csv').values X = dataset[1:, 32:34] y = dataset[1:,...

## Scipy ValueError: Total size of new array must be unchanged

python,numpy,scipy
I am currently using Scipy 0.7.2 with Numpy 1.4.1. My Python version is 2.6.6. I have written a simple code to read a coo sparse matrix from a .mtx file as follows: data = scipy.io.mmread('matrix.mtx') On running the code, I got the following error: Traceback (most recent call last): File...

## Numpy dtype for list with mixed data types

python,numpy
I have a list, my_list, with mixed data types that I want to convert into a numpy array. However, I get the error TypeError: expected a readable buffer object. See code below. I've tried to base my code on the NumPy documentation. my_list = [['User_0', '2012-2', 1, 6, 0, 1.0],...

## Python numpy: reshape list into repeating 2D array

python,arrays,numpy
I'm new to python and I have a question about numpy.reshape. I currently have 2 lists of values like this: x = [0,1,2,3] y = [4,5,6,7] And I want them to be in separate 2D arrays, where each item is repeated for the length of the original lists, like this:...

## add value to each element in array python

python,python-2.7,numpy
I have an array like this a= np.arange(4).reshape(2,2) array([[0, 1],[2, 3]]) I want to add a value to each element in the array. I want my result return 4 array like array([[1, 1],[2, 3]]) array([[0, 2],[2, 3]]) array([[0, 1],[3, 3]]) array([[0, 1],[2, 4]]) ...

## Why my mask failed in Python?

python,numpy
My code: #!/usr/bin/python import numpy as np names = np.array(['Bob', 'Joe', 'Will', 'Bob', 'Will', 'Joe', 'Joe']) data = np.random.randn(7, 4) + 0.8 print (data) mask2= ((names != 'Joe') == 7.0) d2 = data[mask2] print (d2) d3 = data[names != 'Joe'] = 7.0 print (d3) Actually,my intention was to get the...

## Convert one dimensional point list in two dimensional np array

python,numpy
I've one dimensional position list of 10x10 grid: [(0, 0), (0, 1), (0, 2), ..., (9, 9)] I would like numpy array like this (list of 10 length list): array([[ (0, 0), (0, 1), (0, 2), (0, 3), (0, 4), (0, 5), (0, 6), (0, 7), (0, 8), (0, 9)...

## Numpy Indexing: Get every second coloumn for each even row

python,numpy,indexing
I want to select in an numpy array all odd off diagonals. Basically the following: a = np.arange(16).reshape(4,4) array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11], [12, 13, 14, 15]]) b = a[::2,1::2] and at the same time a[1::2,::2] array([[ 1, 3], [...

## Plot only one or few rows of a correlation matrix

python,numpy,matplotlib,heatmap,correlation
I have a correlation matrix named corrdata that I calculated using numpy.corrcoef. Then what I do is extract one or a few rows of this matrix, and now just want to plot them instead of the whole matrix. Because the matrix is no longer square, it is not possible to...

## Using OR operator in NumPy Array to Append multiple items

python-2.7,numpy,arcgis,arcpy
I am using a NumPy array to feature class in Esri ArcGIS. I have a tuple of items that I am appending to a NumPy array. One field/column should be able to use multiple values depnding on the proxy input, i.e. if the category is a bulky item then the...

## Numpy reshape array of arrays to 1D

python,arrays,numpy,reshape
How do I get x to become a 1D array? I found it convenient to create x like this, x=np.array([[0,-1,0]*12,[-1,0,0]*4]) print x print len(x) returns array([ [0, -1, 0, 0, -1, 0, 0, -1, 0, 0, -1, 0, 0, -1, 0, 0, -1, 0, 0, -1, 0, 0, -1, 0,...

## Find Maximum of 3D np.array along Axis = 0

python,arrays,numpy
I have a 3D numpy array that looks like this: X = [[[10 1] [ 2 10] [-5 3]] [[-1 10] [ 0 2] [ 3 10]] [[ 0 3] [10 3] [ 1 2]] [[ 0 2] [ 0 0] [10 0]]] At first I want the maximum along...

## Will numpy.roots() ever return n different floats when a polynomial only has

numpy,scipy
I think the title says it all, but just to be specific, say I have some list of numbers named "coeffs". Assuming the polynomial with said coefficients has exactly k unique roots, will the following code ever set number_of_unique_roots to be a number greater than k? import numpy as np...

## NumPy vs MATLAB

python,matlab,numpy
I've started to use NumPy instead of MATLAB for a lot of things and for most things it appears to be much faster. I've just tried to replicate a code in Python and it is much slower though. I was wondering if someone who knows both could have a look...

## Calculating distances between unique Python array regions?

python,arrays,numpy,scipy,distance
I have a raster with a set of unique ID patches/regions which I've converted into a two-dimensional Python numpy array. I would like to calculate pairwise Euclidean distances between all regions to obtain the minimum distance separating the nearest edges of each raster patch. As the array was originally a...

## Read binary file which has different datatypes

python,numpy,binary-data
Attempting to read a binary file produced in Fortran into Python, which has some integers, some reals and logicals. At the moment I read the first few numbers correctly with: x = np.fromfile(filein, dtype=np.int32, count=-1) firstint= x[1] ... (np is numpy). But the next item is a logical. And later...

## Better image normalization with numpy

python,image,numpy
I already achieved the goal described in the title but I was wondering if there was a more efficient (or generally better) way to do it. First of all let me introduce the problem. I have a set of images of different sizes but with a width/height ratio less than...

## how to calculate a 2D array with numpy mask

python,arrays,numpy
I have a 2 dimension array and based if the value is greater than 0 I want to do a operation (example with x+1). In plain python something like this: a = [[2,5], [4,0], [0,2]] for x in range(3): for y in range(2): if a[x][y] > 0: a[x][y] = a[x][y]...

## Why is my plt.savefig is not working?

python,numpy,matplotlib
I have a simple python code as follows: import numpy as np import matplotlib.pyplot as plt """ Here are the solutions and the plot. """ # Create the axis and plot. plt.axis([0, 10, 0, 10]) axis_x = range(1, 11) grd = [1.1, 2.1, 3.1, 4.1, 5.1, 6.1, 7.1, 8.1, 9.1,...

## OpenCV & Python: quickly superimpose mask over image without overflow

I'd like to superimpose a binary mask over a color image, such that where the mask is "on", the pixel value changes by an amount that I can set. The result should look like this: I am using OpenCV 2.4 and Python 2.7.6. I have a way that works well,...

## multiple iteration of the same list

python,python-2.7,python-3.x,numpy,shapely
I have one list of data as follows: from shapely.geometry import box data = [box(1,2,3,4), box(4,5,6,7), box(1,2,3,4)] sublists = [A,B,C] The list 'data' has following sub-lists: A = box(1,2,3,4) B = box(4,5,6,7) C = box(1,2,3,4) I have to check if sub-lists intersect. If intersect they should put in one tuple;...

## Read a csv with numpy array using pandas

python,csv,numpy,pandas
I have a csv file with 3 columns emotion, pixels, Usage consisting of 35000 rows e.g. 0,70 23 45 178 455,Training. I used pandas.read_csv to read the csv file as pd.read_csv(filename, dtype={'emotion':np.int32, 'pixels':np.int32, 'Usage':str}). When I try the above, it says ValueError: invalid literal for long() with base 10: '70...

## Factorial of a matrix elementwise with Numpy

python,numpy,matrix,factorial
I'd like to know how to calculate the factorial of a matrix elementwise. For example, import numpy as np mat = np.array([[1,2,3],[2,3,4]]) np.the_function_i_want(mat) would give a matrix mat2 such that mat2[i,j] = mat[i,j]!. I've tried something like np.fromfunction(lambda i,j: np.math.factorial(mat[i,j])) but it passes the entire matrix as argument for np.math.factorial....

## Unwrap inner array from a NumPy array

python,arrays,numpy
How can I transform this numpy array: [[[10 10]] [[300 300]] [[10 300]]] Into this one: [[[ 10 10] [300 300] [ 10 300]]] ...

## Insert numpy array to an empty numpy array

python,arrays,numpy,append
I am trying to create an empty numpy array and then insert newly created arrays into than one. It is important for me not to shape the first numpy array and it has to be empty and then I can be able to add new numpy arrays with different sizes...

## Zero out portion of multidim numpy array

python-2.7,image-processing,numpy
I have an numpy array with dimensions (200, 200, 3). It is an RGB image. I also have the (xmin,ymin,xmax,ymax) coordinates of a region of this image that I would like to set to zero. This region should be zero in all three channels. I can of course solve this...

python,numpy,sympy
With numpy, I am able to select an arbitrary set of items from an array with a list of integers: >>> import numpy as np >>> a = np.array([1,2,3]) >>> a[[0,2]] array([1, 3]) The same does not seem to work with sympy matrices, as the code: >>> import sympy as...

## Is it possible to specify the order of levels in Pandas factorize method?

python,numpy,pandas
I am using pandas to factorize an array consisting of two types of strings. I want to make sure that one of the strings "XYZ" is always coded as a 0 and the other string "ABC" is always coded as 1. Is it possible to do this? I looked up...

## numpy.matrix and .shape - which number is rows, and which is column?

python,numpy,matrix
>>> A = np.matrix(np.zeros(2, 3))) >>> A.shape (2, 3) >>> A matrix([[ 0., 0., 0.], [ 0., 0., 0.]]) Does the matrix A have two rows with three zeros or two columns with three zeros?...

## Memory Issue for Array Conversion

python,memory,numpy
If we convert a large array containing 0 and 1 as boolean to another array containing 0 and 1 as float, the size of array would be almost 10 times larger. What is the best way (if any) to handle this issue in python (Numpy) if we need this conversion?

## Creating NxM indices from a Python array

python,numpy,indexing
I would like to take an NxM matrix, for simplicity, we'll use x=np.arange(25).reshape((5,5)) And I would like to create a new matrix, A, in which I can store a node for each element in the first row, its N-direction index in the second row, its M-direction index in the third...

## Finding indices of elements in vector

python,numpy
I have a vector orig which is a p dimensional vector Now, I sampled c elements from this vector (with replacement), lets call it sampled_vec. So basically,sampled_vec has elements from orig Now, I want to find out the indices of these elements (in sampled_vec) from orig. Probably, an example would...

## How to compare list of arrays in python?

python,python-2.7,numpy
I want to compare two variables input_items and temp for equality. To give you an idea of their datatype - print input_items prints - [array([ 50., 1., 0., ..., 0., 0., 0.], dtype=float32), array([ 50., -2., 0., ..., 0., 0., 0.], dtype=float32)] What's the best way to do that in...

## Numpy matrix row comparison

python,performance,numpy,matrix,comparison
The question is more focused on performance of calculation. I have 2 matrix with the same number of columns and different number of rows. One matrix is the 'pattern' whose rows have to be compared separately with the other matrix rows (all rows), then to be able to extract statistical...

## Python np.delete issue

python,numpy
A = np.array([[1,2,3],[3,4,5],[5,6,7]]) X = np.array([[0, 1, 0]]) for i in xrange(np.shape(X)[0]): for j in xrange(np.shape(X)[1]): if X[i,j] == 0.0: A = np.delete(A, (j), axis=0) I am trying to delete j from A if in X there is 0 at index j. I get IndexError: index 2 is out of...

## Fast array manipulation based on element inclusion in binary matrix

python,arrays,numpy
For a large set of randomly distributed points in a 2D lattice, I want to efficiently extract a subarray, which contains only the elements that, approximated as indices, are assigned to non-zero values in a separate 2D binary matrix. Currently, my script is the following: lat_len = 100 # lattice...

## Inconsistency between gaussian_kde and density integral sum

python,numpy,kernel-density
Can one explain why after estimation of kernel density d = gaussian_kde(g[:,1]) And calculation of integral sum of it: x = np.linspace(0, g[:,1].max(), 1500) integral = np.trapz(d(x), x) I got resulting integral sum completely different to 1: print integral Out: 0.55618 ...

## Getting time frequency of number in array in Python?

python,arrays,matlab,numpy
Let's say I have a time series represented in a numpy array, where every 3 seconds, I get a data point. It looks something like this (but with many more data points): z = np.array([1, 2, 1, 2.2, 3, 4.4, 1, 1.2, 2, 3, 2.1, 1.2, 5, 0.5]) I want...

## How to present numpy array into pygame surface?

python,numpy,pygame,pygame-surface
I'm writing a code that part of it is reading an image source and displaying it on the screen for the user to interact with. I also need the sharpened image data. I use the following to read the data and display it in pyGame def image_and_sharpen_array(file_name): #read the image...

## Numerical error when converting array to list

python,arrays,list,numpy,floating-point
I'm doing quite a bit of scientific numerical integration in Python, using Numpy and ode. I use several arrays, and I wanted to turn a 1d array into a list for exporting and easier manipulation. Since then I've found easier and more pythonic methods without resorting to lists, but before...

## Reshaping Pandas data frame (a complex case!)

python-2.7,numpy,pandas
I want to reshape the following data frame: index id numbers 1111 5 58.99 2222 5 75.65 1000 4 66.54 11 4 60.33 143 4 62.31 145 51 30.2 1 7 61.28 The reshaped data frame should be like the following: id 1 2 3 5 58.99 75.65 nan 4...

## How to find the closest corresponding vectors between two coordinate matrices?

python,numpy,vector,euclidean-distance
I have the following problem in Python I need to solve: Given two coordinate matrices (NumPy ndarrays) A and B, find for all coordinate vectors a in A the corresponding coordinate vectors b in B, such that the Euclidean distance ||a-b|| is minimized. The coordinate matrices A and B can...

## Optimizing point - circle distance method

python,opencv,numpy,optimization,cython
I'm implementing a RANSAC algorithm for circle detection in images. I profiled the execution and I get: 13699392 function calls in 799.981 seconds Random listing order was used ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.000 0.000 {time.time} 579810 0.564 0.000 0.564 0.000 {getattr} 289905 2.343 0.000 8.661...

## Numpy has mysteriously stopped working

python,numpy
I was trying to get this ConvexHull function running, and I needed numpy (I think) to get it to work. I'm going to try to get numpy uninstalled and reinstalled, but I'm not sure why/how this happened so that I can prevent it from happening again. While I was running...