python,arrays,numpy,input-sanitization , input/output validation/casting of a numpy calculation

## Question:

Tag: python,arrays,numpy,input-sanitization

This is a situation that happens quite often in my codes. Say I have a function `do_sth(a,b)`, that, only for the sake of this example, simply calculates `a+b`, with `a,b` either 1D `numpy` arrays or scalars. In many occasions, I need the function to broadcast the operation, so that if both `a,b` are 1D arrays, the result will be a 2D array. An example of what I mean follows:

``````do_sth(1,2) -> 3
do_sth([1,2],0) -> array([1, 2])
do_sth(0,[3,4]) -> array([3, 4])
do_sth([1,2],[3,4]) -> array([[4, 5], [5, 6]])
``````

This is a bit similar to how a numpy `ufunc` behaves. A possible implementation follows:

``````from numpy import newaxis, atleast_1d

def do_sth(a, b):
"a,b should be either 1d numpy arrays or scalars"
a, b = map(atleast_1d, [a, b])
# the line below mocks a more complicated calculation
res = a[:, newaxis] + b[newaxis]

conds = [a.size == 1, b.size == 1]

if all(conds):
return res[0, 0]
elif any(conds):
return res.ravel()
else:
return res
``````

As you can see, there's quite a lot of boilerplate. The first question is: is this the right way to do this input/output casting? Is there any reason to not use a decorator to deal with a situation like this? Is there any guideline on the matter?

Moreover, the more complicated calculation, here mocked by the addition, often fails badly if `a` or `b` are numpy arrays with 2D,3D shape for example. I say badly in the sense that the point where the calculation fails is not obvious, or may change with time in different revisions of the code, and it is hard to see the connection between the error and the wrong input shape. I think it is then NOT advisable to put the complicated calculation in a `try/except` block (following python EAFP). In this case, is it correct to check the shape of the 2 arrays at the beginning of the function? Is there any alternative? Is there a numpy function that allows at the same time to convert the input to a numpy array, and also check that the input is compatible with a certain number of dimensions, something like `asarray_withdim(arr,ndim=5)`?

Regarding the use of decorators - I haven't seen much use of decorators in `numpy` code, but I think that's because most of the functionality was developed before decorators become common in Python. If you can make it work, there shouldn't be a any downside (but I'm not an expert with either decorators or `ufunc`).

Non complied numpy functions often have a lot of code that massages the inputs into convenient dimensions. Then they do the core action, followed by final reshaping and type wrapping. They might use functions like np.atleast_2d to ensure there are enough dimensions, and .reshape(-1,1,1) to compress excess dimensions.

`np.tensordot` is an example of one that performs axes transpose plus reshape on the inputs so it can apply the compiled `np.dot`. `np.insert` starts with a number of `ndim` and `isinstance` tests. Special cases are handled early, while the general one is left to the end. `np.einsum` is compiled, but there's a lot of preprocessing being done in C code, before it finally creates an `nditer` object and does the calculation.

# Related:

## Javascript sort array of objects in reverse chronological order

javascript,arrays,sorting
I have an array of objects which holds a list of jobs and I would like to sort them in reverse chronological order as they would appear on a resume for example. I came up with the below solution which 'seems' to work but I was wondering if there is...

## Javascript function to validate contents of an array

javascript,arrays
Here's what is asked: validItems(items) – this function receives a string array of items which are to be for a customer. The function returns an empty string indicating all item codes in the array are valid; otherwise the function returns the first invalid item code in the array. All item...

## Notice: Array to string conversion in “path of php file” on line 64

php,mysql,arrays,oracle
PHP CODE -: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <title>Untitled Document</title> </head> <body> <?php \$rows =0; \$fp = fopen("leave1.csv","r"); if(\$fp){ while(!feof(\$fp)){ \$content = fgets(\$fp); if(\$content) \$rows++; } } fclose(\$fp); //echo \$rows; \$_SESSION['rows'] = \$rows; ?>...

## How does the class_weight parameter in scikit-learn work?

python,scikit-learn
I am having a lot of trouble understanding how the class_weight parameter in scikit-learn's Logistic Regression operates. The Situation I want to use logistic regression to do binary classification on a very unbalanced data set. The classes are labelled 0 (negative) and 1 (positive) and the observed data is in...

## Using counter on array for one value while keeping index of other values

python,collections
After reading the answers on this question How to count the frequency of the elements in a list? I was wondering how to count the frequency of something, and at the same time retreive some extra information, through something like an index. For example a = ['fruit','Item#001'] b = ['fruit','Item#002']...

## pointer to pointer dynamic array in C++

c++,arrays,pointers
I've been having bad luck with dynamic pointers when I want to close it. why the application wrote to memory after end of heap buffer? how can I close my array? int main() { . . int **W; W = new int* [n]; for (int i=1; i <= n; i++)...

## How to use template within Django template?

python,html,django,templates,django-1.4
I have the django template like below: <a href="https://example.com/url{{ mylist.0.id }}" target="_blank"><h1 class="title">{{ mylist.0.title }}</h1></a> <p> {{ mylist.0.text|truncatewords:50 }}<br> ... (the actual template is quite big) It should be used 10 times on the same page, but 'external' html elements are different: <div class="row"> <div class="col-md-12 col-lg-12 block block-color-1"> *django...

## Python recursive function not recursing

python,recursion
I'm trying to solve a puzzle, which is to reverse engineer this code, to get a list of possible passwords, and from those there should be one that 'stands out', and should work function checkPass(password) { var total = 0; var charlist = "abcdefghijklmnopqrstuvwxyz"; for (var i = 0; i...

## Replace nodejs for python?

python,node.js,webserver
i'm working in a HTML5 multiplayer game, and i need a server to sync player's movement, chat, battles, etc. So I'm looking for ways to use python instead nodejs, because i have I have more familiarity with python. The server is simple: var express = require('express'); var app = express();...

## do calculation inside JSONArray in Java

java,arrays,json
I have a simple issue but cannot solve it as I am not very good at algorithm! I have an JSONArray in this form: [{"values":[{"time":1434976493,"value":"50"}, {"time":1434976494,"value":"100"}],"counter":"counter1"}, {"values":[{"time":1434976493,"value":"200"}, {"time":1434976494,"value":"300"}],"counter":"counter2"}, {"values":[{"time":1434976493,"value":"400"}, {"time":1434976494,"value":"600"}],"counter":"total"}] What I want to do is to get the integer value for counter 1 and counter 2 and then divide...

## Identify that a string could be a datetime object

python,regex,algorithm,python-2.7,datetime
If I knew the format in which a string represents date-time information, then I can easily use datetime.datetime.strptime(s, fmt). However, without knowing the format of the string beforehand, would it be possible to determine whether a given string contains something that could be parsed as a datetime object with the...

## Python - Opening and changing large text files

python,replace,out-of-memory,large-files
I have a ~600MB Roblox type .mesh file, which reads like a text file in any text editor. I have the following code below: mesh = open("file.mesh", "r").read() mesh = mesh.replace("[", "{").replace("]", "}").replace("}{", "},{") mesh = "{"+mesh+"}" f = open("p2t.txt", "w") f.write(mesh) It returns: Traceback (most recent call last): File...

## Twilio Client Python not Working in IOS Browser

I have created a simple twilio client application to make phone calls from Web Browser to phones. I used a sample Flask app to generate a secure Capability Token and used twilio.min.js library to handle calls from my HTML. The functionality works fine in Computer Browsers ans Android Phone Browsers,...

## How do I read this list and parse it?

python,list
I'm using requests and the output I get from the sites API is a list, I've been stuck trying to parse it to get the data from it. I use r = requests.get(urlas, params=params) r.json() to get the data I want. Here is a snippet of the list [{'relation_type': None,...

## Strange Behavior: Floating Point Error after Appending to List

python,python-2.7,behavior
I am writing a simple function to step through a range with floating step size. To keep the output neat, I wrote a function, correct, that corrects the floating point error that is common after an arithmetic operation. That is to say: correct(0.3999999999) outputs 0.4, correct(0.1000000001) outputs 0.1, etc. Here's...

## Substring of a file

javascript,arrays,substring
I have a file that is structure like this : var file = "a|b|c|d, a|b|c|d, a|b|c|d, a|b|c|d, a|b|c|d"; Now I would extract all letters "c" and "d" of this file and put those letter in array, structure like this: var array = [ [a,b,1], [a,b,2], [a,b,3], [a,b,4], [a,b,5] ]; How...

## Inserting a variable in MongoDB specifying _id field

python,mongodb,pymongo
I want to insert a variable, say, a = {1:2,3:4} into my database with a particular id "56". It is very clear from the docs that I can do the following: db.testcol.insert({"_id": "56", 1:2, 3:4}) However, I cannot figure out any way to insert "a" itself, specifying an id. In...

## How to pass array in rails 4 strong parameters

ruby-on-rails,arrays
I have to pass a array of food_item_ids in my order_controller. Every order will have many food_items. How can I pass these food_items_id as an array in strong parameters. orders_controller.rb def create @order = Order.new(order_params) if @order.save render :json, @order, status:201, location: [:api, @order] else render :json, { errors: @order.errors...