matrix,stata,correlation , Pairwise correlations over rolling periods ignoring double calculations

Pairwise correlations over rolling periods ignoring double calculations


Tag: matrix,stata,correlation

I am trying to compute pairwise correlations over rolling windows for n= 40 variables where all rolled pairwise correlations for 2 given variables are saved in a new variable.

My dataset has the following structure:

Date        V1   V2   V3  . . .
01/01/2009  0.3  0.6  0.5
02/01/2009  0.1  0.5  0.2 
03/01/2009  0.7  0.1  0.1

The following code does the trick. However, it fails to ignore the lower diagonal of the correlation matrix (it estimates twice the correlation between vars + once the 1s on the diagonal):

ds (Date), not
gl vars `r(varlist)'

local i = 0
local j = 0

foreach current_variable1 in $vars {
local i = `i'+1 
 foreach current_variable2 in $vars {
local j = `j'+1 
    if (`j' > `i') {
     mvcorr `current_variable1' `current_variable2', ///
     generate(corr_`current_variable1'_`current_variable2') window(60) force


In particular the if condition on (j > i) intended to capture the doubled calculation of each pairwise correlation fails to do so. Including the if condition in the mvcorr yields only the error message 'no observations'.

What could be the solution for this issue?


@William Lisowski debugged your code in a comment, but you can simplify the whole procedure.

Create the tuples beforehand using the user-written command tuples (ssc install tuples).

set more off

*----- example data -----

sysuse auto
keep mpg weight price

gen time = _n
tsset time

*----- what you want -----

tuples price mpg weight, min(2) max(2)

forvalues i = 1/`ntuples' {

    local v1 : word 1 of `tuple`i''
    local v2 : word 2 of `tuple`i''

    mvcorr `tuple`i'', generate(`v1'_`v2') window(10) force


@William's advice leads to the same result:

local i = 0
local j = 0

foreach v1 of varlist `vars' {

    local i = `i' + 1 
    foreach v2 of varlist `vars' {

        local j = `j' + 1 
        if (`j' > `i') {
            mvcorr `v1' `v2', ///
            generate(`v1'_`v2') window(10) force


    local j = 0



Clustering a large, very sparse, binary matrix in R

I have a large, sparse binary matrix (roughly 39,000 x 14,000; most rows have only a single "1" entry). I'd like to cluster similar rows together, but my initial plan takes too long to complete: d <- dist(inputMatrix, method="binary") hc <- hclust(d, method="complete") The first step doesn't finish, so I'm...

Numpy and dot products of multiple vector pairs: how can it be done?

I want to get dot product of N vector pairs (a_vec[i, :], b_vec[i, :]). a_vec has shape [N, 3], bvec has the same shape (N 3D vectors). I know that it can be easily done in cycle via function. But cannot it be done somehow simpler and faster?...

Sum of n-1 columns of a double matrix in MATLAB

I have a 1*3000 double matrix. I am doing some calculation and I need to calculate sum of n-1 columns in each step. For example, if I have this matrix: A = [1 2 3 4 5 6] The summation I'm looking for should be like this: B = [0...

Align bars in ciplot

I'm working with the ciplot graphing module for Stata and am encountering a problem with the alignment of bars when I use the by() option. Here's a trivial example demonstrating the issue: webuse citytemp, clear ciplot heatdd cooldd, by(region) horizontal recast(conn) So, the graph shows means and confidence intervals for...

solvePnP: Obtaining the rotation translation matrix

I am trying to image coordinates to 3D coordinates. Using the solvePnP function (in C++)has given me 3X1 rotation matrix and 3X1 translation matrix. But isn't the [R|t] matrix supposed to be 3X4? Any help will be greatly appreciated!...

Matrix Transformation in R - from aggregate output to outer-like matrix

I need to transform the output of an aggregate (mean) into a matrix outer-like style. data(mtcars) aggregate(disp ~ cyl + gear, data = mtcars, FUN = mean ) cyl gear disp 4 3 120.1000 6 3 241.5000 8 3 357.6167 4 4 102.6250 6 4 163.8000 4 5 107.7000 6...

What is an efficient way of multiplying massive polynomials in Java?

I have a function that defines a very long polynomial useful in operations with matrices. The formula multiplies binomials of the form (1+xi*yj) , where i and j are subscripted and correspond to the rows and columns of the matrix in question, respectively. The polynomial that results is the product...

Modify a matrix in ocaml

I want to write a function that is modify every elements inside a matrix. But I have some problem when compiling. Warning 10: this expression should have type unit. I think because all function in ocaml need to return to a value or unit, so if I implement two tasks...

R rounds decimal values in matrix when subsetting

I have a question about subsetting in R. Say I have the following Matrices: Ch1.Amplitude Ch2.Amplitude [1,] 6968.577 9637.309 [2,] 11903.564 11385.656 [3,] 13503.292 9928.314 Ch1.Amplitude Ch2.Amplitude [1,] 11903.564 11385.656 [2,] 2519.582 8042.450 [3,] 9878.749 5899.139 I would like to match row 2 of the first Matrix A with row...

Fortran Seg Fault when assigning Matrices

[UPDATE] the code and a few sentences were changed to reflect a realization explained in my second comment. The code should compile with the line below, however, I have an older gfortran and may not be seeing some errors that you might. gfortran BLU_implementation_copy.f90 -o BLU_implementation_copy.x I'm getting an incredibly...

How to set first column to a constant value of an empty np.zeros numPy matrix?

I'm working on setting some boundary conditions for a water table model, and I am able to set the entire first row to a constant value, but not the entire first column. I am using np.zeros((11,1001)) to make an empty matrix. Does anyone know why I am successful at defining...

find indices x,y of a matrix of specific values in python

I convert a list of integers to a two dimensinal array like this: data = numpy.array( l ) shape = ( 10, 30 ) data = data.reshape( shape ) I try to get the indices x,y of the matrix of the values that are bigger than some threshold and lower...

Translating Stata to R: collapse

Just came across a .do file that I need to translate into R because I don't have a Stata license; my Stata is rusty, so can someone confirm that the code is doing what I think it is? Here's the Stata code: collapse (min) MinPctCollected = PctCollected /// (mean) AvgPctCollected...

Storing columns on disk and reading rows

So I have a C++ code that finds the inverse of a square matrix. The inverse is full and it would be bad to keep it all in memory because I am working with hundreds of thousands of columns. My code generates the columns one by one. After finding the...

How to render a matrix of objects to table with combined cells

i have a matrix of objects: var data = [[ {"value": "1"}, {"value": "2"}, {"value": "2"}, {"value": "4"} ], [ {"value": "1"}, {"value": "2"}, {"value": "2"}, {"value": "4"} ], [ {"value": "1"}, {"value": "3"}, {"value": "3"}, {"value": "5"} ]]; and I would like to render the matirx out as a...

Multipling row and column vector using .* operation

a = 1 2 3 b = 1 2 3 a.*b ans = 1 2 3 2 4 6 3 6 9 I used the .* operator to multiply a row vector and a column vector in Octave to see the results. I dont understand how the answer is obtained....

complexity of generating a sparse matrix

I have a symmetric matrix S(n*n) where approximately 70% of the data are 0. Symmetric matrix I want to convert the symmetric matrix into a sparse matrix with t rows. What would be the time complexity of generating a sparse matrix from the original symmetric matrix? Is it O(n^2), because...

function similar to head() for matrix

I am looking for a function in any package that can print matrix[1:5, 1:5]. head() is good for those that have few columns but not for a large matrix. I know I can create my own function for it but I am wondering if there is a function for it...

Export average value of one variable over 2 other variables

I can use tabout var1 to export a matrix-like object with var1 on one axis, var2 on the other, and the frequencies that each combination of the values var1 and var2 occur. Is it possible to use tabout or some other commmand (preferably native to Stata) to do something...

Inserting One Row Each Time in a Sequence from Matrix into Another Matrix After Every nth Row in Matlab

I have matrix A and matrix B. Matrix A is 100*3. Matrix B is 10*3. I need to insert one row from matrix B each time in a sequence into matrix A after every 10th row. The result would be Matrix A with 110*3. How can I do this in...

Apply a function to all column pairs of two matrices in R

I have two matrices with the same columns and rows names: > metilacion[1:5,1:5] A2BP1 A2M A2ML1 A4GALT AAAS paciente1 0.2804884 0.5816559 1.1814702 -0.6234276 -0.3997400 paciente2 0.5122471 1.2944264 0.5673766 0.4490407 -0.6045510 paciente3 -0.3116356 1.6085049 0.9970350 0.3379215 -0.4787046 paciente4 -0.7220941 0.8771948 2.1445474 -0.5837802 -0.4848246 paciente5 -0.3369999 1.5885716 0.8185654 0.2374583 -0.5698858 > expresion[1:5,1:5]...

Spiral traversal of a matrix - recursive solution in JavaScript

I'm trying to come up with a solution that takes in a matrix like this: [[1,2,3,4], [5,6,7,8], [9,10,11,12], [13,14,15,16]] and returns an array traversing the array as a spiral, so in this example: [1,2,3,4,8,12,16,15,14,13,9,5,6,7,11,10] I'm having trouble getting this recursive solution to work, in which the result array takes the...

I can't fix camera to any object in a 2D Matrix

Problem description: I am trying to develop some 2D game with terrain collision in C# using XNA framework. I have already completed simple terrain generating and object, for example car, can move on this terrain. But now, I get a problem with camera fix to that car. I tried to...

Creating a matrix based on a function in R

I have a symmetric matrix (dimension: 12,000 X 12,000) named A and I want to create another one based on a formula, which depends on the elements position. To explain: I want to create the D matrix (based on the values from A) using the formula: Dij = 1 -...

Analytical solution for Linear Regression using Python vs. Julia

Using example from Andrew Ng's class (finding parameters for Linear Regression using normal equation): With Python: X = np.array([[1, 2104, 5, 1, 45], [1, 1416, 3, 2, 40], [1, 1534, 3, 2, 30], [1, 852, 2, 1, 36]]) y = np.array([[460], [232], [315], [178]]) θ = ((np.linalg.inv( print(θ) Result: [[...

MATLAB removing rows which has duplicates in sequence

I'm trying to remove the rows which has duplicates in sequence. I have only 2 possible values which are 0 and 1. I have nXm which n shows possible number of bits and m is not important for my question. My goal is to find an matrix which is nX(m-a)....

Matlab - Multiply specific entries by a scalar in multidimensional matrix

I'm having problems multiplying specific values within my multidimensional matrix by a scalar. My matrix has the following dimension: size(comDatabe) = 5 10 3 397 10 The third dimension is an x-y-z coordinate frame. Something went wrong and now my y-axis is defined upside down for one subject (#8 out...

Create a Triangular Matrix from a Vector performing sequential operations

I have been trying to solve the following problem. Suppose I have the following vector: aux1<-c(0,0,0,4,5,0,7,0,0,10,11,12) where the numbers represent the number of the row. I want to calculate the distance between the differents elements of this vector fixing the first component, then the second and so on. If the...

A function to calculate cumulative maximum for a double matrix in MATLAB

Is there any function that can calculate cumulative maximum for a double matrix? I have a 1*3000 matrix and I need to calculate the cumulative maximum. For example if the matrix is: A = [8 3 6 7 9 10 6 2 2 3] The cumulative maximum array will be:...

Variable value changing after returning from a function in C

I've been coding for my university where we are working with matrices and I can't find the error in the code that is changing the value of the variable where I save the columns of the matrix. I've tried debugging it and can´t find it, it simply ends the function...

Add a matrix of 2x2 into a vector in c++

I am trying to fill a vector with a matrix of values in c++. I'm not very self confident with this procedure (I don't know well about pointers and I don't know if I need it here) however I am trying this int auxMat[gray.rows][gray.cols]; vector<int> collectionSum; collectionSum.push_back(auxMat); When I try...

Fastest way to copy some rows from one matrix to another in OpenCV

I have a [32678 x 10] matrix (w2c) and I want to copy 24700 rows of it to another matrix(out). I have the index of the rows to be copied in a vector(index). For doing this in matlab I do: out = w2c(index_im,:); It takes approximately 0.002622 seconds. In OpenCV:...

Factorial of a matrix elementwise with Numpy

I'd like to know how to calculate the factorial of a matrix elementwise. For example, import numpy as np mat = np.array([[1,2,3],[2,3,4]]) np.the_function_i_want(mat) would give a matrix mat2 such that mat2[i,j] = mat[i,j]!. I've tried something like np.fromfunction(lambda i,j: np.math.factorial(mat[i,j])) but it passes the entire matrix as argument for np.math.factorial....

Removing rows/columns with only one element from a binary matrix

I'm trying to remove "singletons" from a binary matrix. Here, singletons refers to elements that are the only "1" value in the row AND the column in which they appear. For example, given the following matrix: > matrix(c(0,1,0,1,0,0,1,0,0,1,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1,1,0,0,0,0,1,0,0,0,0,0,1,1), nrow=6) [,1] [,2] [,3] [,4] [,5] [,6] [,7] [1,] 0 1 0...

Extract raw model matrix of random effects from lmer objects (lme4, R)

I have a question regarding the extraction of the (raw) model matrix of random effects from models fitted with lmer (lme4) in R. More specifically, I want to obtain a data frame or a matrix that contains all variables that are involved in random effects terms. The matter is further...

Fast access to matrix as jagged array in C#

I've created a lower triangular distance matrix (because of size issues) as jagged array Note: Distances between objects are symmetric var dm = new double[size][] for (var i = 0; i < size; i++) { dm[i] = new double[i+1]; for (var j = 0; j < i+1; j++) { dm[i][j]...

Join matrices with same values in different vectors in MATLAB

I am trying to merge two matrices that share same values in multiple columns. The following matrices should exemplify my problem and provide a MWE. However, my data is very long size(500000, 4), therefore I am searching an efficient way to merge them. The data consists of option data where...

How do I apply multiple matrices in spriteBatch.Draw()?

I'm making a game in C# and XNA 4.0. In some levels I need a camera matrix to make the level appear to scroll left/right. Additionally, I also need a scaling matrix to scale the graphics when the settings are used to change the window size. Both of my matrices...

Functions with double pointer arrays - find the max in a matrix

Here is the question - Write a program to find the maximum element in a matrix using functions. Function specification: int findMax(int **a, int m, int n) The first argument corresponds to the pointer to the matrix. The second argument corresponds to the number of rows in the matrix. The...

Count of an element in a matrix using Divide & Conquer

I'm starting to learn how to implement divide and conquer algorithms, but I'm having some trouble with this exercise. I have written an algorithm but unfortunately it returns a 0 value. I need to count how many times a number finds in the matrix using D&C. My idea is to...

Find social network components in Stata

[I copied part of the below example from a separate post and changed it to suit my specific needs] pos_1 pos_2 2 4 2 5 1 2 3 9 4 2 9 3 The above is read as person_2 is connected to person_4,...,person_4 is connected to person_2, and person_9 is...

How to solve for matrix in Matlab?

How can I solve , where and and in the least squares sense in matlab? So I'd like to have the minimizing as output....

Computing time complexity of the sparse matrix (2)

I have a data set (D) of (nxd) where n=number of rows and d= number of dimensions, I create a similarity matrix (S)(nxn) by comparing each row of the data set (D) and then convert it into a sparse matrix (tx3) where t is the number of non-zero elements of...

numpy.matrix and .shape - which number is rows, and which is column?

>>> A = np.matrix(np.zeros(2, 3))) >>> A.shape (2, 3) >>> A matrix([[ 0., 0., 0.], [ 0., 0., 0.]]) Does the matrix A have two rows with three zeros or two columns with three zeros?...

Interpolation inside a matrix. Matlab

I have a matrix looks like: 0 0 0 0 0 1 0 0 0 0 0 2 0 0 0 0 0 2 0 0 0 0 0 1 0 1 0 0 0 1 0 4 0 0 0 0 0 3 0 0 6 0 0 4...

Why does this code double transpose a vector - is this a noop?

I have some legacy R code that does: b = t(a) c = t(b) What does this code do? Looks like a noop to me. a is a vector constructed by c(1:20). Edit: bonus points on how to do this better....

How to subset by distinct rows in a data frame or matrix?

Suppose I had the following matrix: matrix(c(1,1,2,1,2,3,2,1,3,2,2,1),ncol=3) Result: [,1] [,2] [,3] [1,] 1 2 3 [2,] 1 3 2 [3,] 2 2 2 [4,] 1 1 1 How can I filter/subset this matrix by whether or not each row has duplicate values? For example, in this case, I would only...

Unable to convert between 1d and 2d array values

I realize this problem and solutions are all over StackOverflow like here however I'm still unable to make this work. Most of the examples say that I just need to multiply the row by the width and add the column meaning the location (4, 3) in a 4x4 square grid...

R - Check different matrices with a possible lag

This issue is quite tricky to explain but I am sure some of you already faced it. So I have two matrix. Matrix 1 (mat 1) and Matrix 2 (mat 2) What I want to do is to record in a third matrix (mat3) the value of mat2, after checking...

How to print the right hemisphere of a square matrix

I am trying to print the right hemisphere of a matrix. If we draw the main and the secondary diagonals in a matrix we see that we got 4 equal parts, the right part is called (in my algorithms textbook) the right hemisphere of a square matrix. For example, in...