FAQ Database Discussion Community

## Rolling average pairwise correlation - code doesn't work as expected

r,statistics,time-series,correlation,xts
I have an xts time series object: head(mtrx) ADS.DE.Close ALV.DE.Close BAS.DE.Close BAYN.DE.Close BEI.DE.Close BMW.DE.Close CBK.DE.Close CON.DE.Close DAI.DE.Close 2007-12-28 01:00:00 51.26 147.95 101.41 62.53 53.00 42.35 21.04 86.06 66.50 2008-01-02 01:00:00 50.00 145.92 100.94 61.45 52.39 42.73 20.75 83.76 64.68 2008-01-03 01:00:00 50.09 144.93 101.60 61.71 51.18 42.09 20.48 81.74 62.91...

## Trying to check data frequency with Pandas Series of datetime64 objects

python,pandas,time-series
I have some time series data that can be 1Hz, 10Hz, or 100Hz. the file I load in happens to be 1Hz: In [6]: data = pd.read_csv("ftp.csv") In [7]: data.Time Out[7]: 0 NaN 1 11:30:08 AM 2 11:30:09 AM 3 11:30:10 AM 4 11:30:11 AM 5 11:30:12 AM 6 11:30:13...

## Use Python/Pandas to Match Sample Pairs Yearly Data

python,pandas,match,time-series,multi-index
While this may start out sounding like as statistics question, please bear with me. I have several calcium concentrations from water samples collected at different sampling locations. The water is resampled at some of the stations on a monthly, yearly, every other year basis. I want to measure yearly and...

## Munging Time Series in Excel

vba,excel-vba,time-series
I want to sort data by the date from latest to earliest. My trouble is that the data i have has dates in mm-dd-yyyy text format. I could easily clean this up using Pandas in python but don't know the tools available in excel. Even when I try to change...

## Adding date tick marks to a Matlab plot

matlab,plot,time-series
I have a plot of time series data, and I would like to replace the tick marks of the x-axis (automatically I have the number of the ordered observations) with the date when the value is observed. I would like to have a tick mark every 5 years for example....

## How to create a function and a loop to calculate growth rates of variables in a data frame in R

r,time-series,growth-rate
New to R and Stack Overflow. Suppose I have the following macroeconomic data loaded into a data frame called testdata in R. > testdata date gdp cpi_index rpi_index 21 2013 Q1 409985 125.067 247.4 22 2013 Q2 412620 125.971 249.7 23 2013 Q3 415577 126.352 250.9 24 2013 Q4 417265...

## Python Pandas: Calculations with Two Different Size Dataframes

python,pandas,time-series
I had two dataframes of differing sizes that I would like to do calculations with. The first dataset is a time series. The second dataset are the long-term monthly averages. The first: year month snow_depth 0 1979 1 18.322581 1 1979 2 11.535714 2 1979 3 5.322581 3 1979 4...

## Time Series Oriented IoT Platform

database,rest,time-series,publish-subscribe,iot
I have an embedded "thing" which generates data samples from several sensors at 1kHz. It has a fairly bandwidth constrained 3G connection to the outside world. Does anyone know of a platform which can provide the following (or at least a subset of the following): A Publish/Subscribe interface to send/receive...

## Fitted values in R forecast missing date / time component

r,time-series,forecasting
I've been doing a variety of models in R with time series data (in XTS format) and I keep running into the same issue where there's no date / time component to the fitted values / forecasts and thus I can't graph them on the same graph as the original...

## SciKit-learn for data driven regression of oscillating data

python,time-series,scikit-learn,regression,prediction
Long time lurker first time poster. I have data that roughly follows a y=sin(time) distribution, but also depends on other variables than time. In terms of correlations, since the target y-variable oscillates there is almost zero statistical correlation with time, but y obviously depends very strongly on time. The goal...

## R Nested For Loop for a Sensitivity Analysis

r,for-loop,nested,time-series
I'm fairly new to R and I've been trying for a while to do something, which I assumed to be very simple, but I keep failing at it (unfortunately for me, it doesn't mean it's not simple!). I have defined a function that takes a time series as an input...

## SQL Server Time Series Modelling Huge datacollection

sql,.net,sql-server,time-series
I have to implement data collection for replay for electrical parameters for 100-1000's of devices with at least 20 parameters to monitor. This amounts to huge data collection as it will be based very similar to time series.I have to support resolution for 1 second. thinking about 1 year [365*24*60*60*1000]=31536000000...

## R: faster alternative of period.apply

r,time-series,apply
I have the following data prepared Timestamp Weighted Value SumVal Group 1 1600 800 1 2 1000 1000 2 3 1000 1000 2 4 1000 1000 2 5 800 500 3 6 400 500 3 7 2000 800 4 8 1200 1000 4 I want to calculate for each group...

## Time series forecasting use SVM

python,time-series
I am trying to set-up a python code for forecasting a time-series, using SVM libraries of scikit-learn. My data consists of X values at a day interval for the last one years, and I need to predict y for a month of the next year . Here's what I have...

## Plot monthly Time series from a data frame with daily data

r,plot,time-series,legend
I have a data set for motor vehicle crashes happening daily in NYC from 1 Jan 2014 to 31 Dec 2012. I want to plot time series of the number of injured cyclists, and motorists, monthly in a single plot. My data looks like this: Date Time Location Cyclists injured...

## R: Volatility function that interprets NAs

r,data.frame,time-series,na
I am looking for help with getting a volatility function to work with my dataframe. In the function below, I'm just trying to get price daily log returns for each security (each column in my data is a different security's prices over time), and then calculate an annualized vol. volcalc=...

## Calculate days since last event in R

r,time-series
My question involves how to calculate the number of days since an event last that occurred in R. Below is a minimal example of the data: df <- data.frame(date=as.Date(c("06/07/2000","15/09/2000","15/10/2000","03/01/2001","17/03/2001","23/05/2001","26/08/2001"), "%d/%m/%Y"), event=c(0,0,1,0,1,1,0)) date event 1 2000-07-06 0 2 2000-09-15 0 3 2000-10-15 1 4 2001-01-03 0 5 2001-03-17 1 6 2001-05-23...

## Using Cassandra for time series data

cassandra,time-series,composite-key
I'm on my research for storing logs to Cassandra. The schema for logs would be something like this. EDIT: I've changed the schema in order to make some clarification. CREATE TABLE log_date ( userid bigint, time timeuuid, reason text, item text, price int, count int, PRIMARY KEY ((userid), time) -...

## Matlab's VAR[X] coefficient constraints for vector time series

matlab,time-series
Matlab's VARMAX model allows the user to set flags that determine whether individual linear coefficients are to be estimated. In particular, vgxset accepts an ARsolve parameter containing flags that determine whether individual time series lag coefficients are estimated. The fact that there are individual scalar flags for each scalar lag...

## How to get a column of lists of contract payout pairs (date:amounts)?

r,time-series,dplyr
I'm trying to add a column of list objects to a data.frame of payments built like ID <- c("A", "B", "B", "c", "A", "B", "c", "c", "A", "B") Date = seq(as.Date("2000/07/01"), as.Date("2000/07/10"), "days") Amt <- rnorm(10, 10, 3) E <- data.frame(Date = Date, ID = ID, Amt = Amt) Date...

## Difference between multi year timeseries and it's 'standard year'

python,datetime,numpy,pandas,time-series
Assume I've a timeseries of a certain number of years as in: rng = pd.date_range(start = '2001-01-01',periods = 5113) ts = pd.TimeSeries(np.random.randn(len(rng)), rng) Than I can calculate it's standard year (the average value of each day over all years) by doing: std = ts.groupby([ts.index.month, ts.index.day]).mean() Now I was wondering how...

## R: Handling subsets using dynlm

r,time-series,subset
I want to compute the following two regressions using R: library("dynlm") zooX = zoo(test[, -1]) lmx <- dynlm(d(Euribor3)~d(Ois3)+d(CDS)+d(Vstoxx)+d(log(omo))+d(L(Euribor3, 1)), data=zooX[1:16]) summary(lmx) zooX = zoo(test[, -1]) lmx <- dynlm(d(Euribor3)~d(Ois3)+d(CDS)+d(Vstoxx)+d(log(omo))+d(L(Euribor3, 1)), data=zooX[17:31]) summary(lmx) The only difference between those two models is the subset (the first[1:16] and the second [17:31]). Now these two...

## Pandas Time-Series: Find previous value for each ID based on year and semester

python,pandas,time-series
I realize this is a fairly basic question, but I couldn't find what I'm looking for through searching (partly because I'm not sure how to summarize what I want). In any case: I have a dataframe that has the following columns: * ID (each one represents a specific college course)...

## Matlab: trying to estimate multifractal spectrum from time series by histogram box-counting

matlab,statistics,time-series,histogram,fractals
I am using the approach from this Yale page on fractals: http://classes.yale.edu/fractals/MultiFractals/Moments/TSMoments/TSMoments.html which is also expounded on this set of lecture slides (slide 32): http://multiscale.emsl.pnl.gov/docs/multifractal.pdf The idea is that you get a dataset, and examine it through many histograms with increasing numbers of bars i.e. resolution. Once resolution is high...

## Plotting multiple time series after a groupby in pandas

python,pandas,group-by,time-series
Suppose I made a groupby on the valgdata DataFrame like below: grouped_valgdata = valgdata.groupby(['news_site','dato_uden_tid']).mean() Now I get this: sentiment news_site dato_uden_tid dr.dk 2015-06-15 54.777183 2015-06-16 54.703167 2015-06-17 54.948775 2015-06-18 54.424881 2015-06-19 53.290554 eb.dk 2015-06-15 53.279251 2015-06-16 53.285643 2015-06-17 53.558753 2015-06-18 52.854750 2015-06-19 54.415988 jp.dk 2015-06-15 56.590428 2015-06-16 55.313752 2015-06-17 53.771377...

## Replace list of permutations with getSymbols data in R

r,time-series,permutation,quantmod
I downloaded some stock data: require("quantmod") s <- c("AAPL", "ADBE", "ADI", "ADP", "ADSK") e <- new.env() getSymbols(s, src='yahoo', from='2015-01-10', env = e ) #get closing prices close <- do.call(merge, eapply(e, function(x) Cl(x))) I found all the pairs of symbol names: #find all the pairwise permutations perm<-combn(s,2) perm [,1] [,2] [,3]...

## Remove values which are surrounded by a certain number of NAs

r,time-series,missing-data
I wish to remove values in a time series which are surrounded by blocks of NA of a certain minimal length. Some toy data: x = seq(0,10,length.out = 100) y = sin(x) + rnorm(length(x), mean=0, sd=0.1) y[20:21] = rep(NA, 2) y[50:54] = rep(NA, 5) y[55:59] = seq(-0.1, -0.8, length.out =...

## R: HAC by NeweyWest using dynlm

r,time-series,regression

## ETS multiplicative trend model written in state space form

time-series,forecasting,state-space
I have an ETS(M,Md,N) model and would like to write it in state space form: yt=w(x{t-1})+r(x{t-1})ɛt xt=f(x{t-1})+g(x{t-1})ɛt For additive trend, the state vector xt=(lt,bt)'. But I have no idea how to write the state vector xt for multiplicative trend. Can anyone help please:D...

## seas start year error R

r,time-series
I am currently trying to deseasonalize data for ARIMA models using the U.S. Census Bureau's package seasonal install.packages("seasonal") you can find a modified version of the data set here. also there is a vignette of the package here which gives very straight-forward directions as to how to install and specify...

## Calculating Active dates based on gap length using Pandas Dataframes

python,date,datetime,pandas,time-series
I'm relatively new to pandas, and trying to figure out what the best way of calculating this information is, so any help is much appreciated. Essentially I have a dataframe that looks like so: id activity_date 1 2015-01-01 1 2015-01-02 1 2015-01-03 2 2015-01-02 2 2015-01-05 3 2015-01-10 And I...

## i10test for integration/stationary time series

matlab,time-series
In http://www.mathworks.com/help/econ/examples/time-series-regression-iv-spurious-regression.html, I am examing the use of the i10test for integration/stationary time series. The online help at http://www.mathworks.com/help/econ/i10test.html shows that this is done through a series of hypothesis tests. From my (OK, limited) exposure to hypthothesis testing, there is usually a threshold for the p-value, e.g. the alpha, that...

## Unable to pass xreg values to hts ARIMA forecast

r,time-series,forecasting
I am trying to pass xreg arguments in my forecast but keep running into an error which says: fc=forecast(gy,fmethod="arima",h=days,method="bu",xreg=z,newxreg=fz) Error in as.matrix(newxreg) %*% coefs : non-conformable arguments In addition: Warning message: In cbind(intercept = rep(1, n), xreg) : number of rows of result is not a multiple of vector length...

## Setting limits with scale_x_datetime and time data

r,ggplot2,time-series
I want to set bounds for the x-axis for a plot of time-series data which features only time (no dates). My limits are: lims <- strptime(c("03:00","16:00"), format = "%H:%M") And my ggplot prints fine, but when I add this to scale_x_datetime scale_x_datetime(limits = lims) I get Error: Invalid input: time_trans...

## Fitting a polinomial curve to time series data

r,ggplot2,time-series,lm
I have a time series graph with monthly article frequency as the y axis. The data looks like this: Count.V Date Month Week Year 2637 6 2006-01-02 2006-01-01 2006-01-02 2006-01-01 406 4 2006-01-03 2006-01-01 2006-01-02 2006-01-01 543 4 2006-01-04 2006-01-01 2006-01-02 2006-01-01 998 3 2006-01-05 2006-01-01 2006-01-02 2006-01-01 1400 4...

## Timeline just with years that shall be ordered and just showing per year how many values are available

d3.js,time-series,timeline,timeserieschart
I would like to use a timeline that just shows years. The special about my data is that I just have years. But these years are ordered. So I know the exact order but not any more detail like day or month. So first of all I would like to...