FAQ Database Discussion Community

## Combining time series data into a single data frame

r,date,data.frame,time-series
I have multiple data frames that look like this: > head(Standard.df) Count.S Date Month Week Year 552 15 2008-01-01 2008-01-01 2007-12-31 2008-01-01 594 11 2008-01-02 2008-01-01 2007-12-31 2008-01-01 1049 10 2008-01-03 2008-01-01 2007-12-31 2008-01-01 511 12 2008-01-04 2008-01-01 2007-12-31 2008-01-01 717 10 2008-01-06 2008-01-01 2007-12-31 2008-01-01 1744 3 2008-01-07 2008-01-01...

## Fitted values in R forecast missing date / time component

r,time-series,forecasting
I've been doing a variety of models in R with time series data (in XTS format) and I keep running into the same issue where there's no date / time component to the fitted values / forecasts and thus I can't graph them on the same graph as the original...

## Matlab: trying to estimate multifractal spectrum from time series by histogram box-counting

matlab,statistics,time-series,histogram,fractals
I am using the approach from this Yale page on fractals: http://classes.yale.edu/fractals/MultiFractals/Moments/TSMoments/TSMoments.html which is also expounded on this set of lecture slides (slide 32): http://multiscale.emsl.pnl.gov/docs/multifractal.pdf The idea is that you get a dataset, and examine it through many histograms with increasing numbers of bars i.e. resolution. Once resolution is high...

## R function to return start & end date of a time series ts() object?

r,datetime,time-series,forecasting
I have created a list of 300 time series. Now I want to create a training sample(by holding out most recent 3 weeks) for each of the time series to build forecast models. So I want to use window function to subset the time series to skip the most recent...

## small line x-axis for years

r,time-series
I have data covering a time period of over 25 years. In the data set are over 6300 days. I would like to show the years on the x-axix. dates <- as.Date(DOL[,1], "%d.%m.%Y") DOL <- as.xts(DOL[,2], dates) plot(DOL, xaxt = "n", main="SMA", ann = FALSE) axis(1, at=as.POSIXct(dates),labels=format(dates,"%Y"),tick=TRUE) title(ylab = "Value")...

## Auto-regressive model prediction decays to flatline

python,time-series,statsmodels,autoregressive-models
Apologies if this is a simple question/error, but when I try and predict a timeseries using statsmodels.tsa AR the prediction flatlines very quickly past the data I have. This doesn't depend on the order of the model or the length of the data used to fit the AR model. What...

## Matlab Reintroduction of AR and GARCH processes

matlab,for-loop,return,time-series,volatility
I am trying reintroduce autocorrelation and heteroskedasticity to my simulated residuals. My simulated (standardized) residuals have the dimension (horizon, nTrials, nIndices). In order to calculate today's mean / variance (i.e. t), I need to use the last periods mean /variance (i.e. t-1) as an input. This is where I am...

## pandas shift time series with missing values

python,pandas,time-series,shift
I have a times series with some missing entries, that looks like this: date value --------------- 2000 5 2001 10 2003 8 2004 72 2005 12 2007 13 I would like to do create a column for the "previous_value". But I only want it to show values for consecutive years....

## Calculate days since last event in R

r,time-series
My question involves how to calculate the number of days since an event last that occurred in R. Below is a minimal example of the data: df <- data.frame(date=as.Date(c("06/07/2000","15/09/2000","15/10/2000","03/01/2001","17/03/2001","23/05/2001","26/08/2001"), "%d/%m/%Y"), event=c(0,0,1,0,1,1,0)) date event 1 2000-07-06 0 2 2000-09-15 0 3 2000-10-15 1 4 2001-01-03 0 5 2001-03-17 1 6 2001-05-23...

## Matlab's VARMAX regression parameters/coefficients nX & b

matlab,time-series
I'm having a bit of trouble following the explanation of the parameters for vgxset. Being new to the field of time-series is probably part of my problem. The vgxset help page (http://www.mathworks.com/help/econ/vgxset.html) says that its for a generalized model structure, VARMAX, and I assume that I just use a portion...

## Munging Time Series in Excel

vba,excel-vba,time-series
I want to sort data by the date from latest to earliest. My trouble is that the data i have has dates in mm-dd-yyyy text format. I could easily clean this up using Pandas in python but don't know the tools available in excel. Even when I try to change...

## Lag dependent variable [closed]

r,time-series
I want to compute the following time series regression using R: $\Delta y_t=\beta_1 \Delta x_t+\beta_2 \Delta z_t+\beta_3 \Delta m_t+\beta_4 \Delta y_{t−1}$ Since I have not that much experience with R I want to ask if the following R code gives me what I want: y <- ts(diff(YY)) x <- ts(diff(XX))...

## Dates on x-axis, time series

r,date,time-series,as.date
I have data covering a time period of over 25 years and I would like to see the years on the x-axis. dates <- as.Date(Dollar[,1], "%d.%m.%Y") Dollar <- as.xts(Dollar[,2], dates) plot(SWEDOLall, xaxt = "n", main="SMA", ann = FALSE) axis.Date(side = 1, dates, at = labDates, format = "%y", labels =...

## R, lag( ) has inconsistent behavior for xts and ts objects

r,time-series,xts
I would like to take a lag of an xts variable, and the lag() function returns a lag. However, if I use it on a ts variable, it gives a lead. Is this a bug, or working as intended? library('xts') a = as.xts(ts(c(5,3,7,2,4,8,3), start=c(1980,1), freq=4)) cbind(a, lag(a)) # provides lag...

## How to create a time series plot in the style of a horizontal stacked bar plot in r

r,plot,time-series,bar-chart
I would like to create a horizontal ‘stacked bar’ type plot in which date runs along the x-axis and my samples appear as bars on the y-axis. In the simple example below, I have three samples (a, b, c) each containing three values (0, 1, 2). I would like the...

## How to best compress timeseries into a different duration?

r,time-series
I've got a timeseries object defined like so: tser <- ts(cumsum(1 + rnorm(48)), frequency = 12, start = c(2010, 1)) The data looks similar to the below (clipped to only show one year) Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 2010 0.6055677 2.8650543 2.6115597 3.1496051...

## multi-monthly mean with pandas' Series

python,pandas,time-series
I have a sequence of datetime objects and a series of data which spans through several years. A can create a Series object and resample it to group it by months: df=pd.Series(varv,index=dates) multiMmean=df.resample("M", how='mean') print multiMmean This, however, outputs 2005-10-31 172.4 2005-11-30 69.3 2005-12-31 187.6 2006-01-31 126.4 2006-02-28 187.0 2006-03-31...

## Relationship between LinearModel & GeneralizedLinearMixedModel classes

matlab,oop,time-series,linear-regression,superclass
Matlab defines LinearModel and GeneralizedLinearMixedModel classes. Browsing the documentation indicates that either (i) one is derived from the other, or (ii) there is automatic conversion. These are complex objects, and I am just starting to explore them, so I apologize if their relationship is obvious, but what exactly is their...

## Arima.sim issues in R

r,math,statistics,time-series,forecasting
I am working on making a prediction in R using time-series models. I used the auto.arima function to find a model for my dataset (which is a ts object). fit<-auto.arima(data) I can then plot the results of the prediction for the 20 following dates using the forecast function: plot(forecast(fit,h=20)) However...

## Pandas Dataframe Plot

python,pandas,time-series,ipython-notebook
I am trying to plot a time series data. The dataframe looks like this [1]:Index ship_date cost_amount 0 1/8/2010 34276 1 1/8/2010 12375 2 1/8/2011 12343 3 2/9/2011 15435 [2]: df1.plot(figsize(20,5)) I am trying to plot the data but for some reason plot doesn't have x-axis in ascending order. How...

## Measure the STD of RMSE

matlab,time-series,forecasting
I'm working on a time series forecasting problem and I would like to confirm if it makes sense to compute the standard deviation of the root mean squared error. If so, is this the correct way? STD_test = std(sqrt((y_real-y_pred).^2)) Also, imagine that the output of the model is 100, the...

## Matlab's VAR[X] coefficient constraints for vector time series

matlab,time-series
Matlab's VARMAX model allows the user to set flags that determine whether individual linear coefficients are to be estimated. In particular, vgxset accepts an ARsolve parameter containing flags that determine whether individual time series lag coefficients are estimated. The fact that there are individual scalar flags for each scalar lag...

## Direct forecast using epsilion-SVR

matlab,time-series,libsvm,forecasting
Is it possible to predict directly into the future using epsilion-svr? My dataset is a univariate time series and has per line a record in this format: Y(t-W), Y(t-W+1), ..., Y(t), Y(t+PH) W is the number of time steps to consider PH controls how many steps into the future I...

## 150x150 crosstab in stata, showing timeseries movement between categories

time-series,stata,crosstab
I'm a bit over my head here and I hope you can help me, or at least point me in the right direction. I got a massive dataset (5.8 mio observations per year, over 14 years), which deals with individuals' occupation over time. I need to sum up the changes...

## ts.intersect does not work with xts objects

r,time-series,xts
The following produces an error a1 = as.xts(ts(rnorm(20), start=c(1980,1), freq=4)) a2 = as.xts(ts(rnorm(30), start=c(1983,1), freq=4)) a = ts.intersect(a1,a2) Error in .cbind.ts(list(...), .makeNamesTs(...), dframe = dframe, union = FALSE) : no time series supplied The documentation says argument should be two or more univariate or multivariate time series, or objects which...

## Dates with month and day in time series plot in ggplot2 with facet for years

r,ggplot2,time-series
I want to have both month and day in the x-axis of the time series plot when using facet for years in ggplot2. My MWE is below: set.seed(12345) Date <- seq(as.Date("2010/1/1"), as.Date("2014/1/1"), "week") Y <- rnorm(n=length(Date), mean=100, sd=1) df <- data.frame(Date, Y) df$Year <- format(df$Date, "%Y") df$Month <- format(df$Date, "%b")...

## Imputing missing values using ARIMA model

r,time-series,missing-data
I am trying to impute missing values in a time series with an ARIMA model in R. I tried this code but no success. x <- AirPassengers x[90:100] <- NA fit <- auto.arima(x) fitted(fit)[90:100] ## this is giving me NAs plot(x) lines(fitted(fit), col="red") The fitted model is not imputing the...

## Plotting Probability Density Heatmap Over Time in R

r,plot,time-series,kriging
Let's say I have the output of a monte-carlo simulation of one variable over several different iterations (think millions). For each iteration, I have the values of the variable at each point in time (ranging from t=1 to t=365). I would like to produce the following plot: For each point...

## geom_vlines multiple vlines per plot

r,ggplot2,time-series,timeserieschart
How can I get ggplot to produce something similar like library(ggplot2) library(reshape2) library(ecp) synthetic_control.data <- read.table("/path/synthetic_control.data.txt", quote="\"", comment.char="") n <- 2 s <- sample(1:100, n) idx <- c(s, 100+s, 200+s, 300+s, 400+s, 500+s) sample2 <- synthetic_control.data[idx,] df = as.data.frame(t(as.matrix(sample2))) #calculate the change points changeP <- e.divisive(as.matrix(df[1]), k=8, R = 400,...

## Pandas Time-Series: Find previous value for each ID based on year and semester

python,pandas,time-series
I realize this is a fairly basic question, but I couldn't find what I'm looking for through searching (partly because I'm not sure how to summarize what I want). In any case: I have a dataframe that has the following columns: * ID (each one represents a specific college course)...

## timeseries fitted values from trend python

python,pandas,time-series,statsmodels,trend
I have daily stock price data from yahoo finance in a dataframe called price_data. I would like to add a column to this which provides the fitted value from a time series trend of the Adj Close column. Here is the structure of the data I am using: In [41]:...

## Different regression output using dynlm and lm

r,time-series,lm
I ran a regression first using lm and then using dynlm(from the package dynlm). Here is what I did using lm: Euribor3t <- ts(diff(Euribor3)) OIS3t <- ts(diff(Ois3)) x <- ts(diff(Eurepo3-Ois3)) Vstoxxt <- ts(diff(Vstoxx)) CDSt <- ts(diff(CDS)) omo2 <- ts(diff(log(Open.Market.Operations))) l1 <- (lag(Euribor3t, k=-1)) axx <- ts.intersect(Euribor3t, OIS3t, x, Vstoxxt, CDSt,...

## Issue with setting up time series correctly in R

r,time-series
I have been trying to do some basic analysis on some timeseries data. However, I keep getting this error on anything I am trying to do Error in decompose(data_ts, type = c("additive")) : time series has no or less than 2 periods I assume the problem is that I am...

## R: Volatility function that interprets NAs

r,data.frame,time-series,na
I am looking for help with getting a volatility function to work with my dataframe. In the function below, I'm just trying to get price daily log returns for each security (each column in my data is a different security's prices over time), and then calculate an annualized vol. volcalc=...

## Calculate the maximum price fluctuation in a 24 hour window

r,time-series
I have a data frame that contains two columns - time and price. It contains a series of observations for price of a certain item at various times. Here is a sample. > df time price 1 2014-12-12 14:57:15 45.81 2 2014-12-12 14:57:15 45.90 3 2014-12-12 15:00:08 45.76 4 2014-12-12...

## SQL Server Time Series Modelling Huge datacollection

sql,.net,sql-server,time-series
I have to implement data collection for replay for electrical parameters for 100-1000's of devices with at least 20 parameters to monitor. This amounts to huge data collection as it will be based very similar to time series.I have to support resolution for 1 second. thinking about 1 year [365*24*60*60*1000]=31536000000...

## R: HAC by NeweyWest using dynlm

r,time-series,regression

## R seasonal decomposition

r,time-series
I simulate a time series with periodic and linear components and try to use the R stl function to analyze it n = 1000 x = ts(0.1*rnorm(n) + sin(6*pi*(1:n)/n) + (1:n)/n,frequency=n) plot(x) stl(x,"per") but get a message Error in stl(x, "per") : series is not periodic or has less than...

## Cassandra storage internal

cassandra,apache-spark,time-series,cql
I'm trying to understand what exactly happens internally in storage engine level when a row(columns) is inserted in a CQL style table. CREATE TABLE log_date ( userid bigint, time timeuuid, category text, subcategory text, itemid text, count int, price int, PRIMARY KEY ((userid), time) - #1 PRIMARY KEY ((userid), time,...

## How to use parameters from data frame in R and loop through time holding them constant

r,nested,time-series,lapply,sapply
I have a function (weisurv) that has 2 parameters - sc and shp. It is a function through time (t). Time is a sequence, i.e. t<-seq(1:100). weisurv<-function(t,sc,shp){ surv<-exp(-(t/sc)^shp) return(surv) } I have a data frame (df) that contains a list of sc and shp values (like 300+ of them). For...

## how to plot multiple time series in the same graph with customized x axis

javascript,django,python-2.7,highcharts,time-series
I want to plot multiple time series in the same graph using the same xaxis witch a customized one. this is my code: in views.py this is my function: def cdr_weekly_comparison(request): #import ipdb; ipdb.set_trace() acc = cdr_data.find() donnees=[] dt = datetime.now() y = dt.year m = dt.month d = dt.day...

## Using Cassandra for time series data

cassandra,time-series,composite-key
I'm on my research for storing logs to Cassandra. The schema for logs would be something like this. EDIT: I've changed the schema in order to make some clarification. CREATE TABLE log_date ( userid bigint, time timeuuid, reason text, item text, price int, count int, PRIMARY KEY ((userid), time) -...

## Replace list of permutations with getSymbols data in R

r,time-series,permutation,quantmod
I downloaded some stock data: require("quantmod") s <- c("AAPL", "ADBE", "ADI", "ADP", "ADSK") e <- new.env() getSymbols(s, src='yahoo', from='2015-01-10', env = e ) #get closing prices close <- do.call(merge, eapply(e, function(x) Cl(x))) I found all the pairs of symbol names: #find all the pairwise permutations perm<-combn(s,2) perm [,1] [,2] [,3]...

## Timeline just with years that shall be ordered and just showing per year how many values are available

d3.js,time-series,timeline,timeserieschart
I would like to use a timeline that just shows years. The special about my data is that I just have years. But these years are ordered. So I know the exact order but not any more detail like day or month. So first of all I would like to...

## Deleting duplicates in a time series

sql-server,duplicates,time-series,sql-delete
I have a large set of measurements taken every 1 millisecond stored in a SQL Server 2012 table. Whenever there are 3 or more duplicate values in some rows that I would like to delete the middle duplicates. Highlighted values in this image of sample data are the ones that...

## Convert data frame with epoch timestamps to time-series with milliseconds in R

r,time-series,xts,zoo
I have the following data.frame: df <- data.frame(timestamp=c(1428319770511, 1428319797218, 1428319798182, 1428319803327, 1428319808478), session=c("A","A","B","A","A")) I'd like to convert this data frame to a time series and work on time windows shorter than one second. I already tried zoo and xts, but I found it difficult to represent the epoch times as...

## Estimating change of a cyclic boolean variable

time-series,sampling,measurement,probability-theory
We have a boolean variable X which is either true or false and alternates at each time step with a probability p. I.e. if p is 0.2, X would alternate once every 5 time steps on average. We also have a time line and observations of the value of this...

## Calculating the difference in dates in a Pandas GroupBy object

python,pandas,time-series
I have a Pandas DataFrame with the following format: In [0]: df Out[0]: col1 col2 date 0 1 1 2015-01-01 1 1 2 2015-01-09 2 1 3 2015-01-10 3 2 1 2015-02-10 4 2 2 2015-02-10 5 2 3 2015-02-25 In [1]: df.dtypes Out[1]: col1 int64 col2 int64 date datetime64[ns]...

## R: faster alternative of period.apply

r,time-series,apply
I have the following data prepared Timestamp Weighted Value SumVal Group 1 1600 800 1 2 1000 1000 2 3 1000 1000 2 4 1000 1000 2 5 800 500 3 6 400 500 3 7 2000 800 4 8 1200 1000 4 I want to calculate for each group...

## R, how to use pch with time series plot

r,time-series,lattice
I would like to plot a time series (meaning line graph with x axis as time) and specify a plotting character to use. None of the following has worked a1 = as.xts(ts(c(5,3,7,2,4,8,3), start=c(1980,1), freq=4)) library('lattice') xyplot(a1, col="red", pch=2) xyplot(a1, col="red", par.settings = list(superpose.symbol = list(col = 1, pch = 2)),)...

## Creating a running counting variable in R?

r,time-series,running-total
I have a dataset of soccer match results, and I am hoping to learn R by creating a running set of ratings similar to the World Football Elo formula. I am running into trouble with things that seem to be simple in Excel aren't exactly intuitive in R. For instance,...

## Adding date tick marks to a Matlab plot

matlab,plot,time-series
I have a plot of time series data, and I would like to replace the tick marks of the x-axis (automatically I have the number of the ordered observations) with the date when the value is observed. I would like to have a tick mark every 5 years for example....

## Time Series Oriented IoT Platform

database,rest,time-series,publish-subscribe,iot
I have an embedded "thing" which generates data samples from several sensors at 1kHz. It has a fairly bandwidth constrained 3G connection to the outside world. Does anyone know of a platform which can provide the following (or at least a subset of the following): A Publish/Subscribe interface to send/receive...

## Trying to check data frequency with Pandas Series of datetime64 objects

python,pandas,time-series
I have some time series data that can be 1Hz, 10Hz, or 100Hz. the file I load in happens to be 1Hz: In [6]: data = pd.read_csv("ftp.csv") In [7]: data.Time Out[7]: 0 NaN 1 11:30:08 AM 2 11:30:09 AM 3 11:30:10 AM 4 11:30:11 AM 5 11:30:12 AM 6 11:30:13...

## ggplot vertical line with date axis R

r,ggplot2,time-series,vline
I'm having problems ploting vertical lines with ggplot in R. I want to draw a vertical line each Sunday on my time serie: VisitDate VisitMonth VisitYear City Weekday VisitWeek Code_CxF Centre Location 1 2014-05-02 05 2014 Barcelona 05Friday 2014-04-28 CxF_BCN CaixaForum Barcelona Catalunya 2 2014-05-03 05 2014 Barcelona 06Saturday 2014-04-28...

## ETS multiplicative trend model written in state space form

time-series,forecasting,state-space
I have an ETS(M,Md,N) model and would like to write it in state space form: yt=w(x{t-1})+r(x{t-1})ɛt xt=f(x{t-1})+g(x{t-1})ɛt For additive trend, the state vector xt=(lt,bt)'. But I have no idea how to write the state vector xt for multiplicative trend. Can anyone help please:D...

## SensorEvent.timestamp and Location.getElapsedRealtimeNanos() Timestamp Delay Offset

java,android,gps,time-series,kalman-filter
I am currently getting timestamps from accelerometers, magnetometers, and gyroscopes and performing sensor fusion with GPS Location on an android device. I am getting the sensor timestamp using SensorEvent.timestamp and Location.getElapsedRealtimeNanos(). My code is as follows: Sensor Timestamp public void onSensorChanged( SensorEvent event ) { if( event.sensor.getType() == Sensor.TYPE_ACCELEROMETER )...

## Cassandra Time-Series: Allow Filtering, Buckets, or Other

database,cassandra,time-series,data-modeling,cql
I know there are many time-series questions on here but mine does not seem to comfortably fit with the given solutions. I am also new to Cassandra so I might be approaching this with the wrong mindset. Bear with me. I am receiving search data in the form: datetime_searched, term_used,...

## Unable to pass xreg values to hts ARIMA forecast

r,time-series,forecasting
I am trying to pass xreg arguments in my forecast but keep running into an error which says: fc=forecast(gy,fmethod="arima",h=days,method="bu",xreg=z,newxreg=fz) Error in as.matrix(newxreg) %*% coefs : non-conformable arguments In addition: Warning message: In cbind(intercept = rep(1, n), xreg) : number of rows of result is not a multiple of vector length...

## How to aggregate time series documents in mongodb

mongodb,mapreduce,time-series,mongodb-query,nosql-aggregation
i have a mongo sharded cluster where i save data from a virtual machines monitoring system (zabbix ecc). Now I want to get some information from the db, for example the avg memfree in the last 2 days of one vm. I read the tutorials about aggregation and also the...

## Forecasting an Arima Model in R Returning Strange Error

r,time-series,shiny,forecasting
I'm working on building a Shiny App for forecasting time series. One component of this is using ARIMA models to forecast. The user specifies the start and end of the historical data, what p, d, and q they would like to use in the ARIMA model (if they don't want...

## Calculating Active dates based on gap length using Pandas Dataframes

python,date,datetime,pandas,time-series
I'm relatively new to pandas, and trying to figure out what the best way of calculating this information is, so any help is much appreciated. Essentially I have a dataframe that looks like so: id activity_date 1 2015-01-01 1 2015-01-02 1 2015-01-03 2 2015-01-02 2 2015-01-05 3 2015-01-10 And I...

## Enlarge time series and fill with -9999 R

r,merge,time-series
I run am model from 2007-01-01 00 to 2013-12-31 23. Not all my observations are that long, they start later and/or end earlier. In that case I want to fill in -9999 values. I have: [1,] "2003 09 01 01" "0" [2,] "2003 09 01 02" "0" [3,] "2003 09...

## plotting; adding own x-axis does not work

r,plot,time-series
I would like to plot time-series data. To illustrate the dates on the x-asis, I first removed the values on the axis to then add my on axsis with the correct dates: set.seed(1) r <- rnorm(20,0,1) z <- c(1,1,1,1,1,-1,-1,-1,1,-1,1,1,1,-1,1,1,-1,-1,1,-1) data <- as.data.frame(na.omit(cbind(z, r))) series1 <- ts(cumsum(c(1,data[,2]*data[,1]))) series2 <- ts(cumsum(c(1,data[,2]))) d1y...