FAQ Database Discussion Community


How to best compress timeseries into a different duration?

r,time-series
I've got a timeseries object defined like so: tser <- ts(cumsum(1 + rnorm(48)), frequency = 12, start = c(2010, 1)) The data looks similar to the below (clipped to only show one year) Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 2010 0.6055677 2.8650543 2.6115597 3.1496051...

Plotting Probability Density Heatmap Over Time in R

r,plot,time-series,kriging
Let's say I have the output of a monte-carlo simulation of one variable over several different iterations (think millions). For each iteration, I have the values of the variable at each point in time (ranging from t=1 to t=365). I would like to produce the following plot: For each point...

Issue with setting up time series correctly in R

r,time-series
I have been trying to do some basic analysis on some timeseries data. However, I keep getting this error on anything I am trying to do Error in decompose(data_ts, type = c("additive")) : time series has no or less than 2 periods I assume the problem is that I am...

Use Python/Pandas to Match Sample Pairs Yearly Data

python,pandas,match,time-series,multi-index
While this may start out sounding like as statistics question, please bear with me. I have several calcium concentrations from water samples collected at different sampling locations. The water is resampled at some of the stations on a monthly, yearly, every other year basis. I want to measure yearly and...

Matlab: trying to estimate multifractal spectrum from time series by histogram box-counting

matlab,statistics,time-series,histogram,fractals
I am using the approach from this Yale page on fractals: http://classes.yale.edu/fractals/MultiFractals/Moments/TSMoments/TSMoments.html which is also expounded on this set of lecture slides (slide 32): http://multiscale.emsl.pnl.gov/docs/multifractal.pdf The idea is that you get a dataset, and examine it through many histograms with increasing numbers of bars i.e. resolution. Once resolution is high...

SQL Server Time Series Modelling Huge datacollection

sql,.net,sql-server,time-series
I have to implement data collection for replay for electrical parameters for 100-1000's of devices with at least 20 parameters to monitor. This amounts to huge data collection as it will be based very similar to time series.I have to support resolution for 1 second. thinking about 1 year [365*24*60*60*1000]=31536000000...

Rolling average pairwise correlation - code doesn't work as expected

r,statistics,time-series,correlation,xts
I have an xts time series object: head(mtrx) ADS.DE.Close ALV.DE.Close BAS.DE.Close BAYN.DE.Close BEI.DE.Close BMW.DE.Close CBK.DE.Close CON.DE.Close DAI.DE.Close 2007-12-28 01:00:00 51.26 147.95 101.41 62.53 53.00 42.35 21.04 86.06 66.50 2008-01-02 01:00:00 50.00 145.92 100.94 61.45 52.39 42.73 20.75 83.76 64.68 2008-01-03 01:00:00 50.09 144.93 101.60 61.71 51.18 42.09 20.48 81.74 62.91...

R Nested For Loop for a Sensitivity Analysis

r,for-loop,nested,time-series
I'm fairly new to R and I've been trying for a while to do something, which I assumed to be very simple, but I keep failing at it (unfortunately for me, it doesn't mean it's not simple!). I have defined a function that takes a time series as an input...

Fitted values in R forecast missing date / time component

r,time-series,forecasting
I've been doing a variety of models in R with time series data (in XTS format) and I keep running into the same issue where there's no date / time component to the fitted values / forecasts and thus I can't graph them on the same graph as the original...

Plotting multiple time series after a groupby in pandas

python,pandas,group-by,time-series
Suppose I made a groupby on the valgdata DataFrame like below: grouped_valgdata = valgdata.groupby(['news_site','dato_uden_tid']).mean() Now I get this: sentiment news_site dato_uden_tid dr.dk 2015-06-15 54.777183 2015-06-16 54.703167 2015-06-17 54.948775 2015-06-18 54.424881 2015-06-19 53.290554 eb.dk 2015-06-15 53.279251 2015-06-16 53.285643 2015-06-17 53.558753 2015-06-18 52.854750 2015-06-19 54.415988 jp.dk 2015-06-15 56.590428 2015-06-16 55.313752 2015-06-17 53.771377...

R function to return start & end date of a time series ts() object?

r,datetime,time-series,forecasting
I have created a list of 300 time series. Now I want to create a training sample(by holding out most recent 3 weeks) for each of the time series to build forecast models. So I want to use window function to subset the time series to skip the most recent...

seas start year error R

r,time-series
I am currently trying to deseasonalize data for ARIMA models using the U.S. Census Bureau's package seasonal install.packages("seasonal") you can find a modified version of the data set here. also there is a vignette of the package here which gives very straight-forward directions as to how to install and specify...

Combining time series data into a single data frame

r,date,data.frame,time-series
I have multiple data frames that look like this: > head(Standard.df) Count.S Date Month Week Year 552 15 2008-01-01 2008-01-01 2007-12-31 2008-01-01 594 11 2008-01-02 2008-01-01 2007-12-31 2008-01-01 1049 10 2008-01-03 2008-01-01 2007-12-31 2008-01-01 511 12 2008-01-04 2008-01-01 2007-12-31 2008-01-01 717 10 2008-01-06 2008-01-01 2007-12-31 2008-01-01 1744 3 2008-01-07 2008-01-01...

Time Series Oriented IoT Platform

database,rest,time-series,publish-subscribe,iot
I have an embedded "thing" which generates data samples from several sensors at 1kHz. It has a fairly bandwidth constrained 3G connection to the outside world. Does anyone know of a platform which can provide the following (or at least a subset of the following): A Publish/Subscribe interface to send/receive...

R: faster alternative of period.apply

r,time-series,apply
I have the following data prepared Timestamp Weighted Value SumVal Group 1 1600 800 1 2 1000 1000 2 3 1000 1000 2 4 1000 1000 2 5 800 500 3 6 400 500 3 7 2000 800 4 8 1200 1000 4 I want to calculate for each group...

Deleting duplicates in a time series

sql-server,duplicates,time-series,sql-delete
I have a large set of measurements taken every 1 millisecond stored in a SQL Server 2012 table. Whenever there are 3 or more duplicate values in some rows that I would like to delete the middle duplicates. Highlighted values in this image of sample data are the ones that...

Converting time series to data frame, matrix, or table

r,time-series,tapply
With reference to this question: transforming a ts in a data.frame and back I have a list of monthly averages that start in May 2012 and go through May 2015. It looks like this initially: head (AVG_LOSCAT2) month AVG_LOSCAT YEAR MONTH 1 2012-05 5.342066 2012 05 2 2012-06 6.544096 2012...

Matlab Reintroduction of AR and GARCH processes

matlab,for-loop,return,time-series,volatility
I am trying reintroduce autocorrelation and heteroskedasticity to my simulated residuals. My simulated (standardized) residuals have the dimension (horizon, nTrials, nIndices). In order to calculate today's mean / variance (i.e. t), I need to use the last periods mean /variance (i.e. t-1) as an input. This is where I am...

ts.intersect does not work with xts objects

r,time-series,xts
The following produces an error a1 = as.xts(ts(rnorm(20), start=c(1980,1), freq=4)) a2 = as.xts(ts(rnorm(30), start=c(1983,1), freq=4)) a = ts.intersect(a1,a2) Error in .cbind.ts(list(...), .makeNamesTs(...), dframe = dframe, union = FALSE) : no time series supplied The documentation says argument should be two or more univariate or multivariate time series, or objects which...

multi-monthly mean with pandas' Series

python,pandas,time-series
I have a sequence of datetime objects and a series of data which spans through several years. A can create a Series object and resample it to group it by months: df=pd.Series(varv,index=dates) multiMmean=df.resample("M", how='mean') print multiMmean This, however, outputs 2005-10-31 172.4 2005-11-30 69.3 2005-12-31 187.6 2006-01-31 126.4 2006-02-28 187.0 2006-03-31...

ggplot vertical line with date axis R

r,ggplot2,time-series,vline
I'm having problems ploting vertical lines with ggplot in R. I want to draw a vertical line each Sunday on my time serie: VisitDate VisitMonth VisitYear City Weekday VisitWeek Code_CxF Centre Location 1 2014-05-02 05 2014 Barcelona 05Friday 2014-04-28 CxF_BCN CaixaForum Barcelona Catalunya 2 2014-05-03 05 2014 Barcelona 06Saturday 2014-04-28...

plotting; adding own x-axis does not work

r,plot,time-series
I would like to plot time-series data. To illustrate the dates on the x-asis, I first removed the values on the axis to then add my on axsis with the correct dates: set.seed(1) r <- rnorm(20,0,1) z <- c(1,1,1,1,1,-1,-1,-1,1,-1,1,1,1,-1,1,1,-1,-1,1,-1) data <- as.data.frame(na.omit(cbind(z, r))) series1 <- ts(cumsum(c(1,data[,2]*data[,1]))) series2 <- ts(cumsum(c(1,data[,2]))) d1y...

how to convert data frame into time series in R

r,time-series
I have one csv file in which I have 2 closing prices of stock(on daily basis) Dates Bajaj_close Hero_close 3/14/2013 1854.8 1669.1 3/15/2013 1850.3 1684.45 3/18/2013 1812.1 1690.5 3/19/2013 1835.9 1645.6 3/20/2013 1840 1651.15 3/21/2013 1755.3 1623.3 3/22/2013 1820.65 1659.6 3/25/2013 1802.5 1617.7 3/26/2013 1801.25 1571.85 3/28/2013 1799.55 1542 I...

Measure the STD of RMSE

matlab,time-series,forecasting
I'm working on a time series forecasting problem and I would like to confirm if it makes sense to compute the standard deviation of the root mean squared error. If so, is this the correct way? STD_test = std(sqrt((y_real-y_pred).^2)) Also, imagine that the output of the model is 100, the...

Difference between multi year timeseries and it's 'standard year'

python,datetime,numpy,pandas,time-series
Assume I've a timeseries of a certain number of years as in: rng = pd.date_range(start = '2001-01-01',periods = 5113) ts = pd.TimeSeries(np.random.randn(len(rng)), rng) Than I can calculate it's standard year (the average value of each day over all years) by doing: std = ts.groupby([ts.index.month, ts.index.day]).mean() Now I was wondering how...

Convert Daily Data into Weekly in R Week Starts on Saturday

r,data.frame,time-series,weekend
I am having trouble converting daily data into weekly using averages over the week. My Data looks like this: > str(daily_FWIH) 'data.frame': 4371 obs. of 6 variables: $ Date : Date, format: "2013-03-01" "2013-03-02" "2013-03-04" "2013-03-05" ... $ CST.OUC : Factor w/ 6 levels "BVG11","BVG12",..: 1 1 1 1 1...

Unable to pass xreg values to hts ARIMA forecast

r,time-series,forecasting
I am trying to pass xreg arguments in my forecast but keep running into an error which says: fc=forecast(gy,fmethod="arima",h=days,method="bu",xreg=z,newxreg=fz) Error in as.matrix(newxreg) %*% coefs : non-conformable arguments In addition: Warning message: In cbind(intercept = rep(1, n), xreg) : number of rows of result is not a multiple of vector length...

how to plot multiple time series in the same graph with customized x axis

javascript,django,python-2.7,highcharts,time-series
I want to plot multiple time series in the same graph using the same xaxis witch a customized one. this is my code: in views.py this is my function: def cdr_weekly_comparison(request): #import ipdb; ipdb.set_trace() acc = cdr_data.find() donnees=[] dt = datetime.now() y = dt.year m = dt.month d = dt.day...

Matlab's VARMAX regression parameters/coefficients nX & b

matlab,time-series
I'm having a bit of trouble following the explanation of the parameters for vgxset. Being new to the field of time-series is probably part of my problem. The vgxset help page (http://www.mathworks.com/help/econ/vgxset.html) says that its for a generalized model structure, VARMAX, and I assume that I just use a portion...

Arima.sim issues in R

r,math,statistics,time-series,forecasting
I am working on making a prediction in R using time-series models. I used the auto.arima function to find a model for my dataset (which is a ts object). fit<-auto.arima(data) I can then plot the results of the prediction for the 20 following dates using the forecast function: plot(forecast(fit,h=20)) However...

ggplot2: arranging multiple boxplots as a time series

r,ggplot2,time-series,boxplot
I would like to create a multivariate boxplot time series with ggplot2 and I need to have an x axis that positions the boxplots based on their associated dates. I found two posts about this question: one is Time series plot with groups using ggplot2 but the x axis is...

Using dplyr to mutate() contract duration, payment counts, and total payout

r,timestamp,time-series,dataframes,dplyr
A simplified structure of my data is as follows: >ID <- c("A", "B", "B", "C", "A", "B", "C", "C", "A", "B") >Date = seq(as.Date("2000/07/01"), as.Date("2000/07/10"), "days") >Amt <- rnorm(10, 10, 3) >E <- data.frame(Date = Date, ID = ID, Amt = Amt) >E Date ID Amt 1 2000-07-01 A 5.9...

Auto-regressive model prediction decays to flatline

python,time-series,statsmodels,autoregressive-models
Apologies if this is a simple question/error, but when I try and predict a timeseries using statsmodels.tsa AR the prediction flatlines very quickly past the data I have. This doesn't depend on the order of the model or the length of the data used to fit the AR model. What...

R: Handling subsets using dynlm

r,time-series,subset
I want to compute the following two regressions using R: library("dynlm") zooX = zoo(test[, -1]) lmx <- dynlm(d(Euribor3)~d(Ois3)+d(CDS)+d(Vstoxx)+d(log(omo))+d(L(Euribor3, 1)), data=zooX[1:16]) summary(lmx) zooX = zoo(test[, -1]) lmx <- dynlm(d(Euribor3)~d(Ois3)+d(CDS)+d(Vstoxx)+d(log(omo))+d(L(Euribor3, 1)), data=zooX[17:31]) summary(lmx) The only difference between those two models is the subset (the first[1:16] and the second [17:31]). Now these two...

Matlab's VAR[X] coefficient constraints for vector time series

matlab,time-series
Matlab's VARMAX model allows the user to set flags that determine whether individual linear coefficients are to be estimated. In particular, vgxset accepts an ARsolve parameter containing flags that determine whether individual time series lag coefficients are estimated. The fact that there are individual scalar flags for each scalar lag...

R - how to calculate “global” monthly means of a zoo object

r,time-series,mean,zoo
Let's say I have this zoo object: library(zoo) df <- structure(list(date = structure(c(0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334, 365, 396, 424, 455, 485, 516, 546, 577, 608, 638, 669, 699, 730, 761, 790, 821, 851, 882, 912, 943, 974, 1004, 1035, 1065, 1096, 1127,...

Calculate days since last event in R

r,time-series
My question involves how to calculate the number of days since an event last that occurred in R. Below is a minimal example of the data: df <- data.frame(date=as.Date(c("06/07/2000","15/09/2000","15/10/2000","03/01/2001","17/03/2001","23/05/2001","26/08/2001"), "%d/%m/%Y"), event=c(0,0,1,0,1,1,0)) date event 1 2000-07-06 0 2 2000-09-15 0 3 2000-10-15 1 4 2001-01-03 0 5 2001-03-17 1 6 2001-05-23...

vgxset command: Q parameter for resulting model object?

matlab,time-series
Matlab's command for defining a vector time series model is vgxset, the formalism for which can be accessed by the command "doc vgxset". It says that the model parameter Q is "[a]n n-by-n symmetric innovations covariance matrix". No description of what it is for. I assumed that it was the...

How to aggregate time series documents in mongodb

mongodb,mapreduce,time-series,mongodb-query,nosql-aggregation
i have a mongo sharded cluster where i save data from a virtual machines monitoring system (zabbix ecc). Now I want to get some information from the db, for example the avg memfree in the last 2 days of one vm. I read the tutorials about aggregation and also the...

python freezes with many repeated pandas calls

python,pandas,time-series,freeze
I'm trying to compute the variance of a time series for many sampling frequencies (the so called signature plot), I used the resample method looping on a set of frequencies but python stops before completing the task (no errors, just freezed). Here the code is var_list = [timeseries.resample(rule=str(int(freq))+'min',how='first').var() for i...

Cassandra Time-Series: Allow Filtering, Buckets, or Other

database,cassandra,time-series,data-modeling,cql
I know there are many time-series questions on here but mine does not seem to comfortably fit with the given solutions. I am also new to Cassandra so I might be approaching this with the wrong mindset. Bear with me. I am receiving search data in the form: datetime_searched, term_used,...

Cassandra storage internal

cassandra,apache-spark,time-series,cql
I'm trying to understand what exactly happens internally in storage engine level when a row(columns) is inserted in a CQL style table. CREATE TABLE log_date ( userid bigint, time timeuuid, category text, subcategory text, itemid text, count int, price int, PRIMARY KEY ((userid), time) - #1 PRIMARY KEY ((userid), time,...

Calculating Active dates based on gap length using Pandas Dataframes

python,date,datetime,pandas,time-series
I'm relatively new to pandas, and trying to figure out what the best way of calculating this information is, so any help is much appreciated. Essentially I have a dataframe that looks like so: id activity_date 1 2015-01-01 1 2015-01-02 1 2015-01-03 2 2015-01-02 2 2015-01-05 3 2015-01-10 And I...

Fitting a subset model with just one lag, using R package FitAR

r,time-series
I am trying to fit a subset model with only lag 4. In the manual it's written "you must use p=c(0,0,0,4) since p=4 will fit a full AR(4)". I did this. #fit a subset model with just lag 4 Fit=FitAR(p=c(0,0,0,4), lag.max = "default", ARModel = "ARz") However, I get the...

Fitting a polinomial curve to time series data

r,ggplot2,time-series,lm
I have a time series graph with monthly article frequency as the y axis. The data looks like this: Count.V Date Month Week Year 2637 6 2006-01-02 2006-01-01 2006-01-02 2006-01-01 406 4 2006-01-03 2006-01-01 2006-01-02 2006-01-01 543 4 2006-01-04 2006-01-01 2006-01-02 2006-01-01 998 3 2006-01-05 2006-01-01 2006-01-02 2006-01-01 1400 4...

Time series forecasting use SVM

python,time-series
I am trying to set-up a python code for forecasting a time-series, using SVM libraries of scikit-learn. My data consists of X values at a day interval for the last one years, and I need to predict y for a month of the next year . Here's what I have...

150x150 crosstab in stata, showing timeseries movement between categories

time-series,stata,crosstab
I'm a bit over my head here and I hope you can help me, or at least point me in the right direction. I got a massive dataset (5.8 mio observations per year, over 14 years), which deals with individuals' occupation over time. I need to sum up the changes...

SensorEvent.timestamp and Location.getElapsedRealtimeNanos() Timestamp Delay Offset

java,android,gps,time-series,kalman-filter
I am currently getting timestamps from accelerometers, magnetometers, and gyroscopes and performing sensor fusion with GPS Location on an android device. I am getting the sensor timestamp using SensorEvent.timestamp and Location.getElapsedRealtimeNanos(). My code is as follows: Sensor Timestamp public void onSensorChanged( SensorEvent event ) { if( event.sensor.getType() == Sensor.TYPE_ACCELEROMETER )...

R: Volatility function that interprets NAs

r,data.frame,time-series,na
I am looking for help with getting a volatility function to work with my dataframe. In the function below, I'm just trying to get price daily log returns for each security (each column in my data is a different security's prices over time), and then calculate an annualized vol. volcalc=...

Calculating the difference in dates in a Pandas GroupBy object

python,pandas,time-series
I have a Pandas DataFrame with the following format: In [0]: df Out[0]: col1 col2 date 0 1 1 2015-01-01 1 1 2 2015-01-09 2 1 3 2015-01-10 3 2 1 2015-02-10 4 2 2 2015-02-10 5 2 3 2015-02-25 In [1]: df.dtypes Out[1]: col1 int64 col2 int64 date datetime64[ns]...

Different regression output using dynlm and lm

r,time-series,lm
I ran a regression first using lm and then using dynlm(from the package dynlm). Here is what I did using lm: Euribor3t <- ts(diff(Euribor3)) OIS3t <- ts(diff(Ois3)) x <- ts(diff(Eurepo3-Ois3)) Vstoxxt <- ts(diff(Vstoxx)) CDSt <- ts(diff(CDS)) omo2 <- ts(diff(log(Open.Market.Operations))) l1 <- (lag(Euribor3t, k=-1)) axx <- ts.intersect(Euribor3t, OIS3t, x, Vstoxxt, CDSt,...

Python Pandas: Calculations with Two Different Size Dataframes

python,pandas,time-series
I had two dataframes of differing sizes that I would like to do calculations with. The first dataset is a time series. The second dataset are the long-term monthly averages. The first: year month snow_depth 0 1979 1 18.322581 1 1979 2 11.535714 2 1979 3 5.322581 3 1979 4...

ETS multiplicative trend model written in state space form

time-series,forecasting,state-space
I have an ETS(M,Md,N) model and would like to write it in state space form: yt=w(x{t-1})+r(x{t-1})ɛt xt=f(x{t-1})+g(x{t-1})ɛt For additive trend, the state vector xt=(lt,bt)'. But I have no idea how to write the state vector xt for multiplicative trend. Can anyone help please:D...

Dates with month and day in time series plot in ggplot2 with facet for years

r,ggplot2,time-series
I want to have both month and day in the x-axis of the time series plot when using facet for years in ggplot2. My MWE is below: set.seed(12345) Date <- seq(as.Date("2010/1/1"), as.Date("2014/1/1"), "week") Y <- rnorm(n=length(Date), mean=100, sd=1) df <- data.frame(Date, Y) df$Year <- format(df$Date, "%Y") df$Month <- format(df$Date, "%b")...

Pandas Time-Series: Find previous value for each ID based on year and semester

python,pandas,time-series
I realize this is a fairly basic question, but I couldn't find what I'm looking for through searching (partly because I'm not sure how to summarize what I want). In any case: I have a dataframe that has the following columns: * ID (each one represents a specific college course)...

value from a past, potentially missing month in dataframe

python,pandas,time-series
Assume I have a DataFrame like the following: Month, Gender, State, Value 2010-01, M, S1, 10 2010-02, M, S1, 20 2010-05, M, S1, 26 2010-03, F, S2, 11 I want to add another column for the given Gender and state from the previous month (or X months past) if it...

Pandas Dataframe Plot

python,pandas,time-series,ipython-notebook
I am trying to plot a time series data. The dataframe looks like this [1]:Index ship_date cost_amount 0 1/8/2010 34276 1 1/8/2010 12375 2 1/8/2011 12343 3 2/9/2011 15435 [2]: df1.plot(figsize(20,5)) I am trying to plot the data but for some reason plot doesn't have x-axis in ascending order. How...

How to use parameters from data frame in R and loop through time holding them constant

r,nested,time-series,lapply,sapply
I have a function (weisurv) that has 2 parameters - sc and shp. It is a function through time (t). Time is a sequence, i.e. t<-seq(1:100). weisurv<-function(t,sc,shp){ surv<-exp(-(t/sc)^shp) return(surv) } I have a data frame (df) that contains a list of sc and shp values (like 300+ of them). For...

plotting a graph with 3 curves time series data

r,plot,time-series
I have a pretty basic questions, but I wasn't able to find an answer. I would like to create a graph with three curves (time series data) without using ts.plot. Here are the three data sets: a1 <- seq(as.Date("2001-01-01"),as.Date("2021-01-01"),"years") a2 <- rnorm(21,10,1) Dollar <- data.frame(a1,a2) dates <- as.Date(Dollar[,1], "%d.%m.%Y",tz="GMT") xtsplot1...

Creating a running counting variable in R?

r,time-series,running-total
I have a dataset of soccer match results, and I am hoping to learn R by creating a running set of ratings similar to the World Football Elo formula. I am running into trouble with things that seem to be simple in Excel aren't exactly intuitive in R. For instance,...

Dates on x-axis, time series

r,date,time-series,as.date
I have data covering a time period of over 25 years and I would like to see the years on the x-axis. dates <- as.Date(Dollar[,1], "%d.%m.%Y") Dollar <- as.xts(Dollar[,2], dates) plot(SWEDOLall, xaxt = "n", main="SMA", ann = FALSE) axis.Date(side = 1, dates, at = labDates, format = "%y", labels =...

Trying to check data frequency with Pandas Series of datetime64 objects

python,pandas,time-series
I have some time series data that can be 1Hz, 10Hz, or 100Hz. the file I load in happens to be 1Hz: In [6]: data = pd.read_csv("ftp.csv") In [7]: data.Time Out[7]: 0 NaN 1 11:30:08 AM 2 11:30:09 AM 3 11:30:10 AM 4 11:30:11 AM 5 11:30:12 AM 6 11:30:13...

Why does R give me a time series of row numbers instead of values?

r,time-series
Why does R give me a time series of row numbers instead of values? I load a CSV with a single column of values in the order I need them. I'm trying to make it a time series. Instead of giving me the values I've entered, R gives me the...

How to create a function and a loop to calculate growth rates of variables in a data frame in R

r,time-series,growth-rate
New to R and Stack Overflow. Suppose I have the following macroeconomic data loaded into a data frame called testdata in R. > testdata date gdp cpi_index rpi_index 21 2013 Q1 409985 125.067 247.4 22 2013 Q2 412620 125.971 249.7 23 2013 Q3 415577 126.352 250.9 24 2013 Q4 417265...

How to create a time series plot in the style of a horizontal stacked bar plot in r

r,plot,time-series,bar-chart
I would like to create a horizontal ‘stacked bar’ type plot in which date runs along the x-axis and my samples appear as bars on the y-axis. In the simple example below, I have three samples (a, b, c) each containing three values (0, 1, 2). I would like the...

geom_vlines multiple vlines per plot

r,ggplot2,time-series,timeserieschart
How can I get ggplot to produce something similar like library(ggplot2) library(reshape2) library(ecp) synthetic_control.data <- read.table("/path/synthetic_control.data.txt", quote="\"", comment.char="") n <- 2 s <- sample(1:100, n) idx <- c(s, 100+s, 200+s, 300+s, 400+s, 500+s) sample2 <- synthetic_control.data[idx,] df = as.data.frame(t(as.matrix(sample2))) #calculate the change points changeP <- e.divisive(as.matrix(df[1]), k=8, R = 400,...

time series data in mongodb - how to query embedded document

mongodb,time-series
I'm working on a design for a time series dataset, basically I have servers I monitor, and I would like to know some metric about it over a period of time. based on http://blog.mongodb.org/post/65517193370/schema-design-for-time-series-data-in-mongodb I created a design that is a document per server+month, and in it an embedded document...

Imputing missing values using ARIMA model

r,time-series,missing-data
I am trying to impute missing values in a time series with an ARIMA model in R. I tried this code but no success. x <- AirPassengers x[90:100] <- NA fit <- auto.arima(x) fitted(fit)[90:100] ## this is giving me NAs plot(x) lines(fitted(fit), col="red") The fitted model is not imputing the...

small line x-axis for years

r,time-series
I have data covering a time period of over 25 years. In the data set are over 6300 days. I would like to show the years on the x-axix. dates <- as.Date(DOL[,1], "%d.%m.%Y") DOL <- as.xts(DOL[,2], dates) plot(DOL, xaxt = "n", main="SMA", ann = FALSE) axis(1, at=as.POSIXct(dates),labels=format(dates,"%Y"),tick=TRUE) title(ylab = "Value")...

Adding date tick marks to a Matlab plot

matlab,plot,time-series
I have a plot of time series data, and I would like to replace the tick marks of the x-axis (automatically I have the number of the ordered observations) with the date when the value is observed. I would like to have a tick mark every 5 years for example....

pandas shift time series with missing values

python,pandas,time-series,shift
I have a times series with some missing entries, that looks like this: date value --------------- 2000 5 2001 10 2003 8 2004 72 2005 12 2007 13 I would like to do create a column for the "previous_value". But I only want it to show values for consecutive years....

R, lag( ) has inconsistent behavior for xts and ts objects

r,time-series,xts
I would like to take a lag of an xts variable, and the lag() function returns a lag. However, if I use it on a ts variable, it gives a lead. Is this a bug, or working as intended? library('xts') a = as.xts(ts(c(5,3,7,2,4,8,3), start=c(1980,1), freq=4)) cbind(a, lag(a)) # provides lag...

i10test for integration/stationary time series

matlab,time-series
In http://www.mathworks.com/help/econ/examples/time-series-regression-iv-spurious-regression.html, I am examing the use of the i10test for integration/stationary time series. The online help at http://www.mathworks.com/help/econ/i10test.html shows that this is done through a series of hypothesis tests. From my (OK, limited) exposure to hypthothesis testing, there is usually a threshold for the p-value, e.g. the alpha, that...

How to get a column of lists of contract payout pairs (date:amounts)?

r,time-series,dplyr
I'm trying to add a column of list objects to a data.frame of payments built like ID <- c("A", "B", "B", "c", "A", "B", "c", "c", "A", "B") Date = seq(as.Date("2000/07/01"), as.Date("2000/07/10"), "days") Amt <- rnorm(10, 10, 3) E <- data.frame(Date = Date, ID = ID, Amt = Amt) Date...

Setting limits with scale_x_datetime and time data

r,ggplot2,time-series
I want to set bounds for the x-axis for a plot of time-series data which features only time (no dates). My limits are: lims <- strptime(c("03:00","16:00"), format = "%H:%M") And my ggplot prints fine, but when I add this to scale_x_datetime scale_x_datetime(limits = lims) I get Error: Invalid input: time_trans...

Relationship between LinearModel & GeneralizedLinearMixedModel classes

matlab,oop,time-series,linear-regression,superclass
Matlab defines LinearModel and GeneralizedLinearMixedModel classes. Browsing the documentation indicates that either (i) one is derived from the other, or (ii) there is automatic conversion. These are complex objects, and I am just starting to explore them, so I apologize if their relationship is obvious, but what exactly is their...

R conditionally matching date-time from one dataframe to closest date-time field in second dataframe

r,datetime,merge,data.frame,time-series
I have two data frames, df.events and df.activ. df.activ has very granular minute level data and an order of magnitude more records (1,000,000+) than df.events which has ~100,000 records, also at minute level granularity. The two dataframes have two common fields, DateTime and Geo. Both DateTime columns are in as.POSIXlt,...

R, how to use pch with time series plot

r,time-series,lattice
I would like to plot a time series (meaning line graph with x axis as time) and specify a plotting character to use. None of the following has worked a1 = as.xts(ts(c(5,3,7,2,4,8,3), start=c(1980,1), freq=4)) library('lattice') xyplot(a1, col="red", pch=2) xyplot(a1, col="red", par.settings = list(superpose.symbol = list(col = 1, pch = 2)),)...

Calculate the maximum price fluctuation in a 24 hour window

r,time-series
I have a data frame that contains two columns - time and price. It contains a series of observations for price of a certain item at various times. Here is a sample. > df time price 1 2014-12-12 14:57:15 45.81 2 2014-12-12 14:57:15 45.90 3 2014-12-12 15:00:08 45.76 4 2014-12-12...

SciKit-learn for data driven regression of oscillating data

python,time-series,scikit-learn,regression,prediction
Long time lurker first time poster. I have data that roughly follows a y=sin(time) distribution, but also depends on other variables than time. In terms of correlations, since the target y-variable oscillates there is almost zero statistical correlation with time, but y obviously depends very strongly on time. The goal...

Convert data frame with epoch timestamps to time-series with milliseconds in R

r,time-series,xts,zoo
I have the following data.frame: df <- data.frame(timestamp=c(1428319770511, 1428319797218, 1428319798182, 1428319803327, 1428319808478), session=c("A","A","B","A","A")) I'd like to convert this data frame to a time series and work on time windows shorter than one second. I already tried zoo and xts, but I found it difficult to represent the epoch times as...

Enlarge time series and fill with -9999 R

r,merge,time-series
I run am model from 2007-01-01 00 to 2013-12-31 23. Not all my observations are that long, they start later and/or end earlier. In that case I want to fill in -9999 values. I have: [1,] "2003 09 01 01" "0" [2,] "2003 09 01 02" "0" [3,] "2003 09...

Plot monthly Time series from a data frame with daily data

r,plot,time-series,legend
I have a data set for motor vehicle crashes happening daily in NYC from 1 Jan 2014 to 31 Dec 2012. I want to plot time series of the number of injured cyclists, and motorists, monthly in a single plot. My data looks like this: Date Time Location Cyclists injured...

Remove values which are surrounded by a certain number of NAs

r,time-series,missing-data
I wish to remove values in a time series which are surrounded by blocks of NA of a certain minimal length. Some toy data: x = seq(0,10,length.out = 100) y = sin(x) + rnorm(length(x), mean=0, sd=0.1) y[20:21] = rep(NA, 2) y[50:54] = rep(NA, 5) y[55:59] = seq(-0.1, -0.8, length.out =...

Forecasting an Arima Model in R Returning Strange Error

r,time-series,shiny,forecasting
I'm working on building a Shiny App for forecasting time series. One component of this is using ARIMA models to forecast. The user specifies the start and end of the historical data, what p, d, and q they would like to use in the ARIMA model (if they don't want...

Timeline just with years that shall be ordered and just showing per year how many values are available

d3.js,time-series,timeline,timeserieschart
I would like to use a timeline that just shows years. The special about my data is that I just have years. But these years are ordered. So I know the exact order but not any more detail like day or month. So first of all I would like to...

Plotting multivariate time-series data in R

r,graph,plot,ggplot2,time-series
My data looks like this: > head(Full.df) Date Month Week Year Count.S Count.G Count.W Count.F 1 2006-01-02 2006-01-01 2006-01-02 2006-01-01 0 7 9 6 2 2006-01-03 2006-01-01 2006-01-02 2006-01-01 0 13 12 4 3 2006-01-04 2006-01-01 2006-01-02 2006-01-01 0 13 15 4 4 2006-01-05 2006-01-01 2006-01-02 2006-01-01 0 20 6...

R: compute waldtest using dynlm

r,time-series
I ran the following two regressions: library("dynlm") library("lmtest") zoop <- (test1[, -1]) f <- d(y) ~ d(x)+ d(z) + d(m) + d(log(p)) m1 <- dynlm(f, data = zoop, start = 1,end = 15) coeftest(m1, vcov=NeweyWest) m2 <- dynlm(f, data = zoop, start = 16,end = 31) coeftest(m2, vcov=NeweyWest) which gives...

R: HAC by NeweyWest using dynlm

r,time-series,regression
This is what I would like to do: library("lmtest") library("dynlm") test$Date = as.Date(test$Date, format = "%d.%m.%Y") zooX = zoo(test[, -1], order.by = test$Date) f <- d(Euribor3) ~ d(Ois3) + d(CDS) + d(Vstoxx) + d(log(omo)) + d(L(Euribor3)) m1 <- dynlm(f, data = zooX, start = as.Date("2005-01-05"),end = as.Date("2005-01-24")) m2 <- dynlm(f,...

Lag dependent variable [closed]

r,time-series
I want to compute the following time series regression using R: $\Delta y_t=\beta_1 \Delta x_t+\beta_2 \Delta z_t+\beta_3 \Delta m_t+\beta_4 \Delta y_{t−1}$ Since I have not that much experience with R I want to ask if the following R code gives me what I want: y <- ts(diff(YY)) x <- ts(diff(XX))...

Estimating change of a cyclic boolean variable

time-series,sampling,measurement,probability-theory
We have a boolean variable X which is either true or false and alternates at each time step with a probability p. I.e. if p is 0.2, X would alternate once every 5 time steps on average. We also have a time line and observations of the value of this...

Using Cassandra for time series data

cassandra,time-series,composite-key
I'm on my research for storing logs to Cassandra. The schema for logs would be something like this. EDIT: I've changed the schema in order to make some clarification. CREATE TABLE log_date ( userid bigint, time timeuuid, reason text, item text, price int, count int, PRIMARY KEY ((userid), time) -...

R seasonal decomposition

r,time-series
I simulate a time series with periodic and linear components and try to use the R stl function to analyze it n = 1000 x = ts(0.1*rnorm(n) + sin(6*pi*(1:n)/n) + (1:n)/n,frequency=n) plot(x) stl(x,"per") but get a message Error in stl(x, "per") : series is not periodic or has less than...

timeseries fitted values from trend python

python,pandas,time-series,statsmodels,trend
I have daily stock price data from yahoo finance in a dataframe called price_data. I would like to add a column to this which provides the fitted value from a time series trend of the Adj Close column. Here is the structure of the data I am using: In [41]:...

Replace list of permutations with getSymbols data in R

r,time-series,permutation,quantmod
I downloaded some stock data: require("quantmod") s <- c("AAPL", "ADBE", "ADI", "ADP", "ADSK") e <- new.env() getSymbols(s, src='yahoo', from='2015-01-10', env = e ) #get closing prices close <- do.call(merge, eapply(e, function(x) Cl(x))) I found all the pairs of symbol names: #find all the pairwise permutations perm<-combn(s,2) perm [,1] [,2] [,3]...

Replace -inf, NaN and NA values with zero in a dataset in R

r,time-series,nan,zoo
I am trying to run some trading strategies in R. I have downloaded some stock prices and calculated returns. The new return dataset has a number of -inf, NaN, and NA values. I am reproducing a row of the dataset (log_ret). Its a zoo dataset. library(zoo) log_ret <- structure( c(0.234,-0.012,-Inf,NaN,0.454,Inf),...

Changing tick intervals when x axis values are dates

r,ggplot2,time-series
Apologies if this is easy to solve, but I can't get my head around it. I have this data frame: > aaci Date Plate.1 Plate.2 Plate.3 Plate.4 Plate.5 Plate.6 Plate.7 Plate.8 Average SE Species 1 2014-06-19 0.0000000 0.0000000 0.000000 0.000000 0.0000000 0.0000000 0.000000 0.0000000 0.000000 0.0000000 aa 7 2014-08-04 7.0057778...

Direct forecast using epsilion-SVR

matlab,time-series,libsvm,forecasting
Is it possible to predict directly into the future using epsilion-svr? My dataset is a univariate time series and has per line a record in this format: Y(t-W), Y(t-W+1), ..., Y(t), Y(t+PH) W is the number of time steps to consider PH controls how many steps into the future I...

Read Data into Time Series Object in R

r,data,time-series
My data looks as follows: Month/Year;Number 01/2010; 1.0 02/2010;19.0 03/2010; 1.0 ... How can I read this into a ts(object) in R?...

Munging Time Series in Excel

vba,excel-vba,time-series
I want to sort data by the date from latest to earliest. My trouble is that the data i have has dates in mm-dd-yyyy text format. I could easily clean this up using Pandas in python but don't know the tools available in excel. Even when I try to change...