Monotonic Detection and Plotting

One important preprocessing step for time series data processing is the detection of monotonic data and transform it to non-monotonic type by using the finite difference operator.

Artificial Data Example

Let's create an artificial monotonic data and apply our monotonic transformer to normalize it. We can use the Plotter filter to visualize the generated data.

using TSML

Random.seed!(123)
pltr = Plotter(Dict(:interactive => false,:pdfoutput => false))
mdates = DateTime(2017,12,1,1):Dates.Hour(1):DateTime(2017,12,31,10) |> collect
mvals = rand(length(mdates)) |> cumsum
df =  DataFrame(Date=mdates ,Value = mvals)
fit_transform!(pltr,df)

Now that we have a monotonic data, let's use the Monotonicer to normalize and plot the result:

using TSML

mono = Monotonicer(Dict())

pipeline = @pipeline mono |> pltr

res=fit_transform!(pipeline,df)

Real Data Example

We will now apply the entire pipeline starting from reading csv data, aggregate, impute, normalize if it's monotonic, and plot. We will consider three different data types: a regular time series data, a monotonic data, and a daily monotonic data. The difference between monotonic and daily monotonic is that the values in daily monotonic resets to zero or some baseline and cumulatively increases in a day until the next day where it resets to zero or some baseline value. Monotonicer automatically detects these three different types and apply the corresponding normalization accordingly.

using TSML

regularfile = joinpath(dirname(pathof(TSML)),"../data/typedetection/regular.csv")
monofile = joinpath(dirname(pathof(TSML)),"../data/typedetection/monotonic.csv")
dailymonofile = joinpath(dirname(pathof(TSML)),"../data/typedetection/dailymonotonic.csv")

regularfilecsv = CSVDateValReader(Dict(:filename=>regularfile,:dateformat=>"dd/mm/yyyy HH:MM"))
monofilecsv = CSVDateValReader(Dict(:filename=>monofile,:dateformat=>"dd/mm/yyyy HH:MM"))
dailymonofilecsv = CSVDateValReader(Dict(:filename=>dailymonofile,:dateformat=>"dd/mm/yyyy HH:MM"))

valgator = DateValgator(Dict(:dateinterval=>Dates.Hour(1)))
valnner = DateValNNer(Dict(:dateinterval=>Dates.Hour(1)))
stfier = Statifier(Dict(:processmissing=>true))
mono = Monotonicer(Dict())
pltr = Plotter(Dict(:interactive => false))

Regular TS Processing

Let's test by feeding the regular time series type to the pipeline. We expect that for this type, Monotonicer will not perform further processing:

• Pipeline with Monotonicer: regular time series

pipeline = @pipeline regularfilecsv |> valgator |> valnner |> mono |> pltr

fit_transform!(pipeline)