Value Preprocessing

In order to process 1-D TS as input for ML model, it has to be converted into Matrix form where each row represents a slice of 1-D TS representing daily/hourly/weekly pattern depending on the size of the chunk, stride, and number of steps ahead for prediction. Below illustrates the processing workflow to Matrify a 1-D TS.

For illustration purposes, the code below generates a Date,Value dataframe where the values are just a sequece of integer from 1 to the length of the date sequence. We use this simple sequence to have a better understanding how the slicing of rows, steps ahead, and the stride to create the Matrified output is generated.

using TSML

lower = DateTime(2017,1,1)
upper = DateTime(2017,1,5)
dat=lower:Dates.Hour(1):upper |> collect
vals = 1:length(dat)
x = DataFrame(Date=dat,Value=vals)

julia> last(x,5)
5×2 DataFrame
│ Row │ Date                │ Value │
│     │ DateTime            │ Int64 │
├─────┼─────────────────────┼───────┤
│ 1   │ 2017-01-04T20:00:00 │ 93    │
│ 2   │ 2017-01-04T21:00:00 │ 94    │
│ 3   │ 2017-01-04T22:00:00 │ 95    │
│ 4   │ 2017-01-04T23:00:00 │ 96    │
│ 5   │ 2017-01-05T00:00:00 │ 97    │

Matrifier

Let us create an instance of Matrifier passing the size of row, stride, and steps ahead to predict:

mtr = Matrifier(Dict(:ahead=>6,:size=>6,:stride=>3))
res = fit_transform!(mtr,x)

julia> first(res,5)
5×7 DataFrame
│ Row │ x1    │ x2    │ x3    │ x4    │ x5    │ x6    │ output │
│     │ Int64 │ Int64 │ Int64 │ Int64 │ Int64 │ Int64 │ Int64  │
├─────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┤
│ 1   │ 86    │ 87    │ 88    │ 89    │ 90    │ 91    │ 97     │
│ 2   │ 83    │ 84    │ 85    │ 86    │ 87    │ 88    │ 94     │
│ 3   │ 80    │ 81    │ 82    │ 83    │ 84    │ 85    │ 91     │
│ 4   │ 77    │ 78    │ 79    │ 80    │ 81    │ 82    │ 88     │
│ 5   │ 74    │ 75    │ 76    │ 77    │ 78    │ 79    │ 85     │

In this example, we have hourly values. We indicated in the Matrifier to generate a matrix where the size of each row is 6 hours, steps ahead for prediction is 6 hours and the stride of 3 hours. There are 7 columns because the last column indicates the value indicated by the steps ahead argument.

Let us try to make a matrix with the size of 6 hours, steps ahead of 2 hours, and a stride of 3 hours:

mtr = Matrifier(Dict(:ahead=>2,:size=>6,:stride=>3))
res = fit_transform!(mtr,x)

julia> first(res,5)
5×7 DataFrame
│ Row │ x1    │ x2    │ x3    │ x4    │ x5    │ x6    │ output │
│     │ Int64 │ Int64 │ Int64 │ Int64 │ Int64 │ Int64 │ Int64  │
├─────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┤
│ 1   │ 90    │ 91    │ 92    │ 93    │ 94    │ 95    │ 97     │
│ 2   │ 87    │ 88    │ 89    │ 90    │ 91    │ 92    │ 94     │
│ 3   │ 84    │ 85    │ 86    │ 87    │ 88    │ 89    │ 91     │
│ 4   │ 81    │ 82    │ 83    │ 84    │ 85    │ 86    │ 88     │
│ 5   │ 78    │ 79    │ 80    │ 81    │ 82    │ 83    │ 85     │