Training and Validation
Let us continue our discussion by using another dataset. This time, let's use CMC dataset that are mostly categorical. CMC is about asking women of their contraceptive choice. The dataset is composed of the following features:
using AutoMLPipeline
using CSV
using DataFrames
cmcdata = CSV.File(joinpath(dirname(pathof(AutoMLPipeline)),"../data/cmc.csv")) |> DataFrame;
X = cmcdata[:,1:end-1]
Y = cmcdata[:,end] .|> string
show5(df) = first(df,5)julia> show5(cmcdata)5×10 DataFrame Row │ Wifes_age Wifes_education Husbands_education Number_of_children_ever_born Wifes_religion Wifes_now_working.3F Husbands_occupation Standard.of.living_index Media_exposure Contraceptive_method_used │ Int64 Int64 Int64 Int64 Int64 Int64 Int64 Int64 Int64 Int64 ─────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 1 │ 24 2 3 3 1 1 2 3 0 1 2 │ 45 1 3 10 1 1 3 4 0 1 3 │ 43 2 3 7 1 1 3 4 0 1 4 │ 42 3 2 9 1 1 3 3 0 1 5 │ 36 3 3 8 1 1 3 2 0 1
Let's examine the number of unique instances for each column:
julia> DataFrame(hcat([length(unique(n)) for n in eachcol(cmcdata)],names(cmcdata)),:auto)10×2 DataFrame Row │ x1 x2 │ Any Any ─────┼─────────────────────────────────── 1 │ 34 Wifes_age 2 │ 4 Wifes_education 3 │ 4 Husbands_education 4 │ 15 Number_of_children_ever_born 5 │ 2 Wifes_religion 6 │ 2 Wifes_now_working.3F 7 │ 4 Husbands_occupation 8 │ 4 Standard.of.living_index 9 │ 2 Media_exposure 10 │ 3 Contraceptive_method_used
Except for Wife's age and Number of children, the other columns have less than five unique instances. Let's create a pipeline to filter those columns and convert them to hot-bits and concatenate them with the standardized scale of the numeric columns.
std = SKPreprocessor("StandardScaler")
ohe = OneHotEncoder()
kohe = SKPreprocessor("OneHotEncoder")
catf = CatFeatureSelector()
numf = NumFeatureSelector()
disc = CatNumDiscriminator(5) # unique instances <= 5 are categories
pcmc = @pipeline disc |> ((catf |> ohe) + (numf |> std))
dfcmc = fit_transform!(pcmc,X)julia> show5(dfcmc)5×24 DataFrame Row │ x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 x15 x16 x17 x18 x19 x20 x21 x22 x1_1 x2_1 │ Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 ─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 1 │ 1.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 1.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 -1.03817 -0.110856 2 │ 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 1.51519 2.85808 3 │ 1.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 1.27202 1.58568 4 │ 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 1.15043 2.43394 5 │ 0.0 0.0 1.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 1.0 0.0 1.0 0.0 0.420897 2.00981
Evaluate Learners with Same Pipeline
You can get a list of sklearners and skpreprocessors by using the following function calls:
julia> sklearners()syntax: SKLearner(name::String, args::Dict=Dict()) where 'name' can be one of: AdaBoostClassifier AdaBoostRegressor ARDRegression BaggingClassifier BayesianRidge BernoulliNB ComplementNB DecisionTreeClassifier DecisionTreeRegressor ElasticNet ExtraTreesClassifier ExtraTreesRegressor GaussianNB GaussianProcessClassifier GaussianProcessRegressor GradientBoostingClassifier GradientBoostingRegressor IsotonicRegression KernelRidge KNeighborsClassifier KNeighborsRegressor Lars Lasso LassoLars LinearDiscriminantAnalysis LinearSVC LogisticRegression MLPClassifier MLPRegressor MultinomialNB NearestCentroid NuSVC OrthogonalMatchingPursuit PassiveAggressiveClassifier PassiveAggressiveRegressor QuadraticDiscriminantAnalysis RadiusNeighborsClassifier RadiusNeighborsRegressor RandomForestClassifier RandomForestRegressor Ridge RidgeClassifier RidgeClassifierCV RidgeCV SGDClassifier SGDRegressor SVC SVR VotingClassifier and 'args' are the corresponding learner's initial parameters. Note: Consult Scikitlearn's online help for more details about the learner's arguments.julia> skpreprocessors()syntax: SKPreprocessor(name::String, args::Dict=Dict()) where *name* can be one of: Binarizer chi2 dict_learning dict_learning_online DictionaryLearning f_classif f_regression FactorAnalysis FastICA fastica FunctionTransformer GenericUnivariateSelect IncrementalPCA KBinsDiscretizer KernelCenterer KernelPCA LabelBinarizer LabelEncoder LatentDirichletAllocation MaxAbsScaler MiniBatchDictionaryLearning MiniBatchSparsePCA MinMaxScaler MissingIndicator MultiLabelBinarizer mutual_info_classif mutual_info_regression NMF non_negative_factorization Normalizer OneHotEncoder OrdinalEncoder PCA PolynomialFeatures PowerTransformer QuantileTransformer RFE RFECV RobustScaler SelectFdr SelectFpr SelectFromModel SelectFwe SelectKBest SelectPercentile SimpleImputer sparse_encode SparseCoder SparsePCA StandardScaler TruncatedSVD VarianceThreshold and *args* are the corresponding preprocessor's initial parameters. Note: Please consult Scikitlearn's online help for more details about the preprocessor's arguments.
Let us evaluate 4 learners using the same preprocessing pipeline:
jrf = RandomForest()
ada = SKLearner("AdaBoostClassifier")
sgd = SKLearner("SGDClassifier")
tree = PrunedTree()using DataFrames: DataFrame, nrow,ncol
learners = DataFrame()
for learner in [jrf,ada,sgd,tree]
pcmc = @pipeline disc |> ((catf |> ohe) + (numf |> std)) |> learner
println(learner.name)
mean,sd,folds = crossvalidate(pcmc,X,Y,"accuracy_score",5)
global learners = vcat(learners,DataFrame(name=learner.name,mean=mean,sd=sd,kfold=folds))
end;┌ Warning: Assignment to `pcmc` in soft scope is ambiguous because a global variable by the same name exists: `pcmc` will be treated as a new local. Disambiguate by using `local pcmc` to suppress this warning or `global pcmc` to assign to the existing global variable.
└ @ learning.md:70
rf_fvh
fold: 1, 0.5457627118644067
fold: 2, 0.5034013605442177
fold: 3, 0.5389830508474577
fold: 4, 0.47619047619047616
fold: 5, 0.5288135593220339
errors: 0
AdaBoostClassifier_hsP
/home/runner/work/AutoMLPipeline.jl/AutoMLPipeline.jl/docs/.CondaPkg/env/lib/python3.11/site-packages/sklearn/ensemble/_weight_boosting.py:527: FutureWarning: The SAMME.R algorithm (the default) is deprecated and will be removed in 1.6. Use the SAMME algorithm to circumvent this warning.
warnings.warn(
fold: 1, 0.5457627118644067
/home/runner/work/AutoMLPipeline.jl/AutoMLPipeline.jl/docs/.CondaPkg/env/lib/python3.11/site-packages/sklearn/ensemble/_weight_boosting.py:527: FutureWarning: The SAMME.R algorithm (the default) is deprecated and will be removed in 1.6. Use the SAMME algorithm to circumvent this warning.
warnings.warn(
fold: 2, 0.5306122448979592
/home/runner/work/AutoMLPipeline.jl/AutoMLPipeline.jl/docs/.CondaPkg/env/lib/python3.11/site-packages/sklearn/ensemble/_weight_boosting.py:527: FutureWarning: The SAMME.R algorithm (the default) is deprecated and will be removed in 1.6. Use the SAMME algorithm to circumvent this warning.
warnings.warn(
fold: 3, 0.5830508474576271
/home/runner/work/AutoMLPipeline.jl/AutoMLPipeline.jl/docs/.CondaPkg/env/lib/python3.11/site-packages/sklearn/ensemble/_weight_boosting.py:527: FutureWarning: The SAMME.R algorithm (the default) is deprecated and will be removed in 1.6. Use the SAMME algorithm to circumvent this warning.
warnings.warn(
fold: 4, 0.5544217687074829
/home/runner/work/AutoMLPipeline.jl/AutoMLPipeline.jl/docs/.CondaPkg/env/lib/python3.11/site-packages/sklearn/ensemble/_weight_boosting.py:527: FutureWarning: The SAMME.R algorithm (the default) is deprecated and will be removed in 1.6. Use the SAMME algorithm to circumvent this warning.
warnings.warn(
fold: 5, 0.535593220338983
errors: 0
SGDClassifier_RCa
fold: 1, 0.49491525423728816
fold: 2, 0.4557823129251701
fold: 3, 0.46440677966101696
fold: 4, 0.4387755102040816
fold: 5, 0.488135593220339
errors: 0
prunetree_YSp
fold: 1, 0.4745762711864407
fold: 2, 0.5034013605442177
fold: 3, 0.4745762711864407
fold: 4, 0.4557823129251701
fold: 5, 0.4745762711864407
errors: 0julia> @show learners;learners = 4×4 DataFrame Row │ name mean sd kfold │ String Float64 Float64 Int64 ─────┼──────────────────────────────────────────────────── 1 │ rf_fvh 0.51863 0.0286669 5 2 │ AdaBoostClassifier_hsP 0.549888 0.0206957 5 3 │ SGDClassifier_RCa 0.468403 0.0231588 5 4 │ prunetree_YSp 0.476582 0.0170585 5
For this particular pipeline, Adaboost has the best performance followed by RandomForest.
Let's extend the pipeline adding Gradient Boost learner and Robust Scaler.
rbs = SKPreprocessor("RobustScaler")
gb = SKLearner("GradientBoostingClassifier")
learners = DataFrame()
for learner in [jrf,ada,sgd,tree,gb]
pcmc = @pipeline disc |> ((catf |> ohe) + (numf |> rbs) + (numf |> std)) |> learner
println(learner.name)
mean,sd,folds = crossvalidate(pcmc,X,Y,"accuracy_score",5)
global learners = vcat(learners,DataFrame(name=learner.name,mean=mean,sd=sd,kfold=folds))
end;┌ Warning: Assignment to `pcmc` in soft scope is ambiguous because a global variable by the same name exists: `pcmc` will be treated as a new local. Disambiguate by using `local pcmc` to suppress this warning or `global pcmc` to assign to the existing global variable.
└ @ learning.md:89
rf_fvh
fold: 1, 0.49491525423728816
fold: 2, 0.5136054421768708
fold: 3, 0.5084745762711864
fold: 4, 0.5510204081632653
fold: 5, 0.5389830508474577
errors: 0
AdaBoostClassifier_hsP
/home/runner/work/AutoMLPipeline.jl/AutoMLPipeline.jl/docs/.CondaPkg/env/lib/python3.11/site-packages/sklearn/ensemble/_weight_boosting.py:527: FutureWarning: The SAMME.R algorithm (the default) is deprecated and will be removed in 1.6. Use the SAMME algorithm to circumvent this warning.
warnings.warn(
fold: 1, 0.5152542372881356
/home/runner/work/AutoMLPipeline.jl/AutoMLPipeline.jl/docs/.CondaPkg/env/lib/python3.11/site-packages/sklearn/ensemble/_weight_boosting.py:527: FutureWarning: The SAMME.R algorithm (the default) is deprecated and will be removed in 1.6. Use the SAMME algorithm to circumvent this warning.
warnings.warn(
fold: 2, 0.5306122448979592
/home/runner/work/AutoMLPipeline.jl/AutoMLPipeline.jl/docs/.CondaPkg/env/lib/python3.11/site-packages/sklearn/ensemble/_weight_boosting.py:527: FutureWarning: The SAMME.R algorithm (the default) is deprecated and will be removed in 1.6. Use the SAMME algorithm to circumvent this warning.
warnings.warn(
fold: 3, 0.559322033898305
/home/runner/work/AutoMLPipeline.jl/AutoMLPipeline.jl/docs/.CondaPkg/env/lib/python3.11/site-packages/sklearn/ensemble/_weight_boosting.py:527: FutureWarning: The SAMME.R algorithm (the default) is deprecated and will be removed in 1.6. Use the SAMME algorithm to circumvent this warning.
warnings.warn(
fold: 4, 0.5544217687074829
/home/runner/work/AutoMLPipeline.jl/AutoMLPipeline.jl/docs/.CondaPkg/env/lib/python3.11/site-packages/sklearn/ensemble/_weight_boosting.py:527: FutureWarning: The SAMME.R algorithm (the default) is deprecated and will be removed in 1.6. Use the SAMME algorithm to circumvent this warning.
warnings.warn(
fold: 5, 0.5694915254237288
errors: 0
SGDClassifier_RCa
fold: 1, 0.4542372881355932
fold: 2, 0.4217687074829932
fold: 3, 0.5084745762711864
fold: 4, 0.4557823129251701
fold: 5, 0.47796610169491527
errors: 0
prunetree_YSp
fold: 1, 0.5457627118644067
fold: 2, 0.47278911564625853
fold: 3, 0.49491525423728816
fold: 4, 0.445578231292517
fold: 5, 0.49491525423728816
errors: 0
GradientBoostingClassifier_Cl1
fold: 1, 0.5559322033898305
fold: 2, 0.5136054421768708
fold: 3, 0.559322033898305
fold: 4, 0.6190476190476191
fold: 5, 0.5525423728813559
errors: 0julia> @show learners;learners = 5×4 DataFrame Row │ name mean sd kfold │ String Float64 Float64 Int64 ─────┼──────────────────────────────────────────────────────────── 1 │ rf_fvh 0.5214 0.0229989 5 2 │ AdaBoostClassifier_hsP 0.54582 0.0222608 5 3 │ SGDClassifier_RCa 0.463646 0.0320887 5 4 │ prunetree_YSp 0.490792 0.0368245 5 5 │ GradientBoostingClassifier_Cl1 0.56009 0.0377878 5
This time, Gradient boost has the best performance.