Training and Validation
Let us continue our discussion by using another dataset. This time, let's use CMC dataset that are mostly categorical. CMC is about asking women of their contraceptive choice. The dataset is composed of the following features:
using AutoMLPipeline
using CSV
using DataFrames
cmcdata = CSV.File(joinpath(dirname(pathof(AutoMLPipeline)),"../data/cmc.csv")) |> DataFrame;
X = cmcdata[:,1:end-1]
Y = cmcdata[:,end] .|> string
show5(df) = first(df,5)julia> show5(cmcdata)
5×10 DataFrame
Row │ Wifes_age Wifes_education Husbands_education Number_of_children_ever_born Wifes_religion Wifes_now_working.3F Husbands_occupation Standard.of.living_index Media_exposure Contraceptive_method_used
│ Int64 Int64 Int64 Int64 Int64 Int64 Int64 Int64 Int64 Int64
─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ 24 2 3 3 1 1 2 3 0 1
2 │ 45 1 3 10 1 1 3 4 0 1
3 │ 43 2 3 7 1 1 3 4 0 1
4 │ 42 3 2 9 1 1 3 3 0 1
5 │ 36 3 3 8 1 1 3 2 0 1Let's examine the number of unique instances for each column:
julia> [n=>length(unique(x)) for (n,x) in eachcol(cmcdata,true)]
ERROR: MethodError: no method matching eachcol(::DataFrames.DataFrame, ::Bool)
Closest candidates are:
eachcol(::DataFrames.AbstractDataFrame) at /home/travis/.julia/packages/DataFrames/oQ5c7/src/abstractdataframe/iteration.jl:173Except for Wife's age and Number of children, the other columns have less than five unique instances. Let's create a pipeline to filter those columns and convert them to hot-bits and concatenate them with the standardized scale of the numeric columns.
std = SKPreprocessor("StandardScaler")
ohe = OneHotEncoder()
kohe = SKPreprocessor("OneHotEncoder")
catf = CatFeatureSelector()
numf = NumFeatureSelector()
disc = CatNumDiscriminator(5) # unique instances <= 5 are categories
pcmc = @pipeline disc |> ((catf |> ohe) + (numf |> std))
dfcmc = fit_transform!(pcmc,X)julia> show5(dfcmc)
5×24 DataFrame
Row │ x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12 x13 x14 x15 x16 x17 x18 x19 x20 x21 x22 x1_1 x2_1
│ Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64 Float64
─────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ 1.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 1.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 -1.03817 -0.110856
2 │ 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 1.51519 2.85808
3 │ 1.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 1.27202 1.58568
4 │ 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 1.15043 2.43394
5 │ 0.0 0.0 1.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 1.0 0.0 1.0 0.0 0.420897 2.00981Evaluate Learners with Same Pipeline
You can get a list of sklearners and skpreprocessors by using the following function calls:
julia> sklearners()
syntax: SKLearner(name::String, args::Dict=Dict())
where 'name' can be one of:
AdaBoostClassifier AdaBoostRegressor ARDRegression BaggingClassifier BayesianRidge BernoulliNB ComplementNB DecisionTreeClassifier DecisionTreeRegressor ElasticNet ExtraTreesClassifier ExtraTreesRegressor GaussianNB GaussianProcessClassifier GaussianProcessRegressor GradientBoostingClassifier GradientBoostingRegressor IsotonicRegression KernelRidge KNeighborsClassifier KNeighborsRegressor Lars Lasso LassoLars LDA LinearSVC LogisticRegression MLPClassifier MLPRegressor MultinomialNB NearestCentroid NuSVC OrthogonalMatchingPursuit PassiveAggressiveClassifier PassiveAggressiveRegressor QDA RadiusNeighborsClassifier RadiusNeighborsRegressor RandomForestClassifier RandomForestRegressor Ridge RidgeClassifier RidgeClassifierCV RidgeCV SGDClassifier SGDRegressor SVC SVR VotingClassifier
and 'args' are the corresponding learner's initial parameters.
Note: Consult Scikitlearn's online help for more details about the learner's arguments.
julia> skpreprocessors()
syntax: SKPreprocessor(name::String, args::Dict=Dict())
where *name* can be one of:
Binarizer chi2 dict_learning dict_learning_online DictionaryLearning f_classif f_regression FactorAnalysis FastICA fastica FunctionTransformer GenericUnivariateSelect IncrementalPCA KBinsDiscretizer KernelCenterer KernelPCA LabelBinarizer LabelEncoder LatentDirichletAllocation MaxAbsScaler MiniBatchDictionaryLearning MiniBatchSparsePCA MinMaxScaler MissingIndicator MultiLabelBinarizer mutual_info_classif mutual_info_regression NMF non_negative_factorization Normalizer OneHotEncoder OrdinalEncoder PCA PolynomialFeatures PowerTransformer QuantileTransformer RFE RFECV RobustScaler SelectFdr SelectFpr SelectFromModel SelectFwe SelectKBest SelectPercentile SimpleImputer sparse_encode SparseCoder SparsePCA StandardScaler TruncatedSVD VarianceThreshold
and *args* are the corresponding preprocessor's initial parameters.
Note: Please consult Scikitlearn's online help for more details about the preprocessor's arguments.Let us evaluate 4 learners using the same preprocessing pipeline:
jrf = RandomForest()
ada = SKLearner("AdaBoostClassifier")
sgd = SKLearner("SGDClassifier")
tree = PrunedTree()using DataFrames: DataFrame, nrow,ncol
learners = DataFrame()
for learner in [jrf,ada,sgd,tree]
pcmc = @pipeline disc |> ((catf |> ohe) + (numf |> std)) |> learner
println(learner.name)
mean,sd,folds = crossvalidate(pcmc,X,Y,"accuracy_score",5)
global learners = vcat(learners,DataFrame(name=learner.name,mean=mean,sd=sd,kfold=folds))
end;┌ Warning: Assignment to `pcmc` in soft scope is ambiguous because a global variable by the same name exists: `pcmc` will be treated as a new local. Disambiguate by using `local pcmc` to suppress this warning or `global pcmc` to assign to the existing global variable. └ @ none:2 rf_ds2 fold: 1, 0.5491525423728814 fold: 2, 0.5068027210884354 fold: 3, 0.5186440677966102 fold: 4, 0.5170068027210885 fold: 5, 0.5491525423728814 errors: 0 AdaBoostClassifier_CwM fold: 1, 0.5254237288135594 fold: 2, 0.5816326530612245 fold: 3, 0.5830508474576271 fold: 4, 0.5 fold: 5, 0.576271186440678 errors: 0 SGDClassifier_p1H fold: 1, 0.5220338983050847 fold: 2, 0.47959183673469385 fold: 3, 0.511864406779661 fold: 4, 0.46598639455782315 fold: 5, 0.46440677966101696 errors: 0 prunetree_SaL fold: 1, 0.4406779661016949 fold: 2, 0.46258503401360546 fold: 3, 0.47796610169491527 fold: 4, 0.48639455782312924 fold: 5, 0.47796610169491527 errors: 0
julia> @show learners;
learners = 4×4 DataFrame
Row │ name mean sd kfold
│ String Float64 Float64 Int64
─────┼────────────────────────────────────────────────────
1 │ rf_ds2 0.528152 0.0197006 5
2 │ AdaBoostClassifier_CwM 0.553276 0.0381887 5
3 │ SGDClassifier_p1H 0.488777 0.0266305 5
4 │ prunetree_SaL 0.469118 0.0180733 5For this particular pipeline, Adaboost has the best performance followed by RandomForest.
Let's extend the pipeline adding Gradient Boost learner and Robust Scaler.
rbs = SKPreprocessor("RobustScaler")
gb = SKLearner("GradientBoostingClassifier")
learners = DataFrame()
for learner in [jrf,ada,sgd,tree,gb]
pcmc = @pipeline disc |> ((catf |> ohe) + (numf |> rbs) + (numf |> std)) |> learner
println(learner.name)
mean,sd,folds = crossvalidate(pcmc,X,Y,"accuracy_score",5)
global learners = vcat(learners,DataFrame(name=learner.name,mean=mean,sd=sd,kfold=folds))
end;┌ Warning: Assignment to `pcmc` in soft scope is ambiguous because a global variable by the same name exists: `pcmc` will be treated as a new local. Disambiguate by using `local pcmc` to suppress this warning or `global pcmc` to assign to the existing global variable. └ @ none:2 rf_ds2 fold: 1, 0.4915254237288136 fold: 2, 0.5034013605442177 fold: 3, 0.4915254237288136 fold: 4, 0.5408163265306123 fold: 5, 0.49491525423728816 errors: 0 AdaBoostClassifier_CwM fold: 1, 0.576271186440678 fold: 2, 0.5544217687074829 fold: 3, 0.5084745762711864 fold: 4, 0.5204081632653061 fold: 5, 0.5491525423728814 errors: 0 SGDClassifier_p1H fold: 1, 0.4915254237288136 fold: 2, 0.45918367346938777 fold: 3, 0.4915254237288136 fold: 4, 0.5136054421768708 fold: 5, 0.49830508474576274 errors: 0 prunetree_SaL fold: 1, 0.488135593220339 fold: 2, 0.5136054421768708 fold: 3, 0.48135593220338985 fold: 4, 0.4387755102040816 fold: 5, 0.488135593220339 errors: 0 GradientBoostingClassifier_fRr fold: 1, 0.5932203389830508 fold: 2, 0.564625850340136 fold: 3, 0.5898305084745763 fold: 4, 0.5510204081632653 fold: 5, 0.5559322033898305 errors: 0
julia> @show learners;
learners = 5×4 DataFrame
Row │ name mean sd kfold
│ String Float64 Float64 Int64
─────┼────────────────────────────────────────────────────────────
1 │ rf_ds2 0.504437 0.0209082 5
2 │ AdaBoostClassifier_CwM 0.541746 0.027248 5
3 │ SGDClassifier_p1H 0.490829 0.0198561 5
4 │ prunetree_SaL 0.482002 0.0271246 5
5 │ GradientBoostingClassifier_fRr 0.570926 0.0194625 5This time, Gradient boost has the best performance.