7 Scikit-learn Tricks for Hyperparameter Tuning - KDnuggets

www.kdnuggets.com

7 Scikit-learn Tricks for Hyperparameter Tuning
Image by Editor

Introduction

Tuning hyperparameters in machine learning models is, to some extent, an art or craftsmanship, requiring the right skills to balance experience, intuition, and plenty of experimentation. In practice, the process might sometimes appear daunting because sophisticated models have a large search space, interactions between hyperparameters are complex, and performance gains due to their adjustment are sometimes subtle.

Below, we curate a list that contains 7 Scikit-learn tricks for taking your machine learning models' hyperparameter tuning skills to the next level.

1. Constraining Search Space with Domain Knowledge

Not constraining an otherwise vast search space means looking for a needle in the middle of a (large) haystack! Resort to domain knowledge — or a domain expert, if necessary — to firstly define a set of well-chosen bounds for some relevant hyperparameters in your model. This will help reduce complexity and increase the feasibility of the running process, ruling out implausible settings.

An example grid for two typical hyperparameters in a random forest examples could look like:

param_grid = {"max_depth": [3, 5, 7], "min_samples_split": [2, 10]}

2. Starting Broadly with Random Search

For low-budget contexts, try leveraging random search, an efficient approach to explore large search spaces, by incorporating a distribution-driven sampling process that samples some hyperparameter value ranges. Just like in this example for sampling over C, i.e. the hyperparameter that controls the rigidness in the boundaries of SVM models:

param_dist = {"C": loguniform(1e-3, 1e2)}
RandomizedSearchCV(SVC(), param_dist, n_iter=20)

3. Refining Locally with Grid Search

After finding promising regions with a random search, it is sometimes a good idea to apply a narrow-focus grid search to further explore those regions to identify marginal gains. Exploration first, exploitation follows.

GridSearchCV(SVC(), {"C": [5, 10], "gamma": [0.01, 0.1]})

4. Encapsulating Preprocessing Pipelines within Hyperparameter Tuning

Scikit-learn pipelines are a great way to simplify and optimize end-to-end machine learning workflows and prevent issues like data leakage. Both preprocessing and model hyperparameters can be tuned together if we pass a pipeline to the search instance, as follows:

param_grid = {
    "scaler__with_mean": [True, False],  # Scaling hyperparameter
    "clf__C": [0.1, 1, 10],              # SVM model hyperparameter
    "clf__kernel": ["linear", "rbf"]     # Another SVM hyperparameter
}

grid_search = GridSearchCV(pipeline, param_grid, cv=5)
grid_search.fit(X_train, y_train)

5. Trading Speed for Reliability with Cross-validation

While applying cross-validation is the norm in Scikit-learn-driven hyperparameter tuning, it is worth understanding that omitting it means a single train-validation split is utilized: this is faster but yields more variable and sometimes less reliable results. Increasing the number of cross-validation folds — e.g. cv=5 — increases stability in performance for the sake of comparisons among models. Find a value that strikes the right balance for you:

GridSearchCV(model, params, cv=5)

6. Optimizing Multiple Metrics

When several performance trade-offs exist, having your tuning process monitor several metrics helps reveal compromises that may be inadvertent when applying single-score optimization. Besides, you can use refit to specify the main objective for determining the final, "best" model.

from sklearn.model_selection import GridSearchCV

param_grid = {
    "C": [0.1, 1, 10],
    "gamma": [0.01, 0.1]
}

scoring = {
    "accuracy": "accuracy",
    "f1": "f1"
}

gs = GridSearchCV(
    SVC(),
    param_grid,
    scoring=scoring,
    refit="f1",   # metric used to select the final model
    cv=5
)

gs.fit(X_train, y_train)

7. Interpreting Results Wisely

Once your tuning process ends, and the best-score model has been found, go the extra mile by using cv_results_ to better comprehend parameter interactions, trends, etc., or if you like, perform a visualization of results. This example builds a report and ranking of results for a grid search object named gs, after having completed the search and training process:

import pandas as pd

results_df = pd.DataFrame(gs.cv_results_)

# Target columns for our report
columns_to_show = [
    'param_clf__C',
    'mean_test_score',
    'std_test_score',
    'mean_fit_time',
    'rank_test_score'
]

print(results_df[columns_to_show].sort_values('rank_test_score'))

Wrapping Up

Hyperparameter tuning is most effective when it is both systematic and thoughtful. By combining smart search strategies, proper validation, and careful interpretation of results, you can extract meaningful performance gains without wasting compute or overfitting. Treat tuning as an iterative learning process, not just an optimization checkbox.

Iván Palomares Carrascosa is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world.

Feeds