profile

Learn Data Science from Data School 📊

Tuesday Tip #14: Ensemble your ML models for better performance 🎯

Published 12 months ago • 1 min read

Hi Reader,

Last week, I had an hour-long chat with my friend Ken Jee for his podcast. Starting tomorrow, you can watch the conversation on the Ken’s Nearest Neighbors YouTube channel!


👉 Tip #14: How to ensemble your ML models

When working on a Machine Learning problem, it’s always a good idea to try different types of models to see which one performs best.

However, you can also use a process called “ensembling” to combine multiple models. The goal is to produce a combined model, known as an ensemble, that performs better than any of the individual models.

The process for ensembling is simple:

  • For a regression problem, you calculate the average of the predictions made by the individual regressors and use that as your prediction.
  • For a classification problem, you can either average the predicted probabilities output by the classifiers, or you can let the classifiers vote on which class to predict.

The idea behind ensembling is that if you have a collection of individually imperfect models, the “one-off” errors made by each model are probably not going to be made by the rest of the models. Thus, the errors will be discarded (or at least reduced) when ensembling the models.


Ensembling in scikit-learn

Here’s a simple example in which I ensembled Logistic Regression and Random Forests using scikit-learn’s VotingClassifier:

Notice that the accuracy of the ensemble (0.725) is significantly better than the accuracy of either individual model. (Check out the full code here.)


Advice for ensembling

➡️ Ensembling is useful any time model accuracy (or another evaluation metric) is your highest priority. Keep in mind that the ensemble will be less interpretable than the individual models.

➡️ It’s ideal to include at least 3 models in the ensemble.

➡️ It’s important that all models you include are performing reasonably well on their own.

➡️ It’s best if the included models generate their predictions using different processes, since they will be likely to make different types of errors. (This is what makes Logistic Regression and Random Forests good candidates for ensembling!)


If you enjoyed this week’s tip, please forward it to a friend! Takes only a few seconds, and it really helps me out! 🙌

See you next Tuesday!

- Kevin

P.S. Gym rats vs data scientists

Did someone awesome forward you this email? Sign up here to receive data science tips every week!

Learn Data Science from Data School 📊

Kevin Markham

Join 25,000+ aspiring Data Scientists and receive Python & Data Science tips every Tuesday!

Read more from Learn Data Science from Data School 📊

Hi Reader, happy Tuesday! My recent tips have been rather lengthy, so I'm going to mix it up with some shorter tips (like today's). Let me know what you think! 💬 🔗 Link of the week A stealth attack came close to compromising the world's computers (The Economist) If you haven't heard about the recent "xz Utils backdoor", it's an absolutely fascinating/terrifying story! In short, a hacker (or team of hackers) spent years gaining the trust of an open-source project by making helpful...

13 days ago • 1 min read

Hi Reader, Today's tip is drawn directly from my upcoming course, Master Machine Learning with scikit-learn. You can read the tip below or watch it as a video! If you're interested in receiving more free lessons from the course (which won't be included in Tuesday Tips), you can join the waitlist by clicking here: Yes, I want more free lessons! 👉 Tip #43: Should you discretize continuous features for Machine Learning? Let's say that you're working on a supervised Machine Learning problem, and...

20 days ago • 2 min read

Hi Reader, I'm so excited to share this week's tip with you! It has been in my head for months, but I finally put it in writing ✍️ It's longer than usual, so if you prefer, you can read it as a blog post instead: Jupyter & IPython terminology explained 🔗 Link of the week Python Problem-Solving Bootcamp (April 1-21) Want to improve your Python skills quickly? There's no better way than solving problems, reviewing alternative solutions, and exchanging ideas with others. That's the idea behind...

about 1 month ago • 3 min read
Share this post