How to evaluate a classifier in scikit-learn

In this video, you'll learn how to properly evaluate a classification model using a variety of common tools and metrics, as well as how to adjust the performance of a classifier to best match your business objectives. I'll start by demonstrating the weaknesses of classification accuracy as an evaluation metric. I'll then discuss the confusion matrix, the ROC curve and AUC, and metrics such as sensitivity, specificity, and precision. By the end of the video, you will have a solid foundation for intelligently evaluating your own classification model.

Download the notebook: https://github.com/justmarkham/scikit-learn-videos

== CONFUSION MATRIX RESOURCES ==
Simple guide to confusion matrix terminology: https://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/
Intuitive sensitivity and specificity: https://www.youtube.com/watch?v=U4_3fditnWg
The tradeoff between sensitivity and specificity: https://www.youtube.com/watch?v=vtYDyGGeQyo
How to calculate "expected value" from a confusion matrix: https://github.com/podopie/DAT18NYC/blob/master/classes/13-expected_value_cost_benefit_analysis.ipynb
Classification threshold graphic: https://media.amazonwebservices.com/blog/2015/ml_adjust_model_1.png

== ROC/AUC RESOURCES ==
ROC Curves and Area Under the Curve: https://www.youtube.com/watch?v=OAl6eAyP-yo
ROC visualization: http://www.navan.name/roc/
ROC Curves: https://www.youtube.com/watch?v=21Igj5Pr6u4
An introduction to ROC analysis: http://people.inf.elte.hu/kiss/13dwhdm/roc.pdf
Comparing different feature sets: http://research.microsoft.com/pubs/205472/aisec10-leontjeva.pdf
Comparing different classifiers: http://www.cse.ust.hk/nevinZhangGroup/readings/yi/Bradley_PR97.pdf

== OTHER RESOURCES ==
scikit-learn documentation on model evaluation: http://scikit-learn.org/stable/modules/model_evaluation.html
Comparing model evaluation procedures and metrics: https://github.com/justmarkham/DAT8/blob/master/other/model_evaluation_comparison.md
Counterfactual evaluation of machine learning models: https://www.youtube.com/watch?v=QWCSxAKR-h0

WANT TO GET BETTER AT MACHINE LEARNING? HERE ARE YOUR NEXT STEPS:

1) WATCH my scikit-learn video series:
https://www.youtube.com/playlist?list=PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A

2) SUBSCRIBE for more videos:
https://www.youtube.com/dataschool?sub_confirmation=1

3) JOIN "Data School Insiders" to access bonus content:
https://www.patreon.com/dataschool

4) ENROLL in my Machine Learning course:
https://www.dataschool.io/learn/

5) LET'S CONNECT!
- Newsletter: https://www.dataschool.io/subscribe/
- Twitter: https://twitter.com/justmarkham
- Facebook: https://www.facebook.com/DataScienceSchool/
- LinkedIn: https://www.linkedin.com/in/justmarkham/

Видео How to evaluate a classifier in scikit-learn канала Data School