Thresholding with Match Template - OpenCV Object Detection in Games #2
Detect multiple objects with OpenCV's match template function by using thresholding. In this tutorial, we dig into the details of how this works.
Full tutorial playlist: https://www.youtube.com/playlist?list=PL1m2M8LQlzfKtkKq2lK5xko4X-8EZzFPI
GitHub Repo: https://github.com/learncodebygaming/opencv_tutorials
OpenCV documentation: https://docs.opencv.org/4.2.0/
Official template matching tutorial: https://docs.opencv.org/4.2.0/d4/dc6/tutorial_py_template_matching.html
1:18 Understanding the result from matchTemplate()
2:25 Using the debugger in VSCode
5:08 Why is the match template result smaller than the original image?
6:18 Using numpy where() to filter the results
7:27 Converting np.where() to a list of pixel positions
9:41 Adjusting the threshold
10:11 Drawing multiple rectangles
12:27 Using TM_SQDIFF_NORMED
With thresholding, we can detect multiple objects using OpenCV's matchTemplate() function. They discuss briefly how to do this in the offical tutorial, but here I will cover it more in-depth.
Previously we used minMaxLoc() to get the best matching position for our needle image, but matchTemplate() actually returns a result matrix for all the positions it searched.
Each value in this matrix represents the confidence score for how closely the needle image matches the haystack image at a given position. The index of the outer dimension represents the Y position, and the index of the inner list represent the X position. For example, the confidence value 0.58679795 in the data [see: https://learncodebygaming.com/blog/thresholding-with-match-template] corresponds to Y = 1 and X = 6. When we overlay the needle image on the haystack image, such that the upper left corner of the needle is placed at pixel position (6, 1), it's match score is 0.58679795.
Note that the resulting matrix size is (haystack_w − needle_w + 1) * (haystack_h − needle_h + 1). This is because there are no meaningful match results when the needle image is partially overhanging the haystack image.
The idea with thresholding is, we want to get the coordinates of all the places where the match confidence score is above some threshold number that we set.
To do that, the documentation suggests we should use the np.where() function. This will give us all the locations above that threshold.
In the result, the first array contains the Y positions, and the second contains the X positions. So in the above example we found two matches above the threshold we set, at positions (1, 1) and (15, 5).
So we have the data we need know, but the format returned by np.where() isn't very convenient to work with. So lets convert it to a list of (X, Y) tuples: list(zip(*locations[::-1]))
In this line of code, [::-1] reverses a numpy array, so that we get the X values to come before the Y. Then the star * unpacks a list, so that we now have two one-dimensional arrays instead of one two-dimensional array. We then use zip() to merge those two lists into a bunch of new lists, each comprised of the elements from the input lists that share the same index. And because zip() actually returns a generator instead of a rendered list, we wrap it all in list() to get the final result we're looking for.
So that's how we've converted the Y and X arrays returned by np.where() into a list of (X, Y) tuples.
Now that we have a list of matching locations, let's draw rectangles around all of those locations. We'll adapt our code from part 1 to do this, and simply loop over all the locations.
Keep adjusting your threshold, and your comparison method, until you get as many correct matches as possible without any false positives. In this case, I achieved the best result by using cv.TM_SQDIFF_NORMED and setting a threshold of values below 0.17 (remember TM_SQDIFF_NORMED uses inverted results).
You'll no doubt notice that you are getting many more location results than the number of rectangles you're seeing on the result image. You'll also notice that some of the rectangles are much thicker than others. This indicates that you're getting many match results that are very close to one another, and your rectangles are overlapping each other.
Visually this may not be too problematic, but if you're trying to count up the number of some item in an image, or if you're searching for screen locations to click on, then you'll need some way to clean this up. And that's what we'll be covering in part 3 of this tutorial: how to group those overlapping rectangles into single detection results.
Read the full written tutorial here: https://learncodebygaming.com/blog/thresholding-with-match-template
Видео Thresholding with Match Template - OpenCV Object Detection in Games #2 канала Learn Code By Gaming
Full tutorial playlist: https://www.youtube.com/playlist?list=PL1m2M8LQlzfKtkKq2lK5xko4X-8EZzFPI
GitHub Repo: https://github.com/learncodebygaming/opencv_tutorials
OpenCV documentation: https://docs.opencv.org/4.2.0/
Official template matching tutorial: https://docs.opencv.org/4.2.0/d4/dc6/tutorial_py_template_matching.html
1:18 Understanding the result from matchTemplate()
2:25 Using the debugger in VSCode
5:08 Why is the match template result smaller than the original image?
6:18 Using numpy where() to filter the results
7:27 Converting np.where() to a list of pixel positions
9:41 Adjusting the threshold
10:11 Drawing multiple rectangles
12:27 Using TM_SQDIFF_NORMED
With thresholding, we can detect multiple objects using OpenCV's matchTemplate() function. They discuss briefly how to do this in the offical tutorial, but here I will cover it more in-depth.
Previously we used minMaxLoc() to get the best matching position for our needle image, but matchTemplate() actually returns a result matrix for all the positions it searched.
Each value in this matrix represents the confidence score for how closely the needle image matches the haystack image at a given position. The index of the outer dimension represents the Y position, and the index of the inner list represent the X position. For example, the confidence value 0.58679795 in the data [see: https://learncodebygaming.com/blog/thresholding-with-match-template] corresponds to Y = 1 and X = 6. When we overlay the needle image on the haystack image, such that the upper left corner of the needle is placed at pixel position (6, 1), it's match score is 0.58679795.
Note that the resulting matrix size is (haystack_w − needle_w + 1) * (haystack_h − needle_h + 1). This is because there are no meaningful match results when the needle image is partially overhanging the haystack image.
The idea with thresholding is, we want to get the coordinates of all the places where the match confidence score is above some threshold number that we set.
To do that, the documentation suggests we should use the np.where() function. This will give us all the locations above that threshold.
In the result, the first array contains the Y positions, and the second contains the X positions. So in the above example we found two matches above the threshold we set, at positions (1, 1) and (15, 5).
So we have the data we need know, but the format returned by np.where() isn't very convenient to work with. So lets convert it to a list of (X, Y) tuples: list(zip(*locations[::-1]))
In this line of code, [::-1] reverses a numpy array, so that we get the X values to come before the Y. Then the star * unpacks a list, so that we now have two one-dimensional arrays instead of one two-dimensional array. We then use zip() to merge those two lists into a bunch of new lists, each comprised of the elements from the input lists that share the same index. And because zip() actually returns a generator instead of a rendered list, we wrap it all in list() to get the final result we're looking for.
So that's how we've converted the Y and X arrays returned by np.where() into a list of (X, Y) tuples.
Now that we have a list of matching locations, let's draw rectangles around all of those locations. We'll adapt our code from part 1 to do this, and simply loop over all the locations.
Keep adjusting your threshold, and your comparison method, until you get as many correct matches as possible without any false positives. In this case, I achieved the best result by using cv.TM_SQDIFF_NORMED and setting a threshold of values below 0.17 (remember TM_SQDIFF_NORMED uses inverted results).
You'll no doubt notice that you are getting many more location results than the number of rectangles you're seeing on the result image. You'll also notice that some of the rectangles are much thicker than others. This indicates that you're getting many match results that are very close to one another, and your rectangles are overlapping each other.
Visually this may not be too problematic, but if you're trying to count up the number of some item in an image, or if you're searching for screen locations to click on, then you'll need some way to clean this up. And that's what we'll be covering in part 3 of this tutorial: how to group those overlapping rectangles into single detection results.
Read the full written tutorial here: https://learncodebygaming.com/blog/thresholding-with-match-template
Видео Thresholding with Match Template - OpenCV Object Detection in Games #2 канала Learn Code By Gaming
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
Fast Window Capture - OpenCV Object Detection in Games #4Grouping Rectangles into Click Points - OpenCV Object Detection in Games #3Template Matching - OpenCV with Python for Image and Video Analysis 11Starbase - Programming In Space? Pre-Alpha YOLOL Guide (March 2020)118 - Object detection by template matchingPerfecting our Woodcutting Bot - Learn JavaScript by playing Runescape 4/4OpenCV Object Detection in Games Python Tutorial #1Codewars Review & TipsRLBot Beginner Tutorial - Learn Python by playing Rocket League 1/3RobotJS Tutorial for Beginners - Learn JavaScript by playing RuneScape 2/4The SIMPLEST Screeps TutorialPython: Real Time Object Detection (Image, Webcam, Video files) with Yolov3 and OpenCVHow to make advanced image recognition bots using pythonColor Botting Basics - Learn JavaScript by playing RuneScape 3/4Your First PyAutoGUI Game Bot: PyAutoGUI Video Game Bot Tutorial 2/4Template Matching using OpenCV and PythonKalman Filter with OpenCV for calculation of trajectories in basketball.OpenCV - Python plays Grand Theft Auto 5 p.2PyAutoGUI not working? Use DirectInputStream Live Video used in Django using OpenCV