FREAK Descriptor
FREAK (Fast Retina Keypoint) has a defined sampling pattern like BRISK. It uses a retinal sampling grid with more density of points near the centre with the density decreasing exponentially with distance from the centre.
FREAK’s measure of orientation is similar to BRISK but instead of using long pairs, it uses a set of predefined 45 symmetric sampling pairs. The set of sampling pairs is determined using a method similar to ORB, by finding sampling pairs over keypoints in standard datasets and then extracting the most discriminative pairs. The orientation weights over these pairs are summed and the sampling window is rotated by this orientation to some canonical orientation to achieve rotation invariance.
The descriptor is built using intensity comparisons of a predetermined set of 512 sampling pairs. This set is also obtained using a method similar to the one described above. For each pair if the first point has greater intensity than the second, then 1 is written else 0 is written to the corresponding bit of the descriptor.
Example
Let us take a look at a simple example where the FREAK descriptor is used to match two images where one has been translated by (50, 40) pixels and then rotated by an angle of 75 degrees. We will use the lighthouse image from the TestImages package for this example.
First, let us create the two images we will match using FREAK.
using ImageFeatures, TestImages, Images, ImageDraw, CoordinateTransformations, Rotations
img = testimage("peppers_color")
Original
img1 = Gray.(img)
rot = recenter(RotMatrix(5pi/6), [size(img1)...] .÷ 2) # a rotation around the center
tform = rot ∘ Translation(-50, -40)
img2 = warp(img1, tform, axes(img1))
To calculate the descriptors, we first need to get the keypoints. For this tutorial, we will use the FAST corners to generate keypoints (see fastcorners).
keypoints_1 = Keypoints(fastcorners(img1, 12, 0.35))
keypoints_2 = Keypoints(fastcorners(img2, 12, 0.35))77-element Vector{CartesianIndex{2}}:
CartesianIndex(169, 22)
CartesianIndex(169, 23)
CartesianIndex(180, 24)
CartesianIndex(181, 24)
CartesianIndex(266, 57)
CartesianIndex(267, 57)
CartesianIndex(267, 58)
CartesianIndex(267, 59)
CartesianIndex(268, 59)
CartesianIndex(268, 60)
⋮
CartesianIndex(500, 465)
CartesianIndex(499, 467)
CartesianIndex(257, 503)
CartesianIndex(255, 504)
CartesianIndex(254, 505)
CartesianIndex(253, 506)
CartesianIndex(458, 510)
CartesianIndex(457, 511)
CartesianIndex(456, 512)To create the FREAK descriptor, we first need to define the parameters by calling the FREAK constructor.
freak_params = FREAK()FREAK(22.0, Vector{Tuple{Float16, Float16}}[[(0.0, 14.664), (12.7, 7.332), (12.7, -7.332), (0.0, -14.664), (-12.7, -7.332), (-12.7, 7.332), (5.5, 9.52), (11.0, 0.0), (5.5, -9.52), (-5.5, -9.52) … (-1.223, -2.117), (-2.445, -0.0), (-1.223, 2.117), (0.0, 1.833), (1.588, 0.9165), (1.588, -0.9165), (0.0, -1.833), (-1.588, -0.9165), (-1.588, 0.9165), (0.0, -0.0)], [(0.3599, 14.664), (12.875, 7.02), (12.516, -7.645), (-0.3599, -14.664), (-12.875, -7.02), (-12.516, 7.645), (5.73, 9.39), (11.0, -0.27), (5.266, -9.66), (-5.73, -9.39) … (-1.273, -2.086), (-2.443, 0.06), (-1.17, 2.146), (0.04498, 1.833), (1.609, 0.8774), (1.564, -0.9556), (-0.04498, -1.833), (-1.609, -0.8774), (-1.564, 0.9556), (-0.0, -0.0)], [(0.7197, 14.65), (13.05, 6.703), (12.33, -7.95), (-0.7197, -14.65), (-13.05, -6.703), (-12.33, 7.95), (5.96, 9.24), (10.984, -0.5396), (5.027, -9.78), (-5.96, -9.24) … (-1.324, -2.055), (-2.441, 0.11993), (-1.117, 2.174), (0.08997, 1.831), (1.631, 0.838), (1.541, -0.9937), (-0.08997, -1.831), (-1.631, -0.838), (-1.541, 0.9937), (-0.0, -0.0)], [(1.079, 14.625), (13.2, 6.38), (12.125, -8.25), (-1.079, -14.625), (-13.2, -6.38), (-12.125, 8.25), (6.188, 9.09), (10.97, -0.809), (4.785, -9.91), (-6.188, -9.09) … (-1.375, -2.021), (-2.438, 0.1798), (-1.063, 2.201), (0.1349, 1.828), (1.65, 0.7974), (1.516, -1.031), (-0.1349, -1.828), (-1.65, -0.7974), (-1.516, 1.031), (-0.0, -0.0)], [(1.4375, 14.59), (13.36, 6.055), (11.92, -8.55), (-1.4375, -14.59), (-13.36, -6.055), (-11.92, 8.55), (6.406, 8.94), (10.945, -1.078), (4.54, -10.016), (-6.406, -8.94) … (-1.424, -1.987), (-2.434, 0.2396), (-1.009, 2.227), (0.1797, 1.824), (1.67, 0.757), (1.49, -1.068), (-0.1797, -1.824), (-1.67, -0.757), (-1.49, 1.068), (-0.0, -0.0)], [(1.795, 14.555), (13.5, 5.723), (11.71, -8.836), (-1.795, -14.555), (-13.5, -5.723), (-11.71, 8.836), (6.625, 8.78), (10.914, -1.347), (4.293, -10.125), (-6.625, -8.78) … (-1.473, -1.951), (-2.426, 0.2993), (-0.954, 2.25), (0.2244, 1.819), (1.6875, 0.7153), (1.464, -1.1045), (-0.2244, -1.819), (-1.6875, -0.7153), (-1.464, 1.1045), (-0.0, -0.0)], [(2.152, 14.51), (13.64, 5.39), (11.484, -9.12), (-2.152, -14.51), (-13.64, -5.39), (-11.484, 9.12), (6.84, 8.62), (10.88, -1.614), (4.043, -10.23), (-6.84, -8.62) … (-1.52, -1.915), (-2.418, 0.3586), (-0.8984, 2.273), (0.269, 1.813), (1.705, 0.674), (1.436, -1.14), (-0.269, -1.813), (-1.705, -0.674), (-1.436, 1.14), (-0.0, -0.0)], [(2.508, 14.45), (13.766, 5.055), (11.26, -9.4), (-2.508, -14.45), (-13.766, -5.055), (-11.26, 9.4), (7.047, 8.445), (10.836, -1.881), (3.791, -10.33), (-7.047, -8.445) … (-1.566, -1.877), (-2.408, 0.418), (-0.8423, 2.295), (0.3135, 1.807), (1.721, 0.632), (1.407, -1.175), (-0.3135, -1.807), (-1.721, -0.632), (-1.407, 1.175), (-0.0, -0.0)], [(2.861, 14.38), (13.89, 4.715), (11.02, -9.67), (-2.861, -14.38), (-13.89, -4.715), (-11.02, 9.67), (7.254, 8.27), (10.79, -2.146), (3.535, -10.414), (-7.254, -8.27) … (-1.611, -1.838), (-2.398, 0.4768), (-0.7856, 2.314), (0.3577, 1.798), (1.736, 0.5894), (1.378, -1.209), (-0.3577, -1.798), (-1.736, -0.5894), (-1.378, 1.209), (-0.0, -0.0)], [(3.213, 14.31), (14.0, 4.37), (10.79, -9.94), (-3.213, -14.31), (-14.0, -4.37), (-10.79, 9.94), (7.453, 8.086), (10.734, -2.41), (3.28, -10.5), (-7.453, -8.086) … (-1.656, -1.798), (-2.385, 0.5356), (-0.7285, 2.334), (0.4016, 1.789), (1.75, 0.5464), (1.349, -1.242), (-0.4016, -1.789), (-1.75, -0.5464), (-1.349, 1.242), (-0.0, -0.0)] … [(-3.564, 14.23), (10.54, 10.2), (14.1, -4.027), (3.564, -14.23), (-10.54, -10.2), (-14.1, 4.027), (3.02, 10.58), (10.67, 2.672), (7.65, -7.906), (-3.02, -10.58) … (-0.6714, -2.35), (-2.371, -0.5938), (-1.7, 1.757), (-0.4456, 1.778), (1.317, 1.275), (1.763, -0.5034), (0.4456, -1.778), (-1.317, -1.275), (-1.763, 0.5034), (0.0, -0.0)], [(-3.213, 14.31), (10.79, 9.94), (14.0, -4.37), (3.213, -14.31), (-10.79, -9.94), (-14.0, 4.37), (3.28, 10.5), (10.734, 2.41), (7.453, -8.086), (-3.28, -10.5) … (-0.7285, -2.334), (-2.385, -0.5356), (-1.656, 1.798), (-0.4016, 1.789), (1.349, 1.242), (1.75, -0.5464), (0.4016, -1.789), (-1.349, -1.242), (-1.75, 0.5464), (0.0, -0.0)], [(-2.861, 14.38), (11.02, 9.67), (13.89, -4.715), (2.861, -14.38), (-11.02, -9.67), (-13.89, 4.715), (3.535, 10.414), (10.79, 2.146), (7.254, -8.27), (-3.535, -10.414) … (-0.7856, -2.314), (-2.398, -0.4768), (-1.611, 1.838), (-0.3577, 1.798), (1.378, 1.209), (1.736, -0.5894), (0.3577, -1.798), (-1.378, -1.209), (-1.736, 0.5894), (0.0, -0.0)], [(-2.508, 14.45), (11.26, 9.4), (13.766, -5.055), (2.508, -14.45), (-11.26, -9.4), (-13.766, 5.055), (3.791, 10.33), (10.836, 1.881), (7.047, -8.445), (-3.791, -10.33) … (-0.8423, -2.295), (-2.408, -0.418), (-1.566, 1.877), (-0.3135, 1.807), (1.407, 1.175), (1.721, -0.632), (0.3135, -1.807), (-1.407, -1.175), (-1.721, 0.632), (0.0, -0.0)], [(-2.152, 14.51), (11.484, 9.12), (13.64, -5.39), (2.152, -14.51), (-11.484, -9.12), (-13.64, 5.39), (4.043, 10.23), (10.88, 1.614), (6.84, -8.62), (-4.043, -10.23) … (-0.8984, -2.273), (-2.418, -0.3586), (-1.52, 1.915), (-0.269, 1.813), (1.436, 1.14), (1.705, -0.674), (0.269, -1.813), (-1.436, -1.14), (-1.705, 0.674), (0.0, -0.0)], [(-1.795, 14.555), (11.71, 8.836), (13.5, -5.723), (1.795, -14.555), (-11.71, -8.836), (-13.5, 5.723), (4.293, 10.125), (10.914, 1.347), (6.625, -8.78), (-4.293, -10.125) … (-0.954, -2.25), (-2.426, -0.2993), (-1.473, 1.951), (-0.2244, 1.819), (1.464, 1.1045), (1.6875, -0.7153), (0.2244, -1.819), (-1.464, -1.1045), (-1.6875, 0.7153), (0.0, -0.0)], [(-1.4375, 14.59), (11.92, 8.55), (13.36, -6.055), (1.4375, -14.59), (-11.92, -8.55), (-13.36, 6.055), (4.54, 10.016), (10.945, 1.078), (6.406, -8.94), (-4.54, -10.016) … (-1.009, -2.227), (-2.434, -0.2396), (-1.424, 1.987), (-0.1797, 1.824), (1.49, 1.068), (1.67, -0.757), (0.1797, -1.824), (-1.49, -1.068), (-1.67, 0.757), (0.0, -0.0)], [(-1.079, 14.625), (12.125, 8.25), (13.2, -6.38), (1.079, -14.625), (-12.125, -8.25), (-13.2, 6.38), (4.785, 9.91), (10.97, 0.809), (6.188, -9.09), (-4.785, -9.91) … (-1.063, -2.201), (-2.438, -0.1798), (-1.375, 2.021), (-0.1349, 1.828), (1.516, 1.031), (1.65, -0.7974), (0.1349, -1.828), (-1.516, -1.031), (-1.65, 0.7974), (0.0, -0.0)], [(-0.7197, 14.65), (12.33, 7.95), (13.05, -6.703), (0.7197, -14.65), (-12.33, -7.95), (-13.05, 6.703), (5.027, 9.78), (10.984, 0.5396), (5.96, -9.24), (-5.027, -9.78) … (-1.117, -2.174), (-2.441, -0.11993), (-1.324, 2.055), (-0.08997, 1.831), (1.541, 0.9937), (1.631, -0.838), (0.08997, -1.831), (-1.541, -0.9937), (-1.631, 0.838), (0.0, -0.0)], [(-0.3599, 14.664), (12.516, 7.645), (12.875, -7.02), (0.3599, -14.664), (-12.516, -7.645), (-12.875, 7.02), (5.266, 9.66), (11.0, 0.27), (5.73, -9.39), (-5.266, -9.66) … (-1.17, -2.146), (-2.443, -0.06), (-1.273, 2.086), (-0.04498, 1.833), (1.564, 0.9556), (1.609, -0.8774), (0.04498, -1.833), (-1.564, -0.9556), (-1.609, 0.8774), (0.0, -0.0)]], Vector{Float16}[[7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165] … [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165], [7.332, 7.332, 7.332, 7.332, 7.332, 7.332, 5.5, 5.5, 5.5, 5.5 … 1.223, 1.223, 1.223, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165, 0.9165]], Tuple{Float16, Float16}[(0.0, 0.0341), (0.02953, 0.01704), (0.02953, -0.01704), (-0.01968, 0.0341), (0.01968, 0.0341), (0.03937, 0.0), (0.01968, -0.0341), (-0.01968, -0.0341), (-0.03937, 0.0), (0.02274, 0.03937) … (-0.0909, 0.05246), (0.0, 0.1364), (0.1181, 0.0682), (0.1181, -0.0682), (0.10223, 0.177), (0.2045, 0.0), (0.10223, -0.177), (0.0, 0.2727), (0.2362, 0.1364), (0.2362, -0.1364)])Now pass the image with the keypoints and the parameters to the create_descriptor function.
desc_1, ret_keypoints_1 = create_descriptor(img1, keypoints_1, freak_params)
desc_2, ret_keypoints_2 = create_descriptor(img2, keypoints_2, freak_params)(BitVector[[0, 1, 0, 0, 0, 1, 0, 0, 0, 0 … 0, 0, 1, 0, 0, 0, 0, 0, 0, 0], [1, 0, 1, 0, 1, 0, 1, 0, 0, 0 … 0, 0, 1, 1, 0, 0, 0, 0, 0, 0], [1, 0, 0, 1, 0, 0, 1, 0, 0, 0 … 0, 1, 0, 1, 0, 0, 0, 0, 0, 0], [0, 1, 0, 0, 0, 0, 0, 0, 0, 0 … 0, 0, 0, 1, 0, 0, 0, 0, 0, 0], [0, 1, 1, 0, 0, 1, 0, 0, 1, 0 … 0, 1, 0, 1, 0, 0, 0, 0, 0, 0], [1, 0, 0, 0, 0, 1, 0, 0, 0, 0 … 0, 0, 0, 1, 0, 0, 0, 0, 0, 0], [0, 1, 0, 0, 0, 1, 0, 0, 1, 0 … 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 1, 0, 0, 0, 0 … 0, 1, 0, 1, 0, 0, 0, 0, 0, 0], [0, 1, 0, 0, 0, 0, 0, 0, 1, 0 … 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 1, 0, 0, 0, 0, 0, 0, 1, 0 … 0, 1, 0, 0, 0, 0, 0, 0, 0, 0] … [0, 1, 0, 1, 1, 0, 0, 1, 1, 0 … 1, 1, 0, 1, 0, 1, 1, 1, 1, 1], [0, 1, 0, 1, 0, 0, 0, 1, 1, 1 … 1, 0, 0, 1, 0, 1, 1, 1, 1, 1], [0, 1, 0, 1, 1, 0, 0, 1, 1, 1 … 1, 1, 0, 1, 0, 1, 1, 1, 1, 1], [0, 1, 0, 1, 1, 0, 0, 1, 1, 1 … 1, 1, 0, 1, 0, 1, 1, 1, 1, 1], [1, 0, 1, 1, 1, 1, 1, 1, 0, 1 … 1, 0, 0, 1, 0, 1, 0, 0, 1, 1], [1, 0, 1, 1, 1, 1, 0, 1, 0, 1 … 1, 0, 0, 1, 0, 1, 0, 1, 1, 1], [1, 1, 1, 0, 0, 0, 0, 1, 1, 0 … 1, 0, 0, 0, 0, 1, 0, 1, 1, 0], [1, 0, 0, 1, 0, 1, 0, 0, 1, 1 … 1, 0, 0, 1, 0, 0, 0, 0, 1, 1], [1, 1, 0, 1, 0, 1, 0, 0, 1, 1 … 0, 0, 0, 1, 0, 0, 0, 0, 0, 1], [0, 1, 0, 1, 0, 0, 0, 0, 1, 1 … 0, 0, 0, 1, 0, 1, 0, 0, 0, 1]], CartesianIndex{2}[CartesianIndex(180, 24), CartesianIndex(181, 24), CartesianIndex(266, 57), CartesianIndex(267, 57), CartesianIndex(267, 58), CartesianIndex(267, 59), CartesianIndex(268, 59), CartesianIndex(268, 60), CartesianIndex(269, 60), CartesianIndex(269, 61) … CartesianIndex(289, 278), CartesianIndex(289, 279), CartesianIndex(288, 280), CartesianIndex(288, 281), CartesianIndex(62, 291), CartesianIndex(62, 292), CartesianIndex(298, 295), CartesianIndex(382, 304), CartesianIndex(382, 306), CartesianIndex(382, 307)])The obtained descriptors can be used to find the matches between the two images using the match_keypoints function.
matches = match_keypoints(ret_keypoints_1, ret_keypoints_2, desc_1, desc_2, 0.1)12-element Vector{Vector{CartesianIndex{2}}}:
[CartesianIndex(410, 443), CartesianIndex(266, 57)]
[CartesianIndex(409, 443), CartesianIndex(267, 57)]
[CartesianIndex(408, 442), CartesianIndex(267, 58)]
[CartesianIndex(409, 442), CartesianIndex(267, 59)]
[CartesianIndex(406, 441), CartesianIndex(268, 60)]
[CartesianIndex(278, 339), CartesianIndex(329, 213)]
[CartesianIndex(282, 331), CartesianIndex(321, 218)]
[CartesianIndex(193, 382), CartesianIndex(424, 218)]
[CartesianIndex(192, 383), CartesianIndex(425, 218)]
[CartesianIndex(285, 296), CartesianIndex(301, 247)]
[CartesianIndex(469, 138), CartesianIndex(62, 292)]
[CartesianIndex(263, 253), CartesianIndex(298, 295)]We can use the ImageDraw.jl package to view the results.
grid = hcat(img1, img2)
offset = CartesianIndex(0, size(img1, 2))
map(m -> draw!(grid, LineSegment(m[1], m[2] + offset)), matches)
grid
This page was generated using DemoCards.jl and Literate.jl.