OPPA European Social Fund Prague & EU: We invest in your future.

Talk Outline appearance based tracking patch similarity using histogram tracking by mean shift experiments, discussion Mean shift Tomáš Svoboda, svoboda@cmp.felk.cvut.cz Czech Technical University in Prague, Center for Machine Perception http://cmp.felk.cvut.cz Last update: April 8, 23 Please note that the lecture will be accompanied be several sketches and derivations on the blackboard and few live-interactive demos in Matlab

Appearance based tracking 2/2 2 2 illustration from []

Histogram based representation 3/2 image patch 3 histogram for 6 bins 5 25 5 2 25 frequency 2 5 5 3 5 5 2 25 5 5 2 25 intensity

Patch comparison 4/2 image patch 3 histogram for 6 bins 5 25 5 2 25 frequency 2 5 5 3 5 5 2 25 5 5 2 25 image patch 5 5 2 25 frequency 5 5 2 25 intensity 8 7 6 5 4 3 2 histogram for 6 bins 5 5 2 25 intensity

histogram difference assume normalized histograms, i.e m u= p u = 5 image patch 3 25 histogram for 6 bins 5/2 2 d = ρ[p, q] 5 2 frequency 5 where ρ[p, q] is the Bhattacharyya coefficient 25 3 5 5 2 25 5 5 5 2 25 intensity ρ[p, q] = m u= pu q u 5 5 image patch frequency 8 7 6 5 4 3 histogram for 6 bins 2 2 25 5 5 2 25 5 5 2 25 intensity

Similarity measured by the Bhattacharyya coefficient 6/2 5 5 2 25 3 35 4 45 5 55 Similarity surface of the frame 26 2 3 4 5 6 7.9.8.7.6.5.4.3.2. The object is the 4 plate and the model is histogram of image intensities. s(y) = m u= pu (y)q u where p(y) is the histogram of image patch at position y and q is the histogram of the template.

Problem: finding modes in probability density 7/2 5 5 2 25 3 35 4 45 5 55 Similarity surface of the frame 26 2 3 4 5 6 7.9.8.7.6.5.4.3.2. the complete enumeration of similarity surface can be costly, can we do it faster and more elegantly?

Density Gradient Estimation 8/2

Mean shift procedure 9/2 3 3 Figure borrowed from [4]

Meanshift segmentation of colours - color distribution /2 4 4 Figure from [2]

Meanshift segmentation of colours - color modes seeking /2 5 5 Figure from [2]

Mean shift segmentation - intensity and space 2/2 u, v are here spatial pixel coordinates different normalization for intensity and spatial coordinates

Multivariate kernel density estimator 3/2 Normal kernel, K N (x) = exp( 2 x 2 ) Epanechnikov kernel, K E (x) = x 2.8.5.6.4.5.2.5.5.5.5.5.5.5.5 Given n data points x i in d-dimensional space R d. f h,k (x) = nh d n i= K ( ) x xi h

Multivariate kernel density estimator 3/2 Normal kernel, K N (x) = exp( 2 x 2 ) Epanechnikov kernel, K E (x) = x 2.8.5.6.4.5.2.5.5.5.5.5.5.5.5 Given n data points x i in d-dimensional space R d. f h,k (x) = nh d n i= K ( ) x xi h looking for extremum of fh,k(x)

Multivariate kernel density estimator 3/2 Normal kernel, K N (x) = exp( 2 x 2 ) Epanechnikov kernel, K E (x) = x 2.8.5.6.4.5.2.5.5.5.5.5.5.5.5 Given n data points x i in d-dimensional space R d. f h,k (x) = nh d n i= K ( ) x xi h looking for extremum of fh,k(x) gradient fh,k(x) =

Kernels 4/2 Normal kernel, K N (x) = exp( 2 x 2 ) Epanechnikov kernel, K E (x) = x 2.8.5.6.4.5.2.5.5.5.5.5.5.5.5 Can be seen as membership function. Remind Kernel density estimation (Parzen method). Remember convolution? 6 6 Taken from http://en.wikipedia.org/wiki/kernel_density_estimation

Mean-shift iterations 5/2 Assuming a reasonable differentiable kernel K, iterate till convergence: y k+ = n i= x ig( y k x i 2 ) n i= g( y k x i 2 ) g is the derivative of kernel profile.

Mean-shift tracking - ratio histogram Ratio histogram: r u = min ( ) qu, p u 6/2 where q is the histogram of the target and p is the histogram of the current frame. w i = r b(xi ) (just binning) Image intensities (or colors) are tranformed into weights, w i, by back projection of the ratio histogram. Mean-shift iterations: y k+ = n i= w ix i g( y k x i 2 ) n i= w ig( y k x i 2 )

Mean-shift tracking - Bhattacharya coeficient model, coordinates x i centered at : 7/2 q u = C n i= k( x i 2 )δ(b(x i ) u) target candidate centered at y: n h p u (y) = C h i= k ( y x i h 2) δ(b(x i ) u)

ms tracking - object and its model 8/2 7 7 Figure from [5]

ms tracking - iterations 9/2 8 8 Figure from [5]

Mean-shift originally from [3]. References 2/2 [] Robert Collins. CSE/EE486 Computer Vision I. slides, web page. http://www.cse.psu.edu/~rcollins/cse486/. Robert kindly gave general permission to reuse the material. [2] Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Analysis, 24(5):63 69, May 22. [3] Keinosuke Fukunaga and Larry D. Hostetler. The estimation of the gradient of a density function, with appilcations in pattern recognition. IEEE Transactions on Information Theory, 2():32 4, January 975. [4] Milan Šonka, Václav Hlaváč, and Roger Boyle. Image Processing, Analysis and Machine Vision. Thomson, 3rd edition, 27. [5] Tomáš Svoboda, Jan Kybic, and Václav Hlaváč. Image Processing, Analysis and Machine Vision. A MATLAB Companion. Thomson, 27. Accompanying www site http://visionbook.felk.cvut.cz.

End 2/2

image patch 3 histogram for 6 bins 5 25 5 2 25 frequency 2 5 5 3 5 5 2 25 5 5 2 25 intensity

image patch 8 7 histogram for 6 bins 5 5 2 frequency 6 5 4 3 2 25 5 5 2 25 5 5 2 25 intensity

image patch 3 histogram for 6 bins 5 25 5 2 25 frequency 2 5 5 3 5 5 2 25 5 5 2 25 intensity

image patch 8 7 histogram for 6 bins 5 5 2 frequency 6 5 4 3 2 25 5 5 2 25 5 5 2 25 intensity

Similarity surface of the frame 26 5 5 2 25 3 35 4 45 5 55 2 3 4 5 6 7.9.8.7.6.5.4.3.2.

Normal kernel, K N (x) = exp( 2 x 2 ).8.6.4.2.5.5.5.5

Epanechnikov kernel, K E (x) = x 2.5.5.5.5.5.5

Normal kernel, K N (x) = exp( 2 x 2 ).8.6.4.2.5.5.5.5

Epanechnikov kernel, K E (x) = x 2.5.5.5.5.5.5

OPPA European Social Fund Prague & EU: We invest in your future.