Aalysis of Algorithms
Aalysis Fram Most algorithms trasform iput objects ito output objects. The ruig time of a algorithm typically grows with the iput size. Average case time is ofte difficult to determie. We focus o the worst case ruig time. Easier to aalyze Crucial to applicatios such as games, fiace ad robotics Ruig Time 20 00 80 60 40 20 0 best case average case worst case 000 2000 3000 4000 Iput Size 2
Aalysis Framework Time Efficiecy Space efficiecy Write a program implemetig the algorithm with iputs of varyig size ad compositio Use a method like System.curretTimeMillis() to get a accurate measure of the actual ruig time Plot the results Time (ms) 9000 8000 7000 6000 5000 4000 3000 2000 000 0 0 50 00 Iput Size 3
Time Efficiecy It is ecessary to implemet the algorithm, which may be difficult Results may ot be idicative of the ruig time o other iputs ot icluded i the experimet. I order to compare two algorithms, the same hardware ad software eviromets must be used 4
Theoretical Aalysis for Time Efficiecy Uses a high-level descriptio of the algorithm istead of a implemetatio Characterizes ruig time as a fuctio of the iput size,. Takes ito accout all possible iputs Allows us to evaluate the speed of a algorithm idepedet of the hardware/software eviromet 5
Pseudocode (.) High-level descriptio of a algorithm More structured tha Eglish prose Less detailed tha a program Preferred otatio for describig algorithms Hides program desig issues Example: fid max elemet of a array Algorithm arraymax(a, ) Iput array A of itegers Output maximum elemet of A curretmax A[0] for i to do if A[i] > curretmax the curretmax A[i] retur curretmax 6
Pseudocode Details Cotrol flow if the [else ] while do repeat util for do Idetatio replaces braces Method declaratio Algorithm method (arg [, arg ]) Iput Output Method call var.method (arg [, arg ]) Retur value retur expressio Expressios Assigmet (like = i Java) = Equality testig (like == i Java) 2 Superscripts ad other mathematical formattig allowed 7
The Radom Access Machie (RAM) Model A CPU A potetially ubouded bak of memory cells, each of which ca hold a arbitrary umber or character 0 2 Memory cells are umbered ad accessig ay cell i memory takes uit time. 8
Primitive Operatios Basic computatios performed by a algorithm Idetifiable i pseudocode Largely idepedet from the programmig laguage Exact defiitio ot importat Assumed to take a costat amout of time i the RAM model Examples: Evaluatig a expressio Assigig a value to a variable Idexig ito a array Callig a method Returig from a method 9
Coutig Primitive Operatios By ispectig the pseudocode, we ca determie the maximum umber of primitive operatios executed by a algorithm, as a fuctio of the iput size Algorithm arraymax(a, ) # operatios curretmax A[0] 2 for i to do 2 + if A[i] > curretmax the 2( ) curretmax A[i] 2( ) { icremet couter i } 2( ) retur curretmax Total 7 0
Estimatig Ruig Time Algorithm arraymax executes 7 primitive operatios i the worst case. Defie: a = Time take by the fastest primitive operatio b = Time take by the slowest primitive operatio Let T() be worst-case time of arraymax. The a (7 ) T() b(7 ) Hece, the ruig time T() is bouded by two liear fuctios
Growth Rate of Ruig Time Chagig the hardware/ software eviromet Affects T() by a costat factor, but Does ot alter the growth rate of T() The liear growth rate of the ruig time T() is a itrisic property of algorithm arraymax 2
Growth Rates Growth rates of fuctios: Liear Quadratic 2 Cubic 3 I a log-log chart, the slope of the lie correspods to the growth rate of the fuctio T ( ) E+30 E+28 E+26 E+24 E+22 E+20 E+8 E+6 E+4 E+2 E+0 E+8 E+6 E+4 E+2 E+0 Cubic Quadratic Liear E+0 E+2 E+4 E+6 E+8 E+0 3
Costat Factors The growth rate is ot affected by costat factors or lower-order terms Examples 0 2 + 0 5 is a liear fuctio 0 5 2 + 0 8 is a quadratic fuctio T ( ) E+26 E+24 E+22 E+20 E+8 E+6 E+4 E+2 E+0 E+8 E+6 E+4 E+2 E+0 Quadratic Quadratic Liear Liear E+0 E+2 E+4 E+6 E+8 E+0 4
Worst-Case, Best-Case, ad Average- Case Efficiecies Algorithm SequetialSearch(A,,K) i 0 while i< ad A[i] K do i i+ if i< retur i else retur - 5
Worst-Case, Best-Case, ad Average- Case Efficiecies The ruig time of this algorithm ca be quite differet for the same list size Worst-case efficiecy of a algorithm is its efficiecy for the algorithm rus the logest amog all possible iputs of that size. The worst-case aalysis provides very importat iformatio about a algorithm s efficiecy by boudig its ruig time from above. I other words, it guaratees that for ay istace of size, the ruig time will ot exceed it. From the SequetialSearch algorithm the worse-case time is T() = 6
Worst-Case, Best-Case, ad Average- Case Efficiecies Best-case efficiecy of a algorithm is its efficiecy for the algorithm rus the fastest amog all possible iputs of that size. First, determie the kid of iput for which the #of operatios will be the smallest amog all possible iputs of size. The, determie #of operatios accordig that kid of iput. From the SequetialSearch algorithm the best-case time T() = 7
Worst-Case, Best-Case, ad Average- Case Efficiecies Average-case efficiecy yield the ecessary iformatio a algorithm s behavior o a typical or radom iput. From the SequetialSearch algorithm, the probability of a successful search is p (0 p ), the probability of the first math occurrig i the i th positio of the list is p/ for every i, the umber of comparisos is with probability of such a search beig (-p). 8
Worst-Case, Best-Case, ad Average- Case Efficiecies Therefore, T avg () = [*p/+2*p/ i*p/+ +*p/]+*(-p) = p/[+2+ +i+ ]+(-p) = p/*(+)/2+(-p) = p(-)/2+(-p) If the search is must be successful (p = ) the average = (+)/2 If the search is must be usuccessful (p = 0) the average = 9
Worst-Case, Best-Case, ad Average- Case Efficiecies A probability distributio of iputs eeds to be obtaied or assumed so that the expected value of the basic #of operatios ca be derived. Probability assumptio is usually difficult to verify, but average-case efficiecy iformatio is eeded. There are may algorithms for which the average-case efficiecy is much better tha the worst-case efficiecy. 20
Asymptotic Notatio (Big-Oh Notatio) Give fuctios f() ad g(), we say that f() is O(g()) if there are positive costats c ad 0 such that f() cg() for 0 Example: 2 + 0 is O() 2 + 0 c (c 2) 0 0/(c 2) Pick c = 3 ad 0 = 0 0,000,000 00 0 3 2+0 0 00,000 2
Big-Oh Example Example: the fuctio 2 is ot O() 2 c c The above iequality caot be satisfied sice c must be a costat,000,000 00,000 0,000,000 00 0 ^2 00 0 0 00,000 22
More Big-Oh Examples 7-2 7-2 is O() eed c > 0 ad 0 such that 7-2 c for 0 this is true for c = 7 ad 0 = 3 3 + 20 2 + 5 3 3 + 20 2 + 5 is O( 3 ) eed c > 0 ad 0 such that 3 3 + 20 2 + 5 c 3 for 0 this is true for c = 4 ad 0 = 2 3 log + log log 3 log + log log is O(log ) eed c > 0 ad 0 such that 3 log + log log c log for 0 this is true for c = 4 ad 0 = 2 23
Big-Oh ad Growth Rate The big-oh otatio gives a upper boud o the growth rate of a fuctio The statemet f() is O(g()) meas that the growth rate of f() is o more tha the growth rate of g() We ca use the big-oh otatio to rak fuctios accordig to their growth rate g() grows more f() grows more Same growth f() is O(g()) Yes No Yes g() is O(f()) No Yes Yes 24
Big-Oh Rules If is f() a polyomial of degree d, the f() is O( d ), i.e.,. Drop lower-order terms 2. Drop costat factors Use the smallest possible class of fuctios Say 2 is O() istead of 2 is O( 2 ) Use the simplest expressio of the class Say 3 + 5 is O() istead of 3 + 5 is O(3) 25
Asymptotic Algorithm Aalysis The asymptotic aalysis of a algorithm determies the ruig time i big-oh otatio To perform the asymptotic aalysis We fid the worst-case umber of primitive operatios executed as a fuctio of the iput size We express this fuctio with big-oh otatio Example: We determie that algorithm arraymax executes at most 7 primitive operatios We say that algorithm arraymax rus i O() time Sice costat factors ad lower-order terms are evetually dropped ayhow, we ca disregard them whe coutig primitive operatios 26
Computig Prefix Averages We further illustrate asymptotic aalysis with two algorithms for prefix averages The i-th prefix average of a array X is average of the first (i + ) elemets of X: A[i] = (X[0] + X[] + + X[i])/(i+) Computig the array A of prefix averages of aother array X has applicatios to fiacial aalysis 35 30 25 20 5 0 5 0 X A 2 3 4 5 6 7 27
Prefix Averages (Quadratic) The followig algorithm computes prefix averages i quadratic time by applyig the defiitio Algorithm prefixaverages(x, ) Iput array X of itegers Output array A of prefix averages of X #operatios A ew array of itegers for i 0 to do s X[0] for j to i do + 2 + + ( ) s s + X[j] + 2 + + ( ) A[i] s / (i + ) retur A 28
Arithmetic Progressio The ruig time of prefixaverages is O( + 2 + + ) The sum of the first itegers is ( + ) / 2 There is a simple visual proof of this fact Thus, algorithm prefixaverages rus i O( 2 ) time 7 6 5 4 3 2 0 2 3 4 5 6 29
Prefix Averages (Liear) The followig algorithm computes prefix averages i liear time by keepig a ruig sum Algorithm prefixaverages2(x, ) Iput array X of itegers Output array A of prefix averages of X A ew array of itegers s 0 for i 0 to do s s + X[i] A[i] s / (i + ) retur A Algorithm prefixaverages2 rus i O() time #operatios 30
Math you eed to Review Summatios Logarithms ad Expoets Proof techiques Basic probability properties of logarithms: log b (xy) = log b x+ log b y log b (x/y) = log b x-log b y log b xa = alog b x log b a= log x a/log x b properties of expoetials: a (b+c) = a b a c a bc = (a b ) c a b /a c = a (b-c) b = a log a b b c = a c*log a b 3
Relatives of Big-Oh big-omega f() is Ω(g()) if there is a costat c > 0 ad a iteger costat 0 such that f() c g() for 0 big-theta f() is Θ(g()) if there are costats c > 0 ad c > 0 ad a iteger costat 0 such that c g() f() c g() for 0 little-oh f() is o(g()) if, for ay costat c > 0, there is a iteger costat 0 0 such that f() c g() for 0 little-omega f() is ω(g()) if, for ay costat c > 0, there is a iteger costat 0 0 such that f() c g() for 0 32
Ituitio for Asymptotic Notatio Big-Oh f() is O(g()) if f() is asymptotically less tha or equal to g() big-omega f() is Ω(g()) if f() is asymptotically greater tha or equal to g() big-theta f() is Θ(g()) if f() is asymptotically equal to g() little-oh f() is o(g()) if f() is asymptotically strictly less tha g() little-omega f() is ω(g()) if is asymptotically strictly greater tha g() 33
Examples 5 2 is Ω( 2 ) f() is Ω(g()) if there is a costat c > 0 ad a iteger costat 0 such that f() c g() for 0 let c = 5 ad 0 = 5 2 is Ω() f() is Ω(g()) if there is a costat c > 0 ad a iteger costat 0 such that f() c g() for 0 let c = ad 0 = 5 2 is ω() f() is ω(g()) if, for ay costat c > 0, there is a iteger costat 0 0 such that f() c g() for 0 eed 5 02 c 0 give c, the 0 that satisfies this is 0 c/5 0 34
Examples 3 2-00+6 = O( 2 )? Yes Because 3 2 >3 2-00+6 3 2-00+6 = O( 3 )? Yes Because 0.0 3 >3 2-00+6 3 2-00+6 = O()? O() Because c*<3 2 whe > c 3 2-00+6 = Ω( 2 )? Yes Because 2.99 2 <3 2-00+6 3 2-00+6 = Ω( 3 )? Ω() Because 3 2-00+6< 3 3 3 2-00+6 = Ω()? Yes Because 0 00 <3 2-00+6 35
Usig Limits for Comparig Orders of Growth lim g()/f() = 0 lim g()/f() = c implies that g() has a smaller order of growth tha f() implies that g() has a same order of growth as f() lim g()/f() = implies that g() has a larger order of growth tha f() Moreover accordig to L Hopital s rule lim g()/f() = lim g ()/f () Ad Stirlig s formula! 2π for very large values of e 36
Usig Limits for Comparig Orders of Growth Compare orders of growth of (/2)(-) ad 2 Compare orders of growth of log () ad (/2) Compare orders of growth of! ad 2 37
Properties of Logarithms Recall, c log c x =x Asymptotically, the base of the log does t matter log b a= log c a/ log c b Thus, log 2 = (/ log 00 2)* log 00, / log 00 2=6.643 (costat) Asymptotically, polyomial fuctio of does t matter log( 473 + 2 ++96) = O(log ) Sice, 473 + 2 ++96 = O( 473 ), ad log( 473 ) = 473*log 38
Aalysis of No-recursive Algorithms Algorithm MaxElemet (A[0 -]) maxvalue A[0] for i 0 to do if A[i] > maxvalue maxvalue A[i] retur maxvalue T() = the umber of times is executed i a array of size Algorithm makes oe compariso o each executio of the loop which is repeated for each value of the i withi boud of ad - = T ( ) i= T ( ) = i= = Θ( ) 39
40 Aalysis of No-recursive Algorithms = = = + = = + + = = 2 0 2 0 2 0 ) ( ] ) ( ) [( ) ( i i i i j worst i i T Algorithm UiqueElemets (A[0 -]) for i 0 to - 2 do for j i+ to do if A[i] = A[j] retur false retur true Iermost loop cotais a sigle operatio (A[i] = A[j]) ) ( 2 2 ) ( 2 ) 2)( ( ) ( 2 ) 2)( ( ) ( ) ( 2 2 2 2 0 2 0 2 0 i i i i Θ = = = = = = =
4 Aalysis of No-recursive Algorithms = 0 k = = = = 0 0 0 ) ( i j k T Algorithm MatrixMulitiplicatio (A[0 -][0 -],B[0 -][0 -]) for i 0 to - do for j 0 to do C[i,j] 0 for k 0 to - do C[i,j] C[i,j] + A[i,k]*B[k,j] retur C There is just oe multiplicatio executed o each repetitio of algorithm s iermost loop. 3 0 0 2 0 0 0 0 ) ( T i j j i j k = = = = = = = = = =
Aalysis of No-recursive Algorithms Algorithm Biary () cout while > do cout cout + /2 retur cout Most frequetly executed operatio here is ot iside the while loop but rather the compariso >. We have to compute the umber of times the loop is executed. Value of is about halved o each repetitio of the loop. The aswer should be about log 2 42
Pla for Aalyzig Efficiecy of Norecursive Algorithm Decide o a parameter (s) idicatig a iput s size. Idetify the algorithm s basic operatio (ier most loop). Check that the umber of times the basic operatios is executed depeded oly o the size of a iput. Set up a sum expressig the umber of times the algorithm s basic operatio is executed. Usig stadard formulas ad rules of sum maipulatio, either fid a closed form formula for the cout. 43
Aalysis of Recursive Algorithms Algorithm Factorial () //! = * *(-)*=(-)!* for if = 0 retur else retur Factorial(-)* The basic operatio is multiplicatio, whose umber of executios = T(). The fuctio F() is computed accordig to the formula F() = F(-)* for > 0 The umber of multiplicatio T() eeded to compute it must satisfy the equality T() = T(-) + for > 0 To compute F(-) To multiply F(-) by 44
Aalysis of Recursive Algorithms Our goal is to solve the recurrece relatio (recurrece) T() = T(-) + Coditio that makes the algorithm stop: if = 0 retur This implies. the calls stop whe = 0, 2. the smallest of is 0. Therefore; T() = T(-) + for > 0 T(0) = 0 // o multiplicatio whe = 0 F() = F(-)* for > 0 F(0) = 45
Aalysis of Recursive Algorithms Method of Backward Substitutios T() = T(-)+ substitute T(-) = T(-2)+ = [T(-2)+] = T(-2)+2 substitute T(-2) = T(-3)+ = [T(-3)+2] = T(-3)+3 T() = T(-i) + i We take advatage of the iitial coditio give that it is specified =0, by substitute i = T()=T(-)+ = T(-i)+i = = M(-)+ = 46
Aalysis of Recursive Algorithms Tower of Haoi (http://www.mazeworks.com/haoi/) ad (http://e.wikipedia.org/wiki/tower_of_haoi) This puzzle cotais disks of differet sizes ad three pegs. First all the disks are o the first peg i order of size, the largest o the bottom ad the smallest o top. The goal is to move all disks to the third pegs usig secod oe as a auxiliary, if ecessary. Oly oe disk ca be moved at a time, ad it is forbidde to place a larger disk o top of a smaller oe. 47
Aalysis of Recursive Algorithms To move > disks from peg to peg 3, we first move recursively - disks from peg to peg 2, the move the largest disk directly from peg to peg 3 ad move recursively - disks from peg 2 to peg 3. If =, move the sigle disk from peg to peg 3. The umber of disks is the choice for the iput size ad movig oe disk is the algorithm s basic operatio. The umber of move T() follow the recurrece equatio: T() = T(-) + + T(-) for > where T() = // iitial coditio 48
Aalysis of Recursive Algorithms T() = 2T(-)+ sub T(-) = 2T(-2) + = 2[2T(-2)+]+ = 2 T(-2)+2+ sub T(-2) = 2T(-3) + = 2 T(-3)+]+2+ = 2 3 T(-3)+2 2 +2+ The ext substitute would be 2 4 T(-4)+2 3 +2 2 +2+ T() = 2 i T(-i)+2 i- +2 i-2 +..+2+ = 2 i T(-i)+2 i - Iitial coditio is for =, which is achieved for i = -, therefore we get followig formula for solutio T()= 2 - T(-(-))+2 - - = 2 - T() +2 - - = 2 - +2 - - = 2-49
Aalysis of Recursive Algorithms Algorithm BiRec () if = retur else retur BiRec(/2)+ From the iitial coditio, the time efficiecy is the umber of additio made by the algorithm = T(). The umber of additio made i computig BiRec(/2) is T(/2) plus oe more additio to icrease the retured valued by. Therefore: T() = T(/2)+ for > T() = 0 //the recursive calls ed whe = 50
Aalysis of Recursive Algorithms Let =2 k the T(2 k ) = T(2 k- )+ for >0 where T(2 0 ) = 0 Backward Substitutio: T(2 k ) = T(2 k- )+ substitute T(2 k- ) = T(2 k-2 )+ = [T(2 k-2 ) +]+ = T(2 k-2 )+2 substitute T(2 k-2 ) = T(2 k-3 )+ = [T(2 k-3 ) + ]+2 = T(2 k-3 )+3 = T(2 k-i )+i = T(2 k-k )+k // let i = k T(2 k ) = T(2 0 ) +k = k Sice =2 k, which also meas k = log 2 therefore: T() = log 2 = Θ(log 2 ) 5
Pla for Aalyzig Efficiecy of Recursive Algorithm Decide o a parameter (s) idicatig a iput s size. Idetify the algorithm s basic operatio (ier most loop). Check that the umber of times the basic operatios is executed depeded oly o the size of a iput. Set up a recurrece relatio, with a appropriate iitial coditio, for the umber of times the basic operatio is executed. Solve the recurrece or ascertai the order of growth of its solutio. 52
Exercises o Recursive Algorithms T() = 3T(/4) + T() = 2T(/2) + 2 53