Asymptotic Time Complexity and the Big-O Notation

(漸近的計算量と O 記法)

Data Structures and Algorithms

3rd lecture, October 1, 2015

http://www.sw.it.aoyama.ac.jp/2015/DA/lecture3.html

Martin J. Dürst

© 2009-15 Martin J. Dürst 青山学院大学

Today's Schedule

• Leftovers from last lecture・Summary of last lecture・Last week's homework
• Comparing Execution Times: From Concrete to Abstract
• Classification of Functions by Asymptotic Growth
• Big-O notation

Leftovers from Last Lecture

Summary of Last Lecture

• There are many ways of describing algorithms: natural language text, diagrams, pseudocode, programs
• Each description has advantages and disadvantages
• Pseudocode is close to structured programming, but ignores unnecessary details
• In this course, we will use Ruby as "executable pseudocode"
• The main criterion to evaluate algorithms is time complexity as a function of the number of (input) data items
• Time complexity is the most important criterion when comparing algorithms

Last Week's Homework 1: Example for Asymptotic Growth of Number of Steps

[昨年度資料につき削除]

Thinking in Terms of Asymptotic Growth

• The execution time of an algorithm and the number of executed steps depends on the size of the input (the number of data items in the input)
• We can express this dependency as a function: f(n) (n is the size of the input)
• Rules for comparing functions:
  • Concentrate on what happens when n increases (gets really big)
    → Ignore special cases for small n
    → Ignore constant(-time) differences (example: initialization time)
  • Concentrate on the essence of the algorithm
    → Ignore hardware differences and implementation differences
    → Ignore constant factors

⇒ Independent of hardware, implementation details, step counting details

⇒ Simple expression of essential differences between algorithms

Solution to Homework 2: Compare Function Growth

[昨年度資料につき削除] 

Using Ruby to Compare Function Growth

• Start irb (Interactive Ruby)
• Writing a loop: (start..end).each { |n| comparison }
• Example of comparison: puts n, 1.1**n, n**20
• Change the start and end values until appropriate
• If necessary, convert integers to floating point numbers for easier comparison
• Define the factulty function: def fac(n) n>1 ? n*fac(n-1) : 1 end

Caution: Use only when you understand which function will eventually grow larger

Classification of Functions by Asymptotic Growth

Various growth classes with example functions:

• Linear growth: n, 2n+15, 100n-40, 0.001n,...
• Quadratic growth: n², 500n²+30n+3000,...
• Cubic growth: n³, 5n³+7n²+80,...
• Logarithmic growth: ln n, log₂n, 5 log₁₀n²+30,...
• Exponential growth: 1.1ⁿ, 2ⁿ, 2^0.5n+1000n¹⁵,...
• ...

Big-O Notation: Set of Functions

Big-O notation is a notation for expressing the order of growth of a function (e.g. time complexity of an algorithm).

O(g): Set of functions with lower or same order of growth as function g

Examples:
n^1.5 ∈ O(n²), n² ∈ O(n²), n³ ∉ O(n²)

Exact Definition of O

∃c>0: ∃n₀≥0: ∀n≥n₀:       f(n)≤c·g(n)   ⇔  f(n)∈O(g(n))

• g(n) is an asymptotic upper bound of f(n)
• In some references (books, ...):
  • f(n)∈O(g(n)) is written f(n)＝O(g(n))
    
  • In this case, O(g(n)) is always on the rigth side
  • However, f(n)∈O(g(n)) is more precise and easier to understand

Example Algorithms

• The number of steps in linear search is: an + b
  ⇒ Linear search has time complexity O(n)
  (linear search is O(n), linear search has linear time complexity)
• The number of steps in binary search is: c log₂ n + d
  ⇒ Binary search has time a complexity of O(log n)
• Because O(log n) ⊊ O(n), binary search is faster

Comparing the Execution Time of Algorithms

(from last lecture)

Possible questions:

• How many seconds faster is binary search when compared to linear search?
• How many times faster is binary search when compared to linear search?

Problem: These questions do not have a single answer.

When we compare algorithms, we want a simple answer.

The simple and general answer is:
Linear search is O(n), binary search is O(log n).

Additional Examples for O

• Linear growth:
  n∈O(n);  2n+15∈O(n);  100n-40∈O(n);  5 log₁₀n+30∈O(n), ...
  

  O(1)⊂O(n);  O(log n)⊂O(n);  O(20 n)=O(4n + 13), ...

• Quadratic growth:
  n²∈O(n²);  500n²+30n+3000∈O(n²), ...
  O(n)⊂O(n²);  n³∉O(n²), ...
• Cubic Growth:
  n³∈O(n³);  5n³+7n²+80∈O(n³), ...
• Logarithmic growth:
  ln n∈O(log n);  log₂n∈O(log n);  5 log₁₀n²+30∈O(log n), ...

Additional Notations: Ω and Θ

• O(g(n)): Set of functions with lower or same order of growth as g(n)
  
• Ω(g(n)): Set of functions with larger or same order of growth as g(n)
• Θ(g(n)): Set of functions with same order of growth as g(n)

Examples:
n^1.5 ∈ O(n²), n² ∈ O(n²), n³ ∉ O(n²)
n^1.5 ∉ Ω(n²), n² ∈ Ω(n²), n³ ∈ Ω(n²)
n^1.5 ∉ Θ(n²), n² ∈ Θ(n²), n³ ∉ Θ(n²)

Exact Definitions of Ω and Θ

∃c>0: ∃n₀≥0: ∀n≥n₀:         c·g(n)≤f(n)   ⇔   f(n)∈Ω(g(n))

∃c₁>0: ∃c₂>0: ∃n₀≥0: ∀n≥n₀:       c₁·g(n)≤f(n)≤c₂·g(n)   ⇔   f(n)∈Θ(g(n))

f(n)∈Θ(g(n))   ⇔   f(n)∈O(g(n)) ∧ f(n)∈Ω(g(n))

Θ(g(n)) = O(g(n)) ∩ Ω(g(n))

Use of Order Notation

• O: Maximum (worst-case) time complexity of algorithms
• Ω: Minimally needed time complexity to solve a problem
• Θ: Used when expressing the fact that a time complexity is not only possible, but actually reached

In general as well as in this course, mainly O will be used.

Confirming the Order of a Function

• Method 1: Use the definition
  Find appropriatie values for n₀ and c, and check the definition
• Method 2: Use the limit of a function
  lim_n→∞(f(n)/g(n)):
  • If the limit is 0:   O(f(n))⊊O(g(n)), f(n)∈O(g(n))
  • If the limit is 0 < d < ∞:   O(f(n))=O(g(n)), f(n)∈O(g(n))
  • If the limit is ∞:   O(g(n))⊊O(f(n)), f(n)∉O(g(n))
• Method 3: Simplification

Simplification of Big-O Notation

• Big-O notation should be as simple as possible
• Examples (for all functions except constant functions, we assume increasing):
  • Constant functions: O(1)
  • Linear functions: O(n)
  • Quadratic functions: O(n²)
  • Cubic functions: O(n³)
  • Logarithmic functions: O(log n)
• For polynomials, all terms except the term with the biggest exponent can be ignored
• For logarithms, the base is left out

Ignoring Lower Terms in Polynomials

Concrete Example:   500n²+30n ∈ O(n²)

Derivation: f(n) = dn^a + en^b = O(n^a) [a > b > 0]

Definition of O: f (n) ≤ cg(n) [n > n₀; n₀, c > 0]

dn^a + en^b ≤ cn^a [a > 0 ⇒ n^a>0]

d + en^b/n^a ≤ c

d + en^b-a ≤ c [b-a < 0 ⇒ en^b-a→0]

Possible values for c and n₀:

• n₀ = 1, c = d+e
• n₀ = 2, c = d+2^b-ae
• n₀ = 10, c = d+10^b-ae

Possible values for concrete example (500n²+30n):

• n₀ = 1, c = 530 → 500n²+30n ≤ 530n² [n≥1]
• n₀ = 2, c = 515 → 500n²+30n ≤ 515n² [n≥2]
• n₀ = 10, c = 503 → 500n²+30n ≤ 503n² [n≥10]

In general: a > b > 0 ⇒ O(n^a + n^b) = O(n^a)

Ignoring Logarithm Base

How do O(log₂ n) and O(log₁₀ n) differ?

(Hint: log_b a = log_c a / log_c b = log_c a · log_b c)

log₁₀ n = log₂ n · log₁₀ 2 ≅ 0.301 · log₂ n

O(log₁₀ n) = O(0.301... · log₂ n) = O(log₂ n)

∀ a>1, b>1:   O(log_a n) = O(log_b n) = O(log n)

Summary

• To compare the time complexity of algorithms:
  • Ignore constant terms (initialization,...)
  • Ignore constant factors (differences due to hardware or implementation)
  • Look at asymptotic growth when input size increases
• Asymptotic growth can be expressed with big-O notation
• The time complexity of algorithms can be expressed as O(log n), O(n), O(n²), O(2ⁿ), ...

Homework

(no need to submit)

Review this lecture's material every day!

On the Web, find algorithms with time complexity O(1), O(log n), O(n), O(n log n), O(n²), O(n³), O(2ⁿ), O(n!), and so on.

Glossary

big-O notation

O 記法 (O そのものは漸近記号ともいう)

asymptotic growth

漸近的な増加

approximate

近似する

essence

本質

constant factor

一定の係数、定倍数

eventually

最終的に

linear growth

線形増加

quadratic growth

二次増加

cubic growth

三次増加

logarithmic growth

対数増加

exponential growth

指数増加

Omega (Ω)

オメガ (大文字)

capital letter

大文字

Theta (Θ)

シータ (大文字)

asymptotic upper bound

漸近的上界

asymptotic lower bound

漸近的下界

appropriate

適切

limit

極限

polynomial

多項式

term

(式の) 項

logarithm

対数

base

(対数の) 低