Quicksort, Average Time Complexity

(クイックソート、平均計算量)

Data Structures and Algorithms

7th lecture, November 10, 2015

http://www.sw.it.aoyama.ac.jp/2016/DA/lecture7.html

Martin J. Dürst

© 2009-16 Martin J. Dürst 青山学院大学

Today's Schedule

• Summary of last lecture
• About the Manual Sorting report
• Quicksort:
  Concept, implementation, optimizations
• Average time complexity
• Sorting in C, Ruby, ...
• Comparing sorting algorithms using animation

Summary of Last Lecture

• Sorting is a very important operation for Information Technology
• Simple sorting algorithms are all O(n²)
• Merge sort (O(n log n)) needs double memory
• Heap sort (O(n log n)) uses a large number of comparisons and exchanges

Report: Manual Sorting: Problems Seen

• 218341.368 seconds (⇒about 61 hours)
• 6¹⁰¹⁰·10^3·1010 (units? way too big)
• O(60000) (how many seconds could this be)
• Calulation of actual time backwards from big-O notation (1second/operation, n=5000, O(n²) ⇒ 25'000'000 seconds?)
• A O(n) algorithm (example: "5 seconds per page")
• For 20 people, having only one person work at the end of the algorithm
• For humans, binary sorting is constraining (sorting into 3~10 parts is better)
• Using bubble sort (868 days without including breaks or sleep)
• Preapare 10¹⁰ boxes (problem: space, cost, distance for walking)
• Forgetting time for preparation, cleanup, breaks,...
• Submitting just a program
• Report too short

Today's Goals

Using quicksort as an example, understand

• Move from algorithmic concept to efficient implementation
• Average time complexity

History of Quicksort

• Invented by C. A. R. Hoare in 1960
• Researched in great detail
• Extremely widely used

Reviewing Divide and Conquer

• Heap sort: The highest priority item of the overall tree is the highest priority item of the two subtrees
• Merge sort: Split into equal-length parts, recurse, merge
  
• Quicksort: Use an arbitrary boundary element to partition the data, recursively

Basic Workings of Quicksort

• Select one element as the partitioning element (pivot)
• Exchange elements so that
  elements smaller than the pivot go to the left, and
  elements larger than the pivot go to the right
• Apply quicksort recursively to the data on the left and right sides of the pivot

Ruby pseudocode/implementation: conceptual_quick_sort in 7qsort.rb

Quicksort Implementation Details

1. Use the rightmost element as the pivot
2. Starting from the right, find an element smaller than the pivot
3. Starting from the left, find an element larger than the pivot
4. Exchange the elements found in steps 2. and 3.
5. Repeat steps 2.-4. until no further exchanges are needed
6. Exchange the pivot with the element in the middle
7. Recurse on both sides

Ruby pseudocode/implementation: simple_quick_sort in 7qsort.rb

Worst Case Complexity

• What happens if the largest (or the smallest) element is always choosen as the pivot?
• The time complexity is Q_w(n) = n + Q_w(n-1) = Σⁿ_i=1 i
  ⇒ O(n²)
• This is the worst case complexity (worst case running time) for quick sort
• This complexity is the same as the complexity of the simple sorting algorithms
• This worst case can easily happen if the input is already sorted

Best Case Complexity

• Q_B(n) = n + 1 + 2 Q_B(n/2)
• Q_B(1) = 0
• Same as merge sort
• ⇒ O(n log n)
• Unclear whether this is relevant

For most algorithms, worst case complexity is very important, and best case complexity is mostly irrelevant. But there are exceptions.

Average Complexity

• Assumption: All permutations of the input values have the same probability
• Q_A(n) = n + 1 + 1/n Σ_1≤k≤n (Q_A(k-1)+Q_A(n-k))
• Q_A(1) = 0

Calculating Q_A

[1] Q_A(n) = n + 1 + 1/n Σ_1≤k≤n (Q_A(k-1)+Q_A(n-k))

[2] Q_A(0) + ... + Q_A(n-2) + Q_A(n-1) =
= Q_A(n-1) + Q_A(n-2) + ... + Q_A(0)

[3] Q_A(n) = n + 1 + 2/n Σ_1≤k≤n Q_A(k-1) [use [2] in [1]]

[4] n Q_A(n) = n (n + 1) + 2 Σ_1≤k≤n Q_A(k-1) [multiply [3] by n]

[5] (n-1) Q_A(n-1) = (n-1) n + 2 Σ_1≤k≤n-1 Q_A(k-1) [[4], with n replaced by n-1]

Calculating Q_A (continued)

[6] n Q_A(n) - (n-1) Q_A(n-1) = n (n+1) - (n-1) n + 2 Q_A(n-1) [[4]-[5]]

[7] n Q_A(n) = (n+1) Q_A(n-1) + 2n [simplifying [6]]

[8] Q_A(n)/(n+1) = Q_A(n-1)/n + 2/(n + 1) [dividing [7] by n (n+1)

Q_A(n)/(n+1) =
= Q_A(n-1)/n + 2/(n + 1) = [repeatedly right side of [8] by using [8]]
= Q_A(n-2)/(n-1) + 2/n + 2/(n+1) =
= Q_A(n-3)/(n-2) + 2/(n-1) 2/n + 2/(n+1) = ...
= Q_A(2)/3 + Σ_3≤k≤n 2/(k+1) [approximating sum by integral]

Q_A(n)/(n+1) ≈ 2 Σ_1≤k≤n 2/k ≈ 2∫₁ⁿ 1/x dx = 2 ln n

Result of Calculating Q_A

Q_A(n) ≈ 2n ln n ≈ 1.39 n log₂ n

⇒ O(n log n)

⇒ The number of comparisons on average is ~1.39 times the optimal number of comparisons in an optimal decision tree

Distribution around Average

• A good average complexity is not enough if the worst case is frequently reached
• For Q_A, it can be shown that the standard deviation is about 0.65n
• This means that the probability of deviation from the average very quickly gets extremely small

Complexity of Sorting

Question: What is the complexity of sorting (as a problem)?

• Most sorting algorithms are O(n log n) (except for simple sorting algorithms)
• The basic operations for sorting are comparison and movement
• For n data items, the number of different sorting orders (permutations) is n!
• With each comparision, in the best case, we can reduce the number of sorting orders to half
• The mimimum number of comparisions necessary for sorting is log (n!) ≈ O(n log n)
• Sorting using pairwise comparison is Ω(n log n)

Pivot Selection

• The efficiency of quicksort strongly depends on the selection of the pivot
• Some solutions:
  • Select rightmost element
    (dangerous!)
  • Use the median of three values
  • Use the value at a random location
    (this is an example of a randomized algorithm)

Implementation Improvements

• Comparison of indices
  → Use a sentinel to remove one comparision
  
• Stack overflow for deep recursion
  → When splitting into two, use recursion for the smaller part, and tail recursion or a loop for the larger part
• Low efficiency of quicksort for short arrays/parts
  → For parts smaller than a given size, change to a simple sort algorithm
  → With insertion sort, it is possible to do this in one go at the very end
  (this needs care when testing)
  → Quicksort gets about 10% faster if change is made at an array size of about 10
• Duplicate keys
  → Split in three rather than two

Ruby pseudocode/implementation (excluding split in three): quick_sort in 7qsort.rb

Comparing Sorting Algorithms using Animation

• Uses Web technology: SVG (2D vector graphics) and JavaScript
• Uses special library (Narrative JavaScript) for timing adjustments
• Comparisons are shown in yellow (except for insertion sort)、exchanges in blue

Watch animation: sort.svg

Stable Sorting

• Definition: A sorting algorithm is stable if it retains the original order for two data items with the same key value
• Used for sorting with multiple criteria (e.g. sort by year and prefecture):
  
  • First, sort using the lower priority criterion (e.g. prefecture)
  • Then, sort using the higher priority criterion (e.g. year)
• The simple sorting algorithms and merge sort can easily be made stable
• Heap sort and quicksort are not stable
  → Solution 1: Sort multiple criteria together
  → Solution 2: Use the original position as a lower priority criterion

Sorting in C and Ruby

• Sorting is provided as a library function or method
• Implementation is often based on quicksort
• Comparison of data items depends on type of data and purpose of sorting
  → Use comparison function as a function argument
• If comparison is slow
  → Precompute a value that can be used for sorting
• If exchange of data items is slow (e.g. very large data items)
  → Sort/exchange references (pointers) only

C's qsort Function

void qsort(
    void *base,        // start of array
    size_t nel,        // number of elements in array
    size_t width,      // element size
    int (*compar)(     // comparison function
        const void *,
        const void *)
  );

Ruby's Array#sort

(Klass#method denotes instance method method of class Klass)

array.sort uses <=> for comparison

array.sort { |a, b| a.length <=> b.length }
This example sorts (e.g. strings) by length

The code block (between { and }) is used as a comparison function

Ruby's <=> Operator

(also called spaceship operator, similar to strcmp in C)

Ruby's Array#sort_by

array.sort_by { |str| str.length }

(sorting strings by length)

array.sort_by { |stu| [stu.year, stu.prefecture] }

(sorting students by year and prefecture

This calculates the values for the sort criterion for each array element in advance

Summary

• Quicksort is another application of divide and conquer
• Quicksort is a very famous algorithm, and a good example to learn about algorithms and their implementation
• Quicksort has been carefully researched and widely implemented and used
• Quicksort is a classical example of the importance of average time complexity
• Quicksort is our first example of a randomized algorithm
• Sorting based on pairwise comparison is Θ(n log n)

Preparation for Next Time

• Think about inputs for which conceptual_quick_sort will fail
• Watch the animations carefully (>20 times) to deepen your understanding of sorting algorithms

Glossary

quicksort

クイックソート

partition

分割

partitioning element (pivot)

分割要素

worst case complexity (running time)

最悪時の計算量

best case complexity (running time)

最善時の計算量

average complexity (running time)

平均計算量

standard deviation

標準偏差

randomized algorithm

ランドム化アルゴリズム

median

中央値

decision tree

決定木

tail recursion

末尾再帰

in one go

一括

stable sorting

安定な整列法

criterion (plural criteria)

基準

block

ブロック