Algorithm Design Strategies

(アルゴリズムの設計方法)

Data Structures and Algorithms

13th lecture, January 9, 2020

http://www.sw.it.aoyama.ac.jp/2019/DA/lecture13.html

Martin J. Dürst

© 2009-20 Martin J. Dürst 青山学院大学

Today's Schedule

• Remaining schedule
• Leftovers/summary of last lecture
• Algorithm design strategies
• Examples of 'difficult' problems

Remaining Schedule

• January 9 (today): 13th lecture (algorithm design strategies)
• January 16: 14th lecture (NP-completenes, reducibility; makeup class)
  Important: Contents of makeup class is part of final exam!
• January 23: 15th lecture (approximation algorithms)
• January 30, 09:30-10:55: Term Final Exam

Summary of Last Lecture

• Dynamic programming is an algorithm design strategy
• Dynamic programming is suited for problems where the overall (optimal) solution can be obtained from solutions for subproblems, but the subproblems overlap
• The time complexity of dynamic programming depends on the structure of the actual problem
• Dynamic programming is closely related to memoization

Algorithm Design Strategies/Methods

• Simple/simplistic algorithms
• Brute force algorithms
• Greedy algorithms
  Find overall optimal solution by selecting locally optimal solutions
• Divide and conquer
  Divide problem into nonoverlapping subproblems
• Dynamic programming
  Find overall optimal solution from solutions for overlapping subproblems

The Knapsack Problem

• A knapsack with capacity c (weight or volume) and
• n items s₁,..., s_n (each with a weight/volume, and in some variations a value)
• Goal: Find the best way to pack the knapsack

Depending on the details of the problem, the best algorithm design strategy is different

Variations of the Knapsack Problem

1. All items are the same; how many items fit in?
   
2. Pack as many items as possible
   
3. Use as much capacity as possible (integer version)
   
4. Maximise value

All Items are the Same

Example: Capacity c = 20kg, weight per item: 3.5kg

'Algorithm': Divide the capacity by the weight per item, round down

Answer for example: 5 items

Design strategy: simplistic 'algorithm'

Simplistic Algorithm

• Sometimes too simple to be called 'algorithm'
• Examples:
  • Select the third-smallest element from a sorted array
  • Obtain the surface of a rectangle from the length of its sides
  • Closed formula of a number sequence
• Often forgotten because computers are so fast now

Pack as Many Items as Possible

Example: Capacity c = 20kg, weight of items: 8kg, 2kg, 4kg, 7kg, 2kg, 1kg, 5kg, 12kg

Algorithm: Sort items by increasing weight, pack starting with lightest item

Answer for example: 5 items (e.g. 1kg, 2kg, 2kg, 4kg, 5kg)

Design strategy: Greedy algorithm

Time complexity: O(n log n)

Greedy Algorithm

• Develop solution step-by-step
• Consider only locally optimal solutions
• Optimal substructure is a precondition, but the structure is different than for dynamic programming
• Time complexities of O(n) and O(n log n) are frequent
• Examples: Calculating change, many planning/scheduling problems, some algorithms on graphs (e.g. minimum spanning tree,...),...

Use as Much Capacity as Possible (Integer Version)

Example: Capacity c = 20kg, weight of items (w_s1,...,w_sn): 8kg, 2kg, 4kg, 7kg, 2kg, 1kg, 5kg, 12kg

Algorithm: Consider subproblems with capacity c' ≦ c using only the first k items s₁,..., s_k (k ≦ n)

Design strategy: Dynamic programming

Recursive problem description:

max_weight(0, ...) = 0 = max_weight(x, {})

max_weight(c₁, {s₁,..., s_k}) = max(max_weight(c₁-w_sk, {s₁,..., s_k-1})+w_sk,
                                                         max_weight(c₁,      {s₁,..., s_k-1}))

Example Solution

Table rows: Items to add (or not) to knapsack

Table columns: Total capacity

Table entries: ×: Item in this row is not used; ○: Item in this row is used; number: max weight possible

How to find the solution: See next slide

How to Find the Solution

• Start at bottom right corner of the table
• If there is a ×, do not include the item on this row, and move up one row.
• If there is a ○:
  
  1. Do include the item on this row
  2. Move left w columns, where w is the weight of the item included
  3. Move up one row
• Repeat until you arrive at a row or column that is 0

Answer for example: 1kg, 7kg, 4kg, 8kg (total: 20kg; other solutions possible)

Time complexity: O(cn)

Value Maximization

Example: Capacity c = 20kg, weight and value of items: 8kg, 500¥; 2kg, 2000¥; 4kg, 235¥; 7kg, 700¥; 2kg, 400¥; 1kg, 100¥; 5kg, 450¥; 12kg, 650¥

Algorithm: Try all possible solutions

Design strategy: Brute force

Implementation: Dknapsack.rb (of various algorithms, only brute force finds optimal solution if weights and values can be non-integers)

Brute Force Algorithm

• Try all solution possibilities (combinations, permutations,...)
• Choose the best solution
• May be very slow
• May be improved by shortcuts (e.g. compare with best solution found up to now)
• For more examples, see next lecture

Variations of the Knapsack Problem (Summary)

1. All items are the same; how many items fit in?
   Solution: Divide capacity by item weight
   Design strategy: Simplistic 'algorithm'
2. Pack as many items as possible
   Solution: Start with lightest items
   Design strategy: Greedy algorithm
3. Use as much capacity as possible (integer version)
   Solution: Consider subproblems with capacity c' ≦ c and items s₁,..., s_k (k ≦ n)
   Design Strategy: Dynamic programming
4. Maximize value
   Design Strategy: Brute force

Algorithm Design Strategies

• Useful when developing algorithms
• Consider design strategies one-by-one
• For some problems, some strategies can be excluded quickly
• Depending on the details of the problem, the best strategy may be different
• For the same problem and the same strategy, there may be several algorithms

Goal for Remaining Time

• Be able to distinguish between 'simple' and 'difficult' problems
• Expand your view, look at the algorithm universe

Example Problem 1: 3-SAT

• n binary variables
• A logical formula using these variables
• The formula is a conjunction of disjunctions (of negations)
• All disjunctions use exactly 3 terms
• Problem: Find values for each variable so that the overall formula becomes true
• Problem variant: Decide whether a solution is possible or not
• Example (' indicates negation):
  (x₁∨x₂∨x₄) ∧ (x₁'∨x₃∨x₄') ∧ (x₂∨x₃'∨x₄) ∧ (x₁'∨x₂'∨x₃')
• Number of possible answers: 2ⁿ
• Time to check all possible answers: O(n2ⁿ)
• Currently, no faster general algorithm is known
• Currently, there is no proof that there is no faster general algorithm

Example Problem 2: Independent Set

• Graph with n vertices
• If two vertices are connected by an edge, there is a conflict
• Problem: Find the largest independent set (i.e. subset of vertices without conflicts)
• Problem variant: Decide whether there is an independent set of size ≧k
• Number of possible answers: 2ⁿ
• Time to check all possible answers: O(n2ⁿ)
• Currently, no faster general algorithm is known
• Currently, there is no proof that there is no faster general algorithm

Example Problem 3: Traveling Salesman

• n towns
• Distances (or time or cost) between each pair of towns
• Problem: Find the shortest (fastest/cheapest) tour that visits all cities exactly once
• Problem variant: Decide whether there is a tour of size (duration/cost) ≦k
• Number of possible answers: n!
• Time to check all possible answers: O(n!)
• Currently, no faster general algorithm is known
• Currently, there is no proof that there is no faster general algorithm

Homework

(no need to submit)

• Review this lecture
• Review Dknapsack.rb
• Find commonalities of 3-SAT, independent set, and traveling salesman problems
• Prepare for term final exam

Glossary

NP-completeness

NP-完全性

reducibility

帰着可能性

approximation algorithms

近似アルゴリズム

brute force

総当たり方、腕力法、虱潰し

greedy algorithm

貪欲アルゴリズム

knapsack problem

ナップサック問題

capacity

容量

closed formula

「閉じた式」

number sequence

数列

minimum spanning tree (problem)

最小全域気 (問題)

independent set

独立集合

conflict

競合

traveling salesman

巡回セールスマン