Turing, Halting Theorem, P, NP, NPC,
Hamiltonian, TSP, SAT, Subset Sum….
Dr. Mustafa Sakalli
Marmara Univ.
Very Preliminary
slights to be
modified, June
of 2009, CS246
CLRS chapter 34
A Turing Machine
Source: Prof Michael Chan’s Discrete Math Notes.
• A Turing machine (only on mind) is a modified or extended version of finite
state machine.
• Imagine a machine M, scanning along a tape which is infinitely long. Tape is
divided into boxes, each of which is marked with an element of a finite
alphabet A. M can be in any of states S. S has a finite number of states.
• Depending on the element M reads and he state it is in, following happens:
– M either leaves or replaces the element it reads in the current block.
– Then, it moves either direction, backward or forward to the adjacent blocks.
– The M either keeps its state or changes to one of the other state states.
States vs elements of A.
A Touring Machine
• Example: A={0, 1, 2, 3, 4}, S={.., 5, …}
– suppose machine is in state 5,
– the element M reads in current tape position is 3,
– and the table given below describes the behavior machine that will take, 1L2
says replace 3 by 1, shift one left and change the state to 2.
• Convention: aDs
– a  {A} any value to be inserted into the current position, in binary case
A={0, 1}.
– D  {L, R}: Direction shift by 1 either L or R. One step at a time.
– s {0, 1, …, n}: a new state to be taken (predetermined number of n
States vs elements of A.
• After subsequent actions machine takes machine may never
• However it is possible to force the machine into halt by
sending TM into a non-existent state.
And it halts here..
Given below, starting in a different situation, then TM does not go into
Designing a k-state TM
• States s{0..k-1}, without halting. State k, is allocated for the halting state.
• Elements binary: A={0, 1}, and this represents natural numbers in the unary
system, it means that a tape consisting entirely of 0's will represent the integer
0, while a string of n 1's will represent a positive integer n. Numbers are also
interspaced by single 0's. For example, the string
…01111110111111111101001111110101110… will represent the sequence … 6;
10; 1; 0; 6; 1; 3… .
here consecutive zeros equal to 0 (therefore two zeros in between denote 0..
• If there is any bit alive (1) in the sequence, then convention is that TM
starts at the left most 1.
• Example. Design a Turing machine to compute the function f(n) = 1 for
every n  N  {0}.
Solution aimed in here is switching situation
01111100011100 into
00000010000010.. And starting from {s, a} = {0, 1}..
to the situation
• Another example f(n)=n+1,
Walks n step backward after completing n+1st digit..
• Check the document for to add to numbers f(n) = n + m for every
n  N  {0}.
• Merging two TMs. Suppose two machines with states {0, …, k1-1,
k1 (halting state)}, and {0, …, k2-1, k2 (halting state)}, and the
same alphabets.
• Merging two to M1M2, then new states: {0, …, k1, k1+1, …,
• The number of instruction in binary case, for n states, 2n, and the
number of choices for any particular instruction is (4n+1), then the
number of different Turing machines is (4(n+1))2n, so a finite
collection of TMs…
• A sub collection Hn of Tn will halt if starts from a blank tape. So,
Hn is a finite collection, therefore theoretically possible to count
the steps each Turing Machine takes before it goes into halt.
• B(M) = The number of steps M takes, before it halts, then the function
BBM(n)=(n) = maxMHn B(M),
is known as the busy beaver problem, which is to determine a computing
program which will give value (n) for every n N.
• They are 5-Tuple Turing Machines
• Deterministic Turing machines
• All machines are started on initially blank tapes
• A Turing Machine that writes the maximum number of 1’s for its number of state
∑(n), Rado’s function, which is the maximum number of 1’s left on the tape. S(n)
maximum number of moves that can be made by n-state halting Turing Machine.
• The search state is extremely large
• Not possible to determine whether a particular Turing machine will halt.
• Neither ∑(n) or S(n) are computable functions.
• A problem is Computable if it can be computed by a Turing Machine.
• If the halting problem were solvable then the busy beaver problem would be
• Logically impossible to have a computing program for every n for which the
function (n) is computable. Therefore it is non-computable. Note that for a
particular n N, possible to compute (n) but what is not possible is that there
cannot exist one computing program that applicable for every case of (n).
State 0
State 1
State 0
State 0
State 1
State 2
Two State Machine
Busy Beaver Problem
Three State Machine
… ∆ ∆ ∆ ∆
1 ∆
1 ∆
1 ∆
1 ∆
1 ∆1 ∆ ∆ ∆ …
∆/1, R
∆/1, L
1/1, L
∆/1, L
∆/1, L
∆/1, R
1/1, R
1/1, R
1/1, R
1/1, R
∆/1, L
1/1, L
1/1, R
∑(n): Is non computable
Busy Beaver Problem
– Proof: ∑(n+1) > ∑(n) for all n. Observe that the function (n): NN is strictly
increasing. (n+1)>(n), Suppose MHn, satisfies B(n) = (n), for which n is halting
state. Now consider using M to construct another machine M’Hn+1. Adding one more
state of instructions, in the table, then the halting state will be n+1 to make new M’ to
halt. We replace halting state adding another intermediate state
– So B(M’)=(n)+1  B(M’)(n+1), which follows (n)<(n+1).
– Second step, any computing program on a computer can be describes in terms of a
TM. But No a TM can exist for every nN states (n).
– Thirs step is compare a collection of Turing Machines
1. Suppose a BB, M1 of f(n)=n+1, M1 has 2 states.. halts in 2 steps!!
2. Another BB, M2 of f(n)=2n, M2 has 9 states and one more BB M3 has k states.
3. Congregated Turing Machine Mall = M1(M2 i)M3, (2+9i+k), it halts after ∑(n) =
2+9i+k 1’s..
4. With i copies of TM, M2 will halt after (2i)!!!.
From Mall (2i)(2+9i+k)..
However expression 2+9i+k is linear in i, such that for sufficiently large i, 2i>(2+9i+k)
which contradicts to observation made in 3.
S(n): Is non computable. Proof: There exists a TM M with n states that writes
∑(n) 1’s on its tape before halting. Such a TM must make at least ∑(n) moves.
Hence S(n) ≥ ∑(n).
• First four pages is the summary of the article “NP completeness of
Minesweeper” by Ian Stewart.
• An algorithm - a procedure solving some problem - in the
computational term: how efficient - the running time - the number
of clock time required to get the answer ~ the initial data?
• Theoretically the main distinction between problems that are of
type P - polynomial time- and those that are not.
• A problem is of type P if it can be solved using an algorithm whose
running time grows no faster than some fixed power of the number
of symbols required to specify the initial data. Problems of this
type are easy to solve.
• Otherwise the problem is non-P. Non-P problems are hard. Of
course it's not quite as simple as that, but it's a good rule of thumb.
• Intuitively, problems in P can be solved efficiently, whereas non-P
problems cannot be solved algorithmically in any practical manner
because any algorithm will take a ridiculously long time to get an
• If a problem is of type P, possible to exhibit it in polynomial time.
For example, sorting a list of numbers or and searching a string
for some sequence of symbols, “which is why commercial
databases can sort data; or word processors can carry out searchand-replace operations”.
• In contrast, Traveling Salesman Problem finds the shortest route
whereby a salesman can visit every city on some itinerary is
believed to be non-P, which has never been proved.
• Hard to prove that a problem is non-P? Because requires to
contemplate all possible algorithms, and show none of which can
solve the problem in polynomial time. A mind-boggling task.
• The best that has been done to date is to prove that a broad class
of candidate non-P problems are all on the same footing, if any
one of them can be solved in polynomial time, then all can be.
The problems involved here are said to have 'nondeterministic
polynomial' running time: type NP.
• NP is not the same as non-P. A problem is NP, for which if it is
possible to check a solution in a polynomial time.
• a jigsaw puzzle, if solved, usually quick to check whether the
solution is right, since the checking runs in polynomial time. But
solving the puzzle, trying every potential solution in turn and
checking each, the number of potential solutions grows much faster
than any fixed power of the number of pieces.
• Many of NP problems have 'equivalent' running times.
• Specifically, an NP problem is said to be NP-complete if the
existence of a polynomial time solution for that problem implies
that all (of its sub) NP problems have a polynomial time solution.
Solving one in polynomial time, means all the other ones are in
polynomial time.
• A vast range of problems are known to be NP-complete. The
P=NP? problem asks whether types P and NP are (despite all
appearances to the contrary) the same. The expected answer is 'no'.
However, if any NP-complete problem turns out to be of type P —
- having a polynomial time solution — - than NP must equal P. We
therefore expect all NP-complete problems to be non-P, not proven
• One simplest known NP-complete problem is SAT, the logical
satisfiability of a Boolean condition.
• Boolean circuits are logic gates AND, OR and NOT. Each gate
may accept a number of inputs, and outputs the logical value of
that combination.
• The SAT problem asks, for a given Boolean circuit, whether there
exist choices of inputs that produce the output T. May sound easy!
if a circuit contains a large/or a huge number of gates and inputs.
• The link to the computer game comes with the consistency
problem, for example consistency of the logical assertion.
• Kaye proves that Minesweeper is equivalent to SAT, in the sense
that for a given Boolean circuit a SAT can be 'encoded' as a
Minesweeper Consistency Problem for some position in the game,
using a code procedure that is running in polynomial time.
• Therefore, solving the Minesweeper Consistency Problem in
polynomial time, would mean solving the SAT problem for that
circuit in polynomial time. In other words, Minesweeper is NPcomplete.
Algorithmic complexity is some measure of algorithmic length,
“how complex a system is in terms of the length of the shortest
computer program, or set of algorithms”, to perform the
computation. Rather information theoretical approach is taken,
– Kolmogorov complexity. The complexity of a string x as the length of its
shortest description p on a universal Turing machine U
(K(x)=min{l(p):U(p)=x}). A string is simple if it can be described by a
short program, like "the string of one million ones", and is complex if there
is no such short description, like for a random string whose shortest
description is specifying it bit-by-bit. Independent of the choice of U,
computer. Having similar properties with Shannon's entropy (information)
S, but K is superior!! to S in many respects!! it says. K is an excellent
universal complexity measure, suitable e.g. for quantifying Occam's razor.
Entities should not be multiplied unnecessarily William of Occam.
Drawback is incomputable, approximated by MML/MDL or Lempel-Ziv
compression or others.
– Algorithmic Probability. . "Solomonoff“ takes a universal computer and
randomly generate an input program. The program will compute some
possibly infinite output. The algorithmic probability of any given finite
output prefix q is the sum of the probabilities of the programs that compute
something starting with q. Certain long objects with short programs have
high probability. Incomputable.
– Descriptional complexity.
• Computational complexity rather deals with the running
time and space efficiencies. Classifies the problems by how
hard they are. e.g. the problem of searching an ordered list at
most has lgn time complexity.
– Universal Search complexity "Levin". Time-bounded, computable,
``Levin'' complexity penalizing a slow program by adding the log of its
running time to its length. This leads to computable variants of AC and
AP. It solves all inversion problems in optimal (apart from some
multiplicative constant) time.
• Solving problems quickly, i.e., in close to linear time, or at least in some fraction
of polynomial time as a function of the input size
• Evidence that many important problems can not be solved quickly.
• NP-complete problems are hard problems.
– Use a heuristic: come up with a method for solving a reasonable fraction of the
common cases.
– Solve approximately: come up with a solution that you can prove that is close to
– Use an exponential time solution: if you really have to solve the problem exactly
and stop worrying about finding a better solution.
• Decision problems
– Given an input and a question of a problem, determine if the answer is yes or no
• Optimization problems
– Find a solution with the “best” value
• Optimization problems can be cast as decision problems that are easier to study
– E.g.: Shortest path: for a G = unweighted directed
• Find a path between u and v that uses the fewest edges
• Does a path exist from u to v consisting of at most k edges?
• Class P are called tractable. Consists of (decision)
problems (solvable in polynomial time), performance is
given i.e worst-case running time as O(nk), for some
constant k, O(n2), O(n3), O(1), O(n lg n)
• Class non-P Problems not in P are called intractable or
unsolvable, non-p examples O(2n), O(nn), O(n!)
– But can be solved in reasonable time only for small inputs
– Or, can not be solved at all
• Do non-polynomial algorithms always perform worse
than polynomial algorithms?
– n1,000,000 is technically tractable, but in reality impossible
– nlog log log n is technically intractable, but in reality it is
• Decision problems as sets of encodings of instances; the encodings that are in
the set are "yes" instances.
Definition: P is the class of decision problems D and there exists an algorithm
A returns "yes" if and only if input I is in D. (i.e., A correctly decides D), and
the running time of A = O(nk) where n is the size of the I to be decided and k is
a constant.
Informally, P is the class of decision problems that can be decided in
polynomial time for the given instances. Such an algorithm A is said to run in
polynomial time.
Examples of class P problems:
Instance: A weighted, directed graph G = (V, E), a pair of vertices v and w,
both in V, and a number k in R.
Question: Is there a path in G from v to w of length at most k? This is in P:
Using Dykstra's Algorithm, it runs in time polynomial in the size of the
instance. Proving the result of the decision problem, use the sequence of
vertices along a path of length at most k as a certificate; verify the length of this
path without having to trust Dikstra's Algorithm.
• Problem: LESS-THAN
Instance: Two integers, n and m.
Question: Is n less than m? The size of the problem is O(log n + log m), using
binary representation of the numbers. This problem can be decided in O(log n +
log m) time, which is linear in the sizes of the numbers (i.e., k=1 in the
definition of P).
• P roughly corresponds to algorithmic notion of efficient computation; if a
problem is in P, then it can be solved efficiently.
• Some problems are not in P. For example, consider this problem:
• Problem: HALTING
Instance: A C program A.
Question: Will A ever call exit()? However, there is no general algorithm for
deciding this problem in polynomial time. (It turns out that there is no algorithm
for doing this at all, in any kind of time, but you don't have to believe that until
• It isn't known for many problems whether or not they are in P. For
example: Problem: GRAPH-ISOMORPHISM
Instance: Two graphs G and H.
Question: Are G and H isomorphic? (i.e., are they the same shape? Can we
relabel the vertices of one to make it identical to the other?) It isn't known
whether GRAPH-ISOMORPHISM is in P.
Non-p problems (intractable)
• Turing 1930, Problems unsolvable by any algorithm. The most
well known is the halting problem: Given an arbitrary algorithm
and its input, will that algorithm eventually halt, or will it
continue forever in an “infinite loop?”
• Hamiltonian Paths: Given a G(V, E), find a path visiting every
vertex just once. Searching such a path is an optimization
problem. Decision problem: But deciding if there is such a path
is a Decision Problem of yes or no.
• TSP: Finding a minimum weight Hamiltonian cycle. TSP whose
all the edge weights is set to one, is a special case of Hamiltonian
Decision problem: For a given G and for a total weight of w. Is
there a TSP with a total weight at most w.
• Can be classified in various categories based on their degree of
difficulty, e.g., NP, NPC, NP-hard
Nondeterministic algorithm has two stage procedure:
1) Nondeterministic (“guessing”) stage: generate randomly an
arbitrary string that can be thought of a candidate solution
2) Deterministic (“verification”) stage: take the certificate and the
instance to the problem and returns YES if the certificate
represents a solution in polynomial time.
3) Given a “certificate” of a solution, verify that the certificate is
correct in time polynomial to the size of the input. Verification
stage is polynomial.
All these ends up to be the statement “Verifiable in poly time:
given a certificate of a solution, could verify the certificate is
correct in poly time.”
Hamiltonian-cycle, given a certificate of a sequence (v1,v2,…, vn), easily
verified in poly time.
3-CNF, given a certificate of an assignment 0s, 1s, easily verified in poly
A problem p NP for an instance, and any other instances of the
problem p NP can be translated as p in poly time.
All current known NP hard problems proved to be NPC.
P  NP, NPC  NP
P = NP (or P  NP, or P  NP) ???
NPC = NP (or NPC  NP, or NPC  NP) ???
P  NP: one of the most perplexing open research problems in
(theoretical) computer science since 1971.
No poly algorithm found for any NPC problem (even though
there are so many NPC problems)
There is no proof of statement that a poly algorithm cannot exist
for any of NPC problems.
Theoretical computer scientists believe that NPC is intractable
(i.e., hard, and P  NP).
P  NP, NPC  NP, P  NPC = 
Why searching for NPC
• If proved NPC, a good evidence for its intractability (hardness).
• No need to waste time on trying to find out an efficient algorithm, rather focus
on an approximate algorithm or solution for a special cases
• Some problems might seem to be easy, but in fact, np hard (NPC).
Decision VS. Optimization Problems
• Decision problem: solution is binary “YES” or “NO”
• Optimization problem: seeking solution of an optimal solution.
• Easier to define reduction from and between decision problems than
optimization. NPC is confined to decision problem. (but also applicable to
optimization problem.)
• Examples:
– SHORTEST-PATH (optimization)
Given G, and vertices u, v, find a path from u to v with minimum number of edges
(and with minimum weight).
– PATH (decision)
Given G, vertices u, v, and k, whether exists a path from u to v consisting of at most
k edges.
• If there is solution for an optimization problem, the algorithm can be used to
solve the corresponding decision problem, and if optimization is easy, its
corresponding decision is also easy. Or if provides evidence that decision
problem is hard, then the corresponding optimization problem is also hard.
• Reduction is a way of saying that one problem is “easier” than another.
• We say that problem A is easier than problem B, (i.e., we write “A  B”) if we
can solve A using the algorithm that solves B.
• Idea: transform the inputs of A to inputs of B
• Given two problems A, B, we say that A is polynomially reducible to B
(A p B) if:
– There exists a function f that converts the input of A to inputs of B in polynomial
– A(i) = YES  B(f(i)) = YES
• A problem B is NP-complete if:
– (1) B  NP
– (2) A p B for all A  NP
• If B satisfies only property (2) we say that B is NP-hard
• No polynomial time algorithm has been discovered for an NP-Complete problem
• No one has ever proven that no polynomial time algorithm can exist for any NPComplete problem
Problem (class) and problem instance
Instance  of decision problem A and instance  of
decision problem B
A reduction from A to B is a transformation with the
following properties:
(Poly) Reduction  Algorithm for B
Decision algorithm for A
The time transformation is polynomial and obviously for the
answer is yes for  - if the answer is yes for ).
If decision algorithm for B is poly, so does A.
1. A is no harder than B (or B is no easier than A)
2. If A is hard (e.g., NPC), so does B.
3. How to prove a problem B to be NPC ??
find a already proved NPC problem A
establish a (poly) reduction from A to B
• Any problem in P is also in NP:
P  NP
• The big (and open question) is whether NP  P or P = NP
– i.e., if it is always easy to check a solution, should it also be easy to
find a solution?
• Most computer scientists believe that this is false but we do
not have a proof NP-complete problems are defined as the
hardest problems in NP. Most practical problems turn out to
be either P or NP-complete.
Implications of Reduction
- If A p B and B  P, then A  P
- if A p B and A  P, then B  P
Problem B
Problem A
Proving Polynomial Time
1. Use a polynomial time reduction algorithm to
transform A into B
2. Run a known polynomial time algorithm for B
3. Use the answer for B as the answer for A
Polynomial time
algorithm to decide B
Polynomial time algorithm to decide A
P & NP-Complete Problems
• Shortest simple path
– Given a graph G = (V, E) find a shortest path from a source
to all other vertices
– Polynomial solution: O(VE)
• Longest simple path
– Given a graph G = (V, E) find a longest path from a source
to all other vertices
– NP-complete
• Euler tour
– G = (V, E) a connected, directed graph find a cycle that
traverses each edge of G exactly once (may visit a vertex
multiple times)
– Polynomial solution O(E)
• Hamiltonian cycle
– G = (V, E) a connected, directed graph find a cycle that
visits each vertex of G exactly once
– NP-complete
More Discussion on Poly time problems
• (n100) vs. (2n)
– Reasonable to regard a problem of (n100) as intractable, however, very few practical problem
with (n100).
– Most poly time algorithms require much less.
– Once a poly time algorithm is found, more efficient algorithm may follow soon.
Poly time keeps same in many different computation models, e.g., poly class of serial
random-access machine  poly class of abstract Turing machine  poly class of parallel
computer (#processors grows polynomially with input size)
• Poly time problems have nice closure properties under addition, multiplication and
Encoding impact on complexity
• The problem instance must be represented in a way the program (or machine) can
• General encoding is “binary representation”.
• Different encoding will result in different complexities.
• Example: an algorithm, only input is integer k, running time is (k).
– If k is represented in unary: a string of k 1s, the running time is (k) = (n) on length-n input,
poly on n.
– If k is represented in binary: the input length n =  log k  +1, the running time is (k) = (2n),
exponential on n.
Ruling out unary, other encoding methods are same.
Examples of encoding and complexity
Given integer n, check whether n is a composite.
Dynamic programming for subset-sum.
Class P Problems
• Let n= the length of binary encoding of a problem (i.e., input size), T(n) is the
time to solve it.
• A problem is poly-time solvable if T(n) =O(nk) for some constant k.
• Complexity class P=set of problems that are poly-time solvable.
Poly Time Verification
• PATH problem: Given <G,u,v,k>, whether exists a path from u to v with at most k
• Moreover, also given a path p from u to v, verify whether the length of p is at
most k?
• Easy or not? Easy..
Poly Time Verification, encoding, and language
• Hamiltonian cycles
A simple path containing every vertex.
HAM-CYCLE={<G>: G is a Hamiltonian graph, i.e. containing Hamiltonian cycle}.
Suppose n is the length of encoding of G.
HAM-CYCLE can be considered as a Language after encoding, i.e. a subset of *
where ={0,1}*.
• The naïve algorithm for determining HAM-CYCLE runs in (m!)=(2m) time,
where m is the number of vertices, m n1/2.
• However, given an ordered sequence of m vertices (called “certificate”), let you
verify whether the sequence is a Hamiltonian cycle. Very easy. In O(n2) time.
Class NP problems
• For a problem p, given its certificate, the certificate can be verified in poly time.
• Call this kind of problem an NP one.
• Complement set/class: Co-NP.
– Given a set S (as a universal) and given a subset A
– The complement is that S-A.
– When NP problems are represented as languages (i.e. a set), we can discuss their
complement set, i.e., Co-NP.
NP-completeness and Reducibility
• A (class of) problem P1 is poly-time reducible to P2 , written as P1p P2 if there
exists a poly-time function f: P1  P2 such that for any instance of p1 P1, p1 has
“YES” answer if and only if answer to f(x) ( P2) is also “YES”.
• Theorem 34.3: (page 985)
– For two problems P1, P2, if P1p P2 then P2  P implies P1  P.
• A problem p is NP-complete if
1. p  NP and
2. p'p p for every p'  NP.
(if p satisfies 2, then p is said NP-hard.)
Theorem 34.4 (page 986)
if any NP-compete problem is poly-time solvable, then P=NP. Or say: if any
problem in NP is not poly-time solvable, then no NP-complete problem is polytime solvable.
SAT satisfiability problem
Boolean combinational circuit
Boolean combinational elements, wired together, inputs and outputs
(binary), the number of outputs to 1. logic gates: NOT gate, AND gate, OR
gate. true table: giving the outputs for each setting of inputs,
true assignment is given for a set of boolean inputs,
satisfying assignment: a true assignment causing the output to be 1. A
circuit is satisfiable if it has a satisfying assignment.
Circuit satisfying problem: given a boolean combinational circuit composed of AND,
OR, and NOT, is it satisfiable?
CIRCUIT-SAT={<C>: C is a satisfiable boolean circuit}
Implication: if a subcircuit always produces 0, then the subcircuit can be
replaced by a simpler subcircuit that omits all gates and just output a 0.
Solving circuit-satisfiability problem
• Intuitive solution:
– for each possible assignment, check whether it generates 1.
– suppose the number of inputs is k, then the total possible assignments are 2k. So the running
time is (2k). When the size of the problem is (k), then the running time is not poly.
Circuit-satisfiability problem is NP-complete
• Lemma 34.5:(page 990)
– CIRCUIT-SAT belongs to NP.
Proof: CIRCUIT-SAT is poly-time verifiable.
– Given (an encoding of) a CIRCUIT-SAT problem C and a certificate, which is an assignment of
boolean values to (all) wires in C.
– The algorithm is constructed as follows: just checks each gates and then the output wire of C:
• If for every gate, the computed output value matches the value of the output wire given in the certificate
and the output of the whole circuit is 1, then the algorithm outputs 1, otherwise 0.
• The algorithm is executed in poly time (even linear time).
An alternative certificate: a true assignment to the inputs.
Lemma 34.6: (page 991)
– CIRCUIT-SAT is NP-hard.
Proof: Suppose X is any problem in NP
– construct a poly-time algorithm F maps every problem instance x in X to a circuit C=f(x) such
that the answer to x is YES if and only if CCIRCUIT-SAT (is satisfiable).
Since XNP, there is a poly-time algorithm A which verifies X.
Suppose the input length is n and Let T(n) denote the worst-case running time. Let k be the
constant such that T(n)=O(nk) and the length of the certificate is O(nk).
Circuit-satisfiability problem is NP-hard
• Idea is to represent the computation of A as a sequence of
configurations, c0, c1,…,ci,ci+1,…,cT(n), each ci can be broken into
– (program for A, program counter PC, auxiliary machine state, input x, certificate
y, working storage) and
– ci is mapped to ci+1 by the combinational circuit M implementing the
computer hardware.
– The output of A: 0 or 1– is written to some designated location in working
storage. If the algorithm runs for at most T(n) steps, the output appears as
one bit in cT(n).
– Note: A(x,y)=1 or 0.
• The reduction algorithm F constructs a single combinational
circuit C as follows:
– Paste together all T(n) copies of the circuit M.
– The output of the ith circuit, which produces ci, is directly fed into the
input of the (i+1)st circuit.
– All items in the initial configuration, except the bits corresponding to
certificate y, are wired directly to their known values.
– The bits corresponding to y are the inputs to C.
– All the outputs to the circuit are ignored, except the one bit of cT(n)
corresponding to the output of A.
• Two properties remain to be proven:
– F correctly constructs the reduction, i.e., C is satisfiable if and only if there
exists a certificate y, such that A(x,y)=1.
Suppose there is a certificate y, such that A(x,y)=1. Then if we apply the bits of y
to the inputs of C, the output of C is the bit of A(x,y), that is C(y)= A(x,y) =1, so C
is satisfiable.
Suppose C is satisfiable, then there is a y such that C(y)=1. So, A(x,y)=1.
– F runs in poly time.
• Poly space:
Size of x is n.
Size of A is constant, independent of x.
Size of y is O(nk).
Amount of working storage is poly in n since A runs at most O(nk).
M has size poly in length of configuration, which is poly in O(nk), and hence is poly
in n.
– C consists of at most O(nk) copies of M, and hence is poly in n.
– Thus, the C has poly space.
• The construction of C takes at most O(nk) steps and each step takes poly time, so
F takes poly time to construct C from x.
• In summary
– CIRCUIT-SAT belongs to NP, verifiable in poly time.
– CIRCUIT-SAT is NP-hard, every NP problem can be reduced to CIRCUITSAT in poly time.
– Thus CIRCUIT-SAT is NP-complete.
NP-completeness proof basis
• Lemma 34.8 (page 995)
– If X is a problem (class) such that P'p X for some P'  NPC, then X is NP-hard.
Moreover, if X NP, then X NPC.
• Steps to prove X is NP-complete
– Prove X NP.
• Given a certificate, the certificate can be verified in poly time.
– Prove X is NP-hard.
• Select a known NP-complete P'.
• Describe a transformation function f that maps every instance x of P' into an instance f(x)
of X.
• Prove f satisfies that the answer to xP' is YES if and only if the answer to f(x)X is
YES for all instance x P'.
• Prove that the algorithm computing f runs in poly-time.
NPC proof –Formula Satisfiability (SAT)
• SAT definition
– n boolean variables: x1, x2,…, xn.
– M boolean connectives: any boolean function with one or two inputs and one
output, such as ,,,,,… and
– Parentheses.
• A SAT  is satisfiable if there exists an true assignment which causes  to
evaluate to 1.
• SAT={< >:  is a satifiable boolean formula}.
• The historical honor of the first NP-complete problem ever shown.
SAT is NP-complete
• Theorem 34.9: (page 997)
– SAT is NP-complete.
• Proof:
– SAT belongs to NP.
• Given a satisfying assignment, the verifying algorithm replaces each
variable with its value and evaluates the formula, in poly time.
– SAT is NP-hard (show CIRCUIT-SATp SAT).
• CIRCUIT-SATp SAT, i.e., any instance of circuit
satisfiability can be reduced in poly time to an instance
of formula satisfiability.
• Intuitive induction:
– Look at the gate that produces the circuit output.
– Inductively express each of gate’s inputs as formulas.
– Formula for the circuit is then obtained by writing an
expression that applies the gate’s function to its input
• Unfortunately, this is not a poly reduction
– Shared formula (the gate whose output is fed to 2 or more inputs of other
gates) cause the size of generated formula to grow exponentially.
• Correct reduction:
– For every wire xi of C, give a variable xi in the formula.
– Every gate can be expressed as xo(xi1 xi2…  xil)
– The final formula  is the AND of the circuit output variable and
conjunction of all clauses describing the operation of each gate. (example
Figure 34.10)
• Correctness of the reduction
– Clearly the reduction can be done in poly time.
– C is satisfiable if and only if  is satisfiable.
• If C is satisfiable, then there is a satisfying assignment. This means that each
wire of C has a well-defined value and the output of C is 1. Thus the assignment
of wire values to variables in  makes each clause in  evaluate to 1. So  is 1.
• The reverse proof can be done in the same way.
Example of reduction of CIRCUIT-SAT to SAT
= x10(x10(x7 x8 x9))
(x9(x6  x7))
(x8(x5  x6))
(x7(x1 x2 x4))
(x6 x4))
(x5(x1  x2))
INCORRECT REDUCTION: = x10= x7 x8 x9=(x1 x2 x4)  (x5  x6) (x6  x7)
=(x1 x2 x4)  ((x1  x2)  x4) (x4  (x1 x2 x4))=….
NPC Proof –3-CNF Satisfiability
• 3-CNF definition
– A literal in a boolean formula is an occurrence of a
variable or its negation. A clause is a logical
variable or a number of logical variables.
– CNF (Conjunctive Normal Form) is a boolean
formula expressed as AND of clauses, each of
which is the OR of one or more literals.
– 3-CNF is a CNF in which each clause has exactly
3 distinct literals (a literal and its negation are
• 3-CNF-SAT: whether a given 3-CNF is
3-CNF-SAT is NP-complete
• Proof: 3-CNF-SAT NP. Easy.
– 3-CNF-SAT is NP-hard. (show SAT p3-CNF-SAT)
• Suppose  is any boolean formula, Construct a binary ‘parse’ tree, with
literals as leaves and connectives as internal nodes.
• Introduce a variable yi for the output of each internal nodes.
• Rewrite the formula to ' as the AND of the root variable and a
conjunction of clauses describing the operation of each node.
• The result is that in ', each clause has at most three literals.
• Change each clause into conjunctive normal form as follows:
– Construct a true table, (small, at most 8 by 4)
– Write the disjunctive normal form for all true-table items evaluating to 0
– Using DeMorgan law to change to CNF.
• The resulting '' is in CNF but each clause has 3 or less literals.
• Change 1 or 2-literal clause into 3-literal clause as follows:
– If a clause has one literal l, change it to (lpq)(lpq) (lpq)
– If a clause has two literals (l1 l2), change it to (l1 l2 p)  (l1 l2 p).
Binary parse tree for =((x1 x2) ((x1 x3)  x4))x2
'= y1(y1  (y2x2))
(y2  (y3  y4))
(y4  y5)
(y3  (x1  x2))
(y5  (y6  x4))
(y6  (x1  x3))
Example of Converting a 3-literal clause to CNF format
Disjunctive Normal Form:
(y1y2x2) (y1y2x2)
Conjunctive Normal Form:
3-CNF is NP-complete
•  and reduced 3-CNF are equivalent:
– From  to ' , keep equivalence.
– From ' to '' , keep equivalence.
– From '' to final 3-CNF, keep equivalence.
• Reduction is in poly time,
– From  to ' , introduce at most 1 variable and 1
clause per connective in .
– From ' to '' , introduce at most 8 clauses for each
clause in '.
– From '' to final 3-CNF, introduce at most 4 clauses for
each clause in ''.
NP-completeness proof structure
NPC proof -- CLIQUE
• Definition: a clique in an undirected graph G=(V,E) is a subset
V'V of vertices, each pair of which is connected by an edge in E,
i.e., a clique is a complete subgraph of G.
• Size of a clique is the number of vertices in the clique.
• Optimization problem: find the maximum clique.
• Decision problem: whether a clique of given size k exists in the
• CLIQUE={<G,k>: G is a graph with a clique of size k.}
• Intuitive solution: ???
CLIQUE is NP-complete
• Theorem 34.11: (page 1003)
– CLIQUE problem is NP-complete.
• Proof:
– CLIUEQE NP: given G=(V,E) and a set V'V as a certificate for G. The
verifying algorithm checks for each pair of u,vV', whether <u,v> E. time:
– CLIQUE is NP-hard:
• show 3-CNF-SAT pCLUQUE.
• The result is surprising, since from boolean formula to graph.
=(x1x2x3)(x1x2x3)(x1x2x3) and its reduced graph G
C1=x1x2 x3
CLIQUE is NP-complete
• Reduction from 3-CNF-SAT to CLUQUE.
– Suppose =C1 C2… Ck be a boolean formula in 3-CNF with k clauses.
– We construct a graph G=(V,E) as follows:
• For each clause Cr =(l1r l2r l3r), place a triple of v1r, v2r, v3r into V
• Put the edge between two vertices vir and vjs when:
– rs, that is vir and vjs are in different triples, and
– Their corresponding literals are consistent, i.e, lir is not negation of ljs .
– Then  is satisfiable if and only if G has a clique of size k.
• Prove the above reduction is correct:
– If  is satisfiable, then there exists a satisfying assignment, which makes at least one
literal in each clause to evaluate to 1. Pick one this kind of literal in each clause.
Then consider the subgraph V' consisting of the corresponding vertex of each such
literal. For each pair vir,vjs V', where rs. Since lir,ljs are both evaluated to 1, so lir is
not negation of ljs, thus there is an edge between vir and vjs. So V' is a clique of size k.
– If G has a clique V' of size k, then V' contains exact one vertex from each triple.
Assign all the literals corresponding to the vertices in V' to 1, and other literals to 1
or 0, then each clause will be evaluated to 1. So  is satisfiable.
• It is easy to see the reduction is in poly time.
• The reduction of an instance of one problem to a specific instance of the other
Traveling-salesman problem is NPC
• TSP={<G,c,k>:
G=(V,E) is a complete graph,
c is a function from VVZ,
kZ, and G has a traveling salesman
tour with cost at most k.}
• Theorem 34.14: (page 1012)
– TSP is NP-complete.
TSP is NP-complete
• TSP belongs to NP:
– Given a certificate of a sequence of vertices in the tour, the verifying algorithm
checks whether each vertex appears once, sums up the cost and checks whether at
most k. in poly time.
• TSP is NP-hard (show HAM-CYCLEpTSP)
– Given an instance G=(V,E) of HAM-CYCLE, construct a TSP instance <G',c,0) as
follows (in poly time):
• G'=(V,E'), where E'={<i,j>: i,j V and ij} and
• Cost function c is defined as c(i,j)=0 if (i,j) E, 1, otherwise.
– If G has a hamiltonian cycle h, then h is also a tour in G' with cost at most 0.
– If G' has a tour h' of cost at most 0, then each edge in h' is 0, so each edge belong
to E, so h' is also a hilmitonian cycle in G.
Subset Sum is NPC
• Subset sum problem: is there a non-empty subset of positive
integers s1, s2, s3, …, sn having a sum exactly equal to zero? Or an
equivalent one asking does any subset sum to an integer t
(knapsack special case)?
• {1, 6, -3, -2, -5, 7, 9}, t =7 kind.. A = [7], [1, 6], [-2, -5, 7, 1, 6], [3, -2, -5, 1, 6, 5].. Many subsets are possible..
• Another special case partition problem: partition set into two
subsets such that total sum is half sum of all elements in the set.
(also a special case of knapsack.)
• NP-Complete, NP-Hard, because, subset problem is a decision
problem (not an optimization prb)..
• Precision of ± 1% amounts solving the problem just for the first 7
digits and + 1 for the sign.. However if there are 2100, bits, this
will be 7% of the constraints solved, think that every bit represents
an answer of yes or no decision for one question, 27 out of 2100.
remaining 293 unsolved problems. The only way subset solution
can be a solution to other NP problems is to solve all the problems
Subset Sum is NPC
• Very much applicable in cryptology against crytoattacks.
• Studied for approximation problems as well.
• SSS can be viewed as depending on two variables. N the number of decision
variables, P is the precision of the problem. Problem is most difficult when both
variables are in the same order, it is exponential. Becomes easier when either of
variables gets smaller.
• The methods practical
– Exhaustive search: when the number of variables N is small
– Dynamic algorithms applicable when the precision is a small fixed number.
• Solution becomes nonexponential and practical if entire solution space is
countable.. Two ways to count, SSS problem in example, 2N possible
combinations of variables that can be counted for small N, or 2P numerical sums
all of which can be counted with the dynamic algorithm.. When N=P, and both
large no solution space can be counted.
• Checking if subset sums to right number, 2N subsets, N elements O(2N N).
Exponential time. This can be reduced by sorting, by splitting N elements into
two sets, and calculating 2N/2 sums, in two arrays, sort, and checking the sum,
one array in ascending and the other in descending order, and starting.
In addressing subset and permutation prblms
Backtracking Branch and Bound
• Subset problem of size n.
 Nonsystematic search of the space takes O(p2n) time, where p is the time
needed to evaluate each member of the solution space.
• Permutation problem of size n.
 Nonsystematic search takes O(pn!) time. P is the same as above.
• A systematic approach is backtracking and branch and bound
perform; often taking less time than the time taken by a exhoustive
• Solution space: A tree structure such that the leaves represent
members of the solution space. For a size n, this tree
– subset problem structure has 2n leaves.
– permutation problem structure has n! leaves.
• Limitations: Memory and time: too big trees require to much
memory and waste to long time.
• Portions of the tree are created by the backtracking and branch and
bound algorithms as needed.
Tree for Subset Problem (2n leaves)
• At level i the members of the solution space are
partitioned by their xi values.
• Members with xi = 1 are in the left subtree.
• Members with xi = 0 are in the right subtree.
x 1= 0
Subset Tree For n = 4
x2 = 0
x2= 0
x3= 0
Permutation Tree (n! Leaves) for n = 3
• At level i the members of the solution space are partitioned
by their xi values.
• Members (if any) with xi = 1 are in the first subtree.
• Members (if any) with xi = 2 are in the next subtree.
x2= 2
x2= 3
x2= 1
x1= 3
x2= 3
x2= 1
x2= 2
Backtracking and bounding, DFS Algo
• Recursive implementation is applicable. Need a stack to retain
the path from the root to the branch traversed. Search from dark
red to light goes on..
• The solution space tree exists only on mind.
Bounding functions:
• While moving deeper if xi==1, s++ (the sum of the subset),
moving backward if xi== 1, s--. If xi == 0, remains indifferent.
• Keeping the track of the sum in a variable of s: s+=s+( forward
xi==1 ) - ( backward xi==1 ).
x1= 0
• T is the target value, visiting a node,
x2= 0
– s==t, terminate,
x2= 0
– s>t backtrack to parent to parent node, else move forward into the
• A second variable r keeping the sum of the nodes not visited
yet. Before moving to a right child, check if the current subset
sum (s + r) >= t. If not, backtrack.
Branch and Bound
• DFS Each move takes O(1) time
back of forward. Space required is
O(tree height).
• Search the tree using a breadth-first
search (FIFO branch and bound).
• Search the tree as in a bfs, but
replace the FIFO queue with a stack
(LIFO branch and bound).
• Replace the FIFO queue with a
priority queue (least-cost (or max
priority) branch and bound).
• The priority of a node p in the
queue is based on an estimate of the
likelihood that the answer node is
in the subtree whose root is p.
• Space required is O(number of
• Finds solution closest to root.
Least-cost branch and bound directs
the search to parts of the space
most likely to contain the answer.
So it could perform better than
• With effective bounding
functions, large instances can
often be solved.
• For some problems (e.g., 0/1
knapsack), the answer (or a very
good solution) may be found
quickly but a lot of additional
time is needed to complete the
search of the tree.
• Time limitations: Run
backtracking if time is feasible.
• May never find a solution if tree
depth is infinite.
Subset Sum is NPC
• SUBSET-SUM={<S,t>: S is a set of integers and
there exists a S'S such that t=sS's.}
• Theorem 34.15: (page 1014)
– SUBSET-SUM is NP-complete.
• SUBSET-SUM belongs to NP.
– Given a certificate S', check whether t is sum of S'
can be finished in poly time.
• SUBSET-SUM is NP-hard (show 3-CNFSATpSUBSET-SUM).
• Given a 3-CNF formula =C1 C2… Ck with
literals x1, x2,…, xn. Construct a SUBSET-SUM
instance as follows:
– Two assumptions: no clause contains both a literal
and its negation, and either a literal or
its negation appears in at least one clause.
– The numbers in S are based on 10 and have n+k
digits, each digit corresponds to (or is labeled by) a
literal or a clause.
– Target t=1…1||4…4 (n 1’s and k 4’s)
– Target t=1…1||4…4 (n 1’s and k 4’s)
– For each literal xi, create two integers:
• vi=0…01(i)0…0||0…01(l)0…01(w)0…0, where xi appears in Cl,…,Cw.
• vi'=0…01(i)0…0||0…1(m)0……01(p)0…0, where xi appears in
Cm,…,Cp. .
• Clearly, vi and vi' can not be equal in right k digits, moreover all vi and vi' in S
are distinct.
– For each clause Cj,
create two integers:
• sj=0…0||0…01(j)0…0,
• sj'=0…0||0…02(j)0…0.
• all sj and sj' are called “slack number”. Clearly, all sj and sj' in S are distinct.
– Note: the sum of digits in any one digit position is 2 or 6, so when there is
no carries when adding any subset of the above integers.
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
• The above reduction is done in poly time.
• The 3-CNF formula  is satisfiable if and only if there
is a subset S' whose sum is t.
– suppose  has a satisfying assignment.
• Then for i=1,…,n, if xi=1 in the assignment, then vi is put in S',
otherwise, then vi' is put in S'.
• The digits labeled by literals will sum to 1.
• Moreover, for each digit labeled by a clause Cj and in its three
literals, there may be 1, 2, or 3 assignments to be 1. correspondingly,
both sj and sj' or sj', or sj is added to S' to make the sum of the digit to
• So S' will sum to 1…14…4.
– Suppose there is a S' which sums to 1…14…4. then S'
contains exact one of vi and vi' for i=1,…,n. if vi S', then set
xi=1, otherwise, vi' S', then set xi=0. It can be seen that this
assignment makes each clause of  to evaluate to 1. so  is