Chapter III: Markov chains

Admistrivia

• Readings: Chapter III up to page 112
• Questions from the homework?

Models

Example from my research

• "Cooperation in the Short and in the Long Run," with H.P. Young, Games and Economic Behavior, 3, (1991), 145 - 156.
  • Game theory
  • Prisoners Dilemma: Cooperate, defect, tit-for-tat
  • We built a model with these three types and modelled evolution of them
  • Draw matrix
• Interested in what happens several moves from now
  • Suppose start in T4T: where will be 2 moves from now?
  • Suppose start in T4T: where will be 10 moves from now?
  • Chapter 4 answers 1000 moves from now.
• Very large sum
• There is a better way
  • P(1 -> ? -> 3) = sum over all possiblilites
  • Write it out
  • Looks fimilar? Row times column of P matrix
• n-step transition is generated by multiplying matrixes

S,s model (see book for details)

• Inventory model: s,S policy (from OPIM)
• inventory always decrease by amount purshased -- unless short fall!
• P(out of stock) = ???
• E(X_t) = ???
• P(X_t > X_t-1) =
• To "fix" first one, make s large
• To "fix" second one make S small
• to "fix" last one, make S-s small
• Oops: can't do all three at once

Genetics model (see book)

Queuing

• Queueing model of the "The Greek Lady"
• Infinite queues as model
• If arrival rate is higher than service rate--it grows with time
• X_t = (birth - death) t approximately
• Queue drop out solves this
• higher service rate solves this

Summary

• Many of these questions are about long run
• X_t not only object of focus
• All these questions can be solved via first step analysis!