# Bayesian Statistics Homework Solutions

### Spring 2011

### Instructor

David Hitchcock, assistant professor of statistics### Syllabus

Syllabus: (Word document) or (pdf document)### Office Hours -- Spring 2011

Mon 1:00 p.m.-2:00 p.m., Tues 11:00-11:55 a.m., Wed 1:00 p.m.- 2:00 p.m., Thu 11:00-11:55 a.m., Fri 10:30-11:30 a.m., or please feel free to make an appointment to see me at other times.Office: 209A LeConte College

Phone: 777-5346

E-mail: hitchcock@stat.sc.edu

### Class Meeting Time

Tues-Thurs 9:30 a.m. - 10:45 a.m., Davis 216### Prerequisites:

STAT/MATH 511 and STAT 515 or equivalent.### Current Textbook (Recommended, not Required for Spring 2011):

Gill, Jeff.*Bayesian Methods: A Social and Behavioral Sciences Approach*, Second Edition. Chapman & Hall/CRC Press, 2007.

### Course Outline:

Topics covered include: Principles of Bayesian statistics; one- and two-sample Bayesian models; Bayesian linear and generalized linear models; Monte Carlo approaches to model fitting; Prior elicitation; Hypothesis testing and model selection; Complex error structures, hierarchical models; Statistical packages such as BUGS/WinBugs, R, or SAS.__ Learning Objectives:__ By the end of the term successful students should be able to do the following:

### Graded Assignments

Two in-class exams, plus a final exam. Occasional homework assignments.### Computing Tips and Examples: R

### Computing Tips and Examples: WinBUGS

- WinBUGS Home Page with instructions for downloading WinBUGS, installing the patch for version 1.4.3, and downloading the key for unrestricted use

### Homework

### Homework Solutions

### Data Sets

### Information about Project

Information about the Project for Spring 2011: (Word document) or (pdf document)### Review Sheets for Exams

### Formula Sheets for Exams

### Exams

- Exam 1: February 8
- Exam 2: March 22
- Final Exam: Monday, May 2 - 2:00 p.m.

MA40189: Topics in Bayesian statistics |

Lectures and timetable information |

Lecturer: | Simon Shaw; s.shaw at bath.ac.uk | |

Timetable: | Lectures: | Monday 09:15 (3W4.7) and Tuesday 14:15 (3W4.7). |

Problems classes: | Thursday 16:15 (3W4.7). |

The full unit timetable is available here. A schedule for the course is available here.

Credits: | 6 |

Level: | Masters |

Period: | Semester 2 |

Assessment: | EX 100% |

Other work: | There will be weekly question sheets. These will be set and handed in during problems classes. Any work submitted by the hand-in deadline will be marked and returned to you. Full solutions to all exercises and general feedback sheets will be made available. |

Requisites: | Before taking this unit you must take MA40092 (home-page). |

Description: | Aims & Learning Objectives: Aims: To introduce students to the ideas and techniques that underpin the theory and practice of the Bayesian approach to statistics. Objectives: Students should be able to formulate the Bayesian treatment and analysis of many familiar statistical problems. Content: Bayesian methods provide an alternative approach to data analysis, which has the ability to incorporate prior knowledge about a parameter of interest into the statistical model. The prior knowledge takes the form of a prior (to sampling) distribution on the parameter space, which is updated to a posterior distribution via Bayes' Theorem, using the data. Summaries about the parameter are described using the posterior distribution. The Bayesian Paradigm; decision theory; utility theory; exchangeability; Representation Theorem; prior, posterior and predictive distributions; conjugate priors. Tools to undertake a Bayesian statistical analysis will also be introduced. Simulation based methods such as Markov Chain Monte Carlo and importance sampling for use when analytical methods fail. |

We won't follow a book as such but useful references, in ascending order of difficulty, include:

- Peter M. Lee,
**Bayesian Statistics: an introduction**, Fourth Edition, 2012.

Very readable, introductory text. This edition is not in the library but the**full text**is available as an e-book here. The Third Edition is available in the library. Further details about the book can be found here. This includes all the exercises in the book and their solutions. - Andrew Gelman, John B. Carlin, Hal S. Stern and Donald B. Rubin,
**Bayesian Data Analysis**, Second Edition, 2004.**512.795 GEL**

A slightly more advanced introductory text with a focus upon practical applications. The**full text**is available as an e-book, either by following the link from the library here or directly here. Further details about the book can be found on Andrew Gelman's pages here. This includes some of the solutions to exercises in the book. Andrew Gelman also has a blog which often raises some interesting statistical topics, frequently related to current news topics. - Christian P. Robert,
**The Bayesian Choice From Decision-Theoretic Foundations to Computational Implementation**, Second Edition, 2007.

This book is not in the library but the**full text**is available here. A really nice book; Christian Robert also has a wide-ranging blog. - Anthony O'Hagan,
**Kendall's Advanced Theory of Statistics Volume 2B Bayesian Inference**, 1994.**512.795 KEN**

A harder, more advanced, book than the previous two but a rewarding and insightful one. It has a good mix of theory and foundations: a personal favourite. - Jose M. Bernardo and Adrian F.M. Smith,
**Bayesian Theory**, 1994.**512.795 BER**

The classic**graduate**text. Develops the Bayesian view from a foundational standpoint. A very readable short overview to Bayesian statistics written by Jose Bernardo can be downloaded from here.

Lecture notes and summaries |

**Lecture notes:**pdf.

**Table of useful distributions:**pdf (Handed out in Problems Class of 15 Feb 18)

**Material covered:**

Lecture 1 (05 Feb 18): | Introduction: working definitions of classical and Bayesian approaches to inference about parameters. |

Lecture overview: pdf. Handwritten notes: pdf. Online notes: p4-5 (middle of Example 3). | |

Lecture 2 (06 Feb 18): | Â§1 The Bayesian method: Bayes' theorem, using Bayes' theorem for parametric inference. |

Lecture overview: pdf. Handwritten notes: pdf. Online notes: p5 (middle of Example 3)-7 (end of page). | |

Lecture 3 (12 Feb 18): | Sequential data updates, conjugate Bayesian updates, Beta-Binomial example. |

Lecture overview: pdf. Handwritten notes: pdf. Online notes: p7 (end of page)-9 (after equation (1.11)). | |

Lecture 4 (13 Feb 18): | Definition of conjugate family, role of prior (weak and strong) and likelihood in the posterior. Handout of beta distributions: pdf. |

Lecture overview: pdf. Handwritten notes: pdf. Online notes: p9 (after equation (1.11))-10 (end of page). | |

Lecture 5 (19 Feb 18): | Example of weak/strong prior finished, kernel of a density, conjugate Normal example. Handout of weak/strong prior example: pdf. |

Lecture overview: pdf. Handwritten notes: pdf. Online notes: p10 (end of page)-13 (prior to equation (1.17)). | |

Lecture 6 (20 Feb 18): | Conjugate Normal example concluded. Using the posterior for inference, credible interval. |

Lecture overview: pdf. Handwritten notes: pdf. Online notes: p13 (prior to equation (1.17))-15 (table in Example 9). | |

Lecture 7 (26 Feb 18): | Highest density regions, Â§2 Modelling: predictive distribution, Binomial-Beta example. |

Lecture overview: pdf. Handwritten notes: pdf. Online notes: p15 (table in Example 9)-19 (equation (2.3)). | |

Lecture 8 (27 Feb 18): | Predictive summaries, finite exchangeability. |

Lecture overview: pdf. Handwritten notes: pdf. Online notes: p19 (equation (2.3))-21 (end of Example 14). | |

Lecture 9 (06 Mar 18): | Infinite exchangeability, example of non-extendability of finitely exchangeable sequence, general representation theorem for infinitely exchangeable events. |

Lecture overview: pdf. Handwritten notes: pdf. Online notes: p21 (end of Example 14)-23 (end of point 3). | |

Lecture 10 (08 Mar 18): | General representation theorem for infinitely exchangeable random variables, example of exchangeable Normal random variables. |

Lecture overview: pdf. Handwritten notes: pdf. Online notes: p23 (end of point 3)-25 (prior to Section 2.3). | |

Lecture 11 (13 Mar 18): | Sufficiency, k-parameter exponential family, sufficient statistics, conjugate priors for exchangeable k-parameter exponential family random quantities. |

Lecture overview: pdf. Handwritten notes: pdf. Online notes: p25 (prior to Section 2.3)-27 (equation (2.22)). |

**Forthcoming material (based upon the 2016/17 schedule):**

Lecture 12 (15 Mar 18): | Hyperparameters, usefulness of conjugate priors. |

Lecture 13 (20 Mar 18): | Improper priors, Fisher information matrix, Jeffreys' prior. |

Lecture 14 (22 Mar 18): | Invariance property under transformation of the Jeffreys prior, final remarks about noninformative priors, Â§3 Computation: preliminary issues. |

Lecture 15 (10 Apr 18): | Normal approximation, expansion about the mode, Monte Carlo integration. |

Lecture 16 (12 Apr 18): | Importance sampling. Basic idea of Markov chain Monte Carlo (MCMC): transition kernel. Handout: pdf. |

Lecture 17 (17 Apr 18): | Basic definitions (irreducible, periodic, recurrent, ergodic, stationary) and theorems (existence/uniqueness, convergence, ergodic) of Markov chains and their consequences for MCMC techniques. The Metropolis-Hastings algorithm. |

Lecture 18 (19 Apr 18): | Example of the Metropolis-Hastings algorithm. Handout of example: pdf. |

Lecture 19 (24 Apr 18): | The Gibbs sampler algorithm and example. Handout of example: pdf. |

Lecture 20 (26 Apr 18): | Overview of why the Metropolis-Hastings algorithm works, efficiency of MCMC algorithms. |

Lecture 21 (01 May 18): | Â§4 Decision theory: Statistical decision theory: loss, risk, Bayes risk and Bayes rule. |

Lecture 22 (03 May 18): | Quadratic loss, Bayes risk of the sampling procedure, worked example . |

gibbs2 | Gibbs sampler for Î¸1|Î¸2 ~ Bin(n, Î¸2), Î¸2|Î¸1 ~ Beta(Î¸1+Î±, n-Î¸1+Î²), illustration of Example 32 (p54) of the lecture notes. Sample plots for n=10, Î±=Î²=1 and n=10, Î±=2, Î²=3: pdf. |

gibbs.update | Step by step illustration of the Gibbs sampler for bivariate normal, X, Y standard normal with Cov(X, Y) = rho; press return to advance. |

gibbs | Long run version of the Gibbs sampler for bivariate normal, X, Y standard normal with Cov(X, Y) = rho. |

metropolis.update | Step by step illustration of Metropolis-Hastings for sampling from N(mu.p,sig.p^2) with proposal N(theta[t-1],sig.q^2); press return to advance. |

metropolis | Long run version of Metropolis-Hastings for sampling from N(mu.p,sig.p^2) with proposal N(theta[t-1],sig.q^2). |

Illustration of Example 30 (p45) of the lecture notes using metropolis.update and metropolis with mu.p=0, sig.p=1 and firstly sig.q=1 and secondly sig.q=0.6: pdf. | |

plot.mcmc | Plot time series summaries of output from a Markov chain. Allows you to specify burn-in and thinning. |

f | Function for plotting bivariate Normal distribution in gibbs.update. |

All above | All of the above functions in one file for easy reading into R; thanks to Ruth Salway for these functions. |

The following functions are for sampling from bivariate normals, with thanks to Merrilee Hurn

gibbs1 | Gibbs sampler (arguments: n the number of iterations, rho the correlation coefficient of the bivariate normal, start1 and start2 the initial values for the sampler). |

metropolis1 | Metropolis-Hastings (arguments: n the number of iterations, rho the correlation coefficient of the bivariate normal, start1 and start2 the initial values for the sampler, tau the standard deviation of the Normal proposal). |

metropolis2 | Metropolis-Hastings for sampling from a mixture of bivariate normals (arguments: n the number of iterations, rho the correlation coefficient of the bivariate normal, start1 and start2 the initial values for the sampler, tau the standard deviation of the Normal proposal, sigma2 the variance of the normal mixtures). |

Question sheets and solutions |

Question Sheet Zero:pdf. | Solution Sheet Zero:pdf.Problems class notes: pdf. |

Question Sheet One:pdf. | Solution Sheet One:pdfProblems class notes: pdf. |

Question Sheet Two:pdf. | Solution Sheet Two:pdfProblems class notes: pdf. |

Question Sheet Three:pdf. | Solution Sheet Three:pdfProblems class notes: pdf. |

Question Sheet Four:pdf. | Solution Sheet Four:pdf (Additional questions only).Full solutions available on 19 Mar 18 Problems class notes: pdf. |

Past exam papers and solutions |

## 0 thoughts on “Bayesian Statistics Homework Solutions”

-->