Bayesian Statistics Homework Solutions


Spring 2011

Instructor

David Hitchcock, assistant professor of statistics

Syllabus

Syllabus: (Word document) or (pdf document)

Office Hours -- Spring 2011

Mon 1:00 p.m.-2:00 p.m., Tues 11:00-11:55 a.m., Wed 1:00 p.m.- 2:00 p.m., Thu 11:00-11:55 a.m., Fri 10:30-11:30 a.m., or please feel free to make an appointment to see me at other times.

Office: 209A LeConte College
Phone: 777-5346
E-mail: hitchcock@stat.sc.edu

Class Meeting Time

Tues-Thurs 9:30 a.m. - 10:45 a.m., Davis 216

Prerequisites:

STAT/MATH 511 and STAT 515 or equivalent.

Current Textbook (Recommended, not Required for Spring 2011):

Gill, Jeff. Bayesian Methods: A Social and Behavioral Sciences Approach, Second Edition. Chapman & Hall/CRC Press, 2007.

Course Outline:

Topics covered include: Principles of Bayesian statistics; one- and two-sample Bayesian models; Bayesian linear and generalized linear models; Monte Carlo approaches to model fitting; Prior elicitation; Hypothesis testing and model selection; Complex error structures, hierarchical models; Statistical packages such as BUGS/WinBugs, R, or SAS.

Learning Objectives: By the end of the term successful students should be able to do the following:

  • Understand the philosophy of Bayesian statistical modeling
  • Understand Bayesian models for numerous common data analysis situations, including prior elicitation
  • Use software such as R, BUGS, or SAS to implement Bayesian analyses
  • Understand basic principles of both conjugate analyses and MCMC-based Bayesian analyses

    Graded Assignments

    Two in-class exams, plus a final exam. Occasional homework assignments.

    Computing Tips and Examples: R

    Computing Tips and Examples: WinBUGS

    • WinBUGS Home Page with instructions for downloading WinBUGS, installing the patch for version 1.4.3, and downloading the key for unrestricted use

    Homework

    Homework Solutions

    Data Sets

    Information about Project

    Information about the Project for Spring 2011: (Word document) or (pdf document)

    Review Sheets for Exams

    Formula Sheets for Exams

    Exams

    • Exam 1: February 8
    • Exam 2: March 22
    • Final Exam: Monday, May 2 - 2:00 p.m.
  • MA40189: Topics in Bayesian statistics


    Lectures and timetable information
    Lecturer: Simon Shaw; s.shaw at bath.ac.uk
    Timetable: Lectures: Monday 09:15 (3W4.7) and Tuesday 14:15 (3W4.7).
    Problems classes: Thursday 16:15 (3W4.7).

    The full unit timetable is available here. A schedule for the course is available here.


    Credits: 6
    Level: Masters
    Period: Semester 2
    Assessment:EX 100%
    Other work:There will be weekly question sheets. These will be set and handed in during problems classes. Any work submitted by the hand-in deadline will be marked and returned to you. Full solutions to all exercises and general feedback sheets will be made available.
    Requisites:Before taking this unit you must take MA40092 (home-page).
    Description:Aims & Learning Objectives:
    Aims:
    To introduce students to the ideas and techniques that underpin the theory and practice of the Bayesian approach to statistics.
    Objectives:
    Students should be able to formulate the Bayesian treatment and analysis of many familiar statistical problems.

    Content:
    Bayesian methods provide an alternative approach to data analysis, which has the ability to incorporate prior knowledge about a parameter of interest into the statistical model. The prior knowledge takes the form of a prior (to sampling) distribution on the parameter space, which is updated to a posterior distribution via Bayes' Theorem, using the data. Summaries about the parameter are described using the posterior distribution. The Bayesian Paradigm; decision theory; utility theory; exchangeability; Representation Theorem; prior, posterior and predictive distributions; conjugate priors. Tools to undertake a Bayesian statistical analysis will also be introduced. Simulation based methods such as Markov Chain Monte Carlo and importance sampling for use when analytical methods fail.

    We won't follow a book as such but useful references, in ascending order of difficulty, include:

    1. Peter M. Lee, Bayesian Statistics: an introduction, Fourth Edition, 2012.
      Very readable, introductory text. This edition is not in the library but the full text is available as an e-book here. The Third Edition is available in the library. Further details about the book can be found here. This includes all the exercises in the book and their solutions.
    2. Andrew Gelman, John B. Carlin, Hal S. Stern and Donald B. Rubin, Bayesian Data Analysis, Second Edition, 2004. 512.795 GEL
      A slightly more advanced introductory text with a focus upon practical applications. The full text is available as an e-book, either by following the link from the library here or directly here. Further details about the book can be found on Andrew Gelman's pages here. This includes some of the solutions to exercises in the book. Andrew Gelman also has a blog which often raises some interesting statistical topics, frequently related to current news topics.
    3. Christian P. Robert, The Bayesian Choice From Decision-Theoretic Foundations to Computational Implementation, Second Edition, 2007.
      This book is not in the library but the full text is available here. A really nice book; Christian Robert also has a wide-ranging blog.
    4. Anthony O'Hagan, Kendall's Advanced Theory of Statistics Volume 2B Bayesian Inference, 1994. 512.795 KEN
      A harder, more advanced, book than the previous two but a rewarding and insightful one. It has a good mix of theory and foundations: a personal favourite.
    5. Jose M. Bernardo and Adrian F.M. Smith, Bayesian Theory, 1994. 512.795 BER
      The classic graduate text. Develops the Bayesian view from a foundational standpoint. A very readable short overview to Bayesian statistics written by Jose Bernardo can be downloaded from here.
    The International Society for Bayesian Analysis (ISBA) is a good starting point for a number of Bayesian resources.
    Lecture notes and summaries
    Lecture notes:pdf.
    Table of useful distributions:pdf (Handed out in Problems Class of 15 Feb 18)

    Material covered:
    Lecture 1 (05 Feb 18): Introduction: working definitions of classical and Bayesian approaches to inference about parameters.
    Lecture overview: pdf. Handwritten notes: pdf. Online notes: p4-5 (middle of Example 3).
    Lecture 2 (06 Feb 18): §1 The Bayesian method: Bayes' theorem, using Bayes' theorem for parametric inference.
    Lecture overview: pdf. Handwritten notes: pdf. Online notes: p5 (middle of Example 3)-7 (end of page).
    Lecture 3 (12 Feb 18): Sequential data updates, conjugate Bayesian updates, Beta-Binomial example.
    Lecture overview: pdf. Handwritten notes: pdf. Online notes: p7 (end of page)-9 (after equation (1.11)).
    Lecture 4 (13 Feb 18): Definition of conjugate family, role of prior (weak and strong) and likelihood in the posterior. Handout of beta distributions: pdf.
    Lecture overview: pdf. Handwritten notes: pdf. Online notes: p9 (after equation (1.11))-10 (end of page).
    Lecture 5 (19 Feb 18): Example of weak/strong prior finished, kernel of a density, conjugate Normal example. Handout of weak/strong prior example: pdf.
    Lecture overview: pdf. Handwritten notes: pdf. Online notes: p10 (end of page)-13 (prior to equation (1.17)).
    Lecture 6 (20 Feb 18): Conjugate Normal example concluded. Using the posterior for inference, credible interval.
    Lecture overview: pdf. Handwritten notes: pdf. Online notes: p13 (prior to equation (1.17))-15 (table in Example 9).
    Lecture 7 (26 Feb 18): Highest density regions, §2 Modelling: predictive distribution, Binomial-Beta example.
    Lecture overview: pdf. Handwritten notes: pdf. Online notes: p15 (table in Example 9)-19 (equation (2.3)).
    Lecture 8 (27 Feb 18): Predictive summaries, finite exchangeability.
    Lecture overview: pdf. Handwritten notes: pdf. Online notes: p19 (equation (2.3))-21 (end of Example 14).
    Lecture 9 (06 Mar 18): Infinite exchangeability, example of non-extendability of finitely exchangeable sequence, general representation theorem for infinitely exchangeable events.
    Lecture overview: pdf. Handwritten notes: pdf. Online notes: p21 (end of Example 14)-23 (end of point 3).
    Lecture 10 (08 Mar 18): General representation theorem for infinitely exchangeable random variables, example of exchangeable Normal random variables.
    Lecture overview: pdf. Handwritten notes: pdf. Online notes: p23 (end of point 3)-25 (prior to Section 2.3).
    Lecture 11 (13 Mar 18): Sufficiency, k-parameter exponential family, sufficient statistics, conjugate priors for exchangeable k-parameter exponential family random quantities.
    Lecture overview: pdf. Handwritten notes: pdf. Online notes: p25 (prior to Section 2.3)-27 (equation (2.22)).

    Forthcoming material (based upon the 2016/17 schedule): 

    Lecture 12 (15 Mar 18): Hyperparameters, usefulness of conjugate priors.
    Lecture 13 (20 Mar 18): Improper priors, Fisher information matrix, Jeffreys' prior.
    Lecture 14 (22 Mar 18): Invariance property under transformation of the Jeffreys prior, final remarks about noninformative priors, §3 Computation: preliminary issues.
    Lecture 15 (10 Apr 18): Normal approximation, expansion about the mode, Monte Carlo integration.
    Lecture 16 (12 Apr 18): Importance sampling. Basic idea of Markov chain Monte Carlo (MCMC): transition kernel. Handout: pdf.
    Lecture 17 (17 Apr 18): Basic definitions (irreducible, periodic, recurrent, ergodic, stationary) and theorems (existence/uniqueness, convergence, ergodic) of Markov chains and their consequences for MCMC techniques. The Metropolis-Hastings algorithm.
    Lecture 18 (19 Apr 18): Example of the Metropolis-Hastings algorithm. Handout of example: pdf.
    Lecture 19 (24 Apr 18): The Gibbs sampler algorithm and example. Handout of example: pdf.
    Lecture 20 (26 Apr 18): Overview of why the Metropolis-Hastings algorithm works, efficiency of MCMC algorithms.
    Lecture 21 (01 May 18): §4 Decision theory: Statistical decision theory: loss, risk, Bayes risk and Bayes rule.
    Lecture 22 (03 May 18): Quadratic loss, Bayes risk of the sampling procedure, worked example .

    gibbs2Gibbs sampler for θ1|θ2 ~ Bin(n, θ2), θ2|θ1 ~ Beta(θ1+α, n-θ1+β), illustration of Example 32 (p54) of the lecture notes. Sample plots for n=10, α=β=1 and n=10, α=2, β=3: pdf.
    gibbs.updateStep by step illustration of the Gibbs sampler for bivariate normal, X, Y standard normal with Cov(X, Y) = rho; press return to advance.
    gibbsLong run version of the Gibbs sampler for bivariate normal, X, Y standard normal with Cov(X, Y) = rho.
    metropolis.updateStep by step illustration of Metropolis-Hastings for sampling from N(mu.p,sig.p^2) with proposal N(theta[t-1],sig.q^2); press return to advance.
    metropolisLong run version of Metropolis-Hastings for sampling from N(mu.p,sig.p^2) with proposal N(theta[t-1],sig.q^2).
    Illustration of Example 30 (p45) of the lecture notes using metropolis.update and metropolis with mu.p=0, sig.p=1 and firstly sig.q=1 and secondly sig.q=0.6: pdf.
    plot.mcmcPlot time series summaries of output from a Markov chain. Allows you to specify burn-in and thinning.
    fFunction for plotting bivariate Normal distribution in gibbs.update.
    All aboveAll of the above functions in one file for easy reading into R; thanks to Ruth Salway for these functions.


    The following functions are for sampling from bivariate normals, with thanks to Merrilee Hurn

    gibbs1Gibbs sampler (arguments: n the number of iterations, rho the correlation coefficient of the bivariate normal, start1 and start2 the initial values for the sampler).
    metropolis1Metropolis-Hastings (arguments: n the number of iterations, rho the correlation coefficient of the bivariate normal, start1 and start2 the initial values for the sampler, tau the standard deviation of the Normal proposal).
    metropolis2Metropolis-Hastings for sampling from a mixture of bivariate normals (arguments: n the number of iterations, rho the correlation coefficient of the bivariate normal, start1 and start2 the initial values for the sampler, tau the standard deviation of the Normal proposal, sigma2 the variance of the normal mixtures).

    Question sheets and solutions
    Question sheets will be set in the Thursday problems class. They will appear here with full worked solutions available shortly after the submission date.
    Question Sheet Zero:pdf.Solution Sheet Zero:pdf.
    Problems class notes: pdf.
    Question Sheet One:pdf.Solution Sheet One:pdf
    Problems class notes: pdf.
    Question Sheet Two:pdf.Solution Sheet Two:pdf
    Problems class notes: pdf.
    Question Sheet Three:pdf.Solution Sheet Three:pdf
    Problems class notes: pdf.
    Question Sheet Four:pdf.Solution Sheet Four:pdf (Additional questions only).
    Full solutions available on 19 Mar 18
    Problems class notes: pdf.

    Past exam papers and solutions
    The exam is two hours long and contains four questions, each worth 20 marks. Full marks will be given for correct answers to three questions. Only the best three answers will contribute towards the assessment. The exam is thus marked out of 60. In the exam you will be given the table of useful distributions: pdf.


    0 thoughts on “Bayesian Statistics Homework Solutions”

      -->

    Leave a Comment

    Your email address will not be published. Required fields are marked *