Multiple testing

• Administrivia:
  • Start reading chapter 12
  • One makeup without name

When is 5% not good enough

• Our current rule is +/- 2
  • +/- 2 SE allows for 5% error
  • Fine if you are only doing a few tests
  • But what if you have 20 variables?
  • You will find one of interest by chance along
  • In this case you need more protection

• Finding other probabilities besides .05
  • JMP output: estimate, SE, t, p-value
  • t = estimate / SE (makes math easier for +/- 2)
  • p-value is our primary tool for protection from multiple comparisons
  • p-values of 1/10 only occur 1/10 times
  • p-values of 1/100 only occur 1/100 times
  • p-values of 1/1000 only occur 1/1000 times

• Bonferroni
  • If you are doing 10 tests, use .05/10 = .005 for target p-value
  • If you are doing 25 tests, use .05/25 = .002 for target p-value
  • If you are doing 1000 tests, use .05/1000 = .00005!

• Why Bonferroni works
  • Suppose you are doing 100 tests and you use .0005 for your p-value
  • Then you expect 100*.0005 = .05 << 1 variables to enter by chance alone
  • So there isn't much chance you will claim a spurious result

Example: Jung's test of the Zodiac

• Whats your sign? (My sign is: Ophiuchus ("The serpent holder" Nov 30th--Dec 17th)
  • Who should you date?
  • Look for successful relationships
  • Lets do a simulation to see what difficulties lie yere
    • add rows (say 150)
    • in the first column (label it "his sign")
    • formula --> random --> uniform
    • (uniform*12) then neumeric --> ceiling
    • Repeat for "her sign"
    • Repeat for "relationship success"

Last modified: Thu Mar 30 09:57:09 2000