Last modified: Thu Oct 6 14:36:52 EDT 2005 by Dean Foster

Statistical Data mining: Alternatives to proper scoring rules

Admistrivia

• OPIM is running a data mining seminar (Tuesday at noon, in G50.) You should all go. This tuesday my friend and long term coauthor and generally great speaker is talking. (He will be talking about calibration.)

2 x 2 table of outcomes

• action taken x outcome
• 4 numbers: TP, FP, TN, FN (P=test result is positive)
• Marginals will be called: P,N, S=sick, H=health
• How much time can we spend analysing them?
• Note: action might be something like: I(p > .5). The p has been lost.

Alternatives to proper scoring rules

• ROC curves
  • Type I, type II error graph: easy to understand: FN/S, FP/H
    • alpha = FP/H
    • beta = FN/S
    • power = (sensitivity) = TP/S
  • ROC is transposed version of this: specificity=TN/H, sensitivity=TP/S (oppisite of error rates)
  • sensitivity = fraction true positives
  • specificity = fraction true negatives
  • Both increase as threashold decreases
  • All concept of claimed probability is lost
• Precision / recall
  • recall = TP/S
  • precision = TP/P
  • F measure = 2 * precision * recall / (precision + recall)
• Area under the curve (ROC typically)
• lift charts
  • Y = TP/P vs P

Look at error graph

• Intepret slope
• Note: parametric curve
• If slope = parameter, called prediction is called calibrated

Calibration

• Plot frequency vs claimed probability
• See handout from talk
• Assuming calibration: AUC, F-measure and others start making sense
• Without calibration: you are getting things truely wrong