Rethinking | 暗微幽光

Autopsy of NHST and Bayesian Models (part 1)

Mon, 01 Jan 0001 00:00:00 +0000

Chapter 2 of Statistical Rethinking: A Bayesian Course with Examples in R and Stan introduces a water tossing example for the demonstration how Bayesain model run through the data based on the researcher’s hypothesis. This chapter defines a three stage process that are used in the coming chapters. We start from a narrated Data Story , then update the models by filling data in, and finally evaluate all the upated models. This literated process shows a clear picture for the learners who have yet stuied statistics before read this book. The readers who have studied statistics, like me, will have a hole in the mind what are the differences between Bayesian methods and the null hypothesis significance testing (NHST) in this process. After read Michael Clark’s Bayesian Basics: A Conceptual Introduction with Application in R and Stan on Stan official site, I require the key to understand the difference between the two golems.

The critical difference is which type of conditional probability the statistical method is used to evaluate the model. Once we collected the data based on some hypothesis, we have the distribution of hypothesis $ p( ) $ and the distribution of data $ p(y) $. NHST compuates the probability we have the data when the hypothesis is true $ p(y|) $. In practical, p value refers to the conditional probability of the null hypothesis, and * confidence interval* suggests a range of plausible outcomes based on the confiditional probability of the real hypothesis. A Bayesian method compuates the probability the hypothesis is approved based on the data we have $ p(|y) $. Because the computation of $ p(|y) $ needs the likelihood $ p(y|) $ and the priori $ p() $, of course a Baysian method cost more steps than NHST.

$ $ has a term parameter in many statstical textbooks. It is a space of numbers that provides the axis where the above probability distribution sit on. Likelihood is the sampling distribution we have to update before we run NHST. As like I show in Learning Sampling Distribution in R Programming, a sampling distribution will approach one $ $ with the increasing sample size. Sample size is the key for NHST because it could change the evaluation criteron on the statistical model.

Here is a pseudo experiment I want to know if a test is success based on the expect value smaller than 5 (Obersvations are 1, 2, 3, 4, 5, 6, 7, 8, 9, 10). I completed an experiment of 25 observations and an experiment 36 observations. In use of NHST with a critical value (p < .05), the criterion change with the sample size. The simulation

set.seed(1)
OBV <- 1:10
Dist25 <- NULL
Dist36 <- NULL
count = 100
while(count > 0){Dist25 <- c(Dist25,mean(sample(OBV, 25,replace = TRUE) ) ); count <- count - 1}
Dist25_Density <- data.frame(Theta = density(Dist25, kernel = "gaussian")$x, Density = density(Dist25, kernel = "gaussian")$y)
CL25 <- max(Dist25_Density$Theta[Dist25_Density$Theta < 5 & Dist25_Density$Density < .05])
print(CL25)  ## The smallest parameter that could make judgement


## [1] 4.152939


count = 100
while(count > 0){Dist36 <- c(Dist36,mean(sample(OBV, 36,replace = TRUE) ) ); count <- count - 1}
Dist36_Density <- data.frame(Theta = density(Dist36, kernel = "gaussian")$x, Density = density(Dist36, kernel = "gaussian")$y)
CL36 <- max(Dist36_Density$Theta[Dist36_Density$Theta < 5 & Dist36_Density$Density < .05])
print(CL36)  ## The smallest parameter that could make judgement


## [1] 4.41705

When this experiment outputs a avarage value between 4.15 and 4.42, would you like to collect the data more than 36 participants? This is the opportunity many researchers have to struggle in their study. Researchers who are educated as like me used to collect the observation till the p value is smaller than .05. Most researchers used to stop the experiment till the 36th participant finished the experiment. However, this treatment lacks of the serious scientific thinking. If the hypothesis has set up the conditions to have the average, the sample size should be appointed before the start of experiment.

The appointment of sample size could be loose when the possible average is uncertain. When the experiment is conducted at first time in the world, NHST without the appointment of sample size could draw the range of averages that might obey the hypothesis. On the other hand, the following studies better has the appointment of sample size. In this case, NHST has to accompany the other statistical evaluation, such as power, effect size, to find the appropriate sample size. Therefore, NHST will perform well when the researcher conducted the first study or have the preparation as well as the first study. This will be the key in my autopsy of Bayesian method.

Check My Tools to Create and Manipulate Golems

Mon, 01 Jan 0001 00:00:00 +0000

Since two weeks before I published this post, I have started read the book Statistical Rethinking: A Bayesian Course with Examples in R and Stan written by Richard McElreath. Richard is the evolutionary anthropologist at Max Planck Institute. He wrote this textbook for the PhD students who will use the Bayesian statistics in their research projects. Compared to the textbooks written by statisians and data scientistst, Richard’s book explain and demonstrate the statistical methods with examples instead of equations. His intention is to help who are not staticians but used to use statistics realize one fact: we rely on the statistical models as the representations of our answers rather than answer the questions by the raw data or naked truth. Many non-statisticians are used to find and learn what kind of methods or apps to deal with their data, but few are interested to know the models under the mentods and apps they are using. The trouble and danger is that they thought their jobs are done when the program printed the tables and figures but these outputs are from the statistical model is unable to answer their question. This situation is originated from many non-statisticans consider themselves the end-users of statistical models. Like any user of a packaged software, non-statisticans have no time to understand how the tools in their hands designed and conducted by statisticans.
Richard introduced the story of golem to raise the non-statsiticans’ attention to the troubles they had made and will make. A statistical model, like a golem, has the power beyond human to finish the work the human are unable to do, for example, trace the passengers’ track from the trillion of camera. Its power could be misused or out of control if we do not understand what is the root of its action. A user of ststistical method, no matter you are or are not stistician, have to keep the awareness of engineer when you are dealing with your data. Today everyone has many easier ways than a decade ago to keep the awareness of engineer. One advantage is that the learning curve for being an part-time hacker is getting smooth. Increasing R apps are opening many windows for who are want to outlook and modify the statistical models.
Since this post, every post listed in the category Rethinking is one of the summaries and feedbacks to Statistical Rethinking: A Bayesian Course with Examples in R and Stan. At first I have to check my toolkits for create and manipulate the statistical models. They are R core and the packages. Years ago I have used to use the packages in my data processing. Now I show them for who start to use R after read this post.

install.packages(c("rpart","chron","Hmisc","Design","Matrix","stringr","lme4","coda","e1071","zipfR","ape","languageR","multcomp","contrast","shiny","ggplot2", "dplyr"))

Some of the packages are learned from I participated in Cousera Data Science. Now I used to use dplyr to process the raw data, and I am learning how to draw the figures I need in use of ggplot2. When this post is published, I have updated my R to R version 3.4.3 (2017-11-30). Through the codes of Heuristic Andrew, here are my installed packages.

ip <- as.data.frame(installed.packages()[,c(1,3:4)])
rownames(ip) <- NULL
ip <- ip[is.na(ip$Priority),1:2,drop=FALSE]
print(ip, row.names=FALSE)


##       Package   Version
##       acepack     1.4.1
##           ape       5.0
##    assertthat     0.2.0
##     backports     1.1.2
##     base64enc     0.1-3
##            BH  1.65.0-1
##         bindr       0.1
##      bindrcpp       0.2
##         binom     1.1-1
##        bitops     1.0-6
##      blogdown       0.4
##      bookdown       0.5
##           car     2.1-6
##       caTools    1.17.1
##     checkmate     1.8.5
##         chron    2.3-51
##          coda    0.19-1
##    colorspace     1.3-2
##      contrast      0.21
##    data.table  1.10.4-3
##     dichromat     2.0-0
##        digest    0.6.13
##         dplyr     0.7.4
##         e1071     1.6-8
##      evaluate    0.10.1
##       Formula     1.2-2
##       geepack     1.2-1
##       ggplot2     2.2.1
##          glue     1.2.0
##     gridExtra       2.3
##        gtable     0.2.0
##         highr       0.6
##         Hmisc     4.0-3
##     htmlTable    1.11.1
##     htmltools     0.3.6
##   htmlwidgets       0.9
##        httpuv     1.3.5
##     iterators     1.0.9
##      jsonlite       1.5
##         knitr      1.17
##      labeling       0.3
##     languageR     1.4.1
##         later       0.6
##  latticeExtra    0.6-28
##      lazyeval     0.2.1
##          lme4    1.1-15
##      magrittr       1.5
##      markdown       0.8
##  MatrixModels     0.4-1
##          mime       0.5
##        miniUI     0.1.1
##         minqa     1.2.4
##      multcomp     1.4-8
##       munsell     0.4.3
##       mvtnorm     1.0-6
##        nloptr     1.0.4
##      pbkrtest     0.4-7
##     pBrackets       1.0
##     pkgconfig     2.0.1
##         plogr     0.1-1
##       plotrix       3.7
##          plyr     1.8.4
##     polspline    1.1.12
##      quantreg      5.34
##            R6     2.2.2
##  RColorBrewer     1.1-2
##          Rcpp   0.12.14
##     RcppEigen 0.3.3.3.1
##      reshape2     1.4.3
##         rlang     0.1.4
##        RLRsim     3.1-3
##     rmarkdown       1.8
##           rms     5.1-1
##     rprojroot     1.3-1
##    rstudioapi       0.7
##      sandwich     2.4-0
##        scales     0.5.0
##         servr       0.8
##         shiny     1.0.5
##          simr     1.0.3
##   sourcetools     0.1.6
##       SparseM      1.77
##       stringi     1.1.6
##       stringr     1.2.0
##       TH.data     1.0-8
##        tibble     1.3.4
##       viridis     0.4.0
##   viridisLite     0.2.0
##        xtable     1.8-2
##          yaml    2.1.16
##         zipfR    0.6-10
##           zoo     1.8-0


print(paste0("There are ",dim(ip)[1], " packages installed in my laptop."))


## [1] "There are 92 packages installed in my laptop."

Richard’s book inspired me help people control their golems/ststistical models in the process of coding. In his book, literature and codes are separated. Readers who are not familiar with coding skill might hardly follow his literature. Literatural coding might be the best way to impliment the Rethinking. I am going to accumulating the codes of Bayesian statistics and thake notes of his and others literatures in the coming posts.

Rethink Significance

Mon, 01 Jan 0001 00:00:00 +0000

To trace the fallacy of use hypothesis testing, I am programming the examples in Dr. Adrianus D. de Groot’s paper The meaning of “signiﬁcance” for different types of research. This paper is published in 1956 in the Dutch journal Nederlands Tijdschrift voor de Psychologie en Haar Grensgebieden. Adrianus De Groot is Dutch psychologist and chess master. During 1950s and 1960s, he suggested the concept of emperical cycle for the researchers who use the statistical tools in the social and behavioral researches.

Emperical cycle

de Groot’s emperical cycle distinguish two types of emperical research: exploratory research and confirmatory research. Exploratory research aims to formulate the hypotheses that covers the process from observation to induction. Confirmatory research attempts to test the predicited concequences based the hypothesis. Thus a Confirmatory research covers deduction, testing, and evaluation.

In the time de Groot started his academic career, the protocol of hypothesis testing has been completed by the leading statisticans, Ronald Fisher, Jerzy Neyman, and Egon Pearson. Psychologists at that time learned and used p value to evaluate the results. de Groot in his paper has suggested that the hypothesis testing is the appropriate tool to evaluate the result of a confirmatory reserach. He in the same paper also discussed the problems that perhapes happen when the hypothesis testing was used to decide the available hypotheses in an exploratory research. There are two cases of explorartoy research discussed in his paper. One has finite hypotheses, and the other has infinite hypotheses. Both cases show up in front of every researcher in anytime and in anywhere. Many researcher struggle how to pick the available hypotheses up according to the data in hands.

Exploratory research has finite hypotheses

The section title in de Groot’s paper is Hypothesis testing research for multiple hypotheses. He presumed the case as follow:

We give N as the number of hypotheses yet to be tested. Every hypothesis is going to be evaluated by the hypothesis testing. We also give n as the number of hypothese successfully passed the test. Every hypothesis have the probability .05 pass the test. This probability refers to the type 1 error in the present hypothesis testing.

Today we have 10 hypotheses (N = 10) to be evaluated by the data. Consider all the consequences, we can calculate all the probabilities given the counts of positive hypotheses.

require(xtable)
N <- 10
n <- 0:10
alpha <- .05
REJECT_P <- 0
PASS_P <- rep(0,length(n))
for(k in n){
        REJECT_P = REJECT_P + choose(N,k)*alpha^k*(1 - alpha)^(N - k) 
        PASS_P[k+1] = 1 - REJECT_P
}
SUCCESS = data.frame(n,PASS_P)
colnames(SUCCESS) <- c("n", "Probability(Positive Results)")
print(xtable(SUCCESS), include.rownames = FALSE, type = "html" )

n	Probability(Positive Results)
0	0.40
1	0.09
2	0.01
3	0.00
4	0.00
5	0.00
6	0.00
7	0.00
8	0.00
9	0.00
10	0.00

plot(PASS_P ~ n, xlab = "Smallest Number of Positive Hypotheses", ylab = "Probability(Positive Results)")

That table is telling us if we wish acquire at least one positive hypothesis, the probability is 0.4012631. That plot shows the probability dramatically decrease with the increasing numbers in our wish. In other words, when we have no precise knowlegde about the use of hyphtesis testing, the more hypotheses we want to induce, the higher risk we get nothing.

Exploratory research has infinite hypotheses

This case is everywhere in this era of big data. de Groot called this case Material-exploration: N becomes unspeciﬁed. In his paper, material refers to data because data is a low frequency word in his era. The researchers in his era has realized the best way to deal with this case is to let the data tell the story. The research on this case obviously is an exploratory research. For the researchers in de Groot’s era and in the era of big data, there are two routes to finish this kind of research project. One route is to label and categorize the hypotheses. This route is the hot topic of machine learning in the present data science. The other route is to decide the possible hypotheses. Today the researchers on this route mostly rely on the multiple variate statistical tools.

de Groot suggests the caution to the research that attempts to decide the potential hypotheses. N is infinite because not all hypotheses could be induced in this case. Whe we have 20 testable hypotheses from 200 potential hypotheses. If a researcher confirmed that 10 of the 20 hypotheses have the positive support by the data, based on the type 1 error, he/she has to understand that 5% hypotheses are positive (10/200) but 50% hypotheses are positive (10/20). This caution is like the misuse of golem that is discussed in Statistical Rethingking. In the timing I am writing this note, this caution implied the researchers who are thinking the potential hypotheses are decided assumed that they have finished a emperical circle.

Role of Preregisteration

Decades of misusing the hypothesis testing have resulted in a bad situation we have to faced. There is no clear cutoff between the exploratory research and the confirmatory research. Many published papers in nature are the exploratory researches, but they are packaged in the form of confirmatory research by the authors (editors and reviewers have the responsibility too). This is why de Groot’s originated paper was translated and published 60 years later. This post is one of the fundemental I will cite when I introduce and discuss the preregistration.

Why we need a readable data table in behavioral scientific research?

Mon, 01 Jan 0001 00:00:00 +0000

A bad case

A few days ago I was dealing with a raw data set as like the following table.

ID	Group	I01	I02	I03	I04	I05	I06
s01	A	3	4	5	1	2	3
s02	A	2	5	3	2	1	1
s03	A	4	5	4	3	1	1
s04	B	2	1	5	4	3	3
s05	B	2	1	2	5	2	1
s06	B	2	3	3	4	2	1

This table is a tiny version I copied from a master graduate’s spss raw data table. The format of the table is frequently seen in the data files for SPSS software. This format originated the “row cognition” for data in every mind who are not familar with the statistical tools. Without the awareness of structure, people used to stack a lot of bricks horizontally. As like the table shows, there are the data of 6 participants. A new statistical tool user tend to fill all the information and data for a participant in a row as possible. This format fits human cognition for the naive thing who never manipulate before. However, the computers recognize the table by column, in other words, a “column cognition”. I show you how computer deal with the summation of one participant’s scores by the following codes.

s01_data <- c(3,4,5,1,2,3)
units <- rep(1,6)
sum(s01_data)

[1] 18

t(units)%*%s01_data


 [,1]

[1,] 18

The function sum is one of the first functions a new statistical tool user have learned at the first class. sum is used to add all the scores of a participant. A new user tend to append the summation to the end of a row. But the last code show you a programing language recognizes the scores of a participant by column. The evidence is that the sturcture of t(units) is row but the structure of s01_data is a column. How do we eliminate the difference between human and machine cognition? Why human tend to deal with the data by row rather by column?

Codes to make the table readable

To make the table readable for the computer, I have to modify the format of this table as like this:

Table <- read.csv("data.csv")
New_Table = data.frame(ID = rep(Table[,1],6), Group = rep(Table[,2],6), Item = rep(names(Table[,3:8]), each = 6), Resp = unlist(c(Table[,3:8])), row.names = NULL)
head(New_Table)


##    ID Group Item Resp
## 1 s01     A  I01    3
## 2 s02     A  I01    2
## 3 s03     A  I01    4
## 4 s04     B  I01    2
## 5 s05     B  I01    2
## 6 s06     B  I01    2

The new table arrange all the raw scores in one column. I create a new column Item in this table. Item is the index like ID and Group. We are able to summarize the total score of each participant by combine the index ‘ID’ and ‘Item’. We also average the group means by the index Group.

Configurate human mind approaching a readable data table

Why we have to make effort for waving a table readable for the computer? It is the human nature we prefer every column filled with meanings in a table. Every item has the reason a researcher put it into the study. Organizing all scores of items in a row occupy the meanings in the human cognition, but the index of items is meaningless for every human mind. A researcher need a table filled with the analyzable data for the statitical tool. In the case of this post, the analyzable data is the summation of all 6 items. A table like this case embodies the desired table for the researcher but needs a column of summarized scores for the computer. This gap has to be fixed before a precise statistical process is conducted. The solution I prefer is design the table format at first. This is the main reason I and my students have to prepare a code book before the start of the experiment.

The secondary solution depends on the packaged functions. There are many raw data table organized like the bad case at first, because the software to collect the raw data and organize them in this way, for example, google From apps. To deal with these data, I have to transform the raw data before I import them to R. A reproducible way is to transform them in use of the R codes. The functions have to isolate which columns are index and which columns are raw responses. To my knowledge, a code book is needed to define the columns to be arranged.

ID	Group	I01	I02	I03	I04	I05	I06
s01	A	3	4	5	1	2	3
s02	A	2	5	3	2	1	1
s03	A	4	5	4	3	1	1
s04	B	2	1	5	4	3	3
s05	B	2	1	2	5	2	1
s06	B	2	3	3	4	2	1

ID	Group	I01	I02	I03	I04	I05	I06
s01	A	3	4	5	1	2	3
s02	A	2	5	3	2	1	1
s03	A	4	5	4	3	1	1
s04	B	2	1	5	4	3	3
s05	B	2	1	2	5	2	1
s06	B	2	3	3	4	2	1

ID	Group	I01	I02	I03	I04	I05	I06
s01	A	3	4	5	1	2	3
s02	A	2	5	3	2	1	1
s03	A	4	5	4	3	1	1
s04	B	2	1	5	4	3	3
s05	B	2	1	2	5	2	1
s06	B	2	3	3	4	2	1