Asking one question and answering another: when decisions and statistical analysis are not aligned.

Yesterday I presented at StanCon. This is the virtual poster I created for the conference, and here is a video talking about it:

My main message is the following:

1. Before doing any analysis, get an understanding of the decision that that you are trying to help people make.
2. Do not present result with incredible certitude. That is, you should communicate the uncertainty in our findings.
3. Do not follow the null hypothesis ritual blindly. There is nothing magical about whether or not you can reject the null hypothesis that an effect is zero. In many cases, zero is not the relevant cutoff for making decisions. Moreover, there is nothing magical about a p-value of .05. Many decision makers would rather you tell them that there is a 70% chance that the intervention is moving the needle than you telling the that you cannot reject the null which is not very helpful to inform decisions.