I would like to ask a question if I have got the confidence interval of the sample parameter by bootstrap and also the zero distribution (null distribution) of the sample parameter by permutation.
How should I show that my effect is robust and significant enough by these two distributions?
My data are the 7 eeg data of A1-A7 and the B stimulus data.
My true value is obtained by leaving one out cv method and trf predict A1-7 with B, obtaining 7 pred acc data, and then averaging them to get the average acc.
My null distribution approach is that leave one out cv method and trf predict A1-7 with B at the same time, disturb B, for example, A1-B correlation, then disturbs B 1 time to get a null distribution, then disturb B 1000 times, and finally get a null distribution of A1-B, then do the same for A2-B, A3-B, all 7 stories have 7 null distribution.
Finally, average 7 null distributions, get 1 null distribution, and compare with the average eeg and stimulus correlation acc to get the significance of permutation.
However, because I only have 7 raw data, I am worried that the reviewers will doubt that the results are unstable, so I want to use bootstrap to get the distribution of the sample parameters.
My new idea is intended to repeat 1000 times
each time from A1-7 eeg data with a put-back sample of 7 times
Then cross-validate by the leave-one-out method to obtain the accuracy parameters of the 7 eeg with stimulus B.
Then the 7 sampled eeg are cross-validated with the disrupted stimulus B by the leave-one-out method to obtain 7 accuracy data of zero distributions.
Finally, we can obtain the null distribution and bootstrap resampling distribution for 7000 data after 1000 repetitions.
Then I want to compare these two distributions to show whether the effect is significant or not.
I would like to ask if this idea is suitable, or if there is a more suitable idea.
