I hope you can point me in the right direction regarding two issues I am having with simulating experiments (1: regarding the legality of my assumption of different from chance, 2: regarding whether my simulation is valid at all).
We conducted a study (here's a poster: http://www.opensourcesci.com/pdf/CrossCultural_colourOdourAssociations.pdf
) where participants smelled a range of odours and assigned each 3 colours that seemed most compatible with that odour. We did this in a variety of cultures. Our plan was to first see if the selection of colours to odours differed from that expected by chance, and then see if colours-odour associations differed over cultures.
My initial thought was to use multinomial theory and determine the likelihoods for a colour being picked 1,2,3,4,5,6,7,8,9 etc times for a given odour, and then to assign colour-odour associations in the actual experiment that differ from that expected by chance when p<0.05. However, the calculations for multinomial theory proved beyond me (because of this issue participants could assign each odour THREE colours but could not assign the same colour more than once).
I then moved on to simulating the experiment. Here, colours chosen for each colour were done so at random. The experiment was simulated 100,000 times and the possibility of a colour being picked, at random, 1,2,3,4,5,6,7,8,9 etc times was outputted (if there was observed 600 indidences of a colour being randomly chosen 4 times, then 600/100000= p= 0.06). As before, I assumed that if the number of times a given colour was picked for the same odour in our experiment was greater than that expected by chance (p<0.05) we had a significant result (H1: 'within a given culture there are culturally specific colour-odour associations').
To test if colour-odour associations differed over cultures, I conducted a further 'higher-order' simulation: I ran two simulations as described before, one for each culture, and looked to see the possibility of those two cultures colour-odour associations picked differing by (root square) 1,2,3,4,5,6,7,8,9 etc times. Issue 1
: Is it legal to assume here that, for a given colour C picked X times, we can say this occured at a below-chance level if the possibility of getting X colours picks in the simulation experiments was below p<.05? Issue 2
: I have yet to find what I have done in textbooks (e.g. 'monte carlo simulation' Mooney; 'Simulation for the Social Scientist' Gilbert & Troitzsch). Textbooks advocate mathematically describing the distribution of each of the factors of your experiment, building these into a model of your experiment, and then simulating this. However, I've simulated the actual experiment itself. Is this legal??! Are there any books or articles you can advise me read on this technique, or is what I've done just wrong?
I very much appreciate your efforts in first wading through my lengthy description and then pondering my questions.
With kind regards,