People are asked to guess the outcome of a coin flip. If they guess correctly, they get a reward. However, people only reveal their initial guess after they've seen the outcome of the coin flip. This allows them the possibility to cheat, by lying about their guess.
Hence, we can measure dishonesty by seeing whether the proportion of correct guesses is significantly greater than 50 %.
Does this study really measure dishonesty? Does it not rather measure carelessness? Here's what I mean. Unless you happen to be a complete idiot, if you are invited to such a study, you know what the point of it is. Everyone with a brain can figure out it's an experiment to test your honesty.
Hence, this knowledge will possibly impact their actions during the experiment. For example, if you are a person who cares a lot about your reputation and perception amongst other people, you might think to yourself
oh no, if I lie about my guess, the experiment might conclude that I'm a dishonest person. I don't want that, so let me just play it honestly.
Now, such a person might very well be extremely dishonest ordinarily, but since they are now in a situation where they know their dishonesty is being measured, they'll try to act honestly.
So in this case, the study would conclude such a person is honest, when in fact they are the quite opposite.
Likewise, an otherwise honest person might look at this experiment and think to themselves
I'm a honest person in life ... but this is just a silly, little game, isn't it? Let me just cheat, it'll be fun and I'll get a reward.
This person is honest, but is just having a little fun with the experiment. Yet the experiment would conclude that this person is actually the unethical one.
Am I right? If so, what does that say about all studies that attempt to measure people's ethics. Don't they all suffer from this phenomenon where the bad people know they are being "watched" and try to act good, while the good people don't really care whether or not they are being watched, since they aren't that insecure about themselves and don't feel the need to convince others they are good?
And if so, what other type of experiment would actually be capable of measuring somebody's ethics without committing these fallacies?