If you're interested in this, Searle has a book Freedom and Neurobiology about these topics, and gave a quick presentation to Google about this book in a Talks@Google from 2007. Like you’re making the case, he points out that it can be very difficult to isolate the philosophical problem of free will when the scientific problems of how we obtain a notion of our own free will are also still outstanding and seem much larger.
The basic problem in his phrasing has to do with an aspect of science, which is causal sufficiency of explanations, “given the forces and mass transfers present near this bridge, it had to collapse.” Mario Bunge’s Philosophy of Science is also some good reading here, it talks at least in volume 2 about how science has a structure of explaining things and therefore science questions are of the form “Why p?” and traditionally you have a structure of “Because of some particular circumstances q, and some laws that guarantee that q implies p.” Logical entailment is thus a deep part of normal science.
And the problem with free will is ultimately a feeling we have that our own choices were not determined by prior circumstance. “I voted for Obama but I could have voted for Romney,” say. And the philosophical question therefore looks at the language or understanding barrier between these two ideas, “I could have done something else” versus “Science proceeds via causally sufficient explanations” to get to some shared understanding where we can see what both of those mean.
Searle accepts “It’s all an illusion” as a valid explanation, albeit not one that he likes very much. So let’s dive deep into what this means: it is the statement that ”your notion that you could have done something else is a fiction that your brain generates to make it feel good about itself, but it does not correspond to any real physical phenomenon.” Sort of like when we say a rainbow is an illusion we mean that “I know it looks like there is a big physical arch out there but if you move around you will discover that there is no fixed prismacolor object corresponding to the arch you see.” Searle thinks that yes, this solves the philosophical problem directly, and so he only has a few meta-level problems with this explanation:
- It seems like the point of most “it’s an illusion” explanations is to “shake” the illusion. Unfortunately, we cannot shake this “we could have done something else” illusion of freedom because we by definition do not have the freedom to do so. You cannot freely voluntarily shake an illusion that you have the ability to freely voluntarily shake it. In the Talks@Google talk he remarks that someone had asked him, if someone proved that free will is conclusively an illusion and everything must be deterministic, would Searle accept it? and Searle had to reply that the question is assuming you had no free will, and we made you aware of it, would you use your free will to accept that conclusion? and the question rapidly eats its own premise.
- It seems like our body is doing a lot of biological work to sustain this illusion whereas simpler structures should be in-principle possible that do the same material outcomes without sustaining this illusion, especially if it is indeed an illusion and “we would do better to live without it,” which seems to be the philosopher’s position.
On the other hand you mention the randomness idea and Searle kind of rejects randomness as a valid explanation in the first place, if my memory serves me correctly. Randomness as an explanation, I would say, has to do with the existence of hidden information, and so the claim is that “I could have chosen otherwise” is really a statement of “I don’t know why I chose what I chose, but there is some reason, I just don’t have access to that reason. It was just one of those random things. Some days I vote for Obama and some days I vote for Romney.” And I think the point is, “no, I can give you my reasons I chose Obama over Romney and I know why I chose what I chose, I just could have chosen something else.” In this respect randomness is talking about a different aspect of determinism than the tension that we have to resolve, and yes it is certainly a valid aspect of determinism but it is not the aspect which we find in tension in our own cases, so you have to eventually “fix” the explanation by adding some other statement, e.g. the illusion idea. So a randomness+illusion explanation would say “You just don't have access to the deterministic causes that made you choose what you chose, so you maybe had reasons which established a 95% chance you voted Obama and a 5% chance you voted Romney within your bounds of what you could know about the system, but you are wrong that you could have chosen something else, in fact your next random variable was going to inevitably be 0.89381... and as long as your conscious reasons generated at least an 89% chance of voting Obama you were going to vote for Obama, with the missing gap of ‘could have’ being a fundamental illusion.” The point is that philosophical randomness does not solve the problem, so you can instead understand this as a scientific randomness which can be left to science and the philosophical aspect is being solved by the illusion explanation.
Searle mentions that he sees another explanation but he doesn’t like it either, which is that more recently, physicists have abandoned causal sufficiency in all of their quantum explanations. He says that the problem here is that usually anything that philosophers have to say about quantum mechanics is hot air at best, but this idea that causal sufficiency is only an approximate principle for our reality that works well for science but is not necessary, should suffice to solve the problem the other way. He just doesn’t like the idea that consciousness should be filed away as a quantum phenomenon when that doesn’t seem terribly likely, so it becomes an open scientific problem how things can inherit quantum non-sufficiency in explanation, without inheriting all of the weirdness of quantum mechanics.