I'm beginning to think about one sort of objection that all conceptual analysts use against experimental philosophy. The objection I have in mind is the one that says conceptual analysis uses intuitions anyone would have if they had the right kind of view in hand. Truth be told, I have never been convinced by this kind of intellectual arrogance. Why assume that conceptual analysis gives us some kind of special access to the right kind of intuitions? And why believe that the right kind of view cannot be had by someone who hasn't been trained to use conceptual analysis?
Suppose that a philosopher said, "any person will say that the brain-in-a-vat scenario is feasible." If a person disagrees with the statement, then the disagreement implies that the philosopher's intuition and the person's intuition are not identical. I assume that the lack of identity is what experimental philosophers exploit to challenge a philosophical position. But what kind of identity is at issue?
I take it that conceptual analysts and experimental philosophers could be talking about two kinds of identity. If they are indeed talking about two different kinds of identity, then the debate goes no where. The conceptual analyst talks about one kind, and the experimentalist another. Therefore, the debate ends in a stalemate.
Let me address the two types of identity I think are present. I will call one a strong version and the other a weak version. The strong version is similar to the logician's view of identity. For this view of identity, P and Q are identical if and only if every property of P is a property of Q, and vice versa (approx. Leibniz's Law, though I think this is debatable). If this is the kind of identity the conceptual analysts are talking about, then they have mistaken their own intuitions for the intuitions of other people.
The argument goes (roughly) as follows: Peoples' intuitions are x. The philosopher's intuitions are y. Every property of x is not a property of y. After all, in my example, people do not say y about the brain-in-the-vat scenarios. Therefore, the philosopher's intuitions and the peoples' intuitions are not identical according to the strong version of identity.
The weak version of identity does not uphold the logician's conception of identity. On the weak version, the peoples' intuitions are the same as the philosopher's intuitions. They are identical because people must realize that their own intuitions are mistaken. People do not have a good command of reasoning philosophically. If they had command of reasoning philosophically, then they would not have reacted to the scenario in the way they did. Once they realize that they have made a mistake, they will agree with the philosopher's intuition. So, peoples' intuitions and the philosopher's intuition are identical.
The weak version of identity asserts that the people are mistaken, while the strong version says that philosophers are mistaken about the peoples' intuition. Conceptual analysts seem to endorse the weak version of identity. They assert that people are generally mistaken. But once people see the error of their ways, they will agree with the philosopher's intuitions. So, the intuitions are identical.
A question remains: If we want a folk account of some philosophical issue, why should we endorse the weak version of identity?
There is one argument I could imagine that supports the weak version of identity. But I think that it is a bit uncharitable toward philosophers generally. So, I don't necessarily believe this is a good argument.
Philosophers are good at reasoning and critical thinking. They can identify when other people make mistakes in their reasoning. Reasoning - broadly speaking - includes intuitions. Therefore, philosophers can identify when people are mistaken about their intuitions. Maybe a George Bealer style argument where only one trained in the art and science of doing metaphysics may properly identify possible metaphysical and logical intuitions. Since people aren't so trained, we can discount whatever they have to say about philosophical positions or thought experiments.
Of course, there is a brief counterargument to that argument. Here goes: Philosophers are people too. People are sometimes bad judges of intuition. So, even philosophers are sometimes bad judges of intuition. To dismiss kind of argument is equally arrogant, but that's what the likes of Bealer does.