There is an important distinction between reality-is-a-simulation and brain-in-a-vat, namely that in the former one's brain is simulated, whereas in the latter one's brain exists outside of the simulation and only one's experience is simulated.
Because of this "stitching together" of two realities, brain-in-a-vat offers far more opportunities to discover the simulation.
But first, we must ask some questions:
- How different is the "real" reality from the simulated one? (Perhaps entities in the "real" reality are not human. Maybe they are advanced beings seeking to understand simpler life forms or vastly different laws of physics.)
- Why is my brain in the vat? (Entertainment? Research? Healthcare? Space travel? Imprisonment?)
- What percentage of other people in this simulation are also brains-in-vats? (If I am the only one, the simulation can do a lot to both hide itself from me, and to only simulate very small parts of the world at a time.)
The easiest way to determine that you ARE a brain in a vat is to be "brought back" into the real world. Short of dying in the simulation, your best bet at doing this is to find a way to force the simulation to do a lot of work, using a lot of processor power, which might annoy your captors sufficiently to get you out. (For example if you build a huge room lined with TVs displaying detailed real-time satellite images of populated parts of the Earth it might force the simulation to make sure that all those areas are simulated in high detail so that you don't notice any discrepancies. Maybe you will be watching pre-recorded footage but if you try enough such ideas you may find something the simulation can't cope with.
There may be some sort of failsafe "get me out of here" thing you can do, but that seems a bit pointless if we aren't given the memory of what it is. On the other hand one might argue that those who have "transcended" or "ascended" through meditation have found the secret to breaking out of this existence. Perhaps when we stop paying attention to our senses we can start to sense the vat we are in---or perhaps it is necessary to disconnect ourselves from our senses in order to be safely disconnected from the simulation!
If you are a brain in a vat, then your simulated brain will appear to contradict the laws of physics at times. If you get a knock on the head in the simulated world, this cannot be reproduced exactly in the real world, and so either your brain and the simulated brain respond differently, or the simulated brain mirrors the real brain and therefore seems to contradict the laws of physics. This might be very hard to detect, but it will be there. Of course, can you really trust the MRI machine or the neurosurgeon not to be manipulated by the simulation so that everything seems fine? Perhaps your best bet is to build your own brain imaging machine. Even then, the simulation could twist reality to lie to you, but the bigger the lies, the greater the chance an inconsistency will arise.
Onto your next question: if I'm a brain in a vat, can I in theory experience the real world?
Sure! If technology exists to feed your senses with an artificial world, it should be even easier to feed your senses with input from a camera, microphone, touch and motion sensors, etc.
Of course, you could be told that you are visiting the real world but are in fact in another simulation. But that's beside the point.
If the real world is utterly different from the simulated one (e.g. it's 6 dimensional, there's no light or sound, and objects can overlap) there might be some serious challenges in interfacing it with a brain that is designed for our world. But consider that we can use infra-red goggles to sense a somewhat unfamiliar world. If we are slowly introduced to a weird universe perhaps we can adapt.
If the universe is just a computer simulation, we cannot tell who is running it or why. Any discrepancies or artifacts of simulation might give us clues, but at the end of the day, we can't be completely sure of anything. The simulation designers have full control of what it's like inside the simulation, so they can hide ALL of their tracks and put in red herrings if they like.