Is it possible to imagine something that is not a combination of what we have already perceived with any of our five senses? I mean, was it possible for humans to think of flying if there were no birds out there? Has there been any study to show whether human's imagination is limited to its perceptions or not?
I don't have much of a personal opinion on this question, but this is an issue that Descartes addresses. It's central to his proof for the existence of God. According to Descartes in the Meditations (specifically Med. 2 or 3), you can only imagine (i.e., for an image or phantasm [which is part of the Aristotelian model]) based on what you have already experienced.
So, for Descartes, to imagine a six-toed wolf involves taking your imagination of a dog or a wolf and its toes and adding an extra one. Or to imagine a flying cat, you mix your images of cats with images you already have of flying animals (or airplanes).
This matters for his proof of God's existence, because he does not think the idea of perfection could come from him or from anything less than perfect, ergo something outside of himself that is perfect exists.
I would guess you're looking post-Kant or so to get to the point where people will take the opposite view that we can engage in genuinely novel imaginings.
I do not think it is possible to imagine something outside of your previous body of knowledge. Even the greatest inventors and most creative writers base their ideas on things that they have already experienced and then they imagine new expansions on top of them.
Consider that we know there are some animals which can see the infra-red spectrum while humans cannot. We have infra-red cameras, but they project their data into our own visible spectrum so that we can see it with our eyes. So, there are no humans who know what infra-red actually looks like (technically there have been a few, but they cannot convey those images to us verbally). Can you imagine what it looks like? It has to be outside the realm of every color you have already seen. Can anyone imagine something outside the visible spectrum?
The idea that it is possible to fly is knowledge: useful or explanatory information. Your question assumes that knowledge can be created from sense perceptions, but this is not true. Explanations do not follow from observations in any sense: observations don't prove ideas or make them one jot more probable. This idea is just another variety of justificationism: the idea that it is possible and desirable to prove ideas true or proably true. In reality, you can't prove any position or show it is probable. Any argument requires premises and rules of inference and it doesn't prove (or make probable) those premises or rules of inference. If you're going to say they're self evident then you are acting in a dogmatic manner that will prevent you from spotting some mistakes. If you don't say they are self evident then you would have to prove those premises and rules of inference by another argument that would bring up a similar problem with respect to its premises and rules of inference.
In reality knowledge grows by processes that create variations on existing knowledge and then select among those variations. For example, your eyes and a lot of the visual information processing stuff in your brain evolved by natural selection. Mutations in the genes coding for that machinery produced variations on that machinery. The animals carrying it then either succeeded or failed in passing on their genes: selection. The variations have to be undirected in the sense that they can't be biased toward the right answer since that answer is not known before the knowledge creation process begins.
When a person creates knowledge he does this by guessing variations that might improve his ideas and then selecting among those variations according to whatever criteria he prefers. People thereby can and do guess about the properties of things they have never seen and can never see, such as the core of the sun. And in a sense all knowledge goes beyond perception because it abstracts from perception. If you see a glass of water on a table, you are thinking about what you see using terms like glass and water that refer not only to the glass and water in front of you but to any glass and any water anywhere.
See "Logic of Scientific Discovery", "Objective Knowledge" chapter 1, "Realism and the Aim of Science" by Karl Popper and "The Fabric of Reality" and "The Beginning of Infinity" by David Deutsch.
Assuming it is possible, I'm not sure how you could tell reliably when a particular imaginative thought surpasses this limitation. The closest thing to a study would be to look back in history and try to find what caused someone to be imaginative.
It may depend on what you believe is generating those imaginative ideas. If you think of the brain as a physical system with an input and output then the imagination is limited to finite combinations of experiences stored in memory imperfectly, combinations of parts of memories sliced and diced (as in dreaming during sleep), then modified by instinct, human psychology, and personal ability at using and practicing creative thought.
My non-expert opinion is that the processes in the mind are done in a language, by which I mean a set of symbols and rules that transform symbols to other symbols, symbols to actions or the reverse. Our instinct gives us a finite number of states that ultimately get converted to symbols, or brain states for pleasure and pain. Experience, education, intelligence and environment adds to this.
If this hidden language has enough generality built into it then ideas that appear to be new can be created. Maybe your example of flying can be imagined if the mind's concepts of space and motion are general enough and reachable with a small enough number of thought transformations without experience of an explicit example. I can't explain exactly how this eureka moment actually happens, maybe visualizing motion close to flying or seeing something jump high enough. It may be completely random or accidental, due to misfiring neurons or other body chemistry.
If you believe in a soul that interacts with the mind then the imagination auto-magically gets any "sensory data" that the soul has access to. The soul could also be limited by logic in a similar way that some people say God is limited and can not go against his own nature.