I don't think that Wittgenstein actually "rigged" his argument from the start by limiting the meaningfulness of propositions to only matters of phenomenal fact. Because first, the Tractatus actually _never_ mentions phenomenal facts, neither at the start, nor at the end.  Second, the Tractatus, more generally, does not discuss _epistemology_ at all. Third, in many comments Wittgenstein does oppose _natural science_ to philosophy. But such remarks start relatively late in the book, in part 4. So it does not seem to be rigged from the start (more about this immediately).

>4.11 The totality of true propositions is the whole of natural science (or the whole corpus of the natural sciences).

>4.111 Philosophy is not one of the natural sciences. (The word ‘philosophy’ must mean something whose place is above or below the natural sciences, not beside them.)

Wittgenstein begins the Tractatus with a treatment of facts. Not phenomenal facts, not natural facts, just facts in the abstract. Quite a bit later, and quite suddenly, he begins to identify facts _simpliciter_ with the facts of natural science. How is this identification to be understood? I don't recall that Wittgenstein gives any positive characterization for natural science. He goes the opposite way: all the sentences that are not semantically problematic, by his lights, he groups indiscriminately under the term "natural science".

"Philosophical" sentences can be problematic in two ways, as suggested in the above remark:

>4.111... The word ‘philosophy’ must mean something whose place is above or below the natural sciences, not beside them.

I interpret it like this: philosophical talk is "above natural science" when it attempts to refer to things that are "too big" i.e. beyond the world, like God or the overall meaning of the world.

>6.4312... The solution of the riddle of life in space and time lies outside space and time. (It is certainly not the solution of any problems of natural science that is required.) 

>6.432 How things are in the world is a matter of complete indifference for what is higher. God does not reveal himself in the world. 

>6.4321 The facts all contribute only to setting the problem, not to its solution.

Philosophical talk is "below natural science" when it attempts to refer to the structures of subjectivity, structures that underly the world (the world that language refers to), including the structures that make language possible.

>5.631 There is no such thing as the subject that thinks or entertains ideas. If I wrote a book called The World as l found it ... this being a method of isolating the subject, or rather of showing that in an important sense there is no subject; for it alone could not be mentioned in that book.

>5.632 The subject does not belong to the world: rather, it is a limit of the world.

All the other sentences are, to Wittgenstein's mind,  semantically unproblematic, and he just wraps them all under the term "natural science". This, without endeavoring to tie them precisely to what is usually referred to by this term.