As you say, the difference between extrinsic (observer-based) and intrinsic (subjective) symbol-manipulation semantics is key to the Chinese Room argument (CRA) (cf. the symbol grounding problem). However, I don't see that the abstract-vs-physical distinction between computation theory and physical computing devices addresses this issue directly.
Computation theory is just another mathematical model, like calculus or geometry. Finding logical "0, 1" symbols in a physical computer is no different than finding straight lines and 90-degree angles in buildings. If we believe that the abstract, mathematically based theories of engineering truly keep skyscrapers from falling and planes in the air, then the dancing voltages in my bank's computer also truly represent the logical "balance" of my logical "checking account". Physical computers "manipulate symbols" to the same degree that buildings "obey statics" and planes, "aerodynamics".
Fortunately, the distinction between idealized math and the real world is not needed to fully understand the CRA. Because it is grounded by Turing machine theory, the CRA's analysis can be much less esoteric and much more precise and objective.
The CRA expertly focuses everyone's attention on the wrong symbols: the Chinese inputs and outputs. Like a magician, Searle, makes you ignore the elephant in the Chinese Room: the program. It, too, is a symbolic input to the Searle-computer. (He also hopes you'll miss the fact that it's in English!)
The program (aka, the rule book) is the key symbolic input because it alone dictates how the Chinese symbols are processed. The only reason the Searle-computer can (and must) process the Chinese symbols purely formally is because he also has the program-rule symbols, which he can (and must) interpret non-formally. Searle directs everyone's attention solely to himself and the Chinese symbols, yet he is wholly superfluous to the computation in question!
The Searle-computer could--and should--be completely removed from consideration. Replace it by a direct, non-programmable implementation of the program, and the room would function identically. Hence, Searle's claim that the room processes the Chinese symbols purely formally (as opposed to just its Searle-CPU) is totally unfounded because he fails to account for the program's own computation (whose existence is irrefutably explained by Turing machine theory).
The degree to which the program's computation interprets the Chinese semantically (or not) remains undetermined. Hence, the CRA establishes absolutely nothing about the semantics in the room regarding the Chinese symbols.
References: [Syntax vs. Semantics], [chineseroom.info] : https://philosophy.stackexchange.com/questions/34358/how-can-one-refute-john-searles-syntax-is-not-semantics-argument-against-stro : http://chineseroom.info/