As you say, the difference between extrinsic (observer-based) and intrinsic (subjective) symbol-manipulation semantics is key to the Chinese Room argument (CRA) (cf. the *symbol grounding problem*).
However, I don't see that the abstract-vs-physical distinction between computation theory and physical computing devices addresses this issue directly.

Computation theory is just another mathematical model, like calculus or geometry.
Finding logical "0, 1" symbols in a physical computer is no different than finding straight lines and 90-degree angles in buildings.
If we believe that the abstract, mathematically based theories of engineering truly keep skyscrapers from falling and planes in the air, then the dancing voltages in my bank's computer also truly represent the logical "balance" of my logical "checking account". Physical computers "manipulate symbols" to the same degree that buildings "obey statics" and planes, "aerodynamics".

Fortunately, the distinction between idealized math and the real world is not needed to fully understand the CRA. Because it is grounded by Turing machine (TM) theory, the CRA's analysis can be much less esoteric and much more precise and objective.

The CRA expertly focuses everyone's attention on the *wrong symbols:* the Chinese inputs and outputs. Like a magician, Searle makes you ignore the elephant in the Chinese Room: the program.
It, too, is a symbolic input to the Searle-computer. (He also hopes you'll miss the fact that it's in English!)

The program (aka, the rule book) is the key symbolic input because *it alone* dictates how the Chinese symbols are processed.
The only reason the Searle-computer can (and must) process the Chinese symbols *purely formally* is because he ***also*** has the program-rule symbols, which he can (and must) interpret *non-formally*.
As a [*universal TM*][1], the Searle-computer's main responsibility centers on the program itself.
Searle directs everyone's attention solely to himself and the Chinese symbols, yet *he* is wholly superfluous to *their* computation!

The Searle-computer and its program input could--and should--both be completely removed from consideration.
Replace it by a direct, *non-programmable* implementation of the program and the room would function identically. Hence, Searle's claim that *the room* processes the Chinese symbols purely formally (as opposed to just its Searle-CPU) is totally unfounded because he fails to account for the program's own computation (whose existence is irrefutably explained by Turing machine theory). 

The degree to which the program's computation interprets the Chinese semantically (or not) remains undetermined. Hence, the CRA establishes *absolutely nothing* about the semantics in the room regarding the Chinese symbols.

References: [Syntax vs. semantics][2], [][3]