A baseline for general social living is that it would be correct for anyone to participate in any plan with some degree of potential for success to save everyone in both cases, as long as it doesn’t put the participants in danger themselves. Non-participation is not a neutral position with respect to such planning - if someone came to you for help and asked you to take part, it’s not an excusing reason to say “it’s nothing to do with me!”
Additionally, involvement isn’t sufficient for moral culpability. The large person can still absolved of any judgement for not jumping in front of the trolley while being involved - even if you’d alerted them to the situation and asked them kindly to roll off the edge to block the trolley, they’d be considered innocent for refusing because intervention requires guaranteed death, and it’s not right to demand martyrdom. Moreover, they’ve not been reasonably informed of the situation, so even if they could have helped, we can’t expect them to have done so as a general matter of course.
If however a bystander could help in some other way that didn’t require personal sacrifice, and chose not to after having been clearly asked, they would not be deemed innocent by virtue of mere non-participation. If the plan presented was a good one, and their support would be reasonably expected to have helped, then declining is ethically negligent, and there is a reasonable case for holding them partially responsible for a negative outcome.