I see your point, but view it more like the Kick Catch interference rule. If someone knocks you into him, it ain't a penalty.