>>1 It isn't always wrong to not do anything to stop something bad from happening, but your case probably isn't one of those scenarios. As far as morality goes, ought implies can. Essentially, it can't be said that you morally ought to do something if you physically cannot do it. Like I wouldn't blame for not stopping an asteroid from hitting Earth.
Your scenario is a bit different, though. Unless the aggressor was built like The Mountain, people could probably have stopped him. In your case, people were actively choosing to not do anything to help, which is wrong. The reason why they can rationalize it away is because there's a mental disconnect between not kicking a guy when he's down and stopping someone else from kicking a guy when he's down (even if both can be stopped by you).
If anyone is actually telling you that you're the dumb one, or that you were in the wrong, for stopping that guy, they are entirely incorrect. They have no ground to stand on. Their position is indefensible and (probably) the only reason they are saying that is because they want to believe it.