How to deal with video arguments and getting harassed by their followers?

I’ve been getting into video arguments with this one person over an academic subject and I was continuing it because they literally have like 80x more followers than me so I thought why not it will give me good exposure. While I did gain some followers I also noticed that their followers are starting to leave tons of insulting comments on my stuff and they’re even harassing my followers in the comments section saying they’re dumb and stuff. I’m still fairly a small account (5k followers) so I don’t really have an “army” of loyal followers to do the same to this person.

I want to continue this argument but I also don’t want these haters to continue harassing me and my followers. I think it’s starting to take a toll on my mental health. But at the same time if I don’t reply I don’t want it to appear like I “lost” and I fear the harassment may get even worse (it seems the constant harassment has halted when I uploaded my response video, but I’m afraid if he replies it will resume unless I keep responding back which I don’t wanna do). What should I do?

I’m also worried if I just keep debating him people may unfollow because they followed for a completely different type of content, not debating others.

Is it fine if I just start blocking all the negative people? What should I do?