how to unfollow?

Is it just me? I'm having trouble unfollowing creators, when I refresh my page the amount I have goes back to where it originally was and I have to keep re unfollowing them.

I don't want so many channels anymore and I struggle to unfollow the ones I don't want anymore ;-;