It is hard on both accounts, yes.
And also: it’s okay to accept people who don’t want a deeper relationship. You can have different friendships at different levels.