Is this enough to leave my job?

Getting moved around within an organization is a fact of life at most places. While I know you don't want to move, at least give it a fair chance for a few months if you get moved to see if you can make it work. Who knows - you may find someone even better to work with.