Too many people allow their ideological identification to drive their beliefs. Since they identify as conservative, and conservatives are strongly against abortion, they feel they must therefore be strongly against abortion (and vice-versa for who those identify as liberals). In the current regime of identity politics most people take a side, and by this I mean they effectively pick a favorite team, and then they support that team on almost everything.
It sounds like you have rejected this regime of identity politics, which is a good thing. Allowing your individual beliefs to be held hostage to a greater identity is not sound thinking. Reality is not a team, reality is not the collectivism of beliefs, it is individual beliefs based on each issue having its own set of facts interacting with your personal values.
Team politics will result in ridiculous personal hypocrisy. For example, Team Conservative hated President Obama wasting the people’s time by playing golf. They attacked him mercilessly for it. But when Trump plays golf more suddenly Team Conservative doesn’t think that’s important at all, even though Trump himself was one of those who attacked Obama for it. By not taking a team, presumably you avoid such hypocrisy.
There is, of course, a cost to rejecting the regime of identity politics. Both teams hate you because you fail to consistently cheer for their side.
My advice is to ignore them. I’ll be on your team if you’ll be on mine.