I'd say most DO have that issue, actually. It doesn't get talked about enough because many people quitting porn want to make it seem simpler or don't want to think too much about sexuality or how they might change, but it is most definitely out there.
When we develop and addiction to porn and take away the porn, the porn brain keeps looking for ways to get its fix. For example, lots of guys end up spending hours looking at social media or some PG thing related to a fetish or whatever because their brain is so desperate for it. It definitely helps to pay attention and be ready for the weird ways that your brain is going to try to get something resembling porn.
One common thing among porn addicts is to see an attractive woman in public, say to himself "that woman has a really hot [blank]", and then go home and find porn with women with a really hot [blank]. So, seeing an attractive woman in public becomes the first domino falling in a porn activity, and the porn addict gets to go home and masturbate to whatever they saw. In other words, a trigger. Take the porn away from the scenario but leave in the attractive women and naturally those women are going to going to stir some very strange reactions in the recovering porn addict. Eventually once the brain is starved of the ability to go home and masturbate to what he's seeing in the mini skirt at Starbucks, the association goes away. Yeah, the women will still be beautiful, but different women will have different levels of beauty to you, and your sense of their beauty will be more holistic. What I mean by that is that a beautiful woman will simply be beautiful... not a combination of a really nice [blank] and set of [blank], like some set of porn like attributes put together like a Mr. Potato Head. Body parts isolated from the whole person are less interesting than the whole person to someone who is recovered to porn, even if the individual body parts themselves aren't interesting enough to stimulate the porn addicted version of the same person. In my experience, seeing a beautiful woman during my porn days made me anxious and bothered, maybe even obsessive, but now it is a pleasant experience. It isn't because I'm trying to get her number now or anything, it is just a nice thing to see a beautiful person, like seeing a beautiful anything when I'm out and about. I acknowledge the beauty, am thankful for being able to see it, and go about my day.
It is a little bit off topic, but check this out. I've posted this countless times, but it so often bears repeating. I found it incredibly helpful in trying to make sense of that aspect of recovery. Sexual tastes change when we quit porn, that much is inevitable. It helps to have some sort of road map.
https://www.youtube.com/watch?v=PvQrFBOyDs0