Does lack of affection hurt a relationship and in what ways?

Me and my fiance get into these little tiffs about lack of affection. He had never slept in the same bed as me or his past girlfriends, he always says that none of them never talked in the relationship and he also says that he never showed attention in those relationships. In my mind and how I feel is that its the lack of affection he gives out makes me feel(and previously them) like I cant communicate with him. There is literally no physical attention, no kissing no hugging no touching cuddling and etc.... When I do try to communicate and ask why and try to explain why its hard to feel emotionally attached to him because of it he shuts me out and says that im wrong and he doesnt see how affection can be that important. Am I wrong? Or what could it possibly be? And if I am right, what are some good ways to approach him? Thank you!