It took a bit of thinking but I've finally worked it out. It's because of two things. One is that both seem to be thoughtlessly generic and basically lazy. All the religious festivals at this time of year - and there are many whether it's the Hindu Festival of Lights Deepavali (sometimes called Diwali) at the beginning of November, the Jewish festival of Hanukkah, St Nicholas Day, the solstice celebrations of Druids, the Christian Christmas or any other of the multitude across the world - have their own significances and to me it seems disrespectful to lump them all together.
The other irritant - and I know this is petty but it bugs me - is that most of these are not 'real' holidays. While there may be time off for some associated with some of them - and many who celebrate them may only enjoy them as a break from work and a secular celebration without acknowledging their religious origins - they certainly don't all have associated holidays. To me - and I willingly admit that I'm probably being pedantic - makes 'Happy Holidays' a nonsense.
So, while I certainly won't be expecting everyone to agree with me or follow my example I will still be wishing people merry Christmas and a happy New Year along with happy Hanukkah, happy Diwali and so on. To me that's feels like being respectful.