As most have already said, it's about riding in your "comfort zone"
I have almost always ridden in all weathers
Mainly due to the fact that I've seldom had the option of an alternative means of transport
But I rode a sidecar outfit for many of those years so the potential problems with wind, snow and ice were never as bad as they have been on a solo
Riding in strong winds takes a certain amount of skill, courage and a LOT of luck
As JP says its all too easy to be blown off course by a sudden gust which can send you careering onto the other carriageway or into the kerb
If you don't feel comfortable riding in "any" adverse conditions, don't do it
Not only will you not enjoy your ride, your safety awareness will be considerably reduced, due to the distraction caused by the conditions, thereby increasing your exposure to other dangers
Those of us lucky enough to have survived the perils of "all weather riding" may occasionally make comments about "fair weather" riders in jest but I would never advocate it as a prerequisite to being a good rider
Whatever and whenever you ride, is purely "personal choice" and should never be influenced by someone else's idea of what makes a proper rider
Just ride when YOU want to