I wouldn't say I'm worried about riding in the rain, I just don't like it. I was riding bikes for 12 years before I passed my car test, I did my fair share of riding in the rain, and these days I just prefer my comfort.
I passed my bike test when it was pissing it down, I think I only passed because the guy wanted to get it over with and get out of the rain.