The likelihood of you catching anything is virtually zero unless you lick your hands thoroughly on a regular basis.
Or eat food with your hands... or rub your eyes... or your nose... or...
I always wash my hands after a sit-downer but for a pee? Not always.
You should. The number of bacteria in your groin area is rather spectacularly high, even when compared with other areas of your skin. It's warm, it's moist (relatively), and pubic hair gives it a lovely, safe environment.
And contrary to the various jokes, you'd actually be better off pissing on your hands than not washing them at all. Unless you're very sick, urine is sterile and slightly acidic -- not a great growth medium for bacteria.