Their eyes usually suck, and their physical response times are usually considerably slower.
I'd rather have them drive slow than drive fast and be that much more dangerous.
Having people go slow like that, and impede traffic probably causes more accidents then you would see happen due to lack of response time. Accidents happen when people are unsure of what the other person is going to do, not because you're driving 20 kph too quickly.
Driving defensively is all well and good, when when you start driving hesitantly, and people don't know what to expect, thats when you're a danger