Not really. For one, most the time you are at such an altitude that minor mistakes have plenty of time to adjust, and even the large ones you are still capable of getting out of.
For two, the faster you go, the simpler the controls become. Fly something slow, you've got to jerk it around. Fly something fast, just give it a small little nudge now and again.