Should I limit my drill press's versatility ?

[geezer]

My 10in Delta benchtop DP is one of the gratifying and useful implements in the shop, even though I have an 8in VS grinder, I have used it to grind, sand, and rasp.
But a guy recently cautioned that any other operations involving lateral forces will impact the accuracy of the drill when used for precise drilling.

Is that a significant issue to be of concern ?

[Ballistic Therapy]

He's right.
If done enough it will make a difference. Especially if you are putting any force against it.
My drill presses are for drilling only.

[mtptwo]

Drill presses are not mills.