Shim it, I'm not sure for very small stuff, but bearing shops supply shafting shims, or put some verniers on the shaft and in the hole, find the difference and shim it yourself. Buy the right thickness shim steel so it has a very small interference fit when rolled into a cylindrical shape. Or if the hole is seriously elliptical, get it machined out a touch and have a sleeve turned up on a lathe.