There's a top-out spring visible in your second and third photos?
Yes, drilling out the damper rods is the preferred option!
The damper rods should be square and centred, but after 40+ years and some questionable machining, anything is possible. Old Yammie forks had fibre sealing rings on the head of the damper rods, which would help to keep them centred and minimise wear if they ran into the inside of the fork tube (along with the obvious task of improving sealing).