Hi ninjamk,
Sounds like you have a stretched outer cable that is acting like a spring being compressed when you apply lever pressure.
Also your shoe friction surface may not be running concentric to your drum.
Scour your brake shoe surface, reassemble, LIGHTLY apply your brake while wheel is rotating. Check for the witness of shinny high spots. Don't over apply. Or you will flex the shoe and create false witness.
If it shows consistent even witness all over your linings,,,,than outer cable stretch is most likely your problem.
Good luck.