So it was time to think outside the box a little. I remembered I had some large fabric covered buttons from Whole Port.
So I got out my pliers and snipped off the hank on the back. These are plastic on the buttons I have, they may be a little harder if your button backs are metal.
After I got the hank off and the back as smooth as I could. I filled the small hole left with E6000 glue, you want to use something stronger than hot glue for this, super glue, gorilla glue, or another type of epoxy will work best.
After I filled the in the hole with my glue. I nailed a finishing nail into the spot I wanted the towel hanger, leave the nail out from the surface so the towel will hang easier. I chose to put mine on the back of our bathroom door, however I know this isn't going to be ideal for everyone.
I let the glue dry for a few minutes so that it was just starting to become tacky and them stuck it on the nail. I chose the finishing nails because the head of them fit perfectly in the little hole left after clipping the hank off.
Because they are tacky they shouldn't move to much while they cure. You want to let them cure for at least 24 hours before hanging any thing on them.