Bucky was never exactly twinky in adulthood, but when he was first captured, he was younger and smaller and prettier. He was attractive to the men, and he was raped enthusiastically. As the decades wore on, though, and he got scruffier and bigger and older-looking, the rapes tapered off.

This has left Bucky feeling innately undesirable, but it's for such a fucked up reason he feels he can't explain it to anybody. I will give you my firstborn if this involved rejecting Steve's romantic overtures; Bucky knows he's disgusting and feels Steve must be making fun of him or doing it out of pity.