Yesss- Exactly. Part of why I imagine he creeps everyone out. What he does explain could make complete sense. He, when he articulates something, sounds scarily reasonable in his logic. He keeps fixating on whoever, hell maybe he is worried about both Peter and Tony. There were enough co-favors to make that scarily likely to him. Use Peter against Tony and Tony against Peter. Peter as the superhuman with a need to protect people, Tony as the elder with a burden of care, and from what Bucky has gleaned Tony always felt like he had to provide for others- the eagerness to please that had him devoting entire floors of his tower to the team, renaming it Avengers Tower, these are not signs that point away from the sort of damage Parties left.

Same thing for Peter, who looks so skinny and frail and combines Steve pre-serum with Super Soldier in a horrible combo pack of things HYDRA would love to tie up and share for a weekend. His drive to help and prove himself is prime fodder for HYDRA recruiting. It all makes complete sense in Bucky's view and from the life experiences he has had.