It's a force of habit that when he's worried about being rejected (though saying that was, essentially, to put the ball in Soldat's court about whether or not he wanted to keep talking to him), that when Soldat makes eye contact Yondu looks away. Down, off to the side. Something burned into him from a young age when he worried about getting in trouble to just to turn to an object and avoid engaging when things might go bad.

"Just sayin' if I ain't too good with it sometimes, that's why. Honestly not had much reason to here, though."

And Ego had played to his lesser demons. That surely he could talk it into being a loophole with the other captains. That as a slave he should understand what it meant to get a lost child back. That he could finally have all the wealth and respect he'd always wanted. "Nobody ever managed to convince me of nothin' like that again, didn't matter how mean I was. Figured out bein' a blood relative meant nothin'."

Just to assure Soldat that if he did stick around, he'd already put himself in a position where he knew better.