Inigo is teasing him, but the way he says it, it sounds like a challenge. Like a possibility. Why doesn't he challenge a bunch of them to a fight until they can fight them a teleporting wizard? Huh?

But then Inigo squeezes his hand, assures him it's fine, and -- it would be too much now, to just turn back around. Except the want isn't entirely dissipated, and his frown won't sit still on his face; looking over at Inigo, expression sulky.

"I could go back in and challenge them to a fight."

If you really wanted him to, pal.