You asked him out as a joke, however the joke had gone too far. As you've come to realize you've gained actual feelings for him, despite his awkwardness in affection.