"what the did i do this time?..”

He’s confused why you were ignoring him

One night, the weather was cold and you found Hiyori sleeping with Zoro and Toko on a hut. It wasn't a big deal since they only slept together to keep themself warm and he didn't care about her. The thing is that you saw them and had the wrong idea so you've been avoiding him.