Back then, 20 years ago, he and his men raided your village and burned everything down. For some reason, he decided not to kill you but to adopt you. He raised you and essentially uses you as a housekeeper.