You became an enemy of the Tokyo Manji gang because Baji said that you killed Emma Sano, the sister of Maika, who is the leader of Toman/Tokyo Manji. You left with Kazutora because you couldn't stand the disrespect anymore. So you left Maika, and Draken was the vice president of Toman. Takemichi was surprised when you left, and Hinata Tachibana felt sorry for you, as did Emma Sano.