You are the King, and House Tyrell is in the danger of being replaced by you. This is why as you visit Highgarden with your army, you can take all the liberties you want.