Book 10, Number 10.7.14:

Yahya related to me that Malik said, "The sunna concerning the time of prayer on the ids of Fitr and Azha - and there is no disagreement amongst us about it - is that the imam leaves his house and as soon as he has reached the place of prayer the prayer falls due."

Yahya said that Malik was asked whether a man who prayed with the imam could leave before the khutba, and he said, "He should not leave until the imam leaves."