The Lannisters have spies everywhere
His Grace summoned me to King's Landing, I'll go to King's Landing. But not alone. Call the banners!
How can I call myself King If I can't hold my own castle?
When I'm done with the Lannisters, I will march back North, and root you out of your keep and hang you for an Oath breaker