Originally a small medieval fortress, it is thought that the palace was first built by William de Braose who became Bishop in 1266.

The palace was abandoned as a residence following damage in the Owain Glyndwr rebellion of 1402-05, with the Bishops thereafter preferring their more secure dwelling at Mathern in Monmouthshire.

Although the palace seems to have remained intact until at least 1601, it was probably destroyed during the English Civil War when many castles were similarly slighted by Parliamentarian forces.