This defensible hilltop was highly desirable as the site of a fortification, and has been occupied since at least the 5th century.
Around 1108, a wooden fort was raised here by Robert FitzMartin, Lord Cemmaes as part of the Norman conquest of Pembrokeshire. His son William later rebuilt the castle in stone. The Welsh lord Gruffydd ap Rhys had by that time recaptured the area; but William married his daughter Angharad, and so re-occupied the castle.