Berry Pomeroy Castle occupies a spur of land falling steeply to the Gatcombe Valley, three miles northeast of Tornes. The ruins of a late medieval castle are juxtaposed with those of a great Tudor mansion. The Pomeroys settled here soon after the Norman Conquest but their castle dates only from the fifteenth century. It is probably the work of Henry Pomeroy who held the manor from 1446 to 1487. The new defenses were doubtless a response to the menace of French raids, the castle being just a few miles inland from Torbay.

Only one side remains of the castle defenses, comprising the gatehouse, the D-shaped Margaret's Tower and the length of curtain between them. Enough survives to show that this was no regular quadrangle. The gatehouse has tall flanking towers with pointed fronts and a long machicolation between them. An arcade, the narrower part having served as the chapel, divides the chamber over the gate passage. A fine fresco here depicting the Adoration of the Magi shows Flemish influence, and its discovery led to the re-roofing of the gatehouse during the restoration of the 1980s. An earth rampart as reinforcement against artillery backs the curtain, and the walls are liberally supplied with gun ports.

The big residential block on the east side of the courtyard incorporates the Pomeroys' hall and solar, but it was transformed in the large-scale rebuilding of the following century. In 1547, Sir Thomas Pomeroy sold the castle to Edward Seymour, Duke of Somerset and Lord Protector. As well as converting the eastern block, which survives as a well-preserved shell, he began an ambitious Renaissance mansion centered upon an immense new hall range on the far side of the courtyard, overlooking the valley. Unfortunately, it is too fragmentary to be readily appreciated. Somerset was executed in 1552 and his son completed the work on a reduced scale.