Caister Castle stands three miles north of Great Yarmouth, not at Caister-on-the-Sea, but a little inland at West Caister. This brick stronghold is a monument to Sir John Fastolf. Fastolf was a distinguished veteran of the Hundred Years War, a knight of relatively humble origin who played an important part in the Lancastrian conquest of northern France.

Falstolf built this castle in 1432-46 when he was enjoying a prosperous retirement. On his death in 1459, Caister passed to the Paston family, whose letters give a first-hand portrayal of life in fifteenth century Norfolk. Unfortunately for the Pastons, the Duke of Norfolk also laid claim to the castle and, when legal means had failed, he set about making good his claim by force. In 1469, he brought a considerable force to lay siege to the castle, which creditably held out for several weeks against the duke's cannon before the inevitable surrender.

Veterans of the French wars built most fifteenth century castles and a number were in fashionable brick. They tended to be showplaces, combining lavish accommodations with a show of strength. Some had a secondary role in coastal defense and Caistor did repulse French raiders shortly after its completion.

Caister was one of the finest of its kind but rather too much was pulled down in the eighteenth century. The castle stands in a wide moat still full of water. It is one of those with an inner quadrangle and a subsidiary base court for retainers. This is less obvious now because the arm of the moat between the two courtyards has been filled in.

There is also part of a third courtyard behind, arrested only by a circular corner tower incorporated in a later house. The base court, of inferior brick, is now fragmentary and the main quadrangle had suffered so much destruction that only its north and west walls still stand.