The castle is built on a hill, in a village named Kastro (castle), on the southernmost cape of Peloponnese. That site is a strategic position overlooking the Ionian sea and the plains of Elis.
Originally, it was built by the Franks, under the rule of Geoffrey II of Villehardouin. It was funded by the incomes of the Latin monasteries of Morea, which he had seized. Its initial name was Mata-Grigfan, given by the Franks, meaning Eye that Oversees the Greeks. During that time, the castle was protecting the famous port of Glarentza, and the town of Andravida, then Princely capital.
The castle changed many names, as it changed hands. After a while, it was named Clermont, or Chlemoutsi in Greek. When it came under Venetian rule, it was named Castel Tornese. Geoffrey's death led to battles between the aristocracy for the castle's control. In 1315, the Catalans take the castle, but soon after it is taken by the Franks that held it until the beginnings of the 15th century.
Carlo Tocco, Count of Cephalonia and Despot of Epirus, seized the castle from the Franks, and in 1427 it passed to the hands of Constantine Palaiologos. No battle decided this change of rule, since the Byzantine emperor married the Count's daughter. In 1687, the Turks took control of the castle for a short while, before the Venetians took the castle from them. In 1826, the castle was attacked and damaged extensively by Ibrahim.
The castle itself is comprised of two enclosures, the external one shaped in a polygon. Today, only traces of the buildings remain, along with a Turkish mosque, and more structures near the walls. Despite its ruined state, though, the castle is one of the best preserved castles of Greece, and is an excellent example of Frankish military architecture.