Inverness was one of the chief strongholds of the Picts, and in AD 565 was visited by St Columba with the intention of converting the Pictish king Brude, who is supposed to have resided in the vitrified fort on Craig Phadrig, on the western edge of the city. A 93 oz (2.9 kg) silver chain dating to 500–800 was found just to the south of Torvean in 1983. A church or a monk's cell is thought to have been established by early Celtic monks on St Michael's Mount, a mound close to the river, now the site of the Old High Church and graveyard. The castle is said to have been built by Máel Coluim III (Malcolm III) of Scotland, after he had razed to the ground the castle in which Mac Bethad mac Findláich (Macbeth) had, according to much later tradition, murdered Máel Coluim's father Donnchad (Duncan I), and which stood on a hill around 1 km to the north-east.
Medieval Inverness suffered regular raids from the Western Isles, particularly by the MacDonald Lords of the Isles in the 15th century. In 1187 one Domhnall Bán (Donald Ban) led islanders in a battle at Torvean against men from Inverness Castle led by the governor's son, Donnchadh Mac An Toisich (Duncan Mackintosh). Both leaders were killed in the battle, Donald Ban is said to have been buried in a large cairn near the river, close to where the silver chain was found. Local tradition says that the citizens fought off the Clan Donald in 1340 at the Battle of Blairnacoi on Drumderfit Hill, north of Inverness across the Beauly Firth.
On his way to the Battle of Harlaw in 1411, Donald of Islay harried the city, and sixteen years later James I held a parliament in the castle to which the northern chieftains were summoned, of whom three were arrested for defying the king's command. Clan Munro defeated Clan Mackintosh in 1454 at the Battle of Clachnaharry just west of the city. Clan Donald and their allies stormed the castle during the Raid on Ross in 1491.