The name is taken from the village's situation on the brook formerly known as the Toller, now called the Hooke. The addition Fratrum is the Latin for brothers and refers to the mediaeval ownership of the manor by the Knights Hospitallers, which distinguishes it from the other Tollers, namely Toller Porcorum and Toller Whelme. It is often referred to as Little Toller.
The village has a notable 16th century farm house, Little Toller Farm, built largely by John Samways, who acquired the estate in 1540 and later occupied by the Fulford family. The church of Saint Basil was rebuilt in the 19th century but possesses not only an unusual Norman font, carved with archaic figures, but also a relief of Saint Mary Magdalene of the same period.