Little more than a mile south of St. Michael's, and about two miles from "Perry Cabin," beautifully situated on Spencer Creek, where it empties into St. Michael's River, is "Spencer Hall," the seat, for several generations, of the Spencers, some of whom gained distinction in the various walks of life, but none of the name now reside in Talbot County. The family was of Norman origin and of noble degree, and can be easily traced to the eleventh century, being descended from Robert le Despencer, Lord Stewart of the household of William the Conqueror, and one of the Norman barons whose name is in the Roll of Battle Abbey, and in the great Domesday Book appears as Robertus Dispensator.
In 1657 there came to Northumberland County, Virginia, Nicholas and Robert Spencer, brothers, of Cople, Bedfordshire, descended in the seventh generation from Robert Spencer, A.D. 1475, younger sons of Nicholas Spencer and his wife, Mary Gostwick, daughter of Sir Edward Gostwick, and a branch of the Northamptonshire family. They were accompanied by the brothers John and Lawrence Washington, also from Bedfordshire, the former being the great-grandfather of Gen. George Washington. Nicholas Spencer, by grants and purchases, came into possession of large tracts of land on the Eastern Shore of Maryland, 1658 to 1666 1. He finally settled in Virginia near the Washingtons. He was later known as Col. Nicholas Spencer, and was Secretary of Virginia,
1679-88. Robert Spencer was born in 1635. After removing from Virginia to Barbadoes, where he remained for several years, he came to Maryland in 1678 and settled in Talbot County in 1683. He died prior to April, 1688. He left an only son, James Spencer, born in Barbadoes in 1667, who came to Talbot after his father's death and
settled on St. Michael's River. He died in 1714, leaving by his first wife, Isabella, four sons, James Spencer, Jr., the founder of "Spencer Hall," Charles, William and Hugh Spencer, and two daughters, Alice and Mary Spencer. The last male owner of "Spencer Hall" was Col. Perry Spencer, (1750-1822), and the property finally passed out of the hands of the Spencers in 1837. The Spencers were never numerous in Talbot, and but one of that family is now living in Maryland, Col. Richard H. Spencer, of Baltimore.