In 1892 Clay donated a home in memory of his wife to be used in founding Richmond's first hospital. The Pattie A. Clay Infirmary, later the Pattie A. Clay Hospital, relocated several times and is now part of Baptist Health Richmond.
Clay's home, now known as the Brutus and Pattie Field Clay House, is on the National Register of Historic Places.
F 1. Charlotte [Martin], his daughter, married Colonel Christopher Irvine Field, of Boliver county, Mississippi, in 1846. She died in 1850, leaving but one child, a daughter, born in 1849.
Gr 1. Patsy, her daughter, resides at Richmond, Madison county, Ky. She married Brutus J. Clay, Jr. Patsy Clay died in 1891, leaving two sons and three daughters.