Eldred Simpkins Dudley (1811-1881), prominent Reverend and farmer in Bourbon County, Kentucky was the son of General James Dudley (1777-1870) and was born in Fayette County, Kentucky. E. S. Dudley (as he was commonly known) was ordained a minister in the mid-1850s and served as the Reverend of Stony Point Church (Baptist) in Bourbon County, Kentucky.
In 1835, E. S. Dudley married Mary Ann Clay (1816-1839) and they had two children, Thomas Dudley (1836- ) and Mary E. Dudley (1838-). After his wife's death in November of 1839, E. S. Dudley married his wife's older sister, Henrietta Clay (1811?-1860?) in February of 1840. E. S. and Henrietta Dudley had at least five children, including Henry Clay Dudley (1840 - ), Nancy Dudley (1842 - ), Letitia Dudley (1843 - ), Henrietta Rebecca Dudley (1845 - ), and James Ambrose Dudley (1847 - 1920).
Mary Ann and Henrietta Clay were daughters of Henry Clay (1779-1863) and natives of Bourbon County, Kentucky. Henry Clay (1779-1863), commonly known as "Colonel Henry Clay," was the 2nd cousin of U.S. Senator Henry Clay (1777-1852). In 1861, a widower for a second time, E. S. Dudley married Hattie Bassett. The couple had no known children.