Harriet died in 1823 and Joseph married Eliza Storrs (1784–1861), sister of William L. Storrs, on December 1 of the following year. With his second wife he was the father of a daughter, Eliza Storrs Trumbull (1826–1862).