James Elliot died while still young of small pox, and his wife Martha Day Elliot and 3 sons (James, Samuel, William) were left in low circumstances. They soon moved to New Salem, MA. Of James and Martha's sons, James became a lawyer, settled in Brattleborough, VT, was a representative in Congress from VT for four years, and held other offices. Samuel was a lawyer in Brattleborough for forty years. William was in trade in Boston, but died in Brattleborough in 1839.