William's black cook Eliza apparently lived with him and he fathered at least three children by her, possibly several others. In the 1870 census we find Eliza Sergin with Canada, Gronison and two other children (Union and Pennie) with the Sergin surname.