According to the Rhoton to Abshier letter of March 2003, Tolbert was born out of wedlock to a woman with the last name of 'Parrish'. He was the grandchild of Mary Gaddy and Littleberry Parrish, and was adopted by Mary's brother Benjamin Gaddy. Tolbert may have come to Hart Co. Kentucky from Halifax or Bedford Co. Virginia. He later moved to Kansas with his wife Rebecca, and their ten children.