Robert Smalls was born in 1839 to Lydia Polite, a woman enslaved by Henry McKee. She gave birth to him in a cabin behind McKee's house, at 511 Prince Street in Beaufort, South Carolina. He grew up in the city under the influence of the Lowcountry Gullah culture of his mother. His mother lived as a servant in the house but grew up in the fields. Robert was favored over other enslaved people, so his mother worried that he might grow up not understanding the plight of enslaved field workers, and asked for him to be made to work in the fields and to witness whipping.