John Caldwell was born on 3 May 1769, the eldest son of Elizabeth Calderwood (d. 1796) and John Caldwell Sr. (1742-1803), a prosperous farmer, miller and linen manufacturer whose estate, Harmony Hill, held under a small head rent from Lord Antrim, was situated near the linen-market town of Ballymoney in north County Antrim.