See notes for the evidence that he was the brother of Jane Richardson, in spite of using a different father's name (William Howard Richardson) in some of his records. It is assumed that his father was actually George (as implied by the 1871 census), who died when James was 12. Possibly William Howard Richardson was a relative.