James, was a commercial hemp farmer and Baptist minister in Kentucky who migrated to Missouri after marriage and helped found William Jewell College in Liberty, Missouri. He was prosperous, acquiring six slaves and more than 100 acres of farmland. Robert James travelled to California during the Gold Rush to minister to those searching for gold and died there when Jesse was three years old.
After the death of her husband and Jesse's father Robert, Zerelda remarried, first to Benjamin Simms and then to a doctor named Reuben Samuel. After their marriage in 1855, Samuel moved into the James home. James had two full siblings: his older brother, Alexander Franklin "Frank" James, and a younger sister, Susan Lavenia James. In addition, Reuben Samuel and Zerelda eventually had four children: Sarah Louisa Samuel, John Thomas Samuel, Fannie Quantrell Samuel, and Archie Peyton Samuel. Zerelda and Reuben Samuel acquired a total of seven slaves who raised tobacco on the farm.
The approach of the American Civil War overshadowed the James-Samuel household. Missouri was a border state, sharing characteristics of both North and South, but 75% of the population was from the South or other border states. Clay County lay in a region of Missouri later dubbed "Little Dixie," where slaveholding and Southern identity were stronger than in other areas. Overall, slaves accounted for 10 percent of the population of the state, but in Clay they were 25 percent.
Clay County became the scene of great turmoil after the passage of the Kansas-Nebraska Act in 1854, when the question of whether slavery would be expanded into the neighboring Kansas Territory dominated public life. Much of the tension that led up to the American Civil War centered on the violence that erupted in nearby Kansas between pro- and anti-slavery militias