Born into a family at the highest tier of planter-dominated Piedmont Virginia society, Madison received the best education possible given the era and location. First, he attended the exclusive school operated by Donald Robertson for the sons of the planter elite, next, for two years he was tutored by an Episcopal priest who was hired by his father to be a live-in teacher for young James. All during his childhood, the sickly youth read widely on his own, mastering Greek and Latin. By age 18 he entered the College of New Jersey, Princeton University of today. Like Jefferson his mentor, Madison depended upon slaves for almost every aspect of his life. When Madison left for Princeton by horseback in July 1769 he was accompanied by a slave named Sawney and at least two other individuals. After he reached Princetown Sawney and the others went back to the plantation. It is believed that he attended Princeton to avoid the relatively unhealthy environment of Williamsburg where the College of William and Mary was generally the site of higher education for the Virginia planter elite.
The old Madison house had as its closest neighbor the home of one Thomas Chew, the long-time sheriff of Orange County. It was known for 1748 burning at the stake of a slave named Eve accused of poisoning the slavemaster. The period's anxieties were highlighted by a court order that placed a murdered black's head upon a pole for public display. The name of a local creek, Negrohead Run was indicative of the key role violence against black slaves played in society during that era.
By the early 1780s, the Madison family possessed well over one hundred slaves, and the Montpelier plantation had more slaves than any other in the county. Madison depended on slave labor to earn his income and admittedly felt financially “unable” to free the human beings he had legal title to. Thus both Madison and his neighbor Jefferson indicated that they could not afford to emancipate their black slave captives. Following the emergence of the anti-colonial movement for American independence and the democratic republican wave of humanist ideology, Madison professed to have developed a distaste for slavery. Like Jefferson and Washington, Madison indicated that he was searching for an alternative means of income that would allow him and his family to continue to enjoy a wealthy and privileged lifestyle. Madison contended, as did Jefferson, that slavery was on the road to gradual extinction on its own. Left alone it would eventually die.
Madison’s key role in the American Revolution, his Federalist Papers and, his role at the Constitution Convention have made him one of the most venerated Founding Fathers. Like Jefferson, he was embarrassed by his ownership of African slaves since it appeared as an obvious contradiction of his revolutionary republican convictions. In 1791 Madison traveled with Jefferson up the Hudson River in New York state. East of Lake George they met a free black who owned a 250-acre farm that he "cultivates with white hirelings and by his industry and good management turns to good account," Madison wrote in his diary of the trip. Madison was unaccustomed to blacks whose personalities were not distorted and oppressed by the burden of slavery and was surprised that the man could read, write "understands accounts" and "is dextrous in his affairs".
At least from the early 1780s Madison seemed especially concerned about northern interference in southern affairs. He indicated his approval of a measure in which the Confederation would maintain a small permanent navy commenting that Southerners should support this move since "without it what is to protect the Southern States for many years to come against the insults & aggressions of their N. Brethren." In the aftermath of the decisive American victory at Yorktown, Madison sought to use their demand for compensation for "slaves & other property" to offset British claims of restitution to Loyalists for loss property.
During his stay in Philadelphia as a member of the Continental Congress Madison's "body servant" slave, Billey, was ruined as a slave in Madison's view. He was so used to life in the free state of Pennsylvania and city life in Philadelphia--a city then enjoying a large, optimistic and developing African-American community--he was no longer, in Madison's view, suitable "to be a fit companion for fellow slaves in Virga." To what extent and in what manner Billey was resisting Madison's effort to ship him back to slavery in Virginia is not known, but Madison did express doubt as to whether it "could be done." The young scholar-politician also indicating that during this era of revolution, he could not "think of punishing him by transportation merely for coveting that liberty for which we have paid the price of so much blood, and have proclaimed so often to be the right, & worthy pursuit, of every human being." Nevertheless, Madison still seemed determined to get as much monetary value from him as possible noting that he could not legally sell him for seven years and that he didn't "expect to get near the worth of him." In all likihood this indicated that he had signed him over to someone on an indentured labor agreement for a sum of cash. Later, after the fulfillment of the agreement Billey would be allowed to be free, although we do not actually know what happened. Even after demonstrating a minimum of empathy with his captive, Madison felt he was entitled to compensation, this is an indication of the limits of his consciousness concerning blacks, slave exploitation and freedom. In his quote Madison confirms that he would pay or expend resources, money or labor in pursuit of white American freedom and that he earlier expended labor, resources, and money to squelch black and African freedom. Continuing to profit from slave exploitation makes it abundantly clear which side of liberty and freedom he stood.
Madison’s motive for professing to desire the abolition of slavery was due in large part to his embarrassment internationally, and its effect of undermining American prestige, particularly among republican and democratic revolutionaries in Europe. Even late in his life, in 1826, Madison complained of the “taunts” that Americans had to endure from Europe due to the institution of slavery. Clearly, Madison had many concrete
Immediately prior to the Revolution, Madison displayed precious little sympathy towards black emancipation in a letter written to William Bradford. The young Founding Father wrote of his fear that if violence broke out between the American colonists and Britain “an insurrection among the slaves may & will be promoted.” He wrote that blacks, “those unhappy wretches,” had organized and chose a leader in order to rise when the British arrived. Madison contemptuously wrote, “they foolishly thought . . . that by revolting to them they should be rewarded with their freedom.” He recommended steps to stop the spread of the insurrectionary mood and to crush those who were organized.
Madison’s ostensible anti slavery efforts peaked during the first session of the U.S. Congress, when he spoke in favor of the creation of a colony on the coast of Africa where ex slaves would be settled. Expressing an idea that he would never relinquish, Madison argued to case for “free” black removal:
[I]t may be remarked as one motive to the benevolent experiment that if such an asylum was provided, it might prove a great encouragement to manumission in the southern parts of the U. S. and even afford the best hope yet presented of putting an end to the slavery in which not less than 600,000 unhappy negroes are now involved.
Late in life, James Madison continued to prove faithful to the slaveholder’s cause by adhering to the curious theory of “diffusion” which maintained that the expansion of slavery was the only way to ensure its eventual demise. British abolitionist Harriet Martineau's 1835 visit to Madison's plantation found the fourth president close to despair that anything could be done about slavery in her perception, that anything could be done about slavery. He readily acknowledged many of its evils, one of which was that the slavemasters themselves lived "in a state of perpetual suspicion, fear and anger."
To his dying day, the prospect of a future civil war, secession, and black insurrection troubled him. The knowledge that the anti-slavery ideals of the American Revolution, at the national level, were supplanted by an expansive and aggressive planter-fueled pro-slavery ideology was perhaps one source of Madison’s sorrow. Clearly, at the level of national politics, the former president had always placed a higher priority on the maintenance of the bond of white unity than on black emancipation. At the personal level, he had consistently placed his personal prosperity and status ahead of any pangs of anti-slavery sentiment or empathy with the plight of the slaves, hundreds of whom he personally held title tof. To conclude that Madison was concerned at all with black emancipation would be giving him more than merely the benefit of the doubt. The facts indicate that his professions of sincerity with regard to the emancipation were only a veil, a democratic cover, of a slaveowner fearful of losing a lucrative source of income, wealth, and status.