Brentwood Hall was conceived and built by Rogers Caldwell, son of Nashville banker James Caldwell. The younger Caldwell made his fortune--reported at one time at $650 million--and made his mark in Tennessee as a financial wizard during the '20s. Caldwell, known at that time as "The J. P. Morgan of the South," had a dream to restore the South which was still reeling from the devastation of the Civil War. He coined the phrase, "We bank on the South" for his financial corporation, Caldwell and Co., which included the Bank of Tennessee. 

Andrew Jackson, a Tennessean and the nation's 7th president, was Caldwell's hero. When planning his Brentwood Hall estate, Caldwell used Jackson's home, the Hermitage, as his model. Brentwood House's oversized Williamsburg brick, said to be imported from Virginia, was never painted white, however, like the Hermitage. The word "Hall" refers to the entire estate, while "House" indicates only the Caldwell residence. 

As a marble engraving on the house attests, the building was completed in 1927. Caldwell's interior designer collected from Europe the outstanding marble fireplaces, mantles, gilt mirrors and some of the fixtures which still grace the mansion. 

Caldwell lost his fortune and, ultimately, his home when the Bank of Tennessee collapsed during the Great Depression. The state gained a judgment against the house for part of Caldwell's debts, though Caldwell continued to live there until 1957. No need to mourn Caldwell's fate, however; he was only forty when the Depression hit, and he continued to play an important role for close to 40 more years in Nashville society and politics from his comfortable ante-bellum home in Franklin. To be invited to one of Caldwell's weekly luncheons was the sure mark of one's own importance to the literary or political community. Hanging in each one of Caldwell's homes was his adage, "While in this house please do not say anything unkind about anyone, bearing in mind that what you think of others is nothing like as important as what others might think of you." Surely Caldwell himself would have nothing negative to say about the events of his life or the important role his home has come to play for the state. 

Buford Ellington, a former commissioner of agriculture, was elected governor in 1958 and set up offices in the then barren house to organize his administration. Following his inauguration, Ellington decided against selling the estate and began moving the agricultural offices, which had outgrown their downtown space, to the mansion. William F. "Red" Moss, who followed Ellington as agriculture commissioner, moved his office into Brentwood House in 1960. 

The 207 acre estate was renamed Ellington Agricultural Center by a resolution of the legislature in 1961 when Frank C. Clement, who once again held the governor's office, established the grounds as the headquarters for the Tennessee Department of Agriculture. Tennessee was the first state to place its agriculture department on an actual farm.