Brown was the middle child of three whose parents suffered from an unhappy marriage, in the Greenpoint neighborhood of Brooklyn, New York. She was the granddaughter of politician Benjamin Gratz Brown. In 1923 she attended Chateau Brilliantmont boarding school in Lausanne, Switzerland, while her parents were living in India and Canterbury, Connecticut. In 1925 she attended The Kew-Forest School. She began attending Dana Hall School in Wellesley, Massachusetts, in 1926, where she did well in athletics. After graduation in 1928, Brown went on to Hollins College in Roanoke, Virginia.