The village derives its name from two settlers, John and Briant Brown. The West Windsor Historical Society is in Brownsville and has a wealth of information on the sheep farms and industries that sustained the early residents.
Just east of Brownsville is the entrance to Ascutney Mountain Resort, which used to be one of the major ski areas in the state, until it closed for good in 2010 and their ski lifts were sold in August 2014.
Three of Vermont's covered bridges are nearby. The "Best's" bridge near Churchill Road and the "Bowers" bridge are listed in the National Register of Historic Places. The "Twigg" bridge on Yale Road was moved to its location by a developer and was heavily damaged by wind in 2002.
While modest in size, Brownsville is home to a number of locally owned establishments including the Brownsville General Store (BGS). The BGS offers a wide variety of goods and services including fresh-baked breads, alcohols, treats and breakfast sandwiches.
Slightly south and west of Brownsville is the Little Ascutney Wildlife Management Area, a state owned conservation area hosting wildlife such as white-tailed deer, fisher, coyotes, bobcats, beaver and otter.