Bungay Castle was built by the Normans but was later rebuilt by Roger Bigod, 5th Earl of Norfolk and his family, who also owned Framlingham Castle. Bungay's village sign shows the castle. The Church of St. Mary was once the church of the Benedictine Bungay Priory, founded by Gundreda, wife of Roger de Glanville.
The town was almost destroyed by a great fire in 1688. The central Buttercross was constructed in 1689 and was the place where local farmers displayed their butter and other farm produce for sale. Until 1810, there was also a Corn Cross, but this was taken down and replaced by a pump.
Bungay was important for the printing and paper manufacture industries. Joseph Hooper, a wealthy Harvard graduate who fled Massachusetts when his lands were seized after the American Revolution, rented a mill at Bungay in 1783 and converted it to paper manufacture. Charles Brightly established a printing and stereotype foundry in 1795. Then in partnership with John Filby Childs, the business became Brightly & Childs in 1808 and later Messrs Childs and Son. Charles Childs (1807–1876) succeeded his father as the head of the firm of John Childs & Son. The business was further expanded after 1876 as R. Clay and Sons, Ltd.
The railway arrived with the Harleston to Bungay section of the Waveney Valley Line opening in November 1860 and the Bungay to Beccles section in March 1863. Bungay had its own railway station near Clay's Printers. The station closed to passengers in 1953 and freight in 1964.
Bungay was a town divided into civil parishes which matched its earlier ecclesiastical parishes dedicated to various saints. In the case of Bungay the parishes were Bungay St. Mary and Bungay Holy Trinity. Most of Bungay Holy Trinity was absorbed into the parish of Bungay St. Mary in 1879. In 1910 the parish of Bungay St. Mary became the civil parish of Bungay.