The White Elm is shown on OS maps as a beerhouse at least as late as 1904. By the 1974 sheet, it's just shown as "The White Elm" - whether or not it was still a beerhouse by that date isn't entirely clear, but it seems likely it wasn't.

It's also reported in Tavern Row (1861), Norton Road (1871/1881) and Broom Hill (1891).

A Railway Tavern is listed in 1851 between Tavern Street and Broom Hill. This has been assumed to be an earlier name for the White Elm. However, no Tavern Street can be found in Woolpit, now or on old maps, and the pub is some distance from the nearest railway line (which is at Elmswell). Could this actually have been a reference to the Elmswell Railway Tavern? This seems more probable.