This old pub has now been integrated into Adnams brewery; the frontage (and maybe the entire building) appears to have been rebuilt.

In 1952, The Brickmakers Arms proprietress was Mrs. Hilda Rose Clark (her husband Frederick Henry died 20/12/1950 aged 64) and her telephone number then was Southwold 3362. Mrs Clark died 30/07/68 aged 70, and is listed in the Waveney cemeteries index as a "Retired Publican".
Incidentally, my late-marrying grandfather, Edward Gaymer Parke, (who was born in 1872) was Company Secretary of Adnams and worked for them for 75 years until he retired at the age of 86 in 1958.from Peter Parke

On 8th November 1863, about 7 o'clock in the evening (a Sunday) Mrs. Ballantine Brown was walking to Church. In Church St. she was startled to find someone walking close beside her - a man dressed in a long "oily" (oil skin coat) and a peculiarly shaped sou'wester - a kind never worn by Southwold or East Coast fishermen.
When she came to the entrance of the yard where the "Brickmakers Arms" public house is, the figure disappeared. She then became dreadfully frightened and went to a friend's house nearby. The following morning she, with other people, were on the Cliff watching a fishing boat (half and halfer) of which E . Palmer was master, beating in for the harbour. Upon seeing them the Wesleyan Minister came and told them that the night before (Sunday) Palmer had been instantly killed by the mast falling on him. The time of the accident corresponded exactly with that in which she had seen the figure in Church St. He was wearing a sou'wester which formerly belonged to a French fisherman which she also identified.A Selection of Ghost Stories Smuggling Stories and Poems connected with Southwold by A. BARRETT JENKINS