It's not so much speed building up, but keeping it from building up in the first place, then braking for junctions, traffic, pedestrians etc.
Since my back cable snapped as I was coming up to an extremely busy roundabout, I've been very cautious with braking.
I do have to do a right turn onto a hill at a reasonable speed, as it's a busy junction, so I'm slowing down as soon as I get over that as that leads onto the roundabout already mentioned, but that cant be avoided if I want to get across outside of 8pm-6am, but I'm certainly not flooring it. id say I was braking for about 150m or so, but the back brake won't stop me (it has pretty new shoes on and is as tight as possible without rubbing whilst riding).
Maybe the brake mechanism is at the wrong angle or something, putting more stress on the blocks?