For reference, this is how to override things on the Module::Build command line.
perl Build.PL --config optimize='-g -pipe'
I seem to recall discussion on the list that it’s going to be better documented in future releases (more end user specific). This will be greatly appreciated.