Depends on what voltage you're talking about, and what your motherboard allows you to do.
There's two voltage setting in all motherboards, CPU Vcore and memory voltage. When you overclock, the ideal scenario to have is to be able to clock the RAM back a bit (say, if you're running DDR400, manually clock it back to 333), then increase your FSB. The more you increase the FSB, the more power the processor is trying to draw. You'll have to increase the Vcore to compensate. That depends a lot on the processor itself, some can happily overclock a good ways without a Vcore bump, others can't get much of anywhere at all.
Memory is much the same. The faster you push it, the more you have to increase the power going to the RAM. Gaming type (marketing term for high end RAM) can accept higher voltages, Value type RAM can't, the chips can't take the power increase.
For your question, though, if you've increased the FSB, and both processor and RAM are speeding along happily with no crashes, BSoD's, or errors, leave the voltage be. You only need to touch those settings (if they're even available in your BIOS) if you start crashing with your overclock, but want to try and keep it.
In either case, Vcore or memory controller, you don't need to make big jumps. The smallest increment you can, stress test, only go within a small range. Too much may get you stable, but you'll roast the CPU quite quickly, it's not built to hande the higher voltages. You'll see some logs, where the Vcore was raised to stablize an OC, but it was only raised .05v.
Faster than the speed of snot
Two wrongs don't make it right, but I sleep pretty good at night