The TED and LIBOR have both been trending upwards.
Even if the LIBOR comes down it won't be the saving grace for ARMS that adjust. Most 5/1 ARMS written in 2003 have a margin of anywhere between 2.5 to 3.5. Brokers and banks are paid more money if the margins are higher. Most borrowers didn't really care because the margin means nothing until the loan adjusts. If you had a 5/1 ARM @ 4.75% and you r margin is 3.0 your new rate would be 6.12%. Not bad, but still a full 1.25% higher than what you were previously paying. On a 500k loan that's another 521 dollars a month.
In addition, most interest only arms convert to principal and interest payments when they adjust, which will add to the payment. If you have a jumbo arm that is about to adjust, you are probably better off not refinancing given today's jumbo rates.