I've been told by numerous individuals (including David from APi and Ralph from Milsport), that the rolling diameter of the alloy which negatively affects handling.
So yes, going to 18" will affect handling, but it's not going to destroy it by any means, some impreza time attack cars run 18s so they can fit bigger brakes.