What car do you have? A WRX in a Newage 2.0? You don't really need the intercooler, but it wont hurt to have it. I'm not sure the modified TD04 will give you that much of a power gain, certainly not over 300bhp, though I guess I could be wrong about that. Probably better looking for a VF34 or 35. You can pick these up for about 4 or £500 second-hand. Yeah, best to take out the uppipe as your cat could break up and get into the turbo - besides, you will get better throughput thus more power without the cat. The sports cat will keep everything legal as well as providing a base for more power. A full decat is likely worth another 30bhp though and will be cheaper to buy since there is no cat (I think). Check out Meercat for that.
Don't worry about the car's driveability at 300+ I'm running 340bhp and lots of torque and my wife whose not the best driver, can handle it no probs.
Good luck with it