Personally I skip Times Sq as it's more like Disneyland than New York City. But hey, if it's your first visit have at it.
First off, if you have a smart phone get a NYC Metro app it will be well worth the couple of bucks you spend on it.
Here's a few sites I'd recommend...
As someone else mentioned: Brooklyn. Spend the day. Take the F train to York St. It will put you in DUMBO (Down Under the Manhattan Bridge Overpass). You can walk along the East River and catch some great skyline views of Manhattan.
Also there's Jane's Carousel set on a spectacular spot between the Brooklyn and Manhattan bridges, in a Pavilion designed by Pritzker Prize winning French architect Jean Nouvel.
There's lots more to see in Brooklyn but if you must, you can than wend your way back over the Brooklyn Bridge to lower Manhattan.
An enjoyable walk is along the High Line. The High Line is a park built on the old Meat Packing District's abandoned elevated rail line so you be above the fray and catch some interesting sites. It's on Manhattan's West Side and runs from Gansevoort St in the Meatpacking District to West 34th St near Penn Station/Madison Sq Garden, between 10th & 11th Ave's. You can catch the A, C, or E to 14th St & 8th Ave. In fact have lunch up there. You stop off at the indoor Chelsea Market. It takes up a whole block enter the 9th Ave ('tween 15th & 16th St's) and exit the 10th Ave side with your snacks. Then meander over to the High Line.
Lastly you should take in the 911 Memorial but you'll need to reserve a space now. You might be too late but it's definitely worth a shot (April's usually not too busy). It's free but they're trying to contain the amount of visitors on the the site. Look around on the web you should be able to find the reservation system pretty easily.
And if you gotta get on top of something, do the Rock rather than the Empire.
--Rich