My favourite things there included the castle (it's admittedly expensive, but it is fairly awesome, so, well, up to you - the main art/history museums of the city have free admission, so it balances out to some extent), climbing Arthur's Seat in Holyrood Park (it's a hill of 250-ish meters, towering over the city and offering a fantastic view, the hike takes 30-45 mins but you may want to wear proper walking shoes if you're going to do it, some parts are more like a mountain hike than like a walk in a city park), the National Museum of Scotland, and just walking around on the Royal Mile and some other parts of town.

As for hostels, the Edinburgh Central Youth Hostel was decent, affordable and located reasonably well - the centre of Edinburgh is small enough that you can do most things on foot, the hostel is maybe ten minutes on foot from Waverley Station, which I'd say is the central point of the city, more or less.

Well, don't miss Arthur's Seat then. Holyrood Park should be good for running as well, if you go around and inbetween the hills instead of climbing them. Though Edinburgh does have a lot of elevation in general, for a city that's maybe 5 km from the sea.

I dare say the people living there will be able to give you better suggestions than mine, but this and Mix' comments should be a start. I love Edinburgh, it's a great city.