Worth checking train times again nearer the time. ScotRail run additional late night services from Edinburgh when the festival fortnight is on.
For parking, at 2.3 metres I wouldnt risk the Waverley station multi storey on New Street. However, as above Regent Road or try Carlton Road at the back of station. 10 minute walk through the station itself and then up the hill to the castle. Keeps you in the right side of the city for heading back North Berwick.