Some of the best clues to the early days of the building come from old maps. The 1775 map suggests a main entrance into the house and grounds directly facing the house frontage (on the A915 mid-way between Durham Wynd and the corner turning up to Upper Largo). However, by 1828 the map shows that the driveway had been remodelled to enter close to the corner of the A915 and the road up to Largo Home Farm (the tree-lined track of which can still be traced today). An 1854 map shows a lot of detail of the grounds. The sketch below (based on the 1854 map) illustrates the key features of Largo House and grounds in the mid-19th century.
"a select party of gardeners and others met in Anderson's Inn, Colinsburgh, and gave a supper to Mr Thomas Stewart, gardener, Largo House, on his leaving this neighbourhood for Edinburgh, in order to testify their respect for him as a professional gardener, and their esteem for him as an intelligent companion and an agreeable friend. Mr S. has lived for 25 years at Largo, and all present expressed their regret on account of his leaving Fife, and their best wishes for his welfare."
During his lifetime, General James Durham had the stable block built (in 1815), the two wings constructed on the rear of the house, had the conservatory added and had the Eagle Lodge and Gate added with a new driveway (all 1831) (source 'Largo: An Illustrated History' by Eunson and Band).