Lee is located in the southwest portion of the town of Lee on the old state road, Route 69. The first post office in the town was established here with Jesse Matteson one of the first, if not the first, postmasters.

Prior to 1812 David Byam built the first grist mill along West Creek. This mill was known as the Pepper Mill.

In 1840 a Methodist Church was built on land donated for that purpose by Martin Baker.

By 1851 four sawmills had been built in this area and a Sash & Blind Factory was operating. There were three tanneries in the area. Vincent Taft kept a tavern here and operated it until the building burned in June 1864.

The Lee Grange was organized in 1867 and celebrated their 125th Anniversary in April 1992.