The town began as a mining settlement. It was surveyed and proclaimed in 1855. Amherst became a municipality in 1858, which included the nearby town of Talbot, though as Talbot increased in size it eventually replaced Amherst as the municipality. Much of the original township has been destroyed by bushfire, and little remains other than the Amherst Cemetery.
Amherst is a town rich in history. It is accredited with the first official gold find in Victoria in 1851 setting off a gold rush stretching all over central Victoria. Although much of the architecture constructed during the gold rush days was destroyed by bush fires, there are still many examples of some of the original buildings at nearby Talbot, just 4km away.
Amherst is considered to be the location where gold was first found in Victoria, and the property still abounds with many signs of the alluvial gold fields. Even today, prospecting still continues in the surrounding areas. The Chinese Garden Cottage is on the site of an old Chinese Garden, run during the gold rush days.