Gardens were first developed at Butler Point in the 1840s and 1850s by William and Eliza Butler, who resided in Butler House. Although little of those original gardens remain today, some trees planted by the Butlers have survived and are now grown to full maturity. These include a giant Magnolia Grandiflora; an equally large fig with its roots and branches overhanging the sea at high tide; and a very early olive that still stands in the garden as an massive old stump with only one regenerating branch now remaining.  The oldest trees are native Pohutukawa (metrosideros excelsa), some of which were ancient at the time Butler arrived. Pohutukawa line the coastline to Butler House, providing a spectacular backdrop to the property. One surviving tree with a trunk circumference of 11.9 metres is the largest in New Zealand.

Gardens have been developed along the waterfront. These are planted with exotic and subtropical trees and shrubs such as Dombeya, Tibouchina, Hibiscus Mutabilis, Dragon Tree (dracaena), Agaves.   Orchards feature Cherimoya, olives, figs, citrus and Macadamia nut trees.

The produce from the Macadamia trees are sold in local stores as well as a small shop adjacent to the Whaling Museum.
​
Pohutukawa trees surround the Butler family cemetery where in season there are Bella Donna lilies, white irises and daffodils. Among the native bush on the property are the rare Elingamita johnsonii, many large puriri (vitex lucens) the shining broadleaf griselinia, large native puka and other coastal plants such as the Cape Reinga Lily (arthropodium) and varieties of flax (phormium tenax).

Many more varieties of native, sub tropical and exotic plants can be found throughout the gardens and grounds.