Elaine and David Rolfe spent 11 years transforming the garden, which featured in a TV show and gardening magazines.
But after Mr Rolfe, 51, was diagnosed with terminal cancer, the couple had to leave the property on the Llanover Estate near Abergavenny, Monmouthshire.
The private estate said the garden was “too much” for new tenants to maintain.
Mr and Mrs Rolfe, who have eight children, have been told to return the garden to rough pasture or face paying to have it gravelled by the estate owners.
“It is heartbreaking to pull the garden up and see all our hard work go to nothing,” said Mrs Rolfe.
“David is too ill to know what is going on, so that is a blessing.”
The gardens at the rented 17th century Ochran Mill featured on a TV gardening show presented by Monty Don and was open to the public every summer with proceeds going to local charities.
The National Gardens Scheme describes the property as “a Grade II listed water mill (not working) in approx. 2-acres”.
It praised the site for “an abundance of colour and lush growth packed with many varieties of herbaceous perennials and colourful shrubs … interest from early spring to late autumn, hellebores, roses, asters and many more”.
But the garden is now being pulled up after Mrs Rolfe invited friends to help themselves to plants and cuttings before the diggers move in.
Horticulturists from Bristol Zoo are also taking some of the more rare plants and shrubs.
“If I had the time I would do it myself, but my husband is so ill I can’t leave him,” said Mrs Rolfe.
“It’s upsetting they are trying to charge me, it’s very stressful for our whole family.
“It has been a bit of a nightmare, we didn’t want to move. I would like the plants to go to a good home.”
The couple and their three youngest children have had to move out of the water mill after Mr Rolfe, a website designer, became ill and was diagnosed with a brain tumour last June.
They have moved into rented accommodation with a downstairs bedroom and bathroom in nearby Abergavenny.
Their daughter Jennifer, 24, said: “We have great memories of the garden from growing up.
“It’s a lot of upkeep but it’s amazing and I would have liked to have seen it preserved.”
James Perks, resident land agent for the Llanover Estate, confirmed the approximate cost to clear the “overgrown” garden and put gravel down would be £5,000.
He said that when Mrs Rolfe has removed what she can, the cost will be re-evaluated.
Mr Perks defended the landlord’s timing of the garden work, given Mr Rolfe’s deteriorating condition.
He said: “If someone creates something extraordinary it’s not going to be sustainable once they have left.
“We have tried to be as sympathetic as we can and to give Mrs Rolfe more time but we have to get it [the home] ready for another tenant.
“We can’t leave the property empty for months.”