That’s not the last of the pain in the gardens, mind you.
The general advice for the Flower Quest is to do it prior to the Infected Crossroads appearing. In the randomizer, it is not infected until you acquire an item I still do not have. In vanilla, that is still the case, but an alternate trigger is slaying the Broken Vessel, which is required to get Monarch Wings, which are required to enter the Royal Waterways without using a Small Key. As I needed Isma’s Tear to enter the Queen’s Gardens (or I could use another route which also causes the Infected Crossroads), there wasn’t a whole lot I could do about it. The next best strategy is to clear the route of most encounters (some things respawn when you so much as leave the room), so I did that.
Beyond just the Flower Quest, the Queen’s Gardens is one of the toughest non-DLC locations in the game. To be fair, it’s a location that is only a required visit if you’re getting one of the harder endings. (The game has five endings and you don’t need to come here for the first one.) It even still has a boss fight for me, and it’s one that’s so hard I avoid it if at all possible in a randomizer. There’s only one other non-dream boss that’s that difficult for me, and I’ll be dodging that one, too. That’s not to say I won’t do either of them, but I’ll wait until I at least have the first ending before I bother with them.