It took me only couple of days, to experience the problem you are facing.
First we have to say, when you use only an iPhone to controll that should work until the phone is on the same network (and only 2.4 GHz but not 5 GHz) as the smart devices.
Second. The control should work from iPhone until you do not leave the network where your smart devices are. I mean away from home controll will not work unless you set up a Homekit Hub - as an Apple TV, HomePod or iPad 4th gen.
Third. I googled for couple of hours to find any reason of “not responding” Unfortunately there are only generic troubleshooting tips available, like check network/WiFi, resart iPhone, disconnect/reconnect WiFi/Bluetooth, sing out and back to your iCloud, or remove and add again your smart devices to HomeKit, or the worst case scenario: remove “my Home” from Homekit, and start all over.
Fourth. Unfortunately there are many other folks out there having the same issue, what even worst is, that many of them have even HomePod or any other possible devices set up as HomeHub, but with the same issue.
So from here, I in my opinion this is not an issue with the smart device, but the Apple device. As you mentioned the manufacturer’s app works just fine, same here, YeeLight I can use to controll even when I’m away from home.
Hope that Apple is aware of this HomeKit “not responding devices” problem, or maybe I will try to get in touch with them to find out the reason why, or we need to wait until this will be fixed with any of the upcoming iOS updates.
Thanks for reading, have a nice one.