I understand you are not seeing this issue in your lab at all. Allow me to share what I’ve done and learned so far. FWIW, I’m a retired Unix/Networking consultant, I have skills and, as demonstrated earlier, the ability to say “I was wrong” when I am (though proudly I don’t need to do that too often). Of course, I have to treat the bulbs as a black box since I don’t (and don’t expect to) have source code.
My setup:
I have 14 rgb bulbs here. I’m toying with my own software and how much I can keep such devices from talking to the internet. They are typically not allowed to talk to the internet but are allowed to talk to DNS. The reason that I allow them to talk to DNS is that they tend to become ‘sluggish’ in answering initial requests, that is, a bulb that has sat idle a long time may take a couple a messages before it responds. Allowing DNS fixed that. This has been true since before fw .51, I just never made much fuss about it as I am certainly working in my own little world here. But, at this point, it may be a hint to help. My servers are all Linux machines though I doubt that has any bearing, the more info you get the better.
Things I’ve tried:
I have tried allowing them freedom to talk to anything and I get a hung bulbs overnight. I’ve tried to capture that with tcpdump, unfortunately the low end hardware I bought to load DD-wrt on is really low end and doesn’t have enough memory to store an overnight tcpdump, pushing it over the net via ssh (while capturing the wireless side, the ssh itself not being in the capture) yields enough packet loss to be confusing.
mDNS?
I do not run mDNS. For some reason I’ve been wondering if that’s factor. With my experiments so far I have yet to investigate that.
Isolating the bulbs
I have isolated all 14 bulbs from the internet including DNS. They cannot talk to anything but my devices for DHCP. The result is, they are sitting idle overnight with no messages being sent to them. I have still had a couple of bulbs hang requiring a power cycle this morning. However…
Tonights Experiment:
Just now, I power cycled all of the bulbs and will leave everything blocked. We’ll see if I have hung bulbs in the morning. I do this because I have a suspicion that the bulbs that hung were not power cycled since I isolated them.
A suggestion to you. Take a dozen or so bulbs and put them in a similar situation, let them connect to a wireless, get a DHCP address and nothing else, no mDNS, no DNS, nada. I would be surprised if you don’t see the problem unless, of course, the experiment I am performing tonights experiment show otherwise. I’ll post the result of that tomorrow morning.
Meanwhile, if there is anything you’d like me to try please let me know. I’d even go so far as to setup some remote access to that isolated network if that would be helpful.
I know you guys are super frustrated and nervous about this. It’s just a software bug, we just need to find it and world will be all happy again. Whatever I can do to help.
We will follow your advice and method to reproduce this problem.