After a few months, I decided to revisit this issue. I changed a few settings on the router and I finally found out a feature why.
I turned off AMPDU function. It seems that the latest firmware for me, changed the behaviour on how the device interacts with AMPDU. AMPDU in my router is used so that multiple data to the same destination will be packed together. This helps enhance performance when there are many small packets, but caused problems with this.