That makes the router the top suspect. If your modem and router are separate units you can easily get a second opinion by testing with a computer connected directly to the modem.
Make sure you have the latest firmware for the router. Then reset it to factory defaults and reconfigure it.