I spend about two hours on my website every day. More than the time frame, focus should be on what should be done on todays schedule, such as removing spam comments, approving comments, replying to the comment, publishing posts etc.
It is not necessary to spend days on the development of the site. Only a few hours daily are enough. Sometimes only 60 minutes are enough for making it successful. Don't take too much stress and take rest too. Don't be afraid of losing ranks, but take care of your health too. If your site ranks higher and you got ill, who will manage the site? All the effort will be lost forever.