Now I got fed up enough with this and examined things a bit. Short googling didn't help straight but I ended up checking out logs from Console. Spotify seemed to have pretty many log entries there so I ended up closing it. That makes sense, since Spotify's P2P architecture is probably sharing music from it's cache on my computer every now and then.
After that, I haven't had any sleep mode issues at least so far... (went to sleep mode only a couple of times so far). But thought to share it in case somebody ends up googling with similar problems.