side channel attacks are like guessing how heavy an object in a room is based on how sweaty someone is after they've gone in and moved it, and how long it took

completely possible, you just haven't thought about it