When To Pass A School Bus

I live in a busy neighborhood. On a two way traffic I was stopped behind a school bus that had it's stop sign out and lights flashing while picking up kids. Once it finished doing that he retracted the stop sign and stopped the lights from flashing. The vehicles from the opposite end started flowing in. I waited for a bit for him to start moving, but he didn't move he just turned the hazards on (2 yellow lights on top of the bus). I then passed by thinking its safte to move as the lights were not flashing and the stop sign was still retracted. as I passed him and went into the round about, I saw a father walking his kid to the bus and simultaneously the bus driver turned on the flashing light and pulled the stop sign out. There were no police officers. I wanted to know if I commited an offence by passing him.

Thanks for your answer in advance

No offense if you started passing him before the red lights and stop sign were reactivated. He turned on the amber hazards to let you know he was not leaving.