I was told that if there is no work being done in a construction zone, say on a Sunday or similar, that you don't have to abide by the posted speed limits in that construction zone, rather you were "allowed" to go the normally posted limit. This doesn't sound right to me, because if they allowed us to go the normal limit, wouldn't they block or hide the construction zone speed limit signs? Thanks.

You must always abide by the posted speed limit, even when workers are not present. The presence of workers only impacts the set fine youd receive if caught speeding (it doubles), but the speed limit itself remains unchanged.

Note: This is assuming its a properly posted sign. Im of the belief that orange and black speed limit signs you sometimes see at construction sites are not legally enforceable (not thats Id recommend ignoring them). The signage should be the typical black and white signs, and there should be signage advising youre entering a construction zone.

As previous stated, workers present only impacts your set fine. You're still speeding whether they are there or not, but your fine increases. The extra fine is there for you to be extra cautious, similar to Community Safety Zones.

The temporary speed limit isn't solely dependent on workers being present. Construction brings different conditions to that portion of the road that may require a lower limit (raised manholes, redirection of traffic, etc). These factors don't change once a workers shift is over for the day.