

Answer quickly, if noon is 0330 what time is dinner, what is a 9-5 job and what time do you expect to have breakfast. There are lots of adjustments you will need to make, whereas with the current system you know that as a general rule you can expect dinner at around 8, most people to work 9-5, and places to serve breakfast at 8 or 9, so you switch your clock when you arrive and you’re done.
If you’re a local who never moved timezones z then yeah it makes no difference what the numbers are, you would get used to waking up at 9PM and switching date midway through the day, there might even be 2 different words for tomorrow, one for the next day one for the next date, but the moment you traveled to a different location all of your years of being used to general time where things happen go out the window, it’s much more of a hassle than adjusting your clock and assuming times will be mostly similar.
So if you’re not allowed to I assume you get paid overtime if you have to stay after hours because of this, not doing so would be illegal in most countries. Knowing that I would get some friends to come over, play some games chat and do whatever we would normally do until 3/4 AM every Friday, get to hang out with friends and get paid for it, win/win. If asked I would just say “well, my friends came over to see me, and I couldn’t ask them to leave”.