When my guests use "Add to calendar," the event is all day. Why?
We let the host enter 'free form' in the Time field (e.g. they can enter "Noon to 4 pm", or "7-ish" and don't force them to select an exact time). Because of this, we can't import a time when you add to your calendar. Instead, the event comes in as "all day" event.
We do this because hosts like the ability to write whatever they want for a "time", and guests know that the event is not a 24-hour affair so they just add or edit the time when they import the event to their calendar.