Time = Activity/Inactivity².

Cognitive tests incorporate time in its various ramifications:
Time of day
Day of month
Month of year

But is time a good proxy for memory?
Yes, if your existence depends on time.
No, if your existence does not.

A marooned human on an island has no need for time.
A human laborer needs time to calculate wages and when to show up for work.
Therefore, time is a proxy for activity and not memory/cognition.