Consciousness is a weird thing - its obviously an illusion that we perceive a timeline that our existence travels along. There's no particular reason why there should be a narrative arc - from time to time (pun intended) our mind wanders away from this arc (reminiscing about the "past", or speculating about the "future", not necessarily in that order - reminiscing about false memories, or living one second ahead of what's actually occurring, and so forth). So while we are all time travellers, mostly travelling at the same rate as the rest of the world, we are occasionally travellers in time, but also in parallel worlds (alternative realities - given we only have our point of view to trust, then these alternates are just as valid as consensus).
Now when it comes to travel in space, time travels slowly (especially waiting in lines at airports:) Perhaps the reason no intelligent aliens have visited us is that the further you go, the longer the security line is, and no intelligent race exists with a long enough life to make it to the head of the line, before embarking for this other earth. In all consciousness, you'd find something better to do with your time, wouldn't you?
All this is intimately connected with death. In the 4th dimension, your narrative arc is still there. But after some point (the point of death, where we are, as Kurt Vonnegut said, all most punctual) the narrator has lost the plot. So for travellers in time and space (or their wives or companions) it seems like this would be the single biggest barrier to visiting the neighbours. This is why we see no aliens, nor people from the far future or the distant past. Not because its technologically impossible, nor because they aren't there, but because it's dead boring.
