Lobotomy

Is it possible that when we pass another person, where ever we may be, that we may know them from where were only new to the world. A baby observes the world with a much different perspective than adults or teenagers, so when we were younger observing everyone and everything that we saw, someone took our interest and we looked at them for a few seconds.
Now, is it possible that 10, 16 or maybe 20 years later that we cross paths with the same person recognising this person, unable to think of where from - completely confusing - the answer is that we saw them when we were a few months old and we remembered this person.
Is this possible?
If you look at someone you recognise, did you know them from five years ago; just passing them on the street.