About loosing the implanted memories

It was a reasonably enjoyable thriller, although nothing special. But there's one thing that they got wrong in my opinion. The movie implied that Kevin Costner could loose the implanted memories and also memory of all he did while under the influence of implanted memories. Now this last part does not make any sense. For instance, why would Costner not remember meeting the wife and daughter? He should remember that even after he looses the implanted memories, because he must have developed personal memories of those events. Sure, he might loose the feelings towards wife and daughter. But surely he cannot un-remember the things that he personally did.