Yes, if nothing exists after sentient life has died out, then belief in eternal life would be "an epistemic mistake." If there is no life after death, then belief in life after death would be an incorrect belief and evaluably false.
Christians believe that everyone will have cognitive experience beyond death and that everyone (Christian and non-Christian) will be "resurrected" (Acts 24:15, John 5:28-29) to stand before God, be judged by him, and either enter heaven (eternal life) or hell (eternal destruction). It is believed this will happen in bodily form, and not as magical spirit beings. Christians believe they will have "glorified bodies" in heaven.
Part of the basis for this belief as well as the glorified body belief is on the resurrection of Jesus Christ described in the Gospels (Matthew 28, Mark 16, Luke 24, John 20-21) where Jesus appeared to numerous groups of people, including his opponents and skeptics which then came to believe, a total of over 500 people over a period of 40 days where the disciples could touch his body and they ate with Him, indicating a physical body. They were fully convinced of seeing a bodily resurrection Jesus and went to their violent, bloody deaths because of their being convinced of this truth.
The rest of the basis for this belief is pretty much in the truth and reliability of the Bible as a whole and of Jesus, who talked of eternal life quite often. Christians would point to evidence from history, archeology, prophecy, and philosophy for corroborating the truth in the Bible. They would use the 3 tests for historical validity, the same for other historical documents: external evidence, internal evidence, and the bibliographic test.
Also note that Christians don't necessarily claim that "near-death experiences" (and certainly not people who claim to go to heaven/hell and come back) validate this belief. Near death, parts of the brain (e.g. the cortex) shut down, facilitating some crazy visions and feelings. While these reports may sometimes come in accord with Christian claims, they aren't really considered good evidence of an afterlife.