"Don't think I can go back to my world or I'd take you up on the offer."

And even if a door does present itself the way that it has for Raleigh, Chuck isn't sure what'll happen if he tries to cross that threshold. He's not alive in that world, he's certain. Maybe he'd just vaporize.