The 36-year-old reality star's mother Bonnie passed away after a long illness in 2013 and his father Jeff died just three months later and now his and former partner Kourtney Kardashian's kids, Mason, 10, Penelope, seven, and Reign, five, have begun asking about their grandparents, he's feeling uncomfortable.
Speaking to Kim Kardashian West on 'Keeping Up With the Kardashians', he said: "Losing my parents is not an easy subject for me to talk about, It puts me in a very vulnerable place, to remember them and think about them."
Scott called his "quintessential fake-uncle", his dad's best friend Dave Hacker, and invited him to visit.
The pair reminisced about the past when Dave came to visit and he and Scott opened huge boxes of old family photos.
However, when it proved too emotional for Scott, he left the room.
Khloe Kardashian said: "I wish Scott didn't get like this.
"I wish he would either talk about how he's feeling, or I think the more he does talk about his parents, the easier it will get. And it's fun to talk about memories. You don't want your parents to just disappear."
Scott - who is now in a relationship with Sofia Richie - eventually returned and showed Reign some of his childhood photos, and later admitted he was glad to have done so.
He reflected: "I think when my parents passed, it was just too hard to talk about and, you know, as much as I kind of was dreading looking through some of these old photos because I thought it was going to bring up sad memories, I'm happy Dave brought them with him.
"Because the truth is, it's brought up so many amazing memories and so many things I've forgot."
And he vowed to talk more to his kids about their late grandparents.
He said: "It's definitely not easy, the fact that I don't have my parents here. And I miss them a lot, but I think the more I talk about my parents, the more my kids will feel connected to them and remember them."