Joe and Frances married on March 24, 1963 but separated during the 1980's. According to King County Property Tax records, Joe gave the house to Frances on Valentine's Day, February 14, 1985 in a settlement. Joe took all of his possessions except for the baby grand piano that he had moved from Detroit. They were officially divorced on September 24, 2002.
Joe married Virginia on Valentine's Day, February 14, 2003. They had first met in the 1970's when Joe was teaching inmates at the Black Academy of Music and Virginia was a social worker working with prisoners.
Virginia was the person who took Joe to the hospital when, on his birthday, he had his first heart attack in 2001. Joe died in the living room of her home in 2008. In addition to inheriting all of his papers, records, recordings, equipment, and musical instruments, she also inherited his debts. Virginia was the person who organized his home going celebration.