Yup! Although I think they had little contact - Vita's only mentioned in passing in the book.

Something I forgot to put in the writeup, which was funny: in the late 19th and early 20th centuries, the accepted time for discreet extramarital encounters was between 4 and 7 PM - because the women's clothes of the day were so complicated! The lady's maid would come in at 4 to get her mistress out of her afternoon tea dress, and would return at 7 to get her into her evening dress. Hence the window of opportunity. :D