Sir Charles Lytton, also known as "The Notorious Phantom" and nicknamed "The Juggler" is Clouseau's archnemesis in the comedy films, and the main protagonist (or antagonist,depending on watching view) of the first film. A wealthy, ageing British playboy, he is the world's greatest catburglar, who keeps his identity a secret and leaves a calling card, a white glove marked with the letter "p". Although he is the villain of the first film, he appears as a supporting character in "The Return of the Pink Panther", "Trail of the Pink Panther" and "Curse of the Pink Panther". He is portrayed by David Niven in all of these, except for "Return of the Pink Panther", where he is played by Christopher Plummer. Lytton has been married twice, lives in a large mansion and does not smoke.

In "The Pink Panther", Clouseau has made it his life's goal to catch The Phantom. What he does not know is that his wife, Simone, is secretly Lytton's lover and partner-in-crime. Lytton generously contributes to putting his nephew, George Lytton, through college, unaware that the youth is a scoundral and is not in college at all. Clouseau, Lytton and George all land up staying at the same ski resort, where Clouseau correctly deduces The Phantom will be staying in the hopes of stealing The Pink Panther diamond from Princess Dala. Lytton and Simone continually their affair right under Clouseau's nose, while Lytton also seduces Princess Dala (by having her dog kidnapped and then resucing it), and George tries to seduce Simone. Clouseau and George both find out that Lytton is The Phantom, and at the Princess' costume party, Clouseau sets a trap. George and Lytton bump into each other, both dressed as gorillas, and both trying to steal the jewel, which has been already stolen by Princess Dala to avoid deportation. Uncle and nephew are both arrested as suspects during the ensuing car chase, but Clouseau is framed during the trial by Dala and Simone. Clouseau enjoys the attention he gets when he is believed to be The Phantom, but Simone feels sorry for him. Lytton and George are unsympathetic.

In "Return of the Pink Panther", both Clouseau and Lytton work seperately to solve the mystery of the jewel being stolen again with the glove having been left behind, albeit not by Lytton. Lytton encounters several thugs and breaks several fingers to try and find out who is framing him, before learning that it was his wife, Lady Claudine, who wanted to add some exitement to their marriage. Clouseau, meanwhile, tries to prove Lytton guilty for revenge. Colonel Sharki, the chief of the Lugash police, tries to kill the Lyttons as he had his own agenda regarding the theft of the diamond, but he is killed by Dreyfus and the diamond is recovered by Clouseau.