Surrey-born Sir Colin made his debut with the orchestra in 1959 and was its longest serving principal conductor.
He worked with the Royal Opera House, the BBC Symphony and English Chamber orchestras and was internationally renowned for his interpretations of Mozart, Sibelius and Berlioz.
Messages of condolence are flooding in to a page on the LSO's website.
The LSO said he would be remembered with "huge affection and admiration".
Antonio Pappano, music director of the Royal Opera House, described Sir Colin's death as a "huge blow".
"We had future plans with him in place but more importantly, his passing represents an end of an era, where grit, toil, vision and energy were the defining elements of a leading international opera house," he said.
"The warmth and excitement of his music-making will be terribly missed. He was a giant. A very sad moment for British music."
Roger Wright - controller BBC Radio 3 and director of BBC Proms - said Sir Colin's "death is an enormous loss to music".
"His last performance at the BBC Proms two years ago, a towering interpretation of Beethoven's Missa Solemnis, was a reminder of the special qualities of his conducting and his big-hearted and deep musicianship," he added.
Born in Weybridge on 25 September 1927, Sir Colin studied clarinet at the Royal College of Music, going on to play in the band of the Household Cavalry during his military service.
He began his conducting career as assistant conductor with the BBC Scottish Symphony Orchestra in 1957, before moving to Sadlers Wells in 1959 as principal conductor and later as musical director.
He became chief conductor of the BBC Symphony Orchestra in 1967 and music director of the Royal Opera House in 1971. Sir Colin conducted the Bavarian Radio Symphony Orchestra between 1983 and 1992.
He was the LSO's principal conductor between 1995 to 2006 and became its president in 2007.
He has also been principal guest conductor with the Boston Symphony Orchestra and New York Philharmonic and in 1990 became honorary conductor of the Dresden Staatskapelle.
Knighted in 1980, Sir Colin was awarded international honours by Denmark, Italy, France, Germany and Finland.
He became a Companion of Honour in the Queen's Birthday Honours in 2001 and received The Queen's Medal for Music in December 2009.
His live recording with the LSO of The Trojans by Berlioz won two Grammys and a Classical Brit Award in 2002 and four years later there was another Grammy for Verdi's Falstaff. He won the Classic Brit Male Artist of the Year Award twice.
In a statement, the LSO said he died after an illness.
Read more: bbc