Historically, the duties of the Provost have been handled by the Provost, the President, and the Vice President for Academic Affairs. Since 1912, the Provost has served as the chief academic officer of the University. The Office of the Provost is responsible for the overall quality of the University's academic programs and faculty, and is a member of all Faculties and administrative boards. In this capacity, the Provost directs the development and implementation of academic plans and policies, and supervises the work of the Faculties, departments, institutes, and research centers. The Provost decides academic appointments, leaves, salaries, recommendations for tenure, and faculty grievances. The Provost is also involved in the creation of the University's annual budget and long-range financial plans, as well as support services for academic activities.

The Office was originally established by the Trustees of the University in 1811 with the simultaneous appointments of Reverend William T. Harris as President and John Mitchell Mason as Provost. The structure was created largely as a political compromise, but did not last: the office was abolished upon Mason's departure in 1816. The office was re-established in 1912 with the appointment of William H. Carpenter, who had previously held several administrative posts under President Nicholas Murray Butler. The University announced that the Provost "would be associated with the President and the Secretary of the university in the consideration and oversight of matters of general university concern and in the preparation of general university business for consideration either by the Trustees, the University Council, or the appropriate Faculty." Upon Carpenter's retirement in 1926, the position was again left vacant. The appointment of Frank D. Fackenthal, then Secretary of the University, re-established the office in 1937, by which time was seen as a second only to that of the President. These appointments were largely consistent with President Butler's management style: a top-down structure of administrators that were well-known and personally loyal to him. Fackenthal served as Provost until his appointment as Acting President upon the retirement of President Butler in 1945, and he retired once President Eisenhower was appointed in 1948. Albert C. Jacobs briefly served as Provost from 1947-1949, but once the new presidency was established under Eisenhower, he resigned and left the University.

The University went through a major administrative reorganization shortly thereafter. In 1949, the University created a new structure of four vice presidents: Vice Presidents of the University (also commonly known as Vice President for Academic Administration or Academic Affairs), Business and Finance, Development, and Medical Affairs. This structure would serve to provide the president with support in key administrative areas. However, the first Vice President of the University, George B. Pegram, was forced to retire in 1950 due to the University's implementation of a new mandatory retirement age for administrators. At this point, the position's duties were given to Grayson Kirk, who had been appointed Provost in 1949. The two positions remained closely related until the permanent abolishment of the position of vice president in 1990.

Grayson Kirk served as Vice President and Provost until 1953, when he was made President of the University. In addition, Kirk served as Acting President from the end of 1950, when Eisenhower was granted leave to serve as commander of NATO. From 1950-1953, Kirk handled two issues of lasting importance. First, Kirk managed Columbia's response to the questions surrounding communism in academia that were raised during the McCarthy era. Second, he proposed to limit the service of lecturers on the faculty to five years. This change, adopted by the Trustees in 1952, ultimately set in motion the adoption of "up or out" rules for faculty appointments shortly thereafter.

John A. Krout succeeded Kirk as Vice President and Provost in 1953. The New York Times described his role as "second in command to the President, chief of the university's educational system and a member of all faculty and administrative boards." He had started at Columbia in 1926, and had previously served as Associate Provost and Dean of the Graduate Faculties. In 1958, Jacques Barzun was appointed as Dean of Faculties and Provost so that Krout could focus on the duties of the vice-presidency. Upon Barzun's appointment, the noted that he would "be responsible for educational administration and liaison in administrative affairs among the university's schools." This structure remained intact when Krout retired in 1962; a new vice-president, Lawrence Chamberlain, was immediately appointed to succeed him. During his tenure, Chamberlain had direct responsibility for the capital gifts campaign, community relations, and student health. He was closely involved with Morningside Heights, Inc., and was instrumental in developing plans to deal with tenants evicted from Columbia-owned buildings and the larger issue of neighborhood conflict resolution. The University's physical expansion and its proposal to build a new gym in Morningside Park continued to strain its relationship with the surrounding neighborhood throughout the 1960s.

The 1960s presented tremendous challenges for the University. While the University had many positive achievements, it also had larger problems. After Butler's administration, the administrative structure was decentralized in a move away from his management style. In addition, the University began to experience financial difficulties by the mid-1960s. To combat this problem, the University launched a major capital campaign in 1965. But by 1967, a major administrative change proved necessary. While President Kirk remained in office, several administrators, including Barzun and Chamberlain, resigned in June 1967. David B. Truman was appointed to be Vice President and Provost. Truman, the Dean of Columbia College, was popular with students, faculty, and alumni. It was thought combining the duties of the two positions would create stronger leadership and help the administration to deal with new operational and fundraising challenges. But the events of 1968 forced additional changes. By early 1969, both Kirk and Truman had resigned and left the University. The administration remained in flux until the installation of President McGill in 1970. Vice President and Provost William Theodore de Bary was appointed in 1971. McGill attempted to calm student protests and neighborhood tensions early in his presidency, and much of the rest of his tenure was spent attempting to deal with the University's financial issues. His administration, and that of de Bary's, were marked by conflict with deans and other administrators in efforts to work through these problems. De Bary resigned in 1978, and McGill left the presidency in 1980.

After the installation of President Michael Sovern in 1980, the University restructured the Office of the Provost. The University created a "tripartite provostship" in which there were three provosts; one each for Health Sciences, Arts & Sciences, and the Morningside Professional Schools. This structure did not work, and the University returned to a single provost, Robert F. Goldberger, in 1982. The 1980s saw a shift in academic priorities. Professional schools rose in importance, as arts and sciences dealt with lower staffing, enrollments, and budgets. Goldberger was instrumental in the University's efforts to use its biomedical research in the corporate world, and helped to establish the science and technology development office and a research center at Audubon Park. This shift caused tension within the University, and contributed to Sovern's departure in the early 1990s.

Provost Jonathan Cole was appointed Vice President of Arts and Sciences in 1987. He became Provost in 1989 after the departure of Goldberger, and remained provost under President George Rupp; his tenure was the second longest of all provosts. Cole was instrumental in developing a long-range plan for the University in 1991, and he was responsible for upgrading academic facilities and support services, departments, health sciences, and the quality of faculty, teaching, research, and students during his tenure as provost. He also started new programs and initiatives to improve research and teaching, including collaborative projects. This was not without controversy: efforts to recruit world-class faculty tightened tenure standards, which frustrated some departments, and led to the creation of a private elementary school, which upset advocates for the local public school system. A collaborative project to create an educational website, Fathom.com, was also criticized and the site ultimately went out of business in 2003. But on the whole, Cole is widely recognized and praised as having been instrumental in the positive academic transformation that took place at Columbia University during the 1990s.

President Lee Bollinger appointed Alan Brinkley as Provost in 2003. Bollinger also redefined the position; he took on direct management of Health Sciences, and also planned to take a more direct role in Academic Affairs. Brinkley came from the History Department, which he had joined in 1991, and had chaired since 2000. As provost, Brinkley focused on academic initiatives that included the increasing the size of the faculty, launching a new science building, reviewing undergraduate education, and increasing the globalization of the University. He also created a Vice Provost for Diversity Initiatives, and was involved in the development of the Office of Work/Life. Brinkley returned to teaching and research in 2009. His successor, Claude M. Steele, came to the University from Stanford, where he had chaired the psychology department (1997-2000), and directed the Center for Advanced Study in the Behavioral Sciences (2002-2009). He began his tenure as Provost in September 2009.

The following list identifies the Provosts of the University, as well as their titles and dates of service:

1811–1816 - John M. Mason, Provost

1816 - Office abolished

1912–1927 - William H. Carpenter, Provost

1927–1937 - Position vacant. Milton Del Manzo served as Provost for Summer Sessions during this period.

1937–1946 - Frank Diehl Fackenthal, Provost

1946–1947 - Position vacant. Dr. Fackenthal served as Acting President.

1947–1949 - Albert C. Jacobs, Provost

1949–1953 - Grayson L. Kirk, Provost

1953–1958 - John A. Krout, Provost

1958–June 30, 1967 - Jacques Barzun, Dean of Faculties and Provost

July 1, 1967–March 14, 1969 - David B. Truman, Vice President and Provost

March 15-August 31, 1969 - Paul D. Carter, Provost, and Polykarp Kusch, Vice President and Dean of Faculties

1969–1970 - Peter B. Kenen, Provost, and Polykarp Kusch, Vice President and Dean of Faculties

1970–1971 - Polykarp Kusch, Executive Vice President for Academic Affairs and Provost

1971–1978 - Wm. Theodore de Bary, Executive Vice President for Academic Affairs and Provost

1978–1979 - Norman N. Mintz, Acting Provost

1979–1980 - Michael I. Sovern, Executive Vice President for Academic Affairs and Provost

1980–1981 - Henrik H. Bendixen, Acting Provost (Health Sciences) and Acting Vice President for Health Sciences, and Peter Likins, Provost (Morningside Professional Schools), and Fritz Stern, Provost (Arts and Sciences)

1981–1982 - Robert F. Goldberger, Provost (Health Sciences) and Vice President for Health Sciences, and Peter Likins, Provost (Morningside Professional Schools), and Fritz Stern, Provost (Arts and Sciences)

1982–1983 - Robert F. Goldberger, Provost (Health Sciences and Morningside Professional Schools) and Vice President for Health Sciences, and Fritz Stern, Provost (Arts and Sciences)

1983–1989 - Robert F. Goldberger, Provost

1987 - Fritz Stern, Acting Provost (Robert F. Goldberger served as Acting President)

1989–1994 - Jonathan R. Cole, Provost

1994–2003 - Jonathan R. Cole, Provost and Dean of Faculties

2003–2009 - Alan Brinkley, Provost

2009–2011 - Claude M. Steele, Provost

2011–2019 - John Coatsworth, Provost (Interim 2011-2012)

2019-2021 - Ira Katznelson, Interim Provost

2021-2023 - Mary Boyce, Provost

2023-2024 - Dennis Mitchell, Interim Provost

2024- - Angela V. Olito, Provost