Tag Archives: MBA Rankings

False precision in MBA rankings

I have finally submitted school data to the various MBA rankings surveys that we will participate in this year. Over the last three years, I have noticed how, with each passing iteration, the publications that run these rankings ask for more and more data points.

 

Take for example, the Business Week MBA survey. It is held every two years and asks more than 100 questions and that’s just for the MBA survey. In addition, I had to fill in another 30 odd data points for their General School survey.

 

Much of the data that is asked for has little relevance to a school such as Cambridge. For example, there is an entire matrix to be filled out about the different states that our US students come from, and which states in the US our alums work in after they graduate. This doesn’t make sense to a non-US school like us especially when we might only have 20 American students in each class. There are also several matrix tables of salary data that I had to fill in. BW asked for mean, median, low and high salaries for our graduates by job function, industry sector, and geography. There is a total of 200 cells to fill in for the salary data alone, and that just accounts for 5 of the 100 questions.

 

The statistically-minded among you will be asking yourself what algorithm could combine so much disparate data into a score that can be used to rank different schools. And the simple answer is that none of this information matters in the final analysis because Business Week does not use any of the data that the school provides. 45% of the Business Week ranking is determined by student survey responses, 45% from a survey of recruiters, and 10% from the number of publications, books and articles published by the school’s faculty.

 

If that’s the case, why collect all this data? Business Week needs to make a return on their investment in the rankings and parsing the school data in different ways can provide material for a stream of articles that will attract more and more visitors and hits on their website. In the runup to the publication of its rankings, Business Week has already posted several articles using the school data to rank top schools by salary, and even schools that have the highest percentage of students employed by specific companies. Data is also posted on a school’s profile page in Business Week’s MBA section.

 

I have no problem with this approach, and in this current age of online transparency, one could pose the question whether there is any problem with so much information other than a few overworked business school staff. My worry is about how this flood of data produces the illusion of certainty in an inherently uncertain and fuzzy world.

 

When all this school data is presented alongside rankings that don’t use the data, people might get the impression that the rankings are based on factual data and not look at the underlying data and methodology of the ranking survey. Everyone would benefit from learning the response rate from recruiters and students to see how reflective the results are. As anyone who has run a survey before, it would also be useful to know how the survey is constructed as survey design can play a very large part in determining responses.

 

But there is a deeper issue at play. Which is whether it makes sense at all to rank one school as being better than another in such a one-dimensional sense as rankings. John Kay, in his book Obliquity, draws a parallel with Peter Weir’s film Dead Poets Society where Robin Williams plays a teacher expected to teach poetry based on a text written by Dr J Evans Pritchard. Pritchard has a theory that the greatness of a poem is the product of its importance and its perfection. In one of the film’s most memorable scenes, Williams incites his class to tear off the pages from Pritchard’s book and gets them to truly appreciate the beauty of literature.

 

Kay makes the good point that it is not unreasonable to ask what are the characteristics of a great poem, but one is asking the wrong question whether Keats’s ‘On First Looking into Chapman’s Homer’ is a greater poem than Whitman’s ‘O Captain, My Captain.’ According to Kay, “the goals of education are known but the quest for clear prioritisation of the incommensurable components of education misconceived.”

 

While I am not advocating that everyone takes inspiration from Robin Williams and tears up the pages of the next MBA ranking, I would encourage people to ask what is the objective(s) of an MBA education (and it is still an education). That’s a more enriching discussion than why alums in the East Coast from one school has a higher median salary than alums in the Pacific Northwest from another school.