02 - Data from the Web

Deadline

Wednesday October 25, 2017 at 11:59PM

Important Notes

  • Make sure you push on GitHub your Notebook with all the cells already evaluated (i.e., you don’t want your colleagues to generate unnecessary Web traffic during the peer review)
  • Don’t forget to add a textual description of your thought process, the assumptions you made, and the solution you plan to implement!
  • Please write all your comments in English, and use meaningful variable names in your code.

Background

In this homework we will extract interesting information from www.topuniversities.com and www.timeshighereducation.com, two platforms that maintain a global ranking of worldwide universities. This ranking is not offered as a downloadable dataset, so you will have to find a way to scrape the information we need! You are not allowed to download manually the entire ranking – rather you have to understand how the server loads it in your browser. For this task, Postman with the Interceptor extension can help you greatly. We recommend that you watch this brief tutorial to understand quickly how to use it.

Assignment

  1. Obtain the 200 top-ranking universities in www.topuniversities.com (ranking 2018). In particular, extract the following fields for each university: name, rank, country and region, number of faculty members (international and total) and number of students (international and total). Some information is not available in the main list and you have to find them in the details page. Store the resulting dataset in a pandas DataFrame and answer the following questions:
    • Which are the best universities in term of: (a) ratio between faculty members and students, (b) ratio of international students?
    • Answer the previous question aggregating the data by (c) country and (d) region.

Plot your data using bar charts and describe briefly what you observed.

  1. Obtain the 200 top-ranking universities in www.timeshighereducation.com (ranking 2018). Repeat the analysis of the previous point and discuss briefly what you observed.

  2. Merge the two DataFrames created in questions 1 and 2 using university names. Match universities’ names as well as you can, and explain your strategy. Keep track of the original position in both rankings.

  3. Find useful insights in the data by performing an exploratory analysis. Can you find a strong correlation between any pair of variables in the dataset you just created? Example: when a university is strong in its international dimension, can you observe a consistency both for students and faculty members?

  4. Can you find the best university taking in consideration both rankings? Explain your approach.

Hints:

  • Keep your Notebook clean and don’t print the verbose output of the requests if this does not add useful information for the reader.
  • In case of tie, use the order defined in the webpage.