For some bizarre reason the default behaviour of pip is to download the package every time. So try:
pip install matplotlib
and see how much fun it is to download a 32MB tarball, have one dependency fail, download again, another dependency etc. You can tell pip to stop being stupid by creating a ~/.pip/pip.conf file with:
[global]
download_cache = ~/.cache/pip
Also, the matplotlib build doesn't die on missing headers at the dependency checking stage, so if you see:
src/_png.cpp:10:20: fatal error: png.h: No such file or directory
src/ft2font.h:16:22: fatal error: ft2build.h: No such file or directory
You need:
pip install numpy
sudo apt-get install libfreetype6-dev
sudo apt-get install libpng-dev