It turns out that you can install TFS with almost all features except with Report Server integration on a Windows 7 machine. You really need Windows Server for that one feature to work.
Re-installing the workstation as we speak :).
It would have been nice if the Administration Console would have shown the reporting item and on the page explained that this requires Windows Server to work.
Thanks to Tarun, here's the doc that explains it:
You can install Team Foundation Server on a client computer that is running one of the operating systems in the next list. However, client operating systems do not support integration with SharePoint Products or reporting. If you want to use either of these features, you must install Team Foundation Server on a server operating system.