In a word...
Everything.
This is, at base, a factual dispute. Is it easy to draw sensitive inferences from phone metadata? How often do people conduct sensitive matters by phone, in a manner reflected by metadata?
We used crowdsourced data to arrive at empirical answers. Since November, we have been conducting a study of phone metadata privacy. Participants run the MetaPhone app on their Android smartphone; it submits device logs and social network information for analysis. In previous posts, we have used the MetaPhone dataset to spot relationships, understand call graph interconnectivity, and estimate the identifiability of phone numbers.
At the outset of this study, we shared the same hypothesis as our computer science colleagues—we thought phone metadata could be very sensitive. We did not anticipate finding much evidence one way or the other, however, since the MetaPhone participant population is small and participants only provide a few months of phone activity on average.
We were wrong. We found that phone metadata is unambiguously sensitive, even in a small population and over a short time window. We were able to infer medical conditions, firearm ownership, and more, using solely phone metadata.
The article, from there, goes into further detail about what they could find out from a single call, depending on who was called, and from multiple calls as well. The researchers were able to track down assault weapon owners, discover someone's major medical conditions, figure out a participant had an abortion, and more.
The authors conclude with:
The dataset that we analyzed in this report spanned hundreds of users over several months. Phone records held by the NSA and telecoms span millions of Americans over multiple years. Reasonable minds can disagree about the policy and legal constraints that should be imposed on those databases. The science, however, is clear: phone metadata is highly sensitive.
If there was every any doubt about how phone metadata records aren't "just metadata", this research out of Stanford should clear that misconception up for you. I urge all of you to contact your Congresscritters and tell them to end this unconstitutional program that infringes on our right to privacy.