Sure, your concern is very real, but let's try our hand at sidestepping a marketing trap.
One of the most profound concepts of marketing is to try to convince people that "X Corp's Subset is the only desirable Subset of the overall Set of Products/Services."
So right now we're unnerved at both Google and Apple and maybe even Microsoft if they decide to issue one of these glasses. But it's the Set of AR glasses that I absolutely believe is (part of) the future of computing. So I think I'd trust a company like maybe Samsung, who isn't on my radar of Evil Companies (correct me if they need to be) just making an platform-agnostic set of hardware AR glasses with adaptors to all the phones.
So then for me the question becomes "knowing the very real data sales issues of Google and Apple, if it takes that kind of money to kick this into top gear, then social fashion progress (avoiding the laugh-at-the-nerd factor because "oh, it's okay NOW that *I* do it" might be the TRUE trade off that personal data. Then we just use the Linux mentality and go off the Google-Apple grid.