Evaluation Framework

Our evaluation process for applications and services attempts to address some of the common barriers to effectively evaluating privacy practices. Privacy concerns and needs vary widely based on the type of application or service and the context in which each is used. For example, it makes sense for a student assessment system to collect a home address or other personal information. However, it would not make sense for an online calculator to collect that same student’s home address or other types of personal information.

Our evaluation process pairs both a transparency evaluation with a qualitative evaluation, which provides the ability to track both which practices a policy discloses and the strengths and weaknesses of how a policy discloses that information in different contexts. Lastly, our evaluation process includes reviewer‐written summaries that highlight the implications of the application or service’s privacy practices alongside the goals and contexts within which the service may be used. These summaries aid in the interpretation of our aggregate details as well as identify any shortcomings in our evaluation process relative to an individual product.

The privacy evaluation process contains four steps:

  1. Overview: Select a product and evaluate the details of the various policies of the application or service.

  2. Triage: Answer brief observational questions not related to the policy text itself but rather relating to a superficial assessment of the vendor's privacy and security practices.

  3. Evaluation: Answer questions about whether or not the text of the policies disclose particular issues. Questions are composed of the following details:

    a. Transparency selection: Do the policies address the issue(s) raised in the question?

    b. Qualitative selection: Do the policies indicate whether or not the vendor engages in the practice described?

    c. Notes: Is there anything noteworthy, exceptional, or egregious regarding the details of the question that should be noted?

    d. Policy references: Can text within the policies be highlighted and associated with the particular question selected?

  4. Summary: Create a general summary of the application or service and describe the relevant policy details.

In addition to engaging in this evaluation process, our team also published a basic Information Security Primer. While we do not run all these additional security-related tests as part of every evaluation, it's a useful resource, and we have used this primer to support multiple products addressing security issues.

Evaluation Process

The privacy evaluation process includes questions organized into categories and sections derived from the Fair Information Practice Principles that underlie international privacy laws and regulations. In addition, the full evaluation questions and the categories that organize them are all mapped to a range of statutory, regulatory, and technical resources that provide background information on why each question is relevant to the privacy evaluation process. For example, the following evaluation question requires a reviewer to read the policies of the application or service and determine whether or not they disclose the issue raised in the question by providing a yes or no response:

Question: Do the policies clearly indicate whether or not the vendor collects personally identifiable information (PII)?

"Image of policy annotator"

If the reviewer responds yes to this question, that means the application or service discloses whether or not it collects personally identifiable information. Given a yes transparent response to this question, the reviewer is then asked a follow-up question of whether or not the application or service discloses they engage in the particular practice described. A yes or no response that personally identifiable information is, or is not, collected will determine the final question points based on whether the practices described are considered qualitatively better or worse for the purposes of our evaluation process. Note that some questions do not have a qualitative component and are purely informational. This includes both questions where there is truly no qualitative value to a response and those questions where determining if a given response is qualitatively better or worse requires additional context outside the scope of the evaluation process.

The Privacy Ratings and Evaluation Scores pages describe in more detail how responses to questions impact the overall score for an application or service.