Evaluation Scores

Every privacy rating includes an overall score. A higher score (up to 100%) means the product provides more transparent and comprehensive privacy policies with better practices to protect user data. The score is best used is as an indicator of how much additional work a person will need to do to make an informed decision about a product. This use is directly related to the core work driving the evaluations: to help people make informed decisions about a service with less effort. The higher the number, the less effort required to make an informed and appropriate decision.

For each question, the score is calculated as follows:

Score Question Response
0.0 Not transparent or unclear
0.5 Transparent, but response is qualitatively worse
1.0 Transparent, and if question has a qualitative component, the response is qualitatively better

Each question contributes one point to the overall possible score and a score is calculated by totalling the points earned for a given set of questions relative to the number of questions in consideration. This allows us to take any subset of questions and generate a score. As described above, a score is calculated by taking the total number of points earned and dividing by the number of questions in consideration. This provides a percentage that allows for easier interpretation across different facets of an evaluation. For instance, our concern category scores utilize 10 questions and statute scores are calculated against the respective number of questions in each breakdown. For the evaluation process, "Transparency" is defined as a measure indicating, of the things we expect to know, are they discussed in a vendor's privacy policies. In addition, "Quality" is defined as a measure indicating, of those things we know about, does the vendor's disclosure about those practices protect personal information, which is considered qualitatively better.

The privacy-evaluation process for an application or service is unique, because it produces a score based on transparency and quality, which are combined into an overall score. These two metrics allow for an objective comparison between applications and services based on how transparent their policies are in explaining their practices and the quality of those practices. Other privacy policy assessment tools have used algorithmic qualitative keyword-based contextual methods that attempt to summarize a policy's main issues. These keyword-based methods, such as the Usable Privacy Policy Project, have been found to produce reliable measures of transparency information about the key issues disclosed in an application or service's policies. However, these methods are not able to capture substantive indicators that describe the meaning or quality of those disclosures. Therefore, our privacy-evaluation process was developed with this limitation in mind to incorporate both qualitative and quantitative assessment methods to capture the differential meaning of each privacy practice disclosed in a vendor's policies with scores.

Score Calculations

From our 2019 State of EdTech Privacy Report, we determined a median overall score of products evaluated in 2019 of approximately 65%. This median is lower than expected, given that these applications and services are intended for children and students. The overall score is calculated based on responses to the Basic Evaluation Questions. Basic evaluation questions were selected to be a representative subset of our full evaluation question set, including all the related questions in the Privacy Ratings, which are a varying and in some cases nonrepresentative subset of the Evaluation Concerns. For example, basic evaluation questions include a subset of questions from all nine evaluation concerns, and, to a varying degree of quality, an overall score may serve as a reliable prediction of a full evaluation.

Overall Score

Compared to 2018, applications and services in 2019 indicate a 16% increase in overall median scores that indicate more transparent and qualitatively better practices across a wide range of privacy practices. In addition, since 2018, the industry has improved with greater transparency and better practices across all basic questions, as seen by scores within the second and third quartiles increasing by roughly 11%. Lastly, because the industry has significantly improved its basic privacy practices since 2018 across all evaluation concerns, outliers denoted with circles in 2019 are now considered below the range of basic industry best practices and should update their terms to reflect the better practices the industry has adopted since last year. Please keep in mind these numbers fluctuate almost daily as evaluations are added and updated, so any examples are best understood as snapshots in time.

To explain how questions affect an overall score, we will look at question 3.2.4 from our published list of questions:

Do the policies clearly indicate whether or not personal information is shared with third parties for advertising or marketing purposes?

At a high level, this question has three possible responses:

  1. The policies say nothing about whether or not personal information is shared with third parties for advertising or marketing purposes.

  2. The policies clearly indicate that personal information is shared with third parties for advertising or marketing purposes.

  3. The policies clearly indicate that personal information is not shared with third parties for advertising or marketing purposes.

The first option -- in calculating the overall score, not sharing any information brings the overall score down the most, because without any information, making a fully informed decision is not possible. The second option -- if a vendor clearly indicates that they do share personal information for marketing purposes -- still earns points toward the overall score, because the disclosure helps a person make an informed decision. Because we look at privacy through the lens of making an informed decision, we encourage transparency in policy disclosures as a needed tool to help people make informed decisions. The third option -- clearly specifying that personal information is not shared increases the overall score. If a vendor is transparent and discloses better practices that protect personal information, the overall score increases the most.

Concern Scores

The privacy evaluation process summarizes the policies of an application or service into concern categories based on a subset of evaluation questions that can be used to quickly identify particular practices of a vendor’s policies. These concerns are composed of evaluation questions that can be used to calculate scores relative to that concern. The privacy Evaluation Concerns are composed of both basic and full questions. As such, a concern with only the basic questions answered is a subset of a concern with all the full questions answered, but it still identifies several critical evaluation questions for a quick comparison between products. A concern with all the full evaluation questions answered provides a more comprehensive analysis and understanding of an application or service’s policies with respect to the specific concern. In addition, the evaluation concerns are organized by two‐word question descriptions used to provide a general understanding of the topics covered by each concern. Each concern has its own concern score, which is calculated as a percentage given the number of questions in each concern.

The concerns help provide focused understanding about the different privacy‐, security‐, safety‐, and compliance‐related issues that compose a particular concern for an application or service. The concerns ultimately provide parents and teachers with more relevant information to make a more informed decision about whether to use a particular application or service based on the concerns that matter most for their kids and students.

Statute Scores

Each statute or regulation is associated with one or more evaluation questions. As such, we can calculate scores for each statute or regulation using only those questions associated with the statute or regulation. Each specific statute or regulation's score serves as an indirect proxy indicating the likelihood of the application or service satisfying all of its compliance obligations.

However, this statute or regulation score only provides an indication of how much additional work may be required to determine whether an application or service is actually in compliance with applicable federal or state law in a specific context. A score of less than 100% indicates that additional information is likely required to determine whether an application or service is compliant in all contexts. A lower overall statute score indicates that an application or service is more likely to be missing information or clarity with respect to particular details that may be pertinent in a specific context or use case. In general, lower scores indicate more work would be necessary to ensure the appropriateness of the application or service in each particular context. On the other hand, a higher score indicates that various contexts are more likely to include the necessary information to determine whether compliance is satisfied for that particular use. Each application or service's legal obligations should only be understood in the context in which it is used.

If you would like to learn more about our evaluation scores and see examples, please download our 2019 State of EdTech Privacy Report.