Evaluation Scores
Basic and Full evaluations include an overall evaluation score. A higher score (up to 100%) means the product provides more transparent and comprehensive privacy policies with better practices to protect user data. The overall score is not an average of the evaluation concern scores, but rather is a percentage of the number of points earned for basic evaluation questions. The score is best used is as an indicator of how much additional work a person will need to do to make an informed decision about a product. This use is directly related to the core work driving the evaluations: to help people make informed decisions about a service with less effort. The higher the number, the less effort required to make an informed and appropriate decision.
For each question, the score is calculated as follows:
| Score | Question Response |
|---|---|
| 0.0 | Not transparent or unclear |
| 0.5 | Transparent, but the response is qualitatively worse |
| 1.0 | Transparent, and if the question has a qualitative component, the response is qualitatively better |
Each question contributes one point to the overall possible score and a score is calculated by totalling the points earned for a given set of questions relative to the number of questions in consideration. This allows us to take any subset of questions and generate a score. As described above, a score is calculated by taking the total number of points earned and dividing by the number of questions in consideration. This provides a percentage that allows for easier interpretation across different facets of an evaluation.
For instance, our evaluation concern scores utilize 10 questions and our evaluation statue scores are calculated against the respective number of questions in each privacy law. For the overall evaluation process, "Transparency" is defined as a measure indicating, of the things we expect to know, are they discussed in a vendor's privacy policies. In addition, "Quality" is defined as a measure indicating, of those things we know about, does the vendor's disclosure about those practices protect personal information, which is considered qualitatively better.
Our privacy-evaluation process for an application or service is unique, because it produces a score based on transparency and quality, which are combined into an overall score. These two metrics allow for an objective comparison between applications and services based on how transparent their policies are in explaining their practices and the quality of those practices. Other privacy policy assessment tools have used algorithmic qualitative keyword-based contextual methods that attempt to summarize a policy's main issues. These keyword-based methods, such as the Usable Privacy Policy Project, have been found to produce reliable measures of transparency information about the key issues disclosed in an application or service's policies. However, these methods are not able to capture substantive indicators that describe the meaning or quality of those disclosures. Therefore, our privacy-evaluation process was developed with this limitation in mind to incorporate both qualitative and quantitative assessment methods to capture the differential meaning of each privacy practice disclosed in a vendor's policies with scores.
To explain how questions affect an overall score, we will look at question 3.2.4 from our published list of questions:
Do the policies clearly indicate whether or not personal information is shared with third parties for advertising or marketing purposes?
At a high level, this question has three possible responses:
-
The policies say nothing about whether or not personal information is shared with third parties for advertising or marketing purposes.
-
The policies clearly indicate that personal information is shared with third parties for advertising or marketing purposes.
-
The policies clearly indicate that personal information is not shared with third parties for advertising or marketing purposes.
The first option -- in calculating the overall score, not sharing any information brings the overall score down the most, because without any information, making a fully informed decision is not possible. The second option -- if a vendor clearly indicates that they do share personal information for marketing purposes -- still earns points toward the overall score, because the disclosure helps a person make an informed decision. Because we look at privacy through the lens of making an informed decision, we encourage transparency in policy disclosures as a needed tool to help people make informed decisions. The third option -- clearly specifying that personal information is not shared increases the overall score. If a vendor is transparent and discloses better practices that protect personal information, the overall score increases the most.
Concern Scores
The privacy evaluation process summarizes the policies of an application or service into concern categories based on a subset of evaluation questions that can be used to quickly identify particular strengths and weaknesses of a vendor’s policies. These concerns are composed of evaluation questions that can be used to calculate scores relative to that concern. The privacy Evaluation Concerns are composed of both basic and full questions. As such, a concern with only the basic questions answered is a subset of a concern with all the full questions answered, but it still identifies several critical evaluation questions for a quick comparison between products. A concern with all the full evaluation questions answered provides a more comprehensive analysis and understanding of an application or service’s policies with respect to the specific concern. In addition, the evaluation concerns are organized by two‐word question descriptions used to provide a general understanding of the topics covered by each concern. Each concern has its own concern score, which is calculated as a percentage given the number of questions in each concern.
The concerns help provide focused understanding about the different privacy‐, security‐, safety‐, and compliance‐related issues that compose a particular concern for an application or service. The concerns ultimately provide parents and teachers with more relevant information to make a more informed decision about whether to use a particular application or service based on the concerns that matter most for their kids and students.
Statute Scores
Each statute or regulation is associated with one or more evaluation questions. As such, we can calculate scores for each statute or regulation using only those questions associated with the statute or regulation. Each specific statute or regulation's score serves as an indirect proxy indicating the likelihood of the application or service satisfying all of its compliance obligations.
However, this statute or regulation score only provides an indication of how much additional work may be required to determine whether an application or service is actually in compliance with applicable federal or state law in a specific context. A score of less than 100% indicates that additional information is likely required to determine whether an application or service is compliant in all contexts. A lower overall statute score indicates that an application or service is more likely to be missing information or clarity with respect to particular details that may be pertinent in a specific context or use case. In general, lower scores indicate more work would be necessary to ensure the appropriateness of the application or service in each particular context. On the other hand, a higher score indicates that various contexts are more likely to include the necessary information to determine whether compliance is satisfied for that particular use. Each application or service's legal obligations should only be understood in the context in which it is used.
If you would like to learn more about our evaluation scores and see examples, please download our 2021 State of Kids' Privacy Report.