Full Security Assessment Questions

These following questions are the foundation of our full security assessment framework. You have several options for navigating these questions, and learning more about data security. This page contains all 80 questions that relate to the privacy and security observational practices of smart devices and mobile applications. This full security assessment integrates the Consumer Reports Digital Standard questions, Ranking Digital Rights questions, and OWASP IoT project questions into a single full security assessment framework.

1:0 Data Privacy

1.1: Privacy Policy

1.1.1: Policy Available

  • Criteria:
    • Are the policies for the specific service available (and not, for example, for the public-facing website)?
  • Indicator:
    • Assess whether the Privacy Policy and Terms of Service are made available.
    • Assess whether the Privacy Policy and Terms of Service are not specific to the vendor's website, but the service or application.

1.1.2: App Policy

  • Criteria:
    • Do Android or iOS app privacy policies link to the same privacy policy URL location as the home page policy?
  • Indicator:
    • Assess whether the mobile App store policies are the same as the policies available for the applications and services.
  • Criteria:
    • Are hyperlinks to the vendor's policies available on the homepage and labeled Privacy Policy or Terms of Use?
  • Indicator:
    • Assess whether the policies are labeled Privacy Policy and Terms of Use and easy to find.

1.1.4: Policy Accessible

  • Criteria:
    • Are the policies available in a human and machine readable format that is accessible on the web, mobile devices, screen readers or assistive technologies?
  • Indicator:
    • Assess whether the policies are provided in accessibility formats.

1.1.5: Policy Registration

  • Criteria:
    • Are the policies available on a new account registration webpage for review prior to a user creating a new account with the service or application?
  • Indicator:
    • Assess whether the policies are available on a new account registration page.

1.2: Account Type

1.2.1: Free Service

  • Criteria:
    • Can you create a free sample account with the application or service?
  • Indicator:
    • Assess whether the application or service provides a free account for use.

1.2.2: Paid Service

  • Criteria:
    • Does the application or service offer a paid or subscription based service?
  • Indicator:
    • Assess whether the application or service requires a fee or subscription to use.

1.2.3: Access Code

  • Criteria:
    • Does the application or service require the purchase of hardware or a School access code to create an account?
  • Indicator:
    • Assess whether the application or service requires an access code to use.
    • Assess whether the application or service requires the purchase of additional hardware to use.

1.2.4: In-App Purchases

  • Criteria:
    • Does the application or service offer In-App-Purchases?
  • Indicator:
    • Assess whether the application or service offers In-App-Purchases.

1.2.5: Child Account

  • Criteria:
    • Does the application or service allow children to create an account?
  • Indicator:
    • Assess whether the application or service allows children to create an account.

1.2.6: Student Account

  • Criteria:
    • Does the application or service allow a student to create an account?
  • Indicator:
    • Assess whether the application or service allows a student to create an account.

1.2.7: Parent Account

  • Criteria:
    • Does the application or service allow a parent to create an account?
  • Indicator:
    • Assess whether the application or service allows a parent to create an account.

1.2.8: Teacher Account

  • Criteria:
    • Does the application or service allow a teacher to create an account?
  • Indicator:
    • Assess whether the application or service allows a teacher to create an account.

1.2.9: Consumer Account

  • Criteria:
    • Does the application or service allow a consumer to create an account?
  • Indicator:
    • Assess whether the application or service allows a consumer to create an account.

1.3: Account Protection

1.3.1: Strong Password

  • Criteria:
    • Does the application or service require the user to create a strong passphrase or complex password?
  • Indicator:
    • Assess whether passwords are required to be of a certain complexity.
    • Assess whether passwords are required or optional to be of a certain length.
  • Procedure Overview:
    • Note if the product requires the password to be of a certain complexity (types of characters, prohibiting repeated characters, etc.).
    • Note if there are requirements for the password to be at least a certain length.
    • Note if there are limitations on how long the password can be.

1.3.2: Existing Password

  • Criteria:
    • Does the application or service require the existing password for the user in order to change the password?
  • Indicator:
    • Assess whether the existing password is needed to change the password.
  • Procedure Overview:
    • Note if the product requires knowledge of the existing password in order to execute a password change.

1.3.3: Password Recovery

  • Criteria:
    • Does the application or service provide a secure password recovery mechanism?
  • Indicator:
    • Assess the solution for a password recovery mechanism.

1.3.4: Password Expires

  • Criteria:
    • Does the application or service provide for the expiration of user passwords?
  • Indicator:
    • Assess the solution to force password expiration after a specific period.

1.3.5: Change Defaults

  • Criteria:
    • Does the application or service require the default username and password to be changed?
  • Indicator:
    • Assess the solution to change the default username and password.

1.3.6: Account Changes

  • Criteria:
    • Does the application or service provide an ability for the user to change their username and password?
  • Indicator:
    • Assess the solution to change the username and password.

1.3.7: Account Lockout

  • Criteria:
    • Does the application or service lockout an account after a specified number of failed attempts for authorization?
  • Indicator:
    • Assess the account lockout mechanism for any application or service interface.
    • Assess the account lockout time period for any application or service interface.
    • Assess the account lockout number of failed attempts for authentication.
    • Assess the account lockout mechanism for password recovery attempts.

1.3.7: Two-Factor Authentication

  • Criteria:
    • Does the application or service provide two-factor authentication for any area where authorized access is required?
  • Indicator:
    • Assess the solution for implementation of two-factor authentication.
    • Assess the solution for two-factor authentication with text messaging.
    • Assess the solution for two-factor authentication with a YubiKey.
    • Assess the solution for two-factor authentication with an application code.

1.4: Device Collection

1.4.1: Personal Information

  • Criteria:

    • Does the application or service collect Personally Identifiable Information (PII)?
  • Indicator:

    • Assess the solution to determine the amount and type of personal information collected by the application or service.

1.4.2: Audio Information

  • Criteria:

    • Does the application or service collect audio data?
  • Indicator:

    • Assess the solution to determine the amount and type of audio data collected by the application or service.

1.4.3: Photo Information

  • Criteria:

    • Does the application or service collect photographic information?
  • Indicator:

    • Assess the solution to determine the amount and type of photographic information collected by the application or service.

1.4.4: Video Information

  • Criteria:

    • Does the application or service collect video information?
  • Indicator:

    • Assess the solution to determine the amount and type of video information collected by the application or service.

1.4.5: Location Information

  • Criteria:

    • Does the application or service collect precise geolocation data?
  • Indicator:

    • Assess the solution to determine the amount and type of geolocation data collected by the application or service.

1.4.6: Health Information

  • Criteria:

    • Does the application or service collect health or biometric data?
  • Indicator:

    • Assess the solution to determine the amount and type of health information collected by the application or service.

1.4.7: Behavioral Information

  • Criteria:

    • Does the application or service collect behavioral data?
  • Indicator:

    • Assess the solution to determine the amount and type of behavorial information collected by the application or service.

1.4.8: Metadata Information

  • Criteria:

    • Does the application or service automatically collect any usage information or metadata?
  • Indicator:

    • Assess the solution to determine the amount and type of usage or metadata collected by the application or service.

1.5: Privacy Controls

1.5.1: Data Access

  • Criteria:
    • Does the application or service provide users the ability to access data in their account?
  • Indicator:
    • Assess the solution for a user to acess and revew data in their account.

1.5.2: Data Modification

  • Criteria:
    • Does the application or service provide users the ability to modify data in their account?
  • Indicator:
    • Assess the solution for a user to edit or modify data in their account.

1.5.3: Data Deletion

  • Criteria:
    • Does the application or service provide users the ability to delete data in their account?
  • Indicator:
    • Assess the solution for a user to delete data in their account.

1.5.4: Account Deletion

  • Criteria:
    • Does the application or service provide users the ability to terminate their account?
  • Indicator:
    • Assess the solution for a user to terminate their account, remove the application from a device, or perform a factory reset or erasure of the device.

1.5.5: Device Permissions

  • Criteria:
    • Does the application or service still provide functionality if all user permissions for data access are declined?
  • Indicator:
    • Assess the solution to determine whether the product still works when all permissions not relevant to the product's functionality are declined.
  • Procedure Overview:
    • Decline permissions not relevant to the product's functionality and verify that product is still functional.
    • Decline use of session cookies or javascript.

1.5.6: Default Privacy

  • Criteria:
    • Does the application or service set privacy settings or protections by default?
  • Indicator:
    • Assess whether the default settings in this product prioritize a user's privacy.
    • Assess whether targeted advertising is off by default.
    • Assess whether the transmission of user communications is encrypted by default.
    • Assess whether end-to-end encryption is enabled by default.
    • Assess whether user interface settings which are optimal for privacy are set by default.
  • Procedure Overview:
    • Review settings available from the user interface, and determine which options would be optimal for privacy considerations.
    • Determine whether those options are selected by default.
  • Criteria:
    • Does the application or service provide a choice for users to consent to provide optional information beyond what is necessary?
  • Indicator:
    • Assess the solution to ensure users are given a choice for data collection beyond what is necessary for the proper operation of the application or service.

1.5.8: Parental Controls

  • Criteria:
    • Does the application or service provide managed accounts or parental controls to provide consent?
  • Indicator:
    • Assess the solution for multi-user environments and ensure it includes functionality for parental controls and method of providing consent.

1.5.9: Child Age Gate

  • Criteria:
    • Does the application or service require a bithdate for account registration to determine if the user is under 13 years of age?
  • Indicator:
    • Assess the solution for account registration for a child under 13 years of ag and if an age gate prevents use of the product.

1.6: Data Security

1.6.1: Web Encryption

  • Criteria:
    • Does the web service use HTTPS for the homepage, login page, or pages accessed while logged in?
  • Indicator:
    • Assess the use of encryption with HTTPS for the homepage, login page, or pages accessed while logged in.
    • Assess whether the transmission of user communications is encrypted by default.
    • Assess whether the transmission of user communications is encrypted using unique keys.
    • Assess whether users can secure their content using end-to-end encryption.
    • Assess whether end-to-end encryption is enabled by default.
  • Procedure Overview:
    • Inspect traffic to determine if SSL encryption is used.

1.6.2: Encryption Required

  • Criteria:
    • Does the homepage, login page, or pages accessed while logged in force encryption back to HTTPS if changed to HTTP?
  • Indicator:
    • Assess whether encryption is required for user information transmitted during log-in, account registration, and usage if switched to HTTP.

1.6.3: Data Secure

  • Criteria:
    • Does the mobile application encrypt communication between devices and the Internet?
  • Indicator:
    • Assess the solution to determine the use of encrypted communication between devices and between devices and the Internet.

1.6.4: Storage Encryption

  • Criteria:
    • Does the mobile application encrypt data stored on the mobile device?
  • Indicator:
    • Assess the solution to determine if collected personal data is properly protected using encryption at rest.

1.6.6: Secure Cookies

  • Criteria:
    • Does the web service require persistent or session cookies to require a secure flag?
  • Indicator:
    • Assess whether the web service uses persistent or session cookies with secure flags to require encryption.

1.6.7: Security Notifications

  • Criteria:
    • Does the application or service provide notification to users for security events?
  • Indicator:
    • Assess the solution to determine if alerts and notifications to the user for security events are available.

1.6.8: Bug Bounty

  • Criteria:
    • Is the company willing and able to address reports of vulnerabilities when they are discovered?
  • Indicator:
    • Assess whether the company has a mechanism through which security researchers can submit vulnerabilities they discover.

1.7: Advertising and Tracking

1.7.1: Advertising Displayed

  • Criteria:
    • Does the application or service display advertisements?
  • Indicator:
    • Assess the solution to determine whether the service displays advertisements.

1.7.2: Advertising Requests

  • Criteria:
    • Does the application or service send advertising requests to third parties?
  • Indicator:
    • Assess the solution to determine whether the services send advertising requests to third parties.

1.7.3: Tracking Requests

  • Criteria:
    • Does the application or service send tracking requests to third parties?
  • Indicator:
    • Assess whether the application or service connects to known advertising or tracking servers or domains.

1.8: Device Security

1.8.1: Traffic Interception

  • Criteria:
    • Does the application or service allow network traffic to be intercepted or modified by man-in-the-middle attacks?
  • Indicator:
    • Assess the solution to ensure network traffic of any type cannot be intercepted, or modified by man in the middle attacks.

1.8.2: Overflow Response

  • Criteria:
    • Does the application or service fail to respond when presented with a buffer overflow, fuzzing or denial of service attack?
  • Indicator:
    • Assess the solution to ensure network services don't respond poorly to buffer overflow, fuzzing or denial of service attacks.

1.8.3 Device Exploits

  • Criteria:
    • Is the mobile product protected from known software vulnerabilities that present a danger from attackers?
  • Indicator:
    • Assess whether the mobile application is secure against known bugs and types of attacks.
  • Procedure Overview:
    • Root/jailbreak the device, configure web proxy and network traffic monitor.
    • Launch target app, create accounts, sign-in, launch any activities available from user interface.
    • Monitor communication between the application on the device and any backend services.
    • Examine file system, database, and logs on the mobile device to determine if sensitive information is stored in a way that could lead to compromise of user data.

1.8.4 Connected Exploits

  • Criteria:
    • Is any connected product protected from known software vulnerabilities that present a danger from attackers?
  • Indicator:
    • Assess whether any connected devices are secure against known bugs and types of attacks.
  • Procedure Overview:
    • Check if using latest version of software.
    • Configure web proxy and network traffic monitor.
    • Create account and sign-in to the installed "out of the box applications.
    • Launch any activities available from the user interface.
    • Monitor communication between the applications on the device, the device itself, and any backend services.
    • Examine file system, database, and logs to determine if sensitive information is stored in a way that could lead to compromise of user data.

1.8.5: Security Firewalls

  • Criteria:
    • Does the web service use firewalls to protect against unauthorized access to server ports and subnet addresses?
  • Indicator:
    • Assess whether the web service utilizes firewalls to protect server addresses.

1.8.6 Web Exploits

  • Criteria:
    • Is the web service protected from known software vulnerabilities that present a danger from attackers?
  • Indicator:
    • Assess whether the browser is secure against known bugs and attacks.
  • Procedure Overview:
    • Identify publicly known vulnerabilities for each browser.
    • Use the original proof of concept code (if known) or devise custom code where necessary, to test the browser for the issue identified in the vulnerability notice.
    • Check if the browser is now protected from the identified vulnerabilities.

1.8.7: Web Vulnerabilities

  • Criteria:
    • Does the web service respond to any XSS, SQLi or CSRF web application vulnerabilities?
  • Indicator:
    • Assess the web interface for XSS, SQLi and CSRF vulnerabilities and other web application vulnerabilities.

1.9: Physical Security

1.9.1: Limit Ports

  • Criteria:
    • Does the device limit access to physical data ports?
  • Indicator:
    • Assess the device to ensure it utilizes a minimal number of physical external ports (e.g. USB ports) on the device.

1.9.2: Disable Ports

  • Criteria:
    • Does the device disable unused physical ports?
  • Indicator:
    • Assess the device to determine if it allows for disabling of unused physical ports such as USB.

1.9.3: Firmware Access

  • Criteria:
    • Does the device allow the firmware to be accessed or modified?
  • Indicator:
    • Assess the device to ensure the device firmware cannot be accessed or modified.

1.9.4: Storage Access

  • Criteria:
    • Does the device allow data storage to be accessed or modified?
  • Indicator:
    • Assess the device to ensure the device data storage cannot be accessed or modified.

1.9.5: Device Encryption

  • Criteria:
    • Does the device encrypt data stored on the device?
  • Indicator:
    • Assess the device to ensure the device data storage is encrypted and cannot be accessed or modified.

1.10 Penetration Testing

1.10.1: Encryption Type

  • Criteria:
    • Does the mobile application encrypt data using standard industry practices and strong hash algorithms?
  • Indicator:
    • Assess the solution to determine if encryption options (e.g. Enabling AES-256 where AES-128 is the default setting) are available.
    • Assess the solution to determine if accepted encryption practices are used and if proprietary protocols are avoided.
    • Assess the solution to determine whether encryption hash algorithms such as MD5, SHA1, SHA256, or SHA512 are used.

1.10.2 Privilege Escalation

  • Criteria:
    • Does the device allow for administrative escalation of privileges?
  • Indicator:
    • Assess the device to determine if it includes the ability to limit administrative capabilities to a local interface only.

1.10.3 Runtime Injection

  • Criteria:
    • Does the application or service respond to untrusted data to an interpreter?
  • Indicator:
    • Assess the device to determine if runtime injection vulnerabilities are present.

1.10.4: Password Cracking

  • Criteria:
    • Does the application or service local password strong enough to prevent cracking attacks?
  • Indicator:
    • Assess the device to determine the relative strength of the local interface passwords to cracking attacks with Hashcat.

1.11: Build Quality

1.11.1: Best Build Practices

  • Criteria:
    • Was the software built and developed according to the industry's best practices for security?
  • Indicator:
    • Assess whether the product was built with effectively implemented safety features.
    • Assess the account lockout mechanism for any web interfaceRun static analysis software to determine what application armoring features are present.
  • Procedure Overview:
    • Are there Stack Guards, and if so, are they effectively implemented?
    • Are all safety features available in the pertinent OS enabled? (e.g., ASLR, CFI, RELRO, DEP, etc.)
    • Are those safety features well implemented and/or enabled with optimal settings? (E.g., High Entropy ASLR, rather than just Dynamic Base on Windows 10)
    • Are the binaries 32 or 64 bit?

1.11.2: Software Libraries

  • Criteria:
    • Does the software use third-party unverified functions, modules, or libraries?
  • Indicator:
    • Assess whether the software does not make use of unsafe functions or libraries.
  • Procedure Overview:
    • Pull out data from the binary that speaks to developer hygiene.
    • Do they use strcpy and other historically unsafe functions?
    • Did the developers use older, less historically safe functions, or newer, safer replacements for those functions?
    • What risks are introduced via the libraries that the binary links to, either directly or indirectly?

1.11.3: Software Complexity

  • Criteria:
    • Was the software built using unnecessary complexity?
  • Indicator:
    • Assess whether the software is overly complex.
  • Procedure Overview:
    • Pull out data from the binary that speaks to code complexity.
    • What is the branch density?
    • How many stack adjusts, function calls, etc are there?
    • How complex is the code?

1.11.4: Complexity Attacks

  • Criteria:
    • Is the software vulnerable to algorithmic exploits known to cause worst case behavior?
  • Indicator:
    • Assess whether the software is vulnerable to algorithmic complexity attacks.
  • Procedure Overview:
    • Perform modified fuzzing to determine the software's vulnerability to algorithmic complexity attacks.

1.11.5: Product Stability

  • Criteria:
    • Does the software perform reliably?
  • Indicator:
    • Assess whether the software is susceptible to crashes.
    • Assess whether if the program is forced to unexpectedly terminate, it shuts down in a safe and responsible fashion.
  • Procedure Overview:
    • Fuzz software to see if and how it crashes.
    • Under appropriate fuzz testing, what was the code coverage, number of crashes, and type(s) of crashes.
    • Are crashes exploitable, or do they simply allow a disruption of service?

1.12: Software/Firmware

1.12.1: Software Updates

  • Criteria:
    • Is the software kept protected with software updates for a clearly defined and communicated period of time (i.e., the product life cycle)?
  • Indicator:
    • Assess the device to ensure it includes update capability and can be updated quickly when vulnerabilities are discovered.
    • Assess whether the product life cycle is communicated to the potential owner before purchase.
    • Assess whether software updates are authenticated.
    • Assess whether automatic software updates are provided.
  • Procedure Overview:
    • Examine software settings and product documentation to determine if automatic software updates are enabled by default or can be enabled by the user.

1.12.2: Update Encryption

  • Criteria:
    • Does the application or service encrypt updates files transmitted to the device?
  • Indicator:
    • Assess the device to ensure it uses encrypted update files and that the files are transmitted using encryption.

1.12.3: Update Notice

  • Criteria:
    • Is the user notified software updates are available or that updates have been installed?
  • Indicator:
    • Assess whether users are notified of software updates.
  • Procedure Overview:
    • If updates are not automatic, examine software settings and product documentation to determine if the product notifies the user if a software update is available, and if that notification is persistent, or if the user can easily determine if a software update is available.

1.12.4: Update Process

  • Criteria:
    • Is the software update process simple for the user to complete?
  • Indicator:
    • Assess whether installation of software updates requires multiple steps to complete.
  • Procedure Overview:
    • Execute the procedure to install a software update and evaluate the ease of installation by comparing against established references.

1.12.5: Signed Updates

  • Criteria:
    • Does the application or service sign and validate update files transmitted to the device?
  • Indicator:
    • Assess the device to ensure is uses signed files and then validates that file before installation.

1.12.6: Automatic Updates

  • Criteria:
    • Can the user change the software update process to be automatic?
  • Indicator:
    • Assess whether software can be kept up-to-date for security issues.
  • Procedure Overview:
    • Check if a later version of software exists but the product cannot be updated to it (e.g. Android devices with pre-KitKat versions).

1.12.7: Firmware Updates

  • Criteria:
    • Is the device firmware able to be modified or updated?
  • Indicator:
    • Assess whether the firmware can be modified or updated by physical or remote access.
    • Assess whether the firmware can accept unauthorized updates.