Full Security Assessment Questions

These following questions are the foundation of our full security assessment framework. You have several options for navigating these questions, and learning more about data security. This page contains all 80 questions that relate to the privacy and security observational practices of smart devices and mobile applications. This full security assessment integrates the Consumer Reports Digital Standard questions, Ranking Digital Rights questions, and OWASP IoT project questions into a single full security assessment framework.

1:0 Data Privacy

1.1: Privacy Policy

1.1.1: Policy Available

  • Criteria:
    • Are the policies for the specific service available (and not, for example, for the public-facing website)?
  • Indicator:
    • Assess whether the Privacy Policy and Terms of Service are made available.
    • Assess whether the Privacy Policy and Terms of Service are not specific to the vendor's website, but the service or application.

1.1.2: App Policy

  • Criteria:
    • Do Android or iOS app privacy policies link to the same privacy policy URL location as the home page policy?
  • Indicator:
    • Assess whether the mobile App store policies are the same as the policies available for the applications and services.
  • Criteria:
    • Are hyperlinks to the vendor's policies available on the homepage and labeled Privacy Policy or Terms of Use?
  • Indicator:
    • Assess whether the policies are labeled Privacy Policy and Terms of Use and easy to find.

1.1.4: Policy Accessible

  • Criteria:
    • Are the policies available in a human and machine readable format that is accessible on the web, mobile devices, screen readers or assistive technologies?
  • Indicator:
    • Assess whether the policies are provided in accessibility formats.

1.1.5: Policy Purchase

  • Criteria:
    • Are the policies available on all product purchase or acquisition web pages?
  • Indicator:
    • Assess whether the policies are available on all product pages.

1.1.6: Policy Registration

  • Criteria:
    • Are the policies available on a new account registration webpage for review prior to a user creating a new account with the service or application?
  • Indicator:
    • Assess whether the policies are available on a new account registration page.

1.2: Account Type

1.2.1: Free Service

  • Criteria:
    • Can you create a free sample account with the application or service?
  • Indicator:
    • Assess whether the application or service provides a free account for use.

1.2.2: Paid Service

  • Criteria:
    • Does the application or service offer a paid or subscription based service?
  • Indicator:
    • Assess whether the application or service requires a fee or subscription to use.

1.2.3: Access Code

  • Criteria:
    • Does the application or service require the purchase of hardware or a School access code to create an account?
  • Indicator:
    • Assess whether the application or service requires an access code to use.
    • Assess whether the application or service requires the purchase of additional hardware to use.

1.2.4: In-App Purchases

  • Criteria:
    • Does the application or service offer In-App-Purchases?
  • Indicator:
    • Assess whether the application or service offers In-App-Purchases.

1.3: Data Privacy

1.3.1: Data Collection

  • Criteria:
    • Does the application or service collect the same amount and type of information specified in the policies?
  • Indicator:
    • Assess the solution to determine the amount and type of personal and non-personal information collected by the application or service.
  • Criteria:
    • Does the application or service provide a choice for users to consent to provide optional information beyond what is necessary?
  • Indicator:
    • Assess the solution to ensure users are given a choice for data collection beyond what is necessary for the proper operation of the application or service.

1.3.3: Data De-identified

  • Criteria:
    • Does the application or service de-identify or anonymize collected information?
  • Indicator:
    • Assess the solution to determine if data is de-identified or anonymized.

1.3.4: Account Deletion

  • Criteria:
    • Does the application or service provide users the ability to terminate their account?
  • Indicator:
    • Assess the solution for a user to terminate their account, remove the application from a device, or perform a factory reset or erasure of the device.

1.3.5: Data Retention

  • Criteria:
    • Does the application or service retain user information after it has been deleted or after the the account has been terminated?
  • Indicator:
    • Assess the solution to determine whether any information from a user's account is still accessible after deletion.

1.3.6: Data Permissions

  • Criteria:
    • Does the application or service still provide functionality if all user permissions for data access are declined?
  • Indicator:
    • Assess the solution to determine whether the product still works when all permissions not relevant to the product's functionality are declined.
  • Procedure Overview:
    • Decline permissions not relevant to the product's functionality and verify that product is still functional.
    • Decline use of session cookies or javascript.

1.3.7: Default Privacy

  • Criteria:
    • Does the application or service set privacy settings or protections by default?
  • Indicator:
    • Assess whether the default settings in this product prioritize a user's privacy.
    • Assess whether targeted advertising is off by default.
    • Assess whether the transmission of user communications is encrypted by default.
    • Assess whether end-to-end encryption is enabled by default.
    • Assess whether user interface settings which are optimal for privacy are set by default.
  • Procedure Overview:
    • Review settings available from the user interface, and determine which options would be optimal for privacy considerations.
    • Determine whether those options are selected by default.

1.4: Web Interface

1.4.1: Weak Passwords

  • Criteria:
    • Does the web service allow weak passwords to be created?
  • Indicator:
    • Assess any web interface to determine if weak passwords are allowed on account creation.
    • Assess any web interface to determine if strong passwords are allowed to be changed to weak passwords.

1.4.2: Strong Passwords

  • Criteria:
    • Does the web service require strong passwords to be created?
  • Indicator:
    • Assess the solution to require strong passwords where authentication is needed.

1.4.3: Password Recovery

  • Criteria:
    • Does the web service provide a secure password recovery mechanism?
  • Indicator:
    • Assess the solution for a password recovery mechanism.

1.4.4 Password Expires

  • Criteria:
    • Does the web service provide for the expiration of user passwords?
  • Indicator:
    • Assess the solution to force password expiration after a specific period.

1.4.5: Change Defaults

  • Criteria:
    • Does the web service require the default username and password to be changed?
  • Indicator:
    • Assess the solution to change the default username and password.

1.4.6: Account Changes

  • Criteria:
    • Does the web service provide an ability for the user to change their username and password?
  • Indicator:
    • Assess the solution to change the web interface username and password.

1.4.7: Account Lockout

  • Criteria:
    • Does the web service lockout an account after a specified number of failed attempts for authorization?
  • Indicator:
    • Assess the account lockout mechanism for any web interface.
    • Assess the account lockout time period for any web interface.
    • Assess the account lockout number of failed attempts for authentication.
    • Assess the account lockout mechanism for password recovery attempts.

1.4.8: Multi-user Roles

  • Criteria:
    • Does the web service provide managed accounts or multi-user roles with separate permissions?
  • Indicator:
    • Assess the solution for multi-user environments and ensure it includes functionality for role separation.

1.5: Web Security

1.5.1: Web Encryption

  • Criteria:
    • Does the web service use HTTPS for the homepage, login page, or pages accessed while logged in?
  • Indicator:
    • Assess the use of encryption with HTTPS for the homepage, login page, or pages accessed while logged in.
    • Assess whether the transmission of user communications is encrypted by default.
    • Assess whether the transmission of user communications is encrypted using unique keys.
    • Assess whether users can secure their content using end-to-end encryption.
    • Assess whether end-to-end encryption is enabled by default.
  • Procedure Overview:
    • Inspect traffic to determine if SSL encryption is used.

1.5.2: Encryption Required

  • Criteria:
    • Does the homepage, login page, or pages accessed while logged in force encryption back to HTTPS if changed to HTTP?
  • Indicator:
    • Assess whether encryption is required for user information transmitted during log-in, account registration, and usage if switched to HTTP.

1.5.3: Two-Factor Authentication

  • Criteria:
    • Does the web service provide two-factor authentication for any area where authorized access is required?
  • Indicator:
    • Assess the solution for implementation of two-factor authentication.
    • Assess the solution for two-factor authentication with text messaging.
    • Assess the solution for two-factor authentication with a YubiKey.
    • Assess the solution for two-factor authentication with an application code.

1.5.4: Secure Cookies

  • Criteria:
    • Does the web service require persistent or session cookies to require a secure flag?
  • Indicator:
    • Assess whether the web service uses persistent or session cookies with secure flags to require encryption.

1.5.5: Security Firewalls

  • Criteria:
    • Does the web service use firewalls to protect against unauthorized access to server ports and subnet addresses?
  • Indicator:
    • Assess whether the web service utilizes firewalls to protect server addresses.

1.5.6 Web Exploits

  • Criteria:
    • Is the web service protected from known software vulnerabilities that present a danger from attackers?
  • Indicator:
    • Assess whether the browser is secure against known bugs and attacks.
  • Procedure Overview:
    • Identify publicly known vulnerabilities for each browser.
    • Use the original proof of concept code (if known) or devise custom code where necessary, to test the browser for the issue identified in the vulnerability notice.
    • Check if the browser is now protected from the identified vulnerabilities.

1.5.7: Web Vulnerabilities

  • Criteria:
    • Does the web service respond to any XSS, SQLi or CSRF web application vulnerabilities?
  • Indicator:
    • Assess the web interface for XSS, SQLi and CSRF vulnerabilities and other web application vulnerabilities.

1.6: Mobile Interface

1.6.1: Weak Passwords

  • Criteria:
    • Does the mobile application allow weak passwords to be created?
  • Indicator:
    • Assess the mobile interface to ensure it disallows weak passwords.

1.6.2: Strong Passwords

  • Criteria:
    • Does the mobile application require strong passwords to be created?
  • Indicator:
    • Assess the mobile interface to determine if the option to require strong passwords is available.

1.6.3: Password Recovery

  • Criteria:
    • Does the mobile application provide a secure password recovery mechanism?
  • Indicator:
    • Assess the mobile interface for password recovery mechanisms.

1.6.4: Password Expires

  • Criteria:
    • Does the mobile application provide for the expiration of user passwords?
  • Indicator:
    • Assess the mobile interface to determine if the option to force password expiration after a specific period is available.

1.6.5: Change Defaults

  • Criteria:
    • Does the mobile application require the default username and password to be changed?
  • Indicator:
    • Assess the mobile interface to determine if the option to change the default username and password is available.

1.6.6: Account Changes

  • Criteria:
    • Does the mobile application provide an ability for the user to change their username and password?
  • Indicator:
    • Assess the ability to change the mobile application username and password.

1.6.7: Account Lockout

  • Criteria:
    • Does the mobile application lockout an account after a specified number of failed attempts?
  • Indicator:
    • Assess the mobile interface to ensure it includes an account lockout mechanism.
    • Assess the account lockout time period for any mobile interface.
    • Assess the account lockout number of failed attempts for authentication.
    • Assess the account lockout mechanism for password recovery attempts.

1.6.8: Multi-user Roles

  • Criteria:
    • Does the mobile application provide managed accounts or multi-user roles with separate permissions?
  • Indicator:
    • Assess the solution for multi-user environments and ensure it includes functionality for role separation.

1.7: Mobile Security

1.7.1: App Encryption

  • Criteria:
    • Does the mobile application encrypt communication between devices and the Internet?
  • Indicator:
    • Assess the solution to determine the use of encrypted communication between devices and between devices and the Internet.

1.7.2: Storage Encryption

  • Criteria:
    • Does the mobile application encrypt data stored on the mobile device?
  • Indicator:
    • Assess the solution to determine if collected personal data is properly protected using encryption at rest.

1.7.3: Two-Factor Authentication

  • Criteria:
    • Does the mobile application provide two-factor authentication for any area where authorized access is required?
  • Indicator:
    • Assess the solution for implementation of two-factor authentication.
    • Assess the mobile interface to determine if it Implements two-factor authentication (e.g Apple's Touch ID).
    • Assess the solution for two-factor authentication with text messaging.
    • Assess the solution for two-factor authentication with a YubiKey.
    • Assess the solution for two-factor authentication with an application code.

1.7.4 Device Exploits

  • Criteria:
    • Is the mobile product protected from known software vulnerabilities that present a danger from attackers?
  • Indicator:
    • Assess whether the mobile application is secure against known bugs and types of attacks.
  • Procedure Overview:
    • Root/jailbreak the device, configure web proxy and network traffic monitor.
    • Launch target app, create accounts, sign-in, launch any activities available from user interface.
    • Monitor communication between the application on the device and any backend services.
    • Examine file system, database, and logs on the mobile device to determine if sensitive information is stored in a way that could lead to compromise of user data.

1.7.5 Connected Exploits

  • Criteria:
    • Is any connected product protected from known software vulnerabilities that present a danger from attackers?
  • Indicator:
    • Assess whether any connected devices are secure against known bugs and types of attacks.
  • Procedure Overview:
    • Check if using latest version of software.
    • Configure web proxy and network traffic monitor.
    • Create account and sign-in to the installed "out of the box applications.
    • Launch any activities available from the user interface.
    • Monitor communication between the applications on the device, the device itself, and any backend services.
    • Examine file system, database, and logs to determine if sensitive information is stored in a way that could lead to compromise of user data.

1.8: Network Security

1.8.1: Traffic Interception

  • Criteria:
    • Does the application or service allow network traffic to be intercepted or modified by man-in-the-middle attacks?
  • Indicator:
    • Assess the solution to ensure network traffic of any type cannot be intercepted, or modified by man in the middle attacks.

1.8.2: Overflow Response

  • Criteria:
    • Does the application or service fail to respond when presented with a buffer overflow, fuzzing or denial of service attack?
  • Indicator:
    • Assess the solution to ensure network services don't respond poorly to buffer overflow, fuzzing or denial of service attacks.

1.8.3: Test Ports

  • Criteria:
    • Does the device provide network test ports used for diagnostics?
  • Indicator:
    • Assess the solution to ensure network test ports are are not present.

1.9: Security Configurability

1.9.1: Password Options

  • Criteria:
    • Does the application or service provide account or password security options?
  • Indicator:
    • Assess the solution to determine if password security options (e.g. Enabling 20 character passwords or enabling two-factor authentication) are available.

1.9.2: Password Length

  • Criteria:
    • Does the software require the user to create a password of certain length?
  • Indicator:
    • Assess whether passwords are required or optional to be of a certain length.
  • Procedure Overview:
    • Note if there are requirements for the password to be at least a certain length.
    • Note if there are limitations on how long the password can be.

1.9.3: Password Complexity

  • Criteria:
    • Does the software require the user to create a complex password?
  • Indicator:
    • Assess whether passwords are required to be of a certain complexity.
  • Procedure Overview:
    • Note if the product requires the password to be of a certain complexity (types of characters, prohibiting repeated characters, etc.)

1.9.4: Existing Password

  • Criteria:
    • Does the software require the existing password for the user in order to change the password?
  • Indicator:
    • Assess whether the existing password is needed to change the password.
  • Procedure Overview:
    • Note if the product requires knowledge of the existing password in order to execute a password change.

1.9.5: Security Logging

  • Criteria:
    • Does the application or service log security related events?
  • Indicator:
    • Assess the solution to determine if logging for security events is available.

1.9.6: Security Notifications

  • Criteria:
    • Does the application or service provide notification to users for security events?
  • Indicator:
    • Assess the solution to determine if alerts and notifications to the user for security events are available.

1.9.7: Bug Bounty

  • Criteria:
    • Is the company willing and able to address reports of vulnerabilities when they are discovered?
  • Indicator:
    • Assess whether the company has a mechanism through which security researchers can submit vulnerabilities they discover.

1.10: Physical Security

1.10.1: Limit Ports

  • Criteria:
    • Does the device limit access to physical data ports?
  • Indicator:
    • Assess the device to ensure it utilizes a minimal number of physical external ports (e.g. USB ports) on the device.

1.10.2: Disable Ports

  • Criteria:
    • Does the device disable unused physical ports?
  • Indicator:
    • Assess the device to determine if it allows for disabling of unused physical ports such as USB.

1.10.3: Unintended Access

  • Criteria:
    • Does the device allow for unintended port or network access?
  • Indicator:
    • Assess the device to determine if it can be accessed via unintended methods such as through an unnecessary USB port.

1.10.4: Firmware Access

  • Criteria:
    • Does the device allow the firmware to be accessed or modified?
  • Indicator:
    • Assess the device to ensure the device firmware cannot be accessed or modified.

1.10.5: Storage Access

  • Criteria:
    • Does the device allow data storage to be accessed or modified?
  • Indicator:
    • Assess the device to ensure the device data storage cannot be accessed or modified.

1.10.6: Device Encryption

  • Criteria:
    • Does the device encrypt data stored on the device?
  • Indicator:
    • Assess the device to ensure the device data storage is encrypted and cannot be accessed or modified.

1.11 Penetration Testing

1.11.1: Encryption Type

  • Criteria:
    • Does the mobile application encrypt data using standard industry practices and strong hash algorithms?
  • Indicator:
    • Assess the solution to determine if encryption options (e.g. Enabling AES-256 where AES-128 is the default setting) are available.
    • Assess the solution to determine if accepted encryption practices are used and if proprietary protocols are avoided.
    • Assess the solution to determine whether encryption hash algorithms such as MD5, SHA1, SHA256, or SHA512 are used.

1.11.2 Privilege Escalation

  • Criteria:
    • Does the device allow for administrative escalation of privileges?
  • Indicator:
    • Assess the device to determine if it includes the ability to limit administrative capabilities to a local interface only.

1.11.3 Runtime Injection

  • Criteria:
    • Does the application or service respond to untrusted data to an interpreter?
  • Indicator:
    • Assess the device to determine if runtime injection vulnerabilities are present.

1.11.4: Password Cracking

  • Criteria:
    • Does the application or service local password strong enough to prevent cracking attacks?
  • Indicator:
    • Assess the device to determine the relative strength of the local interface passwords to cracking attacks with Hashcat.

1.12: Advertising and Tracking

1.12.1: Traditional Advertising

  • Criteria:
    • Does the application or service display traditional advertisements based on webpage context?
  • Indicator:
    • Assess the solution to determine whether the service displays traditional advertisements.

1.12.2: Behavioral Advertising

  • Criteria:
    • Does the application or service provide targeted advertising to a user based on collected information?
  • Indicator:
    • Assess the solution to determine whether the services provide targeted advertisements.

1.12.3: Third-party Trackers

  • Criteria:
    • Does the application or service use trackers on its homepage, registration page, or while a user is logged-in?
  • Indicator:
    • Assess whether the application or service connects to known advertising or tracking servers or domains.

2.0: Data Security

2.1: Build Quality

2.1.1: Best Build Practices

  • Criteria:
    • Was the software built and developed according to the industry's best practices for security?
  • Indicator:
    • Assess whether the product was built with effectively implemented safety features.
    • Assess the account lockout mechanism for any web interfaceRun static analysis software to determine what application armoring features are present.
  • Procedure Overview:
    • Are there Stack Guards, and if so, are they effectively implemented?
    • Are all safety features available in the pertinent OS enabled? (e.g., ASLR, CFI, RELRO, DEP, etc.)
    • Are those safety features well implemented and/or enabled with optimal settings? (E.g., High Entropy ASLR, rather than just Dynamic Base on Windows 10)
    • Are the binaries 32 or 64 bit?

2.1.2: Software Libraries

  • Criteria:
    • Does the software use third-party unverified functions, modules, or libraries?
  • Indicator:
    • Assess whether the software does not make use of unsafe functions or libraries.
  • Procedure Overview:
    • Pull out data from the binary that speaks to developer hygiene.
    • Do they use strcpy and other historically unsafe functions?
    • Did the developers use older, less historically safe functions, or newer, safer replacements for those functions?
    • What risks are introduced via the libraries that the binary links to, either directly or indirectly?

2.1.3: Software Complexity

  • Criteria:
    • Was the software built using unnecessary complexity?
  • Indicator:
    • Assess whether the software is overly complex.
  • Procedure Overview:
    • Pull out data from the binary that speaks to code complexity.
    • What is the branch density?
    • How many stack adjusts, function calls, etc are there?
    • How complex is the code?

2.1.4: Complexity Attacks

  • Criteria:
    • Is the software vulnerable to algorithmic exploits known to cause worst case behavior?
  • Indicator:
    • Assess whether the software is vulnerable to algorithmic complexity attacks.
  • Procedure Overview:
    • Perform modified fuzzing to determine the software's vulnerability to algorithmic complexity attacks.

2.1.5: Product Stability

  • Criteria:
    • Does the software perform reliably?
  • Indicator:
    • Assess whether the software is susceptible to crashes.
    • Assess whether if the program is forced to unexpectedly terminate, it shuts down in a safe and responsible fashion.
  • Procedure Overview:
    • Fuzz software to see if and how it crashes.
    • Under appropriate fuzz testing, what was the code coverage, number of crashes, and type(s) of crashes.
    • Are crashes exploitable, or do they simply allow a disruption of service?

2.2: Software/Firmware

2.2.1: Software Updates

  • Criteria:
    • Is the software kept protected with software updates for a clearly defined and communicated period of time (i.e., the product life cycle)?
  • Indicator:
    • Assess the device to ensure it includes update capability and can be updated quickly when vulnerabilities are discovered.
    • Assess whether the product life cycle is communicated to the potential owner before purchase.
    • Assess whether software updates are authenticated.
    • Assess whether automatic software updates are provided.
  • Procedure Overview:
    • Examine software settings and product documentation to determine if automatic software updates are enabled by default or can be enabled by the user.

2.2.2: Update Encryption

  • Criteria:
    • Does the application or service encrypt updates files transmitted to the device?
  • Indicator:
    • Assess the device to ensure it uses encrypted update files and that the files are transmitted using encryption.

2.2.3: Update Notifications

  • Criteria:
    • Is the user notified software updates are available or that updates have been installed?
  • Indicator:
    • Assess whether users are notified of software updates.
  • Procedure Overview:
    • If updates are not automatic, examine software settings and product documentation to determine if the product notifies the user if a software update is available, and if that notification is persistent, or if the user can easily determine if a software update is available.

2.2.4: Update Process

  • Criteria:
    • Is the software update process simple for the user to complete?
  • Indicator:
    • Assess whether installation of software updates requires multiple steps to complete.
  • Procedure Overview:
    • Execute the procedure to install a software update and evaluate the ease of installation by comparing against established references.

2.2.5: Signed Updates

  • Criteria:
    • Does the application or service sign and validate update files transmitted to the device?
  • Indicator:
    • Assess the device to ensure is uses signed files and then validates that file before installation.

2.2.6: Automatic Updates

  • Criteria:
    • Can the user change the software update process to be automatic?
  • Indicator:
    • Assess whether software can be kept up-to-date for security issues.
  • Procedure Overview:
    • Check if a later version of software exists but the product cannot be updated to it (e.g. Android devices with pre-KitKat versions).

2.2.7: Firmware Updates

  • Criteria:
    • Is the device firmware able to be modified or updated?
  • Indicator:
    • Assess whether the firmware can be modified or updated by physical or remote access.
    • Assess whether the firmware can accept unauthorized updates.