Testing, Validation & Quality Assurance

Non-Functional Testing

18 min Lesson 8 of 10

Non-Functional Testing

A hospital patient-management system that correctly stores every record but collapses under fifty simultaneous logins. An e-commerce checkout that processes payments accurately but exposes card numbers in server logs. A logistics portal that functions perfectly on a desktop browser but is unusable on a tablet in a warehouse. All three systems pass every functional test. All three fail in production in catastrophic ways. The culprit in each case is the neglect of non-functional requirements — and the tests that expose them.

Non-functional testing validates the quality characteristics of a system rather than its behaviors. Where functional tests ask "does the system do X correctly?", non-functional tests ask "how well, how safely, and how usably does it do X?" For systems analysts, understanding these test types is essential: you elicit the non-functional requirements, you set the acceptance thresholds, and you define the criteria by which stakeholders will judge the system fit for release.

The Analyst's Perspective on Non-Functional Quality

Non-functional requirements are notoriously hard to specify because stakeholders rarely volunteer them unprompted. A clinic owner says "the booking system must be fast." Fast means 200 milliseconds to a developer and "does not make me wait" to a receptionist. The analyst's job is to convert these vague expectations into measurable, testable acceptance criteria — and then ensure the test strategy covers them.

The dominant framework for classifying non-functional quality is ISO/IEC 25010, which organises software quality into eight main characteristics: functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, and portability. Three of these characteristics — performance, security, and usability — account for the overwhelming majority of post-launch failures in business systems and deserve deep attention from every analyst.

Performance Testing

Performance testing validates that the system meets its response-time, throughput, and resource-utilisation requirements under specified load conditions. It is not a single test; it is a family of related techniques, each targeting a different failure mode.

Load testing applies the expected production load — number of concurrent users, transaction rate, data volume — and verifies that the system meets its response-time targets under normal operating conditions. For an online store, the requirement might be: "95% of product-search requests must complete within 1.5 seconds when 500 users are active simultaneously."

Stress testing deliberately pushes the system beyond its designed capacity to discover where it breaks and how it recovers. The goal is not to prevent failure but to understand the failure mode. Does the system return a graceful "service temporarily unavailable" message, or does it corrupt data and leave transactions in an unknown state? A logistics firm whose freight-booking platform degrades gracefully during Black Friday peaks retains customers; one that silently drops bookings does not.

Soak testing (also called endurance testing) runs the system at sustained moderate load for an extended period — often 24–72 hours — to expose slow memory leaks, gradual database connection exhaustion, and log-file growth that only manifests over time.

Spike testing simulates a sudden, sharp increase in load — a marketing campaign launches, a news story goes viral, a government deadline triggers a rush of submissions — and verifies the system's behaviour during and after the spike.

The analyst's role in performance testing is not to run load-test scripts — that is the performance engineer's job. Your role is to elicit quantified performance requirements during discovery ("What is the maximum acceptable wait time for a search result?"), document them in the SRS as measurable acceptance criteria, and participate in reviewing the results against those criteria at test completion.

Performance requirements must be quantified with three components: a metric (response time, throughput, CPU utilisation), a threshold (under 2 seconds, at least 200 transactions per second, below 70%), and a load condition (at peak concurrent-user count, on minimum-spec hardware, after 24 hours of operation). Vague requirements produce untestable results.

The four main performance testing types and the distinct failure modes each one is designed to expose.

Security Testing

Security testing validates that the system protects its data and functions from unauthorized access, manipulation, and disclosure. For an analyst, security requirements emerge during elicitation as confidentiality, integrity, and availability constraints — the classic CIA triad. The test strategy must verify that those constraints are enforced.

Authentication and authorization testing verifies that identity controls work correctly: login with invalid credentials is refused; session tokens expire appropriately; a clinic receptionist cannot access billing reports restricted to finance staff; a REST API endpoint that requires an admin role cannot be accessed with a standard-user token.

Input validation and injection testing checks that every point where the system accepts user input is resistant to SQL injection, cross-site scripting (XSS), command injection, and similar attacks. This is especially relevant for analysts because weak input-validation requirements ("the system shall accept user input") often open these vulnerabilities — stronger requirements ("the system shall sanitise all text inputs by stripping HTML tags and parameterising all database queries") close them.

Data-in-transit and data-at-rest testing verifies that sensitive data (passwords, payment card numbers, health records, personal identifiers) is encrypted when stored and when transmitted. A common test checks that sensitive fields are not accidentally logged in plain text during error events — a failure mode that affects systems at every scale.

Penetration testing (pen testing) is a structured, authorised attempt to breach the system using the same techniques an attacker would use. It is performed by security specialists, not the development team. The analyst's contribution is defining the scope: which assets are in scope, what level of access the tester is given (black-box, grey-box, or white-box), and what constitutes a finding severe enough to block release.

Security testing is not optional for systems handling personal data. Regulations such as GDPR, HIPAA, and PCI-DSS impose legal obligations on organizations that process personal, health, or payment data. Failure to test security controls before launch is not merely a technical risk — it can result in regulatory fines, reputational damage, and legal liability. The analyst must surface these obligations during the feasibility and requirements phases, not after development is complete.

Usability Testing

Usability testing evaluates how effectively, efficiently, and satisfactorily real users can accomplish real tasks with the system. It sits at the intersection of the analyst's two core responsibilities: understanding user needs and verifying the system meets them. Usability problems discovered after launch are expensive to fix; discovering them in prototype or UAT is dramatically cheaper.

A typical usability test session presents representative users with a set of realistic task scenarios — "book an appointment for a new patient," "generate the monthly sales report," "add a new supplier and submit a purchase order" — and observes them working through the tasks without assistance. The analyst watches for hesitation, errors, backtracking, and verbal expressions of confusion.

Key metrics captured in usability testing include:

Task completion rate — the percentage of users who complete each task successfully
Time on task — how long it takes a representative user to complete each task; compared against an efficiency target
Error rate — the number of unrecoverable errors per task; an error rate above 10% on a critical task is a significant finding
System Usability Scale (SUS) — a standardised ten-question post-session questionnaire that yields a 0–100 score; an industry benchmark score above 68 is considered acceptable
Accessibility compliance — whether the system meets WCAG 2.1 AA criteria for users with disabilities, a legal requirement in many jurisdictions

What analysts own versus what specialists execute across the three main non-functional testing domains.

Writing Non-Functional Acceptance Criteria

The most practical skill an analyst develops in this area is writing acceptance criteria that are precise enough to be tested. A useful template uses the structure: Given [load/context condition], when [action], then [measurable outcome].

Performance:
  Given 500 concurrent users on weekday peak hours,
  when a user submits a product search query,
  then 95% of responses must be returned within 1.5 seconds
  and no response may exceed 4 seconds.

Security:
  Given any authenticated user without the Finance role,
  when they attempt to access the /reports/revenue endpoint (directly or via URL manipulation),
  then the system must return HTTP 403 and log the unauthorized access attempt.

Usability:
  Given a first-time user who has completed the onboarding walkthrough,
  when they attempt to submit a new purchase order,
  then at least 80% of representative users must complete the task
  within 4 minutes with no more than one navigational error.

Involve stakeholders when setting thresholds. Performance targets should come from business context: a patient-booking portal used by receptionists who handle 30 calls per hour has a different latency tolerance than a self-service kiosk in a waiting room. Security severity ratings should be validated with the legal or compliance team. Usability targets should be set against competitor benchmarks or the current system baseline — not invented arbitrarily by the analyst.

Integrating Non-Functional Testing into the Project Life Cycle

Non-functional testing is not an afterthought applied at the end of development. Performance baselines should be established during sprint reviews or build checkpoints; security scans should run in the CI pipeline; usability walkthroughs should accompany prototype reviews from the earliest design phases. The analyst's role is to ensure that non-functional acceptance criteria appear in the SRS, are allocated to the test plan, and are explicitly tracked to a passed or waived status before release approval is granted.

Findings from non-functional tests feed directly into the defect management process covered in the next lesson. A performance test that reveals a 3-second average response time against a 1.5-second SLA is a defect with a severity rating and an owner, not merely a "known issue to monitor in production."