Survey Theme Discovery and Classification
A US survey specialist needed open-ended responses turned into clear, actionable themes at scale, with sensitive data processed securely in their own AWS environment.
Hours to minutes
for a full round of open-ended responses
Manual quality
benchmarked against hand-coded survey programmes
2 steps
theme discovery, then response classification
The client
- SME in survey analysis
- Education
- United States
Overview
The client designs and analyses surveys, with a strong presence in the education sector. They needed open-ended responses turned into structured themes without manual coding becoming the bottleneck.
Clearlead developed, benchmarked, and deployed a secure two-step theme-then-classify pipeline on AWS, with joint testing before handoff.
They can now deliver interpretable theme prevalence and response labels across programmes, including sensitive material in their own cloud environment.
Methods
- LLMs
- Theme generation
- Text classification
- Qualitative survey analysis
Engagement type
- Design and build
The situation
The client works with education-sector and related survey programmes in the United States, where open-ended answers carry much of the insight. As response volumes grew, manual theme discovery and coding became the limiting step. Analysts needed outputs their end clients could use to see what was going well and what was not.
Survey content can be sensitive. Processing had to run in a secure, compliant AWS deployment under the client's control, not on informal tooling that would not meet their obligations to respondents and partners.
Requirements also varied by programme. A brand-new survey type might need themes discovered from scratch. A familiar instrument in a new subject area might need room for additional themes to emerge when a new issue shows up in the data. In other cases, an established coding frame was enough and the priority was assigning each response to known themes. The solution had to support those modes without a separate rebuild each time.
The engagement
Clearlead led the technical design and build. The core deliverable was a two-step process: derive themes from open-ended responses without requiring human coding first (with the option to supply or adjust themes where the client wanted that), then classify each response against those themes. Reaching a standard the client would trust meant benchmarking against surveys they had already coded manually, and substantial testing of current NLP methods, with particular weight on large language models.
-
Benchmarking against manual surveys so automated theme lists and response labels could be compared to the client's existing analyst work, making quality gaps visible before anything went to production.
-
Experimentation with NLP and LLMs to tune the two-step pipeline until outputs were close to what a human reviewer would produce, not a single fixed model choice dropped in at the end.
-
Two-step methodology separating theme identification from response classification, configurable so a run could do discovery only, assignment to an existing frame only, or both when a programme required it.
-
Iterative refinement with the client through regular review of sample outputs, adjusting prompts, theme definitions, and pipeline behaviour until results held up on real survey material.
-
AWS production deployment so sensitive responses could be processed in the client's own cloud account (secure storage, serverless runs, and managed model access).
-
Handoff and operational testing with the client's team so they could run the system, load new survey data, and confirm end-to-end behaviour before using it on live programmes.
Outcomes
- Actionable insight for the organisations commissioning surveys: clear visibility into what respondents are saying is working and what is not, from structured theme summaries rather than unreadable volumes of raw text.
- A secure, compliant AWS deployment suitable for potentially sensitive open-ended responses, with processing under the client's own cloud account.
- Less tedious manual theme discovery and coding, and a removed bottleneck that had limited how quickly the provider could take on new survey areas or domains.
- A domain-agnostic approach that scales beyond education: the same workflow applies when new programmes or sectors are added, with configuration rather than a ground-up rebuild.
Similar applications
The core techniques apply wherever large volumes of open text need to be coded consistently, and especially where the themes themselves are not known in advance.
Customer and user feedback
Open text responses where the goal is to identify which themes arise and track how their prevalence shifts over time or across customer segments.
Longitudinal theme tracking
Studies run across multiple time periods where understanding how themes shift wave-on-wave matters as much as knowing what themes exist.
Theme discovery without preset categories
Any context where the categories cannot be decided in advance because the goal is to find out what is actually being said, not to verify it against a scheme designed before analysis.
Public consultation analysis
Policy feedback and public consultation exercises where transparent, reproducible methodology matters as much as speed.
Qualitative research at volume
Interview transcripts, focus group outputs, or usability study notes where consistent coding is needed across a large sample but manual reading of every item is not feasible.
Employee engagement surveys
Surveys mixing ratings with free-text comments requiring consistent coding across business units and over time.
Discuss a similar engagement
Book a free 30-minute call. We will answer questions directly and say no when something is not a fit.