The Methodological Problems Hiding in Your Research Tools

Summary: The methodological blind spots in UX research tools have always been a problem. Now that AI is planning and analyzing research, it's gotten worse.

Many user-research platforms have been around for over a decade; some close to two decades. And yet, one thing is apparent to most experienced researchers: many of these tools were not built by expert researchers or with sufficient input from them. At Nielsen Norman Group, we've lamented this for years.

For most of the history of UX research tools, the worst thing a poorly designed platform could do was slow you down or send you looking for a workaround. That's changed. We're now in an era where research tools don't just host your study - they plan it, moderate it, and analyze it for you. If those tools lack a solid foundation in research methodology, the risk is greater than simple inconvenience - it's flawed research presented with confidence on a large scale.

A Quick History of UX Research Tools

Before diving in, it helps to understand the landscape of tools we're talking about and where AI fits into it.