IBM's Chat Analyzer: Concept Testing

Challenge:

IBM’s Client Conversion Experience (CCx) Marketing team set out to explore how the Chat Transcript Analyzer could better inform content decisions across IBM’s Lever pages. The study aimed to uncover how marketing teams currently engage with data, identify gaps in their workflow, and determine how the Analyzer could seamlessly integrate into their existing processes.

At the center of this effort are IBM’s Digital Strategists, who maintain and optimize content across Lever pages to guide users toward IBM’s offerings. Although these strategists are experts in their respective domains, they often face time constraints and depend heavily on familiar tools such as Marketing Performance Manager (MPM) to support their daily work.

To help bridge this gap, IBM sought to increase both awareness and adoption of the Chat Transcript Analyzer—a tool that surfaces trends from thousands of customer support chats. By making these insights more actionable, shareable, and relevant to ongoing content decisions, the team aimed to empower strategists to better understand customer needs and continuously improve the conversion experience.

Strategy:

We conducted 10 moderated in-depth interviews (60 minutes) with key roles across the marketing team

• 3 Lever Leaders

• 6 Digital Strategists

• 1 Digital Lead

The interviews focused on participants’ content update processes, collaboration patterns, preferred tools, and their impressions of a low-fidelity Chat Transcript Analyzer prototype. We explored what makes insights feel relevant and valuable, how information is currently shared across teams, and how new tools can complement existing platforms like Slack and MPM. Rather than testing performance, we sought input on the potential value of the Analyzer and how it could be shaped to support real-world use cases. Feedback was gathered on visual formats, update frequency, data specificity, and expectations for personalization and clarity.

Insights:

1. Relevancy Matters: Strategists want insights tied directly to their specific Lever pages and target audiences.

2. High-Level + Deep-Dive: They prefer quick summaries, but with the option to explore detailed chats or Excel exports.

3. Visual Clarity Is Essential: Participants found charts engaging but often unclear without better labeling and context.

4. Integration With Trusted Tools: MPM is seen as the “source of truth.” Insights should complement the tools they already use.

5. Notification Fatigue is real: Strategists are open to Slack or email alerts, but only if the content is concise and personalized.

   
   

Impact:

1. Insights are shaping improvements to Chat Transcript Analyzer, focusing on clearer formats, personalized delivery, and integration with Slack and MPM.

2. IBM aims to boost adoption and engagement by aligning the tool with real-world needs.

3. Findings are now part of CCx’s roadmap for future data tools.