During a recent engagement leading Product at a high-tech startup, one of my first goals was to strengthen the bridge between Customer Support and Product. I set up recurring syncs with the CS team with two immediate objectives:

  1. Establish a clear pipeline for escalating issues - from critical, time-sensitive bugs to strategic improvements for future sprints.
  2. Identify recurring customer pain points that might not be “on fire,” but were quietly eroding user experience or team efficiency.

The CS team had strong instincts around urgent issues. But they lacked a structured view of broader patterns - repeated questions and friction points that revealed gaps in our product, UX, marketing or support content.

The good news: we had the data. Every customer conversation - starting with our AI chatbot and continuing through human handoff - was stored in ZenDesk.

The challenge? Thousands of transcripts: untouched, untagged and unstructured.

Manual review was a non-starter. And our built-in tools couldn’t extract insights at the level of fidelity we needed.

So I built a solution.

Using AI - But Keeping It Local

Because these transcripts potentially contained sensitive information, I didn’t want to use a cloud-based LLM API. Fortunately, I’ve been experimenting with locally hosted models using Ollama and had an environment ready.

Using under 100 lines of Python, I processed the transcripts in batches - feeding them into both Mistral and Llama 3 models running on a modest AWS instance.

Here’s the base prompt I used:

This is a list of customer-submitted questions from a help form. I would like you to:

1. Summarize the main themes or topics of the questions.
2. Provide a percentage breakdown for each theme based on how frequently it appears in the list.
3. Offer actionable suggestions to reduce customer confusion and improve service in areas related to these themes.

Ensure your response is well-structured and concise, using the following format:
-  Summary of Themes : A short list of the main topics covered in the questions.
-  Percentage Breakdown : A table or bulleted list showing the percentage of questions that fall under each theme.
-  Counting Coverage : add a count to the numbers in each theme, and an "unclassified" theme for those questions that don't fit nicely in other themes. ensure that the count matches the initially number of questions submitted.

Here is the list of questions to analyze: {input_text}

After summarizing each batch, I ran a follow-up prompt to synthesize the outputs into one unified analysis.

Embracing a Probabilistic Mindset

With a deep background in data analytics and product insights, I’m used to deterministic outputs - clear numbers, exact classifications, repeatable results.

LLMs don’t behave that way.

Re-running the same transcripts through the same model produced slightly different categorizations. At first, I saw this as a weakness. But I came to see it more like statistical sampling or qualitative coding.

LLM-based analysis is non-deterministic, and its value lies not in exact counts, but in directional accuracy and speed to insight.

That mindset shift - treating unstructured data analysis as probabilistic - has significant implications for how we evaluate AI-driven outputs moving forward.

What I Uncovered

With just a few hours of coding and machine time, I surfaced:

  • Frequently overlooked issues CS hadn’t fully appreciated
  • Gaps in our website and help center content that were generating support load
  • Opportunities to train our AI chatbot to handle common requests more effectively

These insights led to quick wins across Product, Marketing and Support - without needing a dedicated data team or weeks of effort.

The Takeaway

We’re entering an era where AI can help us extract signal from unstructured noise at unprecedented scale. But that signal is often impressionistic, not exact - and that’s okay.

As modern product leaders, we need to:

  • Get comfortable with non-deterministic outputs
  • Use AI to accelerate insight, not replace judgment
  • Balance speed with interpretability and actionability

This project reinforced a deeper truth: AI won’t replace product work one-to-one - it reshapes it. Yes, it accelerates processes we once saw as bottlenecks, like data analysis. But in doing so, it requires a new kind of fluency: crafting the right prompts, guiding the models through iterative ideation, validating fuzzy outputs and interpreting probabilistic insights in context.

The future isn’t just about AI doing our work faster - it’s about evolving how we think, lead and collaborate. The real value becomes tangible when humans stay in the loop to shape, coach and elevate the output.

That’s where I believe the future of product leadership lies: at the intersection of data, code and human insight.