What Businesses Should Ask When Choosing an AI Model

Business leaders evaluating artificial intelligence (AI) solutions should be cautious about assuming that larger generalist models such as OpenAI’s ChatGPT will always deliver better results than specialist models, according to an analysis published this week by Harvard Business Review. 

While large language models (LLMs) such as ChatGPT, Claude and Gemini perform well in general tasks that call for getting information from a wide variety of sources — such as customer service — they don’t do as well as AI models customized for specific industries in domain-specific tasks.

This might seem obvious, but it pans out in practice. That’s the experience of report authors Amber Nigam, founding CEO of startup Basys.ai, and John Glaser, executive in residence at Harvard Medical School and former CEO of Siemens Health Services. 

The authors built generative AI systems for health insurers to determine whether a doctor’s recommended treatments are covered by the patient’s health insurance policy. They found that choosing the right Gen AI model is crucial to getting optimal business outcomes.

The critical difference to understand between generalist and specialist models is this: Specialized AI models know not just what data to retrieve but also “how that information operates within a specific domain’s decision-making framework,” the authors wrote.

For example, a specialist AI model deciding whether health insurers should approve coverage for a lung cancer patient’s treatment plan must consider whether the chemotherapy qualifies under the policy.

It also must consider the patient’s medical condition, such as end-stage renal disease and a recent hospice referral, that would affect a treatment or decision on insurance coverage, the authors wrote.

A generalist AI model attacking the same problem would look for historical patterns of insurers approving coverage for other patients with similar symptoms. However, “this pattern-matching approach misses the underlying clinical and policy logic that drives these decisions — especially with more complex cases like the one mentioned above,” they said. 

See also: From Buzzwords to Bottom Lines: Understanding the AI Model Types

The “aha” moment for the authors was this: “Why would we try to make the AI think like a computer when it needs to think like a doctor?”

The authors then trained their Gen AI agents to “follow how clinicians read — understanding the structure of charts, moving from sections to subsections, and identifying the right findings in context.”

This understanding will help other enterprises choose the right AI models to deploy for use cases, whatever the industry, according to the authors.

According to a PYMNTS Intelligence report, business leaders are tailoring Gen AI systems to meet their strategic imperatives. Those who work in the goods and technology industries are far more likely to use Gen AI for product design and idea generation — a generalist use case. But services companies use Gen AI for more specific purposes such as crafting better strategic positioning or generating insights faster.

Moreover, while most corporate executives surveyed rated Gen AI highly in terms of its effectiveness at completing specific jobs, they acknowledged that it still requires “extensive” human oversight. That means “the enthusiasm might reflect how quickly adopters have started to find Gen AI useful in precise ways, rather than how ready they view it as actually being,” according to the May 2025 report.

The disconnect between expectations and reality underscores the importance of choosing the right AI model for the job. The authors said companies should ask AI vendors the following questions to avoid common pitfalls:

1. Can the system explain its reasoning transparently? Generalist AI models often provide shallow or opaque answers; specialized systems must show how they use criteria in decision-making and cite standards or precedent.
2. Does the vendor work closely with domain experts and adapt to changing norms? Rich collaboration ensures models embed evolving professional frameworks, not outdated rule sets.
3. Does the solution scale across domains? Avoid siloed point solutions by opting for architectures that integrate professional logic and can expand into new areas like finance, law or engineering.

Read more:

AI Models Are Maturing with Customization as the Next Focus

Report: Reasoning AI Models Fail When Problems Get Too Complicated

Microsoft Plans to Rank AI Models by Safety

The post What Businesses Should Ask When Choosing an AI Model appeared first on PYMNTS.com.