What does it really take to move AI from an impressive demo into everyday clinical practice?
We spoke with Paweł Paczuski, CEO and co-founder of upmedic, about the challenges of deploying AI in healthcare, from verifying AI-generated documentation and integrating with hospital systems to earning clinicians’ trust and scaling beyond early adopters. Paweł’s experience reveals why better models alone are not enough, and why the hardest part of making AI useful often begins after the technology starts working. Read the full interview below.
Most companies have spent the last few years asking what AI can do. You are now dealing with a different question: what happens when people actually start using it? What has surprised you most about that transition?
The healthcare AI landscape is evolving rapidly. We are witnessing weekly model releases, an unprecedented healthcare cybersecurity crisis driven partly by LLM adoption, and the collapse of traditional software business models. Meanwhile, patients present with AI-generated self-diagnoses, and clinicians must continually adapt to new tools and shifting compliance frameworks. What surprises me most is how fluidly clinicians redefine AI. Early in my healthcare journey, speech recognition was considered AI. Recently, a group of clinicians told me that generative AI no longer qualifies as AI to them, their benchmark now is complete workflow automation that eliminates manual tasks entirely.
You were working on structured medical documentation years before generative AI became the topic. What did the arrival of generative AI genuinely change for upmedic?
Positive
Negative
On a high-level, what works beautifully in an AI prototype but tends to break once you put it in front of hundreds or thousands of real users?
With GenAI, text generation comes for free. It has shifted focus from generating text (editing) to verifying it. It does not have to be better. Most of the doctors want to see identical text they would produce without ai. If it takes more time to verify correctness of a text than to write it from scratch, then it is not worth deploying it.
Has working with actual clinicians changed your own definition of what a “good AI product” is?
Working with clinicians showed how different workflows in healthcare. The definition of a good product is not so simple. If we take into consideration only clinicians, it must be easy to learn, customizable, deployable in heterogeneous IT settings. The same solution, when assessed by a different stakeholder, e.g. management, will need to have totally different features to be considered as “good AI product”.
What will be the future of medical documentation?
Medical documentation fundamentally serves as an overview of a clinical encounter. Once privacy and adoption hurdles are resolved, physicians likely won't need to manually draft these notes. Clinical workflows produce various other documentation types, including structured lab results and radiological imaging reports. Centralizing these inputs enables seamless data querying, synthesis, and analysis. By safely archiving primary visit data, we establish an authoritative record of the encounter. Automated documentation generated from these raw inputs can then serve as tailored summaries aligned with specific clinical contexts.
Is there a danger that companies use AI to automate processes they should have redesigned first?
In some cases there is no other way round. Some processes that need to be automated, need to go through several steps of semi-automation.
In medical documentation specifically, how much of the problem is really an AI problem, and how much is a workflow, process, data or UX problem?
The problem itself lies beyond AI or any other hype cycle in computer science. It depends on the technological maturity of a partner how they even define the problem. Sometimes under “documentation” you can find any non-clinical activity like basic computer skills. In such cases you need to start with distilling the expectations and then prioritizing solutions.
GenAI is currently the best way to solve this family of problems that are related to medical documentation, but before it classic speech-to-text also solved a lot of problems. It is a matter of how much repeatability you need to have in a process you try to automate
Have you encountered situations where AI could technically automate something, but after observing how clinicians actually work, you concluded that it shouldn't?
We encounter such situations all the time. Data Layer & System Capabilities: In past projects, we proposed GenAI solutions to convert natural language into SQL queries, only to discover that the generated SQL could query merely a fraction of the dataset. Debugging the GenAI layer becomes extremely challenging when the underlying issue resides in the data access layer. This often necessitates frank alignment meetings to manage customer expectations, as clients frequently anticipate that AI will resolve core data issues without effort on their end. User Experience & Skills: There is a real risk of deskilling and fostering an over-reliance on AI systems. Regulatory Compliance: Healthcare frameworks like MDR and IVDR explicitly specify allowable software functions required to meet designated device classifications.
What production teaches you that a PoC cannot?
It is better to deploy a small scale production setting than a big poc project. Doctors will not assess the technology on “imaginary cases”. They want to see how technology behaves in real-world scenarios. Enterprises really love the concept of PoC, but it is also possible to define a PoC at the enterprise level that is realized as a small production deployment at the clinical leve.
What needs to happen between “the model works” and “the customer is willing to deploy this across their organization”?
The value of the model: is it clearly visible? Is it measurable?
When an AI pilot doesn't progress to a larger deployment, what is usually missing?
The single greatest threat to any project's success is administrative and organizational friction.
Key stakeholders involved
Pre-Project Preparation
Initial steps require securing executive buy-in, resolving compliance requirements, and addressing foundational technical parameters. Additionally, the clinical workflow itself must undergo pilot testing prior to any full-scale rollout.
Project Execution & Alignment
Ongoing coordination with both IT personnel and medical staff is essential. While IT teams are generally accustomed to software deployments, clinicians operate under fundamentally different work dynamics. Because a clinician's primary responsibility is patient care, participating in a pilot introduces an unfamiliar, project-based workload. This shift represents a far greater operational adjustment than they typically anticipate. It is not uncommon to complete all preliminary arrangements with IT and compliance, only to discover that physicians lack the allocated time required to engage with the project.
What is the biggest misconception you encounter among organizations moving from AI experimentation into production?
They usually assume that you need to simply scale the same narrative that worked for the pilot. This is not true as during pilots you focus on the 16% of population (Innovators+early adopters Diffusion_of_innovations) that are open for innovation. The remaining population will need a mix of other mechanisms to fully absorb the complexity of any change.
Doctors are an interesting user group because their time is scarce and their workflows are highly specialized. What have they taught you about designing AI for people who have very little patience for the technology itself?
Even if a clinician has only ten minutes to familiarize themselves with your solution and grasps a mere 2% of its total capability, they must immediately sense that, with your support, that 2% yields immense tangible value.
What does adoption actually look like? Is there usually an initial period in which doctors carefully inspect everything the system produces before their approach starts to change?
Clinicians represent a highly diverse user group. Their attitudes toward software range from total resistance to genuine curiosity, while many view non-medical technology as an unnecessary distraction. When training medical professionals on a new tool, it is essential to deliver immediate, practical value that seamlessly integrates into their daily routine.
Can you see trust developing in the product data. For example through fewer corrections, different usage patterns or users gradually delegating more work to the system?
You've discussed the distinction between a “centaur,” where the human remains in control and AI performs part of the work, and an inverted centaur, where the human starts following the machine. Are there any signals that tell you that the line is being crossed?
Official deployment by a medical facility ensures proper IT oversight and compliance safeguards. Conversely, when clinicians resort to copy-pasting content from public LLMs, patient privacy protections are severely compromised. Indeed, instances have occurred where clinicians effectively acted as meat-proxy for an LLM - even accidentally retaining default outputs like "here is the probable shape of a document I prepared for you" directly in the final record. Keep also in mind that patients also have access to those public tools, so do not be surprised if your LLM will be used to criticise/approve output of patient’s LLM.
Human-in-the-loop is one of the most repeated phrases in enterprise AI. What does meaningful human oversight actually look like in a product used in healthcare?
While human oversight remains indispensable, it is the product's responsibility to make that review loop as efficient and digestible as possible for the user.
Does making an AI system more accurate make the human-in-the-loop problem harder?
Placing a human in the role of final verifier requires a clear distinction between baseline elements that can be trusted implicitly and specific generated outputs that demand manual review. Should clinicians truly be responsible for inspecting every raw input, signal, system parameter, and data conversion step?
When healthcare providers bear full liability for every detail of an output, they are naturally driven to minimize the volume of generated text as a strategy to mitigate legal exposure. Relying on a human-in-the-loop framework is inherently flawed if clinicians are forced to routinely validate fundamental technical baselines - such as display pixel luminance - prior to reviewing clinical imaging. Anything that can be automated in terms of verification ought to be. Existing mechanisms like system watchdogs, alongside deterministic validation of specific inputs and outputs, already serve this purpose without needing one LLM to monitor another LLM. To make this operational, the focus must shift away from unstructured text documentation toward structured data models, enabling automated, logic-driven conclusions.
Increasing model accuracy inadvertently complicates proving when the system errs. In traditional radiology, defining a fuzzy lesion's precise circumference was a frequent topic of debate. As AI automates these measurements, such discussions must be revisited. I anticipate these debates could drag on endlessly without reaching a consensus; in the meantime, a single vendor's approach will gain traction, becoming the widely accepted default standard despite not necessarily being the optimal solution.
You've spoken publicly about hallucinations and AI-generated “slop” in medicine. How do those problems change when AI moves from a general-purpose chatbot into a narrowly defined clinical workflow?
Creating a Knee MRI radiological report is my favorite example of a narrowly defined clinical workflow. While a typical radiologist takes about 20 minutes to produce this document, a swift radiologist equipped with deterministic tools, and no LLMs, can complete it in just 2 minutes. That 10x speedup comes simply from refining image-reading skills and knowing how to utilize existing tools.
Although anyone could replicate this workflow, very few actually do. This stems largely from mindset: most physicians operate on the assumption that their personal method is optimal, while medical conferences tend to center on discussing interesting cases rather than efficiency. However, this culture is starting to shift. For example, radiological competitions are emerging that evaluate speed alongside accuracy in identifying image pathologies. These events demonstrate that analyzing images does not inherently require extensive time.
In tightly constrained clinical domains like this, measuring performance gains from new tools is straightforward. The broader opportunity lies in instructing peers in the techniques used by the fastest radiologists, making adoption more an issue of overcoming ego than solving technological deployment. When patients face 12-week wait times for radiological reports, the potential impact of scaling that 10x throughput across all doctors is immense. What does “accuracy” actually mean for upmedic? A single accuracy percentage seems insufficient when one mistake might be stylistic and another clinically significant. Accuracy of the whole system? It it not so easy to measure. We usually treat it as a measure of how well upmedic generated the text for the clinician so no further changes needed to be added. However, some changes to the text are added by the clinician not because they are pieces of information omitted by the model, but because the clinician observed something and wanted to make a note about it. Rather than aiming for a specific number, we observe how the number of changes needed to be introduced changes, expecting it to decrease over time.
In production AI, is knowing when the model is uncertain sometimes more important than improving its average accuracy?
In our workflow contexts, generating incorrect information is actually preferable to producing no output at all. Refining or fixing inaccurate details is significantly easier for clinicians than adding missing information entirely from scratch. Even though correcting AI errors can be frustrating, doctors use these imperfect outputs as functional placeholders for key facts that were mentioned. Rebuilding that missing context without a initial starting point would impose a far greater operational burden.
How do you test AI for unusual patients, unusual terminology, uncommon cases or ways of speaking that may barely appear in training or evaluation data?
To evaluate models against rare clinical scenarios, complex terminology, and distinct speech habits, we rely on a dedicated benchmark dataset containing historical cases where earlier model iterations failed dramatically. In tandem, a multi-specialty advisory panel of physicians from high-volume tertiary care centers rigorously evaluates every candidate model to confirm clear performance gains before any broader deployment.
What have real users managed to do with the product that your internal testing never anticipated?
We have seen some doctors use our speech recognition to dictate slides for medical conferences. As we recognize specialistic terminology well, we were natural candidate for them to save some keystrokes when preparing lectures.
When something goes wrong, how do you determine whether the problem came from the model, the surrounding product, the input data, the workflow or the user's expectations?
This depends on the deployment but it is a mix of very intensive logging, metrics about surrounding environment, behavioral analysis.
If we stripped the AI model out of the conversation for a moment, what are the hardest engineering problems involved in deploying a product like upmedic?
Although it may not be immediately apparent to the user, upmedic operates fundamentally as a structured reporting platform. We put substantial effort into presenting a seamless, plaintext-like experience for clinicians. Beneath the surface, our documentation editor relies on sophisticated techniques from compiler theory and formal grammar to solve complex engineering challenges. It is fascinating to reflect on the amount of technical heavy lifting required to deliver an interface that feels so straightforward. This underlying architecture offers critical advantages:
Healthcare organizations already have HIS, RIS and other systems that were not designed around today's AI. How much of building an AI company is actually integration engineering?
Our relationship with HIS and RIS systems is a mix of appreciation and frustration. On the positive side, their existence spares us from having to build and maintain an entire EMR infrastructure ourselves. We also benefit from the fact that many of these legacy platforms historically treated medical records as basic plaintext snippets stored in a single database column. The friction arises during integration. Time and again, we have shown that opting for a less conventional integration path avoiding split vendor responsibility yields better results than relying on an official vendor bridge that looks good on an invoice but falls short in day-to-day usability. While upmedic can function effectively without deep standard integrations, many medtech products depend on them heavily. As a result, those companies often bypass markets like Poland in search of regions with more established, accessible APIs.
How much of upmedic's know-how today sits outside the model itself. I’m thinking about workflow design, integrations, domain knowledge, data structures etc. Our models simply reflect the core insights embedded within our internal datasets (tens of millions data samples), enhanced by specialized techniques to extract knowledge more effectively. To support this, our initial priority was creating a robust system for collecting data in a format easily adapted for model training—which required making upmedic a tool doctors genuinely enjoyed using in their daily routines right from the start. Furthermore, a major strategic advantage for us lies in our approach to integration, particularly our ability to streamline connections for the direct benefit of clinicians.
What changes technically and organizationally when you move from a successful deployment in one department or medical specialty to many?
Technically, the consistency of challenges across different medical fields is remarkable. Certain specialties rely heavily on specific scoring scales, whereas others avoid them altogether. In fact, due to the severe lack of standardization, medical records from a cardiologist and an oncologist within the same hospital often share more structural similarities than reports generated by two oncologists at different institutions. From an organizational perspective, institutional politics becomes a critical factor as deployment expands. Even deciding the sequence in which departments receive access to the software can carry substantial weight. Overlooking these seemingly minor details risks triggering resistance among clinical staff, which can stall full-scale adoption for months.
What's harder to scale: the technology or the organizational change around it?
Driving organizational change is far more challenging. Although technical solutions for these issues have existed for years, progress was historically stalled by a combination of limited computer literacy among medical personnel and a lack of institutional drive. Today, generative AI creates substantial buzz, sparking genuine fear of missing out among staff. This momentum empowers healthcare institutions to reframe these legacy challenges, leveraging proven cross-industry methodologies paired with emerging technological paradigms.
What does operating an AI product look like after deployment? How much continuous monitoring, evaluation and adaptation is required compared with conventional software?
Testing traditional software is straightforward by comparison. While users have historically found unexpected ways to break conventional applications, introducing an AI layer makes verifying deterministic input-output pairs vastly more complex. Overseeing a deployed AI system closely parallels post-market surveillance in pharmaceuticals: you release a model, track its adverse effects, document them, and address those issues in the subsequent iteration.
Healthcare gives us an unusually clear version of a question every industry will eventually face: if an AI system contributes to an error, where does responsibility sit? How do you think about that boundary as a product company?
While this remains an open debate with extensive ongoing discussion, my perspective centers on first ensuring clinicians are supported to operate under a true best-effort standard. Evaluating liability becomes far more straightforward when doctors have the necessary time, resources, and institutional backing to practice aligned with current medical advancements. Once those safeguards are in place, it is unreasonable to place the entire burden of risk onto clinicians for utilizing thoroughly evaluated tools. Neither medicine nor AI models possess answers to every scenario, and our policy frameworks must reflect that reality. Holding physicians solely liable for AI-related errors will ultimately discourage adoption or drive AI usage into unmonitored gray areas, creating significant risks for patient data privacy.
How do you decide which decisions the product may assist with, which it may automate, and which should remain firmly with the clinician?
To determine which tasks to automate, assist with, or leave entirely to the clinician, we follow a two-step approach:
This strategy has proven highly effective for guiding our development efforts.
Is regulation currently one of the main obstacles to deploying AI in healthcare, or is that explanation sometimes used to hide more mundane problems such as integration, procurement and organizational readiness?
When an organization possesses true operational readiness, navigating complex regulatory frameworks becomes significantly simpler. Ready institutions actively seek paths to accelerate deployment timelines. Ultimately, organizational readiness and sharp strategic prioritization dictate success because when every priority is treated as critical, none truly are.
The industry has rapidly moved from copilots to agents. From where you sit, does that represent a genuine product shift, or are we once again ahead of what customers actually need?
Although agentic solutions fulfill a genuine need for healthcare facilities, current industry capabilities fall short of effective delivery. Deploying autonomous agents demands higher token throughput and greater computational power. Beyond compute constraints, agentic workflows require significantly richer context than standard copilot setups making the privacy issues of patient data even more apparent. Unlocking these advanced use cases relies on enabling the seamless execution of local LLMs directly on clinician workstations across varied hospital IT infrastructure.
What new safeguards become necessary when AI moves from generating information to acting on information?
Before advancing to this next phase, establishing properly structured data within HIS/RIS environments is crucial. Lacking this foundation makes even standard LLM prompting exceptionally difficult. While standard practice across other sectors, this approach represents a fundamental shift in healthcare thinking.
Do you think we're currently overestimating what better models will solve and underestimating everything that has to exist around the model?
Without a doubt, the second part holds true. We have yet to fully realize the capabilities of smaller, current-generation models operating on consumer hardware, provided they are supported by effective tooling and robust context engineering. Regarding future advancements, significant developments lie ahead beyond traditional LLMs. Notably, leading conversational systems already combine symbolic reasoning with “classic” architectural methods.
What separates organizations that successfully operationalize AI from those that accumulate impressive PoCs that never become part of the business?
In the healthcare sector, it is remarkable how frequently partnerships begin without upfront financial discussions. Commercial enterprises operating in the market must fund their operations, making it essential to establish that services are not provided gratis. Conversely, interactions often occur with counterparts who merely wish to experiment. Conducting a Proof of Concept without aligned commercial terms remains an unviable pursuit, regardless of how prestigious the partner’s brand may appear on a corporate website.
Five years from now, what would convince you that AI genuinely transformed healthcare rather than simply adding another layer of software to it?
I witness this shift every day. Regardless of whether medical staff embrace it, many patients are already using LLMs to discuss their conditions, analyze their health data, and present doctors with hypotheses to verify or expand upon. Medical knowledge has become accessible and easily understandable to curious laypeople, marking the crucial first step. Importantly, this evolution requires no complex integration with EMR systems, as it relies entirely on data stored in personal phone-based health vaults. My hope is that this will empower patients to receive effective care at the outpatient clinic level, eliminating two-year wait times for specialist appointments. In this model, doctors act as essential gatekeepers, ensuring patients receive serious medications, invasive diagnostics, and complex procedures only when truly necessary.