How Prompt-Based Development Revolutionizes Machine Learning Workflows
Discover how prompt-based development can cut project timelines, reduce costs, and offer an agile approach to innovation.
In a previous blog post, we introduced a five-phase framework to plan out Artificial Intelligence (AI) and Machine Learning (ML) initiatives. We looked at an exemplary healthcare project with a potential problem of high readmission rates. We also discussed how to tailor AI to predict readmissions, offering a targeted solution that enhances patient care while effectively cutting costs and improving the patient’s well-being. Today, we delve into the traditional ML workflow and explore how prompt-based development accelerates the process by enabling parallel idea experimentation with minimal investment.
The Traditional Machine Learning Workflow
Initiating a traditional ML project begins with collecting data. In the healthcare scenario, this means collecting electronic medical records (EMR). This data then undergoes manual cleaning to address inconsistencies, from measurement outliers to data entry mistakes. Duplicated records are identified and rectified. Afterward, the data is labeled to create training and testing datasets.
Following data engineering, we transition to algorithm and model selection, a choice influenced by both the business case and the type of available data.
For instance, the decision between unsupervised and supervised learning, like opting for K-Means for cluster analysis or SVM for classification problems, plays a crucial role. The selected model may be trained on labeled datasets to effectively differentiate patients with high readmission risks from those with lower risks. Subsequently, data scientists evaluate the model’s accuracy, precision, and recall metrics to pinpoint high-risk patients.
Once the model achieves optimal performance, it’s deployed to operational environments to forecast real-world hospital recurrence rates. The emphasis then shifts to collecting real-time patient insights. Incorporating these insights into the model’s dataset ensures its precision and relevancy by reflecting emerging health trends and new patient data. This comprehensive workflow forms the foundation of traditional ML processes.
Traditional Machine Learning Workflow Challenges
Challenges within the traditional ML workflow are not to be underestimated. Common iterative hurdles include:
Data Scarcity — In most cases, model training requires a vast quantity of high-quality data. Its absence can cause compromises to the model’s performance and precision, affecting the reliability of the end product.
Manual Annotation — An often-underestimated task is the manual labeling of data post-collection, where manual annotations are required to clarify the data.
Model Retraining Costs — During the model training and evaluation phases, we might observe a deficiency in high-variance data, making it daunting to test our hypotheses. Leading to potential model retraining and increased costs in time and resources.
These hurdles hinder the ability to validate assumptions and hypotheses, making such initiatives challenging without thorough planning. But what if we could develop the proof-of-concept (POC) in a fraction of the time?
How Prompt-Based Development Aligns with Agility
Prototyping ML models is notoriously difficult. Fortunately, the barrier to experimenting with multiple approaches is lowered by recent advancements in large language models (LLMs), coupled with the emerging field of prompt engineering.
To draw a comparison, picture LLMs as a toolbox with tools for handling different activities and tasks. Prompt engineering is the manual detailing the tools you should use for a particular task. The more explicit your instructions (prompt), the more capable the toolbox (LLM) will be at solving your business needs.
Leveraging the toolbox (LLMs) at our disposal in tandem with a manual (prompt engineering) brings us to a holistic approach to developing ML solutions called prompt-based development (PBD). How does this help us in practice?
Prompt engineering introduces specialized techniques that allow a machine learning model to make accurate predictions with limited labeled data — zero-shot and few-shot prompting.
Think back to the last time you were in a foreign country and came across a sign you couldn’t understand. Suddenly, you spot a familiar symbol that allows you to comprehend its message without knowing the language. Similarly, with zero-shot prompting, you prompt an LLM with no examples, which the model can still deduce, comprehend, and deliver based on the prompt and prior knowledge.
Now, imagine you took language classes and learned common phrases. During your trip abroad, you came across similar sentences with minor changes, which you could still understand. That’s the equivalent of using a few-shot prompt for an LLM. With this method, you guide the model to solve a particular task, using some examples of similar tasks so you can improve its output.
Prompt-Based Development Advantages
In the realm of healthcare data curation, traditional methods involve laboriously identifying electronic medical records, removing duplicates, and annotating data — a process stretching over months. Enter prompt-based development (PBD), where the use of large language models (LLMs) and textual prompts streamlines and quickens this task significantly. This accelerated pace allows for early assessment of technical feasibility and prompt adjustments based on data quality.
Contrast this with the rigidity of the traditional ML workflow, where changes in business requirements may necessitate retraining a model from scratch. By using PBD, we can instead change our prompting strategy and re-design the prompts used during development to accommodate changes to business requirements, enabling us to be more flexible and faster in our development approach.
Furthermore, prompt-based development presents the opportunity for parallel hypothesis testing. We can simultaneously explore other assumptions besides validating our initial hypothesis on the correlation between readmission rates and discharge time. This may include other variables, such as treatment plan adjustments, comorbidities in patients, or the variability of seasonal fluctuations in their impact on hospital readmission rates. Once we have a firm grasp on the hypotheses, we can transition to deploying the model on a live environment in “shadow mode.” This guarantees the model’s operation without impacting real-world decisions, while its inferences are scrutinized and assessed against pre-defined expectations.
The subsequent comparison of model metrics and outputs offers a pivotal moment. If the results are promising, we can consider revisiting the prototype model to develop it further into a more polished MVP and allow it to make real-world decisions. Alternatively, discrepancies in the model’s performance due to technical or ethical concerns present an opportunity to return to the drawing board and revisit our hypotheses.
Ethical and Technical Considerations
Although prompt-based development presents the opportunity for speedy delivery, shipping an ML model should be done responsibly. While it looks fantastic to build a model from scratch in a fraction of the time it usually takes, there are some crucial aspects to pay attention to, such as bias or unfairness.
Ethically, it is crucial to reduce biases and stereotypes in the data to the utmost. For example, consider a data visualization that identifies lower-income neighborhoods as high risk for hospital readmission within 30 days post-discharge. Limiting ourselves only to that data source could reinforce harmful biases. Through the help of a subject matter expert, we can introduce new and relevant data sources and adjust our prompts to include further nuances such as access to primary care and the health check frequency, resulting in a better-informed and tailored approach to patient care.
Furthermore, a subject matter expert is still required to assess the model’s performance before it can start making impactful decisions. This holds particular significance in healthcare applications and projects, especially in safeguarding a patient’s protected health information.
However, the swift nature of prompt-based development shouldn’t compromise best practices. Rigorous testing, validation, and attention to technical details, such as code quality and library vulnerabilities, are indispensable to prevent incurring technical debt. Balancing speed with ethical and technical diligence is the key to unlocking the true potential of prompt-based development.
Conclusion
Prompt-based development is transforming machine learning initiatives, enabling us to hypothesize, experiment, and strategize with different approaches. It also significantly reduces project timelines, often at a fraction of the time and cost of the traditional approach. Each organization’s journey is unique, and the emergence of prompt-based development allows it to put its value propositions to the test and validate if it is one step closer to its strategic goal. Can your current or future ML projects benefit from quicker validation? If so, consider taking the PBD approach.
Original post found here.
Authored by Georgi Naydenov and Bobby Bahov.
About Georgi Naydenov:
Georgi is a versatile professional with deep expertise and a broad understanding across multiple areas. His proficiency lies in his capability to identify, analyze in depth, and synthesize data to allow for the maximization of business value. His proficiencies extend to both healthcare and energy analytics.
His enthusiasm for data and AI is unwavering, particularly in the prompt engineering area. Over the past year, he’s been actively researching ways to integrate foundation models, such as GPT-4, Vicuna, Claude, Falcon, and Llama 2, into his daily workflows and projects. Georgi is enthusiastic about providing guidance on mitigating language models’ hallucinations and optimizing their efficiency, thereby enabling the implementation of the models’ output for tangible product development. In his leisure time, Georgi is infatuated with metalcore concerts and mosh pits that complement his journey of self-learning the guitar.
About Bobby Bahov:
With more than a decade of experience in software product development, Bobby seamlessly navigates the intersection of cutting-edge technology and business efficacy. Starting his career as a software developer, he quickly advanced into project and product management roles. He has managed projects and consulted a varied clientele — from startups to large corporations, as well as governmental organizations and NGOs. He has also spearheaded multiple innovative ventures and programs focused on AI tools, satellite data, and space technology.
An alumnus of the Rotterdam School of Management with a Master’s degree in Business Information Management, he’s currently pursuing a Ph.D. researching AI simulations, digital twins, and synthetic data. Today, as a Senior Business Analyst at Mentormate, he continues to be a driving force for innovation — and make no mistake, he’s still one of the biggest geeks you’ll find around.
