AI LLMs Predict Job Transitions and Redefine Data Science

Artificial intelligence (AI), particularly large language models (LLMs), is revolutionizing data science by enabling the incorporation of vast amounts of information into statistical models. LLMs, which are trained on next-word prediction, have become highly proficient at generating coherent text. This capability has led researchers to explore whether LLMs can predict more than just words, such as job transitions.

Predicting Job Transitions with LLMs

Social scientists are interested in understanding career evolution for various reasons, including:

• Government planning: Predicting job transitions can help governments prepare for economic disruptions caused by declines in sectors like retail or manufacturing, or by AI-driven job displacement.
• Company support: Companies can use these predictions to design programs that help employees manage their career paths or assist laid-off workers.

To investigate this, a team at the GSB Capital Social Impact Lab at Stanford GSB utilized a database containing 40 years of worker survey data about careers. They translated this survey data into textual resumes and fed them into an open-source LLM, effectively customizing it. The model achieved astounding accuracy, predicting a person's next job more effectively than any other model.

The core insight was that, to an LLM, a sequence of jobs is analogous to a sequence of words.

Broader Applications Beyond Employment

This methodology is not limited to employment. The techniques used to model worker careers can be applied to analyze data in various other domains:

• E-commerce: Analyzing customer journeys.
• Medicine: Understanding sequences of tests, evaluations, and treatments for patients.

The Importance of Fine-Tuning

While LLMs possess extensive knowledge, they often lack the granularity required for specific predictions, such as a worker's next job given their entire work history. To address this, the researchers employed a method called fine-tuning.

Fine-tuning involves:

1. Starting with an off-the-shelf AI model developed by large companies.
2. Training this model further on a particular field of expertise.

This process allows the AI to become highly proficient at a specific task. For example, through fine-tuning, the AI not only learns that an engineering manager often follows an engineer role but also understands how the probability of becoming an engineering manager changes with increasing experience as an engineer.

The Future of Data Science with AI

The ability to adapt AI tools designed for language analysis to solve diverse problems across various scientific fields, including those not traditionally considered "text problems," is a significant development. This marks just the beginning of a new era in data science.

The future of data science will likely involve:

• Hybrid systems: Combining fine-tuned LLMs with rules-based decision-making and traditional machine learning.
• Increased efficiency: Widespread adoption of customized data science methods will enhance efficiency for companies, including small businesses.
• Diverse applications: These methods could assist in software development, business optimization, and even predicting medical treatment outcomes.

The advent of these AI capabilities is fundamentally changing the process of scientific discovery, prompting a re-evaluation of established methodologies.