Data Scientist Shortage in the Generative AI Era

Today, the world of technology is changing at such a fast rate, with generative AI making huge impacts throughout organizations in terms of productivity and efficiency. The media is filled with negative talk of impending job losses and concerns that humans will be ousted from their work due to the rate of current advances. But is this fact or fiction? At smartR AI we believe there will be another outcome, and that organizations will need to bolster their teams with relevant skills to take full advantage of the opportunities that lie ahead with generative AI. A good example is that of data scientist professionals, who are already in short supply.

One of the most promising areas to make data scientists more productive is within generative AI. The rise of generative AI will provide data scientists powerful new capabilities to organize, synthesize and analyze data. This enhanced productivity will uncover deep insights and enable data-driven decision-making at unprecedented levels. However, while generative AI will increase the productivity of data scientists, I believe it will lead to an even more demand for data scientists.

When spreadsheets were introduced in the 1980s the world of finance was transformed. However, rather than losing jobs, finance grew from about 10% of U.S. GDP in 1980 to 20% by the early 2000s. While some low-level jobs were replaced, such as bank tellers, the headcount grew with new high skilled jobs in trading, financing, hedge funds, and other financial services.

A similar revolution is about to take hold within the data science environment due to generative AI. Mirroring the 1980s spreadsheets phenomenon that permeated business functions in the 1980s and 90s, generative models enable data experts to provide and deliver a far greater impact. With generative AI’s exponential data growth across industries, this emerging skill is crucial for competitiveness. Rather than reducing headcount, generative AI will drive a surging demand for data scientists familiar with its capabilities and limitations. Yet market dynamics fail to provide adequate data science talent.

Even before AI, the world faced a dangerous shortage of data professionals. According to Microsoft’s DEGREE + DIGITAL report (based on LinkedIn data), there’s no skill set that shows a more significant disparity between supply and demand. Also, based on the U.S. Bureau of Labor Statistics projections, the demand isn’t expected to let up anytime soon. The Bureau’s Office of Occupational Statistics and Employment Projections reports that the data science field is expected to grow by 36% from 2021-2031 – significantly faster than the average profession.

Additionally, external unstructured data is growing 62% annually (Gartner, Beauhurst et al). This type of data has been hard to utilize, as it doesn’t lend itself to standard analysis.  The proliferation of data new platforms, like JIRA, slack and Teams, has only added to the data chaos.

Generative AI has the ability to absorb and organize all data, much like the way spreadsheets organized financial data. This tool will allow data scientists to do more, and faster than ever before making data scientists much more valuable than ever before, thus stimulating the demand for this skillset.

With generative AI, you can pull data from multiple sources, such as databases, emails, collaborative tools, data lakes, SharePoint, spreadsheets, documents, PDFs, Websites, CRM, and ERP systems, and combine it into a cohesive whole. Instead of struggling to figure out how to extract the information from all these different sources, you can now just ask plain English questions. That ability to interact and experiment quickly allows data scientists to rapidly prototype and find the right data sources to answer questions, and it’s handed to them within moments.

By providing data scientists with the ability to combine disparate data sources, and mine data more effectively, will lead to deeper insights.

At smartR AI, I like to say “the world is drowning in data, but starving for knowledge.” Generative AI bridges this gap. By quickly and efficiently organizing data, that knowledge can now be extracted much more easily.

But ease of use isn’t the only factor that needs to be considered. We all know data is knowledge, and knowledge is power, and therefore your valuable data always needs to be kept private and secure. Who wants to gain deep insights on your market, and then find your competitors and other major technology corporations have access to that valuable data? So, my advice is this: choose, implement, and use a generative AI solution that ensures privacy, ease of use, and make sure you work with a company that continuously innovates. Innovation should be a daily habit

In a world where rapid change in technology is the norm, as it’s becoming with generative AI, the drought in data scientists is a critical factor that needs to be taken seriously. However, generative AI solutions can help organizations overcome some of their concerns, by providing existing staff with the ability to gain deep insights, analyze and supercharge data more easily and effectively without the knowledge this scarce commodity, data scientists, bring to the table.