In 2012, this Harvard Business Review article argued that data science will be the sexiest job in the 21st century. At the time, data science was new and unheard of, with companies eager to use data scientists to revolutionize their practices. Is it still the sexiest job now? Well sort of, but not really. The field has gone through some significant transformations since these “wild west” early days. Now, data science as a discipline has become more streamlined and specialized.
Often key data scientists have slightly different titles like machine learning specialist, data engineers, etc. Machine learning and AI technology have changed the way data is processed and analyzed. This has automated parts of the tasks that data scientists have spent a long time working on, such as data cleaning (which still can take a long time) and initial data exploration, shifting the work necessary for humans to perform in the field to more specialized and fringe tasks. For example, many data scientists have become machine learning specialists focusing on fine-tuning these models or communication specialists focusing on how to use their business expertise to communicate complex findings with stakeholders and help decide what they should do about the results.
I think more than the technology, what has driven the specialization is the routinization of data science processes within an organization. Gone are the days of a lone data scientist at a company doing cutting edge work by themselves just figuring out what is possible. Data science as a field has fallen within the discipline and expectations of corporate bureaucracies. In its early stages, most data scientists worked alone or in small teams doing pioneering, experimental work figuring out how to apply the tools of the field to their organization in ways that people did not know were possible. That can still be the case. In every job I have had as a data scientist, for example, I have been the first data scientist in the entire organization or specific department I work in. But this is increasingly rare. Data science is now mostly one department at an organization, doing important but predictable routine work. All white collar professions get grafted in the “corporate machine” like this overtime.
Recent AI technology has contributed to this too by automating many of the low-level data processing and analyzing tasks so that non-specialists can perform them on their own. This is great, increasing the accessibility of tasks once considered obscure or even “magical” by regular people. Back in the day, to do much of any data modeling, you had to code it yourself, requiring a level of programming knowledge that was beyond a typical office worker or manager. That’s why they needed to hire a data scientist to analyze the data themselves. I hope in the long run using AI tools to tinker with data themselves and try out different theories will increase the data literacy and skillsets of regular professionals. It also means that data scientists are increasingly spending less time on these tasks and have moved to more complex, specialized work that still require quite a bit of technical human thinking.
Another factor that has driven this routinization is the increase in the number of people studying and doing data science. As demand for data science increase, more people have tried to become a data scientist, whether by receiving a degree in it or transitioning their careers into the field. This has led to more data scientists in the market. If this trend continues, eventually the field will become oversaturated, but the demand still seems to be higher than the supply, with more open jobs than people able to fill them.
This has still redefined what data science is. When many people join a field, it becomes difficult to maintain the same level of pioneering eclecticism. Instead, the types of tasks people do become routinized and standardized to provide consistency for a larger number of people, paralleling the transformation Max Weber describes religious movements undergoing from a charismatic leader to a routine social institution.
All of this leads to the current state of data science. This is not necessarily bad, but it is different. So, is data science still the sexiest job? Yes and no. Some of its specialist roles like machine learning specialist, I think, better maintain the excitement and cutting edge of that moniker. It’s still in high-demand, however, a fine field to work in.
