Not All Software Engineers Can Become Good Data Engineers

Abstract

The amount of data that businesses collect and analyze has been rapidly increasing, which has triggered an increase in big data teams. With the growth of both the number and size of big data teams, specialized roles are starting to be defined. One such role is the data engineer, who focuses on ensuring that the data is easily available for advanced analytics. Via a case study, this paper explores the role of the data engineer and the key characteristics that enable someone to be a good data engineer. The paper also explores if good software engineers could become good data engineers. Our findings show that the knowledge and skills required to be a data engineer are significantly different from those required to be a software engineer. Hence, not surprisingly, we found that that not all software engineers could become good data engineers.