Experience of the Architectural Evolution of a Big Data System

Abstract

This paper presents the evolution of a hybrid Big Data architecture over 7 years to adapt to changes in user requirements and technological evolution. This architecture is developed and used by one of the main telco companies in our country. Currently, the result of the presented work is used in many projects providing key information for the operation of the company.

We describe the initial architecture, its main shortcomings, and the current architecture as well as the challenges we met in the process. The main lessons learned are related to the need of modularity and flexibility in the architecture of Big Data systems.