Big Data and Data Lake Platform Implementation on AWS for an Agribusiness Multinational

01The Client

Large French multinational in the agribusiness sector, with operations in Brazil, seeking to become a data-driven organization.

02The Challenge

• Data duplication across systems and organizational departments
• Lack of data governance and formalized security policies
• Security risks and non-compliance with LGPD (Brazilian data protection law)
• Low efficiency in the use and access to corporate information
• Fragmented and unstandardized data, hindering decision-making

03Implemented Solution

• Implementation of a Big Data and Data Lake platform on AWS using S3, Spark, Kinesis, DynamoDB, Redshift, and Athena
• Design of a scalable architecture for data ingestion, storage, and consumption in the cloud
• Data governance with catalog, data dictionary, lineage, and LGPD-aligned security policies
• Integration of data sources via ETL/ELT pipelines with Talend
• Development of inbound and outbound APIs for standardized data integration and consumption
• Deployment of an internal API Marketplace for business units to access, test, and consume data

04Strategic Differentials

• Phased approach ensuring a solid infrastructure foundation before environment evolution
• Data governance by design, with complete technical documentation (manuals, policies, data dictionary)
• LGPD compliance natively integrated into the data architecture
• API Marketplace as a mechanism for democratizing and standardizing data access for internal users

05Results Achieved

• Elimination of data duplication and fragmentation with a single, reliable source
• Greater agility in accessing and consuming information across business units
• LGPD compliance and reduced data security risks
• Qualified decision-making support with traceable, high-reliability data
• Scalable environment capable of absorbing new data sources without architectural rework