Data Processing Layer Optimization in an RTB Platform

About the Client

HIRO Media is a worldwide Internet media company that’s been delivering programmatic video advertising technology services for ad buyers, ad sellers, and content creators since 2006. It has grown to become one of the industry leaders providing one of the most advanced Agile prediction-based RTB systems. Their proprietary technology enables publishers and advertisers to optimize their performance and unfold new revenue streams.

Technologies Used

Goal

Initially, the client used Microsoft SQL Server, which requires yearly license payments per CPU core. As a result, the total cost of ownership was increasing because of the increase in volume. The client wanted to optimize spend, and migrate to Hadoop. It entailed a grounds-up reengineering of the data processing layer.

Work Description

As our first step, we explored the way the original system worked. We examined the Bid Data pipeline operations on both high- and low-levels and analyzed the ETL process between input and output servers. Then we developed a migration plan based on the accumulated data, which comprised two main tasks:

• Rewrite the data processing code to use Hadoop and create required Hadoop Jobs;
• Carry out data migration from SQL servers to the Hadoop ecosystem.

Our team closely interacted with the client, conducting daily meetings via Skype, and providing weekly builds through the TeamCity CI server. We employed TDD practices ensuring high-quality standards.

Work Stages

1. Examine the existing architecture
2. Draw up a migration plan
3. Rework the existing code
4. Execute data migration
5. Provide Quality Assurance
6. Deploy in production

Results

As a result, the client was able to cut down the total costs of ownership several times. Moreover, the new system introduced greater scalability and decreased batch processing time.

Gallery