Data Processing Layer Optimization In An RTB Platform

Case Study

Big Data

6 input SQL Servers;

Data processing cluster;

20 output SQL Servers;

1MM logical transactions per day;

80TB of data.

Challenge

Decrease cost of ownership;

Decrease batch processing time;

Introduce more scalability.

How we approached the project

Before: 28 “output” servers (2 virtualized servers per 1 physical unit). 2 Windows Server licenses for 8 cores each = 16 per physical unit. 2 MS SQL Server licenses for 8 cores each = 16 per physical unit.
  • Analysis

    SQL is not good at storing and processing big data;

    Microsoft “per-core” licensing model for Windows and SQL Server is not cost effective for big data processing.

  • Proposal

    Replace SQL “output” servers with a Hadoop cluster running on Ubuntu Server nodes;

    Adapt the processing logic to work with Hadoop.

  • Incremental development

    Weekly sprints;

    Daily reports to the customer during the Skype stand-up meetings;

    Weekly builds via continuous integration server TeamCity;

    Ensured high quality standards by employing TDD practices.

  • After:

    28 “output” servers (2 virtualized servers per 1 physical unit);

    28 Ubuntu Server nodes (unlimited cores);

    Hadoop.

Our clients say

Let Us Contact You

  • Fill out the form below and we'll get in touch within 24 hours