About the Client

Our customer was large company that provided editorial management software for journals, media organizations, publishers, content makers, and other market players. They worked with more than 200 partners worldwide and had 20,000 active users monthly.

Goal

The idea was to create a pipeline that could pull articles from the Internet and yield the extracted knowledge. The pipeline was conceived to consist of steps, or components. The extracted knowledge was to be saved into a database and then arranged for possible use. Each component consumed and produced data items of certain type.

Work Description

We divided the projected system into three main modules and dealt with each one separately:

1. Retrieval Perform periodic and/or real-time data retrieval from external data sources through available APIs and HTML content scraping techniques.

2. Processing Analyse, categorise, and store the retrieved data in a manner most efficient for end user presentation.

3. API Presentation A set of RESTful services that allow the front-end to consume available data on a per-user basis.

Work Stages

1. New customer-specific architecture design
2. System implementation
3. Pilot design; user feedback acquisition
4. Feedback-based system rework
5. Project transfer to open beta test.

Results

The technical solution we delivered allowed the client to significantly reduce the amount of manual labor done by their content team and optimize their efficiency by 300%. As a result, the client was able to reduce operational expenses and redirect resources to other projects.