Performance Saver for Hot Topic Issue

A mini-guide about searching for hot topic

Photo credit: Pixabay

We designed and implemented a lot of search platforms like this blog mentioned: Tricks to make your search 10X faster using Solr and Cassandra. We can gain many benefits from search engines and NoSQL databases. Please read the above blog if you have any design questions.

In this blog, we will discuss some other situations we may face when designing or even after implementing the search platform.

Agenda

  • Situation statement and motivation
  • New approaches
  • Benefits from new design
  • Summary

Situation statement and motivation

After considering the speed is our first citizen at here and the data size, we really consider NoSQL plus Search Engine together.

In some situations, instead of just fetching raw data, we still need to deal with how to deal with aggregation data. Then MongoDB will be a good choice as detail-level storage. And CosmosDB-Mongo API will be another good choice for HIPAA data.

In our previous blog, we are using the below initial design to achieve those requirements.

Figure 1 - initial design

Based on Figure 1, the performance of our process does well until facing some hot topics data points. We have around 0.3% of data with massive records (more than 500,000). Because CosmosDB has some performance issues, we tried to do more brainstorming to find two backup plans below:

  1. Doing pre-calculation before loading,
  2. Horizontally splitting the hot topic data — just put the 0.3% data into another collection + Improve the index design.

After doing the testing and PoC for those methods, there are some weak parts.

  1. There won’t have dynamic aggregation if we use pre-calculation because we cannot do one more level aggregation for some methods such as mod, mid, or more complicated processes.
  2. Because the data size is super large even for 0.3% of data, even if those data are all under one shard, the ComosDB still needs to process those amounts of data which impacts the performance when doing aggregation. The worse case will case the timeout issue.

New approaches

Challenges

  • How to use existing kits to solve this problem?
  • How can we balance effort and performance?
  • How can we provide dynamic aggregation?

We find a way to do an Elastic Search as a “Load Balancer” with some minor changes which can help with the performance. I propose two methods below, with some tradeoffs between fetching performance and dynamic aggregation.

Solutions

  1. Comprehensive dynamic aggregation and acceptable performance
Figure 2 - comprehensive dynamic aggregation and acceptable performance

In Figure 2, we keep the same data in CosmosDB Mongo and BigQuery. In the Elastic Search index, we have one field to decide which source the user can fetch the data from (CosmosDB Mongo or BigQuery).

In this design, we have the following pros and cons:

  • We don’t have complex loading logic because we mimic the same data on both databases.
  • We have high flex for aggregation.
  • We can reduce the fetching time to 10 seconds, but it is still not a perfect performance.

2. Part dynamic aggregation and super performance

Figure 3 - part dynamic aggregation and super performance

In Figure 3, the design principle is the same as in figure 2. We replace the BigQuery with another Mongo collection and do the pre-calculation for all kinds of aggregation.

In this design, we have the following pros and cons:

  • We will have complex loading and fetching logic because there are two different data formats — flatten and aggregation.
  • We have an aggregation limitation for 0.3% of data.
  • We can have perfect fetching performance

Benefits from new designs

I created a metric for normal and hot topics among all three methods. Both new approaches improve fetching time from 30 seconds to 10 seconds or even milliseconds.

+-----------+------------+---------------------+-----------------+
| Size | CosmosDB | CosmosDB+BQ+nonagg | CosmosDB+agg |
+-----------+------------+---------------------+-----------------+
| normal | 267 (ms) | 267 (ms) | 267 (ms) |
| hot topic | 29383 (ms) | 10000 (ms) | 200(ms) |
+-----------+------------+---------------------+-----------------+

Besides Improving fetching time, we can gain the following benefits also:

  • Easy to setup for both new approaches
  • Elastic Search as a “Load Balancer”
  • Backup
  • Saving huge costs for using additional sources.

Summary

Using new approaches can improve some search platform issues quickly. There are still some other potential solutions that we can discuss in our future blog.

Performance Saver for Hot Topic Issue was originally published in Walmart Global Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Article Link: Performance Saver for Hot Topic Issue | by Jiazhen Zhu | Walmart Global Tech Blog | Medium