redis
LearnHowTos & TutorialsSolutionsCaching architectureHow to use Redis for Write-behind Caching

Learn

How to use Redis for Write-behind Caching

Prasan Kumar
Author
Prasan Kumar, Technical Solutions Developer at Redis
Will Johnston
Author
Will Johnston, Developer Growth Manager at Redis
GITHUB CODE

Below are the commands to clone the source code (frontend and backend) for the application used in this tutorial

git clone https://github.com/redis-developer/ebook-speed-mern-frontend.git

git clone https://github.com/redis-developer/ebook-speed-mern-backend.git

What is write-behind caching?#

Imagine you've built a movie streaming app. You used MongoDB as your data store, and as you needed to scale you implemented caching using Redis. This allows you to drastically speed up reads. However, now you are experiencing slowness when writing to MongoDB.

For example, maybe you want to allow users to continue watching movies where they last left off. This requires you to store the timestamp of where a user is when they decide to pause the movie. With millions of users, this is starting to cause MongoDB to slow down when you have peaks in demand.

You need a way of flattening the peaks in demand, allowing you to write data quickly and then persist it to MongoDB when the demand dies down. What you need is called the "write-behind pattern."

The pattern is simple, your application writes data to Redis and then asynchronously data gets written to MongoDB. Write operations are queued up so that the application can move on quickly and the cache can catch up over time. However, this does mean there is a short time when the data between the cache and the system of record is inconsistent.

Below is a diagram of the write-behind pattern for the application:

The pattern works as follows:

  1. 1.The application reads and writes data to Redis.
  2. 2.Redis syncs any changed data to the MongoDB database asynchronously.

There are two related write patterns, and the main differences between them are as follows

Write Behind

Write through

Syncs data asynchronously

Syncs data synchronously/ immediately

Data between the cache and the system of record (database) is inconsistent for a short time

Data between the cache and the system of record (database) is always consistent

Learn more about Write through pattern

Why you should use Redis for write-behind caching#

Consider Redis with this pattern when you need to

  1. 1.Flatten peaks in demand: Under stress, an application may need to write data quickly. If your application needs to perform a large number of write operations at high speed, consider Redis. The programmability capabilities of Redis make sure the data stored in the cache is synced with the database.
  2. 2.Batch multiple writes: Sometimes it's expensive to write to a database frequently (for example, logging). In those cases, it can be cost-effective to batch the database writes with Redis so that data syncs at intervals.
  3. 3.Offload the primary database: Database load is reduced when heavy writes operate on Redis, So we can spread writes to improve performance during the peak time of application usage.

Redis programmability for write-behind caching using RedisGears#

Tip

You can skip reading this section if you are already familiar with RedisGears)

What is RedisGears?#

RedisGears is a programmable serverless engine for transaction, batch, and event-driven data processing allowing users to write and run their own functions on data stored in Redis.

Functions can be implemented in different languages, including Python and C, and can be executed by the RedisGears engine in one of two ways:

  1. 1.Batch: triggered by the Run action, execution is immediate and on existing data
  2. 2.Event: triggered by the Register action, execution is triggered by new events and on their data

Some batch type operations RedisGears can do:

  • •Run an operation on all keys in the KeySpace or keys matching a certain pattern like :
    • •Prefix all KeyNames with person:
    • •Delete all keys whose value is smaller than zero
    • •Write all the KeyNames starting with person: to a set
  • •Run a set of operations on all(or matched) keys where the output of one operation is the input of another like
    • •Find all keys with a prefix person: (assume all of them are of type hash)
    • •Increase user's days_old by 1, then sum them by age group (10-20, 20-30 etc.)
    • •Add today's stats to the sorted set of every client, calculate last 7 days average and save the computed result in a string

Some event type operations RedisGears can do:

  • •RedisGears can also register event listeners that will trigger a function execution every time a watched key is changed like
    • •Listen for all operations on all keys and keep a list of all KeyNames in the KeySpace
    • •Listen for DEL operations on keys with a prefix I-AM-IMPORTANT: and asynchronously dump them in a "deleted keys" log file
    • •Listen for all HINCRBY operations on the element score of keys with a prefix player: and synchronously update a user's level when the score reaches 1000

How do I use RedisGears?#

Run the Docker container:

docker run -p 6379:6379 redislabs/redisgears:latest

For a very simple example that lists all keys in your Redis database with a prefix of person: create the following python script and name it hello_gears.py :

gb = GearsBuilder() gb.run('person:*')

Execute your function:

docker exec -i redisgears redis-cli RG.PYEXECUTE "`cat hello_gears.py`"

Using gears-cli#

The gears-cli tool provides an easier way to execute RedisGears functions, specially if you need to pass some parameters too.

It's written in Python and can be installed with pip:

pip install gears-cli
gears-cli hello_gears.py REQUIREMENTS rgsync

Usage:

gears-cli --help
usage: gears-cli [-h] [--host HOST] [--port PORT]
[--requirements REQUIREMENTS] [--password PASSWORD] path [extra_args [extra_args ...]]

RedisGears references#

Write behind caching in a NodeJS application with Redis and MongoDB#

Demo application#

The demo application used in the rest of this tutorial showcases a movie application with basic create, read, update, and delete (CRUD) operations.

The movie application dashboard contains a search section at the top and a list of movie cards in the middle. The floating plus icon displays a pop-up when the user selects it, permitting the user to enter new movie details. The search section has a text search bar and a toggle link between text search and basic (that is, form-based) search. Each movie card has edit and delete icons, which are displayed when a mouse hovers over the card.

GITHUB CODE

Below are the commands to clone the source code (frontend and backend) for the application used in this tutorial

git clone https://github.com/redis-developer/ebook-speed-mern-frontend.git

git clone https://github.com/redis-developer/ebook-speed-mern-backend.git

To demonstrate this pattern using the movie application, imagine that the user opens the pop-up to add a new movie.

Instead of the application immediately storing the data in MongoDB, the application writes the changes to Redis. In the background, RedisGears automatically synchronizes the data with the MongoDB database.

Programming Redis using the write-behind pattern#

Developers need to load some code (say python in our example) to the Redis server before using the write-behind pattern (which syncs data from Redis to MongoDB). The Redis server has a RedisGears module that interprets the python code and syncs the data from Redis to MongoDB.

Loading the Python code is easier than it sounds. Simply replace database details in the Python file and then load the file to the Redis server.

Create the Python file (shown below, and available online). Then update the MongoDB connection details, database, collection, and primary key name to sync.

movies-write-behind.py
# Gears Recipe for a single write behind

# import redis gears & mongo db libs
from rgsync import RGJSONWriteBehind, RGJSONWriteThrough
from rgsync.Connectors import MongoConnector, MongoConnection

# change mongodb connection (admin)
# mongodb://usrAdmin:[email protected]:27017/dbSpeedMernDemo?authSource=admin
mongoUrl = 'mongodb://usrAdmin:[email protected]:27017/admin'

# MongoConnection(user, password, host, authSource?, fullConnectionUrl?)
connection = MongoConnection('', '', '', '', mongoUrl)

# change MongoDB database
db = 'dbSpeedMernDemo'

# change MongoDB collection & it's primary key
movieConnector = MongoConnector(connection, db, 'movies', 'movieId')

# change redis keys with prefix that must be synced with mongodb collection
RGJSONWriteBehind(GB,  keysPrefix='MovieEntity',
                  connector=movieConnector, name='MoviesWriteBehind',
                  version='99.99.99')
WHAT IS A REDISGEARS RECIPE?

A collection of RedisGears functions and any dependencies they may have that implement a high-level functional purpose is called a recipe. Example : "RGJSONWriteBehind" function in above python code

There are two ways to load that Python file into the Redis server:

  1. 1.Using the gears command-line interface (CLI)

Find more information about the Gears CLI at gears-cli and rgsync.

# install
pip install gears-cli
# If python file is located at “/users/tom/movies-write-behind.py”
gears-cli --host <redisHost> --port <redisPort> --password <redisPassword> run /users/tom/movies-write-behind.py REQUIREMENTS rgsync pymongo==3.12.0

2. Using the RG.PYEXECUTE from the Redis command line.

Find more information at RG.PYEXECUTE.

# Via redis cli
RG.PYEXECUTE 'pythonCode' REQUIREMENTS rgsync pymongo==3.12.0

The RG.PYEXECUTE command can also be executed from the Node.js code (Consult the sample Node file for more details)

Find more examples at Redis Gears sync with MongoDB.

Verifying the write-behind pattern using RedisInsight#

Tip

RedisInsight is the free redis GUI for viewing data in redis. Click here to download.

The next step is to verify that RedisGears is syncing data between Redis and MongoDB.

Insert a key starting with the prefix (that's specified in the Python file) using the Redis CLI

Next, confirm that the JSON is inserted in MongoDB too.

You can also check RedisInsight to verify that the data is piped in via Streams for its consumers (like RedisGears).

How does all that work with the demo application? Below is a code snipped to insert a movie. Once data is written to Redis, RedisGears automatically synchronizes it to MongoDB.

BEFORE (using MongoDB)
...
//(Node mongo query)
if (movie) {
  //insert movie to MongoDB
  await db.collection("movies")
           .insertOne(movie);
}
...
AFTER (using Redis)
...
//(Redis OM Node query)
if (movie) {
  const entity = repository.createEntity(movie);
  //insert movie to Redis
  await moviesRepository.save(entity);
}
...

Ready to use Redis for write-behind caching?#

You now know how to use Redis for write-behind caching. It's possible to incrementally adopt Redis wherever needed with different strategies/patterns. For more resources on the topic of caching, check out the links below:

Additional resources#

Caching with Redis

General

Last updated on Aug 28, 2024
PreviousSource Code Tip
NextHow to use Redis for Write through caching strategy