firebase

Take control of your backend with Firebase Cloud Functions (II)

Sprint Board Guardian and Project Happiness Keeper. Dog walker, husband and enthusiastic sci-fi reader.

Use Firebase Realtime Database to implement an easy yet powerful API cache for your mobile apps.

In the previous post, we learned how to use Firebase Cloud Functions to clean up a backend response and make it more mobile friendly.

In this post, we are going to show how to use Firebase Realtime Database to save that cleaned response, using it as a cache. This will prevent calling the original backend API too frequently as well as unnecessary transformations.


Why use a cache at all?

The cloud function we developed in part one was used to fetch raw data from our backend and transform it into something easier to process by a mobile app.
But, in the majority of cases, there’s no need to fetch new fresh data every time a client requests it: cheaper, cached data is good enough.

What to cache and for how long will depend on several factors unique to your case:

In all these cases it seems wise to establish a data validity policy and rely on already cleaned cache data instead of fetching the raw data for every request.

As an added bonus, having our cached data in Firebase allows us to share cache logic across multiple clients.

Persist cleaned up model to Firebase

Continuing with the example introduced in part one, what we are going to save is the cleaned up feed from The Guardian. The feed, once cleaned looks like this:

There are three things to consider for this example:

  1. Saving cleaned up data into our project’s Firebase database.
  2. Checking if there is valid cached data before fetching new.
  3. Cache invalidation policy: deciding when this cache is not valid anymore.

1. Saving transformed data into Firebase database

In this gist, you can find the full code for fetching and transforming data we did in part one.

Let’s start by refactoring that code into something more Promising

exports.fetchGuardian = functions.https.onRequest((req, res) => {  
   return request(URL_THE_GUARDIAN)
       .then(data => cleanUp(data))
       .then(items => response(res, items, 201))
});

function request(url) {  
   return new Promise(function (fulfill, reject) {
       client.get(url, function (data, response) {
           fulfill(data)
       })
   })
}

function response(res, items, code) {  
   return Promise.resolve(res.status(code)
       .type('application/json')
       .send(items))
}

This code is equivalent to the one in the previous post, but by using Promises, the flow is easier to understand and modify.

I’m omitting here the cleanUp function as it’s not relevant, but check the previous post if you are interested in it.

The first thing we are going to do is saving items in Firebase database before returning them to the client. That can be done by modifying the previous code and adding a call to save(items) in the Promises chain:

exports.fetchGuardian = functions.https.onRequest((req, res) => {  
   return request(URL_THE_GUARDIAN)
       .then(data => cleanUp(data))
       .then(items => save(items))
       .then(items => response(res, items, 201))
});

function save(items) {  
   return admin.database().ref('/feed/guardian')
       .set({ items: items })
       .then(() => {
           return Promise.resolve(items);
       })
}

With admin.database().ref(‘/feed/guardian’) we obtain a reference to a path in our database.
.set({items: items}) pushes the items array to that path in the database with the key "items" and returns an empty Promise.

Finally, when the promise is fulfilled (meaning the writing process is done) we return a new Promise with the original items array to continue the chain.

In Firebase this will generate:

2. Checking cached data before fetching new

At this point, we are persisting the cleaned up data in our database but we are not doing anything with it.

The next step is to check if there’s saved data in Firebase database before making an HTTP request to our backend.

exports.fetchGuardian = functions.https.onRequest((req, res) => {  
   return admin.database().ref('/feed/guardian')
       .once('value')
       .then(snapshot => {
           if (isCacheValid(snapshot)) {
               return response(res, snapshot.val(), 200)
           } else {
               return request(URL_THE_GUARDIAN)
                   .then(data => cleanUp(data))
                   .then(items => save(items))
                   .then(items => response(res, items, 201))
           }
       })
});

function isCacheValid(snapshot) {  
   return (snapshot.exists())
}

What changes from step 1 is that we read from the database before fetching the feed.

As we already know, admin.database().ref('/feed/guardian') is a path in the database. With .once('value') we read the values at that path once and return a Promise with them.

What we do afterward is simple:

3. Cache invalidation policy

Finally, we need to set rules for cache validity. To keep this example simple we are going to consider the data is valid for 1h since it’s saved. After that time, the next request should fetch new items and replace the cached ones with them.

To do that we need to save the fetching time along with the items when we persist them. We need to modify our save function like this:

function save(items) {  
   return admin.database().ref('/feed/guardian')
       .set({
           date: new Date(Date.now()).toISOString(),
           items: items
       })
       .then(() => {
           return Promise.resolve(items);
       })
}

This will produce a new field in our database:

The last bit is to use this date to check how old our cached data is. If the cached data was saved less than one hour ago we’ll consider it valid. Otherwise, we invalidate the cache by fetching fresh data and overriding it.

function isCacheValid(snapshot) {  
   return (
       snapshot.exists() &&
       elapsed(snapshot.val().date) < ONE_HOUR
   )
}
function elapsed(date) {  
   const then = new Date(date)
   const now = new Date(Date.now())
   return now.getTime() - then.getTime()
}

All together

The code is also available here as a gist snippet

const ONE_HOUR = 3600000

var functions = require('firebase-functions');  
const URL_THE_GUARDIAN = "https://www.theguardian.com/uk/london/rss"

var Client = require('node-rest-client').Client;  
var client = new Client();

const admin = require('firebase-admin');  
admin.initializeApp(functions.config().firebase);

exports.fetchGuardian = functions.https.onRequest((req, res) => {  
    var lastEdition = admin.database().ref('/feed/guardian');
    return lastEdition
        .once('value')
        .then(snapshot => {
            if (isCacheValid(snapshot)) {
                return response(res, snapshot.val(), 200)
            } else {
                return request(URL_THE_GUARDIAN)
                    .then(data => cleanUp(data))
                    .then(items => save(lastEdition, items))
                    .then(items => response(res, items, 201))
            }
        })
});

function save(databaseRef, items) {  
    return databaseRef
        .set({
            date: new Date(Date.now()).toISOString(),
            items: items
        })
        .then(() => {
            return Promise.resolve(items);
        })
}

function request(url) {  
    return new Promise(function (fulfill, reject) {
        client.get(url, function (data, response) {
            fulfill(data)
        })
    })
}

function response(res, items, code) {  
    return Promise.resolve(res.status(code)
        .type('application/json')
        .send(items))
}

function isCacheValid(snapshot) {  
    return (
        snapshot.exists() &&
        elapsed(snapshot.val().date) < ONE_HOUR
    )
}

function elapsed(date) {  
    const then = new Date(date)
    const now = new Date(Date.now())
    return now.getTime() - then.getTime()
}

function cleanUp(data) {  
    const items = []
    const channel = data.rss.channel

    channel.item.forEach(element => {
        item = {
            title: element.title,
            description: element.description,
            date: element.pubDate,
            creator: element['dc:creator'],
            media: []
        }
        // Iterates through all the elements named '<media:content>' extracting the info we care about
        element['media:content'].forEach(mediaContent => {
            item.media.push({
                url: mediaContent.$.url,                // Parses media:content url attribute
                credit: mediaContent['media:credit']._ // Parses media:cretit tag content
            })
        });
        items.push(item);
    });
    return Promise.resolve(items);
}

Stay tuned!

This is part two of a three-part series of articles about Firebase.

In the final installment we'll learn how to use Google Cloud Natural Language API from our Firebase Cloud Function to enrich the backend response, for example, adding sentiment analysis.


Find me on Twitter @lgvalle, I'd love to chat about Android, Firebase & Cloud Functions.

Enjoyed this article? There's more...

We send out a small, valuable newsletter with the best stories, app design & development resources every month.

No spam, no giving your data away, unsubscribe anytime.

About Novoda

We plan, design, and develop the world’s most desirable software products. Our team’s expertise helps brands like Sony, Motorola, Tesco, Channel4, BBC, and News Corp build fully customized Android devices or simply make their mobile experiences the best on the market. Since 2008, our full in-house teams work from London, Liverpool, Berlin, Barcelona, and NYC.

Let’s get in contact

Stay in the loop!

Hear about our events, blog posts and inspiration every month

Subscribe to our newsletter