@adlrocha - How to make your web app work offline
Service workers, caches, IndexedDB and PWA.
I am working on a side-project where I am building a web app. I want to be able to use this application from any of my devices (laptop, mobile device, etc.). My network connection is usually stable when I use it from my laptop, but this is not always the case from my mobile device, so I started wondering, “what can I do to ensure that my app works also offline, and that all my work and interactions with it are saved and successfully synced once I recover my Internet access?”.
I thought at first that finding all the documentation I required to approach this task would be straightforward, “someone must have done this before, right?”. Nothing could be further from the truth. I had a hard time until I managed to find all the concepts I needed in order to successfully take my app offline. Consequently, this publication is an attempt to collect in the same place all the technologies you will find useful when embarking on the adventure of making your app “offline-compatible”.
Progressive Web Apps (PWA)
This first concept is not directly related to the offline management of your app, but is something I found really useful when approaching the development of web applications where we need a good user experience both in browsers, desktop and mobile devices. Progressive Web Apps provide an “installable”, app-like experience on desktop and mobile that are built and delivered directly via the web. They're web apps that are fast and reliable. And most importantly, they're web apps that work in any browser.
Progressive Web Apps can run in a browser tab, but are also installable. Bookmarking a site just adds a shortcut, but an installed Progressive Web App (PWA) looks and behaves like all of the other installed apps. It launches from the same place as other apps launch (in the case of mobile devices). You can control the launch experience, including a customized splash screen, icons, and more. It runs as an app, in an app window without an address bar or other browser UI. All of the PWA-like behavior of your web app will be specified in a manifest.json file, and in order to know the level of PWA your web app has, you can always run the following audit.
Remember, it's critical that an installable PWA is fast and reliable. Users who install a PWA expect that their apps work, no matter what kind of network connection they're on. It's a baseline expectation that must be met by every installed app. Even more, if you follow the guidelines and usual technologies for building PWA it will be easier for you to generate a native mobile-app from your PWA source code using technologies such as Ionic, React Native or Apache Cordova.
To learn more about the development of PWA you can follow this tutorial.
Service Workers
Let’s move on to our matter at hand, how can we make our web app to work offline? The answer to this lies in the service workers of the browser. Service workers are JavaScript code that runs in the background of your website, even when the page is closed. For offline uses, one of their goals is to store network requests or images in the browser cache.
A service worker is a bit like a proxy server between the application and the browser. With a service worker, we can completely take over the response from an HTTP request and alter it however we like. This is a key feature for serving an offline experience. Since we can detect when the user is disconnected, and we can respond to HTTP requests differently, we have a way of serving the user files and resources that have been saved locally when they are offline. Today, they already include very useful features such as push notifications and background sync.
A service worker has a lifecycle that is completely separate from your web page. To install a service worker for your site, you need to register it, which you do in your page's JavaScript. Registering a service worker will cause the browser to start the service worker install step in the background. Typically during the install step, you'll want to cache some static assets. If all the files are cached successfully, then the service worker becomes installed. If any of the files fail to download and cache, then the install step will fail and the service worker won't activate (i.e. won't be installed). If that happens, don't worry, it'll try again next time. But that means if it does install, you know you've got those static assets in the cache.
When installed, the activation step will follow and this is a great opportunity for handling any management of old caches, which we'll cover during the service worker update section. After the activation step, the service worker will control all pages that fall under its scope, though the page that registered the service worker for the first time won't be controlled until it's loaded again. Once a service worker is in control, it will be in one of two states: either the service worker will be terminated to save memory, or it will handle fetch and message events that occur when a network request or message is made from your page.
And in case you were wondering how hard it is to install a service worker for your application, this piece of code checks if the service worker API is available, and if it is, the service worker at /sw.js is registered once the page is loaded. Thus, the code to be executed by the service worker is the one included in ./sw.js. Two things to bear in mind while using service workers is that your application must use HTTPS (if this is not the case service workers won’t work for security reasons), and that the service worker script is accessible at http://myapp.me/sw.js.
if ('serviceWorker' in navigator) {
window.addEventListener('load', function() {
navigator.serviceWorker.register('/sw.js')
.then(function(registration) {
// Registration was successful
console.log('ServiceWorker registration successful with scope: ', registration.scope); },
function(err) {
// registration failed
console.log('ServiceWorker registration failed: ', err); }); });}Cache, Indexed DB and Offline Storage
So using service workers we managed to install a background process in our users browsers that will be responsible for running the actions for the offline operation of our app, but from where do we get all the data required for the operation if our app is offline? Here is where the browser’s CacheAPI and IndexedDB come into play.
Cache API
The CacheAPI is used for storing and retrieving network requests and corresponding responses. These might be regular requests and responses created in the course of running your application, or they could be created solely for the purpose of storing some data in the cache. The caches only store pairs of Request and Response objects, representing HTTP requests and responses, respectively. However, the requests and responses can contain any kind of data that can be transferred over HTTP. In this Cache we will store information such as the static files and scripts required for the offline operation of our application, and is the one that enables the “proxy-like” operation of service workers we mentioned above.
The following piece of code is an example of the installation of a service worker that stores static files in cache for their future use offline:
self.addEventListener('install', (event) => {
event.waitUntil(async function() {
const cache = await caches.open('mysite-static-v3');
await cache.addAll([
'/css/whatever-v3.css',
'/css/imgs/sprites-v6.png',
'/css/fonts/whatever-v8.woff',
'/js/all-min-v4.js'
// etc ]);
}()); });
Sample operation of service worker and Cache API (source: https://jakearchibald.com/2014/offline-cookbook/)
And what if we want to include a “Read Later” or “Save for offline” functionality to the content of our app, we can still use CacheAPI for this? The answer is yes. You can have a button in your app that triggers the storage of a specific content in the cache. The following code would do the work. We are storing an article in cache whenever an event is triggered:
document.querySelector('.cache-article').addEventListener('click', async (event) => {
event.preventDefault();
const id = this.dataset.articleId;
const cache = await caches.open('mysite-article-' + id);
const response = await fetch('/get-article-urls?id=' + id);
const urls = await response.json(); await cache.addAll(urls); });
Save for later scheme (source: https://jakearchibald.com/2014/offline-cookbook/)
IndexedDB
Is the CacheAPI the only place where we can store information for our offline operation in the browser? Not at all. IndexedDB is a low-level API for client-side storage of significant amounts of structured data, including files/blobs. IndexedDB lets you store and retrieve objects that are indexed with a key; any objects supported by the structured clone algorithm can be stored. You need to specify the database schema, open a connection to your database, and then retrieve and update data within a series of transactions.
In short, it is like having a client-side database ready to store whatever you want. Unlike in the CacheAPI where all the information has to be stored in the form of Request and Response objects, in IndexedDB we can store actual blobs of data, enabling us with the storage of content of a larger size (check here the limits of storage of IndexedDB to have an idea of the size of the content you could store). Another advantage over CacheAPI? You can make SQL-like queries over IndexedDB. You don’t have to use it as a “proxy" for requests like in the Cache API. Thus, with IndexedDB we can add to our application such a cool functionality as the storing of videos to watch offline (just like in the Netflix App!). We can include a service worker and a “Watch Offline” button that triggers the storage of the video in IndexedDB for its posterior retrieval whenever we want to watch it (even offline).
PouchDB
So our app is ready to be used offline, but what if we want to interact with it while offline and sync our inputs whenever we recover our Internet connection, is this possible? Of course it is! PouchDB is an open-source JavaScript database inspired by Apache CouchDB that is designed to run well within the browser. It enables applications to store data locally while offline, then synchronize it with CouchDB and compatible servers when the application is back online, keeping the user's data in sync no matter where they next login.
Thus, including CouchDB in your application will enable you with the power of storing changes locally in the client until an Internet connection is recovered and the data can be shared with the backend. Imagine that one of your users have been working through the app in a really long form (like for instance, in the submission of his next LastBasic project), and at the moment of submitting his form, he doesn’t have Internet connection, and he won’t recover it for the next hour. It would be awful for our service to make users waste their time submitting the form again. Fortunately, PouchDB’s sync solves this problem.
PouchDB can be used in line with CacheAPI and IndexedDB, or it can completely replace IndexedDB for the “blob-like” storage. PouchDB’s API makes it easier to use and interact with than IndexedDB, as you can see in this simple example where we are storing and syncing onchange some data with PouchDB:
var db = new PouchDB('dbname');
db.put({_id: 'dave@gmail.com',
name: 'David',
age: 69 });
db.changes().on('change', function() {
console.log('Ch-Ch-Changes'); });
db.replicate.to('http://example.com/mydb');
Wrap up!
So this is all for today, folks! I hope that after this publication you have a better idea of how to “suit up” your app for its operation offline. For me this was an unknown field until I started exploring it in depth. Let me know if there are any additional resources useful for the implementation of offline apps.
References
Many more I can share with you on-demand (I didn’t want to overwhelm you with references).