30 Jul 2019, 20:37

External Blog Post Roundup

You may notice that I have not blogged in a while, at least not on this site. Recently I have been blogging on my current employers (SitePen) blog on a whole host of different topics. I thought it may be useful to round these up and link to them here for reference. This is the list:

I will aim to keep this list updated with any additional posts in the future. Let me know if there’s anything specific you’d like to hear about!

01 Oct 2018, 20:37

A Quick Look at WorkerDOM

In the browser, many operations occur on a single main thread. Due to the number of things we need to handle as web developers such as styling, DOM updates, fetches, data transforms, timers and so forth we can end up with a lot going on that thread. Unfortunately, this thread is also responsible for handling user inputs and rendering to the screen, which are user critical requirements and have large impacts on their experience. Here if we have tasks that run for long periods this can block responding to user input and rendering new frames.

In the browser we have access to threading via Web Workers. Web Workers allow us to run JavaScript in a separate thread whilst also being able to message back to the main thread. The limitations of Web Workers are that they have transfer times for data between the main and worker thread, limited transfer types (i.e. Functions and Errors aren’t allowed) and also perhaps most importantly no DOM access (i.e. document.getElementByID for example). I have experimented and written about transfer times for workers, showing that for small loads the fear is probably over played. However a core aspect of frontend development is, maybe somewhat obviously, trying to update what is happening on the screen (render new content). This makes the DOM limitation pretty painful for those using Web Workers, as it mostly limits it to computational logic which we then proxy back to the main thread.

On mobile devices, where performance perhaps matters the most, we’re now seeing low-end devices with multiple cores (for example a Nokia 1 and the Micromax Bharat Go both have quadcore processors). Arguably this is where multiple threaded code would probably have the largest benefit. However, we still have a particularly single-threaded approach to writing browser code. This is why it’s interesting to see the emergence of the recently announced WorkerDOM library from the AMP team, which aims to let developers leverage workers more easily, by removing the aforementioned limitation of lack of DOM access (or at least perceived lack of DOM access).

What is WorkerDOM?

WorkerDOM is a library which you can include in your applications and pages. It provides much of the regular DOM API and in turn proxied access to the real DOM on the main thread, passing mutations to the DOM over to the main-thread to be handled. This encourages a lot of logic that was previously happening on the main thread to be handled by the Web Worker, leaving the main thread to handle user input and DOM changes. The library itself is written in TypeScript and is compiled down to both global variable and module formats.

Setting up WorkerDOM

You can include WorkerDOM by installing it via NPM like so:

npm install @ampproject/worker-dom

or if you use Yarn then:

yarn add @ampproject/worker-dom

Alternatively you could use a script tag directly, using a CDN, in the following fashion:

<script src="https://unpkg.com/@ampproject/worker-dom@0.1.2/dist/index.js"></script>

We can actually take this a step further as WorkerDOM is distributed as both a global (aliased as MainThread) and an ES Module, allowing for the import syntax. This means you can do the following;

<script src="https://unpkg.com/@ampproject/worker-dom@0.1.2/dist/index.mjs" type="module"></script>
<script src="https://unpkg.com/@ampproject/worker-dom@0.1.2/dist/index.js" nomodule defer></script>

Here only browsers that understand modules (all modern browsers, except Samsung Internet now support modules) will receive the module script.

First Steps

WorkerDOM allows us to expose a specific part of the DOM to be upgraded, we can do this by doing something we would probably not normally do; we set the src attribute on the containing element we want to upgraded to work with WorkerDOM. The src attribute is updated to be the name of the script we’re interested in running as our worker script. For the purposes of this blog, we are going to generate some prime numbers and render them into the DOM. For our index.html file, we might start with something like this in our case:

<body>
    <div src="primes.js" id="primes">
    </div>
</body>

So now we have an element that we can upgrade in our body. We need to declare explicitly that we want to upgrade the element using the upgradeElement method. We can do that like so:


<script type="module">
    import {upgradeElement} from '/dist/index.mjs';
    upgradeElement(document.getElementById('primes'), '/dist/primes.mjs');
</script>

<script nomodule async=false defer>
    document.addEventListener('DOMContentLoaded', function() {
        MainThread.upgradeElement(document.getElementById('primes'), '/dist/primes.js');
    }, false);
</script>

You can see we take both the module and none module code paths allowing us to handle both scenarios. This means we can now begin looking at our actual worker logic!

The Worker File

The coolest part about WorkerDOM is that it allows us to behave as if we have DOM access in the worker thread. That means we have access to properties like document.createElement for example. Continuing on with the concept that we want to generate primes and add them to the DOM, we could do something like this:


const startNumber = 1;

function generatePrimes() {

    while (document.body.firstChild) {
        document.body.removeChild(document.body.firstChild);
    }

    const numDivs = 1000;
    const limit = startNumber + numDivs;
    const primes = sieveOfEratosthenes(limit); // An algorithm for generating primes up to 'limit'

    const div = document.createElement('div');
    div.className = 'parent';

    for (let i = startNumber; i < limit; i++) {
        const numberDiv = document.createElement('div');
        numberDiv.className = "number";
        const numberText = document.createTextNode(i);
        if (primes.has(i)) {
        numberDiv.style.fontWeight = 'bold';
        numberDiv.style.color = '#240098';
        }
        numberDiv.appendChild(numberText);
        div.appendChild(numberDiv);
    }

    document.body.appendChild(div);
    startNumber += numDivs;

}

setTimeout(generatePrimes, 0); // Not sure why we need this in a timeout?
document.body.addEventListener('click', generatePrimes);

This code will create divs with numbers in them, with prime numbers being highlighted and put in bold. The numbers will update on click on the document. Notice how we can behave as if this worker code is on the main thread, with access to DOM APIs. It’s worth pointing out the main difference here is that the "primes" div from the index.html is considered our document.body here.

As it stands, WorkerDOM is currently in alpha, and as such is still being worked on. Hopefully, this post has given you a reasonable overview of the library, and if you are interested in learning more about WorkerDOM I would recommend these resources:

31 Jul 2018, 20:37

Creating Your Own OS Zoomstack Tiles

In a past life I worked for the Ordnance Survey, specifically at the Geovation Hub in London working with startups doing interesting things with geospatial tools and data. One thing we experimented with there was Ordnance Survey based Vector Tiles. Since I left the OS has now invested a full open data Vector Tile product called Zoomstack. I thought it was a good chance to give it a whirl!

You can get access to Zoomstack by signing up for the trial. Once you have the trial you will receive an email where you can download the tiles in a variety of formats including a Vector Tile API, GeoPackage, a MBtiles file or a PostGIS dump. The purpose of this post is to produce another format, which is the raw Vector Tiles themselves (protobuf files/ .pbfs). To do this we are going to download the MBTiles file and then run it through a tool called tile-join which comes with Mapbox’s tippecanoe. There are probably many other ways we could potentially perform this extraction, and probably even serve them straight from a database directly using a server tool like t-rex or tegola, but in this case will just be creating the raw protobuf files.

Installing tippecanoe and tile-join

On Mac this is fairly straight forward using Brew:

brew install tippecanoe

On Ubuntu (and flavours) you can get the necessary dependencies, compile and install doing:

sudo apt-get install build-essential libsqlite3-dev zlib1g-dev
make
make install

On Windows this is more complicated but there are some thoughts on doing it on the GitHub issues page for tippecanoe.

To confirm that tile-join is successfully installed, and type:

tile-join

into your command line. You should see a corresponding output with some usage hints. Something like this:

Usage: tile-join [-f] [-i] [-pk] [-pC] [-c joins.csv] [-x exclude ...] -o new.mbtiles source.mbtiles ...

Generating the Vector Tiles

Now that we know it works, we can use tile-join to generate the necessary files like so:

tile-join --output-to-directory="./tiles" OS-Open-Zoomstack.mbtiles -z14 --no-tile-size-limit --no-tile-compression

You could also remove --no-tile-compression if your web server can support serving gzipped files. This is definitely recommended for production deployments as gzip tiles will be much lighter over the wire, but for local development uncompressed is fine. One last thing; this process might take a little while!

Locally hosting tiles

Now we are going to host them locally. I chose to use http-server, which you can install using npm (node package manager) like so:

npm install -g http-server

Now http-server should be available as a global command, and we can run it in our tile root directory like so:

http-server --cors

--cors provides us with Cross Origin Resource Sharing support which can be handy if we need to speak from other domains (for example Maputnik). If you did want local gzip compression you could rename all the compressed .pbf files to .pbf.gz with a script and add --gzip as a command flag and http-server should pick them up. You could also add http caching using -c60 (where 60 is the number of seconds) but locally this might cause you some pain if you’re trying to change things rapidly.

Now, assuming you were serving tiles from a tiles folder and you called the above command from the root folder contain tiles, you could get at a specific tile like so:

http://127.0.0.1:8080/tiles/14/8128/5494.pbf

Try opening that in your browser (Chrome, Firefox etc) to prove that it works!

Plugging it into a library

For simplicity in styling here I am going to use Mapbox GL, but we could easily be using the Esri JavaScript API, OpenLayers or potentially even Leaflet (with the right plugin). Basically any library which supports Vector Tiles.

Here is how we’d setup a map in JavaScript:


    mapboxgl.accessToken = 'pk.eyJ1IjoiamFtZXNtaWxuZXIiLCJhIjoiY2prNzA2aGprMWt0MTNydGhuMWs3NG13NSJ9.hRj9NRx4ROa0QDkT4t9XdQ';
    var map = new mapboxgl.Map({
        container: 'map',
        style: 'zoomstack.json'
    });

For styling purposes I took the a premade style from the Ordnance Survey styling repository. You can just download to the root directory where your index.html is and save it with a sensible name as .json file. You will need to make some adjustments to the original OS styling file to make it work with our new setup, such as change the tile url to be the new tiles API format like so:

	"composite": {
		"tiles": [
			"http://127.0.0.1:8080/tiles/pbftiles/{z}/{x}/{y}.pbf"
		],
		"type": "vector"
	}

Using sprites locally

You can download the sprites from the Ordnance Survey folder Mapbox GL styles folder on GitHub, place them locally for your web server to pick up. You can reference them in your style json like so:

	"sprite": "http://127.0.0.1:8080/sprites/sprites",

Custom Fonts

Custom glpyhs can be provided to your styling file to add flair to your map. Let’s say you have a font tha you like from a provider like Google Fonts and then use those with your tiles. You can do this by downloading a fonts ttf file and then running it through fontnik. You can install fontnik locally using npm (npm install fontnik) and then run the build-glyphs in the node_modules folder for fontnik. For example let’s say we downloaded the Roboto TTF file, we can build those glyphs using:

mkdir fonts/Roboto\ Regular
node_modules/fontnik/bin/build-glyphs path/to/source/Roboto-Regular.ttf path/to/fonts/Roboto\ Regular

Note escaping the spaces in the path names using the backslash. After this you can edit your style file to provide the web server glyph URL like so:

	"glyphs": "http://127.0.0.1:8080/path/to/fonts/{fontstack}/{range}.pbf",

Then in a layer layout property you could defined font-text like this:

	"text-font": [
		"Roboto Regular"
	]

Custom styling using Maputnik

Maputnik provides a GUI styling tool you can use to create your own custom styles. You can load your local style (zoomstack-json for example) right into Maputnik and making edits. Check it out at https://maputnik.github.io/editor. Here’s an example of a little styling I did using Material Design based colours and fonts:

Conclusion

Using the .mtiles file you can generate you own tiles to play around with. You could use these locally or host them somewhere on the web like an Amazon S3 bucket, or a web server of your choosing. You’ll want to figure out how to serve the compressed files for best performance. You can also use fantastic base stylings from Ordnance Survey as starting points for your own custom styling for the maps, and edit them manually using a text editor or using a GUI like Maputnik.