22 Jul 2017, 11:30

Using Go to Plot Global Data

Wanting to explore more around geospatial tools with Go, this week I’ve been exploring Globe. Globe creates wireframe global visualisations of data sets, provided they contain latitude and longitude points. Here’s an example of the API from the README:

    shops, err := LoadCoffeeShops("./starbucks.json")
    if err != nil {
        log.Fatal(err)
    }

    green := color.NRGBA{0x00, 0x64, 0x3c, 192}
    g := globe.New()
    g.DrawGraticule(10.0)
    for _, s := range shops {
        g.DrawDot(s.Lat, s.Lng, 0.05, globe.Color(green))
    }
    g.CenterOn(40.645423, -73.903879)
    err = g.SavePNG("starbucks.png", 400)
    if err != nil {
        log.Fatal(err)
    }

I decided that I wanted to build a web service that would allow users to generate an image (PNG) using some sort of interface. With a few free evenings I got to work!

The Backend

For all request serving activity I used the built in http package. Go has a fantastic standard library for building web apps. However I did find it a little tough to find sensible examples of file upload handling.

For extracting the geospatial elements of the data, I used Paul Mach’s GeoJSON package for extracting latitude and longitudes from GeoJSON and then used the standard library CSV package for the doing the same operation with CSVs.

Finally for the drawing; we use the aforementioned Globe package to do the actual drawing. This was all relatively straightforward minus hitting one issue around trying to use RGBA colors rather than NRGBA (non-alpha-premultiplied 32-bit color) colors with the library. It turns out NRGBA produces a lot cleaner results, you can see the issue on the Globe package here.

The app is hosted on Heroku. It took me a little while to get to grips with deploying Go apps on the platform (leveraging Go’s vendoring system was a first for me), but apart from that it was pretty painless. The GitHub push-to-deploy integration is also a very nice touch!

The Frontend

The frontend uses jQuery (boooo) as a general DOM/request helper library. It would have been relatively straight forward to replace these with smaller modular libraries but I was feeling a tad on the lazy side (Pull Request anyone?). For the user interface we leverage Skeleton CSS for the layout.

Spectrum.js is used as the colour selector, which we pass to the backend to generate the dots. To get unique file names, I leveraged a UUID library that prevents clashes.

Lastly, and potentially most interestingly, we make use of Turf.js to find the centroid of the latitude/longitudes. For those of you who aren’t familiar Turf.js is a geospatial analysis library for JavaScript, with a very powerful set of tools at its disposal for dealing with geometries. Mapbox’s csv2geojson was also used to convert CSVs over to GeoJSON so that it can work alongside Turf.js (it uses GeoJSON as it’s operating format). Overall I feel like this was a pretty nice solution as it allows for ‘smart defaults’ of the center of where the image is generated, without the user having to guess/analyse there data extensively.

Demo

The service allows you to create images like this:

You can check out the app at https://globe-plotter.herokuapp.com and have a little play. Try this earthquake GeoJSON file from the USGS if you’re looking for and example.

25 Jun 2017, 14:59

Icon Font to SVG Icons: A worked example

There has been an on going discussion about wether Icon Fonts or SVG Icons are ‘better’ for usage on the web. This css-tricks article outlines the introductory differences between the two nicely. The author concludes that “If you can go IE 9+ / Android 3+, inline SVG is better at pretty much everything than icon fonts”. This view has been debated somewhat, with Tyler Sticka laying down a solid case for moving away from Icon Fonts and Ben Frain making a succinct rebuttal. The general consensus within the web development community appears to have been to move over to SVGs however.

One notable benefit I saw for moving to inline SVG was another opportunity to prevent unnecessary render blocking; the process by which content such as CSS/JS/Fonts slow down the rendering path for a page, preventing meaninful content getting to the screen whilst they load.

With this in mind I decided to use my own blog as a way to explore how to make this transition and critically examine any benefits (or problems) it might hold. This post will lay out my approach for swapping out my blogs icon fonts (on the left hand panel, or a the top if you’re on mobile!).

Steps

  1. Remove the icon font CSS

    In this case we were using Font Awesome. Font Awesome is a great resource, and I really admire what they’ve done. Of course however, this post is about removing icon fonts, so out it must go!

        <link href="//maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css" rel="stylesheet">
    
  2. Remove the icons themselves

    Here we remove the icons where necessary. In my case I was using 5 icons from an icon font within the sidebar. We simply delete the relevant icons from the HTML, for example the Twitter icon:

        <li class="nav-item">
            <a class="pure-button" href="https://twitter.com/{{ . }}">
                <i class="fa fa-twitter"></i> 
                twitter
            </a>
        </li>
    
  3. Grab the SVG Icons you need

    For the SVG Icons I am choosing to use edent’s SuperTinySocialIcons. These are very efficient SVG icons, covering the majority of your favourite companies/logos. This won’t cover none-social icons (arrows etc) that you might need, in my case I got the cube icon from the The Noun Project and ran it through Jake Archibald’s SVG optimiser. For a more robust approach checkout Sara Soueidan’s superb blog post about using fontello-svg to take Fontello (an aggregator for web fonts) fonts and convert them down to SVG.

  4. Inline the SVG Icons

    You might choose to link out to an external SVG asset, however for performance reasons I have inlined the SVG right into the sidebar, for example:

        <a class="pure-button" href="https://twitter.com/{{ . }}">
        twitter
        <svg viewBox="0 0 512 512" xmlns="http://www.w3.org/2000/svg">
            <rect fill="#1da1f3" height="512" rx="15%" width="512"/>
            <path d="m456 133c-14 7-31 11-47 13 17-10 30-27 37-46-15 10-34 16-52 20-61-62-157-7-141 75-68-3-129-35-169-85-22 37-11 86 26 109-13 0-26-4-37-9 0 39 28 72 65 80-12 3-25 4-37 2 10 33 41 57 77 57-42 30-77 38-122 34 170 111 378-32 359-208 16-11 30-25 41-42z" fill="#fff"/>
        </svg>
        </a>
    
  5. Making necessary edits

    We need to do a little bit of extra work to take the coloured icons and convert them to monochrome. Taking the Twitter icon for example, we can get rid of the rectangle entirely to remove the light blue box:

        <svg class="svg-icons" viewBox="0 0 512 512" xmlns="http://www.w3.org/2000/svg">
            <path d="m456 133c-14 7-31 11-47 13 17-10 30-27 37-46-15 10-34 16-52 20-61-62-157-7-141 75-68-3-129-35-169-85-22 37-11 86 26 109-13 0-26-4-37-9 0 39 28 72 65 80-12 3-25 4-37 2 10 33 41 57 77 57-42 30-77 38-122 34 170 111 378-32 359-208 16-11 30-25 41-42z" fill="#fff"/>
        </svg>
    
  6. Scale them correctly

    Using a CSS class we can use vertical-align: middle to align the icons vertically and set the width to be in fitting with the adjacent text (16px). We could also just use direct inline CSS if desired.

Outcomes

By default Font Awesome CSS was 5.83Kb over the wire (23.18Kb uncompressed), plus the actual font itself 55.45kb (WOFF2 is compressed). That’s 61.28kb in total, plus two network requests. Using Fontello, I was able to get the font down to 3.17kb (!) by removing all the unused icons. So in total, at it’s most optimised we get down to 9.0kb compressed and two network requests (if uncached).

Inlining the SVGs in the sidebar for the index page takes the HTML page weight up from the original 8.85Kb (compressed) to 9.68Kb (compressed) increasing it by 0.83kb compressed. Using icon fonts in this case eliminated two network requests for render blocking content, and removed 8kb in page weight in the best case and over 60kb in the worst case scenario (using the default Font Awesome CDN CSS link). Lastly, I would say even if you don’t want to make the switch, at least investigate condensing your icon font usage usage down to the icons you use explicitly, using a tool like Fontello.

Things to note:

  • I didn’t do any scientific experiments on actual page load times (boooo I hear you shout), however there has been some evidence to suggest there are some marginal losses in rendering using SVGs.
  • IE9+ for support for SVGs (hopefully no longer an issue for 99% of developers)
  • Similarly older versions of Android lack support (2.3)

17 Jun 2017, 19:28

Deploying a Static Blog with Continuous Integration

In recent years, static sites have seen a resurgence. In this post we’ll explore how to use Hugo, a static site generator, in conjunction with a remote web server and a continuous integration provider. Although the site generation software here is Hugo it could easily be another provider such as Jekyll. The web server will be hosted on a Digital Ocean droplet running Nginx, and CircleCI will be used for for the continuous integration (often abbreviated to simply “CI”). Although it is difficult to speak in specifics due to the multitude of alternatives out there, it should be (hopefully) fairly straightforward to deduce a similar process for other providers.

Firstly, for those of you unfamiliar what Continuous Integration is, here is a definition:

“Continuous Integration is a DevOps software development practice where developers regularly merge their code changes into a central repository, after which automated builds and tests are run” (Amazon Web Services, 2017)

Generally CI is leveraged when there are many developers checking in code to a distributed version control system to ensure the integrity of software before its release to some environment. However we can leverage another one of its key tenets, which is that the building, deployment and testing is being done by a remote service, saving us time and energy. This in turn allows you to focus on writing the blog posts and not how or where the content needs to be deployed.

Some of you may be deploying your static Hugo site (or other static blog) via a bash script or some sort of manual process. This guide will explain how to deploy your blog every single time you commit that code to GitHub, or other cloud based git provider.

Why Bother?

There are a few benefits to setting up CI with your static blog:

  • Avoid repetitive local build and production steps, offload work to another machine
  • Those build steps will always run, so you can’t forget to do them locally (i.e. for theme CSS/JS modification etc)
  • As you need to do it to deploy, it helps you remember to commit your code
  • Clone code to any machine and edit posts from there without worrying about build/deployment dependencies or processes
  • Allows you to edit/create posts using GitHub’s user interface directly
  • Helps keep secrets and sensitive information out of your source code

Setting Up Circle CI and SSH Keys

You will need to register for an account with CircleCI and associate it with your GitHub account. From here you can begin to build GitHub projects from CircleCI. Check out this guide from CircleCI for more specifics regarding that process.

Before we dig into the details, it is important to explain that in order to get the Circle CI talking to our remote server (this case a Digital Ocean droplet), we must setup SSH keys. SSH allows keys allow us to be authenticated by the server whilst avoiding the use of passwords via Public-key cryptography.

It is a little outside the scope of this guide to delve into setting up the keys, however a great guide to generating SSH keys is available from GitHub here, and an explanation of how to use them in conjunction with Circle CI can be found in their docs here. Once you have generated the keys and registered them with CircleCI you can move onto the next section.

Environment Variables

It is desirable to avoid storing any secrets or sensitive information inside the blogs source code itself. One way to do this is to use environment variables on the CI server. CircleCI provides a UI for setting environment in the project settings, on the left hand panel.

For example in this case, the user login password is set as DIGITALOCEAN and the IP of the server is DIGITALOCEAN_IP. This also makes them reusable should you need to use them more than once in your configuration scripts.

The Configuration File

Once our target server and CircleCI are setup with the right keys we can begin to look into how to deploy from CircleCI. With CircleCI we provide a configuration file in the form of ‘circle.yml’. The config is written in YAML which is a minimalist markup language, often used for configs. Here is what my specific circle.yml file looks like:


dependencies:
  pre:
    - wget https://github.com/gohugoio/hugo/releases/download/v0.23/hugo_0.23_Linux-64bit.deb
    - sudo dpkg -i hugo*.deb
    - sudo apt-get install sshpass rsync
    - cd ./themes/impurehugo/ && npm install 

deployment:
  prod:
    branch: master
    commands:
      - cd ./themes/impurehugo/ && npm run build
      - hugo -v
      - sudo sshpass -p "$DIGITALOCEAN" rsync -avz ./public $DIGITALOCEAN_IP:/var/www/html/
      
test:
  override:
    - "true"

You can see in the dependencies section we install hugo at the current latest release (v0.23), and we install the package. We also install sshpass as we will need it to login on the target deployment server.

In my case I do a little extra work with npm and gulp to do some preprocessing (JavaScript/CSS minification, image compression) so we change to the target theme folder and do an npm install there to ge the node dependencies and install gulp. This step can be skipped if you aren’t interested in these preprocessing steps.

With deployment, we pull from master and then we do the frontend pre-processing previously mentioned with gulp compress. After this we run Hugo with the verbose flag (useful for debugging purposes if necessary). Next because Hugo only produces static assets, we can simply move them over to our target server. Here we use rysnc to copy the files to the remote server. rsync has the benefit over scp that it only transfers files that have changed since the last upload. However, you could use something like scp if you are so inclined.

Lastly we override the tests section to simply pass if as our only real major concern is if there are errors in the actual build and deployment steps.

Conclusion

This should hopefully give an overview of how to setup continuous integration with Hugo, and hopefully enough inspiration to adapt it to work with other static site providers if necessary. I really welcome any improvements, requests for clarity of other feedback you might have for me. Feel free to reach out to me on Twitter, or drop me an email.