Harvey's Blog

Setting Up Automated Database (PostgreSQL) Backups Using Node.js and Bash

Harvey Delaney — Sat, 25 Sep 2021 01:59:22 GMT

In the event of a hardware or software failure, you risk losing your application’s entire database along with your important data. So obviously you should have a disaster recovery plan in place that will cause the least user disruption and data loss.

To be sure you can recover your data with minimal data loss:

Have a detailed backup of your database.
Restore it in production in case of failure.

You can schedule automated database backups and save yourself from the hassle of doing them manually. This way if there is a database server failure, you can always use the latest backup to restore the database.

In this post, you will learn how to use Node.js to run the bash commands. These commands will prepare a backup of the PostgreSQL database. This database backup will be uploaded on another server using Node.js. Then we will write a cron job to schedule this database backup process so it can be performed in a defined window. Let’s get started.

Taking PostgreSQL database backup

The pg_dump is a utility used to take the logical backups of a PostgreSQL database. It is automatically installed with the PostgreSQL installation. Below is the syntax to use the pg_dump command for backups.

 ubuntu@ubuntu:~$ pg_dump [connection options] [options] [database name]

NOTE: This post contains commands specific for PostgreSQL 13. For other versions, commands may vary.

The following connection options are used with the pg_dump command to take the database backup.

-U option specifies the database user used to run the command.
-h is used to specify the hostname for the PostgreSQL server. It may be an IP address or a DNS name.
-p is used to specify the port the database server is listening on.

Other options for database backup include:

-f specifies the name of the output file.
-F specifies the output file format.
-d specifies the database name to have a backup of.

If you wish to dig deep and explore more options, there is a list of them here.

Advancing ahead, in order to take backup of the entire database using the pg_dump utility along with the necessary options, use this syntax:

ubuntu@ubuntu:~$ pg_dump -U admin -h localhost -p 5432 -f db_backup.tar -F t -d pg_database

The above command will generate a file db_backup.tar, which will be the backup of the entire PostgreSQL database named pg_database. We will run this command using Node.js code and then upload the backup file to another server.

Running the bash command using Node.js

In this section, we will write a Node.js code that will run the above mentioned bash “pg_dump” command to take the PostgreSQL database backup and then upload the backup file to another server.

First of all, install the required dependencies using the npm, which is a package manager for Node.js. We will use the dotenv and @getvim/execute packages. The dotenv is used to manage environment variables and we will use it to pass the database connection parameters. The @getvim/execute package will be used to run the bash commands in Node.js code.

ubuntu@ubuntu:~$ npm install dotenv @getvim/execute

Create a file .env in the root directory of the Node.js project and enter the database parameters like database user, host, and database name.

DB_USER=admin
DATABASE=pg_db
DB_HOST=localhost
DB_PORT=5432

Now create a file index.js and write the Node.js code to take the backup of the database.

// importing required modules
const { execute } = require('@getvim/execute');
const dotenv = require('dotenv').config();

// getting db connection parameters from environment file
const username = process.env.DB_USER;
const database = process.env.DATABASE;
const dbHost = process.env.DB_HOST;
const dbPort = process.env.DB_PORT;

// defining backup file name
const date = new Date();
const today = `${date.getFullYear()}-${date.getMonth()}-${date.getDate()}`;
const backupFile= `pg-backup-${today}.tar`;

// writing postgresql backup function
const takePGBackup = () => {
    execute(`pg_dump -U ${username} -h ${dbHost} -p ${dbPort} -f ${backupFile} -F t -d ${database}`);
    .then( async () => {
    	console.log(`Backup created successfully`);
    })
    .catch( (err) => {
    	console.log(err);
    });
}

// calling postgresql backup function
takePGBackup();

The above code gets db parameters from the .env file and then takes the backup from the PostgreSQL database. Now this database backup needs to be compressed, as the original database backup may be very large in size.

Extend the existing takePGBackup() function and add the code to compress the backup file. Before writing the code, install the gzipme package that is used to compress the files.

ubuntu@ubuntu:~$ npm install gzipme

After installing the package, do not forget to import fs and gzipme packages into the index.js file using the require() methos.

const { execute } = require(‘@getvim/execute’);
const dotenv = require(‘dotenv’).config();
const compress = require(‘gzipme’);
const fs = require(‘fs’);
const takePGBackup = () => {

execute(`pg_dump -U ${username} -h ${dbHost} -p ${dbPort} -f ${backupFile} -F t -d ${database}`)
    .then( async () => {
        // add these lines to compress the backup file
        await compress(backupFile);
        fs.unlinkSync(backupFile);
        console.log("Zipped backup created");
    })
    .catch( (err) => {
   		console.log(err);
    });
}
// calling postgresql backup function
takePGBackup();

So far, this code takes the backup from the database and compresses it into a ZIP file. Now write a function to upload this backup to another server. This function will make a POST request with the backup file in the body to another server.

Next on the to-do list is to install the axios and form-data modules. The axios modules will be used to make a POST request while the form-data module will be used to send the file in the body of the POST request.

ubuntu@ubuntu:~$ npm install axios form-data

const { execute } = require(‘@getvim/execute’);
const dotenv = require(‘dotenv’).config();
const compress = require(‘gzipme’);
const fs = require(‘fs’);
const axios = require(‘axios’);
const formData = require(‘form-data’);
// function to upload backups
const uploadBackups = (backupFile) =>
{
    // making post request using axios
    const form = new formData();
    form.append('file', backupFile)
    axios.post('http://backup-server.url/api/upload-form', form, { headers: form.getHeaders()});
    .then(result => {
        fs.unlinkSync(backupFile);
        console.log(result.data);
    })
    .catch(err => {
    	console.log(err);
    })
}

// calling uploadBackups function
uploadBackups(backupFile);

Schedule the backups using cron job

You want to be sure your data is backed up regularly, otherwise if an event causes you to lose your database, your backup will only be as current as the last time you remembered to back it up. With all the precautions you take, from elaborate passwords to gaming VPNs, it would be a shame if you suffered a major setback simply because you didn’t schedule your backups.

Up till now, the code to take the automated backup and upload it to the server is complete. Now it is time to schedule the execution of this code. We will use cron job to schedule this task.

Cron job is a command line program used to schedule tasks on Linux machines. The task may be a command or a script and can be scheduled using a set of fields. This is the syntax to use the cron job:

minute: This is the first field from the left and defines the minute of the hour the job runs. It ranges from 0 to 59.
hour: Second field from the left and specifies the hour. It has values from 0 to 23 where 0 is 12 at midnight.
dayOfMonth: Third field from the left and specifies the day of the month the command runs. It has values from 1 to 31.
month: Fourth field from the left and specifies the month the commands run. It has values from 1 to 12.
dayOfWeek: Fifth field from the left and specifies the day of the week the task runs. It has values from 0 to 6 with 0 being Sunday.
Command: The last field is the command that is executed on a specified schedule.

In order to run this code using cron job, first find the location of the node binaries using the which command as cron job accepts the absolute path of the command.

ubuntu@ubuntu:~$ which node

Open the crontab in the terminal using the -e option along with the crontab command.

ubuntu@ubuntu:~$ sudo crontab -e

It will open a crontab. At the end of the file, use the following pattern to run the code every day at 8:00 AM.

0 8 * * * /usr/local/bin/node

The asterisk (*) in the above pattern means every possible value for the field. There are some other options that can be used to customise the schedule of the cron job. Explore further and play around with all the different options here. After scheduling the job, verify by listing all the scheduled jobs using the following command.

ubuntu@ubuntu:~$ sudo crontab -l

At the end of the file, it lists all the scheduled jobs. In order to remove the scheduled job, use the following command.

ubuntu@ubuntu:~$ sudo crontab -r

After applying any change to the crontab, restart the cron service.

ubuntu@ubuntu:~$ sudo systemctl restart cron

ubuntu@ubuntu:~$ sudo service cron reload

Wrapping up

In this post, we’ve looked at how to use the Node.js program to make database backups. We’ve identified how to schedule this task using cron job and how to use bash commands inside our Node.js code to automate database backup, compress the backup, and upload the compressed database backup to another server for disaster recovery.

Creating your own Vue Component Library

Harvey Delaney — Tue, 11 Aug 2020 12:28:31 GMT

Previously, I wrote an article outlining the steps required to get a React component library up and running. You can read it here: https://blog.harveydelaney.com/creating-your-own-react-component-library

It's had some pretty good feedback so I decided to branch out and create another component library template for Vue! I've not had much experience with Vue before this article and was a great learning experience.

Vue Component Library Code

If think you'll find it helpful to have the code up while you read through the article, have a look at the following GitHub repository I created:

HarveyD/vue-component-library

A project skeleton to get your very own Vue component library up and running using Rollup, Typescript + Vue - HarveyD/vue-component-library

HarveyDGitHub

Component Library Overview

We'll be using the following to create the library:

Vue (obviously)
TypeScript
Rollup

I'll also be showing you how to publish your component library to NPM and how to consume it.

Creating the Component Library

Project Setup

Initialise NPM - npm init to generate a basic package.json file.

Initialise Git - git init.

We're going to have the following structure for our component library:

.gitignore
package.json
rollup.config.js
tsconfig.json
src/
  SampleComponent/
    SampleComponent.vue
  index.ts
  shims.tsx.d.ts
  shims.vue.d.ts

We will be outputting our bundled + compiled files into the lib directory.

Vue

Since we're building a Vue component library, we need to install Vue and it's required dependencies to get it working. Run:

npm i -D vue vue-class-component vue-property-decorator @vue/compiler-sfc vue-template-compiler

You'll notice we're installing these as devDependencies which won't be included when we install the component library in another project. We should expect the project that installs the component library already has Vue installed.

To make sure this is the case, we can specify Vue dependencies as peerDependencies. To do this, open package.json and add:

...
"peerDependencies": {
    "vue": "^2.6.11",
    "vue-class-component": "^7.2.3",
    "vue-property-decorator": "^8.4.2"
  },
  ...

When the component library is installed as a dependency, but the consuming project does not have the peerDependencies, a warning will be output.

Vue Component

What use is a component library without any components? Let's add our first Vue component. Within src/SampleComponent/SampleComponent.vue add:

src/SampleComponent/SampleComponent.vue

Next, let's expose this component in the root file of our library src/index.ts:

import SampleComponent from "./SampleComponent/SampleComponent.vue";

export { SampleComponent };

TypeScript

Install TypeScript by running:

npm i -D typescript

After installing TypeScript, create a tsconfig.json file. Within tsconfig.json and add:

{
  "compilerOptions": {
    "declarationDir": "lib",
    "target": "es5",
    "module": "es2015",
    "sourceMap": true,
    "declaration": true,
    "importHelpers": true,
    "strict": true,
    "experimentalDecorators": true
  },
  "include": ["src/**/*.ts", "src/**/*.vue"]
}

To get Vue types working with TypeScript, we need to introduce some shims. In src/shims-tsx.d.ts add:

import Vue, { VNode } from 'vue'

declare global {
  namespace JSX {
    // tslint:disable no-empty-interface
    interface Element extends VNode {}
    // tslint:disable no-empty-interface
    interface ElementClass extends Vue {}
    interface IntrinsicElements {
      [elem: string]: any
    }
  }
}

And in src/shims-vue.d.ts add:

declare module '*.vue' {
  import Vue from 'vue'
  export default Vue
}

Rollup

First, add Rollup and some additional Rollup plugins that'll help us get our component library transpiled and bundled:

npm i -D rollup rollup-plugin-peer-deps-external @rollup/plugin-node-resolve @rollup/plugin-commonjs rollup-plugin-typescript2 rollup-plugin-vue

In rollup.config.js add:

import peerDepsExternal from "rollup-plugin-peer-deps-external";
import resolve from "@rollup/plugin-node-resolve";
import commonjs from "@rollup/plugin-commonjs";
import typescript from "rollup-plugin-typescript2";
import vue from "rollup-plugin-vue";

import packageJson from "./package.json";

export default {
  input: "src/index.ts",
  output: [
    {
      format: "cjs",
      file: packageJson.main,
      sourcemap: true
    },
    {
      format: "esm",
      file: packageJson.module,
      sourcemap: true
    }
  ],
  plugins: [peerDepsExternal(), resolve(), commonjs(), typescript(), vue()]
};

Let's go through this config file and understand what it's doing.

Input

This points to src/index.ts. Rollup will build up a dependency graph from this entry point and then bundle all the components that are imported/exported.

Output

This is an array with two config objects. These two configs tell Rollup to output two bundles in two different JavaScript module formats:

CommonJS - CJS
ES Modules - ESM

IMPORTANT: You'll notice in output, values from package.json are used. In package.json, add:

...
  "main": "lib/index.js",
  "module": "lib/index.esm.js",
...

This tells packages that install our component library where the entry point of our component library is. We re-use these values in rollup.config.js to instruct Rollup where to output our bundled file.

Plugins

peerDepsExternal - if you remember earlier, we specified some peerDependencies. This plugin will take these dependencies and assign them to the external field in Rollup. Any module in external is not added to the Rollup bundle. This will reduce our bundle size as we already know these dependencies will be present in any project that consumes our library.
resolve (@rollup/plugin-node-resolve) - efficiently bundles third party dependencies we've installed and use in node_modules
commonjs (@rollup/plugin-commonjs - enables transpilation into CommonJS (CJS) format
typescript (rollup-plugin-typescript2) - transpiles our TypeScript code into JavaScript. This plugin will use all the settings we have set in tsconfig.json
vue (rollup-plugin-vue) - allows us to create our Vue components as Single-File Components (SFCs) and still have Rollup bundle them as expected

Running Rollup

Now we need a way to be able to instruct Rollup to do its thing. In package.json, add:

...
  "scripts": {
    "build": "rollup -c"
  },
  ...

The -c flag tells Rollup to use the rollup.config.js file we've created.

Try running it by running npm run build, you should see:

Then in /lib you should something like:

If this is true, your component library has successfully been configured. It's now time to use it!

Note: it's probably worth having the following .gitignore:

node_modules
lib

Final NPM Config

As a reference, here's what your final package.json should look like:

{
  "name": "vue-component-library",
  "main": "lib/index.js",
  "module": "lib/index.esm.js",
  "scripts": {
    "build": "rollup -c"
  },
  "peerDependencies": {
    "vue": "^2.6.11",
    "vue-class-component": "^7.2.3",
    "vue-property-decorator": "^8.4.2"
  },
  "devDependencies": {
    "vue": "^2.6.11",
    "vue-class-component": "^7.2.3",
    "vue-property-decorator": "^8.4.2",
    "@rollup/plugin-commonjs": "^14.0.0",
    "@rollup/plugin-node-resolve": "^8.4.0",
    "@vue/compiler-sfc": "^3.0.0-rc.5",
    "rollup": "^2.23.1",
    "rollup-plugin-peer-deps-external": "^2.2.3",
    "rollup-plugin-typescript2": "^0.27.2",
    "rollup-plugin-vue": "^5.1.6",
    "typescript": "^3.9.7",
    "vue-template-compiler": "^2.6.11"
  }
}

Publishing the Component Library

Now, it's time to publish our component library. We can either publish our library on public NPM registry, or we can find a self-hosted private NPM registry alternative like Verdaccio. After you've made your choice and configured NPM to point to the registry, run:

npm publish

NPM will then read files to see what files it should include when publishing. In our case it's lib, so it'll publish the bundled/transpiled code output by Rollup.

You should see something like:

For example, I've published a version of this Vue component library at: https://www.npmjs.com/package/harvey-vue-component-library

Note: You have to make sure that the files are available in lib before publishing.

Using the Component Library

Locally

We don't have to publish the component library to an NPM registry before installing our component library and test out our components.

Let's say you had a React project on your local machine called harvey-test-app. In harvey-test-app run (making sure the path is correct to your component library):

npm i -D ../vue-component-library

This creates vue-component-library as a symlinked dependency. Read more at: https://docs.npmjs.com/cli/link

From NPM Registry

Let's say your component library is available at: https://www.npmjs.com/package/harvey-vue-component-library, you would install it in another Vue project by running:

npm i -D harvey-vue-component-library

Using Components

Make sure that you register your Vue component. Then you can simple import and use components in another project like:

Here's a Code Sandpit with an example:

GitHub Repository

Again, here's the GitHub repo I created with everything in this article:

HarveyD/vue-component-library

A project skeleton to get your very own Vue component library up and running using Rollup, Typescript + Vue - HarveyD/vue-component-library

HarveyDGitHub

Choosing a Usenet Provider - a Beginner's Guide

Harvey Delaney — Sun, 31 May 2020 11:21:39 GMT

I've recently started to use Usenets over torrents for downloading content. Read about my Usenet articles below:

One thing I spent the most time on in my Usenet setup was the research into what Usenet provider to use. I found that it is most important service that you will choose within your Usenet setup. Usenet providers often have paid subscription plans. Whenever I pay for a service (especially a subscription purchase) I want to make sure that I choose the best provider and plan that meets my needs.

This article will guide you through what I think the most important things to consider are when choosing a Usenet provider. I recommend you consult other articles to make sure what I've said here aligns with my recommendations here!

I've identified a number of "core" features/offerings that are the most important things to consider when choosing a Usenet provider. I'll also cover "nice to have" features/offerings which are some unessential that Usenet providers bundle within their plans to sweeten the deal.

Core

Download speeds
Download limits
Number of connections
Binary retention duration
Available server regions
Pricing and subscriptions

Nice to haves

Trial periods
Customer support
SSL
VPN
Newsreaders / Search / Browser

Core

Download Limits

The most important aspect to consider for a provider. This is the amount of data that you're able to download per month. Usenet providers will usually advertise this amount clearly. The amounts are usually specified in GB/month. Some select (more expensive) plans will offer unlimited download limits.

Try to select a plan that will closely meet your download amount needs (and you can save money!).

An example of a plan with a set download limit/month

Download Speeds

This is the maximum download speed that their servers will provide you. Please note, this is a maximum, not a minimum amount, so expect to see speeds below this "maximum" amount.

I find this to be the most important point due to what Usenet providers are used for. I don't think anyone has ever said that their download speeds were too fast and requested for them to be slower. Since Usenets provider are used by people to download content, people want to download their content quick!

Make sure you're choosing a provider that meets your download speed needs.

Number of Connections

At first, you may not think this applies to you, but it does! This is NOT the number of simultaneous downloads that can occur, but the number of Usenet file downloads that can happen in parallel.

A file stored on the Usenet is split into multiple parts. Your download client will attempt to download these file parts concurrently. The higher the amount of connections, the higher the amount of files you can download at any single time. This is important for downloading files and even more important for downloading multiple files at the same time.

This has a direct impact on your download speeds. To help visualise this, I've experimented with downloading a single file using a maximum of 1 connection, 15 connections and 30 connections using my download client (NZBGet):

Connections (1)

Using 1 connection, I was only able to reach speeds of ~1.3mbps:

Connections (15)

Using 15 connections, I was able to reach speeds of ~5 MBPS:

Connections (30)

Using 30 connections, I was able to reach speeds of ~5.6 MBPS (max of my ISP):

Please note, the above will vary based on the content you're downloading in addition to your network speeds, it's just a little guide :).

Binary Retention Duration

A "binary" is a fancy word for a non-text file. This includes images, videos, executables etc. Once a file is uploaded to the Usenet network, a Usenet provider will attempt to store that file on their servers for a set amount of time (usually specified as days).

The higher this retention period is, the more likely you'll find older content. I've found that the upper bound of what's offered is ~4300 days. This means you if you join a Provider that offers that retention period, they will likely have files uploaded to the Usenet around 11.5 years ago.

So, if you know that you're only going to be downloading recently released content, you could save money by choosing a provider that has a shorter retention period. However, if you're a power user, I highly recommend not skimping on this aspect.

I've experienced using a provider which had an advertised 1100 day retention and a provider with a 4300 day retention. My experience with the 1100 day retetion provider was quite poor (and I almost stopped using Usenet because of it). Most content I tried to download, I would run into failures. The failures were encountered because some "blocks" that made up a file were missing. This is likely due to the fact the retention period was on the lower end:

Once I switched to the 4300 day retention provider, my experience drastically improved. Most, if not all, of my content downloaded successfully the first time. For this reason, I recommend not cheaping out on a provider with a high retention rate and save yourself some sanity :).

Available Server Regions

This is a point that often is overlooked, yet still important. Most people understand that the closer you are geographically to a server, the faster your latency and download speeds will be. This is also the case for Usenet providers - you want to be a close as possible to their servers.

A Usenet provider will have servers in a set number of regions. I've found that most Usenet providers don't clearly advertise what server regions they have available. But you'll find that most have US and EU servers.

You'll often have to dig through their support documentation to find out what servers are available, for example, I found the available regions (US, EU, NL, DE) for Newshosting here:

Pricing and Plans

All of the above points are variable based on how much you're willing to pay! The more expensive a plan, the better your Usenet experience will be.

There are two different ways that you can gain access to a Usenet provider's services:

Subscription
Block amounts

Subscription models are the most common way to gain access to a Usenet provider. You'll pay a set amount per month to gain access to the service. You can also pay for a year of monthly payments upfront which will reduce the amount you pay per month.

An example of a cheap, monthly subscription plan

An example of a more expensive unlimited subscription plan

Block amounts are plans that offer a fixed amount of storage for a one time payment. These plans allow you to use a Usenet provider's service up until you have downloaded the agreed upon "block" amount. I would be wary of the expiration of these blocks, make sure they don't expire earlier than you expect.

An example of a one time payment block account

Block amount plans are ideal for people new to Usenet. Whereas subscription plans are better for power users as they usually have unlimited downloads and amounts.

Nice to haves

There is a lot of competition in the Usenet provider space. Each provider can compete on the core offerings (that we covered previously), but it's become increasingly difficult to compete on price/core offerings alone (they have to pay for server and storage costs somehow!). So, to help convince people to use their service, providers offer additional benefits to help new customers choose their service over other providers.

Trial Periods

Most of the Usenet provider services are confident enough in their services to offer a free trial to all new customers. I highly recommend using these trials to see if the provider meets your needs.

These free trials will often have a set amount of days and download amounts (in GB) that you can use the service for before the trial ends. You will have to provide your credit card information for the trial. I recommend being conscious of these numbers if you don't want to accidentally get charged when you weren't ready to commit to a plan.

SSL (Security)

I was considering categorising this under "core" offerings, but it's become so common now that I've moved it here. Double check that the provider you're signing up with provides downloads over SSL. This allows you to download your content over an encrypted connection - so no one can spy on what you're downloading! If the provider doesn't offer SSL, I recommend not using them.

You can usually identify this by seeing if they provide SSL Ports on their support page:

VPN (Virtual Private Network)

A number of Usenet provider's monthly subscription plans come with VPN services bundled. This can be super handy for you if you have currently have a VPN, but if you do, then there's less incentive to sign up with that provider.

These VPN services are usually not as good as standalone services, but they are still nice to have if you don't have one already.

Newsreaders / Usenet Search / Browsers

Some Usenet providers also grant you access to their own Usenet searching software. Sometimes they call them "Newsreaders", browsers, search or something similar. I like to think of these as a form of Usenet indexer (think Google, but for Usenet, but not as good). These searching tools simply allow you to browse the files within the Usenet.

I have found these bundled indexers to be quite poor. They don't accurately find what I'm looking for and always return unrelated results. However, they can be a good introduction into Usenets. It can be an easy way to start downloading Usenet content immediately. Since they're bundled for free within certain plans, there's no harm in giving them a try.

My personal recommendation here is that you find an independent Usenet indexing service. These services are specialised in organising, categorising and searching Usenet content. They also provide an API which allows you to automate your Usenet download setup (essential for me).

Customer support

I've found that I have not needed to use customer support with any Usenet provider before. I would recommend looking at the provider's support page in addition to understanding if there's any live customer support provided. This may prove useful if you're new to using Usenets.

Although, I think most help and resources relating to Usenets that you will need can be found on the world wide web ;).

Conclusion

That's it! That's all the points that I considered when selecting a Usenet provider. In summary, find a Usenet provider that most accurately meets your core needs. But also don't forget to consider if the nice to haves are... nice to have!

Setting up Plex Media Server on your Unraid Server (2020)

Harvey Delaney — Sat, 30 May 2020 09:48:24 GMT

We've all tried to catalogue our downloaded media content manually on our machines. As our media content library grows, it becomes a headache to use and manage. Downloaded media content names are borderline illegible, you forget what you have and haven't watched and when you want to watch something, you have to scour through tens, hundreds or thousands of files.

Here's where Plex comes to the rescue.

Plex magically scans and organizes your files, automatically sorting your media beautifully and intuitively in your Plex library.

Your Media | Plex

Stream your media with a consistent experience on all devices.

Plex

Think Netflix, but for your own media content. Plex groups similar media titles, remembers what you have, and haven't watched, where you are up to in videos, has excellent search functionality, provides additional information derived from media content metadata and much, much more.

Plex was one of the biggest reasons behind me buying and setting up my NAS. I wanted the server to hold all my media content and all of Plex's features helped me justify the costs associated with building the NAS.

In this article, I'm going to go through how I set up Plex on my Unraid NAS Server!

Note: This article assumes that you've already setup Unraid. If you haven't check out this article.

Create an account on Plex

Navigate to https://www.plex.tv/, click Sign up, enter your email, password and create your account! All pretty standard stuff.

Before we install Plex on our server, we need to create an Unraid "Share" that is dedicated to holding our media content that we want Plex to catalogue.

Open up Unraid, click on the Shares tab then click Add Share:

On the Share Settings page, fill out Share name and Comments and click Add Share:

Once this share has been created, I recommend creating folders that separate the type of media content you wish to store. To do this, open Unraid in your file browser (\\{UNRAID_IP}) on Windows.

For example, I created folders called Photos and Videos. So I now have the two directories:

\\{UNRAID_IP}\Media\Photos
\\{UNRAID_IP}\Media\Videos

Installing Plex

Community Applications

We're going to be using Docker to help us install/manage Plex. Unraid has excellent support for Docker and comes with it already set up (one of the biggest reasons why I chose the Unraid OS for my NAS).

Before we install Plex, we need to install an Unraid plugin which makes it super easy to search for and install Docker images/containers. The plugin is called Community Applications.

Navigate to Plugins -> Install Plugin, add the following URL and click Install:

https://raw.githubusercontent.com/Squidly271/community.applications/master/plugins/community.applications.plg

You will then have access to a new tab called Apps.

Plex Docker Image

Navigate to the new Apps tab, search for "Plex" and click the install icon for Plex Media Server by plexinc (the official Plex Docker image):

You'll be taken to a screen called Add Container.

There are three fields you'll have to add:

Container Path: /transcode (Host Path 2): /tmp/
Container Path: /data (Host Path 3): /mnt/user/{YOUR_NEW_SHARE_NAME}
Container Variable: PLEX_CLAIM (Key 1): Navigate to https://www.plex.tv/claim/. Log in, then copy paste the Plex Claim code here.

Click Apply, then the Docker image will download and the Docker container will be set up.

Setting up Plex

Now that we've setup our Plex Docker container, we have to perform some final steps within our Plex instance before we can start using it.

Navigate to Docker, left click the Plex logo and click WebUI:

You'll be redirected to Plex server: http://{UNRAID_IP}:32400 (which I recommend bookmarking in your browser).

Now we need to have Plex create libraries for our 2 content directories, Videos and Photos.

Click the settings icon in the top right:

In the side menu, under your new server click Library. Then click Add Library.

First, let's setup a Plex library for Videos. In the popup, select the TV programmes library type, enter a name for your library then go to Add folders:

Click Browse for Media Folder and under data, you'll find the two folders we created in our Media share. Click Videos and click Add.

Then click Add Library:

The reason why it's a bit tricky to find these folders is because we've mapped the data directory in Plex to our Media share (if you look at the Docker container settings).

Repeat the process for Photos, but using the Photos library type. Go back home, and you should see the new Photos and Videos library. If you have any content in these directories, I recommend running Scan Library Files, so the Plex catalogue gets updated (you should do this anytime you add new content):

That's it! Plex is now set up on your Unraid server.

Now, move all your existing media content to these new directories we created in our Unraid share. Don't forget to run Scan Library Files anytime you add new content.

Note: You can also visit https://www.plex.tv/, login and click Launch to watch your media content (as opposed to opening the Unraid Plex web instance).

Next steps

I've written a few articles which provide instructions on how to set up software that automates the downloading of media content. Check them out if you're interested in setting up such a thing:

Switching from Torrents to Usenets - The Why and How

Harvey Delaney — Sat, 30 May 2020 05:11:51 GMT

Everyone knows about Torrents. There's just about any kind of file that are shared in a P2P manner. Torrents only work because of the generosity of people donating their computing and networking resources by seeding (uploading) content for others to download.

As you probably know (if you've found this article) there exists something called the Usenet. Usenets can be used to achieve the same things torrents do, but with a number of benefits!

I discovered Usenets a few months ago while surfing the web and decided to give them a go. I found Usenets to provide a much better means to downloading media content. I haven't since looked back at torrents.

This article is a beginner friendly guide to using Usenets. I'll be covering what Usenets are, weighing the pros and cons of torrents vs Usenets and why/how I switched from using torrents to Usenet. I'll be using Newshosting, NZBGeek and NZBGet to download content.

Overview

Torrents

Before we go into Usenets, let's cover what we know about torrents - how people use them and the pros/cons they have.

Usage Flow

Here's how the average torrent user would download content:

Browse your favourite torrent indexer and look for your desired content
Either download or get the magnet URL (a URL pointing to where the torrent file lives) for a torrent file
Open the torrent file or enter the magnet URL into your torrent client
Wait for the download to complete
Enjoy the downloaded content
Seed your downloaded content so others in the network can download it too (optional)

Pros and Cons

I've constructed a table of my views of the pros and cons of torrents below. Please note, these are only my opinions:

+	Torrenting is free. It doesn't require any paid subscriptions to any services for you to download your content!
+	Setup and configuration to use Torrents is simple and can be automated!
+	Torrents are well known method of downloading content. More people know about torrents and have a basic understanding of how they work already.
-	Torrents rely on seeders. Low amount of seeders will result in slow download speeds.
-	Seeding is usually required while/before downloading
-	Torrents can have malicious or low quality files. Although, this largely depends on the indexers you use and I've personally found this to be very rare if you ignore all .exe files

VPN

If you're using torrents, I recommend using a VPN to keep your torrenting activity private. My voice of VPN is PrivadoVPN. I've found it to be cheaper than other VPN services and hasn't slowed down my torrent downloads at all. They provide a free option (10GB free every month) which you can try out here:

Fast and Secure VPN You Can Trust | PrivadoVPN

PrivadoVPN is the fastest, most private VPN service on the planet. Protect yourself online wherever you go with our fast & easy-to-use VPN that you can trust.

PrivadoVPN

Usenets

The Usenet has a lot of history behind it. In summary, it was originally designed as a bulletin-board service. Usenet eventually became a popular place to store and sort any kind of file. An organisation called Newzbin created the NZB file which pointed to where files existed on the Usenet. A whole ecosystem around Usenet and the NZB file then grew until it became what it is today.

Usenets are different to torrents in that files are stored on centralised servers. downloaded from centralised servers (your Usenet provider), as opposed to from other multiple other "peers" like you do in Torrents.

Usage flow

You log onto your Usenet indexer service and search for the files you wish to download
You obtain the NZB (.nzb) file associated with the content. The NZB file is equivalent to the .torrent file.
You open the NZB file in your Usenet downloader
The Usenet downloader communicates with the provider to access the file(s) associated with the NZB
The file is downloaded

As you can see, the flow of using Usenets are very similar to torrents! I'd say the main difference is that you get faster, higher quality downloads from Usenets, but at a price.

Pros and Cons

+	Usenets providers provide unlimited download speeds. You are only limited by your network
+	Most Usenets providers have SSL ports so no one can snoop on what you are downloading and your IP address is kept private
+	Usenet providers, indexers and downloaders all have a large amount of support, documentation and automation functionality available as well as having large, active communities supporting them
+	Don't have to seed (upload) before/while downloading
-	To use Usenets, you need a subscription to a Usenet provider and indxer service. These subscription costs money
-	Usenets are a less popular, unfamiliar, unknown alternative to torrenting. There may be some apprehension in using Usenets for these reasons (new things can be scary)

Services

Before we start, we'll need to choose and configure three services:

Usenet indexer (paid)
Usenet provider (paid)
Usenet downloader (free)

Usenet Indexer

A Usenet indexer can be thought of as the torrent equivalent of a torrent indexer! These services provide a web interface or a programmatic means to search for NZB files (via API). For this article, we'll be focusing on the web interface way to find NZB files.

A Usenet indexer works similar to how search engines work. They scour the Usenet and use metadata available in the NZB's header to catalogue and organise files it discovers.

Using a Usenet indexer is not mandatory. Some Usenet providers give you a "Newsreader" or their own web interface to search files. Although, I find with some services that it can be tricky to navigate and find the content you're looking for. I recommend using an indexer, as they're highly specialised and are designed to making your search for content easier!

Usenet Providers

A Usenet provider is the most important service that we need. It provides servers that fetch the files from the Usenet and serves them to us.

When looking at choosing a provider, we should be aware of four main things:

Download speeds - The maximum download speed that their servers will provide you. Please note, this is a maximum, not a minimum value, so expect to see speeds below this "maximum" amount.
Download limits - This is the amount of data that you can download from your provider per month.
Retention periods - This is the number of days that the provider will keep a file after it's posted. The higher this retention period is, the more access to older files you'll have.
Connections - At first, you may not think this applies to you, but it does! As previously mentioned, a file stored on the Usenet is split into multiple parts. Your download client will attempt to download these file parts concurrently. The higher the amount of connections, the higher the amount of files you can download at any single time.

With these points in mind, let's choose a provider!

Recommended Providers

Newshosting

Newshosting is my personal choice and what I'll be using for the article. Newshosting is one of the most popular Usenet providers out there (and for good reasons). They have been in the game since 1999, have super long retention period (4299 days!), an easy to use web interface, unlimited downloads and uncapped speeds.

On certain Newshosting plans, a zero-log VPN service called PrivadoVPN comes for free. I've been using PrivadoVPN and have found the speeds, available regions and VPN desktop app to be top notch.

I'm using the special discounted Newhosting ~~$12.95 USD~~ $8.33 USD monthly subscription plan - exclusive to this article!

If you're only looking to give Usenets a go, I highly recommend using Newshosting's free 30GB 14 day trial.

Two other providers I've used before and would recommend are:

Recommended Indexers

NZBGeek (Harvey's Pick)

My personal choice, I chose NZBGeek because it was on sale for Black Friday as well as having good reviews and a good reputation for quality indexes. It's made searching for content easy and has most of what I'm looking for.

If you're not looking to use a dedicated Usenet indexer, I recommend using Easynews which comes with an inbuilt indexer (web search only).

Other indexers I would recommend looking at are:

NinjaCentral
Miatrix
GingaDaddy
DrunkenSlug (if you can get a referral)

Usenet Download Client

A Usenet download client is an application that takes an NZB file and works with your Usenet provider to download the files onto your computer.

There are two main contenders in the Usenet downloader space. Sabnzbd and NZBGet. Both have a large community, lots of features and lots of support. You can't go wrong choosing either.

I personally went with NZBGet and will be using it for this article.

Download NZBGet here.

Download Sabnzbd here.

Using the services

Obtaining Provider Credentials

Once you have signed up with your Usenet provider of choice, you'll either be e-mailed, or have the credentials available on their web portal. These credentials are necessary to start downloading content and are:

Server Address
Port
Username
Password

As previously mentioned, I'm using Newshosting. Your Username and Password are what you used to sign up to Newshosting. Server Address and Port can be found on their support page:

Configuring Usenet Download Client

Once you have downloaded and installed NZBGet, open it! It will open in your browser at the address: 127.0.0.1:6789. Bookmark this page and tag it as NZBGet.

Once NZBGet is opened, navigate to Settings:

Navigate to Paths and update the MainDir and DestDir fields to where you want NZBGet to download your content to:

Then navigate to News-Servers:

You'll need to change the following values:

Name - any name you want to use to identify this server e.g. Newhosting
Host - the Server Address we obtained earlier e.g.
Port - the port we obtained earlier. Usually it's 119 for unencrypted and 563 for SSL (encrypted). I highly recommend using 563.
Username - username we use to log into our Provider service.
Password - password we use to log into our Provider service.
Encryption - switch to No if the port value is 119 or Yes if the port value is 563.
Connections - enter the maximum amount of connections supported by your provider. My Newshosting subscription had a max of 60!

Scroll down and click TestConnection to make sure we've configured everything correctly. If it succeeds, click Save all changes at the bottom left of the screen:

You'll then need to reload NZBGet:

NZBGet supports having multiple providers. So if you decide to add another one, you can simply click Add another Server and repeat the steps.

Using Usenet Indexer

Log into your indexer, and then search for your desired content:

Click on the search result that you think will have the closest match to what you're looking for:

Inside, you need to find the "Download" icon which looks like a cloud. There's two different ways of obtaining the NZB file you need:

Download the NZB file onto your computer
Copy the URL to where the NZB file lives

I prefer going with option 2, but it's up to you!

Downloading Content

Now with your NZB file, or URL to NZB file, open up NZBGet and go to the Downloads tab. Click + Add in the top left:

In the popup:

If you have the NZB URL, paste it into Add from URL.
If you have the file, upload it Add local files.

Then click Submit.

NZBGet will do it's thing by communicating with your Usenet provider and start the download!

Once the download has complete, navigate to where we setup NZB to save our downloads to and we should see our fresh content there!

That's it! Hopefully the whole process wasn't as daunting as you originally thought. As previously mentioned, I don't mind paying the subscription to gain access to a more reliable, faster, secure way of downloading media content!

Automating your Usenet setup

If you find that performing your Usenet downloads manually is becoming a hassle, read the next article in the series to help automated your media downloading workflow: https://blog.harveydelaney.com/configuring-your-usenet-provider-and-indexer-with-sonarr-radarr/

Creating your own mini Redux in React using useReducer, React Context and TypeScript

Harvey Delaney — Mon, 13 Apr 2020 06:08:08 GMT

I was working on building out an organism level component for the RateSetter component library. This component used a number of atom/molecule level components from the library in addition to having a sizeable amount of logic to maintain it's state (it's a medium sized form with a number of different API calls).

Normally to solve a problem like this, I would look to use Redux and potentially Redux Saga for global state and side effects management for my application/component.

However, this component was meant to be distributed across a number of other React projects. The other projects may or may not be using Redux and Redux Saga and I didn't want to enforce the usage of these libraries in order to be able to use this component.

I had experimented with using React Context before but found that the Contexts I built became too large and maintainable that I just ended up switching to Redux + Redux Saga.

From surfing the web, I found inspiration to utilise a very handy React hook: useReducer in combination with React Context to build my own mini Redux!

The basic idea of this approach is that I'd create a Provider which would export:

The current state of the reducer (our "store")
The dispatch method given to us by useReducer.

That way any component that sat under the Provider would have access to the "global" state and also be able to dispatch actions which would update the global state!

I've also used useEffect in the store to handle basic side effects. For example, when a value changes in the store, it can fire an API request to retrieve some value, then save it in the store.

Project

For this article, I'm going to be creating a contrived project that will:

Allow the user to increment and decrement a count which starts at 0
Allow the user to select an option. This option will dictate how much each increment/decrement will add/subtract from the current value
On change of values, post the value to an API (a side effect). The API we'll be using is: https://docs.magicthegathering.io/#api_v1cards_get. It will return the details of a Magic the Gathering card with the ID we supply: https://api.magicthegathering.io/v1/cards/{ID}
Save the card data in our store and display it

As previously mentioned, to help us create this project with a global state, we're going to be using useReducer and React Context. I love TypeScript and will be using it to provide static type checking for our mini Redux solution.

I've made the project we're going to build available on GitHub if looking at the code directly helps you more:

HarveyD/mini-redux-example

An example project created for: https://blog.harveydelaney.com/creating-your-own-mini-redux-in-react/ - HarveyD/mini-redux-example

HarveyDGitHub

Project Structure

package.json
src/
  App.tsx
  store/
    store.actions.tsx
    store.tsx
    store.types.ts
  components/
    CardDetails.tsx
    CardInput.tsx

Store

Let's start off by creating the global state of our project. But before that, let's identify all the Actions that our application will need. I've come up with:

Incrementing the current count by the change value
Decrementing the current count by the change value
Seting the change value
Seting the API response

Actions - `store.actions.ts`

First, let's create the actions we will be using for our application. In store.actions.ts add all of the following:

import { ICardDetails } from "./store.types";

export enum ActionType {
  IncrementId = "counter/increment",
  DecrementId = "counter/decrement",
  SetChangeValue = "value/change/set",
  SetCardDetails = "api/set"
}

interface IIncrementId {
  type: ActionType.IncrementId;
}

interface IDecrementId {
  type: ActionType.DecrementId;
}

interface ISetChangeValue {
  type: ActionType.SetChangeValue;
  payload: number;
}

interface ISetCardDetails {
  type: ActionType.SetCardDetails;
  payload: ICardDetails;
}

export type Actions =
  | IIncrementId
  | IDecrementId
  | ISetChangeValue
  | ISetCardDetails;

export const IncrementId = (): IIncrementId => ({
  type: ActionType.IncrementId
});

export const DecrementId = (): IDecrementId => ({
  type: ActionType.DecrementId
});

export const SetChangeValue = (value: number): ISetChangeValue => ({
  type: ActionType.SetChangeValue,
  payload: value
});

export const SetCardDetails = (response: ICardDetails): ISetCardDetails => ({
  type: ActionType.SetCardDetails,
  payload: response
});

src/store/store.actions.ts

First we're defining the interfaces of our four actions. Having a payload is optional, and is the data our future reducer will be receiving.

Defining an interface for each action may seem like it doesn't produce much value and adds unnecessary overhead. But the value that is obtained by doing this is from our IDE's intellisense feature. By utilising TypeScript's Discriminated Unions (as you can see in Action), our IDE is able to infer the type of our payload given the type of action. I'll go into more detail about this when we create our reducer.

We then create an action for each interface that we defined. If the actions requires a payload, we accept it as a parameter and assign it to payload. Creating all our actions in this way allows us to dispatch actions like:

dispatch(SetCardDetails(details));

as opposed to:

dispatch({ type: ActionType.SetCardDetails, payload: details });

Store - `store.tsx`

Now with our actions created, let's get to writing our store. In store.tsx add:

import React, { createContext, useReducer, useEffect } from "react";

import { SetCardDetails, Actions, ActionType } from "./store.actions";
import { ICardDetails } from "./store.types";

interface IStoreState {
  id: number;
  changeValue: number;
  cardDetails: ICardDetails | null;
}

interface IAppContext {
  state: IStoreState;
  dispatch: React.Dispatch;
}

const initialState: IStoreState = {
  id: 1,
  changeValue: 1,
  cardDetails: null
};

const store = createContext({
  state: initialState,
  dispatch: () => null
});

const { Provider } = store;

const reducer = (state: IStoreState, action: Actions) => {
  const { id: count, changeValue } = state;

  switch (action.type) {
    case ActionType.IncrementId:
      return {
        ...state,
        id: count + changeValue
      };
    case ActionType.DecrementId:
      return {
        ...state,
        id: count - changeValue
      };
    case ActionType.SetChangeValue:
      return {
        ...state,
        changeValue: action.payload
      };
    case ActionType.SetCardDetails:
      return {
        ...state,
        cardDetails: action.payload
      };
    default:
      return state;
  }
};

const AppProvider = ({ children }: { children: JSX.Element }) => {
  const [state, dispatch] = useReducer(reducer, initialState);

  useStoreSideEffect(state, dispatch);

  return {children};
};

const useStoreSideEffect = (
  state: IStoreState,
  dispatch: React.Dispatch
) => {
  useEffect(() => {
    fetch(`https://api.magicthegathering.io/v1/cards/${state.id}`)
      .then(async (res) => {
        const data: { card: ICardDetails } = await res.json();
        dispatch(SetCardDetails(data.card));
      })
      .catch((err) => {
        // do some error handling!
        console.error(`Failed to load card with ID: ${state.id}`);
      });
  }, [state.id, dispatch]);
};

export { store, AppProvider };

src/store/store.tsx

Let's step through this file to understand what's going on:

We define the shape of our store state, and create an initial state for it.

We define what we want to be shared by our Context (state + dispatch) and use createContext to create the initial state of our Context. () => null gets around type errors for: React.Dispatch.

We extract Provider from the createContext. We use it to create a Higher Order Component called AppProvider.

We define our reducer to be used in useReducer. It accepts a "previous" state and an action (from the Union Type we've defined). It handles each action type we've created and depending on what type/payload was passed, returns a "new" state. We're careful to not mutate the state and always return a new state given the action type + payload.

Since we utilised Discriminated Types, our IDEs Intellisense is able to identify what the payload within certain cases must be:

In AppProvider, we utilise useReducer to maintain the state of our store. We pass it the reducer function we just created and the initial store state we've created before. From useReducer we get state and dispatch. state holds the current state of our store, and dispatch is a function that accepts an action of type Actions which will be fed to our reducer to update the store state.

Side Effects

We utilise the power of useEffect to perform side effects on changes in our store.

The useStoreSideEffect function takes state and dispatch, "listens" for an update to the id field saved in our store. Once there's an update, it'll perform the API request to the MTG API to retrieve the card details. Once the request is finished, the SetCardDetails action is dispatched to save it in our store.

You can probably see how this is pretty handy. It's definitely not as good as Redux side effect management libraries such as Thunk or Saga, but it's still pretty good and super simple to setup.

Components

With our store and actions all complete, it's time to utilise them within our app.

`CardInput.tsx`

This component will allow the user to increment/decrement (using buttons), update the change value (via a dropdown) and display the current ID. We utilise useContext to retrieve our state and dispatch from the Provider.

import React, { useContext, useCallback } from "react";
import { store } from "../store/store";
import {
  IncrementId,
  DecrementId,
  SetChangeValue
} from "../store/store.actions";

const CardInput = () => {
  const {
    state: { id, changeValue },
    dispatch
  } = useContext(store);

  const decrementEvent = useCallback(() => dispatch(DecrementId()), [dispatch]);
  const incrementEvent = useCallback(() => dispatch(IncrementId()), [dispatch]);
  const changeValueEvent = useCallback(
    (event: React.ChangeEvent) =>
      dispatch(SetChangeValue(Number(event.currentTarget.value))),
    [dispatch]
  );

  return (
    
      
      Current ID: {id}
      
      
      
    
  );
};

export default CardInput;

src/components/CardInput.tsx

CardDetails.tsx

This is a simple component that grabs the card details from our store and renders them. Again, we utilise useContext to get this data.

import React, { useContext } from "react";
import { store } from "../store/store";

const CardDetails: React.FC = () => {
  const {
    state: { cardDetails }
  } = useContext(store);

  if (!cardDetails) {
    return null;
  }

  return (
    
      {cardDetails.name}
      {cardDetails.manaCost}
      {cardDetails.text}
    
  );
};

export default CardDetails;

src/CardDetails.tsx

`App.tsx`

To be able to utilise useContext and access state and dispatch, we need to wrap our app with AppProvider:

import React from "react";
import { AppProvider } from "./store/store";
import CardInput from "./components/CardInput";
import CardDetails from "./components/CardDetails";

const CardApp = () => (
  <>
    
    
  
);

const App = () => (
  
    
  
);

export default App;

App.tsx

By running our app and clicking increment a few times, we can see it's retrieving card details:

Again, this project is a very contrived example that I whipped up pretty quickly. Some important enhancements we'd want to add to the project would be:

loading indicators
error handling
validation
debouncing

Hope this article was enough to help you get started building out your own mini Redux!

Extending create-react-app to make your own React CLI Scaffolding Tool

Harvey Delaney — Sun, 05 Apr 2020 13:29:15 GMT

At RateSetter, we've recently spun up a bunch of new React projects. Our React projects use create-react-app with specific config for the app to be deployed and hosted by our platform.

To spin up a new frontend project, teams would often clone preexisting React projects, delete irrelevant files and update files with the new name of the app. This resulted in problems such as git history not being cleaned from the cloned project, teams wasting time cleaning up the project and errors arising from when config wasn't correctly updated for the new project.

I saw a real need for a frontend project scaffolding tool. The tool needed to run create-react-app and then augment it with all the configuration and templates necessary to get a frontend app up and running at RateSetter.

I've never made a CLI tool before this, so it was an excellent opportunity to learn how to do so. This article focuses on how I built it, and what I learned from building it.

Building the Scaffolding Tool

I've created a GitHub repository for the scaffolding tool. It might help to have this open while following this article:

HarveyD/create-frontend-app

Contribute to HarveyD/create-frontend-app development by creating an account on GitHub.

HarveyDGitHub

Feel free to clone, fork, star or suggest improvements for it :).

Overview

I designed the CLI tool to have the following flow:

Ask the user for the app name (must follow create-react-app's naming convention)
Present the user with a list of available front-end libraries/frameworks
Present a list of options that the user can choose from to augment their frontend application with
Run create-react-app followed by:
Installing additional NPM dependencies
Adding/modifying package.json entries
Adding templates

This article will focus on creating the CLI tool by augmenting create-react-app - a very popular React scaffolding too. Since we are pragmatic engineers, we will be structuring our CLI in an extensible manner to allow future support of other frontend libraries/frameworks (Angular and React).

For the React side of the tool, we're going to provide the user with three options they can augment their React app with:

Adding Redux and Redux Saga
Adding a Mock API server. This will be a simple Node/Express server that can run alongside the app to provide mock API responses
Adding static code analysis tools (Prettier, TsLint, StyleLint)

Scaffold Dependencies

I've uncreatively called the tool: create-frontend-app. Create that directory and initialise Git (git init) and NPM (npm init). Then, add the following project structure:

.gitignore
package.json
index.js
react/
    config/
        mockApi/
            templates/
                ...
            index.js
        redux/
            templates/
                ...
            index.js
        staticCodeAnalysis/
            index.js
    reactApp.js

We will need to install three Node packages:

inquirer - provide our users with an interactive CLI interface
ora - loading spinner to display while our CLI is running commands
fs-extra - extension utils to fs that helps make our CLI code more concise
colors - to provide our users with a colourful CLI experience :)

In frontend-scaffold-cli run:

npm install --save inquirer ora fs-extra

Option Config Setup

As previously mentioned, we're going to allow the user to choose to add up to three optional augmentation options:

Adding Redux/Redux Saga
Adding static code analysis tools (Prettier/StyleLint/EsLint)
Adding a mock API server

My aim for this tool was to have one config file for each augmentation option that that drives what each will provide the app. I've created the following (TypeScript) schema of how I thought I could best handle the requirement of:

Installation of additional NPM dependencies
Additional/modified package.json entries
Additional/updated project files

interface IConfig {
	name: string;
	description?: string;
	dependencies: string[],
	devDependencies: string[],
	packageEntries: Array<{
    	    key: string;
            value: string
	}>,
	templates: Array<{
            path: string;
            file: IFile;
	}>
}

name is the name of the config.
description is a description of what selecting this augmentation will provide to the user. It will be output to the user and asked as a question.
dependencies / devDependencies are a list of NPM packages that will be installed into the project
packageEntries is a key value pair list which will be added to the project's package.json
templates are a list of path/file entries. The CLI will iterate through each, create a new file using the specified template at the path location.

The config I've created for the Mock API option looks like:

const apiIndex = require("./templates/apiIndex");
const mockController = require("./templates/mockController");

module.exports = {
  name: "withMockApi",
  question: "Do you want to include a local mock API using Node and Express?",
  dependencies: [],
  devDependencies: ["express", "body-parser"],
  packageEntries: [
    { key: "proxy", value: "http://localhost:9001" },
    {
      key: "scripts.dev",
      value: "run-p start mock-api",
    },
  ],
  templates: [
    { path: "mock-api/index.js", file: apiIndex },
    { path: "mock-api/mockController.js", file: mockController },
  ],
};

react/config/mockApi/index.js

You'll notice that we're importing apiIndex from "./templates/apiIndex". This is a template file that should be added into the project and looks like:

module.exports = `
const express = require('express');
const app = express();
const bodyParser = require('body-parser');
const port = 9001;

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

const mockController = require('./mockController');

app.get(
  '/mock-endpoint',
  mockController.mockGetEndpoint
);

app.post(
  '/mock-endpoint',
  mockController.mockPostEndpoint
);

app.listen(port, () => console.log('Example app listening on port: ' + port + '!'));
`;

react/config/mockApi/apiIndex.js

It simply exports the contents of the template as a string. This string content will be used to construct template files in projects created by the tool. All the other template files have similar formats, but I won't display them here because there are too many!

You can see all the config files for the other two options in my GitHub repo:

Redux/Redux Saga: https://github.com/HarveyD/create-frontend-app/tree/master/react/config/redux
Static Code Analysis: https://github.com/HarveyD/create-frontend-app/tree/master/react/config/staticCodeAnalysis

We then need to create an index.js file in root of /config to import and export the three config files:

const withStaticCodeAnalysis = require("./staticCodeAnalysis");
const withRedux = require("./redux");
const withMockApi = require("./mockApi");

module.exports = [withStaticCodeAnalysis, withRedux, withMockApi];

react/config/index.js

The CLI Entry-point

With our config all sorted, we now have to construct the base of our CLI. This entry point needs to extract two pieces of information from our user: the desired app name and framework/library. In our entry point (index.js) add the following:

#!/usr/bin/env node
const inquirer = require("inquirer");
const reactApp = require("./react/reactApp");

const askAppQuestions = () => {
  const questions = [
    {
      type: "input",
      name: "appName",
      message:
        "What name do you want to give your app (should be in kebab case format: `your-app-name`)?"
    },
    {
      type: "list",
      name: "appType",
      message: "What framework do you want to use?",
      choices: ["react", "angular", "vue"]
    }
  ];
  return inquirer.prompt(questions);
};

const appDict = {
  react: reactApp
};

const run = async () => {
  const answer = await askAppQuestions();
  const { appName, appType } = answer;

  // Todo: Perform some validation on appName here to make sure it's kebab case
  if (!appName || appName.length <= 0) {
    console.log(`Please enter a valid name for your new app.`.red);
    return process.exit(0);
  }

  const app = appDict[appType];

  if (!app) {
    console.log(
      `App type: ${appType} is not yet supported by this CLI tool.`.red
    );
    return process.exit(0);
  }

  const appDirectory = `${process.cwd()}/${appName}`;

  const res = await app.create(appName, appDirectory);

  if (!res) {
    console.log("There was an error generating your app.".red);
    return process.exit(0);
  }

  return process.exit(0);
};

run();

index.js

Here we're using inquirer to prompt users to:

Enter an app name
Select an app type from 3 options: React, Angular and Vue

It'll find the next part of the CLI tool using the app type selected, which it'll pass the current directory and app name.

As you can see, appDict only has an entry for React at the moment as we're only focusing on React in this article.

React CLI

If React is selected,index.js will run the function in react/reactApp.js with the name and directory of our app. Let's build it out.

react/reactApp.js is a relatively big file, so we'll be breaking it down. First let's add the dependencies required:

require("colors");
const shell = require("shelljs");
shell.config.silent = true;
const inquirer = require("inquirer");
const fse = require("fs-extra");
const set = require("lodash.set");
const ora = require("ora");

const reactConfigList = require("./config");

react/reactApp.js

Now, we create the default function to be exported by reactApp.js:

module.exports = async (appName, appDirectory) => {
  const selectedConfigList = await askQuestions(appName, appDirectory);

  await createReactApp(appName);
  await installPackages(selectedConfigList);
  await updatePackageDotJson(selectedConfigList);
  await addTemplates(selectedConfigList);
  await commitGit();

  console.log(
    `Created your new React app with settings: ${selectedConfigList
      .map(_ => _.name)
      .join(", ")}. cd into ${appName} to get started.`.green
  );

  return true;
};

react/reactApp.js

This function is performing a number of steps. Each step is separated out into a function and returns a promise (hence why we're awaiting them). We'll flesh out the functions one at a time.

askQuestions

The first thing we need to do is to iterate through our list of augmentation options and ask the user if they want to include it within their app.

const askQuestions = async () => {
  const selectedConfigList = [];

  const questions = reactConfigList.map(config => ({
    type: "list",
    name: config.name,
    message: config.question,
    choices: ["yes", "no"]
  }));

  const answers = await inquirer.prompt(questions);

  reactConfigList.forEach(config => {
    const matchingAnswer = answers[config.name];

    if (matchingAnswer && matchingAnswer === "yes") {
      selectedConfigList.push(config);
    }
  });

  return selectedConfigList;
};

react/reactApp.js

Here we're transforming our list of configs into a list of inquirer questions. We then ask the user each question sequentially. We then return a list of options that were selected as yes. This list is passed to subsequent steps in the CLI.

createReactApp

The most essential function of our CLI tool is this function! We need to run create-react-app to have something we can augment!

const createReactApp = appName => {
  const spinner = ora("Running create-react-app...").start();

  return new Promise((resolve, reject) => {
    shell.exec(
      `npx create-react-app ${appName}`,
      () => {
        const cdRes = shell.cd(appName);

        if (cdRes.code !== 0) {
          console.log(`Error changing directory to: ${appName}`.red);
          reject();
        }

        spinner.succeed();
        resolve();
      }
    );
  });
};

react/reactApp.js

We're utilising ora to show the user a spinner while create-react-app is doing it's thing.

We utilise shelljs to execute npx, which simply runs create-react-app with the app name provided to the CLI tool in index.js. This is wrapped in a Promise due to shelljs being asynchronous, but only providing a callback to indicate its completion.

Once it's done we need to change the current directory to this freshly created React app, stop the spinner and resolve our Promise.

installPackages

This function installs all the dependencies and dev dependencies our augmentations require.

const installPackages = async configList => {
  let dependencies = [];
  let devDependencies = [];

  configList.forEach(config => {
    dependencies = [...dependencies, ...config.dependencies];
    devDependencies = [...devDependencies, ...config.devDependencies];
  });

  await new Promise(resolve => {
    const spinner = ora("Installing additional dependencies...").start();

    shell.exec(`npm install --save ${dependencies.join(" ")}`, () => {
      spinner.succeed();
      resolve();
    });
  });

  await new Promise(resolve => {
    const spinner = ora("Installing additional dev dependencies...").start();

    shell.exec(`npm install --save-dev ${devDependencies.join(" ")}`, () => {
      spinner.succeed();
      resolve();
    });
  });
};

react/reactApp.js

Here, for each selected augmentation provided from askQuestions, we're grabbing the list of NPM dependencies it requires and installing them! We use shelljs to run the NPM command to install the dependency! We perform the same thing for dev dependencies.

updatePackageDotJson

This function updates package.json by adding entries, or modifying them.

const updatePackageDotJson = (configList) => {
  const spinner = ora("Updating package.json scripts...");

  const packageEntries = configList.reduce(
    (acc, val) => [...acc, ...val.packageEntries],
    []
  );

  return new Promise((resolve) => {
    const rawPackage = fse.readFileSync("package.json");
    const package = JSON.parse(rawPackage);

    packageEntries.forEach((script) => {
      // Lodash `set` allows us to dynamically set nested keys within objects
      // i.e. scripts.foo = "bar" will add an entry to the foo field in scripts
      set(package, script.key, script.value);
    });

    fse.writeFile("package.json", JSON.stringify(package, null, 2), function (
      err
    ) {
      if (err) {
        spinner.fail();
        return console.log(err);
      }

      spinner.succeed();
      resolve();
    });
  });
};

react/reactApp.js

First we grab all the entry modifications we need to perform from our selected config list. Then we use fse to grab the existing contents of package.json and parse it. We loop over the list of entries, and use lodash set to output the key value pair in package.json. We then use fse to overwrite the old package.json with the new, mutated package.json.

addTemplates

This function adds the template files associated with the augmentation option to our new project.

const addTemplates = configList => {
  const spinner = ora("Adding templates...");

  const templateList = configList.reduce(
    (acc, val) => [...acc, ...val.templates],
    []
  );

  return new Promise(resolve => {
    templateList.forEach(template => {
      // outputFile creates a directory when it doesn't exist
      fse.outputFile(template.path, template.file, err => {
        if (err) {
          return console.log(err);
        }
      });
    });

    spinner.succeed();
    resolve();
  });
};

react/reactApp.js

Similar to the updatePackageDotJson function, we merge the templates we need to add from all the selected augmentation configs. For each, we utilise fse to output the contents of the template at the desired path.

commitGit

create-react-app intialises a GIT repository. Since we added and modified a number of files we will have a number of untracked changes. We want our user to have a fresh experience, free of these untracked files. So we simply add all the files and commit them!

const commitGit = () => {
  const spinner = ora("Committing files to Git...");

  return new Promise(resolve => {
    shell.exec(
      'git add . && git commit --no-verify -m "Secondary commit from Create Frontend App"',
      () => {
        spinner.succeed();
        resolve();
      }
    );
  });
};

react/reactApp.js

That's it! View the file in it's entirety at: https://github.com/HarveyD/create-frontend-app/blob/master/react/reactApp.js

Running

Let's test out our new CLI tool. Run node index.js or npm run start (if main in package.json points to index.js) and you should see:

Followed by our three options:

It'll then run all the steps:

If everything went okay, you should see test-app within your CLI tool:

It should have all the dependencies, templates and package.json entries we specified!

Publishing and Consuming

Before publishing, make sure to add the following to your package.json:

  ...
  "bin": {
    "": "./index.js"
  },
  ...

package.json

This allows the user to execute the script globally once the CLI tool is installed globally via NPM!

Now all there's left to do is publish your new CLI tool to NPM! To find out how to do this, follow: https://blog.harveydelaney.com/setting-up-a-private-npm-registry-publishing-ci-cd-pipeline/

After publishing your library to an NPM registry, you should be able to install it by running: npm i -g create-frontend-app.

Then by running create-frontend-app, you should be presented with the CLI's questions! After answering the questions and letting the tool run, your new project should be created in the directory you rancreate-frontend-app.

Integrating React Components into an Angular 2+ Project

Harvey Delaney — Tue, 31 Mar 2020 10:45:29 GMT

In my final few weeks at RateSetter, I was working on one last, organism level component to add to our RateSetter Component Library.

We had created multiple "vehicle lookup" UIs that were implemented differently across four different projects (two React and two Angular 2+). When we wanted to add features to this vehicle lookup UI, we found it difficult and time consuming to go through each project, understand how the vehicle lookup was implemented and then update it.

We decided to create one component that would be consumed by each of the UI projects. This allowed us to write code once in the component library, publish it and then simply update the component library dependency in each project. It resulted in more rapid, bug-free development.

I whipped up the vehicle lookup component using React (with TypeScript), Hooks and Context and added it to our component library. Integrating the component within React projects was trivial, but integrating it within Angular 2+ projects was an unknown for me.

After a bit of research and experimenting, I found that integrating a React Component within an Angular 2+ project is surprisingly simple! Follow along to find out how I did it.

Note: This article uses react + react-dom @ 16.13.1 and Angular @ 9.1.0. I used @angular/cli to spin up a basic Angular project called react-integration-app.

Integration!

React Component

Let's say you wanted to integrate the following React component into an Angular 2+ project:

import React, { useCallback, useState } from 'react';

export interface IFeelingFormProps {
  name: string;
  onSubmit: (feelingUpdate: string) => void;
}

const FeelingForm: React.FC = ({ name, onSubmit }) => {
  const [currentFeeling, setCurrentFeeling] = useState('');

  const onFeelingChange = useCallback(
    (event: React.ChangeEvent) => {
      setCurrentFeeling(event.currentTarget.value);
    },
    []
  );

  const onSubmitEvent = useCallback(() => {
    onSubmit(`${name} is feeling: ${currentFeeling}`);
  }, [name, currentFeeling]);

  return (
    
      How are you feeling?
      
      
    
  );
};

export default FeelingForm;

Dependencies

To get React components rendering in your Angular project, we need to install React. In react-integration-app run:

npm i --save react react-dom

It also helps to have React types installed:

npm i -D @types/react @types/react-dom

Also, don't forget to install your component library!

Angular Wrapper

In your Angular project, create a new directory in /src called react-feeling-form and in it create react-feeling-form.component.ts. In it, add:

import {
  Component,
  OnChanges,
  Input,
  Output,
  EventEmitter,
  AfterViewInit
} from '@angular/core';
import * as React from 'react';
import * as ReactDOM from 'react-dom';
import { FeelingForm } from '@harvey/harvey-component-library';
import { IFeelingFormProps } from '@harvey/harvey-component-library/build/feeling-form/feeling-form';

@Component({
  selector: 'app-react-feeling-form',
  template: ''
})
export class ReactFeelingFormComponent implements OnChanges, AfterViewInit {
  @Input() name: string;
  @Output() submitEvent = new EventEmitter();

  public rootId = 'feeling-form-root';
  private hasViewLoaded = false;

  public ngOnChanges() {
    this.renderComponent();
  }

  public ngAfterViewInit() {
    this.hasViewLoaded = true;
    this.renderComponent();
  }

  private renderComponent() {
    if (!this.hasViewLoaded) {
      return;
    }

    const props: IFeelingFormProps = {
      name,
      onSubmit: (res: string) => this.submitEvent.emit(res)
    };

    ReactDOM.render(
      React.createElement(FeelingForm, props),
      document.getElementById(this.rootId)
    );
  }
}

Before trying to understand what's being done here, have a good read through React without JSX.

We've created a template with a div element that will act as our container to mount our React component on. We're binding the id attribute to the rootId field in our Angular component to make it easier to find in the DOM.

We're accepting accepting props that match what our React component requires - @Input() for name and an @Output() Event Emitter for our onSubmit callback.

ngAfterViewInit is used to inform our component that our container (view) has loaded and is ready to have the component mounted on it. It also performs the first render.

If we don't check the view has loaded before attempting to mount our React component (which will happen because ngOnChanges runs before ngAfterViewInit), we'll run into this error:

ngOnChanges is used to "re-render" our React component. Anytime one of the Angular inputs change, so should the props provided to our React component. From React's documentation:

If the React element was previously rendered into container, this will perform an update on it and only mutate the DOM as necessary to reflect the latest React element.

React.createElement is used to create our React component, passing through the props our Angular component received. This is identical to in JSX.

ReactDOM.render takes our freshly created React component and mounts it to our container element (which is retrieved from the DOM by rootId).

Important: We will also need to add the following to tsconfig.json:

...
"skipLibCheck": true,
"allowSyntheticDefaultImports": true
...

skipLibCheck will prevent TypeScript compiler issues when that occur when React is present. allowSyntheticDefaultImports is needed to prevent errors due to how React is exported.

Also, don't forget to add the new component to your app.module.ts Declaration field.

Using the wrapper

You would then use the wrapper component like any other Angular component. For example:

import { Component } from '@angular/core';

@Component({
  selector: 'app-root',
  template: `
    
      Below is the React Component!
      
    
  `
})
export class AppComponent {
  submitEvent($event: string) {
    alert($event);
  }
}

After running ng server you should see:

And by clicking Send feeling you'll see the alert:

Component Library Styles

If your component library exports styles as a standalone CSS file, you'll need to add it to angular.json under the styles field, for example:

...
 "styles": ["src/styles.scss", "./node_modules/@harvey/harvey-component-library/build/styles.css""]
...

Setting up a Private NPM Registry and Publishing CI/CD Pipeline

Harvey Delaney — Mon, 30 Dec 2019 07:00:01 GMT

This article follows on from how I setup a React component library at work. Read it here: https://blog.harveydelaney.com/creating-your-own-react-component-library/.

With the RateSetter component library all set up, we now needed a way to privately host, distribute and manage versioning it. Private hosting was required as we only wanted engineers at RateSetter to see and use the package. Versioning was essential as we needed to be able to manage this evolving library being used in a number of projects throughout the company. The component library being an NPM package met all these requirements.

Note: For this article I'll be creating a scoped NPM package:

Scopes are a way of grouping related packages together, and also affect a few things about the way npm treats the package.

Choosing an NPM Registry

First up, we have to choose how we want to host our registry. There are two options for hosting a private NPM registry for your organisation:

Paid

The benefits of paid service is that you receive a working solution out of the box. There is little to no configuration required. You also receive great documentation and support that help you use the service. The downside is that you have to pay for the service and risk having your packages removed if they violate the services T&Cs or compromised in the unlikely event of a breach.

Some of these paid services are:

NPM ($7 / user / month)
ProGet ($1000 / year)
MyGet ($165 / year)
JFrog Bintray ($150 / month)

It's important to note that while some services are more expensive that others, they offer additional features like hosting other package libraries like NuGet (.Net), Maven (Java) and Gems (Ruby).

Free

Benefits of free services are that they are... free and all the packages are hosted on your servers. The downside is that there is usually overhead with setting up these services as they are required to be self-hosted.

Some examples of free services are:

Setting up the Private NPM Registry

We chose ProGet to host our private NPM registry that would initially host the component library, and later, all other future RateSetter NPM packages.

The main reason behind this decision was that we were currently self-hosting ProGet which was being used for our NuGet packages. It made sense to just use this and add an NPM registry alongside the NuGet registry.

However, if we didn't have ProGet already setup, I would have chosen Verdaccio. It's very popular (8.5k GitHub stars), actively maintained as of (21/12/2019), has many questions asked/answered about it on Stack Overflow in addition to excellent documentation on it's website.

Manual Publishing

As mentioned at the start of this article, we will be publishing to a scoped package registry. For this example, I'm using the scope: @harvey and the package name: react-component-library.

You'll need some information handy from your NPM registry before we can publish our first package:

Your package repository
Registry URL
Email, Username, Password that you wish to use to log in to the registry

With your registry URL, run:

npm config set @harvey:registry http://YOUR_REGISTRY:PORT

This command saves an NPM config that points all scoped registry requests to our private NPM registry.

Now create a new user by running:

npm adduser --registry http://YOUR_REGISTRY:PORT

Note: you have have seen other articles use: npm login. npm login is an alias to adduser and behaves exactly the same way.

Follow the interactive tool by inputting a Username, Password and Email address. The command will then communicate with the registry to create configuration that are saved under .npmrc. .npmrc (aka NPM Runtime Configurations) can be found at: /Users/Username/.npmrc (on Windows). For example, for me it created: //localhost:4873/:_authToken="MaDSxMyURlERcdThvWbg6A==". This file and it's contents are used by NPM to authenticate all requests made to registries (public and private).

Read more about .npmrc here: https://docs.npmjs.com/files/npmrc

Now in package.json in your package repository, change the name from package-name to @scope/package-name, for example:

"name": "react-component-library", => "name": "@harvey/react-component-library",

NPM will then know to publish to your scoped, private registry as opposed to the public NPM registry.

Now running npm publish in your repository's root directory should work:

Automated Publishing (Pipeline)

Manual publishing can work if you're working solo or as a small team. But it'll get to a point where it becomes too hard to keep track of what versions of the package had been deployed. It is also risky and error prone as developers will have permissions to publish on their local machines and could publish packages without running tests or without having their code reviewed through a pull request.

For this section, I'll be going through how I automated NPM publishing by creating our NPM CI publishing pipeline using BuildKite.

First we need to create a BuildKite agent, BuildKite pipeline and link the pipeline to our library repo on GitHub. Read my previous article to find out how: https://blog.harveydelaney.com/setting-up-buildkite-and-your-first-ci-pipeline-in-2-hours/.

We will be defining our pipeline steps through .yml and .bash scripts in the .buildkite directory of our package repository.

Note: To have BuildKite read from the .buildkite directory, you need to add buildkite-agent pipeline upload to the Commands to Run section in your pipeline:

Our pipeline will have two steps:

Run linting and tests
Build, package and publish the package to our private NPM registry

Step 1 - Linting and Tests

The first step on your BuildKite pipeline will be a simple one liner: npm run lint && npm run test.

These commands help make sure code warnings/errors aren't present and components are behaving as expected before the package is published to the registry. You can add other checks such as Prettier to this step as well.

Our pipeline.yml under .buildkite will be:

steps:
  - label: "Install and Lint"
    command: "npm run lint && npm run test"

Step 2 - Build and Publishing

Before Step 2, we will be adding a Block Step. This Block Step will allow the engineer to elect what kind of update they pushed to the component library. We use Semantic Versioning to help us determine what type of release it should be (Major | Minor | Patch):

steps:
  - label: "Install and Lint"
    command: "npm run lint && npm run test"
          
  - block: Publish to NPM
    fields:
    - select: "NPM Package Version"
      key: "npm-semver-type"
      required: true
      options:
        - label: "Major"
          value: "major"
        - label: "Minor"
          value: "minor"
        - label: "Patch"
          value: "patch"

The above .yaml file creates a block step that presents three options to the engineer: Major, Minor and Patch. The selection is saved to the BuildKite build meta-data which can be accessed in proceeding steps.

Now we need to add the final step, Publish:

steps:
  - label: "Install and Lint"
    command: "npm run lint && npm run test"
          
  - block: Publish to NPM
    fields:
    - select: "NPM Package Version"
      key: "npm-semver-type"
      required: true
      options:
        - label: "Major"
          value: "major"
        - label: "Minor"
          value: "minor"
        - label: "Patch"
          value: "patch"

  - label: "Publish"
    command: "bash ./.buildkite/publish.bash"

As you can see, our final step is running a bash command as we need to run a number of commands. The publish.bash file will be:

#!/usr/bin/env bash

# INCREMENT PACKAGE VERSION
semverIncrementType="$(buildkite-agent meta-data get npm-semver-type)"
npm version $semverIncrementType

# INSTALL REQUIRED PACKAGES
npm ci
npm run build

# CONFIGURE NPM SETTINGS
echo "@harvey:registry=http://YOUR_REGISTRY:PORT" >> .npmrc
echo "unsafe-perm=true"
echo "//YOUR_REGISTRY:PORT/:_password:YOUR_NPM_PASSWORD" >> .npmrc
echo "//YOUR_REGISTRY:PORT/:username:YOUR_NPM_USERNAME" >> .npmrc
echo "//YOUR_REGISTRY:PORT/:email:YOUR_NPM_EMAIL" >> .npmrc
echo "//YOUR_REGISTRY:PORT/:always-auth=false" >> .npmrc

# Publish!
npm publish

# UPDATE PACKAGE VERSION IN REPOSITORY
git push

Let's break this bash script down. First, we're grabbing the version from the block step by using buildkite-agent meta-data get and using that to increment the version of the package appropriately - read more about npm version here.

We then get our package ready to be published by running npm ci && npm run build which will produce our built package in the output folder of /build.

The next block of echos redirecting to an .npmrc file may seem a bit strange. It's how I decided configure the build instance to setup the correct NPM settings for publishing to my private NPM registry. It creates all the settings that we created for in the Manual Publishing section above. It was done like this as npm adduser is an interactive command that can't be interact with in our pipeline. You could also replace _password, username and email with a single _authToken field instead.

Next, the script runs the npm publish command!

Finally, we need to update our repository to have the latest version by simply pushing to it (requires the BuildKite agent to have write permissions to your repository). We only have to push because npm version adds a commit with the latest package version in package.json and package-lock.json. This will only occur if publishing is successful.

End Result

Once an update has been pushed to your library repository, BuildKite should read the config we created in .buildkite in the repository then generate and kick off the pipeline:

Clicking Publish to NPM should present the modal:

On clicking continue, the publish.bash script should be run. If everything is successful, you should see your new package was published, in addition to the package.json version being incremented appropriately:

Now go and enjoy the benefits of having your own private NPM registry and NPM publishing CI/CD pipeline!

Configuring your Usenet Provider and Indexer with Sonarr/Radarr on Unraid

Harvey Delaney — Sun, 01 Dec 2019 10:22:32 GMT

Last year I installed Unraid on my NAS in addition to Sonarr/Radarr/Deluge. This setup helped me automatically download and manage my media content. Read more about how I set it up here.

Recently, I was surfing the web and discovered Usenets. To me, Usenets were a new, different method to download media content! I decided to weigh up the pros and cons of torrents and Usenets by giving Usenets a go!

NOTE: this article assumes you've already set up Unraid/Sonarr/Radarr, follow the above article if you haven't.

Torrent Setup

You probably know what torrents are and how they work. If not, read more here.

A typical torrent setup using Sonarr/Radarr might work as follows:

Use Sonarr/Radarr to select a media file to download
Sonarr/Radarr communicates with torrent indexers
Sonarr/Radarr adds the torrent file to Deluge and kicks off the download
Deluge downloads the content and once complete, will copy the file over to a completed directory
Sonarr/Radarr detects the file has been complete, renames and copies the file to the appropriate media directory used by Plex

My take on the pros and cons of using torrents are:

+	Torrenting is free. It doesn't require any paid subscriptions to any services for you to download your content!
+	Setup and configuration to use Torrents is simple and can be automated!
+	Torrents are well known method of downloading content. More people know about torrents and have a basic understanding of how they work already.
-	Torrents rely on seeders. Low amount of seeders will result in slow download speeds.
-	Seeding is usually required while/before downloading
-	Torrents can have malicious or low quality files. Although, this largely depends on the indexers you use and I've personally found this to be very rare if you ignore all .exe files

Usenet Setup

Usenet has a lot of history behind it. In summary, it were originally designed as a bulletin-board service. Usenet eventually became a popular place to store and sort any kind of file. An organisation called Newzbin created the NZB file which pointed to where files existed on the Usenet. A whole ecosystem around Usenet and the NZB file then grew until it became what it is today. Usenets are different to torrents in that files are downloaded from a single server, as opposed to from other multiple other "peers" like you do in Torrents.

An automated Usenet download flow would work as follows:

Use Sonarr/Radarr to select a file or wait for a file to be released
Sonarr/Radarr communicates with a Usenet Indexer to find a matching file
Using the index, Sonarr/Radarr sends the file location to a Usenet Downloader
The Usenet Downloader communicates with a Usenet Provider which serves the content to the downloader
Once the download is complete, it will copy the file to a completed directory
Sonarr/Radarr detects the file has been complete, renames and copies the file to the appropriate media directory

My take on the pros and cons of Usenets are:

+	Usenets providers provide unlimited download speeds. You are only limited by your network
+	Most Usenets providers have SSL ports so no one can snoop on what you are downloading and your IP address is kept private
+	Usenet providers, indexers and downloaders all have a large amount of support, documentation and automation functionality available as well as having large, active communities supporting them
+	Don't have to seed (upload) before/while downloading
-	To use Usenets, you need a subscription to a Usenet provider and indxer service. These subscription costs money
-	Usenets are a less popular, unfamiliar, unknown alternative to torrenting. There may be some apprehension in using Usenets for these reasons (new things can be scary)

In both approaches, there are positives and negatives. As previously mentioned, I decided to give Usenets a go to see if I would like them better than torrents.

Before we get started, we'll need to pick a provider and an indexer to use. There are many indexers and providers out there, I'll just be suggesting a few of them. I would encourage you to do your own research as well.

Recommended Usenet Providers

Newshosting (Harvey's Recommendation)

Newshosting is my personal choice and what I'll be using for the article. Newshosting is one of the most popular Usenet providers out there (and for good reasons). They have been in the game since 1999, have super long retention period (4299 days), an easy to use web interface, unlimited downloads and uncapped speeds. Another plus was that it integrated seamlessly with Sonarr and Radarr.

On certain Newshosting plans, a zero-log VPN service called PrivadoVPN comes for free. I've been using PrivadoVPN and have found the speeds, available regions and VPN desktop app to be top notch.

I'm using the special discounted Newhosting ~~$12.95 USD~~ $7.95 USD monthly subscription plan - exclusive to this article!

If you're only looking to give Usenets a go, I highly recommend using Newshosting's free 30GB 14 day trial.

Two other providers I've used before and would recommend are:

Read this article to learn more about choosing the best Usenet provider for you.

Recommended Usenet Indexers

NZBGeek (Harvey's Pick)

My personal choice, I chose NZBGeek because it was on sale for Black Friday as well as having good reviews and a good reputation for quality indexes.

Other indexers I would recommend are:

NinjaCentral
Miatrix
GingaDaddy
DrunkenSlug (if you can get a referral)

Usenet Downloaders

Sonarr/Radarr are configured to use a number of download clients. Sonarr/Radarr support four different Usenet clients:

Sabnzbd
Nzbget
Pneumatic
UsenetBlackhole

I was tossing up between Sabnzbd and NZBGet (due to both having a high amount of features/support/community). I ended up going with NZBget as I preferred the UI of it slightly more and will be using it for this article.

UnRAID Setup

In UnRAID, navigate to Plugins and open the Community Applications plugin (assumed to have been installed already). Search for nzbget and click Install:

Use the default port (6789) and setHost Path 2 to /mnt/user/Downloads/NZB and hit Done:

Once the Docker image has been downloaded and container set up, open NZBGet in your browser. You'll be prompted for a username and password which are:

Username: nzbget
Password: tegbzn6789

Then in the top menu select Settings. Then select News-Servers on the left:

Your Usenet provider should provide four essential pieces of information:

Server address
Port (usually 119 for unencrypted and 563 for encrypted)
Username
Password

All three of these details were available in Newshosting immediately after I joined the monthly subscription plan. Enter them in under the correct fields:

Set Encryption to Yes and change the port from 119 -> 563 if you want to encrypt your Usenet download traffic (highly recommended).

Test the connection and then click save all changes in the bottom left and you'll be prompted to restart NZBGet. Restart and NZBGet will be all ready to use!

Since we're configuring both Sonarr and Radarr to work with NZBGet, we need to add two categories on NZBGet. Do this by going to Settings -> Categories -> Add Another Category. Enter Radarr in the Name field then click Save:

Repeat this for Sonarr as Category2.

Radarr (and Sonarr) Setup

With our Usenet download client and indexer (doesn't require setup) all ready, now we just have to configure Sonarr/Radarr to use them.

I'll just be going through how to set Radarr up - Sonarr will be identical, you'll just have to repeat the steps.

Add Indexer

Step one is to add the indexer. Navigate to Settings -> Indexers -> Add Indexer (+):

Under Usenet -> Newznab -> Presets, find your indexer of choice (I use NZBGeek). Then simply add the API Key you got from your indexer, click Test and if everything is okay, click Save:

Now if you navigate to any content you have and perform a manual search - you will see a number of results from NZBGeek:

Add Download Client

Navigate to Settings -> Download Client -> Add new client (+):

Then select NZBGet (first option):

Then in the popup, add:

Host: IP of your Unraid Server
Port: 6789
Username: nzbget
Password: tegbzn6789

Note: you'll notice that Category has Radarr. If you are setting up Sonarr, you need to make it Sonarr which will map to the correct category that we added to NZBGet previously.

Test it and if successful, click Save.

That's it! You've setup your Usenet indexer and Usenet download client.

NOTE: Remember to repeat the above steps for Sonarr!

Testing it out

In Radarr, search for some new content and click Add and Search. After a few seconds, you should see your new media file start downloading in NZBGet:

After the download has complete, NZBGet will move the file to the /downloads/completed/radarr directory. Radarr will then copy the file over to your Media directory automatically. You can then open up your Plex server, refresh your library and watch your new media content!

Maintaining code formatting and quality automatically on your front-end projects using Prettier, ES/TSLint and StyleLint

Harvey Delaney — Sun, 24 Nov 2019 08:04:53 GMT

At work, I've spun up a handful of React projects that have a number of engineers working on it (onshore and offshore). When reviewing code, I'm very particular about how code is formatted and it's quality. This is probably an artifact of me always using formatters and linters in my projects, so it's become a personal standard of mine to make sure these things are done correctly.

While reviewing pull requests, I find myself sometimes pointing out code formatting (inconsistent spacing, incorrect indentation, etc) and quality issues (using any, using var etc). I try to avoid doing so as I don't think it should be part of the code review - everyone knows how frustrating it is to have formatting issues pointed out. So I began thinking of ways that I could encourage adherence to formatting/quality standards without having to point them out or fix them myself. The answer was to automatically enforce (not just recommend) by using a collection of formatters/linters.

Prettier

My absolute favourite code formatter is Prettier! I've used Prettier on every single JavaScript project I've created at home and at work. It's an opinionated code formatter who's opinion I happen to strongly agree with. It formats all code very clean, easy to read format. To start off with I just used the Prettier VSCode extension which I could just run over code that I'm working on. Sometimes I forgot to run it so I then move onto adding it to run on file save, and added a script to run over all my files in package.json.

From their website, Prettier is described as:

Prettier is an opinionated code formatter. It enforces a consistent style by parsing your code and re-printing it with its own rules that take the maximum line length into account, wrapping code when necessary.

It has support for JavaScript, JSX, Angular, Vue, Flow, TypeScript, CSS, Less, and SCSS, HTML, JSON, GraphQL, Markdown and YAML.

Installation and configuration

Installing Prettier is simple, run:

npm i --D prettier

There are a number of formatting rules that Prettier provides, I'm a fan of just using the default ruleset. However, I still think it's handy to have a Prettier config file just to make sure everyone is using the same Prettier config and in case new formatting rules need to be added. Create prettierrc.json file then place the following in it:

{
  "trailingComma": "es5",
  "tabWidth": 4,
  "semi": false,
  "singleQuote": true
}

I would recommend installing the Prettier VSCode extension so you can see formatting errors in real-time.

Now we have to add an NPM script to run Prettier over all of our files and a script that runs a check over our files. In package.json add:

"prettier": "prettier --write src/**/*.{js,tsx,scss}"
"prettier:check": "prettier --list-different src/**/*.{js,tsx,scss}"

Make sure to update these scripts to run on files types that you're using in your project. I use prettier:check in the pre-commit hook (to be discussed later) and have the prettier option to just allow people to run Prettier over all their code and do it's magic.

ES/TSLint

For a majority of my projects I use TSLint by Palantir. However, I would use ESLint for linting any JavaScript I write. TSLint is described as:

TSLint is an extensible static analysis tool that checks TypeScript code for readability, maintainability, and functionality errors. It is widely supported across modern editors & build systems and can be customized with your own lint rules, configurations, and formatters.

TSLint can be used as a code formatter, but I prefer using Prettier for that. We will only be enabling rules that pertain to code quality issues. Examples of these issues might be that the any type is being used, a console.log is present or using let when const should be used.

Installation and configuration

Install TSLint by running:

npm i -D tslint

Then create a tslint.json file and place the following config inside:

{
  "defaultSeverity": "error",
  "extends": ["tslint:recommended"],
  "jsRules": {},
  "rules": {
    "object-literal-sort-keys": false,
    "ordered-imports": false,
    "quotemark": [true, "single", "jsx-double"],
    "trailing-comma": false,
    "semicolon": [true, "always", "ignore-bound-class-methods"],
    "arrow-parens": [true, "ban-single-arg-parens"],
    "interface-over-type-literal": false
  },
  "rulesDirectory": []
}

Obviously feel free to add/remove rules as you prefer, but make sure that the rules here don't conflict with Prettier, for example, the quotemark rule by default wants double, whereas we've configured Prettier to use single quotemark.

We also need to introduce an NPM script to run TSLint over all our files, in package.json scripts add:

"tslint:check": "tslint -c tslint.json 'src/**/*.{ts,tsx}'"

I would also recommend installing the TSLint VSCode extension, again, to see linting errors in real-time.

StyleLint

I'm a huge fan of using CSS preprocessors like Sass over CSS (I just cannot live without nesting) and I was using SassLint for all my linting needs. SassLint worked fine, but I wanted to use a more generic style linter that could be used across multiple stylesheet type. A bit of research lead me to StyleLint. I haven't looked back since.

It has support for SCSS, Sass, Less and SugarSS. It's described as being:

A mighty, modern linter that helps you avoid errors and enforce conventions in your styles.

StyleLint provides hundreds of options that you can configure it with. It also has a number of great "base" rule-sets that you can simply install from NPM, add one line in package.json and have everything just work. For example, there are base rule-sets for CSS, SCSS and LESS. It's also super easy to extend and override these rule-sets.

Installation and configuration

We'll just be configuring StyleLint for css. Install StyleLint using:

npm i -D stylelint

The easiest way to get up and running with StyleLint is to extend the basic CSS rule-set. This requires an additional install:

npm i -D stylelint-config-recommended

Now create a .stylelintrc.json and in it place:

{
  "extends": "stylelint-config-recommended"
}

You can read all the basic rules provided by this here. You can also easily extend/override rules by adding a rules key like:

{
  "extends": "stylelint-config-recommended",
  "rules": {
    "at-rule-no-unknown": [ true, {
      "ignoreAtRules": [
        "extends"
      ]
    }],
    "block-no-empty": null,
    "unit-whitelist": ["em", "rem", "s"]
  }
}

Again, StyleLint has a VSCode extension I recommend installing. Now let's add a package.json script:

"stylelint:check": "stylelint \"src/**/*.css\"",

Pre-commit hooks

With Prettier, ES/TSLint and StyleLint installed and configured, we just need a way to enforce these have been run before the pull request is created!

An excellent NPM library that was recommended to me by a co-worker is called pre-commit. pre-commit allows you to run an NPM script before a git commit is performed. When this script is run and an error is thrown, the commit won't occur. This is incredibly useful and I've utilised it to enforce the engineer has run Prettier, Es/TsLint and StyleLint before submitting their code for review.

To get it working we first have to install it:

npm i -D pre-commit

Before we use pre-commit, create a new package.json script:

"lint-all": "npm run prettier:check && npm run tslint:check && npm run stylelint:check"

This script will run all of our formatters/linters sequentially and output any errors. This is the script we will get pre-commit to run.

In package.json add:

  "pre-commit": [
    "lint-all"
  ],

Now, anytime a git commit is run, Prettier, TSlint and StyleLint will all be run over your code. If there are any errors from them, the commit will fail and the errors will be output. Only once all errors are resolved will the commit work.

Hope this article helps you maintain your front-end project's code quality + formatting without having to point these our in pull requests!

Creating a React Component Library using Rollup, Typescript, Sass and Storybook

Harvey Delaney — Sat, 23 Nov 2019 01:45:06 GMT

At work we have a number of front-end projects that are worked on by different project teams. Each of these projects are designed by our internal designer and had a lot common with them (same inputs, buttons designs etc). Up until now, we had no shared style-sheets or components. Components across projects were created from scratch, rewritten or copy pasted over each time. I saw a real need for an internal component library. This component library would enable teams to pull down an NPM package, import components, provide some props and have their components ready to use.

I strongly believed the component would help speed up front-end development by removing the need to write/copy paste components over to new projects. I thought it would also be great to learn how to create, manage and help adopt a component library used in production across multiple engineering teams and projects. So I created a React component library!

I wrote this article to share my experience around researching and building the component library. It''ll also cover how you can build your own React component library!

Component Library Overview
Component Library Research
- Existing Component Libraries
- Component Library Creation Tools
Custom Library Technology Choices
Creating the Component Library
Publishing the Component Library
Using the Component Library
- Installing from NPM Registry
- Installing Locally
Adding More Components
Introducing Code Splitting
Further steps

Component Library Overview

There were a number of requirements I had for the component library. Those requirements being that it should:

Use React
Use Sass
Use TypeScript (and bundle/export TypeScript types)
House a number of components
Have each component thoroughly tested (using Jest/React Testing Library)
Bundle and transpile the components and be able to be published to an internal (or external) NPM registry as a single package
Have components that are able to be displayed and interacted with on an internal (or external) website without having to set the library up locally
Be versionable using Semantic Versioning

Just give me the component library code!

Sometimes it's just easier to clone a repository as opposed to following along with the article creating everything from scratch, I get it. For your convenience I've created a GitHub repository which has all the configuration needed for creating your own component library:

HarveyD/react-component-library

Contribute to HarveyD/react-component-library development by creating an account on GitHub.

HarveyDGitHub

I've created branches for different variants of the component library that people have requested:

Give it a star if it helps you out :). If you feel like it can be improved, please raise a pull request or a GitHub issue!

Component Library Research

Existing Component Libraries

Before I created our own component library, I had a look around to see what other React component libraries were out there and whether we could use them. Turns out there's quite a few great libraries such as:

Our designer had created the designs for our components with specific styles that none of these libraries could easily achieve without us going into the internals of the components and altering them. So I decided it would be better to create our own components.

Component Library Creation Tools

I then had a look around at a number of React project skeletons I could use to bootstrap the component library. Here are the libraries I considered:

Create React App - "Set up a modern web app by running one command"
Create React Library - "CLI for creating reusable react libraries"
NWB - "a toolkit for React, Preact, Inferno & vanilla JS apps, React libraries and other npm modules for the web"

All of these libraries had excellent documentation, support and a number of features that made it easy to get different flavours of React project/libraries up and running. But there was always an issue I ran into that hindered my ability to configure the library to meet my requirements (since they abstracted config away).

Create React App is designed for creating a web application, not a library. NWB was promising, but didn't have great support for TypeScript when I used it. Create React Library had inflexible config, making it tricky to get my library transforming and bundling the Sass files correctly.

Due to all the above not matching the requirements I had for the component library, I decided to write all the config from scratch! This approach gave me with much more control around the config of the project.

Custom Library Technology Choices

Before I began building out the React component library, I needed to choose the languages and libraries to use to build the library. As you've seen in the article's title, I've chosen TypeScript, Sass, Rollup and Storybook. Here's my rationale behind choosing them.

TypeScript

TypeScript is my go to language for all my front-end projects. TypeScript provides type safety over your functions (and components). I've found that this type safety has helped me:

Catch runtime errors at compile time
Understand how to use functions/components without looking at the code itself
Write code quicker (thanks to my IDE's autocomplete)
Refactor code faster and with more confidence
Write code that is easier to read and understand

Building the component library using TypeScript allows you to easily bundle the types of your components for no extra work! Anyone who is using TypeScript and installs/uses the library will be eternally grateful to you (it is also kind of expected nowadays).

There is an overhead of creating types/interfaces for all your functions/components, but it is something that will save you ALOT of time later down the track.

Sass

Sass is my go to library for writing CSS. It's a CSS pre-processor that augments CSS with a number of handy features. It allows you to write CSS with variables, nesting, mixins, loops, functions, imports and more! The features I use most are variables and nesting.

Sass has always helped me write CSS faster, that's less verbose, is more maintainable and with less repetition.

If you don't want to use Sass, I'll also be showing you how to use CSS Modules, LESS, or Stylus to build your component library.

If you want to use styled-components, check out this branch of my component library repo.

Rollup

My decision for a JavaScript module bundler to use was between Webpack and Rollup. I decided to use Rollup for the component library after researching what the differences between Webpack and Rollup are. This article goes into depth about these differences. The take-home is that:

Use Webpack for apps, and Rollup for libraries

This isn't a hard and fast rule, it's more of a guideline (which I have opted to follow). Rollup can be used to build apps and Webpack can be used to build libraries.

I also recommend reading this Medium article and this Stack Overflow post that further discuss the difference between Rollup and Webpack.

Storybook

I found out about Storybook in an interview I had with a web agency a few years ago. They explained they were using Storybook to help them create, experiment and display their React components. After the interview, I did my own research and experimentation with Storybook and fell in love with the tool. Storybook felt like a perfect fit for the component library.

Storybook provides a sand-boxed environment for your front-end projects that helps you to develop your components in isolation. The sandbox environment encourages engineers to create components that aren't tied in with logic within your application (e.g. coupled with and reliant on a Redux store). This results in components that are more generic and more re-usable.

Storybook also has the ability to be exported as a static webpage. You can easily host these files which will allow everyone in your organisation to see how components look/interact without having to clone and setup the repository locally. This is perfect to help out our product manager and designer friends!

For example, I've exported the component library Storybook files built in this article and currently host it on my Express server here. Here it is in an iframe:

Creating the Component Library

First, we have to initialise NPM (npm init), set the name field to react-component-library and initialise Git (git init).

We're creating a React component library and need React to help us build our components:

npm i --save-dev react react-dom @types/react

We will also configure react and react-dom as Peer Dependencies. Having them as peer dependencies will mean that once our library is installed in another project, they won't be automatically installed as dependencies. Instead, NPM will provide soft assertion by outputting a warning if a matching version of the dependencies haven't been installed alongside our component library.

We will later introduce a Rollup plugin called rollup-plugin-peer-deps-external. This plugin prevents packages listed in peerDependencies from being bundled with our component library (reducing our bundle size by 109kb).

Create an entry in package.json called peerDependencies and add react and react-dom. I've specified the version target as >=16.8.0 as we want a version of React that supports React Hooks (released in 16.8.0):

...
"peerDependencies": {
  "react": ">=16.8.0",
  "react-dom": ">=16.8.0"
}
...

package.json

Now let's create the skeleton of our library by creating the following files and directories:

.storybook/
  main.js
.gitignore
package.json
rollup.config.js
tsconfig.json
jest.config.js
jest.setup.ts
src/
  TestComponent/
    TestComponent.tsx
    TestComponent.types.ts
    TestComponent.scss
    TestComponent.stories.tsx
    TestComponent.test.ts
  index.ts

Folder Structure

We will output our transpiled, bundled files within a build directory.

In TestComponent.tsx we will have a very simple component:

import React from "react";

import { TestComponentProps } from "./TestComponent.types";

import "./TestComponent.scss";

const TestComponent: React.FC = ({ theme }) => (
  
    I'm the test component
    Made with love by Harvey
  
);

export default TestComponent;

src/TestComponent/TestComponent.tsx

With the prop types imported from and defined in TestComponent.types.ts:

export interface TestComponentProps {
  theme: "primary" | "secondary";
}

And in TestComponent.scss we will have:

.test-component {
    background-color: white;
    border: 1px solid black;
    padding: 16px;
    width: 360px;
    text-align: center;
    
    .heading {
        font-size: 64px;
    }

    &.test-component-secondary {
        background-color: black;
        color: white;
    }
}

src/TestComponent/TestComponent.scss

src/index.ts will be used as the entry point for Rollup. We will use a pattern called Barrel Exports to expose our components in the entry point. We do this by importing, then exporting all our components. Components exported here will be bundled by Rollup. In this file add:

import TestComponent from "./TestComponent/TestComponent";

export { TestComponent };

src/index.tsx

We will leave TestComponent.stories.js and TestComponent.test.tsx empty for now.

Adding TypeScript

Run npm i -D typescript and in tsconfig.json add:

{
  "compilerOptions": {
    "declaration": true,
    "declarationDir": "build",
    "module": "esnext",
    "target": "es5",
    "lib": ["es6", "dom", "es2016", "es2017"],
    "sourceMap": true,
    "jsx": "react",
    "moduleResolution": "node",
    "allowSyntheticDefaultImports": true,
    "esModuleInterop": true
  },
  "include": ["src/**/*"],
  "exclude": [
    "node_modules",
    "build",
    "src/**/*.stories.tsx",
    "src/**/*.test.tsx"
  ]
}

tsconfig.json

This is the config I recommend using. It's important to take note of "declaration": true and "declarationDir": "build". This will generate and place the types of our components within our build folder. Feel free to adjust this config to your liking.

Adding Rollup

Now we need to install Rollup in addition to some plugins required for our component library to be transpiled and bundled correctly:

npm i -D rollup rollup-plugin-typescript2 @rollup/plugin-commonjs @rollup/plugin-node-resolve rollup-plugin-peer-deps-external rollup-plugin-postcss node-sass

In rollup.config.js add:

import peerDepsExternal from "rollup-plugin-peer-deps-external";
import resolve from "@rollup/plugin-node-resolve";
import commonjs from "@rollup/plugin-commonjs";
import typescript from "rollup-plugin-typescript2";
import postcss from "rollup-plugin-postcss";

const packageJson = require("./package.json");

export default {
  input: "src/index.ts",
  output: [
    {
      file: packageJson.main,
      format: "cjs",
      sourcemap: true
    },
    {
      file: packageJson.module,
      format: "esm",
      sourcemap: true
    }
  ],
  plugins: [
    peerDepsExternal(),
    resolve(),
    commonjs(),
    typescript({ useTsconfigDeclarationDir: true }),
    postcss()
  ]
};

rollup.config.js

Lets run through the everything in this config file.

`input`

points to src/index.ts. Rollup will build up a dependency graph from this entry point and then bundle all the components that are imported/exported.

`output`

is an array with two objects, each specifying output config. We will be outputting two bundles in two different JavaScript module formats:

CommonJS - CJS
ES Modules - ESM

This is because we want to support tools that use CommonJS (Webpack, Node.js) and tools that work with ES Modules (Webpack 2+, Rollup). Read more about CJS, ESM and other output formats in Rollup's documentation.

ES Modules have a static module structure. This enables bundling tools that target ESM to perform tree shaking. Tree shaking is a process of dead code elimination where Bundlers will attempt to only bundle code that is being used.

CommonJS modules have a dynamic module structure. This makes it difficult for bundling tools that target CommonJS to perform tree shaking. This means that even if only one component is imported from our library, all components will be bundled.

Read more about CJS, ESM and tree shaking here.

We will import the filename of our desired CommonJS and ES Modules index file from package.json. The main field in package.json points to our bundled CommonJS entry point and the module field points to our bundled ES Modules entry point.

If a tool can support ESM, it'll use module otherwise it'll use main.

We will be outputting all our bundles to the build directory. Add the main and module fields in package.json in addition to the files field to instruct NPM what files to include when our component library is installed as a dependency:

...
"main": "build/index.js",
"module": "build/index.es.js",
"files": ["build"],
...

package.json

`plugins`

is an array of 3rd party Rollup plugins. The plugins I've included are ones that are required to bundle the component library. A complete list of plugins can be found here. Let's go through all the plugins we're using:

peerDepsExternal (rollup-plugin-peer-deps-external) - prevents Rollup from bundling the peer dependencies we've defined in package.json (react and react-dom)
resolve (@rollup/plugin-node-resolve) - efficiently bundles third party dependencies we've installed and use in node_modules
commonjs (@rollup/plugin-commonjs) - enables transpilation into CommonJS (CJS) format
typescript (rollup-plugin-typescript2) - transpiles our TypeScript code into JavaScript. This plugin will use all the settings we have set in tsconfig.json. We set "useTsconfigDeclarationDir": true so that it outputs the .d.ts files in the directory specified by in tsconfig.json
postcss (rollup-plugin-postcss) - transforms our Sass into CSS. In order to get this plugin working with Sass, we've installed node-sass. It also supports CSS Modules, LESS and Stylus. I recommend reading the documentation here if you want to use a different CSS pre-processor and/or to learn all the other settings available.

Other useful plugins you might want to add are:

@rollup/plugin-images - import image files into your components
@rollup/plugin-json - import JSON files into your components
rollup-plugin-terser - minify the Rollup bundle

Running Rollup

Now we need to add the first package.json script entry that will run our Rollup bundling process:

...
"scripts": {
  "build" "rollup -c"
}
...

package.json

The -c argument is short for --config which accepts a Rollup config file name as a parameter. If no file is provided it'll attempt to use rollup.config.js within the same directory.

Now by running npm run build you should see Rollup do it's thing and create a /build folder that will contain the compiled (CJS and ESM) component library, ready to be published to an NPM registry:

Note: you might want to create a .gitignore file and add node_modules and build to it.

Adding Storybook

We will be adding Storybook to our component library manually.

Install Storybook for React, it's core dependencies (Babel) and Webpack loaders:

npm install --save-dev @storybook/react @babel/core babel-preset-react-app babel-loader sass-loader

Storybook uses Webpack to serve and load JS modules. It ships with default config outlined here. Since we are using Sass and TypeScript, we'll need to extend the default config with additional Webpack rules to get Storybook working with our library.

Yes, it's less than ideal that we have to configure both Webpack and Rollup for our component library. Until Storybook supports Rollup or Webpack becomes the recommended module bundler for JavaScript libraries, we'll have to stick with this!

In .storybook/main.js and add:

const path = require("path");

module.exports = {
  stories: ["../src/**/*.stories.tsx"],
  // Add any Storybook addons you want here: https://storybook.js.org/addons/
  addons: [],
  webpackFinal: async (config) => {
    config.module.rules.push({
      test: /\.scss$/,
      use: ["style-loader", "css-loader", "sass-loader"],
      include: path.resolve(__dirname, "../")
    });

    config.module.rules.push({
      test: /\.(ts|tsx)$/,
      loader: require.resolve("babel-loader"),
      options: {
        presets: [["react-app", { flow: false, typescript: true }]]
      }
    });
    config.resolve.extensions.push(".ts", ".tsx");

    return config;
  }
};

.storybook/main.js

Here, we're instructing Storybook where to find our stories, specifying our Stoprybook addons (none so far, but check out all the addons available) and using sass-loader + babel-loader to compile our Sass + TypeScript files.

Storybook has style-loader and css-loader as dependencies already. Hence why we only had to install sass-loader.

We also need to create package.json script entries for Storybook:

...
  "scripts": {
    ...
    "storybook": "start-storybook -p 6006",
    "storybook:export": "build-storybook",
    ...
  }
...

package.json

storybook:export is optional! It allows you to export Storybook as static files that can can be served anywhere. This is helpful for showing off your components to non-technical members of your team. You can chuck the files into an S3 Bucket, use GitHub pages or spin up a custom ExpressJS server - the choice is yours!

That's it! Storybook is now configured for our component library.

Now, we have to create stories for TestComponent. Open src/TestComponent/TestComponent.stories.tsx and place:

import React from "react";
import TestComponent from './TestComponent';

export default {
  title: "TestComponent"
};

export const Primary = () => ;

export const Secondary = () => ;

src/TestComponent/TestComponent.stories.tsx

This is a very basic story, showing the two variants of our component.

Now run npm run storybook and Storybook will run it's magic and load up your components at http://localhost:6006. Storybook should look like:

Adding Jest and React Testing Library

Maintaining a high level of test coverage on components is extremely important for your component library. We need to have confidence that when we make changes to our components that we won't be breaking how the component is being expected to behave in another project. For testing our React components, we will be using Jest and React Testing Library.

Start by installing Jest, React Testing Library:

npm i --save-dev jest ts-jest @types/jest identity-obj-proxy @testing-library/react @testing-library/jest-dom

In jest.config.js add:

module.exports = {
  roots: ["./src"],
  setupFilesAfterEnv: ["./jest.setup.ts"],
  moduleFileExtensions: ["ts", "tsx", "js"],
  testPathIgnorePatterns: ["node_modules/"],
  transform: {
    "^.+\\.tsx?$": "ts-jest"
  },
  testMatch: ["**/*.test.(ts|tsx)"],
  moduleNameMapper: {
    // Mocks out all these file formats when tests are run
    "\\.(jpg|ico|jpeg|png|gif|eot|otf|webp|svg|ttf|woff|woff2|mp4|webm|wav|mp3|m4a|aac|oga)$":
      "identity-obj-proxy",
    "\\.(css|less|scss|sass)$": "identity-obj-proxy"
  }
};

jest.config.js

The creator of react-testing-library has also created a library called jest-dom. It extends Jest, providing a number of helpful Jest matchers like toHaveClass, toHaveAttribute, toBeDisabled and so on. If you want to add it, in jest.setup.ts add:

import "@testing-library/jest-dom";

jest.setup.ts

And add two new scripts in package.json to run the tests:

...
"scripts":
    {
        ....
        "test": "jest",
        "test:watch": "jest --watch",
        ....
    }
...

package.json

test should be used on your CI/CD pipeline and test:watch should be used when you're running your tests locally (they will re-run whenever a file is changed).

Then in TestComponent.test.tsx create two simple tests:

import React from "react";
import { render } from "@testing-library/react";

import TestComponent from "./TestComponent";
import { TestComponentProps } from "./TestComponent.types";

describe("Test Component", () => {
  let props: TestComponentProps;

  beforeEach(() => {
    props = {
      theme: "primary"
    };
  });

  const renderComponent = () => render();

  it("should have primary className with default props", () => {
    const { getByTestId } = renderComponent();

    const testComponent = getByTestId("test-component");

    expect(testComponent).toHaveClass("test-component-primary");
  });

  it("should have secondary className with theme set as secondary", () => {
    props.theme = "secondary";
    const { getByTestId } = renderComponent();

    const testComponent = getByTestId("test-component");

    expect(testComponent).toHaveClass("test-component-secondary");
  });
});

src/TestComponent/TestComponent.test.tsx

After running npm run test:watch you should see Jest run and output:

Indicating we have set it up correctly!

Final NPM Config

After going through these steps, your package.json should look like:

{
  "name": "react-component-library",
  "version": "1.0.0",
  "main": "build/index.js",
  "module": "build/index.esm.js",
  "files": [
    "build"
  ],
  "scripts": {
    "build": "rollup -c",
    "test": "jest",
    "test:watch": "jest --watch",
    "storybook": "start-storybook -p 6006",
    "storybook:export": "build-storybook"
  },
  "peerDependencies": {
    "react": ">=16.8.0",
    "react-dom": ">=16.8.0"
  },
 "devDependencies": {
    "@babel/core": "^7.9.0",
    "@rollup/plugin-commonjs": "^11.1.0",
    "@rollup/plugin-node-resolve": "^7.1.3",
    "@storybook/react": "^5.3.18",
    "@testing-library/jest-dom": "^5.5.0",
    "@testing-library/react": "^10.0.2",
    "@types/jest": "^24.0.24",
    "@types/react": "^16.9.12",
    "@types/react-dom": "^16.9.8",
    "babel-loader": "^8.1.0",
    "babel-preset-react-app": "^9.1.2",
    "identity-obj-proxy": "^3.0.0",
    "jest": "^24.9.0",
    "node-sass": "^4.14.1",
    "react": "^16.13.1",
    "react-dom": "^16.13.1",
    "rollup": "^1.27.4",
    "rollup-plugin-copy": "^3.3.0",
    "rollup-plugin-peer-deps-external": "^2.2.0",
    "rollup-plugin-postcss": "^3.1.2",
    "rollup-plugin-typescript2": "^0.27.0",
    "sass-loader": "^8.0.0",
    "ts-jest": "^24.2.0",
    "typescript": "^3.7.2"
  }
}

Publishing the Component Library

To publish our library, we first have to make sure that we have run Rollup (npm run build) and the transpiled/bundled library code exists under /build. We can utilise the NPM script prepublishOnly to make sure build is run before publish occurs. Add the following to package.json:

...
"scripts": {
    ...
    "prepublishOnly": "npm run build"
  },
...

package.json

Next, we have to choose an NPM registry to which we want to upload our library to. The easiest option is to use the public NPM registry. Other private (self hosted) alternatives are Verdaccio and ProGet. Create an account, then using those credentials (username and password) log into NPM by running: npm login.

Next, make sure the name of the package in package.json is something you desire and the version is 1.0.0 (to begin with):

{
  "name": "react-component-library",
  "version": "1.0.0",
  ...
}

The name in this article uses react-component-library which has already been taken - so choose something unique. For example, I've used harvey-component-library.

Now run:

npm publish

Any files that are under the directories outlined in files in package.json will be uploaded to the registry. For us, this will be all the files under /build which is the output from Rollup. You should see:

Anytime you want to update your library, you will have to increment the version in package.json following the Semantic Versioning guide. For example, if I had a "patch" update, I'd change the version from 1.0.0 to 1.0.1 and run npm publish.

Using the Component Library

Installing from NPM Registry

Let's say we've followed the above steps to publish our component library to NPM at: https://www.npmjs.com/package/harvey-component-library. We would install the component library as a dependency like we would any other NPM package:

npm install --save harvey-component-library

Installing Locally

We probably don't always want to have to publish and then install the component library to see new updates when using the library in other React projects. Fortunately, we don't have to publish the component library to an NPM registry before installing our component library and test out our components.

Let's say you had a React project on your local machine called harvey-test-app. In harvey-test-app run (making sure the path is correct to your component library):

npm i --save ../react-component-library

This will install your local instance of the component library as a dependency to harvey-test-app! This establishes a symlink from the component library to the dependency in the consuming project. Anytime an update is made to the library, it will immediately be reflected in the consuming project. Read more about NPM link here.

Using the Components

For either option, you would then go into your project consuming the component library (harvey-test-app) project and import our TestComponent like:

import React from "react";
import { TestComponent } from "react-component-library";

const App = () => (
    
        Hello I'm consuming the component library
        
    
);

export default App;

Running harvey-test-app should now successfully render our TestApp component!

CRA app consuming our TestComponent component

Check out this Code Sandpit snippet to see how easy it is to use the component!

Adding More Components

The way we've structured our project allows for any number of components to be added. You can copy the TestComponent folder, rename and then build out your new component.

Update 26/04/2020: I found the above steps to create a new component were cumbersome and time consuming. I decided to create a Node script that helps generation of new components:

Check out util/create-component.js in my GitHub repository if you want to implement something similar.

For both approaches, make sure you export your new component in index.ts otherwise it won't be picked up and bundled by Rollup.

Introducing Code Splitting (optional)

We can introduce code splitting that enables projects consuming our library to import one or a few components, instead of the whole library. This can help us achieve smaller bundle sizes in projects using our component library without tree shaking.

You'll only see benefit from this if you have a large component library with many components OR if the projects consuming the library do not use modern module bundlers that can perform tree shaking.

We will be utilising Rollup's Code Splitting functionality to help us achieve this.

Note 1: I opted to only target one module format (CJS). This reduces one level of nesting when importing components (import X from library/build/cjs/component/component -> import X from library/build/component/component). You can still target two formats if you want.

Note 2: It's also worthwhile to introduce an index.ts file to each of our components. The index.ts file can either contain our component, or import/export the component from another file. This reduces another level of nesting when importing components (import X from library/build/component/component -> import X from library/build/component). Here is an example.

Note 3: I've found that there is an issue with Rollup code splitting and bundling dependencies from node_modules. As such, I've removed @rollup/plugin-node-resolve when code splitting until I can find a better solution.

There are some changes we have to make to rollup.config.js and package.json. First, in rollup.config.js, update input, output and perserveModules to:

{
  input: ["src/index.ts", "src/TestComponent/TestComponent.tsx"],
  output: [
    {
      dir: "build",
      format: "cjs",
      sourcemap: true
    }
  ],
  preserveModules: true,
  ...
}

rollup.config.js

Instead of specifying one index file, we're telling Rollup to use a number of index files. This is how we instruct Rollup to perform code splitting. Add a path to all components you want to be split in the array.

We can use @rollup/plugin-multi-entry and provide appropriate file name patterns instead of having to add each of our components.

Setting preserveModules to true is required to maintain the directory/file structure of our components after being compiled. Without it, Rollup won't co-locate our compiled components and their associated type files correctly:

We're updating output.file to output.dir because Rollup bundles with code splitting, it produces output in multiple files/directories instead of one file.

In package.json remove the module entry as we're only shipping CJS:

{
  ...,
  "main": "build/index.js"
  "files": [
    "build"
  ],
  ...
}

package.json

That's it!

After building/publishing/installing our updated component library, we should be able to import components in two ways:

import { TestComponent } from 'react-component-library';

Option A

import TestComponent from 'react-component-library/build/TestComponent';

Option B

Option A will import TestComponent along all the other components available in the library. This will increase your overall bundle size (assuming there's no tree shaking).

Option B will only import TestComponent. This approach can significantly reduce the amount of code that is sent to the client.

See the component library with code splitting on this branch. Or checkout this commit to see what changes are required.

Further Steps

Setting up a Private NPM Registry, CI Pipeline and Consuming the Library

The next steps to get this component library was to setup a private NPM registry and pipeline which automatically publishes/maintains library versioning. I'll be covering this aspect in another blog post which you can read at: Setting up a Private NPM Registry and Publishing CI/CD Pipeline

Maintaining Code Quality in your Component Library

As this component library will likely be contributed by other engineers on your team, you'll probably want to maintain a high level of code quality by using static code analysis. Follow along with: Maintaining Code Formatting and Quality Automatically to help achieve this.

GitHub Repository

Again, I've created a GitHub repository with all the code necessary to create your own component library. Feel free to clone, fork or star it if it helps you!

HarveyD/react-component-library

A project skeleton to get your very own React Component Library up and running using Rollup, Typescript, SASS + Storybook - HarveyD/react-component-library

HarveyDGitHub

Setting up BuildKite and your first Continuous Integration pipeline in 2 hours

Harvey Delaney — Sat, 16 Nov 2019 03:40:30 GMT

I've written a few blogs in the past about how I setup my continuous integration pipeline for www.harveydelaney.com (https://blog.harveydelaney.com/jenkins-build-test-deploy-node-app/ and https://blog.harveydelaney.com/setting-up-jenkins-on-docker/). My Jenkins pipeline has been quite reliable and has done everything I needed.

At RateSetter we use Buildkite for our Continous Integration. I've found it to be an excellent tool that provides a simple, clean UI for all your pipelines in addition to having great documentation and an easy to use API. It also has excellent support for emojis!

When I joined RateSetter, all the BuildKite configuration had already been done. I had created new pipelines, but I wanted to learn how to set everything up from scratch. So I made it a task this weekend to migrate my CI pipeline from Jenkins to BuildKite and document it all here.

Please note, this article only covers how you can setup your pipeline from BuildKite's UI and not from writing BuildKite config files.

BuildKite Agent

For this blog, I'll be using an Azure Ubuntu VM.

After creating an account on BuildKite, it prompted me to run some commands on a VM. So I headed over to Azure to spin up a Standard B1s machine (1 virtual CPU and 1GB RAM, 0.5GB RAM will have troubles installing the agent):

After the VM was spun up, I SSHed into the machine and ran the all steps listed under setting up an Ubuntu agent on BuildKite:

After running all these commands, got a nice little popup and then could setup my first Pipeline!

Creating my first Pipeline

I wanted to create a pipeline for Discaper. Discaper uses NextJS for the front-end ( basically a React application that has server side rendering) and Node for the back-end. I want to have two pipelines, one for the front-end and one for the back-end. The pipelines will perform as follows:

Detect a push to the master branch of Discaper on GitHub
Clone from GitHub onto the BuildKite agent
Build the project: Run npm ci && npm run build && npm run export to create an out folder that contains all necessary files to deploy a NextJS app - read more about static NextJS exports here
Prepare the assets: Compress the contents of the out folder as a tar file then upload it as a BuildKite Artifact (so it can be downloaded and used in a proceeding BuildKite step)
Block the pipeline: have a step that requires me to press it before a deployment occurs. Read more about block steps here
Deploy the assets: On block step click, download tar from BuildKite artifacts and thenit to Discaper's VM (using SCP) where it'll be uncompressed and served!

Configuring Build Agent

First step is was to generate an SSH key. It was important to create this while logged in as the buildkite-agent user, on my agent I ran:

sudo su buildkite-agent
ssh-keygen

Then added it as a deploy key for Discaper (Discaper is a private GitHub repository).

Read more about GitHub Deploy keys here. If you're going to be adding multiple deploy keys, read more about how to do that in my other blog post: https://blog.harveydelaney.com/configuring-multiple-deploy-keys-on-github-for-your-vps/. You should also read BuildKite's documentation around adding SSH keys.

I also needed to setup NodeJS and NPM as we'd need this to build Discaper by following the steps here.

Creating the Build step

After adding a deploy key so that the Agent can clone the repository, I created a new Pipeline with Discaper's GitHub URL and added the script:

npm ci
npm run build
npm run export

Automated GitHub Deployments

BuildKite then prompted me with a list of steps required to setup GitHub webhooks:
"Follow the instructions below to integrate Buildkite with your GitHub repository to automatically create new builds when you push new code."

After doing so, I went back to my BuildKite pipeline, it was already to perform the first build!

Since I just wanted to test if the agent worked, I created a small commit in Discaper and pushed it to GitHub. BuildKite knew there was a push event and triggered a new build. BuildKite ran the build commands, NextJS did it's magic and I got my first successful BuildKite build:

Uploading Artifacts

Now that my agent could clone the repository from GitHub and generate a successful build, I needed to compress and package the contents of the out folder, then tell BuildKite to upload the tar file to it's artifacts and block the pipeline.

To upload artifacts, I went back to Discaper's Pipeline Settings and added the following command to my step:

buildkite-agent artifact upload discaper-assets.tar.gz

This tells BuildKite to upload discaper-assets.tar.gz at the very end of the step, read more about how BuildKite Artifacts work here:

To test this out, I simply triggered another build and could see that BuildKite uploaded all the files in out were compressed then uploaded to the pipeline's artifacts, which could be viewed in the Artifacts tab:

Block Step

Creating the block step from the BuildKite UI is trivial. I simply had to go into Pipeline Settings and add a new state, being Wait and block pipeline, I gave it the label: Deploy to Production along with a nice little rocket emoji:

Creating the Deploy step

My deploy step was relatively simple, although you may want to handle deploying assets in a different way. My strategy was to download discaper-assets.tar.gz, copy it over to my Discaper server, uncompress the tar file then serve the static files.

First, I created an additional step called Deploy to Production and told BuildKite to download discaper-assets.tar.gz, copy it over to my Discaper server and then execute a bash script on the server (that uncompressed the tar then re-runs the server with the new files copied 0ver):

The commands in the screenshot are:

buildkite-agent artifact download discaper-assets.tar.gz /tmp
scp /tmp/discaper-assets.tar.gz discaper@discaper-ip:~
ssh discaper@discaper-ip "./publish.sh"

Running this step in my Pipeline gives the output:

All Together

Hope this article helped you setup your first BuildKite pipeline. From here, you can (and should!) extend your pipeline to do more things like:

Run linting/tests
Deploy to to your test/staging environment
Cleaning up environments

Creating Breadcrumb Structured Data for your React/Typescript Website

Harvey Delaney — Mon, 19 Aug 2019 11:58:15 GMT

Recently, I've been working a my new website called Discaper. My vision for Discaper is that it will hopefully become the Zomato of Escape Rooms! Meaning that it'll be the place you go to to find the most suitable escape room for you.

Anyway, I've implemented a breadcrumb component for Discaper that helps users navigate throughout the website. For example, for the Assassin in the Club Escape Room, has the breadcrumbs of: Home -> Australia -> NSW -> Sydney -> Escape Hunt Sydney -> Assassin in the Pub. Clicking on any section of the breadcrumb will then show you all places/rooms in the next level contained within it. For example, clicking Australia will show you all states that have escape rooms in Australia. Or clicking Escape Hunt Sydney will show you all the escape rooms that are operated by Escape Hunt Sydney.

I've noticed on other big websites that results in Google Search give a neat little breadcrumb result. For example, if I searched Indu (a restaurant I went to last weekend) on Google, Zomato's search result looks like:

You can see Zomato has a nice little breadcrumb result: https://www.zomato.com › Australia › Sydney › City of Sydney › CBD. I wanted this for Discaper! Currently a result on Discaper looks like:

It turns out there's a relatively easy way to get these search results. You have to implement something called Structured Data and search engines consume this to give neat little results like Zomato.

Discaper is made using NextJS, React and TypeScript and I wanted to document how I got structured data working for Discaper using these technologies!

Structured Data Breadcrumb Component

For this example, let's say we want to display the breadcrumbs for our "Place", which looks like: Home -> Country -> State -> Region -> Place.

Let's also say we have our breadcrumb list with an interface of

export interface IBreadcrumb {
  description: string;
  url: string;
}

and we have our breadcrumb data for our place as:

  const breadcrumbList: IBreadcrumb[] = [
    {
      description: "Home",
      url: "/"
    },
    {
      description: "Australia",
      url: "/australia"
    },
    {
      description: "NSW",
      url: "/australia/nsw"
    },
    {
      description: "Sydney",
      url: "/australia/nsw/sydney"
    },
    {
      description: "Harvey's place",
      url: "/australia/nsw/sydney/harveys-place"
    }
  ];

The first thing I did was went to the Google documentation for how they wanted the BreadCrumb structured data to be... structured. This documentation gave me an overview, but I found the examples to be lacking.

So, I looked for a concrete example and found one here on Stack Overflow. This post basically gave the outline of what a valid structured data breadcrumb HTML should look like:


  
    
      Root page
    
    
  
  
    
      Category page
    
    
  
  
    
      This page

User FlameStorm also outlined that you can use Google's Structured Data Testing Tool to find out whether your structured data is valid or not. Try pasting the above HTML into it!

Using this example, I created a component that allowed me to render the breadcrumbs that are structured data compliant (with Google at least).

I created the component BreadCrumb that looks like:

The gotchas here are to remember that, in React, attributes have different names to what they normally would be (took me a while to figure out why React wasn't rendering things correctly!).

So for example:

would be:

in React.

Example usage

Let's use our component and see if our HTML is structured data compliant.

Rendering:

Would give us:


  
    Home
  
  
    Australia
  
  
    NSW
  
  
    Sydney
  
  
    Harvey's place

Putting this HTML into the Structured Data Testing Tool gives us a valid result!

If this helped, give https://gist.github.com/HarveyD/7964ac480fe7a6fbb0d85004d3959d39 a star :). Enjoy your breadcrumbs!

Also, put https://www.discaper.com/room/assassin-in-the-pub into the Google Structured Data Testing tool! Pretty cool right?

Update 25/08/2019: Assassin in the Pub now has the search result:

Running a Test with Multiple Test Cases using Jest/Enzyme (React)

Harvey Delaney — Sat, 10 Aug 2019 07:18:01 GMT

Last year I wrote an article which outlines some tips I had for running a test with multiple test cases using Jasmine: /running-multiple-test-cases-in-jasmine/

Both at work and on my personal side projects, I've started to use React a lot more than Angular. Since Create React App comes with Jest/Enzyme out of the box, I thought I would write another article outlining how I run multiple test cases in Jest/Enzyme for my React projects!

Simple Tests Cases

If we had the following simple class that we wanted to test:

export default class CalculatorService {
  average = numberList =>
    numberList.reduce((acc, val) => acc + val, 0) / numberList.length;
}

We might have created a test for our CalculatorService like:

import CalculatorService from "./calculator.service";

describe("CalculatorService", () => {
  let calculatorService;

  beforeEach(() => {
    calculatorService = new CalculatorService();
  });

  describe("average", () => {
    it("should average the number list correctly", () => {
      const numberList = [1, 4, 5, 10];
      const res = calculatorService.average(numberList);

      expect(res).toEqual(5);
    });
  });
});

But now we want to easy run multiple test cases without creating a new statement each time. The best way that I've found to structure my tests this way is by running a forEach loop over an it statement. It's important that we have the forEach outside of the it statement otherwise we won't be able to provide information to the it statement that will inform us which test case in particular has failed. This is how I would write test cases for CalculatorService:

  describe("average", () => {
    const testCases = [
      {
        numberList: [1, 2, 3],
        expected: 2
      },
      {
        numberList: [1, 4, 10],
        expected: 5
      }
    ];

    testCases.forEach(test => {
      it(`should correctly find the average of: ${test.numberList} which is: ${
        test.expected
      }`, () => {
        const res = calculatorService.average(test.numberList);

        expect(res).toEqual(test.expected);
      });
    });
  });

As you can see, having the it statement outside allows us to write descriptive test cases which will provide an output of:

This allows us to pin-point exactly which test case has failed and rectify it.

React Components

Applying this forEach loop methodology to testing React components using shallow rendering from the Enzyme library works very well! For this example, I'll be testing the following React component:

import React from "react";
import "./List.css";

const List = ({ items, itemClickEvent }) => (
  
    {items.map(item => (
       itemClickEvent(item.id)}
        className="item-container"
      >
        {item.heading}
        {item.description}
      
    ))}
  
);

export default List;

I was able to create 3 test cases using the forEach structure:

import React from "react";
import User from "./User";
import List from "./List";
import { shallow } from "enzyme";
import { configure } from "enzyme";
import Adapter from "enzyme-adapter-react-16";

configure({ adapter: new Adapter() });

describe("List", () => {
  const testItems = [
    {
      id: 45,
      heading: "Test Item",
      description: "This is a test item"
    },
    {
      id: 75,
      heading: "Test Item 2",
      description: "This is a test item"
    },
    {
      id: 90,
      heading: "Test Item 3",
      description: "This is a test item"
    },
    {
      id: 150,
      heading: "Test Item 4",
      description: "This is a test item"
    },
    {
      id: 270,
      heading: "Test Item 5",
      description: "This is a test item"
    }
  ];

  let testCases = [
    {
      itemAt: 0,
      expectedToCallWith: 45
    },
    {
      itemAt: 2,
      expectedToCallWith: 90
    },
    {
      itemAt: 4,
      expectedToCallWith: 265
    }
  ];

  testCases.forEach(test => {
    it(`should call correctly call itemClickEvent with ${
      test.expectedToCallWith
    } when item at: ${test.itemAt} is clicked`, () => {
      const mockItemClickEventHandler = jest.fn();
      const props = {
        items: testItems,
        itemClickEvent: mockItemClickEventHandler
      };

      const listComponent = shallow();
      listComponent
        .find(".list-container")
        .children()
        .at(test.itemAt)
        .simulate("click");

      expect(mockItemClickEventHandler).toHaveBeenCalledWith(
        test.expectedToCallWith
      );
    });
  });
});

A failure of a test would give a helpful error message of:

React Component Snapshots

Again, this can be applied to React Snapshots. If we had the component:

import React from "react";
import "./user.css";

const User = ({ imgUrl, firstName, lastName, age, address }) => {
  return (
    
      
      
      {`${firstName} ${lastName}`}
      {age}
      {address}
    
  );
};

export default User;

We can easily create multiple Snapshot test cases using react-test-renderer like so:

import React from "react";
import renderer from "react-test-renderer";
import User from "./User";

describe("User", () => {
  describe("Snapshots", () => {
    let testCases = [
      {
        imgUrl: "https://test.com",
        firstName: "Harvey",
        lastName: "Delaney",
        age: "69",
        address: "123 Fake Street"
      },
      {
        imgUrl: "https://test2.com",
        firstName: "John",
        lastName: "Doe",
        age: "35",
        address: "123 Real Street"
      }
    ];

    testCases.forEach(test => {
      it(`should have the correct snapshot for ${test.firstName} ${
        test.lastName
      }`, () => {
        const userComponent = renderer.create().toJSON();
        expect(userComponent).toMatchSnapshot();
      });
    });
  });
});

This will output two Snapshots:

// Jest Snapshot v1, https://goo.gl/fbAQLP

exports[`User Snapshots should have the correct snapshot for Harvey Delaney 1`] = `

  
  
  
    Harvey Delaney
  
  
    69
  
  
    123 Fake Street
  

`;

exports[`User Snapshots should have the correct snapshot for John Doe 1`] = `

  
  
  
    John Doe
  
  
    35
  
  
    123 Real Street
  

`;

I hope this article has helped give you a bit of insight into how duplicated tests can be written more concisely by opting to use a simple forEach loop!

Harvey's Blog

Setting Up Automated Database (PostgreSQL) Backups Using Node.js and Bash

Taking PostgreSQL database backup

Running the bash command using Node.js

Schedule the backups using cron job

Wrapping up

Creating your own Vue Component Library

Vue Component Library Code

Component Library Overview

Creating the Component Library

Project Setup

Vue

Vue Component

{{ headingText }}

{{ bodyText }}

TypeScript

Rollup

Input

Output

Plugins

Running Rollup

Final NPM Config

Publishing the Component Library

Using the Component Library

Locally

From NPM Registry

Using Components

Hello I'm consuming the component library

GitHub Repository

Choosing a Usenet Provider - a Beginner's Guide

Core

Nice to haves

Core

Download Limits

Download Speeds

Number of Connections

Connections (1)

Connections (15)

Connections (30)

Binary Retention Duration

Available Server Regions

Pricing and Plans

Nice to haves

Trial Periods

SSL (Security)

VPN (Virtual Private Network)

Newsreaders / Usenet Search / Browsers

Customer support

Conclusion

Setting up Plex Media Server on your Unraid Server (2020)

Create an account on Plex

Setting up an Unraid share

Installing Plex

Community Applications

Plex Docker Image

Setting up Plex

Next steps

Switching from Torrents to Usenets - The Why and How

Overview

Torrents

Usage Flow

Pros and Cons

VPN

Usenets

Usage flow

Pros and Cons

Services

Usenet Indexer

Usenet Providers

Recommended Providers

Newshosting

Recommended Indexers

NZBGeek (Harvey's Pick)

Usenet Download Client

Using the services

Obtaining Provider Credentials

Configuring Usenet Download Client

Using Usenet Indexer

Downloading Content

Automating your Usenet setup

Actions - `store.actions.ts`

Store - `store.tsx`

`CardInput.tsx`

`App.tsx`