Parsing a CSV file using Node/JavaScript

Parsing a CSV file using Node/JavaScript

javascript Dec 01, 2018

Last year, I was heavily interested into crypto currency (check out my Crypto Coaster website. This interest was driven by monetary reasons. This combined with the fact that I was a greedy, emotional trader resulted in a large number of panic driven coin buys and sells. This year, I decided to offload all my crypto currency in favour of less speculative investments like index traded funds.

I downloaded all the trades I did as a CSV file, which had thousands of rows. I simply could not have analysed it manually. So I decided to use the programming language I'm strongest at: JavaScript, to analyse my trade data.

I found it harder than expected to find out an up to date article on how to read and parse a CSV file using Node. So hopefully my article helps someone trying to achieve the same thing.

Prerequisites

For this tutorial, I'm using Node v8.11.4, I would recommend installing Node 8+ if you want to follow this tutorial.

Packages

We will be using the csv-parse npm package so go ahead an install it:

npm install csv-parse

CSV File

Of course, you'll need a CSV file you are processing, for example puposes I've generated a CSV file containg 1000 randomly generated users using Mockaroo. There are 7 columns: (id, first_name, last_name, age, email, gender and country), the first 5 rows are:

id,first_name,last_name,age,email,gender,country
1,Malchy,Teliga,55,mteliga0@un.org,Male,Mexico
2,Yasmeen,Ruddlesden,89,yruddlesden1@bigcartel.com,Female,China
3,Fonzie,Skeldon,22,fskeldon2@dedecms.com,Male,Portugal
4,Krystalle,Meineken,7,kmeineken3@apple.com,Female,Russia
5,Garnet,Jossum,19,gjossum4@berkeley.edu,Female,China

This CSV is saved as data.csv.

The Code!

Reading and transforming the data

Create a file called calculate.js (or whatever you want really):

var fs = require("fs");
var parse = require("csv-parse");

var csvFile = "data.csv";

class User {
  constructor(id, firstName, lastName, age, email, gender, country) {
    this.id = id;
    this.firstName = firstName;
    this.lastName = lastName;
    this.age = age;
    this.email = email;
    this.gender = gender;
    this.country = country;
  }
}

const processData = (err, data) => {
  if (err) {
    console.log(`An error was encountered: ${err}`);
    return;
  }

  data.shift(); // only required if csv has heading row

  const userList = data.map(row => new User(...row));

  analyseUsers(userList);
}

fs.createReadStream(csvFile)
  .pipe(parse({ delimiter: ',' }, processData));

What we're doing here is:

  1. Defining the shape of our Users
  2. Reading the CSV using fs (which comes with Node) and piping the output into csv-parse
  3. Shifting the array (removing the CSV headings: id,first_name,last_name,age,email,gender,country)
  4. Mapping each entry in the array entries into a User object

Analysing the data

Now, we are free to process our list of users in our analyseUsers function. For example, we can find the average age of our users:

const analyseUsers = userList => {
  const ageSum = userList.reduce((acc, val) => acc += val.age, 0);
  const averageAge = ageSum / userList.length;
  console.log(averageAge);
}

Or the most common country:

const analyseUsers = userList => {
  let countryCountDict = new Map();

  userList.forEach(user => {
    let countryEntry = countryCountDict.get(user.country);

    const newVal = countryEntry === undefined
      ? 0
      : countryEntry + 1;

    countryCountDict.set(user.country, newVal);
  });

  let maxEntry = ['', Number.MIN_SAFE_INTEGER];

  for(const entry of countryCountDict.entries()) {
    if (entry[1] > maxEntry[1]) {
      maxEntry = entry;
    }
  }

  console.log(maxEntry);
}

Run the file using node calculate.js.

Good luck with analysing your data using Node!

Tags

Harvey Delaney

Front End Engineer II at Amazon Web Services

Exclusive Usenet provider deals

Harvey's essential software engineering books

1

The Pragmatic Programmer: From Journeyman to Master

2

Clean Code: A Handbook of Agile Software Craftsmanship

3

Code Complete: A Practical Handbook of Software Construction

4

Design Patterns: Elements of Reusable Object-Oriented Software

Harvey is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to amazon.com
Great! You've successfully subscribed.
Great! Next, complete checkout for full access.
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.