CSV to Firestore

If you want to migrate your existing database to Firestore, you might be wondering… How do I import JSON or CSV to Firestore?. In this lesson, I will provide you with a reliable pipeline for reading raw data and parsing it to a writable format for Firestore.

Everybody has different data modeling needs, so I will teach you how to quickly roll out your own NodeJS CSV-to-Firestore command line import tool. Its purpose is to take your local files (CSV or JSON format in this case), parse it to an array of JS objects, then use the Firebase admin SDK to perform an atomic batched write. Not only will this handle migrations, but it will also give you a solid development utility that you can extend with additional functionality.

CSV format was chosen for this demo because it is commonly used as an export format for relational SQL databases and MS Excel spreadsheets. If you want the opposite, watch episode 69 to learn how to [export Firestore to CSV](https://angularfirebase.com/lessons/csv-exports-from-firestore-database-with-cloud-functions/).

Step 1 - Initial Setup

Let’s go ahead and create an empty folder and initialized NPM.

mkdir firestore-importer
cd firestore-importer
npm init

Setting up a NodeJS TypeScript Project

You can write this utility in vanilla JS, but TypeScript will dramatically improve tooling and productivity long-term.

npm install -D typescript
npm install -D @types/node 

touch tsconfig.json

mkdir src
touch src/index.ts

Now update your tsconfig.json with the following content. Basically, we’re just telling TS to compile our code in the /dist folder as commonjs that can be understood by NodeJS.

{
    "compilerOptions": {
        "outDir": "./dist/",
        "noImplicitAny": false,
        "module": "commonjs",
        "target": "es5",
        "allowJs": true,
        "sourceMap": true,
        "moduleResolution": "node",
        "lib": [
            "es2015"
        ],
        "types": [
            "node"
        ]
    },
    "include": [
        "src/**/*"
    ]
}

At this point, you can run tsc from the command line to compile your code.

Install the Firebase Admin SDK

npm install firebase-admin --save

You will need to download your service account from the Firebase admin console. Save it in the root of this project and name it credentials.json. This will give your local app full access to the project and bypass all security rules.

Make sure to keep your credentials private. If using git, add the line `credentials.json` to your *.gitignore* file.

Install Commander

Commander.js is a tool that makes it easy to parse command line arguments in Node. We will use it to pass a local file path and Firestore collection path argument to the migration command.

npm install commander --save

Install FS Extra and CSVtoJSON

Lastly, let’s install FS Extra to interact with the local file system. When it comes to CSV, there are a bunch of different Node packages, but CSVtoJSON works especially well for this task because it has a callback that emits each row from the spreadsheet as JSON.

npm i --save csvtojson fs-extra
npm i -D @types/{csvtojson,fs-extra}

The final initialization logic should look something like this.

#!/usr/bin/env node

import * as admin from 'firebase-admin';
import * as fs from 'fs-extra';
import * as args from 'commander';
import * as csv from 'csvtojson';

var serviceAccount = require("../credentials.json");

admin.initializeApp({
  credential: admin.credential.cert(serviceAccount)
});

const db = admin.firestore();

Step 2 - Parsing Raw Data

Our next step is to read a raw file, then convert it to a JavaScript object that can be used as the document data in Firestore.

Reading a JSON File

Reading a raw JSON file is an easy one-liner thanks to fs-extra.

fs.readJSON('hello.json');

CSV to JSON

Reading a CSV is a bit more work. CSVtoJSON uses callbacks, but we will Promisify it in the next section. It gives us a handful of listeners that emit data when a row or document is finished processing.

csv()
  .fromFile(path)
  .on('json', (row) => {
    // emits each row
  })
  .on('end_parsed', (data) => {
    // emits all rows
  })
  .on('error', err => {
    // handle errors
  })
})

Step 3 - Building your own CLI Tool in Node

While it’s possible to process command line arguments in Node without any dependencies, I highly recommend the Commander.js package to make life easier for your team.

Here’s how we want our CLI command to work:

fire-migrate --src bunnies.csv --collection animals

It should read the CSV source file, then write each row as a document in Firestore. Accessing arguments from the command is as simple as defining them as an options.

args
.version('0.0.1')
.option('-s, --src <path>', 'Source file path')
.option('-c, --collection <path>', 'Collection path in database')
.option('-i, --id [id]', 'Optional field to use for document ID')
.parse(process.argv);


// Now use the args in your script

const file = args.src;
const colPath = args.collection;

As an added bonus, we get instant documentation for the CLI tool.

fire-migrate --help

Full Firebase CLI Code

Now it’s time to put everything together into a CLI tool that we can actually use.

#!/usr/bin/env node

import * as admin from "firebase-admin";
import * as csv from "csvtojson";
import * as fs from "fs-extra";
import * as args from "commander";

args
  .version("0.0.1")
  .option("-s, --src <path>", "Source file path")
  .option("-c, --collection <path>", "Collection path in database")
  .option("-i, --id [id]", "Field to use for document ID")
  .parse(process.argv);

// Firebase App Initialization
var serviceAccount = require("../credentials.json");

admin.initializeApp({
  credential: admin.credential.cert(serviceAccount)
});

const db = admin.firestore();

// Main migration function

async function migrate() {
  try {
    const colPath = args.collection;
    const file = args.src;

    // Create a batch to run an atomic write
    const colRef = db.collection(colPath);
    const batch = db.batch();

    let data;
    if (file.includes(".json")) {
      data = await fs.readJSON(file);
    }

    if (file.includes(".csv")) {
      data = await readCSV(file);
    }

    for (const item of data) {
      const id = args.id ? item[args.id].toString() : colRef.doc().id;

      const docRef = colRef.doc(id);

      batch.set(docRef, item);
    }

    // Commit the batch
    await batch.commit();

    console.log("Firestore updated. Migration was a success!");
  } catch (error) {
    console.log("Migration failed!", error);
  }
}

function readCSV(path): Promise<any> {
  return new Promise((resolve, reject) => {
    let lineCount = 0;

    csv()
      .fromFile(path)
      .on("json", data => {
        // fired on every row read
        lineCount++;
      })
      .on("end_parsed", data => {
        console.info(`CSV read complete. ${lineCount} rows parsed.`);
        resolve(data);
      })
      .on("error", err => reject(err));
  });
}

// Run
migrate();

Compile the Code and Link the Command

To connect our Node executable file to the local command line PATH, we need to register it in the bin object in package.json.

  "bin": {
    "fire-migrate": "dist/index.js"
  }

You can compile the source code and link the command by running:

tsc && npm link

The End

As you can see, creating your own CLI tool for Firebase development is pretty simple. You can expand on this code to build additional utilities that increase your development productivity.