Batch insert in MongoDB using node.js and Monk

As a test, I have decided to use node.js and Monk to mimic a bulk import in MongoDB, from a source file containing a json array. I say « mimic », as Monk does not implement MongoDB bulk insert per se.

My first try was to insert the whole ‘data’ array in a ‘film’ collection, parsed from the json file (the initialization code is shown further down).

It really inserts each array item in Mongodb as an individual document, with its own id, which is fine.

But the issue was that on execution, the script was stuck somewhere, and the prompt never showed backed  until I hit ctrl-C. Browsing Mongodb with the ‘mongo’ shell showed that the docs were nevertheless stored in the database.

Some would just add a …

… *after* the insert directive, but it would be missing the point: the insert function call is asynchronous, and the insertion could still be running when the close directive is sent, ending with an exception of type « Connection Closed By Application ».

This happens if you don’t import a JSON array as a block, but instead have to insert a larget set of individual items (e.g. from a set of individual files), which takes some time.

The good practice (*) is therefore to close the db from the insert callback, after the last item has been processed. In the following code, I use insert > count > close for the demonstration.

Note that the variable i cannot be used as a test item for asserting the end of the loop, as i may be equal to data.length *before* the first insertion callback is called.

(*) Disclaimer: I’m still a newcomer to node.js, therefore there may be better ways to achieve what I want; welcome if you can propose something more effective …

 

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">

WordPress Anti Spam par WP-SpamShield