Build A CRM Chrome Extension To Manage Over 10k Google Contacts

Hello,

this post is about creating a Chrome Extension to enrich the functionalities of Google Contacts and to manage more than 10k entries in an easy way.

The story starts one month ago, when I was confronted with the following requirements:

  1. “Build a Chrome Extension that is capable of managing all our Google Contacts (12k+)”
  2. “We want a simple login”
  3. “We want inplace editing for some cells, as well as batch editing for a subset of all cells”
  4. “The extension should support fuzzy search, sorting, inverse search(exclude matching entries), multimail, quick filtering by age, notes/tags, …” – unfortunately, Google’s Contacts API is lacking a lot or nearly all this functionality right now

I will just go over the main challenges and outline my approach here. However, if you are facing similar issues and stumbled upon this post in search for a comprehensive solution, or if you are searching for a really powerful, lowcost and customizable Customer relationship management (CRM) tool, leave a comment and maybe I can help you by going further into detail.

General Discussion

Q: Why build another solution? There are existing CRM!
A: We wanted our own tailored solution. Nothing more or less. CRM are expensive and offer a lot of functionality that is not requested but paid. Moreover, when I started my work, the client had already used Highrise and wanted for some reason to get back to a simple Google contacts solution.

Q: Why should I create a Chrome Extension? JavaScript is slow and ugly!
A: To have everything in one place – the browser. No installation needed, painless distribution across multiple departments and maintenance in one place.

Q: But it is slow to manage more than 10k contacts in the browser!
A: The final solution was blazing fast: fuzzy search over 10k+ entries with more than 7 fields each, in under 3 sec!

Q: Only spammers have more than 10k contacts!
A: The client used a signup form for volunteers, which was connected to the Contacts API. No spam.

Q: You cannot have more than 10k contacts on Google! Source: http://www.feld.com/archives/2011/02/dear-google-i-have-more-than-10000-contacts.html
A: You can! And this article and the 100 comments are a great example, for why people would not use Google Contacts to manage their contacts. However after fighting all the troubles, when everything is set up, one will have a powerful, lowcost, leightweight CRM in the browser!

Let’s start!

Login

As stated in the Contacts API v3, Google wants apps to authorize via OAuth2.
In this blog post from 2011, Boris Smus introduces his library for OAuth authorization in Chrome Extensions. This library was then updated to OAuth2. There is a GitHub Repository as well (authored on 24 Dec 2013). However, Boris marked this repo as deprecated: “The Chrome Identity API makes this drop-in library obsolete.”

I tried all of the mentioned solutions and none of them was satisfying. The Identity API triggered a Chrome login and all the bookmarks, the history etc was synced. Imagine this happening in a team with five or more persons working on the contacts database. The effort, needed to get the other strategies to run was really high. The only solution that worked out of the box and was very stable, was this one, which is used in the official google sample applications as well:
http://src.chromium.org/viewvc/chrome/trunk/src/chrome/common/extensions/docs/examples/extensions/oauth_contacts/
https://developer.chrome.com/extensions/samples#search:contacts

The Google APIs Client Library in JavaScript seems to be a suitable solution, but it will be quite hard to get in running in your extension. There is a post on Stackoverflow for those who are interested.

Fetching 10k Contacts At Once

…is not possible with the current API. Google would just make a cut after a certain amount of data. Feel free to correct me if I’m wrong.

My first apporach followed the Documentation, but failed, i.e. was not returning all contacts.

Then, after a while, I found this little note, saying:

Note: The feed may not contain all of the user’s contact groups, because there’s a default limit on the number of results returned.
For more information, see the max-results query parameter in Retrieving contact groups using query parameters.

This was not really informative but at least I knew it was not my fault.

The second attempt was to load the data in pages or chunks. One can achieve this by passing the API the query parameters ‘max-results’ and ‘start-index’:

start-ndex: The 1-based index of the first result to be retrieved (for paging).

And on the Youtube API one can read this, more dteailed explanation:

The start-index parameter specifies the index of the first matching result that should be included in the result set. This parameter uses a one-based index, meaning the first result is 1, the second result is 2 and so forth. This parameter works in conjunction with the max-results parameter to determine which results to return. For example, to request the second set of 10 results – i.e. results 11-20 – set the start-index parameter to 11 and the max-results parameter to 10.

And from the Analytics Devguide page:

start-index=10
Optional.
If not supplied, the starting index is 1. (Result indexes are 1-based. That is, the first row is row 1, not row 0.) Use this parameter as a pagination mechanism along with the max-results parameter for situations when totalResults exceeds 10,000 and you want to retrieve rows indexed at 10,001 and beyond.

I did this in the following way:

However, this was returning the same pages, regardless of the specified start-index.

I tried also to specify a very large number for max-results, but nothing was working for me.

max-size:
The maximum number of entries to return. If you want to receive all of the contacts, rather than only the default maximum, you can specify a very large number for max-results.

Finally, I got the fetching working with following trick:

  • First, I split my large group(>10k) into two parts.
  • Second, I queried the entries groupwise

Contact__Split

One comfortable way of doing this by using a Promise library like Q.
However, one thing that took me a couple of hours was to figure out how to query a group. The API is really simple on this:

To retrieve a single group, send an authorized HTTP GET request to the contact group’s edit URL.
https://www.google.com/m8/feeds/{userEmail}/full/{groupID}

In all following examples the groupID is just displayed as ‘groupID’:

After trying all conceivable possibilities I figured out that the mysterious groupID is a full URL. This means one would have to request in the following way:

When all my results returned I felt happy and depressed at the same time.

Updating Multiple Hundreds Of Contacts

Data Table API and Inplace Editing

I think the following section can be interesting for a lot of people regardless of this whole Chrome extension, CRM stuff, since I found many many questions on blogs and forums on how to create an efficient and free inplace editing for the DataTable plugin. I came up with the following solution and after a little bit of performance twaeking I can say that it works very well for my purpose.

DataTables Inverse Search

Leave a Reply

Your email address will not be published.

Time limit is exhausted. Please reload the CAPTCHA.