Working with the Evernote API in Rails

Many people find Evernote to be an invaluable tool for organizing their life. The Evernote API provides has some really great features to take advantage of, including document download and OCR (optical character recognition). Whichever features you choose to investigate, be sure to read their terms of service so that you are within the guidelines of API usage.

So much of business is sorting and organizing data. Having API access to Evernote gives you superb leverage in harnessing the data you have. You can improve on the Evernote ecosystem by bringing your technology to the table.

We'll start from a place where Rails is already installed and Devise is configured for handling user sessions. Before we dive into the Rails bits, we need to create some credentials for accessing the Evernote API.

1. Go to dev.evernote.com and click on GET AN API KEY.

2. Fill out your details, and then choose either basic or full access. If you get any errors, check for invalid characters. You will be emailed a copy of the credentials as well as see them in the results page.

3. Save your credentials, as the site will never share these with you again. At this point you now have working credentials for the API Sandbox.

4. If you would like to activate the credentials for public use, go to dev.evernote.com and click the Resources dropdown menu. Click Activate an API Key on the menu. Once you fill out the pop-up form and click Submit, the credentials will work on both the sandbox and personal Evernote accounts.

Getting set up will take quite a few steps. But they're not that complex, so it shouldn't be too difficult to follow along.

First we need to add the dependencies to our Gemfile.

# Gemfile
# Get & store Evernote API credentials
gem 'omniauth'
gem 'omniauth-evernote'
# Evernote API
gem 'evernote_oauth'

These three gems will provide everything you need for handling Evernote's API. Go ahead and run bundle for the gems to be grabbed for your Rails site.

First we'll build a controller for handling the OmniAuth response (since the gems will handle the request). Execute the following command:

rails g controller OmniAuth

Next lets create an initializer for Evernote with OmniAuth.

# config/initializers/omniauth.rb
# Place the credentials in a secure place
# (do not place them in a file saved to your projects repository)
evernote = {
  consumer_key: ENV['EVERNOTE_KEY'],
  consumer_secret: ENV['EVERNOTE_SECRET'],
  sandbox: true
}
evernote_site = evernote[:sandbox] ? 'https://sandbox.evernote.com' : 'https://www.evernote.com'
Rails.application.config.middleware.use OmniAuth::Builder do
  provider :evernote,
           evernote[:consumer_key],
           evernote[:consumer_secret],
           client_options: {site: evernote_site}
end
OmniAuth.config.on_failure = OmniAuthController.action(:failure)

The reason we generated the controller before the initializer is because the last line of the initializer is pointing to a method we will soon define in the controller. If the controller wasn't created yet, you would not be able to run any part of the application.

Next we need to place the appropriate routes in our routes file.

# config/routes.rb
Rails.application.routes.draw do
  # omniauth
  get '/auth/:provider/callback' => 'omni_auth#callback'
  get '/auth/failure' => 'omni_auth#failure'
end

The OmniAuth gem sets auth request urls for outgoing authorizations and the return failure path.

By default, OmniAuth will configure the path /auth/:provider. It is created by OmniAuth automatically for you, and you will start the auth process by going to that path.

Also of note, if user authentication fails on the provider side, OmniAuth will by default catch the response and then redirect the request to the path /auth/failure, passing a corresponding error message in a parameter named message. You may want to add an action to catch these cases.

For the model, we'll go with a social network agnostic API token model. Many sites use the same OmniAuth/OAuth kinds of credentials, so it makes sense to organize it as one table in the database. Type this command to generate our model:

rails g model Social kind:integer:index username:string:index uid:text:index token:text token_expires:datetime verified:datetime user:references

Take note! Before you migrate it, update the migration file and set the username to have a blank string as a default.

# db/migrate/20170116234721_create_socials.rb
class CreateSocials < ActiveRecord::Migration[5.0]
  def change
    create_table :socials do |t|
      t.integer :kind
      t.string :username, default: ""
      t.text :uid
      t.text :token
      t.datetime :token_expires
      t.datetime :verified
      t.references :user, foreign_key: true
      t.timestamps
    end
    add_index :socials, :username
    add_index :socials, :kind
    add_index :socials, :uid
  end
end

And now you may migrate it.

rake db:migrate

Now to update the social model, we'll add an enumerator for the kinds of social network models available and throw in a validator for kind.

# app/models/social.rb
class Social < ApplicationRecord
  belongs_to :user
  enum kind: [:evernote]
  validates_presence_of :kind
end

And give the user model its relation to the Social model.

# app/models/user.rb
class User < ApplicationRecord
  has_many :socials, dependent: :destroy
end

Next we'll fill out the controller to handle a response from Evernote for our token request.

# app/controllers/omni_auth_controller.rb
class OmniAuthController < ApplicationController
  rescue_from OAuth::Unauthorized, with: Proc.new{redirect_to root_path}
  def callback
    case params['provider']
    when 'evernote'
      current_user.socials.evernote.where(username: request_auth["extra"]["raw_info"].username).
        first_or_create.update(
          uid: request_auth["uid"].to_s,
          token: request_auth["credentials"]["token"],
          token_expires: Time.at(request_auth['extra']['access_token'].params[:edam_expires].to_i).utc.to_datetime,
          verified: DateTime.now
      )
        flash[ :notice ] = "You've successfully Authorized with Evernote"
        session[:return_to] ||= root_path
    end
    redirect_to session.delete(:return_to) || root_path
  end
  def oauth_failure
    redirect_to root_path
  end
  private
  def request_auth
    request.env['omniauth.auth']
  end
end

When a user approves access for their account to your app, Evernote sends back a hash of different bits of information. This controller will save the relevant data for a user token to the database.

There's a lot going on here, so I'd like to cover how this handles the relation of Evernote record to user.

Calling socials on current_user safely ensures we're only working with what the current user owns. The evernote method on the socials collection is a helper method built by Rails' enum code and further scopes the collection down to only entries of the kind :evernote.

The where clause for username should be obvious and is why we had to set the default to a blank string in the model. Otherwise, it would crash here.

The first_or_create takes into account all of the previous scopes declared and autofills those in a new entry if one does not exist.

Now we are guaranteed a proper Evernote entry for our current user, and we can update the specific token values with the update method.

The session[:return_to] is optional. I use this as a way to have my site store what page the user was last on; there may be different kinds of pages they've left that they may want to return to.

I set the ||= to root_path for this example so it will work for you out of the box. But if you wanted to have a different page to reroute to per OmniAuth kind, you can put that in here instead. You're also free to change the failure page results to your liking.

For the view, you only need to put a link to /auth/evernote and the gems handle the rest.

<%= link_to 'Evernote Authorization', '/auth/evernote' %>

Now that you have enough to successfully connect and store Evernote tokens, you may use the API.

The API is a bit strange in design, in my opinion, and the naming scheme takes a little getting used to. What I would call folders are 'Notebooks', and entries in the folder are 'Notes'. Notes may have multiple media items attached to them.

I'm going to give you a grab bag of goodies in an example from a business card importer. Finding things in the API may take quite some time to get used to, so hopefully this will provide some insight.

This example assumes there is a Card model that uploads image data with Cloudinary.

module CardImporter
  def self.get_cards(notebook, user_id)
    user = User.find(user_id)
    token = user.socials.evernote.first.try(:token)
    client = EvernoteOAuth::Client.new( token: token )
    note_store = client.note_store.listNotebooks( token ).
                   detect { |i| i.name =~ /#{notebook}/i }.
                   note_store
    notes = pick_notes_with_images(token, note_store)
    notes.each do |note|
      note_resources(note.guid, token, note_store).
        select {|i| i.mime =~ /image\/(?:jpg|jpeg|png|gif)/}.
        each do |resource|
          card = create_card(
            resource.data.body,
            user,
            note.guid,
            resource.mime.split('/')[-1].gsub("jpeg", "jpg"),
            note.title
          )
          # enqueue OCR work here if card exists
        end
    end
  end
  private
  def self.grab_notes(user_token, note_store)
    page_size = 1000
    filter = Evernote::EDAM::NoteStore::NoteFilter.new(
      order:         Evernote::EDAM::NoteStore::NoteFilter::ORDER
      #              Evernote::EDAM::NoteStore::NoteFilter::ASCENDING
      #              Evernote::EDAM::NoteStore::NoteFilter::WORDS
      #              Evernote::EDAM::NoteStore::NoteFilter::NOTEBOOKGUID
      #              Evernote::EDAM::NoteStore::NoteFilter::TAGGUIDS
      #              Evernote::EDAM::NoteStore::NoteFilter::TIMEZONE
      #              Evernote::EDAM::NoteStore::NoteFilter::INACTIVE
      #              Evernote::EDAM::NoteStore::NoteFilter::EMPHASIZED
    )
    spec = Evernote::EDAM::NoteStore::NotesMetadataResultSpec.new(
      includeTitle: Evernote::EDAM::NoteStore::NotesMetadataResultSpec::INCLUDETITLE,
      #includeContentLength: Evernote::EDAM::NoteStore::NotesMetadataResultSpec::INCLUDECONTENTLENGTH,
      #includeCreated: Evernote::EDAM::NoteStore::NotesMetadataResultSpec::INCLUDECREATED,
      #includeUpdated: Evernote::EDAM::NoteStore::NotesMetadataResultSpec::INCLUDEUPDATED,
      #includeDeleted: Evernote::EDAM::NoteStore::NotesMetadataResultSpec::INCLUDEDELETED,
      #includeUpdateSequenceNum: Evernote::EDAM::NoteStore::NotesMetadataResultSpec::INCLUDEUPDATESEQUENCENUM,
      #includeNotebookGuid: Evernote::EDAM::NoteStore::NotesMetadataResultSpec::INCLUDENOTEBOOKGUID,
      #includeTagGuids: Evernote::EDAM::NoteStore::NotesMetadataResultSpec::INCLUDETAGGUIDS,
      #includeAttributes: Evernote::EDAM::NoteStore::NotesMetadataResultSpec::INCLUDEATTRIBUTES,
      includeLargestResourceMime: Evernote::EDAM::NoteStore::NotesMetadataResultSpec::INCLUDELARGESTRESOURCEMIME #,
      #includeLargestResourceSize: Evernote::EDAM::NoteStore::NotesMetadataResultSpec::INCLUDELARGESTRESOURCESIZE
    )
    note_store.findNotesMetadata(user_token, filter, 0, page_size, spec).notes
  end
  def self.pick_notes_with_images(user_token, note_store)
    grab_notes(user_token, note_store).select {|i|
      i.largestResourceMime =~ /image\/(?:jpg|jpeg|png|gif)/
    }
  end
  def self.note_resources(guid, user_token, note_store)
    note_store.getNote(user_token, guid, false, true, true, true).resources
  end
  def self.create_card(data, user, evernote_guid, format='',name='')
    format.prepend('.') unless format[0]['.']
    t = Tempfile.new(['card', format], Dir.tmpdir, 'wb')
    t.binmode; t.write data; t.rewind
    card = Card.create!(user_id: user.id, picture: t, name: name, evernote_guid: evernote_guid)
    t.delete
    card
  end
end

As you can see there are many intricacies to navigating around the API. The get_cards method here lets you pick a specifically named 'notebook' to go through. It filters out all the individual 'notes' to ones with images in it. It then goes through all the images and renders the raw data from the image to a file (which internally Cloudinary uploads to a host).

I've included all the optional parameters for note_store.findNotesMetadata in the grab_notes method as commented out lines. You are required to pass in at least some of these parameters.

If you do plan on using OCR data from Evernote, let me tell you first that the recognized text is unsorted. Everything is in word blocks with the pixel locations given for the corners around each word.

To use this, you will need to rate the distance between blocks and prioritize between horizontal and vertical. Once that's done, you will have to filter out what kind of data is present (address, email, etc.). I wrote a gem back in 2014 that rates proximity of boxes called prox_box should you be interested.

The learning curve may be a bit larger to get into Evernote's API, but the rewards are probably well worth it. With such a vast and powerful resource within your reach, why wouldn't you use it?

But yes, be forewarned it may cost you a great deal of time to implement features. Hopefully what I've provided here will save you most of that time. If you need to look up some specific information, search dev.evernote.com.

Again, I'd like to remind you to read their policy before making a product based around Evernote. They are a business offering an amazing service, and taking the time to follow their guidelines will make for good business for both you and them.

I wish you all the best in your development endeavors.