Image showing Nodz linking the right nodes together.

Overfeeding, taking Drupal's Feeds module to the edge - Entity Reference Import

19.06.14

Getting data into Drupal: While the first part dealt with the import of multilingual data, we’re now looking at importing relational data.

Importing in Drupal: relational data

In part 1 of the overfeeding article I explained how to import multilingual data and expanded on how to link those nodes together on import. In this second part, the focus will be on something similar: relational content. From an architecture perspective, this means different objects or tables with references to each other; in Drupal, it means multiple content types with entity references between them.

Importing relational data is a bit more complicated than the multilingual import, but we’re going even further: After having explained the basics of importing relational data with entity references, I’ll show how to combine this with the multilingual import so that we can have a multilingual relational import.

What we need

We’ll build on top of the same module stack from the first part:

And we’ll add a few to meet our new needs:

Entity reference is the base here, without it we just don’t really have relational data. To work with Feeds we need Feeds Tamper: String to ID, which allows us to perform a node look up based on import data and set an entity reference. Views is used for the advanced setup of Feeds Tamper: String to ID so that we can customise the lookup query which matches one of our import fields to a node in Drupal.

For an even more advanced setup with i18n and entity references it helps to have Internationalization contributions, which allows the synchronisation of entity references. While it’s not strictly necessary for the import, it helps keeping order afterwards when editing data.

First things first: Importing references

The hard part is to define the mapping between the entities. I was lucky to be importing from another database and all my entities had unique ids. To avoid a rather abstract description, I’ll use a more concrete example of what I was importing:

  • Travel Offers
  • Travels

Travel Offers represent a generic item with the shared data; Travels represent a specific instance of the Travel Offers with certain information such as a date unique to that item. So both had a set of unique IDs, and the Travels had a reference field containing the IDs of the travel offers they relate to. Thus, as far as the implementation in Drupal goes we have two content types, one of which has a reference to the other.

Image showing how the two content types are linked.
The Travels have a reference to the Travel offers.

Note that if you don’t have unique IDs for each of the Travel Offers and Travels you could always generate them through a preprocessing step. However, in the end you still need a reference from one entity type to the other, even if it’s the title string, or you’ll have no other option but to manually link the data.

It’s necessary that you keep this reference ID field when importing the parent entity and the child entity, even if Drupal will use the nid instead. Otherwise, we’ll have no chance to look it up later.

I’m breaking down the import process into three steps (I am assuming that the Travel Offers have already been imported as shown in part 1):

  1. Setup the Node processor Mapping
  2. Setup the Lookup view
  3. Setup the Feeds Tamper

1. Setup the Feeds mappings

Mappings can be done as shown in part 1 of the article, except for mappings of entity references. This is where Feeds Tamper: String to ID comes in. We set up a mapping from the imported reference field to the entity reference field of the node on the Feed Node Processor mappings page. In my case, I had:

  • a Travel Offer number
  • which I wanted to link to a parent Travel Offer entity ID (Entity reference by Entity ID)

We also have to set up the tamper Convert string into entity ID on this mapping, since as long as the imported field isn’t a node ID, the import wouldn’t work. Depending on what reference we are importing, there are two options:

  • It is the actual title of the node we want to reference. Entityreference autocomplete does the job, skip to step 3.
  • It is not the title field. In this case, we need to create a view which looks up a node ID based on the reference field.

2. Setup the mapping view

If we add the tamper page it give us some hints:

  • Takes a string as a ‘contextual filter’ argument
  • Displays ‘fields’
  • Returns a row containing just an ID as the first column (not linked)

This is why we have to set up the view first. Mine is called Feeds Travel Linking, I selected neither create page nor create block, and preselected the content type to be my travel offers type. This will create a view with a Master display. As for the settings:

  • Format
    • Format: Unformatted list
    • Show: Fields
  • Fields:
    • Content: NID (not linked, no label)
  • Filter Criteria:
    • Content: Type (= Travel Offer) - Note that this was automatically set when the view was created.
  • Advanced
    • CONTEXTUAL FILTERS
    • Content: SN_V (This was the name of my reference field in travel offers)
      • When the filter value is not available: Hide View

If we know one of the references, we can now use view’s auto preview to test it: Throw the number in and punch it. It should only return a node ID here. The two key elements are that we only have one contextual filter and only return one unformated unlinked node ID.

3. Setup the tamper

Now that we have the view needed to fill it out, we can create the tamper. Select Convert string into entity ID, from there we have 3 settings:

  • Lookup method ⇒ Views
  • Choose the view ⇒ Master (of the Feeds Travel Linking View)
  • Choose ID field ⇒ NID

When we run the import, tamper will now pass the reference to our view, which in turn will return a node ID. The tamper will then set the Entity Reference node ID with this value – and we have a working reference!

Multilingual and relational

Back to the big picture: I have Travel Offers, each of which has multiple Travel instances. In part one of the article I showed how to do multilingual import with the Travel Offers. I just showed how to import the Travels and link them to Travel Offers. However, there’s one piece missing still, because the Travels are multilingual as well.

We can reuse the tricks explained in part one but it’s not going to be enough. I again implemented the hook_feeds_after_save(). to set the translation ID, so that the nodes are linked to their translations (or set as the primary node). But then the references also need to be localised:

  • The English Travels should point to the English Travel Offers
  • The German Travels should point to the German Travel Offers

Because the title can be different in the two languages, we can’t rely on autocomplete but need to extend the setup. Since we can only have one contextual filter with Feeds Tamper: String to ID views, we need two views, one which looks up the English node IDs and one which looks up the German node IDs. Given that we can only have one Feeds Tamper string to ID view per feed, this means that we also need two feeds. So to extend the diagram above, we’re now going to double up on everything and add more linking.

Image showing how the content types are linked once translation is taken into account.
Taking into account that we have two languages, we have to double up on everything.

We can reuse the view defined above and clone it so that we have the following setup:

  • Feeds Travel Linking EN
    • Different settings:
    • Filter Criteria:
      • Content: Language (= EN)
  • Feeds Travel Linking DE
    • Different settings:
    • Filter Criteria:
      • Content: Language (= DE)

Image showing a screenshot of the view settings.
The view settings for the “Convert string into entity ID” look up.

Then our feeds import for the english language will use the Feeds Travel Linking EN for its Convert string into entity ID tamper and the feed import for the german language will use the Feeds Travel Linking DE for its Convert string into entity ID.

Image showing a screenshot of the tamper "Convert string into entity ID."
Selecting the view in the “Convert string into entity ID” tamper.

Conclusion

In this article I showed how Feeds Tamper: String to ID can be used to perform a two step import to get linked nodes into Drupal from an external source. This requires first importing the parent entities, and then, assuming we have a reference key value available, setting up a tamper to look up the linked node on import and set the entity reference correctly.

Taking it a step further, even if the top level entity is multilingual we can still import a linked entity. This requires a more complex setup, as we need to ensure that we’re linking the imported entity to its translation, but also set up an entity reference to the correct language.

Still confused? Feel free to leave a comment!