Tallan's Technology Blog

Tallan's Top Technologists Share Their Thoughts on Today's Technology Challenges

Creating a SharePoint BCS .NET Connectivity Assembly to Crawl RSS Data in Visual Studio 2010

Overview

In this post, I’ll walk you though how to create a SharePoint 2010  BCS .NET Connectivity Assembly in Visual Studio 2010.

We’re going to create an assembly that will allow us to Crawl RSS Feeds using BCS.

Creating a BCS .NET Connectivity Assembly project in Visual Studio 2010

First, create a new project selecting the Sharepoint –> 2010 –>Business Data Connectivity Model.  I’ll call this project RSSModel.

image

At the SharePoint Customization Wizard screen, select Deploy as a Farm Solution.  Be sure to put in the address of your base web application.

image

Click FinishVisual Studio will create the new project.  Once complete you should be presented with the model designer surface as below:

image

Updating the Default Project for Our Custom BCS Model

Setting the Model Name

The model is created using default model and entity names.  Lets change those first.  In solution explorer, change the name of the model (BdcModel1) to something that describes our project better.  Lets change it to RSSItemModel.

image

Also, change BdcModel1.bdcm to RSSItemModel.bdcm.

image

Creating Entity Classes

Next, we’ll want to change the default Entity1.cs and Entity1Service.cs.   Since we will be returning RSS Item entities here, lets change the Entity1.cs file to RSSItem.cs, and the Entity1Service.cs  file to RSSItemService.cs.

image

(Generally, when you change a class file name, Visual Studio will automatically rename the class.  I’ve noticed in my environment this has not happened 100% of the time.  Also, changing the BDCModel folder name does not automatically change the namespaces on the class files below it.  Open the RSSItem and RSSItemSErvice files and update the class names and namespace names as seen below)

image

Next, lets update our Entity class.  We’re interested in indexing RSS Item entities here.  I’m going to use the Google News feed available at http://news.google.com/news?pz=1&cf=all&ned=us&hl=en&output=rss

Below is a snippet of XML from that feed:

   1: <item>

   2: <title>

   3: Romney to go on the attack in Pennsylvania, but Democrats beat him to it - ABC News

   4: </title>

   5: <link>

   6: http://news.google.com/news/url?sa=t&fd=R&usg=AFQjCNFG5igf5H4QkI0TF1qlu2MZ3CkE7w&url=http://abcnews.go.com/Politics/OTUS/romney-attack-pennsylvania-democrats-beat/story?id%3D16793372

   7: </link>

   8: <guid isPermaLink="false">tag:news.google.com,2005:cluster=35185807926259</guid>

   9: <category>Top Stories</category>

  10: <pubDate>Tue, 17 Jul 2012 13:05:54 GMT</pubDate>

  11: <description>

  12: <table border="0" cellpadding="2" cellspacing="7" style="vertical-align:top;"><tr><td width="80" align="center" valign="top"><font style="font-size:85%;font-family:arial,sans-serif"><a href="http://news.google.com/news/url?sa=t&amp;fd=R&amp;usg=AFQjCNFezv7cH-723chb5gH-K-tEJcNx9g&amp;url=http://abcnews.go.com/blogs/politics/2012/07/for-mitt-romney-the-tax-questions-cometh-the-note/"><img src="//nt2.ggpht.com/news/tbn/gnUN3X5no-RSqM/6.jpg" alt="" border="1" width="80" height="80" /><br /><font size="-2">ABC News</font></a></font></td><td valign="top" class="j"><font style="font-size:85%;font-family:arial,sans-serif"><br /><div style="padding-top:0.8em;"><img alt="" height="1" width="1" /></div><div class="lh"><a href="http://news.google.com/news/url?sa=t&amp;fd=R&amp;usg=AFQjCNFG5igf5H4QkI0TF1qlu2MZ3CkE7w&amp;url=http://abcnews.go.com/Politics/OTUS/romney-attack-pennsylvania-democrats-beat/story?id%3D16793372"><b>Romney to go on the attack in Pennsylvania, but Democrats beat him to it</b></a><br /><font size="-1"><b><font color="#6f6f6f">ABC News</font></b></font><br /><font size="-1">PITTSBURGH--Mitt Romney returns to Pennsylvania today, where he&#39;s expected to debut a more aggressive message against President Barack Obama&#39;s policies and his ties to political donors. The move comes after days of blistering attacks from Democrats on <b>...</b></font><br /><font size="-1"><a href="http://news.google.com/news/url?sa=t&amp;fd=R&amp;usg=AFQjCNEJ56sur53oL3Tla-aUg7eR_xw9Jg&amp;url=http://www.foxnews.com/politics/2012/07/17/romney-goes-on-offense-in-pennsylvania-obama-collects-cash-in-texas/">Romney goes on offense in Pennsylvania, Obama collects cash in Texas</a><font size="-1" color="#6f6f6f"><nobr>Fox News</nobr></font></font><br /><font size="-1"><a href="http://news.google.com/news/url?sa=t&amp;fd=R&amp;usg=AFQjCNEzsSksuLwvtXu0QLDdRF0N8mzuQQ&amp;url=http://www.cbsnews.com/8301-505267_162-57473573/obama-romney-keep-sniping-over-outsourcing/">Obama, Romney keep sniping over outsourcing</a><font size="-1" color="#6f6f6f"><nobr>CBS News</nobr></font></font><br /><font size="-1"><a href="http://news.google.com/news/url?sa=t&amp;fd=R&amp;usg=AFQjCNFQl5WbHEerR5Wb-QIcLmJOle_Ivg&amp;url=http://www.politico.com/news/stories/0712/78589.html">Cutter won&#39;t drop the &#39;f-bomb&#39;</a><font size="-1" color="#6f6f6f"><nobr>Politico</nobr></font></font><br /><font size="-1" class="p"><a href="http://news.google.com/news/url?sa=t&amp;fd=R&amp;usg=AFQjCNGe7Sfp2rArUyKWtkM1iAYOigcXbA&amp;url=http://content.usatoday.com/communities/theoval/post/2012/07/new-ads-allege-romney-is-hiding-something-in-tax-returns/1"><nobr>USA TODAY</nobr></a>&nbsp;-<a href="http://news.google.com/news/url?sa=t&amp;fd=R&amp;usg=AFQjCNGfQt1g6m38uNRHeuHBYin5UYyAaA&amp;url=http://www.oregonlive.com/news/oregonian/david_sarasohn/index.ssf/2012/07/refusing_to_release_tax_return.html"><nobr>OregonLive.com</nobr></a>&nbsp;-<a href="http://news.google.com/news/url?sa=t&amp;fd=R&amp;usg=AFQjCNE4NNtbG7mJBx3Q5ZshuBrfZNuEBQ&amp;url=http://stream.wsj.com/story/campaign-2012-continuous-coverage/SS-2-9156/SS-2-33747/"><nobr>Wall Street Journal</nobr></a></font><br /><font class="p" size="-1"><a class="p" href="http://news.google.com/news/more?pz=1&amp;ned=us&amp;ncl=d4mg_JiQN8FwJnMFVeUDI-eeUF3BM&amp;topic=h"><nobr><b>all 12,234 news articles&nbsp;&raquo;</b></nobr></a></font></div></font></td></tr></table>

  13: </description>

  14: </item>

Using the fields found in the RSS Snippet above, we’ll construct the RSSItem class as below

   1: namespace RSSModel.RSSItemModel

   2: {

   3:     /// <summary>

   4:     /// This class contains the properties for Entity1. The properties keep the data for Entity1.

   5:     /// If you want to rename the class, don't forget to rename the entity in the model xml as well.

   6:     /// </summary>

   7:     public partial class RSSItem

   8:     {

   9:         public string Guid { get; set; }

  10:         public string Title { get; set; }

  11:         public string Link { get; set; }

  12:         public string Category { get; set; }

  13:         public DateTime PubDate { get; set; }

  14:         public string Description { get; set; }

  15:     }

  16: }

Next, lets update the RSSItemService class.  The sample class created for us had 2 methods, ReadItem() and ReadList().  These are the two methods needed in an External Content Type in order to consume information in SharePoint.  Since we’re not interested in Insert, Update, or Delete operations, we’ll just modify the two methods here to return information for RSS Items in the feed.

Note: Shortcuts were taken here in the interest readability and post size.  Ensure all production code uses proper exception handling and null checking.

   1: public class RSSItemService

   2:    {

   3:        private static string _feedUrl = "http://news.google.com/news?pz=1&cf=all&ned=us&hl=en&output=rss";

   4:        /// <summary>

   5:        /// Note: This is for a quick sample.  Ensure production code has proper exception handling and checks for valid return streams

   6:        /// </summary>

   7:        /// <returns></returns>

   8:        private static XDocument GetFeed()

   9:        {

  10:            return XDocument.Load(new StreamReader(HttpWebRequest.Create(_feedUrl).GetResponse().GetResponseStream()));

  11:        }

  12:        /// <summary>

  13:        /// This is a sample specific finder method for Entity1.

  14:        /// If you want to delete or rename the method think about changing the xml in the BDC model file as well.

  15:        /// </summary>

  16:        /// <param name="id"></param>

  17:        /// <returns>Entity1</returns>

  18:        public static RSSItem ReadItem(string id)

  19:        {

  20:            var foundItem =

  21:                GetFeed().Descendants("channel").Descendants("item").Where(item => item.Element("guid").Value == id).FirstOrDefault();

  22:            if (foundItem == null) return null;

  23:                return new RSSItem()

  24:                {

  25:                    Guid = foundItem.Element("guid").Value,

  26:                    Title = foundItem.Element("title").Value,

  27:                    Link = foundItem.Element("link").Value,

  28:                    Description = foundItem.Element("description").Value,

  29:                    PubDate = DateTime.Parse(foundItem.Element("pubDate").Value)

  30:                };

  31:        }

  32:        /// <summary>

  33:        /// This is a sample finder method for Entity1.

  34:        /// If you want to delete or rename the method think about changing the xml in the BDC model file as well.

  35:        /// </summary>

  36:        /// <returns>IEnumerable of Entities</returns>

  37:        public static IEnumerable<RSSItem> ReadList()

  38:        {

  39:            return GetFeed().Descendants("channel").Descendants("item").Select(item => new RSSItem()

  40:                                                                                           {

  41:                                                                                               Guid =

  42:                                                                                                   item.Element(

  43:                                                                                                       "guid").Value,

  44:                                                                                               Title =

  45:                                                                                                   item.Element(

  46:                                                                                                       "title").Value,

  47:                                                                                               Link =

  48:                                                                                                   item.Element(

  49:                                                                                                       "link").Value,

  50:                                                                                               Description =

  51:                                                                                                   item.Element(

  52:                                                                                                       "description").

  53:                                                                                                   Value,

  54:                                                                                               PubDate =

  55:                                                                                                   DateTime.Parse(

  56:                                                                                                       item.Element

  57:                                                                                                           ("pubDate").

  58:                                                                                                           Value)

  59:                                                                                           });

  60:        }

  61:    }

 

Updating the Model in the Designer

Next we’ll update the RSSItemModel in the designer.  First, open the designer surface by double-clicking RssItemModel.bdcm in solution explorer.

Click somewhere in the background of the designer surface.  If the Name property in the  Properties pane still says BdcModel1, be sure it gets changed to RSSItemModel:

image

Next, select the “Entity1” entity on the designer surface.  In the Properties window, change the Name property to RSSItem and the Namespace property to RSSModel.RSSItem.

image

Next, click Identifier1 under Indentifiers in the RSSItem entity on the model surface.  Change the name to Guid, which is the identifier for our External Content Type.  Ensure the Type Name property reflects the .NET type of the field we’re using for the identifier (in this case, System.String).

image

Next, we’ll edit the BDC Methods.  First, click the BDC Method Details tab at the bottom of the editor windows.

image

Expand the ReadList node. Click Entity1List under the Type Descriptor column.  Click the drop down arrow and select <edit>. In the Properties Box, change the Name property to RSSItemList

image

Expand the Instances nodes under ReadList.  Select the ReadList node.  In the Properties box, change the Default Display Name property to RSSItemList.

image

Next expand the ReadItem node.  Click Identifier1 in the Type Descriptor column.  Click the drop down arrow and select <Edit>.  In the Properties window, change the Name property to the name of your identifier field (in our case, Guid).

image

Next, click Entity1 in the Type Descriptor column.  Click the drop-down arrow and select <Edit>.  Change the Name property in the Properties window to RSSItem.

image

Expand the Instances node under the ReadItem node. Click the ReadItem node under Instances.  In the Properties window, change the Default Display Name property to RSSItem.

image

Next, click the BDC Explorer tab, which should be next to the Solution Explorer tab.

image

Updating Type Descriptors for Entities in our BDC Method Instances

First, change the name of the LobSystem note in the tree, directly under the RSSItemModel node, currently named BdcModel1. Change it to RSSItemModel.

Next, expand the RSSItem node, and expand the ReadItem and returnParameter nodes beneath it.  Expand the RSSItem node.  You should see something similar to the following:

image

First, click the Identifier field.  in the Properties window, be sure the Identifier property reads Guid.  If not, set the Identifier property to Guid.  Next, change the Name property to Guid and ensure the Type Name property is set to System.String.

image

Next, select the Message node under RSSItem in the tree.  Change the Name property to match the next property in our entity (in our case, Title). Ensure the Type Name property matches the .NET Type of that property in our entity (in our case, System.String).

image

Now, we need to add Type Descriptors for the remaining columns in our entity.  For each remaining property in our entity, right click the RSSItem node under returnParameter in the BDC Explorer tool window and click Add Type Descriptor.

 

image

Fill in the Name and Type Name properties for each remaining property using the values in the table below:

Name

Type Name

Link

System.String

Category

System.String

PubDate

System.DateTime

Description

System.String

You should end up with an RSSItem tree resembling the following:

image

Next, expand the ReadList, returnParameter, and RSSItemList nodes.  Select the Entity1 node under RSSItemList.  Right click the node and click Delete.

 

image

Next, select the RSSItem node under ReadItem –>returnParameter that we just edited above.  Right click the node and click Copy.

image

Next, right click the RSSItem node under ReadList –> returnParameter and click Paste.

image

This should result in a ReadList node that resembles the screenshot below.

image

Next, build the solution.

Note: If you encounter build errors, check your RSSItemService.cs to be sure stubs for BCS model operations have not been added by the designer to the bottom of your class.  If they have, delete them, leaving the ReadItem and ReadList methods we changed earlier.

Setting Required BCS Custom Properties

Site URL Property

We have to set a BCS Custom Property to allow us to successfully deploy the BCS model.  For some reason, the SharePoint Configuration Wizard sets the Site URL project property, but doesn’t create the SiteUrl property in our Feature that is needed for successful deployment.

To add this property, click RSSItemModel in Solution Explorer.  in the Properties window, click on (Collection) next to Feature Properties and click the ellipsis () button.

image

In the Feature Properties dialog, click Add.  Set the Key property to SiteUrl and the Value property to the URL you selected in the SharePoint Configuration Wizard when you created the project. (Note: this should be the same value as the Site URL property in the project properties)

image

Setting BCS Custom Properties to Make BCS Model Searchable In SharePoint 2010 Search Service Applications

Just as with the SiteUrl BCS Custom Property, there are other custom properties we need to set in order to make the External Content Type we’ve created usable in SharePoint Search as a Line of Business content source.

First, we need to set the ShowInSearchUI property on the LobSystemInstance to make the external content type show up in the Content Sources properties in the SharePoint Search SSA (and/or FAST Search Content SSA)

Open the model design surface by double-clicking RSSItemModel.bdcm in the Solution Explorer tool window.  Click the BDC Explorer tab (next to Solution Explorer), and expand the Model, RSSItemModel, RSSItemModel, and LobSystemInstances nodes.  Click the node under LobSystemInstance, which should be named RSSItemModel.  If it still reads BdcModel1, not, rename it to RSSItemModel.

Click (Collection) next to Custom Properties in the Properties tool window and click the elipsis () button.

image

In the Property Editor window, type ShowInSearchUI in the Name column and hit the enter key.  Double-click System.String in the Type column, and select System.Boolean from the drop down box. Double-click the empty Value column and enter true in the box.  The result should look like the screenshot below.  Click the OK button when done.

image

Next, we need to identify the RootFinder method instance.  The RootFinder method instance allows SharePoint to know which method to use to enumerate the entity collection for search.

Click the RSSItem entity in the designer pane and select the BDC Method Details tab in the bottom tool window.

Under the Methods node, click the ReadList node.  Click (Collection) next to Custom Properties in the Properties tool window and click the elipsis () button.

image

In the Property Editor box, set the Name column to RootFinder, type to System.String, and Value to x. Click OK when done.

image

Next, expand the Instances node under Methods –> ReadList in the BDC Method Details tool window.  Click ReadList. Click (Collection) next to Custom Properties in the Properties tool window and click the elipsis () button.

image

In the Property Editor box, set the Name column to RootFinder, type to System.String, and Value to x. Click OK when done.

image

Setting the Title Property for our BCS Model

Next, we’ll set the Title field for the entity.  If this BCS Custom Property is not set, any search results that are returned from this content source will have the name of the BCS Profile Page as the title, and they will all be identical.  This makes the search results less valid.

In the BDC Explorer tool window, click RSSItem under Model –> RSSItemModel –> RSSItemModel. Click (Collection) next to Custom Properties in the Properties tool window and click the elipsis () button.

image

In the Property Editor window, add a new property.  In the Name column, enter Title. In the Type column, set System.String.  In the Value column, we will put the name of the property on our entity that we want to use as a title for search results.  In this case, it’s Title. Click OK when done.

 

image

Now, build and deploy the project.

Verifying Successful Deployment of the .NET Connectivity Assembly

Configuring the External Content Type in the BCS Service Application

Open up Central Administration on your SharePoint 2010 instance, and navigate to your BCS Service Application Instance.  (Central Admin –> Manage Service Applications).  If the .NET Connectivity Assembly has been successfully deployed, you will see the RSSItem entity listed in the table as below:

image

Click the checkbox next to the RSSItem entity in the table, and click the Configure button on the toolbar in the Profile Pages section.

image

In the Configure External Content Type Profile Page Host select the SharePoint site where you want your ECT Profile Pages created. Click OK at the bottom of the popup window when done.

image

Check the checkbox next to RSSItem in the table and click Create/Upgrade in the Profile Pages section of the toolbar.

image

A Loading… popup window will appear while the profile pages are created.  When completed, an informational dialog will appear. Click OK.

image

Another window will be displayed when the profile page has been successfully created.

image

Next, check the checkbox next to RSSItem again, and click Set Object Permissions in the Permissions section of the toolbar.

image

In the Set Object Permissions box, add appropriate permissions for the search service account, content access account, and any app pool accounts for any web applications that will be using, consuming, or displaying data from the RSSItem External Content Type.

Click OK when done.

Setting up a Content Source for Crawling our BCS Entity Data

In Central Administration, navigate to your Search SSA.  This will either be your SharePoint Search SSA if using SharePoint Server Search, or your FAST Content SSA if using FAST.

Click Content Sources under crawling.  In Manage Content Sources click New Content Source.

image

In the Name field, type a name for this content source.  I’ll enter RSS.  For Content Source Type select Line of Business Data.  This should change the rest of the page to show external data sources.  Under External Data Source, you should now see an enabled drop-down labeled Select the Business Data Connectivity Service Application, and two radio buttons, one entitled Crawl all external data sources in this Business Data Connectivity Service Application, and one entitled Crawl selected external data source.

Select the second radio button, and check RSSItemModel.

image

Under Start Full Crawl, check Start full crawl of this content source, and click OK.

Let the crawl start up and complete.  You can click the Refresh button to monitor the crawl, or look at the Crawl History table on the Search Administration page.

Note: If your crawl doesn’t execute successfully, look at the crawl log for error messages.  (messages sometimes take a few minutes to appear in the log after the crawl completes)  I had unsuccessful crawls due to an account I missed putting in the BCS Set Object Permissions dialog.

image

Validating Search Results

Once the crawl is complete, go to your Search Center page and execute a search for a term you expect to see in the RSS Feed.  I believe there was an article about the president in the top feeds, so I’ll search Obama in my search center.

image

As you can see, we get 2 results, and the Title of the search result is the value from the Title element in the RSS feed, which we set using the Custom BCS Property.

That’s it.  Please leave any questions in the comments section!

6 Comments. Leave new

Reddy Kadasani
August 22, 2012 5:03 pm

So a few additional things to remember:
1. The ReadItem method is called by the crawler for each item being crawled. So make sure you implement the ReadItem method as well (in addition to the ReadList) method.
2. If you need to debug through your code, the process you need to attach to is mssdmn.exe, NOT w3wp or mssearch. If you have more than one instance of this process, attach to all of them.

Thank you so much!
It wasted my whole day to research the ECT.I dont know where was the issue. I just follow your blog and solved the issue. You saved my Saturday Night!
Thank again!

Thanks, this finally resevlod my issue.With this solution I can replace the news section on our WSS home page with a blog (subsite) as there appears to be no other way to display content of subsites.All of this is required to allow users to post news items with an easy way for them to upload pictures (post directly from MsWord)One small adjustment I had to make was to create an RSS reader account on our WSS site with read rights and specify that account in the login tab of the data source properties.Thanks!

I’ve done the same steps as mentioned in the article. But still I’m not able to see my external data source while creating new content source. Please help.

Now I’m able to get the data source while creating a content source. But ACL is not applied to the results. Any help please.

poulomi das
June 30, 2014 6:50 am

ok, i have a problem.
i have done exactly the same process, literally copied it. But once i build it and opened my central admin to check the business data connectivity service, the rssitem is not showing in the list

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

\\\