SharePoint MetaData and Classification

Using AD Groups in a Claims Based Web Application

I ran across an interesting little side effect of altering my claims enabled web application in SharePoint 2010 the other day I thought would be useful for others to know about. The situation has to do with accessing AD Groups from within the SharePoint people picker in a claims enabled environment.

Initially one of my SharePoint web applications was configured to allow authentication either through NTLM or through a Trusted Claims Provider (which was ADFS 2.0 sitting in front of AD of course). I could select which method I wanted to use when I navigated to the site through the default sign in page which presented a drop down list of the 2 authentication providers. At any time I could authenticate through NTLM or through the claims provider. This was all standard stuff so far.

I decided that I should clean up this web application since I was only using claims based authentication, and I removed the option for NTLM authentication. This just made the environment a little cleaner from a usage and demo perspective.

Then I logged into my web application through the ADFS 2.0 provider and proceeded on with additional work. I was configuring an existing library with a new metadata column – it was a People and Group column. My intention was to use this column in a demonstration where I was selecting groups were associated with specific items in the library. I wanted to be able to specify a different group for different documents in the library. In particular I wanted to use Active Directory groups and I already had a number of groups setup. So, I add my column and tried to select the AD groups I already had been using for some time. It turned out that my SharePoint people picker could no longer resolve the groups that I was searching for.

I checked over my AD installation. I checked that my groups were configured correctly. But nothing seemed out of the ordinary, nor would any changes here to improve the situation. Then I noticed that my people picker was missing the Active Directory node – it looked like this:

I was no longer able to search against any Active Directory groups!

So, with my upcoming demonstration looming, to try to resolve this I returned to Central Admin and the Manage Web Application page. I reconfigured the authentication providers for this web application so that once again I could sign in through either NTLM or my trusted ADFS 2.0 provider. I logged in once again using the ADFS 2.0 provider, went to add a group to my metadata column an I could find my AD groups again. My people picker now looked as follows:

Notice the Active Directory node on the left side. I could now find and select my ‘JointChiefs’ group and add it to my column (note I need to choose the one on the right side that is under the Active Directory heading.

So, it turns out that even if I’m always signing into SharePoint through my ADFS 2.0 claims provider, I need NTLM or Kerberos configured as an authentication provider for my web application in order to access Active Directory entities like groups in the SharePoint people picker. Please keep in mind that this is without any customization of the people picker using a custom claim provider.

Hope this helps someone that might run into the same situation.
– Antonio

SharePoint Security Paper: Harness the Power of Claims to Protect Information

The topic of claims, or trusted user attributes, is an established and well defined concept within the
identity management space. In SharePoint 2010, Microsoft introduced support for a claims-based identity model. Claims-based identities can be used in Microsoft SharePoint to enhance the process of user authentication. By harnessing the power of claims and extending its use to authorization, organizations can also effectively manage security and governance policies for accessing valuable information assets. It is important to
understand the many concepts of the claims-based identity paradigm in order to implement granular,
scalable and dynamic security within Microsoft SharePoint.

Organizations can implement and effectively leverage user claims in Microsoft SharePoint 2010 to securely share sensitive information. By implementing claims-based authorization in SharePoint and leveraging the benefits of trusted user attributes, organizations can solve many of the security challenges they face in their collaboration and file sharing environments. Further, it becomes evident that in order to properly manage security and fully realize the potential of claims, organizations must have solutions that automatically enforce access control and security policies on all of their SharePoint content.

This new white paper describes how organizations can implement and effectively leverage user claims in Microsoft SharePoint 2010 to securely share sensitive information.  Download the “Harness the Power of Claims to Protect Information” white paper, or go to our webpage, and learn how you can successfully implement claims-based authorization across all the SharePoint content in your organization.

Using Microsoft Word Quick Parts to Add Metadata to SharePoint

I recently came upon a new way to add metadata to SharePoint.  Using Quick Parts in Word documents allow users to quickly add metadata without them even knowing about it!

Typically Quick Parts would be used in templates or forms where you want users to fill in standard fields.  This can also be done using Microsoft InfoPath, but that would require a more sophisticated user or administrator to set it up.

The steps to set this up using Quick Parts are as follows:

1. Create the custom columns in SharePoint that will be used to store the metadata.
2. Add the Quick Parts to a Microsoft Word document that is created in SharePoint. These Quick Parts will point to the metadata columns just created in SharePoint.
3. Save this Word document as a template in the SharePoint library
4. Users can then open the template when they click "New" in the SharePoint Document Library
5. Users populate the Quick Part with the metadata, either by typing it in, or by selecting from a dropdown.
6. When the doucment is saved into SharePoint, the metadata is saved into the SharePoint columns.

This is explained in more detail in an excellent blog by Kevin DeRudder called Using quickparts in document templates.

Here is my quick example. I created a document library that is used to save administrative documents for a cruise ship company.   I added 2 new columns to the library called Cruise Month, which captures which month the ship sailed, and Cruise Classification, which captures the classification of the document. (Click image to get larger view)

I then created a word document and inserted these columns from Quick Parts

Here is the Word document.

I then save the Word document locally, and copy this document to the SharePoint document library template folder.

Then when users click on New document in Sharepoint, this template is opened.  The users can change the values of the Quick Parts:

When the user saves the document back to SharePoint the values in the Quick Parts are automatically added to the appropriate columns in SharePoint.

 

  

New File Share to SharePoint Migration Utility from Microsoft

Microsoft is releasing a new Powershell script to allow customers to upload files directly from a file share to a SharePoint document library.  In addition, the script will allow files that have metadata to carry that metadata into SharePoint.  The script was developed by the group responsible for the Windows Server 2008 R2 File Classification Infrastructure and supports both SharePoint 2007 and SharePoint 2010.  If you are not familiar with the File Classification infrastructure, it allows organizations to assign classification metadata to files on a Windows Server 2008 R2 file server.  For more on File Classification infrastructure see one of my earlier posts on this topic.

The business reason behind the script is that a lot of organizations still have thousands of documents on server file shares.  At some point these organizations will want to migrate those files into SharePoint to make them more available for collaboration, and to facilitate management of the files. FCI allows organizations to assign metadata to all the files on a file server, and this new script will enable transfer of the files and the associated metadata into SharePoint.

The combination of FCI and SharePoint overcomes a major hurdle in using the full power of SharePoint -populating files with metadata.   Metadata allows users to filter and search information in SharePoint in a much more powerful way, especially with SharePoint 2010 features such as navigation hierarchies and key filters. Many customers say they can get their files into SharePoint, but they have no idea how to assign metadata to all these files.  FCI along with this new script allows organizations to overcome this hurdle. 

Getting metadata into SharePoint provides benefits for user search and file disposition and can also be used to strengthen security via products like our Metadata Security for SharePoint. 

The new script is called FciSharePointUpload.ps1 and is available from Microsoft at this download site.

The script provides two options:

Option 1: Upload a file to a SharePoint Library - SharePoint 2007, SharePoint 2010.

 

FciSharePointUpload.ps1 –file <file path> -url <target site> -libPath <relative library path> [-name <target name>] [-sourceAction {keep|delete|url}] [-targetAction {overwrite|skip|fail}] [-propertyAction {copy|ignore}] [-user <user name> -password <password>]

 

Option 2: Upload a file to SharePoint using the official file web service that is coupled with a content organizer (e.g.: Record center) – SharePoint 2010 only.

 

FciSharePointUpload.ps1 –useOfficialFileWebService –file <file path> -url <target site> [-name <target name>] [-contentType <content type name>] [-sourceAction {keep|delete|url}] [-propertyAction {copy|ignore}] [-user <user name> -password <password>] [-additionalProperties] <property list>

 

The parameters are as follows:

• file: path to the file to upload (required)
• url: URL to the SharePoint site (required). Example: -url ”http://sharepoint/sites/mariant”
• libPath: relative path to the document library within the site (required), Example: -path ”Shared Documents”
• name: name of the target file (optional). If not specified, it defaults to the source file name as specified by the “file” argument. Example: -name Test.docx
• contentType: name of the content type to use for the document (optional). If not specified, defaults to “Document”
• sourceAction: indicates how to handle the source file after a successful upload (optional) - 1) keep - leave the file in place (default) 2) delete: delete the file;3)url: replace the file with a shortcut to the uploaded document’s URL
• targetAction: specifies how to handle existing destination documents (optional)  - 1) overwrite, replace (default) , 2) skip: leave existing version and continue , 3) fail: return error
• ropertyAction: specifies FCI property transfer mode (optional) - 1) copy: transfer properties (default) , 2) ignore: do not transfer properties, let SharePoint choose defaults
• user: user name to use for authentication (optional). The argument is ignored if “password” is not specified.
• password: password to use for authentication (optional). The argument is ignored if “user” is not specified.
• additionalProperties: list of extra properties to use for the document (optinal). The argument is followed by a space-separated list of strings. Each group of three strings is interpreted as the name, type and value of the property to add, in this order. The list size must be a multiple of three. These additional properties are always used, regardless of the propertyAction argument value. Example: -additionalProperties PII Boolean True BusinessImpact Choice High

Here is an example I created.  I created a file share called "Merger".  I then tagged all the files in this share with metadata property called CorpClassification, and a value of "CONFIDENTIAL" using FCI (you can't actually see the metadata in the file system).

I then created a column called CorpClassification in a SharePoint library called Merger, and using the script I copied these three files into the SharePoint library.  As you can see the metadata has been transferred with the files.

Be careful when creating the property and column type in FCI and SharePoint.  These have to match up as per the script requirements, or the script will not work.  In my case, the FCI property was of type "Choice", this means I had to create a column of type Choice - Checkboxes in SharePoint. If I tried to define any other column type (e.g Choice - Dropdown) the script will not work.

Using SharePoint 2010 Content Organizers

Well, today is the last day here at the Microsoft SharePoint conference.  It's been a busy week.  Being at the show you can feel the momentum Microsoft has built with SharePoint.  There is a tremendous community of skilled SharePoint developers, administrators that have bought into the SharePoint platform. SharePoint 2010 will propel this another step forward.

For SharePoint 2010, Mcrosoft has introduced a couple of new methods for getting files and metadata into SharePoint,  so I thought I would make that my topic today.

When first getting started with SharePoint, organizations often want to migrate infromation stored on file shares up into SharePoint.  In addition, they would like to populate some metadata with the files transferred into SharePoint.  The metadata can be used for search, security, or retention.  In SharePoint 2010 Microsoft has introduced a new feature called the Content Organizer.  The Content Organizer is a way to route new files into the appropriate folder, document library, records center etc. The content organizer allows you to specify rules so that when files enter the drop-off folder or library, rules are applied to route the files to the correct location.

A site administrator can define a content organizer that will route new documents based on metadata.  For example, all files tagged as "Finance" could be routed to a Finance document library, and all documents tagged as HR could be routed to an HR document library. In addition, if files have been tagged as "Archive"  they could be routed to a Records Center.  Using metadata with the Content Organizer feature provides a myriad of document management possibilities. It's a great way to migrate a large number of file share documents up into appropriate libraries or folders in SharePoint 2010.  If you have files that have never been tagged with metadata you can use a tool like the Titus Labs Shell Extension, or you could use the new File Classification Infrastructure that is part of Windows 2008 R2 as explained in one of our earlier posts.

In order to turn on a Content Organizer go to Site Settings, pick Content Organizer Rules and create your rules. When creating a rule you can use any of the metadata fields associated with the file. You can also have the content organizer automatically create subfolders inside a /Pages library if the target location is getting too big. Here is a look at one of the screens.

In addition to bulk uploading files shares into SharePoint using the Upload Multiple Documents method, Microsoft will also be providing a script which can be used to automate the process of moving file shares into SharePoint. This can also be used with SharePoint 2007.

Microsoft Embraces Metadata in SharePoint 2010

I'm at the Microsoft SharePoint conference this week and it's all about SharePoint 2010.  It seems to be a huge release for Microsoft with a wide range of new features for end-users, IT pros and developers.  I took in the general keynotes yesterday and then finished the day at the "ECM for the Masses" presentation.  Because our focus is on metadata I was especially pleased with this last presentation that started to dig into all of the new metadata features and how they can be used for document and information management.

Microsoft is really embracing the use of metadata in SharePoint 2010.  There are a number of new features that help the user generate valid metadata and help users more effectively search and filter information in SharePoint based on this metadata.  I won't be able to cover all the new stuff here, but will be covering this in more detail over the coming weeks and months.

I guess the most important new feature is what Microsoft is calling Managed Metadata Services.  This is a service that can be activated via Site Settings that allow organizations to define their metadata structure and the metadata that is valid for use within SharePoint.  This defined taxonomy can be hierarchical if it fits the organization.  Users can quickly navigate the tree when creating content to assign relevant metadata. The Managed Metadata Service is an Enterprise service.  A configured metadata service can be used to provision metadata across multiple farms in the organization.  As single document ibrary can consume from multiple metadata providers.

The metadata can be assigned to documents both within SharePoint, but also directly within Office 2010 documents using the properties features in Office 2010 Backstage. Within the properties in Backstage a user can click on a SharePoint custom column name (e.g Project) and then navigate the metadata tree to assign the properties.

In addition to the managed metadata service, Microsoft is also embracing folksonomies or ad-hoc metadata in SharePoint 2010.  Users can add their own tags to information.  This can be done in document libraries but also in the new social media features such as wikies, blogs etc. SharePoint will keep track of these tags and make them available next time a user wants to assign a tag.  SharePoint tracks all these add-hoc tags as well as the managed metadata in what they are calling "SharePoint Keywords".

So what is so great about all of these new metadata features?  Well it's how they can be used that's powerful.  Microsoft has introduced the concept of "navigators" in SharePoint 2010 that will allow users to find information very quickly.  These navigators can appear in a left hand pane of SharePoint. By clicking on metadata tags or by navigating a metadata hierachy in the navigator, the user can narrow down the scope of the information they are looking at in SharePoint.  For instance the user could navigate to Finance metadata, and then perhaps to mergers, to find all of the documents that have been tagged as "merger" in the document library.  This is great stuff for e-discovery also.

Well. that's it for now.  More to come later this week...Charlie

Other Useful SharePoint Metadata Blog Posts

Here are some other blog posts that cover SharePoint metadata that I've found useful:

Top 10 SharePoint Metadata Tips for 2008 - Part I

Top 10 SharePoint Metadata Tips for 2008 - Part II

Updating Metadata on Files in SharePoint: Low Cost, High Reward

IDC Survey Says Metadata Management Important in SharePoint

Using metadata instead of folders

Use Quick Parts to Display Metadata fields

Enjoy the weekend reading!

Metadata or Content Types?

Content types seem to be a “hot” item in SharePoint.  But customers sometimes get confused about when to use content types.   Content types can contain custom metadata requirements, but as we’ve mentioned custom columns is normally the way to get metadata into SharePoint.  Custom columns work great if all of the documents in your document library will have the same metadata fields, but what happens if I want to collect different metadata for documents in the same document library? That’s where content types come in.

Content types are a means of encapsulating a data schema and making it independent of a SharePoint document library. A content type can include the following information:

• The metadata, or properties, you want to assign to this type. These are represented by columns added to the document library when you add the content type.
• Custom New, Edit, and Display forms to use with this content type.
• Workflows available for items of this content type.
• For document content types, the document template on which to base documents of this type.
• Any information necessary for custom solutions associated with this content type. You can store this information in the content type as one or more XML documents.

Consider a document library for all documents that relate to a legal department.  The library may contain contracts and also legal guidelines.  For contracts we may want to collect the signing authority as metadata.  For legal guidelines we may want to collect the department it was written for.  We won’t want to prompt the user for a signing authority when they create a new Legal Guideline, as that metadata is only applicable to contracts. The solution to this problem is to create two content types.  In this scenario we will want to create a Contract content type and a Legal Guideline content type. When a new document is created, SharePoint will prompt the user for the content type (contract or legal guideline), and once selected, the appropriate metadata prompt will be displayed.  The examples below show the metadata prompt when a new contract, or legal guideline is created.

 

 

Content types by themselves can also be considered metadata.  Knowing the content type will tell you something about the document.  For example, knowing the document is of content type “Contract” tells you something about that document.  You may want to sort or search based on this content type e.g. find all Contract content types in the Legal document library.  So content types by themselves, even without additional custom columns, also act as valuable metadata.

Content types can be defined at the site level so that they can be reused in many document libraries.  Custom columns can be defined at either the document library or the site level.  I would recommend you use content types if you non-standardized metadata, or if you have templates or workflow you want to associate with certain types of documents.  If you have standardized metadata or a classification taxonomy you want to apply to all the documents in your organization, I would use custom columns defined at the site level.

To define a content type, go into Site Settings, select Site content types, and click the Create icon.  Once you have given the content type a name and clicked OK, you will be able to define advanced properties.  Scroll down this page and add the columns you want to collect as metadata for this content type.

How to get Metadata into SharePoint (Part 1)

Last week we talked about SharePoint metadata and why it’s good to have metadata.  This week we are going to talk about how you can get metadata into SharePoint document libraries.  Many of our customers are interested in using metadata for document security, search or retention decisions, but they have never added any metadata to their document libraries.  So their first question is – How do I get metadata into SharePoint? The best way to get metadata into SharePoint will vary depending on whether you are creating new documents, or whether you want to add metadata to existing documents.  It’s easier to add metadata when you are creating a new document, but there are always some situations where you have some old documents that you added to SharePoint without any metadata.  Now you need to go back and add metadata.

Let’s first look at some of the ways of getting metadata into SharePoint when you are creating new documents.  The first thing we need to do is to make sure a document library column exists that can hold the metadata.  In some cases you may want to populate some of the default columns that already exist for document libraries (like document title, file size, copy source etc), but more likely you will want to create metadata which is useful for your organization.  To do this you will create a custom column, either directly in your document library, or as a site column which can be used by all document libraries in the site. Administrators can create new document library columns by Editing the Document Library Settings.  Under the column setting, select Create Column.  Now this is where it gets interesting.  You need to decide what type of information you are creating and how you want to collect it.  Do you want the user to type in the metadata, do you want them to select from a dropdown list, or do you want to get the information in some other way.  You also need to decide if the metadata is required for every entry. Let’s say we want to create a column called Department that will specify what department created the document.

 

Figure 1 - Create Department column

If you want to make sure the document always has some value for the metadata column you are creating, an easy way to do this would be to select Yes for Require that this column contains information. When a user is uploading or saving a new document to this document library they will be prompted for the metadata.

  

Another way of making sure there is some metadata in the column is to assign a Default value for the column.  In this way, if the user does not specify a value, the default value will be assigned for that document.  This works well if almost all the documents will have specific metadata.  For example if the SharePoint document library is owned by marketing we know that 99% of the time the proper value for Department will be Marketing. Default values may not work so well if the metadata can vary a lot.  Lazy users that are given a default may just accept it in which case you may get misleading metadata.  In this situation it may be best not to assign a default value.  

Assigning default values can be valuable if you want to bulk load documents with metadata.  If you want all of the documents to have the same metadata you can assign default values for columns and then do a bulk upload.  For example, if you want to upload a number of documents for the marketing department that are all related to the new MP3 project, you can assign Marketing as the default value for the Department column and MP3 as the default value for the Project column.  When you bulk upload several documents to this library they will appear with the correct metadata.  

 

If you create new documents from within SharePoint, and you are using Office 2007 to create the documents, the Document Information Panel can be used to enter metadata. For example, if you click on the New button in the library and Microsoft Word is the default template you will see the Document Information Panel appear.

 

Figure 2 -Document Information Panel

If a column is required, the user will not be able to save the document until they have provided a value for the column.

 

The Document Information Panel can also be used to enter the standard document metadata such as Author, Title, Subject etc.

Our document classification product, called Titus Labs Document Classification can also be used to create metadata which can be imported into SharePoint.  We’ll cover this topic in more detail on another blog post next week.   Check back on Wednesday for more on how to get metadata into SharePoint.

Charlie

 

What is SharePoint Metadata?

I’m sure you’ve heard this one before - metadata is data about the data. I remember the first time I heard that and it sounded like a bunch of theoretical gobbledygook. Data about the data? An easier way to think about metadata is to think of metadata as attributes of the object.  Let’s take a book as an example.  Attributes of the book could include the title, author, date of publication, subject and ISBM number.  All of these attributes can be called metadata of the book.

In our context – SharePoint document libraries - metadata is data about the documents.  These can be any properties that describe the attributes of the document.  This is easier to understand because you will actually see the metadata in a SharePoint document library.  By default SharePoint captures certain metadata such as document title, document type, last modified date etc.  These are displayed in SharePoint as columns. 

Figure 1 - Metadata displays as columns

So why do I need metadata?  Why not just store my documents and move on.  Well let’s go back to our example of a book.  How do you find a book at the library?  You search by title, author etc.  Ahh, so we can use metadata to make search easier. Often you can decide what additional metadata might be needed by thinking about how you want to use the metadata – for instance for search, security or retention. If you want to use metadata for retention, think about what type of metadata you will need to determine the retention of a document.  For example, we may want to know if the document is an official company record before we decided how long to retain it.  In this case we may want to create metadata called ‘Company Record”,  which would have a value of either yes or no.

Typically a document will have more default metadata than is displayed to the users in SharePoint.  Administrators can modify the view.  The view will define which columns are displayed to the user when they go to the document library.  When an administrator modifies the view for a document library, they can see alot of the metadata that is by default associated with the documents.  In many cases, not all of this metadata exists, for instance, the metadata field in Check in Comment probably doesn`t exist unless someone has checked in document and added some comments.

Figure 2 - Standard metadata

In addition to this default metadata maintained by SharePoint, it’s easy to add additional metadata to SharePoint document libraries by creating custom columns.  Custom columns will represent metadata that is especially meaningful to your organization, so you will want to record it when documents are created.    You can capture any metadata for a document by creating a custom column.  For example we may want to add metadata about what department created the document, and what project it is associated with.  In this case we could create two custom columns called Department and Project.

Figure 3 - Department and Project metadata added

Those are the basics on what SharePoint metadata is.  Next week we intend to do another post that will describe how to get this metadata into SharePoint.

Charlie