Practical Metadata

Data.gov

2010-02-12T10:42:00.002-05:00

Most of my time has been consumed analyzing data.gov data.

Please see the data.gov collaboration site and share you ideas on improving data.gov.

I also have an idea there on metadata and asking for help in identifying the right metadata elements for the data.gov template.

Hope to see you there!

Digging out from the avalanche

2008-11-27T10:23:00.000-05:00

Hi Everyone,

Been incredibly busy with the new business ... head underwater the last few months. Finally clawing my way back to the surface!

Though not about metadata, read a great article a few days ago on a security expert named Kaminsky and how he discovered and helped fix a huge security hole in the domain name system. I liked this article on so many levels - I highly recommend you check it out.

In regard to metadata, most of my work these days are revolving around IBM's Infosphere product line. Especially there business glossary and it's ability to provide line-of-sight between your business end-users and your physical IT assets.

More soon ... - Mike

Crazy busy!

2008-06-08T12:02:00.000-04:00

Well, recently joined a startup as the Chief Technology Officer.
See our web page for details!

[LTMP] Link Title Metadata Proposal

2008-02-17T12:25:00.003-05:00

Link title metadata has become common place on news aggregrator/ranking sites like reddit, digg, delicious, and many others. A search today for "NSFW" on google returned 7,350,000 hits.
For those new to link metadata, NSFW is an acronym for "Not Safe For Work". This is Metadata in the title of a link to provide additional information about the linked resource for the benefit of readers. A tag of "NSFW" means that the end-user should NOT click on the link if they are at work or they could get in trouble. Adding such descriptive metadata to the title of a link is a great service to end-users. I applaud whoever started this trend. However, I have recently seen ambiguous and inconsistent variations on the use of link title metadata which means it may be time for a standard. For example, I recently saw [PIC] as a keyword; of course, PIC is not an acronym and right down lower on the same page was [Image] which is a much better description. Why do we need a standard? Browsers and sites (like reddit) may want to rely on this metadata for their customers - for example, a site may want to place all NSFW content in a separate section of their site. To do this, programs needs to be able to rely on the metadata ... fortunately, by following a few simple rules we can accomplish this.

LTMP Proposed Standard:

Enclose Link Title Metadata keywords in brackets; for example, [NSFW] or [SFW].
Link Title Metadata should either be at the beginning or end of a link title.
Reserve all capitals in a keyword for acronyms. Use a capitalized keyword for non-acronyms; for example, [Image] or [Video] or [Song].
The community should develop a list of well-known LTM keywords. I have also seen [Tutorial], [], [Graph], [Cartoon], [Non-Linkjacked], etc. Here is an initial list with definitions.
Link Title Metadata can have multiple words in a keyword; however, multiple keywords should be separated by a ',' (comma). For example, [NSFW, Image].
Plural keywords should be used when appropriate; for example, [Image] versus [Images].
[optional] If you have an authoritative source for the keyword, that can optionally be added with a "source" keyword followed by a ':' (colon) followed by the metadata keyword. An example of this would be "[W3C:Standard]OWL 2.0 is out!".

What do you think?

Speaking on Metadata at the DAMA conference

2008-02-02T11:18:00.000-05:00

I just finished my presentation on "Effective Metadata Design" for the upcoming DAMA conference in San Diego. This is a presentation on the Appendix in my new book by outskirts press.

The presentation delves into metadata design by focusing on the seven specific techniques in metadata design: identification, static measurement, dynamic measurement, degree, categorization, relationships and commentary. Each is presented with examples on application of the techniques. Hope to see you there!

On another note, I will also be speaking at FOSE and a few other events in the works...

See you at the conferences!

Awarded United States Patent 7299408!

2007-11-21T12:34:00.000-05:00

See the announcement on my website for details!
It is important to note that this directly relates to the concept of Information MVC and the case study in my new book: Information As Product.

Book Launch, GCN, DAMA and Cringley!

2007-11-12T17:02:00.000-05:00

Lots of things going on lately! The top news would be the availability of my new book. You can order now at http://www.outskirtspress.com/daconta
So far, reviews our very positive with several government folks considering it a "must-read"!

Second, my latest GCN column is out and discusses the high-costs of ambiguity. In fact, it raises some issues relating to this blog - especially the ambiguity, for some, between metadata and data.

Third, I will be speaking at the next DAMA conference, on effective metadata design. This talk will cover all aspects of metadata raised in my new book and teach practitioners how to take their metadata management to the next level! Hope to see you there!

Finally, Robert Cringely has a new article out talking about a new opportunity for Google as a metadata provider.

It is good to live in such interesting times! Best wishes, - Mike

Reviewers Wanted for my New Book!

2007-10-16T21:18:00.000-04:00

Hi Everyone,
Are you interested in reviewing my new book? If so, go to my website and contact me via the contact form for details. Please put in the subject line [IAP REVIEWER]. I will not ask, question or comment on the contents of your review in any manner - that is your unfettered opinion (and I wouldn't have it any other way). If you post your review online, I will send you an autographed copy of the book.

The book covers many of the concepts put forth on this blog. Specifically there are two key sections on metadata - chapter 3 on the Information Catalog and Appendix A on Effective Metadata design.

I will have many more posts and excerpts from the book in the following months.

Regards, - Mike

Good Interview on XML and Data Management with Owen Ambur

2007-08-19T08:18:00.000-04:00

An interview with Owen Ambur,a former senior architect in the Interior Department and leader of the XML community practice, entitled, The Data Landscape, is available on GCN. I worked with Owen on the data reference model and other XML related activities. Owen was a passionate XML advocate and we held many superb discussions on how to fix data management. We both argued strongly for a more concrete approach to the DRM with concrete XML schemas for the reporting of data assets. Good to hear that Owen is doing well and staying involved. It was a pleasure to work with him.

Can Data Management help to Stop Fraud?

2007-08-16T19:34:00.000-04:00

This article reports how a company fraudelently billed the pentagon over $900,000 for shipping. Sounds to me like poor data validation. If the billing system uses well-defined metadata fields, this type of anomaly should be easily caught. Seems like the pentagon should get some best practice from the fraud detection division of the credit card companies!

New GCN Column

2007-06-24T06:23:00.000-04:00

I have been swamped and remiss in updating this blog. My latest Government Computer News column is out. The column series is called "Reality Check" and I always strive to drill down beneath the symptoms to the root causes underneath (revealing the underlying "realities"). In this month's installment, I look for the killer app in SOA. Next month, I will be examining Rich Internet Applications (JavaFx, Flex, Silverlight, AJAX). Enjoy!

Step one is Transparency!

2007-04-02T16:48:00.000-04:00

New article out over at GCN on transparency. It is critical to understand how without accessibility, metadata catalogs are worthless. So, transparency is first.
Best wishes, - Mike

Google's Approach to Image Metadata

2007-03-15T10:57:00.000-04:00

See how Google is trying to get better image search results: http://images.google.com/imagelabeler/.

This application essentially makes a "game" for you and a random internet partner to label random images to help improve the quality of Google's image search results.. Very interesting.

One of my friends said, "They've actually made assigning metadata to their images kind of fun!"

-Kevin T. Smith

Simpler, XHTML Friendly Metadata Format: RDFa

2007-02-16T12:19:00.000-05:00

Good article on XML.com about RDFa. Finally, the benefit of triples in an easy-to-use syntax that we can leverage on the web. Can someone please tell Google right away that we need them to announce they will exploit a "Document-Date" RDFa field so we can do a google search and check a box for "sort by MOST RECENT", so we don't have to waste hours reading old documents when we know a more recent and relevant one exists!
That current pitfall just bit me when I was researching whether the SBA was going to raise the size limits for small businesses. I had heard there was some recent news on this but the google search was bringing up old stuff from 2004. Very frustrating! Page rank is not always the most relevant criteria for search... remember - it is all about the 5W's.

The Machine is Us/ing Us

2007-02-14T13:00:00.000-05:00

Jim Feagans just shot me a URL for a neat video on the evolution of "smart data" on the web. It is very cool that an Anthropology teacher gets this! He comes to the conclusion that it is right and good (and beneficial) to teach the computer.
Thanks Jim!

Here is a transcript of the video:
Text is linear
Text is unlinear
Text is said to be unlinear
Text is often said to be unlinear
Text is unlinear when written on paper
Digital text is different.

Digital text is more flexible.
Digital text is moveable.
Digital text is above all…hyper.
Digital hypertext is above all…
hypertext is above all…
hypertext can link
hypertext can link
here
here
or here…
virtually anywhere
anywhere virtually
anywhere virtual
The WayBack Machine
http://yahoo.com
Take Me Back
Oct 17, 1996
Yahoo
View Source
Most early websites were written in HTML
HTML
HTML was designed to define the structure of a web document.
p is a structural element referring to “paragraph”
LI
LI is also a structural element referring to “List Item”
As HTML expanded, more elements were added.
Including stylistic elements like B for bold and I for italics
Suck elements defined how content would be formatted.
In other words, form and content became inseparable in HTML
Digital Text can do better.
Form and content can be separated.
http://www.cnn.com
RSS XML
View Source
XML was designed to do just that.
http://www.cnn.com/?eref=rss_topstories
same with
CNN.com
and
and virtually all other elements in this document.
They describe the content, not the form.
So the data can be exported,
free of formatting constraints.
Latest News
Anthro Blogs (124)
Savage Minds
8apps: Social Networking for Productive People
WORLD CHANGING ANOTHER WORLD IS HERE
Antrho Journals (124)
University of California Press
Journals Digital Publishing
Current Anthropology
AESonline.org
Google
With form separated from content, users did not need to know complicated code to upload content to the web,
I’m Feeling Lucky
Create Blog
Name Your Blog
Beyond Etext
http://beyondetext.blogspot.com
Choose a template
Your blog has been created!
Monday, January 29, 2007
Hello World!
POSTED BY PROFESSOR WESCH AT 8:14 PM 0 COMMENTS
There’s a blog born every half second
and it’s not just text…Search
YouTube
Broadcast Yourself
This is a video response to The Beauty of Being Human
flickr
Ahoy mwesch!
Upload Photos
Anthropology club
Created by you.
KSU Anthropology club
Club Photos
Google
XML facilitates automated data exchange
two sites can “mash” data together
flickr maps
I’m Feeling Lucky
Limelight
Fluffy and white
Brushy Creek
Tokyo Delve’s Sushi B..
Who will organize all of this data?
TAG
del.icio.us
digital ethnography hypermedia anthropology
save
Who will organize all of this data?
We will.
You will.
Google
XML + U & Me create a database-backed web
a database-backed web is different
the web is different
the web
we are the web
I’m Feeling Lucky
WIRED
We Are the Web
by Kevin Kelly
“When we post and then tag pictures
teaching the Machine to give names,
we are teaching the Machine.
Each time we forge a link,
we teach it an idea.
Think of the 100 billion times per day humans click on a Web page
teaching the Machine”
the Machine
Diigo
Highlight
Highlight and Sticky note
Mwesch’s private note
the machine is us
Digital text is no longer just linking information…
Hypertext is no longer just linking information…
The Web is no longer just linking information…
The Web is linking people…
Web 2.0 is linking people…
…people sharing, tracing, and collaborating…
Wikipedia
Web 2.0
edit this page
We’ll need to rethink a few things…
We’ll need to rethink copyright
We’ll need to rethink authorship
We’ll need to rethink identity
We’ll need to rethink ethics
We’ll need to rethink aesthetics
We’ll need to rethink rhetorics
We’ll need to rethink governance
We’ll need to rethink privacy
We’ll need to rethink commerce
We’ll need to rethink love
We’ll need to rethink family
We’ll need to rethink ourselves.
by Michael Wesch
Assistant Professor of Cultural Anthropology
Kansas State University

Metadata about Ontology Metadata

2007-02-07T17:49:00.000-05:00

I am at the RSA conference this week, where I am speaking about SOA Security. In a few months, I am going to speak with Eric Monk at Semtech 2007 about techniques for associating security classification & other metadata with subject-predicate-objects in persistence stores without using reification. I have written about only one approach (and it is kind of a hack), so let me know what you think: http://home.comcast.net/~kevintsmith/resteasy.html

-Kevin T. Smith

The semantics of Microformats

2007-01-02T14:00:00.000-05:00

Very interesting blog post by Alex Faaborg, a Mozilla engineer, on incorporating microformats into Mozilla. I have stressed focusing on the interrogatives in many conference talks and I encourage everyone to download the Operator plug in to try this out.

Metadata Experts Wanted!

2006-12-17T10:12:00.000-05:00

Hi Everyone,

I am busily working on my next book. In fact, I will be taking two weeks off around the holidays to make some serious headway. If you are interested in reviewing chapters for the book, email me at mdaconta AT oberonassociates.com (no mailto link to avoid spam robots). I plan on having the first batch of chapters ready for review in mid-january. I have changed the scope of this book several times and refined its approach and content. I am really happy with the book's new direction and am confident this will be an important book (note: if you think you know the subject matter - you probably don't because I have very recently changed the book title and outline). It is really shaping up nicely. I am only looking for a select few reviewers because I will be mailing out actual printed copies instead of electronic. If I know you - no need to send a resume - if we have never met and you want to be considered then email me your resume. I will get back to you after the holidays ...

Thanks, - Mike

Awesome Semantic Web Application!!

2006-11-25T21:09:00.000-05:00

Don't walk, run over to pandora.com for a glimpse at a robust music ontology in action! This website is a recommender system that uses a music ontology (smartly called the Music Genome Project) to recommend music similar to your known favorites by decomposing your preference into its component parts. I have been using this for a few days and my initial impression is that it works great! Kudos!
How many more "Genome" (i.e. Semantic Web) projects will we see in the coming years? If this is any indication, my guess is many, many ...

Torching Doctorow: Part 1

2006-11-25T19:47:00.000-05:00

Cory Doctorow wrote an article in 2001 called "Metacrap" where he purports to expose seven fatal flaws to reaching a "metatopia" where there is a "world of exhaustive, reliable metadata."

Before we dig into his 7 "straw-men", let's examine a little metadata about Cory Doctorow to attempt to determine his qualifications for making such assertions. First, it is interesting that he does not post any qualifications or any references to back up any of his assertions in the article. We do not know how much, if any, metadata he has actually created. Secondly, wikipedia claims that he only has a high school degree. Third, he is certainly not a practicing IT professional, per his own website. Given the above three things, we must take his assertions with a big, grain of salt. However, given that there are a number of blog entries and links to his article it is worthwhile to at least examine his arguments. Of course, publicity is not any indicator of truth and I surmise that most links to his article are merely people commiserating the fact that metadata is hard to do right. Fortunately, there are those of us that still believe that "hard" does not equate to "wrong" and that this is a temporary state due to lack of expertise. More on that later ... let's get back to Doctorow.

Strawman #1. People Lie. Doctorow uses this to attack the reliability of metadata. His argument is that because metadata "lives in a competitive world", people will lie to gain advantage. Frankly, this is a ridiculous statement because all metadata does NOT live in a competitive world. In fact, the most important metadata, or enterprise metadata, will not live on the "wild, uncontrolled internet" but in the controlled, corporate intranet.
So, let's debunk this in a number of ways:
a. People without access to my metadata can lie all they want and it won't affect me.
b. People lie more when they can lie without attribution. That is why Wikipedia has so many problems (lack of attribution).
c. People lie about both data and metadata but Doctorow is not saying we should distrust the entire internet.
So, the real point here is that Non-attributed data and metadata, of any type, is untrustworthy. Fortunately, every good metadata development process includes attribution via governance so this is truly a tangential argument (at best).
d. Metadata is not the victim of people lying but a cure to that problem. For example, a metadata attribute of "reliability" which is used in a number of very credible organizations is quite effective in measuring the trustworthiness of the source of information. Of course, in some cases, capturing lineage can replace additional metadata attributes for judging reliability.

Daconta's Counterpoint #1. Reliable data and metadata is properly attributed.

Next time we will examine Straw-man #2...

Geospatial Metadata

2006-11-05T16:19:00.000-05:00

Ok, moving on from the application metadata example (by the way, that was very helpful in ferreting out the principles of metadata design) - I am now researching interrogative metadata (who, what, when, where, why). The basic principle here is that the interrogatives are powerful vertices or axes to describing our data assets. In that, I am starting with "where" and geospatial metadata. Of course, there is a wealth of information on creating and using geospatial metadata. The Federal Geospatial Data Committee has a good site here. Also, geospatial one stop has some good pages on metadata creation.
More on this soon...

Application Metadata and an Ontology

2006-10-01T21:53:00.000-04:00

In ferreting the facets of metadata design, I am using "Application Metadata" as a test case. Part of the metadata for an application would be one or more categorization schemes. Here is the start of a categorization scheme for the "purpose" attribute in our application metadata. This simple taxonomy/ontology (created using Protege from Stanford) was developed via a simple categorization exercise on the applications in my start menu on this laptop. The purpose of formalizing this taxonomy was to make it a "formal taxonomy" as expressed in my article on XML.com.
There are some weaknesses in this taxonomy like the "Utilities" class because it is a generic grouping where its subclasses could easily intersect with other classes (which would not work if you wanted each subclass to be disjoint).
This was a useful exercise that did help me as part of my research into metadata design and the facets of a good metadata record. If you have ideas on that let me know or post comments here. Examples of metadata facets are identification, static measurement, dynamic measurement, categorization, etc.
Let's dig in and work out the details of metadata design! More soon...

Solving the All-Data Problem #1: The Data Optimization Pyramid

2006-09-16T16:31:00.000-04:00

The figure on the right depicts the "data optimization pyramid". This is the key scoping strategy that all organizations should use to apply reasonable management techniques to their data by first realizing that "all data is not equal". This means that you do not apply management techniques to all your data because not all of your data is meaningful or relevant. The key is developing an understanding and management strategy to allow relevant information to rise to the top. This may mean that you apply some brute force techniques to the Unmanaged data (like unstructured data) in order to assist you in the "bubbling up" process. But on the other side of the coin, labor-intensive tasks like tagging are reserved for a smaller subset of the data. Ok, you should have noticed that the pyramid is not complete ... here is where I need your help to find a good name for the top tier. At the top is critical data that you want to automate rules against, require a guaranteed level of fidelity or is highly relevant to a particular high-value ad-hoc community of interest. So, what do you think that top-tier should be called?
Some options:

Augmented Data
Refined Data
Formal Data
Critical Data

Comments very welcome ...

Gargoyles and the All-Data Problem

2006-08-30T22:09:00.000-04:00

Overly-simplifying assumptions and sweeping generalizations are the kiss of death for effecting change. In the data space, we have universal problems like information overload that are massive in scale. Thus the laissez-faire folks (like Cory Doctorow and Clay Shirky) seem to basically assume that the scope is just too large for any meaningful top-down solution.
Let's take a brief diversion here with an amusing, fictional analogy. In the fictional novel, Snow Crash, by Neal Stephenson there are characters called "Gargoyles" (as I attempted to sketch here) that are derided by the protagonist as goons because they attempt to collect any and all data in their vicinity with the hope something is of value. The derision is chiefly aimed at the brute-force strategy of collecting everything. The protagonist's derision is attacking the lack of discrimination in the method. I call this very lack of discrimination - the "All Data Problem". In fact, I would assert that the laissez-faire camp make this same mistake.
The "All Data Problem" makes a blanket assumption that our solution must be applied to all data and therefore will run headlong into failure due to massive scope. But what must be understood clearly is that the "All-Data Problem" is a flawed argument for one simple reason:
All Data is Not Equal.
Thus, for example, all data does not require metadata... all data does not require formal semantic models ... it is foolhardy to apply web-based principles applicable to the internet as a whole to an organization's data.
All Data is Not Equal.
I have a solution to the "All-Data Problem" that I will demonstrate over the next few posts. First, let's examine a precedent... In the National Information Exchange Model, I established the following scoping principle as shown in the diagram on the right. Universal entities were shared among all domains, common entities were shared among two or more domains and domain-specific entities were only used within their domain. Each type of entity required different degrees of harmonization and governance.
All Data is Not Equal...

Semantic Metadata Management and SOA Metadata

2006-08-25T12:37:00.000-04:00

Several interesting acquisitions have occured in the intersection of the metadata registry and SOA spaces. BEA acquired flashline and WebMethods acquired Cerebra. Having known Jeff Pollock for a few years, I am glad to see a semantic web company seen as a valuable commodity. Jeff is a fellow John Wiley author who co-authored the book (which I recommend as a good read) entitled Adaptive Information.

What does this mean? Primarily two things - first, it is a good example of Combinatorial experimentation which I discuss in my Semantic Web book as a brute-force approach to experimentation (fueled by the internet) where we continually combine things to search for the right combination which leads to productivity leaps.

Of course, I am not saying that combinatorial experimentation is the first or only reason why these acquisitions were made ... BEA and WebMethods are better businesses than that. However, you create new combinations when the current approach is not as effective as it should be.

Thus, the realization that semantics and metadata are critical to implementing the SOA vision.

Metadata design

2006-08-19T11:46:00.000-04:00

Currently working on a demonstration of the principles of metadata design. I am considering a resource type of which to use as an example. Please comment if you have suggestions. I want to use something that has not been done many times over (like music). My current thinking is to do metadata design on desktop applications. One ramification of this would be a requirement to change the definition of metadata from just a description of data to a description of an electronic resource (as clearly both applications and services would not be considered data).
Your thoughts and suggestions are welcome...

Metadata Definition Tweak

2006-07-30T09:42:00.000-04:00

I have slightly modified my Re-definition of metadata with one small but important change. Here is the latest:

Metadata – an external description of a distinct data resource. Common usages for metadata include providing the context of the data resource, managing its lifecycle and extending it to new uses. An example of metadata is the external description of an audio file specifying the artist that created it, when it was created, the length or play time, and the genre of music it belongs to. The purpose of metadata is to manage and improve the use of data thereby turning it into a strategic information asset.

The change comes at the end in the purpose statement for metadata. The statement makes an assertion that metadata is critical to transform data into information. I have heard information defined as "data in context". While that is certainly not a complete definition, it makes sense to think of information as a "refinement of data" much in the same way that raw materials get refined into finished products.

A Call to Arms for Digital Photo Metadata

2006-07-21T19:50:00.000-04:00

Embedded Image metadata is hot! This is the second time this week, people have raised this issue to me.
First, Clay Robinson of OSD informed me about a successful pilot he led on searching embedded photo metadata in JPEGs using a Google appliance. His pilot took advantage of metadata defined in the IPTC Core for XMP.
Secondly, I see that the Stock Artists Alliance has just issued a "Metadata Manifesto".
Here is a snippet:

Imagine a world where metadata is ubiquitous. It’s a world where images can be easily located
and identified by anyone, anywhere. Creators can transmit their images to distributors and users,
who instantly integrate these into their systems. Image users can track their digital assets using
fully automated systems. A registry—now in development by the Picture Licensing Universal
System (PLUS)—will link every image to current information about its source and owner.
To realize that future, we propose three guiding principles as our “metadata manifesto.”

Metadata is essential to identify and track digital images.
Ownership metadata must never be removed.
Metadata must be written in formats that are understood by all.

The full manifesto can be downloaded here and it is well worth the read.

Realtors, Metadata and gaining Public Consciousness

2006-07-03T14:04:00.000-04:00

Hi Everyone,
I was away on vacation last week to beautiful, keystone resort in Dillon, Colorado. Highly recommended vacation spot ... everyday was a new adventure in whitewater rafting, mountain biking, downhill biking, and hiking... Ok, back to metadata.

Spoke with my realtor, Glenn McFeeters, this weekend who works for Coldwell Banker. After discussing my kitchen remodeling, Glenn told me that he was surprised that before he met me he never heard of metadata but now he sees it and hears it mentioned all the time. He said, whatever I was doing, it was definitely working.

Of course, I certainly cannot take credit for this myself. I do agree with him that "metadata" is "crossing the chasm" from early adopters to mainstream acceptance. Part of this is the broader notion of data management now having successfully crossed that chasm as a necessary component of every CIO's responsibility. The management of data and the collection of proper metadata to insure that data is well-managed is now understood as critical to insure data integrity, security and quality.

I encourage all my colleagues to continue educating the public on this. In fact, I closed the conversation with Glenn by stressing the importance of laypeople's understanding of metadata. Because a layperson understanding the benefits of metadata and requesting such from their Information Technology vendors moves markets much faster than technologists preaching to the choir. The boulder is cresting the hill ... one final push should send it careening forward...

The New Face of Information Sharing

2006-06-05T21:28:00.000-04:00

I recently wrote this Op/Ed piece for Government Computer News on Information Sharing. While the article focuses more on an analogy about the tipping point for information sharing it does highlight the role of the FEA Data Reference Model in supporting communities of interest (COI).

For those new to the FEA DRM, it is an architectural metamodel to be used as a pattern for assessing and optimizing agency data architectures. In the article, I argue that it is a key element in an agency's preparedness to support an ad-hoc COI.

The Data Management/Discovery Connection

2006-05-31T21:50:00.000-04:00

Jim Rapoza of eWeek has a great article on how desktop search is not as effective as his metadata-enhanced MP3 Collection. To quote, "Because of the way that music ripping and creation encourages proper tagging, it's very easy for me to find and sort music based on, for example, artist and musical style."
Thus, my favorite example of metadata (music metadata) strikes again and continues to make headway into the public consciousness...

Metadata Definition: Reloaded

2006-05-27T11:51:00.000-04:00

Here is an improved attempt at a metadata definition:

Metadata – an external description of a distinct data resource. Common usages for metadata include providing the context of the data resource, managing its lifecycle and extending it to new uses. An example of metadata is the external description of an audio file specifying the artist that created it, when it was created, the length or play time, and the genre of music it belongs to. The purpose of metadata is to manage and improve the use of data as a strategic asset.

I believe this is an improved version that fixes the "data about data" problem, avoids the laundry list problem while giving some usage examples and closes with a clear statement of purpose. Next steps will be to apply the definition and drill down into the common usages I state in the definition: data asset context, data asset lifecycle management, and data asset extension (or repurposing). Many of these will be tested in a new contract we just won to provide data management support (more on that after the press release).

Content Metadata (aka "Content Tagging")

2006-05-18T14:18:00.000-04:00

A good article on XML.com discussing the benefit of adding metadata tags in the content of news stories.
While, in general, I agree with the author's approach. There are some difficulties and some push back in content tagging. In fact, I have personally been in numerous briefings where content tagging was bashed as a pipe dream, overly burdensome for content creators, not worth the cost and several other criticisms. That is why I have favored a hybrid strategy where you combine an entity extractor for universal attributes and focus the content creators on more domain specific tagging.
That brings us to the difficulty of knowing what to tag. As entity extractors improve we will see them evolve beyond who, what, when and where to ontology-driven extractors that can focus on domain-specific knowledge.
I believe that with sites like del.icio.us we are seeing the public accept tagging as a natural activity and therefore the search engine vendors should be keenly aware of this and begin exploiting it to improve the precision of results. To even spur the common adoption of tagging content creation tools should do a better job of quantifying the utility of adding tagging to a user by providing some kind of statistical side panel on related documents as you add more tags. In other words, something that immediately allows the end-user to see the benefit that adding content metadata has on the ability for consumers to find their information once they publish it in the organization.

Metadata in URIs

2006-05-13T09:48:00.000-04:00

An important W3C Technical Architecture Group (TAG) finding has just come out entitled "Metadata in URIs". For those who have heard some of my presentations, you know that I am in favor of "semantic identifiers" in support of the notion of "semantic chains". My justification for both ideas is based upon the understanding that "semantics are infinite" and that applications need "just enough semantics" to get the job done. Given that it is crucial to deliver semantics in small quantities at the closest point to the user in conjunction with a link back to a larger pool of metadata. The document has a great example about URIs used in the public that I feel highlights the point well. Highly recommended reading...

Army CIO on Metadata

2006-05-03T09:34:00.000-04:00

I met Lieutenant General Steven Boutelle in August 2004 when I was speaking at the Army Knowledge Management conference in florida. He had read and liked my book on the semantic web. He was kind enough to actually kick-off my talk by exhorting the packed audience to listen carfefully to what I had to say. An article came out today where he discusses his current priorities and metadata ranks right up there! It is clear from the article that he both understands metadata (specifically his comments about authoritative data sources) and how critical it is in an SOA environment. Makes me proud to be a former Army Intelligence Officer... ;^)

My Favorite Metadata Example

2006-05-02T21:58:00.000-04:00

In my opinion, the best and most intuitive example of metadata is digital music. I recently wrote a song about the unique name of my company, Oberon Associates, Inc. Here is the Dublin Core attributes of the song:

Title: O' Oberon
Creator (author): Michael C. Daconta
Subject or keywords: song,"Oberon Associates, Inc.",company,tribute
Description: A song about the company I work for, Oberon Associates, Inc., and the varying semantics of the word "Oberon". Specifically there are 6 references to the word Oberon and 2 literary references.
Date: 2006-05-03
Type: audio/mpeg
DCMI Type: [None]CollectionDatasetEventImageInteractive ResourcePhysical ObjectServiceSoftwareSoundText
Format: audio/mpeg 2446489 bytes
Identifier: http://www.daconta.us/Media/oberon-song.mp3
Language: en
Coverage: http://purl.org/dc/dcmitype/Sound
Rights management: (c) 2006 Michael C. Daconta
(Note: if you want the lyrics or chords to the song, you can get see them www.daconta.us).

You could enter these into dcdot to create an RDF version of the metadata. This song has 6 references to the semantics of Oberon and 2 literary references. In later posts, I will explore those semantics by creating richer metadata descriptions. Without the metadata, this song's value is almost counter-productive because people are very sensitive to their music. Music is powerful because it elicits emotion. Try randomly blaring a piece of music to someone you know nothing about and see what kind of reaction you get. Most of the time it will be negative. People strongly rely on music metadata (genre, artist, decade, etc.) to guide them to the right music. Metadata is all about guiding you to the "right" resource.

United 93, Information Sharing and Metadata

2006-04-30T20:32:00.000-04:00

Saw the movie United 93 tonight. Amazing and very intense. I cannot stress enough that it does not disrespect the victims of 9-11. It is also not a political movie. It does not take sides. It's purpose is to first give you the context and then put you in a seat with the heroic passengers of that tragic flight. It succeeds.

Being an information technology professional and the former metadata program manager for the department of homeland security ... I have to admit to you that I left the movie theater very frustrated and pissed off. The movie makes it very plain to see the ramifications of conflicting information, vague information and too much information -- in a crisis the manual systems are too slow!!! What else is it going to take for this to sink in? That brings us squarely into the realm of metadata -- metadata is not confusing or complex. It is contextual information on your data like lineage (do you trust this data?), security (should you see this?), privacy (does this violate a citizen's reasonable expectation for "personal space") efficiency (what is the cycle time from publishing to discovery and use?) and summarization (this is about subject 'x' and it has geographical information so it can be plotted on a map). You cannot have information sharing without metadata. Period.

We solved this problem! The Data Reference Model (DRM) clearly spells out how to assess, organize and create a data architecture that supports information sharing. And yes, it does require a metadata registry/repository. It does require data stores to be treated as "ENTERPRISE ASSETS". Before I left DHS, there was a strategic memo at the very top tier of DHS that stated this and that policy memo was meeting resistance by some components. That is just plain wrong. I sincerely hope that situation has been fixed with the new policy organization that was stood up under the Second Stage Review (2SR).

But there is plenty of blame to go around. Congress did not fund the Metadata Center of Excellence. Period. Why? The unofficial answer was that we needed to leverage the component assets better? Huh? That answer makes no sense because metadata and data standards must be centralized -- that is the definition of a standard -- a single source of truth! How do we fix this? Congress must be better educated as to the purpose and utility of metadata. This means that us technical weanies have to stop bickering amongst ourselves (specifically Clay Shirky's "metacrap" -- I will debunk this article in a series of later posts. Let me clearly state here that his arguments are faulty and specious). There is a technical camp that preaches "google as a silver bullet" as to why we don't need metadata or any form of data tagging. I hope google with their "do no evil" mantra will get off the fence and come down on the right side of this issue. Of course, the implicit metadata that they use in their page-rank algorithm sure as heck is metadata. It is just metadata created by others and not by themselves. Secondly, their ad-sense program definitely uses an ontology. So, anyone that thinks Google is a testament against metadata does not understand Google.

Since, I am doing a bit of bomb-throwing here; let me also state plainly that Microsoft needs to support the Web Ontology Language (OWL) and stop trying to only support things it can control (I will blog more on the details of this later). WinFS without OWL is a poor copy-cat of Apple's spotlight. When will they learn that such a machiavellian attitude is a losing long-term solution. Sure, you can get some points in the short run. You can even become the richest person in the world. But you won't build anything that lasts. You won't inspire anyone. You won't ever be a company that garners respect... those things last.

Let me leave this entry on a bright note -- the FAA has an excellent data management program that includes both a Metadata Registry/Repository and good data policy. Finally, Ben Sliney (who played himself in the movie), is a credit to himself, the FAA and his Country. I especially loved the scene when he grounds all air traffic over the objection of "how much money it will cost". Some people DO GET IT...

The CIO and "Information as an Asset"

2006-04-30T11:59:00.000-04:00

Excellent audio cast on by Peter Burris on Information Management and specifically "information as an asset". Every CIO should listen to this. He stresses the role of mastering our metadata to accomplish the compliance requirements and to achieve information superiority.
Highly recommended listening!

Exciting Data Warehouse Conference

2006-04-18T17:25:00.000-04:00

The April 13 Data Warehouse Conference by the Data Management Forum was a very successful and exciting event!
I recommend everyone examine Bill Inmon's new DW2.0 (TM) concept that raises the profile of metadata in the data warehouse.
I gave a talk on how the FEA DRM standardizes the metadata of data architecture. I will post the brief shortly. Especially important is my recent work on the V0.3 version of the DRM XML Schema (both XSD and RDF compliant). It is important to remember that the XML for the FEA DRM is to enable DRM-compliant metadata registry/repositories to share DRM metadata and NOT a mandate to "XML-ify" everything in the universe (though I have been wrongfully accused of trying to do that).

The "disparate data problem": myth busters #1

2006-04-05T09:46:00.000-04:00

I recently received an interesting email trail on multiple government officials discussing how they are working on ... with varying degrees of success "the disparate data problem". Unfortunately, most engineers know this is not a technical problem... it is a cost/benefit problem. It is easy to expose data ... a simple combination of web services and a registry solves the problem. The real issues are what to expose, what to tag and who to burden with the responsibility. That's a management issue.

So, we need to keep the proper perspective... Disparate Data is not a problem... Understanding your business and mission well enough and how data supports that mission is the real issue. Here is an analogy I find useful: Data is to an organization's mission like combat support is to combat units. Data done right is a combat multiplier... but there are two things there that hold it back ... first, focusing too much on the combat and not on what supports the combat and secondly not having a clean methodology to see the line of sight between mission and data.

It is that line of sight process that the FEA DRM begins and which I am continuing to work on ...

Biometrics Metadata

2006-03-24T16:54:00.000-05:00

I have been swamped the last few days assisting our developers on the Biometrics Automated Toolset (BAT) debugging and redesigning a distributed matching engine. Besides the critical nature of a rapid solution to fix bugs in a fielded system, it was great to be back coding. Those who know me, know that I never pass up an opportunity to code.
That brings up the issue of Biometrics metadata and person-centric metadata. On the subject of Biometrics data, there are multiple standards defined by NIST on biometric data. As for biometrics metadata it typically falls into a few types:

person biographic information
contextual information like a match from a matching engine
contact information based on a screening

An important issue relating to biometrics is distinguishing between person and identity. I recently wrote an internal Oberon newsletter article that defined the two as follows:

Person — a unique human being whose uniqueness is verifiable via unmodifiable characteristics.
Identity — a set of human characteristics for asserting “person-hood”; often captured in credentials.

How you model these is critical and which one goes on a watchlist also has important ramifications to the actions that should be taken.

Biometrics metadata is one of my top candidates for important areas to focus on. Most people intuitively get the importance of data and metadata about people; however, as the person/identity issue points out ... we are still not modeling this category properly.

The Government Metadata Controversy

2006-03-20T10:49:00.000-05:00

At the end of last year, Government Computer News published the article, "Metadata Not Essential For Search", based on the results of a Request for Information asking about "Efficient and Effective Information Sharing".

The report said that the results "overwhelmingly" support the hypothesis that “..for the majority of government information, exposing it to indexing with commercial search technology is sufficient to meet the information categorization, dissemination, and sharing needs of the public and as required by law and policy.”

Of course, the word "overwhelmingly" was used because 56% of the respondents supported the hypothesis.. One respondent stated that "Search technology has progressed far enough so that manual categorization and metadata tagging of textural documents is no longer necessary and any perceived gain in accessibility does not justify the cost of categorization."

This is the "Google as a Silver Bullet" argument, but brings about some interesting thoughts.

Precise vs. "Good Enough" Results - Can or should the government settle for "good enough" results when indexing their non-marked-up data with COTS tools out of the box?

Availability of Data - The metadata markup process (when it is not automated) may limit how soon that data is available. This is an issue that I think the government is painfully aware of.

Hybrid Solutions? A Focus on Rules for Pattern Recognition, Auto-Tagging? - This is what I call "guess metadata" (metadata that is determined by a computer process, and not a man-in-the-loop). One thing that the report didn't really focus on was that much time and effort is spent defining pattern recognition rules for concepts/keywords for automated markup of metadata in the search engine indexing process. These rules are utilized at indexing time, and increase the likelihood of good results vs. a "search engine out of the box" solution.

From a business perspective, looking at the results of the respondents in this paper, we do need to recognize the need to get the data out faster with minimal impact to organizations making data available. In doing so, we can focus on rules and a pattern-recognition process at pre-indexing time. At indexing time, this process can tag the data with agreed-upon metadata standards, even tying elements of the data to classification taxonomies and ontologies. Of course, IMHO, any automated process is still "guess metadata". Is "guess metadata" good enough?

Away for a few days

2006-03-19T16:59:00.000-05:00

Been away down to South Carolina... Updated the Metadata Types page with a few more entries. Also working on the next version of the definition which I will post soon.

Blog Project #2: Metadata Types

2006-03-14T21:43:00.000-05:00

We have been working through redefining metadata, which is the first project for this blog. Though we are not nearly done, it is time to introduce a related project to catalog all the various types of metadata. I began a list tonight and posted it here.
What do you think are the key types of metadata?
Do you agree with the "traditional types" like structural, descriptive and administrative metadata?
How do these types square up against our definition?
Personally, I have a problem with structural metadata as I think it blurs the line between data and metadata which I believe is harmful.
Getting a good authoritative set of metadata types (and their definitions) will help us understand and explain the purpose of metadata. From the looks of it, I think we will have an interesting time winnowing down the list!
Another interesting side effect of collecting this list will be an interesting snapshot of our current understanding (or lack thereof) of just what constitutes metadata.

Strawman definition and Open XMP

2006-03-13T21:18:00.000-05:00

Here is my first cut at a strawman definition:
● Metadata – an external description of a distinct data resource to provide context, metrics or amplification.

Let me know what you think...
On another note, Adobe announced Open XMP. I am looking forward to checking this out. It is good to see Adobe take a leadership role in this space. Note: XMP uses the W3C RDF specification.

Speaking on April 13, 2006 on Information Sharing

2006-03-08T08:59:00.000-05:00

Visit dmforum for information on my upcoming presentation on information sharing. Bill Inmon and Dick Burk are also speaking. My presentation will focus on Information Sharing and specifically the metadata of the Federal Enterprise Architecture Data Reference Model. Over the next few months I want to refine that metadata and even work on the next version of the XML Schema. If you would like to work with me on this, feel free to email me at my oberon associates address.

Hysteresis, Cost and "About-ness"

2006-03-07T12:03:00.000-05:00

An article from Sean McGrath on Metadata asserts that there is a natural delay between creating information resources and understanding them. This natural delay is called hysteresis. His solution to this problem is to let creators create content and categorizers (or metadata experts) add the metadata at a later time. I disagree with this as a general principle because it increases the cost side of the metadata equation without being able to concretely define the commensurate value received. In an ideal world, the majority of metadata can be added at the point of creation with the flexibility to add additional metadata afterwards. I previously worked on a project called the Virtual Knowledge Base where we devised a scheme to enable multiple layers of metadata where we differentiatied between machine-generated versus human-generated metadata (and allowed multiple layers of both).
Our key takeaway principle is this: Don't put the metadata cart before the business value horse.
Since metadata is extrinsic information, it is either automated (ideal solution and zero cost), semi-automated and done at the point of creation (small cost) or worth the cost of adding it after the fact. The last option is the last one for good reason.
The article uses the term "about-ness" to describe metadata and I think this is a superb, though made-up, word. Let's add that one to key terms in our new definition of metadata. Our current list of terms for our new definition of metadata are:

description
context
"about-ness"

The Good, the Bad and the Ugly of Administrative Metadata

2006-03-06T17:41:00.000-05:00

Administrative metadata is descriptive information about the creating and managing of information assets like author, creation date, etc. The most well known standard for tracking this type of information is the dublin core metadata initiative.
The good news is that this type of metadata is well-understood, fairly easy to implement and even partially supported in most editing applications like MS Word, Photoshop, etc.
The bad news is that this type of metadata is so ubiquitous that it is intimidating to manage and therefore seen by many managers as overkill. Think of the number of records required in a metadata repository to catalog all the administrative metadata on every document, diagram, photo and report created in your organization. It is a scary proposition for large organizations and if you cannot cleary define the benefit gained from such management, such an undertaking is dead-on-arrival. So, unless you control the digital production process (and some organization's do) where this step can be automated or semi-automated, I do not recommend you try to capture this metadata. That brings us to the ugly...
Several recent stories (here and here), have highlighted the danger of unmanaged and sometimes hidden metadata in photos and documents that embarrass the organizations that created them. Does this mean we are back to recommending storage of all administrative metadata to eliminate such gaffes? No. It means that organization's do need to be aware of the metadata that editing programs store and then determine which digital data are important enough to require control of the production process.
A good resource would be a listing of all the metadata stored by current popular editing tools... if you know of such a resource (or have time to develop one), please let me know.

"data about data" considered harmful!

2006-03-05T11:55:00.000-05:00

The classical definition of metadata is what I refer to as the "geek definition" because it includes a nice double entendre with its recursiveness. Unfortunately, this definition is like nails on a chalk board to business managers. Business managers don't want the clever definition... they don't want to know how it is cool to be both metadata about some other data and data at the same time which can have metadata pointing to it which can then be data for more metadata pointing to it, on and on ... until their eyes glaze over. In essence, that is a technically accurate but practically foolish definition. I am working on a better definition and your feedback is welcome.

Here is my initial requirement for a new definition:

Clearly distinct from data. Metadata is NOT the same as data and you must clearly differentiate between the two.

Let's begin the work of fixing this mess... and please, hold the cleverness.

Story clarification

2006-03-04T10:15:00.000-05:00

Had dinner on Thursday with two good friends, Kevin Smith and Danny Proko. At dinner, Kevin said that he had read an article on me and that I had "slammed my CIO" (CIO is Chief Information Officer). Being that Kevin is a really smart guy, if he misread the article entitled "Metadata Dreams Adrift", than I am sure many others have also.

Of course, the misunderstanding comes from the pullout quote where I say, "The Homeland Security Department’s cross-cutting mission will not be successful without cross-cutting IT. ... the way you get that is with a strong CIO". Of course, that is only part of the story. So, let me add the 'rest of the story' here to clear this up. First, I was absolutely NOT slamming my CIO. The comment was made about the position of CIO and not the person who is the CIO. I was asserting, as many others have asserted, that in order for DHS to be successfully integrated, in this case specifically Information Technology integration, the CIO should have the component CIOs directly reporting to him in order to successfully carry-out the numerous cross-cutting missions of the department. While that decision is way above my former paygrade, I said it because I believe that is what it will take to be effective. But don't get me wrong... I am proud of the work DHS is currently doing and the fine group of people working hard under extreme pressure and public scrutiny.

At a later time, I will address the reasons why I left DHS in more detail ...

Hope that clears things up... for those interested... In this blog, I will be dissecting the role of metadata within the federal goverment and what changes need to be made, both in industry and goverment, to have it fulfill its promise. Believe me, there are many, many changes that need to be made and us techies are a big part of the problem. The way we explain metadata and the current raft of products that support it lack the necessary focus and value proposition to go mainstream. I hope to change that.

- Mike

What is this?

2006-03-03T13:10:00.000-05:00

Hi Everyone,

This will be a blog on my personal experiences and strategic analysis of the art and science of designing metadata to productize and manage are vital data assets.

- Mike