Composing the Semantic Web

Validating schema.org Microdata with SPIN

2011-07-05T10:57:00.000-07:00

The new 3.5.1 version of TopBraid Composer introduces some initial features to import, browse, edit and analyze Microdata. I wrote about this in a previous blog entry - if you want to try those features just download TBC's evaluation version, keeping in mind that Microdata support is still at an early stage and that, for example, the parser isn't fast yet. Today I will focus on a SPARQL-based approach for validating schema.org Microdata using SPIN inside of TopBraid.

I have published a library of SPIN constraints at http://topbraid.org/spin/schemaspin. This library currently includes 11 types of integrity constraints covering various aspects on the schema.org ontology, as shown below.

The most basic tests are making sure that schema.org properties can only be used at classes with matching domains, and that the values of those properties match their declared ranges. These tests alone may help you identify many potential errors at edit time. Another generic test is associated with owl:IrreflexiveProperty, making sure that a value of children, colleagues, follows, knows etc doesn't point to itself.

Other tests check for various family relationships (e.g. children must be born after their parents, children cannot contain cycles and birthDate must be before deathDate), emails must match certain regular expressions, and the ranges of longitudes and latitudes. The following TBC screenshot shows an example of an invalid longitude:

A really powerful demonstration of ontology reuse and linkage is the constraint that validates currency codes. There is a finite vocabulary of those, including EUR and USD. The schemaspin ontology simply imports the QUDT namespace that already defines all of those abbreviations, and reports an error if an unknown currency is used on a Microdata page. The following screenshot shows the underlying SPIN constraint (note that this spin:constraint is attached to a property metaclass that marks all properties holding currencies as values):

The constraints above are just a beginning. Much more interesting constraints could be defined if more data is published on the Semantic Web, e.g. by asking the Sindice SPARQL end point with the SPARQL SERVICE keyword to validate that a link to a Person really describes a known http://schema.org/Person, or to compare prices to make sure that my offering of a product is currently the lowest price on the market. The open architecture of SPIN and the richness of SPARQL makes adding these and domain-specific constraints easy and enjoyable.

SPARQL Web Pages made easy

2011-07-03T16:25:00.000-07:00

SPARQL Web Pages (SWP, aka UISPIN) is a templating language for HTML and XML formats that operate on RDF data. In a nutshell, SWP makes it possible to embed SPARQL expressions and queries directly into web page snippets, and to link an RDF or OWL ontology with such SWP snippets. SWP can also be used to generate JSON callback results to support Ajax-style patterns. This basically means that application developers can cover the whole software stack ranging from model to control to view with RDF-based representations only.

At TopQuadrant, we have meanwhile made substantial use of SPARQL Web Pages in internal and customer-facing projects, and have introduced several SWP enhancements with TopBraid 3.5. One thing that several people asked for was SWP support for stand-alone web pages that are not necessarily linked to specific classes in an ontology. In response to this, we have introduced *.swp files, which can be used like PHP or JSP documents with a TopBraid server.

In order to create such SWP files, go to File > New > SPARQL Web Pages file. This will create a stub file with some content that will help you get started:

Through its base platform Eclipse, TopBraid Composer includes a powerful HTML editor, and TBC 3.5.1 includes syntax highlighting for SWP built-ins (see above, you may need to associate *.swp files with the HTML content type in the Eclipse preferences as described in the TBC Help).

As soon as you have created this file, you can immediately execute it via the personal TopBraid Live server that is built into TopBraid Composer. Just visit http://localhost:8083/tbl/test.swp in your browser:

The example above takes one argument (test) and inserts this into the greeting. The expression {= ?test } will insert the current value of the SPARQL variable ?test into the output document. In our example, ?test is fetched from the URL arguments via the built-in function ui:param(). The demo page then creates a simple loop over all instances of kennedys:Person in the query graph and inserts them into an unordered HTML list. The actual query graph is specified using the ui:setContext tag - if this isn't present it will use the default graph.

For a more complete live demo of SWP, visit http://spinservices.org:8080/spin/doc.swp providing documentation of the GoodRelations ontology:

For anyone with experience in hand-editing HTML and JSP or PHP, SPARQL Web Pages should look quite familiar. In fact, SWP borrows ideas from other well-known languages such as loops, assignments, if-then-else branching and user-defined tags, but is 100% SPARQL. With RDF nodes as first-class citizens, this language is IMHO an attractive alternative to projects that use RDF as their primary data representation or integration format.

Microdata and RDFa in TopBraid Composer

2011-06-09T00:24:00.000-07:00

The next release of TopBraid Composer will include comprehensive support for editing and processing schema.org Microdata, and will also have improved support for RDFa. TopBraid is an extension of Eclipse and thus inherits a lot of goodness from the platform, including a very nice HTML editor. It was straight-forward and highly desirable to extend TopBraid with native support for those Web Data formats. Here is a preview of what it will look like.

Working with Microdata and RDFa

When I started exploring Microdata for my own web site, I created a new Eclipse project within TopBraid Composer containing the HTML, CSS and image files for the site.

While I was adding the Microdata tags to the HTML documents, I quickly discovered that RDF based tooling can be extremely helpful to make sure that the published metadata is consistent and of good quality. For example, data about entities (such as the http://schema.org/Person about myself) is split across multiple HTML pages: the front page contains my address, but my personal page contains information about my children. In such cases it is important that both pages use the same identifiers for the same Linked Data entities. This becomes even more important if we want to link to external standard vocabularies, such as ontologies about units, countries or product categories.

Linked Web Data is much more useful than isolated data snippets on individual pages.

As a result of this, I introduced the notion of Web Data Sites into TopBraid Composer - collections of pages in the same folder and its sub-folders. Right click on the project above and select New > Microdata Site File (or RDFa Site File). This opens a wizard with an option for default ontologies to include. For Microdata this is obviously the schema.org namespace, but any other RDF vocabulary can be added later:

This creates a site file (*.mds) that acts as a placeholder for all RDF triples on the HTML pages within the same folder and its subfolders. The site file can be opened like any other RDF data source, it can be imported into other data models, etc. When opened, it will scan the HTML files and always automatically stay up to date when the data on the HTML is changed.

The screenshot below (click on the image for the full size) shows some of the new TBC capabilities in practice.

You can see that TopBraid has built-in views to browse the class hierarchy, properties and instances. These are powerful mechanisms to navigate through the data space that is encoded in the HTML pages. In the example above, you can see that my current Microdata pages contain information about three Persons, as well as various address and location objects. The class tree shows the number of instances of each class. A double-click on an instance will display it on a form. You can see the form view of the resource http://knublauch.com (representing myself as a schema:Person) on the right. Here is a larger view, with the details of one of the children objects opened up:

Alternative views such as graphs and smart browser displays are also built-in. Here is a TBC graph view of some instances:

Analyzing Web Data with SPARQL and SPIN

You can also run SPARQL queries over this data:

We have a lot of SPARQL-based features built into the TopBraid platform, including the rule and constraint language SPIN (now a W3C Member Submission). SPIN is useful to define model-based integrity constraints, and I have started to create a SPIN constraints library for the schema.org namespace. Currently this checks that the value type of properties on the HTML pages matches the range defined by the ontology, but more checks will be added, for example regular expressions of emails, country abbreviations etc. More on this in a separate entry some day.

Editing Microdata and RDFa

Once you have checked constraints and the system reports a violation, you can navigate to the source of the violation on the form of the relevant instance. From those forms, you simply need to double-click on the icon to the left of the value to navigate to the HTML source code:

At this stage, the circle is completed and are in HTML document where you can fix problems (e.g. a misspelled email address). Save the HTML file, and the RDF triple (on the form and elsewhere) will update automatically.

The HTML editor in TopBraid Composer has been enhanced with syntax highlighting for the Microdata attributes such as itemprop. And more is on its way...

Harvesting Microdata and RDFa from the web

In addition to editing and processing local Web Data files, TopBraid can also be used to work with external mark-up from existing pages. TBC Version 3.5 had already introduced the Web Data Basket, and we have extended this to also support Microdata. The mechanism is simple yet powerful: you install a small Firefox extension that will send the pages you visit to your locally running TopBraid Composer. This will collect all RDF metadata contained on the visited pages, and make it available to the RDF, OWL and SPARQL machinery of TBC. This means you can simply browse the web and you will automatically get the stream of RDF triples into your working environment.

Faceted Search with TopBraid and SWP

2011-04-26T22:41:00.000-07:00

Many Semantic Technology companies offer some kind of faceted browsing tool. With TopBraid 3.5 it was time for TopQuadrant to say "me too", and add some unique capabilities into the mix.

The main idea of faceted browsing is to allow users to narrow down a set of objects by selecting properties that the sought-after objects must possess. For example, if you search for people in the infamous Kennedy ontology, you may want to find all instances of Person that went to the same university and share the same profession. TopBraid's faceted search component follows a user interface paradigm made popular by FreeBase Parallax: you start with a set of all Persons and the system will compute how many matches are in each category. Clicking on a category will narrow down the set, and you can add the next condition. The following screenshot illustrates this, with the facet "alma mater" narrowed down to "Harvard University".

TopBraid's faceted search is implemented by a collection of TopBraid Live servlets and a JavaScript UI library. What you see on the screen above is in fact a web browser embedded into TopBraid Composer. The default stylesheet is simple and can be customized, and it's also possible to use the same JavaScript library in completely different web applications.

One distinguishing capability of TopBraid's Faceted Search support is its customizability. SPARQL Web Pages (SWP) technology can be used to customize the visual appearance of the preview results on the right hand side. A key benefit of SWP is the linkage between ontologies and user interface snippets. Basically, SWP allows you to attach HTML snippets to any RDFS or OWL class in your domain model using the property ui:instanceView, and the system is then able to dynamically select the best suitable visualization for any object that it gets. For example, the visualization for kennedys:Person can be changed as shown below.

The faceted search component looks for visualizations marked with ui:id="facetSummary", and will display them as shown below.

Further customizations are possible without any programming: For example you can specify which properties shall be visible by default, and which properties shall not be selectable as facets.

There is more to be said about this new capability. But if you just want to get started, use TBC-ME 3.5, select the class that you want to search instances of, switch to the Browser tab and pick the facet.ui:SearchView view in the drop down list. Note that on Windows this currently does not work because Eclipse includes an outdated internal web browser. Please use the button Open current page in external browser. Like with any new feature, we appreciate your feedback.

TopBraid Composer's Web Data Basket: Collecting Linked Data while you browse

2011-04-26T22:10:00.000-07:00

One of the little new features in TopBraid Composer 3.5 is the Web Data Basket view. This can be used to incrementally download Linked Data (either RDFa or RDF) while browsing the web. The best way to experience this is by getting a small TBC Firefox extension. This will add a tiny TopBraid button to the lower right corner of your browser.

Click on this button while TopBraid Composer is executing, and all RDF data encoded on the currently visited page will be added to TBC's Web Data Basket:

While this Basket displays the raw triples, it also has options to add the loaded triples into the current model. For example, you will get a proper foaf:Person for David Bowie if you visit his DBpedia page:

In order to facilitate the use of this data, TopBraid Composer will automatically add missing imports to namespaces such as foaf and skos. When you follow a hyperlink in your web browser, the basket will get more content. This means that the system will accumulate any Linked Data into TopBraid as you navigate through the web.

This little Web Data Basket makes it easy to collect Linked Data without having to leave your favorite tools. I think it provides a fine example of how Linked Data could be used, e.g. to build up a shopping list of products backed with GoodRelations data.

SPINMap: SPARQL-based Ontology Mapping with a Graphical Notation

2011-04-21T16:28:00.000-07:00

One of the new features in the upcoming TopBraid 3.5 release is called SPINMap. SPINMap is a SPARQL-based language to represent mappings between RDF/OWL ontologies. These mappings can be used to transform instances of source classes into instances of target classes. This is a very common requirement to create Linked Data, for example starting with spreadsheets, XML files or databases, but also from one domain-specific ontology into a more generic one. As a first impression, here is a picture of SPINMap in action:

If you would like to learn about this with a visual demo, please take a look at the

SPINMap Tutorial Video

In the rest of this blog entry I will cover similar content to the video, but with screenshots and prose.

Introduction to SPINMap

SPARQL is a rich language that can be used for many purposes. The SPARQL CONSTRUCT keyword is particularly useful to define rules that map from one graph pattern (in the WHERE clause) to another graph pattern. This makes it possible to define sophisticated rules that map instances from one class to instances of another one.

The SPIN framework provides several mechanisms that make the definition of such SPARQL-based mapping rules easier. In particular, SPIN makes it easy to associate mapping rules with classes, and SPIN templates and functions can be exploited to define reusable building blocks for typical modeling patterns.

The SPINMap vocabulary (http://spinrdf.org/spinmap) is a collection of reusable design patterns that reflects typical best practices in ontology mapping. SPINMap models can be executed in conjunction with other SPARQL rules with any SPIN engine. The main advantage of SPINMap is that it provides a higher-level language that is suitable to be edited graphically. TopBraid Composer 3.5 provides a visual editor that makes it easy to establish ontology mappings using drag and drop, and filling in forms.

It is a good practice to store the ontology mapping rules in files separate from the source and target files. The mapping file only needs to import the SPINMap namespace (which in turn imports SPIN etc). The easiest way to get started is to use File > New > RDF/OWL/SPIN File... and then to activate the check box for "SPINMap Ontology Mapping Vocabulary", as shown below.

This will create an empty file importing http://topbraid.org/spin/spinmapl. As a next step, you should drag the source and target ontologies into the Imports view so that those get imported into the mapping ontology. Then select the class you want to start mapping, and switch to the Diagram tab. In the example below, the source ontology A defines a class a:Person, and we want to map it into the target class b:Customer.

Use drag and drop (e.g. from the Classes view) to add other classes to the Diagram. If the SPINMap namespace is present, the Diagram will provide additional capabilities and use a different layout algorithm than usual. If you move the mouse over a class, a triangular anchor point will appear in the upper right corner of the class box. It will turn green if you move the mouse over it, and if it can be made the source of a mapping. Click on this and keep the mouse button pressed to establish a link to another class. Move the mouse over the incoming upper anchor of the target class and release the mouse. A dialog like the one below will appear.

This dialog is used to create a "mapping context" that is later used to determine how the target instances shall be selected from the source instances. In particular this is used to construct URIs from the values of a given resource, e.g. so that a:Instance-0-1 is turned into b:John-Smith. The dialog provides a collection of target functions that can be used for that purpose. You simply need to pick an appropriate function and fill in the blanks to establish a mapping context. In the example screenshot, a new URI is constructed from the values of the source properties a:firstName and a:lastName and a provided URI template. This assumes that those properties together serve as unique identifiers, similar to primary keys in a database. Other algorithms can be created if needed through SPIN functions.

As soon as you have filled in all required arguments of the mapping context function, the preview panel of the dialog will give you an idea of how the resulting values will look like. When you are happy with this, press OK.

The resulting context will be displayed with a yellow graph node as shown below.

If you ever need to edit this context node again, e.g. to change the URI template, just double-click on it. Right-clicking the node opens a context menu with an option to delete it.

Once a context has been established between two classes, the user interface makes it possible to add transformations. In the example above, the source class has a property a:dob that holds date of birth values as raw strings, such as "30/04/1985". We want to map this into the target property b:birthDate, which is a well-formed xsd:date in the format "1985-04-30". TopBraid's SPARQL library provides a built-in function spif:parseDate to make this task easier. Use the mouse to draw a connection from a:dob to b:birthDate. A dialog such as the following will appear.

In this dialog you can either manually select a transformation function, or check if the system has any suggestions for you, on the Suggestions tab. In this case, the system suggests spif:parseDate with pre-defined patterns to convert raw dates into valid xsd:date literals. Pressing OK, this creates a mapping transformation as shown below.

At any point in time, TopBraid Composer makes it easy to try the mapping out. Assuming TopSPIN is the selected inference engine, just press the Run Inferences button in the main tool bar to see the results.

As you can see above, each instance of the a:Person class has been mapped into a corresponding instance of b:Customer. The URI of the target resources has been generated using the string insertion template based on first name and last name. Furthermore, proper birth dates have been generated from the raw source strings. The context menu of the Inferences view provides options to assert the resulting RDF triples if desired, or you can use the Triples View to move them elsewhere.

It is possible to add any number of other transformations in similar ways. Some transformations take more than one argument. In that case, additional input anchor points will be displayed, as shown for the node "concat with separator" below.

Note that a complex example like above uses a number of different design patterns. Some additional of those patterns are explained in the tutorial video, that I would strongly recommend if you want to save time with this technology.

Understanding and Extending SPINMap

The mini tutorial above might be enough for many users to get started. For advanced users with knowledge of SPIN, the following background may be helpful to understand how SPINMap works, and how it can be extended.

SPINMap is an entirely declarative application of SPIN. This means you can explore the mappings generated by the visual editor from an RDF perspective, e.g. using TBC forms. In the example above, the form for a:Person displays a collection of SPIN Template calls:

You can drill into the templates by opening up the + sign that appears when you hover the mouse over the template icon.

The example above illustrates that SPINMap is based on a (small) collection of generic templates, such as spinmap:Mapping-2-1 which represents a mapping from 2 source properties into 1 target property. Each of those templates a linked to a spinmap:Context which is used at execution time to determine the target URIs. Furthermore, the argument spinmap:expression points to a SPARQL expression, SELECT or ASK query, or even a constant URI or literal that is used to compute the target value from the source value(s). The SPINMap templates are using the function spin:evalto evaluate those expressions at execution time. When executed, the expression will be invoked with pre-assigned values for ?arg1, ?arg2 etc, based on the current values of spinmap:sourcePredicate1 on the source instances.

Since in practice any SPARQL function can be used as spinmap:expression, users can also add their own SPIN functions where appropriate. It is also possible to use the built-in SPARQL functions such as xsd:string().

The mapping context uses a similar mechanism, also based on spin:eval to create target URIs. You can open any instance of spinmap:Context to see how this is done.

In the example above, the target function spinmapl:buildURI2 is used to derive a new URI from two input properties and a template. You are free to define your own target functions there, as long as they are instances of spinmap:TargetFunction (and subclass of spinmap:TargetFunctions).

If you are writing your own functions, or want to make the system smarter, you can add your own spinmap:suggestionXY values to the functions. These are SPARQL CONSTRUCT queries that may construct zero or more instances of the function, with partially filled in fields, as well as a spinmap:suggestionScore. See the function spif:parseDate for an example of what can be done with this mechanism.

SPIN is a W3C Member Submission

2011-04-04T17:12:00.000-07:00

The SPARQL Rules language SPIN has evolved over the last couple of years as an integral part of TopQuadrant's TopBraid Suite. SPIN started during a discussion between Dean Allemang and myself, in which we brainstormed about having an RDF syntax for SPARQL. I went ahead and implemented this based on Jena's ARQ API, and the result eventually became the SPIN RDF Syntax. This was no rocket science, because similar ideas of representing higher level languages by means of RDF blank node structures had been explored by OWL and SWRL.

Prior to our work on SPIN, we had already experimented with various mechanisms to link SPARQL queries with RDF data structures, so that they could be shared as query libraries. TopBraid veterans may remember the sparql:query property that was introduced to store SPARQL queries (as strings) together with RDF models. So while I was working on the SPIN RDF Syntax, I noticed that we now have a much better way of achieving this goal. A quick cross-reference to object-oriented languages led to me select properties such as spin:rule and spin:constraint to point from a class to a SPARQL query, expressed in RDF. This later became the SPIN Modeling Vocabulary.

Once I had the rules and constraint mechanism in place, I noticed that many rules and constraints were following similar patterns, with just one or two values different in each rule. This led to the creation of SPIN Templates. Templates then became the foundation of user-defined SPIN Functions. With those two pieces in place, SPIN suddenly became a language that was fundamentally different (and better) than what similar languages such as SWRL provided, because it became possible for users to define their own modeling vocabulary, and even extend the expressivity of SPARQL.

The first version of SPIN was published as part of TopBraid Composer in January 2009. Since then, it was positively received by our user community and practical use cases have enabled us to fine tune and extend the language over the years. Now, around three years after its first experimental versions, we found the time was right to officially share SPIN with the broader community, and make clear that it is not a proprietary TopQuadrant technology. Together with James Hendler and Kingsley Idehen, we put together a SPIN W3C Member Submission that has just been published on the W3C site.

The status of a Member Submission means that TopQuadrant encourages other tool vendors to also provide SPIN implementations, and as I have heard there is work in progress already. The Member Submission also indicates that SPIN may play a role as input to future revisions of other standards such as RIF. This is all very good. Of course a full spec of SPIN as an official W3C standard would be even better, but going through the whole standardization process is a long and difficult journey. Given that SWRL had become a similar de-facto standard with Member Submission status alone indicates to me that SPIN has good chances of achieving the same. In fact I strongly believe that the fact that SPIN is based on SPARQL will be crucial in winning the hearts and minds of many Semantic Web and Linked Data enthusiasts. SPIN can co-exist with other languages including OWL 2 RL and SKOS. SPIN doesn't require any special execution engine apart from a SPARQL store. The learning curve is very low for anyone who already knows SPARQL. SPIN is part of the Semantic Web technology stack.

A good place to start learning SPIN is the TopBraid SPIN page, with screenshots and links to a tutorial. For programmers, there is an open source SPIN API available.

A Textual Syntax for SPARQLMotion

2011-01-25T17:36:00.000-08:00

SPARQLMotion is an RDF-based scripting language that is suitable to be presented and edited graphically to form data processing pipelines. Many of our customers are using SPARQLMotion and we are constantly extending and refining the tools to make it more powerful. One of the recent enhancements that made it into TopBraid 3.4 is support for an alternative textual notation for SPARQLMotion. A spec for this notation can be found here:

http://sparqlmotion.org/smXMLSyntax.html

An example of how this XML-based notation for SPARQLMotion can be used is shown in the TopBraid Composer screenshot's sm:bodyScript field below.

The script above is stored in the same RDF-based format like other SPARQLMotion scripts, and can still be visualized graphically:

In the past few years since SPARQLMotion was created, several people had asked about a notation that can be edited with conventional text editing tools to create scripts. Among the advantages of a text-based notation is that it becomes easier to perform large-scale refactorings to move things around. It is sometimes simply faster, plus there is no need to "invent" artificial URIs for the nodes in a script. A great plus of the XML-based notation is that it becomes easy to insert SPARQL Web Pages (aka UISPIN) snippets directly into a single document. This can significantly accelerate the development of SPARQL-based web services.

The XML notation does have various limitations though. In particular it is only suitable for a subset of SPARQLMotion - there is no concept of multiple predecessor nodes in a linear notation.

Understanding SPARQL Rules with the SPIN Statistics View

2011-01-10T18:59:00.000-08:00

One of the new features of TopBraid Composer 3.4 is a new view called SPIN Statistics. Whenever this view is open and you run some SPARQL Rules, this will record the execution time of each individual rule. When completed, you can browse the performance characteristics of each rule, grouped by invocations or by the associated class. In the following screenshot, the SPIN rules for OWL 2 RL executed over the pizza ontology are recorded:

According to the statistics in this screenshot, the rule that implements the transitivity of rdfs:subClassOf has taken the largest fraction of the time. This can help identify performance bottlenecks, and may also be useful to understand better what happens inside of the rule engine. The view can be filled incrementally, e.g. to accumulate how certain rules fire over different data sets.

The Teamwork Ontology

2010-12-13T17:44:00.000-08:00

The newly released TopBraid Suite 3.4 introduces the Enterprise Vocabulary Net (EVN), an out-of-the-box solution for web-based development and management of interconnected controlled vocabularies. The focus of this product for now is on collaborative editing of SKOS models, and the EVN link above will lead you to screenshots showing this in action. I should write a couple of blog entries about this, but here is one about one particular aspect of the EVN system - its support for keeping track of changes (on the SKOS models) in a multi-user environment.

The requirements of EVN include the ability of teams to collaborate on edits on an enterprise vocabulary. Imagine a media company wants to build a semantic web model of concepts that are relevant in its domain. This media company would have domain experts on the "news" channel, and for this news channel, a controlled vocabulary would be needed to be able to categorize incoming news items so that they can be processed more easily down the road. A simple SKOS hierarchy would define standard identifiers (URIs) for things like Sports, and under that there would be sub-concepts such as RacquetSports, with further specializations including Tennis and Badminton.

In order to build such a controlled vocabulary, the media company would create a team of domain experts, and each group of experts would collaborate independently to fill in the various sub-trees of the overall enterprise vocabulary. In the TopBraid EVN system, this is implemented through an editing process in which different people can play different roles. Assume the sports experts want to add another bunch of categories to classify the various kinds of football, then they would open a so-called "working copy". The working copy is a logical extension of the master model, but also includes local changes that are not yet visible to the rest of the team. The sports team can experiment with arbitrary edits in their working copy sandbox. When done, they can change the status of their working copy to trigger a review process. At this stage, no further edits will be done, until a reviewer (who "owns" the master copy) had a chance to OK or reject those edits. If OK, the working copy will be retired, and all changes will be applied to the master copy and thus published to the rest of the enterprise. If rejected, the sports people may want to make additional edits.

In order to support such workflows, we have designed an RDF based framework for tracking and managing changes on RDF models. This framework, internally called "teamworks support" is based on the teamwork ontology (http://topbraid.org/teamwork), an excerpt of which is outlined in the class diagram below, together with some technical details.

The teamwork ontology represents changes (teamwork:Change) made by users (sioc:UserAccount) on governed resources. The governed resources are either a whole model (the master copy, an instance of owl:Ontology), or working copies (teamwork:Tag). If a group of users wants to add a new category of sports concepts, then it would create a teamwork:Tag and give it a label such as "Add football sports". This will become the container of any number of smaller changes, each represented as instances of teamwork:Change, which are recorded together with a time stamp and creator. Each Change points to one or more added or deleted RDF triples, where the triples are stored as reified statements of the class teamwork:Statement. A Change can be associated with a working copy (teamwork:Tag) via the property teamwork:tag. The collection of Change objects associated to a Tag form a group of edits that can be tracked in the workflow using the property teamwork:status at the teamwork:Tag. Example status values are teamwork:Uncommitted, teamwork:FrozenForReview and teamwork:Rejected.

The teamwork user model manages user accounts as well as the roles that each user can play within a working copy and the master vocabulary. This is done through the sub-properties of teamwork:role: viewer (read-only), editor (write access) and manager (write access and workflow control).

The teamwork triples above are stored in separate graphs, called the teamwork graphs, that are linked to the edited vocabularies via a file ending with .tch.*. For example, the graph example.tdb may have a companion graph stored in example.tch.tdb. As soon as TopBraid finds a .tch file with a matching name, it will put the ontology under teamwork control, which means that changes will be automatically tracked whenever someone writes the the graph. Furthermore, all graphs under teamwork control will show up on the log in page of the EVN application. The following figure illustrates this set up:

The teamwork graph will contain any metadata about the changes, i.e. any teamwork:Change objects, the reified triples, information about the working copies etc. It may easily become larger than the actual main model, because it can keep track of the whole audit trail, and may use four triples for each changed triple. When a user logs into a working copy, the system creates a logical view - a graph that does not necessarily have a physical representation, but may be populated on demand. When created, this logical view will check the teamwork database for any uncommitted changes associated with the given working copy. Those changes will then be visible to the editing user, without having to be materialized in the actual master copy database. When the user makes some edits, the changes will only be logged into the teamwork database. Changes only make it into the master database, if the manager approves them. From that moment on, they will become automatically visible to any existing logical view.

There is much more to say about this whole architecture, but I'll leave these details for later. It suffices to say that the user interfaces of EVN and TopBraid Composer will shield the user from all those details. But if you ever want to get low level access to those teamwork repositories yourself, you can easily do that: just open the .tch files in TopBraid Composer and browse the content. If you have made some edits from EVN, you can see which triples are impacted in the teamwork graph. You can use SPARQL to query the changes, e.g. to find all changes that mention a given resource, or all changes within a certain time interval. You can also use scripting languages like SPARQLMotion to perform batch operations on the teamwork repositories.

The fact that we are using RDF all the way down makes the TopBraid teamwork support a very transparent and consistent architecture for change management.

UISPIN is now SPARQL Web Pages

2010-11-28T15:03:00.000-08:00

As part of the official release of UISPIN 1.0 with TopBraid Suite 3.4, we have decided to give it a more compelling and more descriptive name: SPARQL Web Pages (SWP). There are no technical implications of the new name - it's mostly to communicate better what this technology is all about: Creating HTML (and XML) documents by embedding SPARQL expressions and queries into template documents. The namespaces and many of the technical references will continue to use the term UISPIN, which illustrates the lower-level technical aspect (of being the User Interface layer of the SPIN framework).

RDFS Plus as a subset of OWL RL in SPARQL Rules

2010-09-01T03:13:00.000-07:00

There is a growing interest in the OWL RL profile of the Web Ontology Language, e.g. see recent postings on Semantic Overflow or Dean Allemang's blog entry. OWL RL has been defined by the W3C working group as a subset of OWL 2 that can be formally represented by rules. There are lots of rule engines out there that can be used to represent this subset. I had written about our implementation of OWL RL using SPARQL Rules (SPIN). This implementation can be downloaded and used with SPIN aware tools, such as TopBraid Composer. A human readable display of those rules can be found here: OWL 2 RL in SPARQL.

Some people wonder about the limitations of OWL RL - it does not cover the full expressiveness of OWL after all. How could something so limiting be of any value at all? But this question is only relevant if you are sold into selecting the OWL vocabulary in the first place. This isn't as obvious as it may seem. There is a huge interest in lighter knowledge representation and modeling formalisms such as SKOS, and the emerging Linked Data cloud also only makes use of a very limited subset of OWL.

If you have read Dean Allemang's and Jim Hendler's Working Ontologist book, you may have heard about an even smaller subset of OWL, sometimes called RDFS Plus. This is basically RDF Schema, plus some carefully selected terms from OWL. As an exercise, I recently went through a list of features that roughly cover what RDFS Plus might look like, which Jim had sent to me. All I had to do was to pick those rules from the OWL RL implementation that mentioned the terms from the RDFS Plus vocabulary, and collected them as SPIN Templates in the file at

http://topbraid.org/spin/rdfsplus

A human-readable rendering of this namespace with a list of SPARQL CONSTRUCT rules is here:

RDFS Plus in SPARQL

This exercise illustrates how easy it is to define any number of languages simply in terms of rules. There is a rule for rdfs:subClassOf, a rule for owl:inverseOf etc. When executed together, a small set of rules can lead to interesting inferences. But now take this one step further, and add support for any other vocabulary, such as SKOS:

http://topbraid.org/spin/skosspin

See this blog posting on how to use this for SKOS constraint checking. And then take this idea another step further, and add some rules for other popular vocabularies such as FOAF or Good Relations, or add support for unit conversion using SPIN. Suddenly you can express relationships between things brought together from multiple online shops with different units, that vaguely define subclass or broader/narrower relationships between each other.

You can basically select whatever kinds of inferences you need for your application. And you can execute them all with the same engine, you can edit them all with the same tool, and you can debug, profile and optimize them with the same platform. OWL RL and RDFS Plus are really good building blocks, but they are part of a family of languages, including the many languages (aka vocabularies and ontologies) that are still to come. I wrote before about how different this approach is from a the hard-coded, closed world view of some OWL fanatics, who still insist that moving beyond OWL DL is evil, and that all you need is a tableau reasoner. Rule-based approaches like SPARQL Rules (SPIN) make it easy to define your own language, or to assemble your own language from existing components.

WHERE OWL fails

2010-04-07T18:42:00.000-07:00

Paul Hermans has written an insightful series of blog entries (part 1, part 2, part 3, part 4) in which he reports on his experiences trying to express some SKOS integrity constraints and inference rules with OWL 2. After failing to express those things with OWL 2, he then demonstrates how his goals can be easily achieved with SPARQL with the help of the SPIN framework.

His conclusions (the last sentences from his blog entries):

"If you speak SPARQL fluently, it is fairly easy to define constraints on your RDF data using SPIN."
"And the winner for constraint S13 is clearly SPIN."
"Once again fairly easy to do with SPIN; a long study of the particularities of OWL2 DL restrictions to find out that this constraint cannot be expressed in OWL2 DL."
"SPIN wins again."

Paul is in no way associated with TopQuadrant and we have not asked him to create those write ups for marketing purposes - I discovered them by chance. Paul appears to be fluent with a large variety of technologies and makes balanced use of whatever tools and languages are most useful for his given tasks.

So why does OWL fail in those examples? In my opinion, these examples expose a fundamental design limitation of OWL: OWL is hard-coded against specific design patterns, but anything that goes beyond those patterns cannot be expressed. Furthermore, the choice of supported design patterns is misguided by theoretical assumptions about DL inferencing that are quite often irrelevant for practical purposes.

Let's look at a longer version of this answer. The data model of the Semantic Web is a graph structure consisting of RDF triples. The strengths of RDF is that people can define their own ways of representing data and knowledge, and thus create arbitrary RDF graph patterns. Users are free to define classes with any number of associated properties, forming larger structures that go far beyond the triple level.

In order to check constraints or execute rules on those graph structures, a general graph matching language is needed. A strong candidate for this is SPARQL, especially its WHERE clause. The WHERE clause is able to match fairly complex sub-graph patterns and provides variable bindings that can be used to report constraint violations or to fire the right hand side of a rule.

OWL on the other hand side is not able to represent arbitrary RDF graph patterns, but only a sub-set of those patterns that the designers of OWL found useful. Many of those patterns have seemingly arbitrary restrictions, as illustrated by Paul's examples (e.g., mixing different property types is not allowed in property chains). OWL 2 and some of its implementations such as the OWL API have driven this approach to extremes, making it not even possible to represent those patterns syntactically. This is because OWL 2 is not based on the RDF data model and therefore cannot talk about RDF in general.

So if you want to ask a question that the OWL 2 designers have not anticipated, then you cannot use OWL.

To make matters worse, OWL 2 is heavily influenced by research from the field of Description Logics, which many real-world users find both artificial and unhelpful. The goal of DL is to find a "tractable" sub-set of logic that allows inference engines to "guarantee" that all possible questions will be answered in finite time. While this sounds like an attractive value proposition from a theoretical point of view, practical evidence shows that the sub-set selected for OWL DL does not cover enough real-world use cases (see Paul's entries). Furthermore, there is enough practical evidence suggesting that while OWL DL inferencing may terminate in finite time, this time might be after the heat death of the universe and therefore completely useless. Just look at the mailing list archives of popular OWL DL inference engines to read about complaints of how slow those engines are in the real world. With SPARQL and SPIN you can of course also create very slow queries, but at least you have much greater flexibility and expressivity. And like with any language, a fair amount of engineering and experience allows you to prevent performance pitfalls. You also cannot expect to throw any complex query at your SQL database and expect ideal response times. Engineering is needed.

In defense of OWL, there are lots of useful design patterns encoded in this language, and it is great that the community has a standard vocabulary to talk about classes and things like property cardinalities. There needs to be some standard to capture ontology design patterns, and OWL does a good job for many of them. But this makes OWL just one out of a catalog of vocabularies, on the same level as SKOS or FOAF or SIOC or GoodRelations. It's simply a good vocabulary to talk about classes, while SKOS is a good vocabulary to talk about taxonomies and GoodRelations is a good vocabulary to talk about business.

But for anything that is actionable for the real world, a combination of various vocabularies and a rich constraint and rule language like SPIN is needed.

The SPIN Technology Stack

2010-04-03T10:00:00.000-07:00

Regular readers of this blog may notice that I am a big fan of SPARQL-based technologies. In fact, most of my work in the last couple of years went into defining extensions to SPARQL, and implementing editing and debugging tools for those extensions. The results of this work have been made core features of the TopBraid Suite, one of the most successful industrial semantic development platforms.

The first of those SPARQL extensions was SPARQLMotion that somehow emerged out of discussions on scripting languages between me and Dean Allemang (wow: he has a Wikipedia page) and other TopQuadrant colleagues. SPARQLMotion was published in the end of 2007 and has matured considerably in the last year, often driven by real-world feedback from our growing user base. SPARQLMotion is a visual scripting language that simplifies the development of data processing pipelines. In addition to its use in TopBraid Composer, SPARQLMotion scripts can be executed as TopBraid Live web services or used to drive TopBraid Ensemble applications.

A year later, at the end of 2008, we published the SPARQL Inferencing Notation (SPIN), a SPARQL-based rule and constraint checking language. SPIN also greatly extends SPARQL itself through its support for user-defined functions, magic properties and templates. SPARQLMotion now uses parts of SPIN for its base infrastructure, and SPIN functions can also be used in SPARQLMotion scripts. Major application areas of SPIN range from ontology mapping to rule-based systems and even computer games.

The newest addition to this family is UISPIN, published as a beta release with TopBraid Composer 3.3. UISPIN makes it possible to link RDF and OWL models with user interface descriptions that can be rendered as HTML or SVG documents. UISPIN will enable the creation of a new generation of dynamic business applications in which the rendering of content is entirely model-driven. UISPIN is also based on SPIN and borrows its ideas of procedural attachment to classes and pre-bound variables.

Taking a step back, we can now draw a pretty picture to illustrate how those languages fit perfectly together, forming the SPIN Technology Stack:

This pragmatic collection of technologies offers a fairly complete infrastructure for projects based on linked data and semantic web. Based only on RDF and SPARQL as well as bits of RDF Schema (and, if you like, OWL), the SPIN Stack covers a wide range of business needs:

RDFS/OWL + SPIN: rich, self-describing domain models
SPARQLMotion: executable behavior
UISPIN: model-driven dynamic user interfaces

Software developers may recognize that those three pieces correspond to the well-known Model-View-Controller (MVC) architecture pattern. The SPIN Stack now basically covers all aspects of classical software architecture. These languages are careful extensions of the official Semantic Web standards that take RDF/OWL and SPARQL out of the research labs and into the real world.

And the best thing is that there is a lot of energy behind SPARQL (with SPARQL 1.1 on its way) and lots of other extensions, online SPARQL end points and efficient SPARQL databases on the market. As this market grows and more and more developers become familiar with SPARQL, the SPIN Technology Stack will be a safe investment for companies that wish to create flexible solutions based on smart, self-describing data.

UISPIN Example: Documenting SPIN Functions

2010-04-02T10:00:00.000-07:00

The UISPIN framework can be used to create HTML documents from templates that contain embedded SPARQL queries and expressions. You can attach those templates to the classes of your domain model to help the system find the most suitable rendering of the instances. In this blog entry I show how this technique can be applied to create ontology documentation. In particular I create a UISPIN model for rendering SPIN functions, but similar ideas can be applied to other language elements such as OWL classes. This example also demonstrates many key features of UISPIN, including user-defined elements and control structures.

The end result of this example will look like the following. Whenever a user navigates to a SPIN function (here: spl:object), then the Browser will display the HTML shown below.

The resulting HTML page displays the qname of the function in the heading, then the comment of the function, then a section listing all arguments of the function and finally the body query (if one exists).

In order to produce such a rendering, I have created a new file that contains the UISPIN definitions for the SPIN ontology. This file imports the SPIN namespace (http://spinrdf.org/spin) and the TUI namespace (http://uispin.org/tui, which in turn imports the HTML support for UISPIN). Since I want to document all SPIN functions, I have attached the following ui:instanceView to the class spin:Function.

The UISPIN snippet above basically states that all instances of spin:Function shall be rendered as a HTML div element containing a h2 etc. The snippet contains many SPARQL expressions in the {= ... } notation, and those expressions are executed when the page is being rendered. As usual, the variable ?this points to the current instance, i.e. an instance of spin:Function, and functions such as spl:object can be called to conveniently retrieve properties of ?this.

The snippet above also contains the control element ui:if that only inserts its child elements if the ui:condition evaluates to true.

Finally, there is also a user-defined UISPIN element, spin.ui:ArgumentsList, which is defined as shown in the next TopBraid Composer screenshot.

A user-defined UISPIN element has a unique identifier (URI) so that it can be shared and reused on the Semantic Web. Then it can have any number of arguments, using the spl:Argument SPIN template at spin:constraint. By the way this is the same design pattern as for user-defined SPIN functions, templates and SPARQLMotion modules.

The core of any user-defined UISPIN class though is its ui:prototype. This is the snippet that will be inserted into the document where the element is used. In the prototype, the values of the arguments are pre-bound to the variables shown in bold face. In the example above, the value of the spin.ui:module argument will be bound to the variable ?module in the prototype. If you scroll up two screenshots, you can see that the call of the spin.ui:ArgumentsList element includes a value for spin.ui:module. This means that the current value of the variable ?this will be inserted as ?module into the prototype. The prototype itself contains a control element of type ui:forEach that traverses all arguments of the current module, and creates one table row and table data element for each of them.

Finally, the prototype also makes use of the control element ui:resourceView, which will insert the default rendering of the ui:resource into the target document. The values of ui:resource in this case are the arguments (?arg), and those are instances of spl:Argument. This class has the following ui:instanceView attached to it.

While the above snippet may be hard to read due to its formatting, you may see that it will display the predicate's qname, followed by its value type, followed by the word [Optional].

Let's quickly summarize the main concepts from this example:

You can attach HTML snippets to classes using the property ui:instanceView.
The variable ?this points to the current instance of the class.
Views may contain user-defined elements, which have a prototype.
In the prototype, the arguments of the element are pre-bound as variables.
ui:if can be used to insert an HTML block based on a condition being true.
ui:forEach can be used to repeat a block for each row of a SPARQL query.
ui:resourceView will insert the default rendering (e.g. ui:instanceView) of a given resoource.

Please give UISPIN a try by downloading TopBraid Composer ME 3.3. and let us know your feedback on the topbraid-users mailing list. Note that UISPIN is still in beta stage, and your input may shape its future.

Charts and Business Reports with UISPIN

2010-04-01T10:00:00.000-07:00

UISPIN makes it easy to develop new components that can be used like HTML tags in a document. Those components may comprise of a complex snippet of HTML and other XML-based languages such as SVG. UISPIN components can be published on the web, where they are identified by their URI. UISPIN Charts is a library of reusable components for visualizing data on charts and maps. Based on Google Charts and Google Maps, there is no need to install anything - the components are entirely declarative and ready to use.

The UISPIN Charts page shows many example screenshots of the various kinds of charts and maps that are currently supported: Pie charts, Bar charts, Map charts, and Google Maps. More will be added in the future.

Here is a step-by-step tutorial illustrating how to use those charts. We will create the pie chart shown in the following screen shot.

Let's first look into the RDF model of the data that we want to visualize. This data is about election results, and is downloadable as a Turtle file from here. Download it into your TopBraid Composer ME 3.3 workspace to follow the exercise.

The data file defines the classes shown in the diagram below. Instances of the class ex:Election point to a collection of ex:ElectionResults. Each result links a party with a percentage (an xsd:float literal).

Here is a form view of the ex:Election instance that we want to visualize. There are four ex:ElectionResults and they are blank nodes.

In order to use the chart components, we need to create a file that imports the UISPIN charts vocabulary. We can either import this namespace into the data document itself, or we can create a new document that imports both the charts and the data model. Since we do not want to clutter the data file, we create a new RDF file and drag the charts namespace as well as the data model into the Imports View. The charts namespace is part of the system ontologies, under the TopBraid/UISPIN project. The imports view should look similar to the following screenshot.

Having both the domain model and the charts namespace in the same RDF model allows us to create links between the Election classes and suitable visualizations. Select the class ex:Election so that it shows up on the form. You will now see a property ui:instanceView on the form. This is where we will put the definition of the pie chart later.

But first let's think about the actual data. Pie charts visualize slices of data in proportion to one another. Each slice can have a label (here: the name of the political party). These value-label pairs form a table, and this table is the input to the pie chart. Select the instance of ex:Election (ex:AustralianFederalElection2006) and use the SPARQL View to run the following query:

This query returns the numeric value in the first column, and the label in the second. This happens to be exactly the format that the charts:PieChart component expects. Note that the query above uses a built-in feature of TopBraid Composer: in the SPARQL view, the variable ?this is pre-bound with the currently selected resource.

Go back to the class ex:Election and locate the ui:instanceView property on its form. We could now add an empty row and type the UISPIN snippet of the pie chart in, but in this exercise we use a different route: Open the context menu behind the ui:instanceView property name and select Create blank node... which opens a class selection dialog as the following.

You need to select the class charts:PieChart here. In order to find it quickly, click on the Group by namespace button in the lower left corner. After clicking on OK, a new anonymous instance of charts:PieChart will be created and displayed as a nested form as shown next.

This nested form will enumerate all potential arguments of the pie chart class, including charts:label and ui:resultSet. Scroll down to find the widget for ui:resultSet, add an empty row and copy and paste the SPARQL query from the query view into it:

Now, go to charts:label and select Add SPARQL expression from its context menu. This will open up an empty row, in which you can paste the expression ui:label(?this), which will insert the name of the current election into the pie chart. Finally, set html:width and html:height to 500 by 240 as shown below:

That's the complete definition of our pie chart, and closing the nested form you can see its XML syntax:

You could have entered this directly in XML syntax instead of going through the form. In any case, we can now go back to the instance of ex:Election and click on the Browser tab at the bottom of the editor window to get the screenshot from the top of this article.

This pie chart is now the default visualization of all instances of ex:Election. But since the charts:PieChart element is just one among many other available elements (including the whole range of HTML tags), we can turn the pie chart into a comprehensive report that contains other views (such as a charts:BarChart), an HTML table and whatever else we want to display.

UISPIN: Creating HTML and SVG Documents with SPARQL

2010-03-31T15:31:00.000-07:00

UISPIN is a new framework that can be used to link RDF resources with user interface descriptions that can be rendered as HTML or Scalable Vector Graphics (SVG) pages. Based on SPIN, UISPIN makes it easy to embed SPARQL expressions into XML snippets, so that the content of those snippets can be dynamically assembled into complete XHTML documents and SVG graphics. Here is an example of a typical UISPIN snippet (for rendering SKOS concepts):

The UISPIN snippet above would be rendered in HTML as follows:

UISPIN is comparable to template languages such as JSP and PHP, but is natively optimized for the Semantic Web and Linked Data. Among others, UISPIN makes it possible to define new XML tags that encapsulate reusable (HTML/SVG) building blocks, and to share those building blocks on the Web in RDF.

TopBraid Composer ME 3.3.0 is the first tool that provides comprehensive support for editing, debugging and browsing UISPIN documents. UISPIN itself and its implementation are still in beta stage, and we publish it to encourage early adopters to play with this new technology and provide feedback. We will move quickly to bring UISPIN to a stable state in the coming months.

There is already a large body of documentation and examples online, and I will report about more case studies and examples on this blog in the coming weeks. To get an overview, I recommend walking through the mini tutorial on uispin.com. This will give you a good idea of the basic concepts and might even be sufficient to get you started. For details and the official specification, please check uispin.org.

Converting UML Files to RDF via Java/EMF Objects and SPARQLMotion

2010-03-30T20:49:00.000-07:00

TopBraid 3.3 comes with two new SPARQLMotion modules that can be used to convert arbitrary EMF files (including UML and XSD) to RDF. The underlying mechanism is very simple and extremely flexible:

Eclipse is able to load .uml files into a MOF-based Java object model called EMF.
SPARQLMotion provides a new module sml:ImportJavaObjectsFromEMFFile that takes a UML file as argument and loads its EMF Java objects into a SPARQLMotion variable.
SPARQLMotion provides a new module sml:ConvertJavaObjectsToRDF that can traverse arbitrary Java object graphs to create RDF objects with the same structure.
Now that we have the UML file represented in RDF, we can apply arbitrary transformations, in particular SPIN rules (SPARQL CONSTRUCTs) to create whatever ontology we like.

In an example SPARQLMotion script, this process looks as follows:

In the import step, we have to specify the file name (here: an example UML file) and the name of an output variable, that allows subsequent steps to access the EMF objects:

Next, we instruct the SPARQLMotion engine to convert the Java objects to RDF. This will start at the object specified by the variable uml, and then call all public getXY and isXY methods to collect the properties of the objects. The algorithm will include adjacent Java objects, as long as they are from a list of classes and packages specified under sml:javaClass:

As the output of this step, the module has produced an RDF graph containing instances as well as class and property definitions, all derived from the Java objects using reflection. All Java classes are mapped into a default namespace (abbreviated with the prefix "uml" in the screenshot below):

A TopBraid Composer class diagram shows that the generated classes also have RDF properties associated with them, all automatically derived from the Java objects:

Now that we have those RDF/OWL classes, we can attach SPARQL CONSTRUCT queries as SPIN rules to them. (Here, we use a property umlspin:mappingRule which is a subproperty of spin:rule which has max iteration count set to 1 to speed up the mapping):

These two example SPIN rules traverse the generated UML classes starting at the current class ?this, and create corresponding OWL classes and rdfs:subClassOf relationships between them. Of course, additional mapping rules are needed to represent the other UML elements such as associations and attributes. In our example SPARQLMotion script above, those SPIN rules are loaded from a generic mapping file umlspin.ttl. Note that the rules above call a user-defined SPIN function umlspin:getOWLClass that creates a URI resource from a namespace and a UML name.

With those rules in place, we just need to run the SPIN rules engine (TopSPIN) and get an OWL class hierarchy from whatever has been defined in the UML file:

The approach presented here is not completely out-of-the-box yet, but you can see that it offers a great deal of flexibility because developers can add or change any number of SPIN rules to fine tune the mapping. For example, in one case you may want to create owl:ObjectProperties from UML associations, while in other cases a reified object might be a better choice.

The SPIN based importer raises the level of abstraction and makes it possible to define complex mappings without having to work with a programming language like Java. This empowers a larger group of people to contribute mapping rules. The UML/MOF world consists of many related standards, and the SPIN/SPARQLMotion-based mapping enables the domain experts of those standards (e.g. mechanical engineers) to fine tune the importers.

Finally, other EMF-based importers exist for formats such as XML Schema, and the approach presented above would of course also work for any other Java object model, as long as it provides access to its properties using getXY and isXY methods. You can basically connect your own native Java objects into an RDF triple store by providing a bit of glue code (as a SPARQLMotion module similar to the EMF loader above).

The SPARQLMotion Debugger

2010-03-25T18:49:00.000-07:00

TopQuadrant's visual data processing language SPARQLMotion is now in routine use in many projects around the world. Its graphical notation and rich set of features makes it a powerful language to import, process and export linked data from almost any format, using Semantic Web technologies. Over the last two years, TopBraid's SPARQLMotion engine has become increasingly robust and it is now fair to say that it is becoming a de-facto standard language for linked data (at least in the professional world where the TBC-ME price tag is not an obstacle).

An important part of turning a (formerly) experimental language into an industry-strength solution is to provide tools that aid developers in the construction and testing phases of their project. In this spirit, we have added a graphical SPARQLMotion Debugger to TopBraid Composer 3.3. This debugger makes it possible to interrupt an executing SPARQLMotion script, to introspect into variable bindings and arguments, to ask test queries against the current state, and to display the current RDF graph.

Let's look into debugging the SPARQLMotion script shown below.

There are now two execution buttons at the top of the SPARQLMotion graph editor. The green arrow executes the script normally. If a module has been selected, then the script will only execute until this point. The green bug button will execute the script but immediately open the debugger window. The debugger will also be shown whenever a module is reached that has a breakpoint attached to it. To set a breakpoint, select a module and then press the button with the small blue dot. In either case, the debugger will show up as shown below.

The left part of the debugger displays the currently executed modules, including those from nested sub-scripts (shown indented under their parents). You can set additional breakpoints there to make sure you don't miss anything important. The main area of the debugger can be switched between three different tabs, including the Variables tab shown above. This tab displays all current input variable bindings, allowing you to see the exact state of the engine when it enters the current module. Below the variables, you can see the input arguments to the current module. The values of those arguments are shown in exactly the same way as the engine will interpret them, e.g. they will have string templates applied to them already.

The next tab can be used to ask arbitrary SPARQL queries to further explore what the current module will see when it executes. If the module takes a SPARQL query as an input argument, then this will be suggested as the initial query.

Finally, if you wonder about the structure of the input RDF graphs, you can switch to the third tab (shown below). The tree structure of the graphs represents the imports closure (sub-graphs). Having this view will be helpful if you wonder why certain queries don't work as expected - they may operate on a different set of input graphs than you may believe.

When all is said and done, you can use the buttons at the bottom to either continue the execution up to the next breakpoint, or to step into the next executing module. Note that the debugger will also show up when you run web services and TopBraid Ensemble callbacks.

The new SPARQLMotion Debugger addresses one major challenge that many users have: understanding exactly what's happening. SPARQLMotion is now no longer a black box, and there is no longer the need to create artificial debugging and trace output to trace what's happening behind the scenes. I am confident this debugger will significantly reduce the learning curve and create many happy SPARQLMotion users.

Magic Properties with SPIN

2009-11-02T09:23:00.000-08:00

Magic Properties (aka property functions) are a popular extension point supported by many SPARQL engines. Looking like normal triple matches, magic properties can be used to dynamically compute property values even if there are no corresponding triples in the actual model. For example, in the kennedys ontology, a magic property called :grandParent could be used to dynamically find grandparent relationships even though only the direct parent relationships are asserted in the model:

In the example above, the SPARQL engine will use some internal "magic" to figure out which values need to be returned for the ?grandParent variable. How this internal magic is implemented is left to the particular engine, and in the case of Jena this is simply a Java method call in which people can add their own custom code.

These magic properties can be extremely powerful, and often are the only escape mechanism if the default expressivity of SPARQL isn't good enough for a task. However, the current mechanisms of defining such magic properties require low-level (Java) programming and leads to queries that are neither transparent to the end user, nor platform independent.

The new version of SPIN 1.1 addresses those shortcomings and provides a powerful mechanism for defining magic properties entirely in RDF. Let's look at an example first. The following TopBraid Composer 3.2 screenshot shows the definition of the magic property :grandParent:

A magic SPIN property is an instance of the property metaclass spin:MagicProperty. Like regular SPIN functions, magic properties can define arguments that represent the left hand side of the magic property. Here, the argument sp:arg1 has type kennedys:Person indicating that this property can be applied to subjects of type Person. The results of the right hand side of the triple are computed by the nested SPARQL query specified as spin:body. In this body, the variable ?arg1 represents the Person whose grand parents we are asking for. When a SPIN-aware SPARQL engine hits the triple pattern with the :grandParent predicate in it, it will execute the nested body query and bind the variable on the right with the results of the SELECT query. The results in the first screenshot (JosephKennedy and RoseFitzgerald) have been computed this way: the grand parents are the parents of the parents.

An interesting characteristic of magic properties is that they may be applied in any "direction", i.e. either with a variable on the subject position, a variable as object or even in both places. For example, the :grandParent relationship can be queried to find all grand children of RoseFitzgerald (including JohnKennedyJr):

In this case, the nested SPARQL query is simply executed with different bindings (?arg1 is left blank and ?grandParent is pre-bound with RoseFitzgerald). We can also leave both sides blank and the system will return all existing grand parent/grand child relationships in the whole model.

Now comes the really interesting bit. Think about it: this is a mechanism that can be used to define new (SPARQL) predicates entirely based on other SPARQL queries. These SPARQL queries may be recursive and may use other magic properties. The SPARQL engine can walk through the results delivered by one magic property and, while doing this, can consider any other kind of background knowledge that it needs to compute to fulfill its task. If a solution does not lead to new matches, the system will backtrack and try the next one. In traditional rule languages (incl. Prolog) this technique is called backward chaining. Each user-defined magic SPIN property can be regarded as a rule that is fired on demand to support the task of solving a goal (finding matches).

Magic SPIN properties greatly extend the power of SPARQL, and do so in a very transparent and Semantic Web compliant way.

SPARQL Debugger and Profiler

2009-11-02T09:19:00.000-08:00

SPARQL queries may get complex and often contain a series of operators such as triple matches, filter clauses, optional statements and unions. In many cases, building correct and efficient queries using those elements requires multiple attempts. This is often because it is difficult to see what the SPARQL engine is doing under the hood.

As of version 3.2, TopBraid Composer (Maestro Edition) introduces a Debugger View that has been designed to help query designers by providing an interactive view into a query at execution time. The Debugger can be used to:

Display the internal data structure (algebra) that the SPARQL engine is using to execute the query
To walk through the execution of the query
To display intermediate variable bindings
To execute a query until it reaches a certain break point
To collect statistics about the number of query steps involved in every operator

This is a sophisticated feature that can be extremely powerful, but it also requires a decent understanding of how SPARQL works. People with some programming background may find the debugger more useful than beginners.

Background

Let's start with some background on how SPARQL works. The key feature of SPARQL is the WHERE clause, which contains conditions that return variable bindings based on the query graph. Depending on the type of query, these variable bindings are then returned in the SELECT clause, or used to CONSTRUCT new triples. In order to understand the characteristics of a query, we therefore need to focus on the WHERE clause, and modifiers such as ORDER BY.

When a query is processed, the SPARQL engine will first create an internal data structure, called Algebra. The query is executing this data structure, while the textual representation with keywords such as SELECT and WHERE only serves as the user interface and query exchange language. It is important to understand that different query syntaxes may be converted into the same algebra data structure, and that the same query syntax may be rendered into different algebras depending on the choices of the SPARQL engine. In particular, the engine may decide to optimize certain patterns. Seeing the internal data structure will often lead to surprising results.

For example, the query

SELECT ?subject ?object
WHERE {
   ?subject rdfs:subClassOf ?object .
   ?subject rdfs:label ?label .
   FILTER (fn:starts-with(?label, "C")) .
}

is converted into a SPARQL Algebra data structure that can be rendered into textual form as follows:

(project (?subject ?object)
(filter (fn:starts-with ?label "C")
  (bgp
    (triple ?subject rdfs:subClassOf ?object)
    (triple ?subject rdfs:label ?label)
)))

In this notation, you can see that the SPARQL engine is working with a nested tree structure of operators such as project, filter and bgp. These operators typically have attributes or child elements, for example bgp (Basic Graph Pattern) contains a list of triple patterns to evaluate. Note that it is up to the SPARQL engine to optimize those operators so that they lead to fewer database requests. In particular, multiple triple matches may be combined to a single operation to benefit from server-side joins.

The Algebra operators form a tree structure, in which each node returns an iterator of variable bindings. For example, the basic graph pattern above may return the binding ?label="Matriarch"^^xsd:string, ?object=Person, ?subject=Matriarch. These bindings are served up to the next operator in the processing pipeline. For example, the filter operator will evaluate whether the ?label starts with "C" and then pass the same bindings to its parent operation, the project step. Thus, each operator receives a stream of input bindings and may create new bindings for its parent. The final results of the query are the output bindings of the root operator (here, project).

The algorithm of the SPARQL engine (at least the Jena ARQ engine used by Composer) follows the evaluation model above, and builds a nested hierarchy of iterators. Each of those iterators has two main functions:

hasNext to check whether there is any additional result binding available
next to return the next binding and move to the next step

The SPARQL Debugger allows users to trace the evaluation of the SPARQL engine to see when hasNext and next are invoked, and whether they return new variable bindings. When you step through the query manually, the user interface will display arrows in different colors to illustrate the current position, and display variable bindings in a table.

Getting Started with the Debugger View

TopBraid Composer's SPARQL Debugger is accessible from multiple places:

In the SPARQL View, use the Debug query button to open the currently edited query in the Debugger view.
In the context menu of spin:rule property values on the form, use Debug SPIN query to debug the SPIN rule. Note that this may add a clause ?this rdf:type [Class] to the query, just like the SPIN engine would do.

As shown above, the Debugger is split into multiple areas. The main area on the left displays the query. Two different renderings are available:

Algebra displays the internal Jena ARQ data structure
SPARQL displays the same query in pseudo-SPARQL, which is derived from the Algebra (reverse engineered if you want) and therefore may look different from the original SPARQL syntax.

You can switch between those two renderings at any time and will get the same features. Depending on your experience, the SPARQL view might be easier to get started, but the Algebra view is much more informative as it provides the real underlying data structure.

Both textual views have a control bar on their left border. This bar displays the current step position with an arrow, and also displays any break points. Double-click on the bar to set or remove break points.

On the right hand side, a table displays current variable bindings. These are only filled if the current execution step is "green", i.e. is returning the next operation with some values.

The tool bar contains buttons to control the behavior of the SPARQL view. The following buttons are available:

Run continues the execution of the query until the next break point.
Step Over continues the execution of the query to the next step.
Profiler activates or de-activates profiling. When activated, the engine will record the triple matches (findSPO) queries done against the triple source, and display the resulting statistics behind each operation.
Layout vertically switches between vertical and horizontal layout. Horizontal layout is better suitable for "wide" queries, vertical layout for lots of variable bindings.
Preferences opens up the SPARQL Preferences page, which may impact how the SPARQL engine optimizes the query algebra. Changes will have effect after restarting the query.

Profiling

If profiling mode has been activated (and, if needed, the query restarted), then the query engine will operate on a modified triple store which, as a side effect of running queries, will also collect statistical information. In particular, this records which find(SPO) queries have been made, and how many triples have been iterated over. The accumulated numbers of those calls (find/triples) will be displayed for each operation in cyan color on the right edge of the text display. In the following example, there are two triple patterns that lead to low-level queries.

In the above screen shot, the upper triple match just leads to a single query (for all rdfs:subClassOf triples), and this query has (so far) returned 15 matches. The second (optional) triple match has been executed 15 times (for each of the former matches), but only five results were found so far. From these numbers you can see that the optional operation is far more "expensive" in terms of number of queries than the subClassOf query.

To summarize, TopBraid's new SPARQL debugger is a powerful low-level tool for people who want to understand the execution behavior of a query or SPIN rule in the SPARQL engine. The debugger and its profiler can provide valuable insights that may lead to much more efficient queries.

Fixing Constraint Violations with spin:fix

2009-10-30T13:29:00.000-07:00

Constraint checking is a popular feature of many Semantic Web tools to ensure that instance data meets the design objectives attached to classes and properties in an ontology. Data entry tools like TopBraid Composer and Ensemble use the SPARQL-based constraint checking language SPIN to make sure that users get warnings if the data they are entering is violating constraints. If a violation is reported, the user would read through the violation message and then change the data on the form.

One of the new features of SPIN 1.1 is the property spin:fix which can be used to let the system suggest operations that would repair a constraint violation automatically. The basic idea is that spin:fix can be attached to the spin:ConstraintViolation produced by a spin:constraint query to link the violation with one or more SPARQL Update requests. Those SPARQL Update requests may INSERT or DELETE triples to create a state in which the constraint is no longer violated. If spin:fixes are created, then the user interface may suggest them to the user with a single click. In the TopBraid Composer 3.2 screenshot below, the resource InvalidSquare violates the constraint that instances of Square must have equal width and height:

The following spin:constraint (attached to the Square class) implements the suggestions above:

You can see that the constraint creates a spin:ConstraintViolation that is attached to two instances of :SetObject via spin:fix. The class :SetObject is a SPIN template with the following definition:

The SPIN templates suggested as fixes could come from a library of re-usable building blocks. I assume that most constraint violations can be repaired by replacing, adding or deleting certain triples, and these cases can be generalized easily. Furthermore, the spin:constraints themselves can be generalized into templates, so that the ontology designer in the end just needs to pick the correct template to get a really powerful and convenient constraint checking mechanism.

Also note that the constraint fixing mechanism above could be useful beyond user interfaces, for example to automatically repair incoming data streams in web service calls.

OWL 2 Support in TopBraid Composer

2009-10-27T19:33:00.000-07:00

Following today's announcement that OWL 2 has become an official W3C recommendation, I am pleased to announce that TopBraid Composer 3.2 (to be published by the end of this week) has comprehensive OWL 2 support. Here is a sampler of some of the new capabilities.

Property Chain Axioms can be used to define relationships between multiple properties, for example to define that an uncle is the brother of a parent. In OWL 2 mode, TopBraid's Properties form contains a new widget for entering such property chains using owl:propertyChainAxiom. Another kind of new property axioms, owl:propertyDisjointWith can be edited on the same page.

User-Defined Datatypes are a mechanism of narrowing down datatype properties to specific value ranges, such as integer > 0. In typical cases, such datatypes are entered as allValuesFrom restrictions on the class form. We use the Manchester Syntax for that purpose:

In addition to class axioms, user-defined datatypes can also be used as global rdfs:ranges:

OWL 2 Class Axioms including qualified cardinality restrictions and all other features supported by the Manchester Syntax can be entered on the class forms:

All other OWL 2 extensions such as new property meta-classes, keys and syntactic sugar can be edited through the generic RDF editing capabilities of TopBraid - the extended OWL 2 system vocabulary has been very helpful for this. Of course, TopBraid Composer can load and save any OWL 2 file in formats such as RDF/XML or Turtle.

At the time of writing this, I am not aware of any OWL 2 compliant inference engines that we could freely distribute with TopBraid Composer. Currently available options include OWL RL engines such as SPIN or Oracle 11g RDF. I am sure more will follow, and anyone in the community is invited to contribute plug-ins to those inference engines that we cannot legally ship with our platform, as separate downloads.

The new OWL 2 support is available in all editions of TopBraid Composer, including the Free Edition.

RDFex: Partial ontology imports

2009-10-19T15:15:00.000-07:00

One of the overall design goals of Linked Data and the Semantic Web is vocabulary re-use. Instead of having thousands of "Person" classes, new ontologies should attempt to re-use existing Person definitions, such as those found in the FOAF namespace. This schema re-use makes it easier for Semantic Web agents to link data together, and potentially reduces the maintenance costs as it becomes possible to benefit from the whole infrastructure and community around those shared ontologies.

However, there are a couple of well-known reasons why such a re-use is not always feasible or desirable, leading to situations in which developers feel they need to reinvent the wheel. One particular problem is that the OWL construct for linking vocabularies (owl:imports) has all-or-nothing semantics: if my ontology owl:imports the FOAF namespace, then I would suck all definitions of FOAF into my own model, even though I just care about one or two concepts. The result is that in Semantic Web inference engines, browsers or editors, my ontology will be full of definitions that are just distracting, or unnecessarily increase the complexity of my model. Since owl:imports is not the ideal mechanism, people sometimes simply extract term definitions from other files and paste them into their own files - look for example at the bottom of this file. This, of course, leads to other maintenance problems and is generally not a clean approach.

For an internal project in which I wanted to re-use parts of the SIOC namespace, I have implemented a new web service called rdfex.org. RDFex is a very simple, yet IMHO elegant, mechanism for using owl:imports to import snippets of other namespaces without having to copy and paste them. The basic idea is that the rdfex.org server can be used as a proxy for various popular ontologies (including DC, FOAF and SIOC), so that users can specify which classes, properties and individuals from those namespaces they would like to import (using owl:imports). For example, the proxy ontology http://rdfex.org/foaf/Person,firstName represents all triples defining the class foaf:Person and the property foaf:firstName, including their rdf:type, rdfs:labels, rdfs:comments and any relationships between those terms (such as rdfs:domain and rdfs:range). Any combination of those resources is available because the result will be dynamically assembled at request time.

The upcoming release of the TopBraid platform 3.2 also has native support for those rdfex imports, so that the system can do this extraction from local copies instead of having to go to the web. TopQuadrant is committed to supporting this service in the future, so please feel free to use it if you find it useful.

Currency conversion with the Units Ontology, SPARQLMotion and SPIN

2009-09-04T10:00:00.000-07:00

As the Linked Data cloud is steaming ahead, the first large online stores (recently: BestBuy.com) are publishing their products in a machine-friendly RDF format. In order to exchange product information in a meaningful way, the various product vendors should use a shared (or at least mappable) vocabulary to represent prices, so that internet search engines and crawlers can better compare values. In the real world, three-letter currency codes such as USD, EUR and AUD are being used. These abbreviations are a standard vocabulary, but having just a string representation is an error-prone strategy - e.g. the string "AMD" could be either the Armenian Drams currency, or a computer chip manufacturer.

We have recently published a comprehensive units ontology (QUDT) that also includes all of the world's currencies. For each currency, a globally unique identifier (URI) is used. These URI resources also point back to the currency abbreviation, using a property called qud:abbreviation. A good strategy for ontology designers would be to use those URIs as ranges of their properties, instead of string codes. Using those standard URIs will have some benefits down the road...

This morning I have created a SPIN library of currency conversion functions that can be used to convert between various currencies, using the very latest conversion factors. These functions can be used in SPARQL queries or SPIN rules. For example, we have a function currencies:getRateByCodes that gets the latest exchange rate between two currencies, specified by the three-letter codes:

This particular SPIN function is backed by a SPARQLMotion script that takes two currency codes (arg1 and arg2) as input and then calls an external web service to retrieve the latest exchange rate. The web service delivers XML, and a simple XPath expression is used to extract the value into an xsd:float RDF literal:

We can drill into the Call web service module to see that it is simply calling a REST web service, inserting the two arguments into the URL string:

Based on this low-level function, we can define additional higher-level functions. The following screenshot shows the complete definition of a SPIN function that takes two qud:CurrencyUnits as input, then gets their respective abbreviations, and finally calls the getRateByCodes function:

Whenever the function is called, any SPIN-aware SPARQL engine will simply execute the SELECT query specified as the body of the function. This means that new functions are built by combining other functions. The same mechanism was used to define the convertLiteral function, as shown below:

The function above makes good use of the QUDT units ontology to shield the user from any low-level details. All the user needs to do is to specify the range of a property to be, say, unit:USDollar, and then all future values of that property will be stored as literals with that unit as datatype. The SPIN function can then look at the literal to find out about its currency. Then, the function can look up the unit's abbreviation and make the corresponding web service calls to fetch the current exchange rate. Finally, a simple multiplication returns the desired new value.

Once these functions are defined, they can be used in various places. For example, here is a class Product, which has two properties: one stores the price in US Dollars, and another represents the price in Australian Dollars. Only the USD price needs to be entered, because the class also has a SPIN rule attached to it which will automatically do this computation as an inference:

An example Product instance, with the inferred (blue) AUD price is shown below:

These rules can of course be generalized further into SPIN templates, or the range of the target property could be used instead of hard-coding the target currency anywhere. There are really endless possibilities here.

Units ontology with SPIN support published

2009-08-31T12:40:00.000-07:00

My co-workers at TopQuadrant have just published a new OWL ontology about Quantities, Units, Dimensions and Datatypes (QUDT). This is a result of a long term, ongoing project with NASA AMES, and our friends at NASA have permitted us to publish those ontologies to encourage the wider use outside of NASA.

The QUDT ontology is very carefully designed and provides comprehensive coverage of almost every unit of measurement that is known to humankind. For example, it defines the unit Centimeter as follows:

Each unit has a stable URI, making it possible to link to it from your own domain models in a reliable way. For each unit, the ontology defines some useful metadata including abbreviation, a link to DBpedia and a categorization of units into groups, such as length units.

I think this units ontology can fill an important gap in the current Semantic Web and Linked Data efforts. Numeric data without any formalized units is pretty useless for machines, and sometimes even for humans. Currently, a unit may be mentioned somewhere hidden in a comment or not at all, but the QUDT ontology allows ontology designers to clearly specify these implicit assumptions. With explicitly modeled units, linked data can be processed and transformed in much more useful ways. For example, if a height it specified in Centimeters, then a smart linked data browser can automatically translate it into Feet for US American readers.

There are two main ways of using the units ontology: you can use the unit resources to "annotate" your properties with a dedicated property such as qud:units. The values of your property would use built-in datatypes such as xsd:double. The other alternative is to embed the unit directly into the literals. For this use case, all units have also been declared to be rdfs:Datatypes. This makes it possible, to assign units as rdfs:ranges of a property as shown below:

Here, the property height has the range unit:Centimeter. An example instance would then show up like this:

The specific height will then be stored in RDF literals such as "8380"^^unit:Centimeter. (The upcoming version 3.2 of TopBraid Composer will show the unit in parantheses behind the property name, but I didn't want to play tricks here).

Now that the units have been formalized in an ontology, new ways of working with numeric data become possible. As described earlier, the SPIN framework can be used to define new SPARQL functions which can then be used to do things like unit conversion. We have published a SPIN Library which contains some generic unit conversion functions. For example, the qudspin:convert function can be used to convert any numeric value from a source unit (here: unit:Centimeter) to a target unit (unit:Foot):

If the units are used as datatypes, then the function qudspin:convertLiteral can be used, saving one argument in the function call:

In the following example, we iterate over all instances with a :height property and display the height (in cm) as well as the converted height (in feet) using the SPIN function:

Such unit conversion tasks have been made possible by adding conversion multipliers to the QUDT ontology. SPIN functions can use this extra metadata to drive mathematical computations. The function qudspin:convert is backed by a SPARQL query as shown below:

The SPIN framework makes it possible to define such SPARQL functions (and rules and constraints) in a completely declarative way. No extra hard-coding of anything is needed. Any SPIN-aware SPARQL engine can simply look up the definition of the qudspin:convert function on the web and learn about the underlying mathematics. Likewise, there is no need for humans to worry about the calculations themselves - they can treat the SPIN function as a black box.

The easiest way to play with this (for example using TopBraid Composer Free Edition) is to add an owl:imports statement to http://topbraid.org/spin/qudspin

Ontology Mapping with SPIN Templates

2009-08-21T13:44:00.000-07:00

The question of how to transform data from one ontology to another comes up again and again, most recently in a question on the W3C Semantic Web list. The requirement is very real: for example, assume you have a class Person (with firstName/lastName) and a class Member (with fullName), and you want to construct one Member for each Person, so that the fullName is derived by concatenating firstName + " " + lastName. So basically you want to transform some (legacy) data into a format that some other application can understand.

Ideally, there should be a reusable standard mapping ontology for this purpose, which is also executable and user-friendly in visual editing tools. I am not aware of such a standard ontology, but I know how it could be built. Clearly, the typical complexity of such mapping tasks goes beyond what is provided by modeling languages like OWL. A graph matching language like SPARQL with rich built-in functions will be better suited. SPARQL CONSTRUCT queries can be used to define such mappings, as described on this blog three years ago.

The SPARQL Inferencing Notation (SPIN) provides a framework for organizing such SPARQL CONSTRUCT queries in a way that is easy to maintain and efficient to execute. In the following example I will walk through the steps needed to create a generic mapping ontology for tasks such as the one above, using SPIN Templates. The example is intentionally held very simple. The resulting file can be downloaded here and you can use TopBraid Composer (even the Free Edition) to execute it.

Let's assume we have two ontologies, person and member with the following classes:

An example instance of the source ontology may look like the following, with values for firstName and lastName filled in:

SPARQL can be used to create a mapping so that all instances of Person become Members, with a fullName derived from firstName and lastName. We would need two CONSTRUCT queries: one that adds the rdf:type triple to make the Persons also Members, and one that concatenates the firstName and lastName values into the fullName. You could attach those CONSTRUCT queries as SPIN rules to the classes as shown below. Note that the variable ?this means "for every instance of the class Person".

This mechanism will work fine, we can press the inferences button to run the SPIN rule engine and it will create the new RDF triples:

We can see that the Person is now also a Member with a fullName:

However, the solution above requires that the person creating the mapping is familiar with SPARQL. Additionally, the transformations can not easily be reused and similar SPARQL queries need to be entered the next time a string concatenation is required.

SPIN Templates can be used to encapsulate SPARQL queries so that they can be reused and edited easily. In the screenshot below I have replaced the hard-coded SPARQL queries with two SPIN template calls, which actually do the same but in a much nicer way:

Another way of visualizing these is using TopBraid's Diagram facility:

Let's look behind the scenes. The two entries under the spin:rule property are now SPIN Template Calls. A Template Call is an instance of a SPIN Template, but with arguments filled in. Here is the definition of the first SPIN Template, the concatenation rule:

The SPIN Template above is wrapping a SPARQL CONSTRUCT query (under spin:body). Templates can take arguments (under spin:constraint), which define how the template can be invoked. The values of the arguments will be "inserted" as variable bindings into the SPARQL query. In the example above, there are three arguments (sourceProperty1, sourceProperty2 and targetProperty) which are referenced in the body query as variables ?sourceProperty1 etc. In order to use such a template, the user simply needs to select the source class, go to "Create from SPIN Template..." under spin:rule, and fill in the arguments, as shown below.

The resulting Template Call will be associated with the class Person as a spin:rule, so that the SPIN (mapping) engine will infer the same new triples. The main achievement though is that the string concatenation module has now been generalized and could be reused in other ontologies. Since SPIN Templates are represented entirely in RDF, they can be shared on the web. Creating a library of such mapping modules would be a great topic for a Master's Thesis...

SPIN Diff: Rule-based Comparison of RDF Models

2009-07-28T09:59:00.000-07:00

One of the new features of the Maestro Edition of TopBraid Composer 3.1 is a simple yet very flexible diff tool that can be used to compare two versions of an RDF file or database. Diffing is a common requirement of collaborative modeling work, but conventional text-based diff tools often fail miserably on RDF-based data.

TopBraid's Diff feature is based on the SPARQL-based rule language SPIN. Our approach works as follows:

We have defined a simple diff ontology that defines classes and properties to describe differences between two RDF models.
The diff engine creates instances of those diff ontology classes, e.g. diff:AddedTripleDiff.
The rules that construct those diff instances are expressed as SPARQL CONSTRUCT queries in SPIN.
Any RDF-based browser such as TopBraid Composer or Ensemble can be used to browse the results of the diff engine - they are simply another RDF model.
Likewise, any RDF-based processing language such as OWL, SPIN or SPARQLMotion can be used to post-process the output of the diff engine, for example to create more detailed diff reports for specific ontology design patterns.

Let's have a look at a simple example with the out-of-the-box behavior first. The example ontology contains a class Person with an owl:Restriction attached to it, as well as labels and comments. I have made some edits to the file and ran the diff tool from TopBraid Composer's Model menu. After selecting the old version of the file, the diff engine runs the rules encoded in the diff.owl file and produces a new file containing the change records. There is a stored SPARQL query in the diff file, defined as a SPIN Template, which can be used to conveniently list all Diff objects in a table. Use the Run SPARQL Query from SPIN Template button in Composer's main tool bar to open a diff report such as the following:

The diff output file also imports the current domain ontology, and you can therefore double-click on any diff entry to navigate to the relevant subject or predicate. A closer look reveals that the diff reports consists of instances such as the following:

This instance of type diff:AddedTripleDiff states that a new rdfs:comment "A human being" has been added to the class difftest:Person. As you can see, the details of the particular change can be queried using properties such as rdf:subject.

The instance above has been created with the following SPIN rule, which is attached to the diff:AddedTripleDiff class:

The trick is that the diff rules can query the old and the new triples using pre-defined named graphs (diff:new and diff:old), and then construct response objects based on comparisons across those two graphs. In the example above, a helper method from the SPIN Standard Modules Library spl:hasValue is used to check whether a given object has already been present in the old graph.

As you can see, this whole diff engine is very much model-driven and not only operates on RDF, but also produces RDF as output, and uses SPARQL rules to query the RDF graphs. The particular collection of rules to fire is stored in the diff.owl file. This makes the engine very adjustable and extensible. For example, let's assume that we want the engine to be smarter about OWL syntax and create custom diff objects for changes to OWL restrictions. In order to do that, we create a new class diff:ChangedRestrictionDiff, and attach the following diff:rule to it:

This will then create more helpful diff messages such as "owl:minCardinality restriction on property firstName at class Person changed filler from 0 to 1". You can create similar custom-tailored rules for other ontology design patterns.

The generic approach presented here has the advantage of being very flexible. The current limitations are cases such as global renaming of a resource, which would currently be regarded as a collection of add and delete triple events. Higher-level rules are needed to analyze the lower-level diff results for such patterns and combine those "add/delete triple" events into "rename" events. Fortunately, given the open architecture of the system, anyone is able to contribute such changes without necessarily having to write any Java code. The beauty of generic model-driven engines is that this power is transferred to the end users. Furthermore, the behavior of the system is transparently encoded into rules that can be viewed by anyone, including software agents on the Semantic Web.

Announcing TopBraid Composer Free Edition

2009-06-15T14:48:00.000-07:00

I am thrilled to announce that TopQuadrant has just released a completely free version of its Semantic Web development tool TopBraid Composer! See the press release here. In a nutshell, the Free Edition provides the basic editing capabilities of the Standard Edition, i.e. you can use it to load, edit and save RDF/OWL files using TopBraid's editing forms. Furthermore, the Free Edition has excellent SPARQL support, including full support for the SPARQL Inferencing Notation (SPIN). Like all editions of TopBraid Composer, the Free Edition is based on Eclipse and provides a stable and well-documented environment for newcomers to Semantic Web technology.

Download TopBraid Composer Free Edition

SemTech pre-announcements

2009-06-12T09:13:00.000-07:00

I am looking forward to the Semantic Technology Conference in San Jose, which starts on Monday. My employer TopQuadrant will be presenting a lot of new technology there and we will be making a couple of hopefully very interesting announcements. If you are in the area, please join us for the second TopBraid Users Group Meeting on Thursday morning, or visit our company booth in the exhibition hall during the show. We'll be giving demos of new capabilities throughout those events, and you can stop by to meet the team and ask questions. We would also like to talk to you if you are looking for a job.

Additionally, my colleague Robert Coyne will be co-presenting a talk on rich information space browsing. Dean Allemang will be leading a hands-on session on deploying solutions with TopBraid Suite on Thursday afternoon. Finally, I'll have the opportunity to talk about our object-oriented semantic web rule language SPIN on Wednesday afternoon. One interesting aspect of this talk is that it is entirely based on free software, but that's all I am allowed to say for now :)

Interacting with XML-based web services using SPARQLMotion

2009-05-19T20:07:00.000-07:00

One of the new features of TopBraid 3.0 is improved handling of web services that return XML results. We have added two new module types to SPARQLMotion that can be used to walk through XML documents via XPath. Here is an illustrated example of how to use these capabilities. You can also download and play with the example script.

The first step is a web service call using sml:ImportXMLFromWorkspace:

In this example we are using a fixed URL from the geonames.org web service:

We can run this trivial script with the debug button to examine the XML result in TopBraid's console:

Our query takes a postal code as input and delivers an XML structure of matching Australian suburbs. We want to copy some of the result values into our simple Suburbs ontology, and capture name, latitude and longitude of each suburb.

In order to do that, we want to walk through all code tags of the XML result. We add an sml:IterateOverXPath module that takes the ?xml from the previous step as input:

At this stage, Composer offers a convenience feature: right click on the IterateOverXPath module and select "Set XPath...": this will execute the script up to that module and open up an XPath creation dialog. This dialog can be used to select the path to the elements that we want to iterate over:

After we have defined and assigned the XPath, we can now specify what we want to do with each code element from the XML document. In order to do so, we create the body of the iteration module as shown in the overview below. This body will be repeated for each code element. The ?xml variable is then bound to the current code element as root.

Each of the three sml:BindByXPath modules has a local XPath starting at the respective code element. Again, we can use the "Set XPath..." feature to let the SPARQLMotion engine execute up to the module and open an XPath editor dialog:

We simply need to pick the example instance data (here, the specific latitude value) and the dialog will generalize it into an XPath expression in the text field. This will bind the specified text node to the variables ?name, ?lat and ?long. In the final step, we define a SPARQL Construct query using sml:ApplyConstruct to create RDF triples for those values:

The complete script can now be executed, and SPARQLMotion we can infer the resulting triples so that we can browse them in TopBraid:

The result is that we have converted the XML data from the web service call into instances of the my:Suburb class in our ontology:

SPARQLMotion is of course more general than that, and you can combine such web service calls with any other type of processing supported by the engine. You can also use the same technique to interact with other XML documents such as local files.

TopBraid Suite 3.0 is here!

2009-05-01T11:13:00.000-07:00

Roughly nine months after our previous official TopBraid Composer release, we have today released TopBraid Suite 3.0! This is not just another Composer version, but a significant milestone for TopQuadrant's product offerings. For the first time, we have integrated and aligned the releases of Composer with an all-new TopBraid Live and TopBraid Ensemble.

Ensemble has been redesigned completely and makes the development of Semantic Web applications easier than ever - completely without programming in the traditional sense. The best way to experience these new capabilities is to start TBC Maestro and then open a web browser at http://localhost:8083/tbl. This will open up the TopBraid Live welcome screen from which you can open an Ensemble application on whatever ontology you have in your Composer workspace, such as the following:

By default, Ensemble is a generic linked data browser and editor with features such as trees, graphs, tables, charts, search and maps. There are two main innovations of Ensemble that make it different from similar frameworks though:

Customizing an application is easy: for example, if you want to change the kind of hierarchy to display in the tree, just configure it in TBE and save your application so that others can use it too. We have a purely model-driven architecture that represents the state of each application in RDF.
Adding new functionality is easy: you can define SPARQLMotion scripts on the server and make them accessible via buttons in TBE. For example, you can define new operations that manipulate the currently selected tree node in whatever way you like, or you can import new data from RSS feeds or whatever else SPARQLMotion allows you to do. And if that's not enough, you can of course extend SPARQLMotion on the server.

There are several other major innovations in TopBraid 3.0, including SPIN which I have written about in many previous blog entries. We hope you enjoy the new TopBraid!

Upcoming talks at JavaOne and SemTech

2009-04-29T14:51:00.000-07:00

June will be conference month for me, and I'll give two presentations on my favorite topic of how to use RDF and SPARQL to develop model-driven applications. The first talk will be at JavaOne 2009 in San Francisco, entitled "Developing smart Java code with Semantic Web technology". The second talk at Semantic Technology Conference 2009 in San Jose is called "SPIN: An object-oriented framework for business rules using SPARQL". I hope to meet some of you there!

Event-Condition-Action Rules with SPARQLMotion

2009-03-16T17:05:00.000-07:00

Several customers have requested a mechanism to trigger certain actions in response to changes to an RDF source. This can be used, for example, to update entries in some database when new triples come in, or send a message to someone to notify her about changes, or simply to synchronize dependencies among multiple data spaces. For the new beta 2 release of TopBraid, I have added a mechanism that can execute arbitrary SPARQLMotion scripts in response to changes to a model.

Here is an example of how it works. Download 3.0 and drop this file into your workspace. The file contains a SPARQLMotion script. The file is ending with .sms.n3 which instructs TopBraid to load this file into memory when the system starts up. The script (shown below) contains a module of type sml:TrackChanges. Any script with such a module will be executed whenever a change has been made to the current RDF model in TopBraid.

To play with this script, open the kennedys.owl ontology from TopBraid Composer's standard examples folder. Now, change the firstName or lastName of any Person in the model: you will see that the (full) name of the person also changes. This change is performed through SPARQL DELETE and INSERT calls at the end of the script above. This is similar to the SPIN-based mechanism for updating display labels that I described earlier today, but the changes are asserted and not only inferred.

The basic mechanism is as follows: The TrackChanges module produces instances of the class Change from a simple change ontology. Each Change points to reified triples (instances of rdf:Statement) via change:added or change:deleted. The modules that follow the TrackChanges module can now query the Change objects and decide what to do with it. In the example above, the script iterates over all subjects that have changed their firstName or lastName and then runs the aforementioned DELETE and INSERT calls. Any other module could also be inserted though, for example to send an email, write something to a log or whatever else SPARQLMotion offers.

For the next release I plan to complete the integration of this mechanism with TopBraid Ensemble. Since Ensemble applications store their internal state in RDF, this would make it possible to perform changes to the user interface (alerts, status changes, updating charts) as a result of changes to the database. This will make it possible to implement knowledge-based stock tickers, new forms of social networks, intelligent collaboration platforms, pro-active agents etc - all with a declarative engine and no low-level coding involved apart from SPARQL. Stay tuned...

Using JavaScript to Define SPARQL Functions

2009-03-16T13:27:00.000-07:00

Many real-world SPARQL queries make heavy use of built-in functions for tasks such as string processing and mathematical calculations. SPARQL comes with a pre-defined set of such built-in functions. However, in practice, these built-in libraries are frequently extended to solve specific problems that have not been anticipated by the language designers. Such extensions are typically implemented natively for a specific SPARQL execution engine, for example in Java. Needless to say, this is not a solution in the spirit of the (Semantic) Web, because it leads to a Tower of Babel with all kinds of dialects and platform-specific extensions.

We have proposed SPIN Functions as one possible extension mechanism for SPARQL, that allows anyone to derive new SPARQL functions by combining other SPARQL functions and query templates. In general, SPIN functions are Semantic Web resources that can be referenced by their URI to get a description of the function's arguments, return value and executable body. However, even this approach does not cover all possible use cases, because it is still limited by the lower-level SPARQL operations and functions. Many problems can only be solved with a general-purpose programming language.

TopBraid 3.0.0 beta 2 now introduces an extension of the SPIN functions mechanism that can be used to define new SPARQL functions with the help of JavaScript. In a nutshell, a SPINx function is a SPIN function that points to a snippet of JavaScript, or a JavaScript file, which will be executed whenever the function is run. The arguments of the SPARQL function call are made available as arguments to the corresponding JavaScript function. Here is a simple example:

    ex:square
      a spin:Function ;
      rdfs:subClassOf spin:Functions ;
      spin:constraint
          [ a spl:Argument ;
            rdfs:comment "The value to compute the square of" ;
            spl:predicate sp:arg1 ;
            spl:valueType xsd:float
          ] ;
      spin:returnType xsd:float ;
      spinx:javaScriptCode "return arg1 * arg1;" .

The function above can be called such as LET (?sq := ex:square(4)) to calculate the square value of the argument. In addition to having inline code via the spinx:javaScriptCode property, there is an option to simply link to a .js file that contains a function with the same name.

This simple approach (a variation of which had been proposed by Gregory Williams) greatly extends the expressive power of SPARQL through a relatively platform independent mechanism - JavaScript interpreters such as Mozilla Rhino are widely available on all major platforms. We have selected JavaScript for three major reasons: First, JavaScript (known as ECMAScript) is a well-established interpretable language that many users are already familar with. Second, JavaScript is web-friendly: you can reference scripts via URLs and scripts may import other scripts from the web as well. Finally, JavaScript has an attractive security model that will make it more difficult to create malicious SPARQL extensions (of course, details would need to be fleshed out).

The expressivity of JavaScript is great for many problems that are beyond the capabilities of SPARQL and its built-in features. You can freely express if-then-else conditions, loops, sub-procedure calls etc, and thus gradually build up your own function libraries in a portable fashion. A limitation right now is that the JavaScript-based SPARQL functions (in TopBraid) do not have any mechanism yet to access the current RDF graph at execution time. This would be needed to implement things like walking an rdf:List or computing average values etc; in short any use case that requires more background knowledge than what has been explicitly passed into the function as arguments. However, this is limitation can be fixed, for example by defining a small collection of built-in call back functions such as find(S,P,O) in JavaScript. There is a large body of related work on JavaScript RDF APIs that may also be leveraged for that purpose.

Generalizing Graphs into SPARQL Queries

2009-03-16T13:07:00.000-07:00

The new TopBraid Composer 3.0 beta 2 now includes a simple visual SPARQL query editor that has been suggested by my colleague Dean Allemang. This query editor is a feature of Composer's graph browser. Dean has discovered that he is using the Graph browser very frequently to explore complex relationships between RDF resources, and asked me: wouldn't it be cool if I could take a snapshot of this particular graph and generalize it so that I can find all sub-graphs of the model with similar structure? Take the example below:

Here, the user has started to explore the RDF graph at a specific instance of Person (Alina Mojica). Opening up the has gender relationship displays the link to the resource female. In the graph above, this link to female as well as the year of birth 1965 have been "fixed", while all other values remain variable. Pressing the star button above the graph now creates a SPARQL query:

Executing this SPARQL query (with another mouse click on the same button) gives us all female Persons that were also born in 1965. The same approach can be used to turn more complex graph patterns into templates - just fix those values that you want to keep and leave the rest as variables.

A very similar graphical SPARQL editor, executing in a rich internet application based on Flex, is now also part of the new TopBraid Ensemble.

Deriving human-readable labels with SPIN

2009-03-16T11:49:00.000-07:00

I know, it took me ages to get there, but TopBraid Composer 3.0.0 beta 2 finally introduces the ability to switch the whole user interface to display human-readable labels instead of resource identifiers such as qnames. There is a button in the tool bar that instructs TopBraid to use any values of rdfs:label or sub-properties thereof, and otherwise fall back to qnames. For example, in the following screen shot, Alfred Tucker's wife is displayed as "Kym Smith" instead of "kennedys:KymSmith".

I believe the lack of this (trivial looking) feature has been a major disappointment for people coming from Protege or comparable tools :) Anyway, now that we have this feature in, let's do something cool with it. You may notice that the Kennedy ontology above contains three naming properties: first name, last name and name, where name can basically be derived by concatenating first and last names. In the kennedys.owl that is found in the examples folder of TBC, the name property has been asserted and manually maintained by the editors. A smarter way is to use a SPIN rule to do this maintenance work for us:

Here, the spin:rule gets the current values of firstName and lastName for the given instance of Person (?this), creates a full string (?name) using a SPARQL built-in function call, and then infers this ?name as value for the name property. Turn on incremental SPIN inferencing and the value will update after each change.

The beauty of this approach is that you can now use arbitrary SPARQL expressions to derive labels. And, as we all know, SPARQL is very expressive and becomes even more expressive every day...

Rapid Semantic Web Application Development with TopBraid Ensemble

2009-03-16T10:08:00.000-07:00

TopQuadrant's product family, called TopBraid Suite, consists of the semantic modeling tool TopBraid Composer, a semantic server called TopBraid Live, and a client-side framework called TopBraid Ensemble. In the new Composer 3.0.0 beta 2 release we have finally aligned all these products into a single development and testing environment: When you download the Maestro edition of beta 2, you also get a personal edition of TopBraid Ensemble, ready to use without further download or installation needs!

Let me show you how this works in practice. Launch TBC-ME 3.0.0 beta 2, and open a web browser. Then go to the URL http://localhost:8083/tbl. The following screen will show up:

The TopBraid Ensemble team has put great efforts into refactoring and generalizing the Ensemble framework, so that it is now a comprehensive development framework for dynamic business applications. Ensemble applications consist of configurable rich-client components such as trees, forms and graphs (all based on Adobe Flex). These components can be re-arranged and re-wired in many ways, to customize an application's appearance and behavior for specific needs.

A freshly installed Ensemble comes with a default application that contains a pre-defined configuration of all built-in components. You can select this default application and run it on any RDF/OWL model from your workspace. The following screen shows the default Ensemble application on the geotravel.owl example ontology:

Ensemble is entirely driven by RDF/OWL models. Not only are the user interface components such as trees and forms driven by the underlying ontology, but the state of the whole application is represented by a flexible RDF-based data model. For example, if the selection of the Tree component (on the left) changes, then some RDF triples in the application's data model will change. Other components can react on those changes, and update their own internal state. If the selection of the tree changes, then the Results Grid (upper right) will display the instances of the selected class. This behavior is "soft-coded" though, and can be changed with a few mouse clicks. Below is a screenshot of a configuration dialog that can be used to customize and re-wire the components:

As shown above, it is possible to change certain properties for each component. For example, the root object and the transitive property of the tree component can be changed dynamically. Similarly, components can listen to events or publish events for other components to consume. This makes it very easy to create custom applications, for example, an application that displays a hierarchy of SKOS terms using the skos:broader property instead of a class tree. Needless to say, various user interface settings such as the choice of available buttons and style sheets are configurable as well. Any end-user with sufficient privileges can make those adjustments and then only needs to press the Save Application button to store this configuration and share it with her co-workers.

Like the rest of the TopBraid Suite, Ensemble is currently in beta stage and is very much work in progress. Now that the generic architecture and framework have been implemented, the team will focus on fleshing out the many details to make each component as flexible, powerful and visually appealing as possible. Expect significant improvements over the next few months - your feedback is appreciated in the meantime. Yet, our customers are not limited to the built-in components. We do provide a documented client-side API that allows anyone to add new Flex/ActionScript components. Furthermore, it is possible to drive parts of the application using server-side SPARQLMotion scripts.

Using Ensemble as the foundation for custom rich internet applications means that you can re-use large chunks of client-side components as well as server-side infrastructure. This infrastructure is well aligned with Composer and its features, such as the various database back-ends. Instead of re-inventing the wheel, you can focus on building the ontologies for your particular domain, and put them to work in your team.

TopBraid SPIN API now Open Source

2009-03-07T10:40:00.000-08:00

Since the release of SPIN a few weeks ago, we already had several requests from people who would like to integrate SPIN functionality (SPARQL-based constraint checking, inferencing, user-defined functions) into their own applications. In order to encourage the wider adoption of SPIN in the community, we have therefore decided to make the key features of our SPIN implementation open source.

The TopBraid SPIN API is built on Jena and provides the following features:

Converters between textual SPARQL syntax and the SPIN RDF Vocabulary
A SPIN-based constraint checking engine (via spin:constraint)
A SPIN-based inferencing engine (via spin:rule and spin:constructor)
Support to execute user-defined SPIN functions using Jena/ARQ
Support to execute user-defined SPIN templates

The license has been selected to allow open source projects (from universities etc) to use SPIN without further complications. For closed source users, we offer a commercial license that also provides business users assurance and support. With this policy we hope to encourage researchers to provide their (open source) implementations back to the community to help the SPIN community grow.

The SPIN API is currently in beta, and the official 1.0 release is scheduled in conjunction with TopBraid 3.0 in the next few weeks. We appreciate your feedback in the meantime.

The Object-Oriented Semantic Web with SPIN

2009-01-18T10:06:00.001-08:00

We have recently introduced the SPARQL Inferencing Notation (SPIN) as a mechanism that uses SPARQL to formalize the meaning and behavior of Semantic Web concepts. SPIN provides a light-weight RDF vocabulary that can be used to attach rules (SPARQL CONSTRUCT queries) and constraint checks (ASK queries) to class definitions.

The design of SPIN was driven by requirements that we (at TopQuadrant) have collected from customers from "real-world" semantic technology projects. For many of those customers, the current stack of Semantic Web languages is simply not enough. OWL has been designed for certain use cases, but it fails to meet other requirements due to its open-world assumption. As Stefan Decker points out in the entertaining "An OWL 2 Far?" panel at the 2008 ISWC conference, OWL cannot even be used to check whether an instance of a class meets the cardinality restrictions. In the same panel, Tim Finin argues that OWL does not need to be the only language on the Semantic Web stack, and that instead families of languages are needed depending on the use case. So while OWL is great for some modeling tasks, there is room (and demand) for other modeling paradigms.

In this article I will introduce SPIN from another angle to illustrate what makes it different, and why I believe that it provides good solutions to many real-world requirements. My main point is that SPIN is borrowing good practices from object-oriented programming and modeling languages and integrates object-oriented techniques with the flexible architecture of Semantic Web to produce a new way of working with linked data.

As a "Hello World" example let's start with a simple ontology about geometric shapes. It defines a class Rectangle with the following characteristics (screenshot from TopBraid Composer 3):

In this RDF/SPIN ontology, a Rectangle is a class that has three properties. The values of width and height are specified by the user, and a SPIN rule is used to compute the value of the area property by multiplying width and height. Let's have a look at the rule first. A SPIN rule is a SPARQL CONSTRUCT query that has been attached to a class using the spin:rule property. The CONSTRUCT query can use whatever feature of SPARQL it wants, for example to do mathematical calculations.

A key contribution of SPIN is to introduce a mechanism that allows users to organize those SPARQL queries in a natural, object-oriented way. SPIN rules are not just plain lists of rules like in comparable rule languages (SWRL etc). Instead, the "procedural attachment" of SPIN means that you can arrange the rules in the class hierarchy where they belong. This follows the OO principles of abstraction and encapsulation. Since the rules (and constraints) are attached to classes, any human or agent who looks at the ontology can quickly understand the meaning of the classes and properties. Furthermore, the rules are "scoped" so that tools are better guided when they need to execute the rules and constraints. For example, TopBraid Composer's SPIN engine comes with a mode in which it does incremental inferencing. This means that whenever someone changes the values of width or height, then the value of area will update automatically, as shown with the example instance below:

Now let's extend the example and introduce a sub-class of Rectangle called Square. Squares inherit all characteristics of Rectangles, but they are constrained in so far that width and height of a Square must be equal. This fact is expressed using a SPIN constraint as shown below:

The constraint is an ASK query that must evaluate to false for all instances of the associated class (Square). Again, any SPARQL graph matching pattern can be tested here, and the dedicated variable ?this is used to access the current instance as a starting point. Here is an example instance of the class Square:

You can see that editing tools can use the constraint definitions to verify user input, and to provide warnings (yellow markers) if values violate a constraint. Since the SPIN constraints are scoped to the class, these tests can be performed very efficiently, only on the instance that the user is currently looking at. In TopBraid Composer, constraints can be incrementally tested after each editing step.

The image above also illustrates another object-oriented aspect of SPIN, namely inheritance: The class Square is a sub-class of Rectangle and thus also inherits the rule that calculates the area from width and height. The same inheritance mechanism applies to constraints, and sub-classes can further constrain or specialize the meaning of its parents.

Now let's look at the meta-modeling capabilities of SPIN, and its closed-world semantics. Previous articles on this blog introduced SPIN Functions and SPIN Templates. These are really powerful mechanisms that allow anyone to create their own modeling vocabulary. SPIN Functions are SPARQL functions that can be used in FILTER or LET statements. SPIN Templates are re-usable SPARQL queries that can be instantiated with parameters. In particular you can use pre-defined templates instead of typing in SPARQL queries by hand. SPIN comes with a library of such pre-defined functions and templates to support common modeling patterns. One of the templates from this library, called spl:Attribute, can be used to link a class with a property, specifying min/max cardinality, value type and default value all in a single place. Here is an example attribute definition from the Rectangle class:

This attribute definition is an instance of the spl:Attribute template, attached to Rectangle via spin:constraint. Its equivalent in OWL would be a collection of three owl:Restrictions attached to the class via rdfs:subClassOf. However, if we look at the definition of the spl:Attribute template, we can see that the semantics are different:

The query above is the body of the spl:Attribute template. When used as a spin:constraint, the system will report constraint violations whenever an instance is found that violates one of the three conditions specified in the query. Among then, is the test for the minimum and maximum number of values: if a Rectangle has less than one or more than one value for width or height, then the system will report an error. If a SPIN template is found, the engine will substitute the occurrences of the argument variables (such as ?predicate and ?minCount) with the provided arguments. In other words, instead of using the template spl:Attribute, the class Rectangle could also have the following constraint, encoded directly as a SPARQL ASK query:

However, such SPARQL queries are complicating and hard to maintain, so that wrapping those queries into re-usable SPIN templates is a far better approach. Again, the principle of object-oriented encapsulation is put to use in SPIN. The user of a template does not need to understand all the low-level details of the underlying SPARQL. Similarly, a user does not need to understand the detailed definitions of SPIN functions (such as spl:objectCount and spl:instanceOf shown above), but instead he or she can simply pass in the arguments and leave the details to the engine. Using the SPIN templates mechanism, even the source code becomes easy to read:

I have uploaded the complete example source code of the spinsquare.n3 file. You will notice that SPIN uses a triple-based RDF representation of SPARQL to overcome the limitations of a pure textual syntax.

To summarize, SPIN applies concepts from object-oriented languages to the Semantic Web. The meaning and behavior of classes can be formalized in a machine-executable way using rules and constraints, and those rules and constraints are scoped locally to aid the execution engine in achieving best performance. Furthermore, SPIN can be used to define new modeling vocabularies, such as attribute definitions with closed-world semantics. If other semantics are needed, then they can be represented as well, as I have shown in a previous posting on expressing OWL 2 RL in SPIN.

The expressive power of SPARQL and its wide adoption in industry means that languages like SPIN could go a long way to extend the reach of Semantic Web technology into areas where other languages fail. SPIN combines a powerful rule language with meta-modeling mechanisms including templates and functions. Instead of limiting ourselves to a single modeling paradigm or language, SPIN makes it possible to create task-specific new languages, or domain-specific services such as unit conversion. All those SPIN-based languages are completely self-describing and thus only require a single unified execution engine. Thinking about this on the Semantic Web scale means that linked data can be published and used in much more powerful ways than currently envisioned.

SPARQLpedia as an Example SPARQLMotion Web Application

2009-01-09T09:36:00.001-08:00

TopQuadrant has recently launched SPARQLpedia, a new web service that allows users to share SPARQL queries and to search for queries that others have submitted. The submitted queries are managed on a server-side RDF database together with searchable metadata such as author and submission date. Here is a screenshot of a simple SPARQLpedia web search interface:

Pressing the Search button will display a list of matching database entries:

And the user can click on any of the search results to display details (and execute the selected query):

The underlying services of SPARQLpedia can also be called as (REST-based) web services, as described on the API page.

In a sense, SPARQLpedia is a typical web application:

The application's designers have prepared a database to store the entries
Users can add or delete entries from that database
Users can search for entries in the database by various criteria
A HTML web interface can be used to interact with the database
It can also be accessed programmatically via web services

In this blog article, I will give some details on how SPARQLpedia was implemented and highlight the role of SPARQLMotion as its server-side scripting language.

sparqlpedia.org hosts a standard Apache Tomcat server that runs a TopBraid Live 3.0 beta application. Installing this server was straight forward and basically included dropping the TopBraid Live war file into the tomcat applications folder. TopBraid Live itself is a generic application development framework that makes tons of RDF/OWL related services available through its APIs. There are two server-side APIs in TopBraid Live:

The SPARQLMotion API is an entirely model-driven way of creating web services based on the collection of services wrapped as SPARQLMotion modules.
The TopBraid Live Java API can be used to access and extend the capabilities of the server. In our example, we only needed it to add some new specialized SPARQL functions.

Setting up the Database

The first step of developing SPARQLpedia was to set up a database with an associated base schema. We are using TopBraid Composer for this purpose. The entries in the database are themselves SPARQL queries (but it's easy to translate this to databases hosting product data, academic publications, medical records or whatever). SPARQL queries are entered as strings, but the database stores them in the SPIN RDF Syntax, because this will later allow us to run sophisticated queries on various aspects of the query that would be difficult to achieve if we only had the string representation. So, as a start, we have defined an empty schema ontology that imports the SPIN (sp) namespace. The rest of our database schema is simple: we store Entries that have been submitted by Users, as illustrated in the class diagram below.

The schema is a collection of RDFS classes and RDF properties. The class spedia:Entry only has a single subclass spedia:QueryEntry, but we may want to add additional types of entries later, such as discussion threads or votes. The schema is stored in a file spedia.owl in our Eclipse workspace.

Next we create a persistent database that will contain the submitted instances. For the simplicity, we use the Sesame 2 native Java back-end in TopBraid, but we could have also used any of the other database types supported by TopBraid, including AllegroGraph and Oracle. Our Sesame database imports the spedia.owl from above and its files will be also stored in the workspace. We give it the base URI http://sparqlpedia.org/public, so that we access it later as a named graph from our SPARQLMotion scripts.

Setting up the SPARQLMotion scripts

SPARQLMotion is a visual semantic web scripting language that can be used to build data processing pipelines through a graphical user interface. Typical SPARQLMotion scripts take some input, do some processing and then create some output. TopBraid Composer 3, Maestro Edition is used to build SPARQLMotion scripts, so everything we do (from schema definition and database maintenance to the implementation of the services) is done within a single uniform environment. Let's have a look at an example script, the outline of which is shown in the following screenshot.

This SPARQLMotion script (stored in an OWL file deleteQuery.sms.n3 in the workspace) implements the functionality to delete a query from the repository. The script takes two arguments as input:

The uri identifying the query that shall be deleted
The password of the submitting user - in SPARQLpedia only the original author of a query can also delete it

SPARQLMotion scripts should be laid out (and read) from top to bottom, i.e. you see input coming in from the top, then the input will be processed through a pipeline and finally some results are returned. Each node in the diagram is of a certain module type, and the SPARQLMotion modules library provides a comprehensive list of frequently needed data processing tasks. The deleteQuery script has two exit points, marked by the two red icons at the bottom:

The left end module is used when the script is called via the web service API and just returns the string "OK" as its result.
The right end module is used when the script is called to render an HTML page.

Both end modules have the same type sml:ReturnText but return different mime types. The rest of the script is the same in both cases, i.e. the arguments and the steps to perform the actual deletion are used independently of whether we use the web service API or the HTML call. The web services themselves are declared as subclasses of spin:Functions as shown below:

The class spedia:deleteQueryBase is an "abstract" base class of the two different services and defines the arguments with the usual SPIN function syntax:

The two non-abstract subclasses of deleteQueryBase "inherit" the argument declarations but also point to the SPARQLMotion module in the script that creates the result. For example, the web service deleteQueryHTML has the module ReturnHTML as its return module:

Once the function has been stored in the workspace, it is accessible through a REST URL call, such as http://sparqlpedia.org:8080/tbl/server/tbl/servlet?action=sparqlmotion&id=deleteQuery&uri=...&password=... At development time, we can run the same script within TopBraid Composer ME by hitting localhost:8083 instead. Or we can debug the script manually through the debug button in the graph view of TBC. This has the benefit that we can look at each intermediate step of the script and inspect the state of the triple store and variable bindings with a few mouse clicks.

A Closer Look at the SPARQLMotion Script

Let's walk through the example script from above. The script starts with two Argument modules, which are placed automatically by TopBraid based on the function definition. Technically, these are the same instances of spl:Argument as shown as spin:constraints on the deleteQueryBase class. Here is a screenshot of the form for the password argument:

Each argument can declare a value type that can be used for error checking, and will be used to transform the REST arguments (always strings) into the correct kind of RDF literals or resources. All downstream modules of the SPARQLMotion script can now access the value of the password argument as a SPARQL variable called ?argument. The other argument of the service is accessible as ?uri. The next module Check entry exists is of the type sml:AssertTrue (new in 3.0) and simply verifies that the provided URI is in fact a valid entry in the database:

When this module is executed, the specified ASK query will be executed. If the query returns false, the script will exit with an error message. The error message in constructed from the template given as sml:text, in which {?uri} is substituted with the actual argument value. But wait, which triple store does this query run on? If you scroll up to the script's overview you can see that the Check entry exists module also has the Connect to DB module as one of its predecessors. As usual in SPARQLMotion, the triples represented by the predecessors will be visible in the queries downstream. As shown in the following picture, Connect to DB just opens the Sesame database:

The module above connects to the database via its base URI, so we could later replace it with some other kind of database with the same base URI. Just in case our Sesame DB would explode in size... Ok, by now we have reached the stage where we have verified that the provided ?uri is in fact a valid instance of spedia:QueryEntry in our database. Next, let's validate the password. Another sml:AssertTrue module is used for that purpose as shown below.

They key aspect of the Check password module is an ASK query that gets the spedia:User object that has submitted the query with the given ?uri and then checks whether this user has the provided ?password. It throws an error if the password in the database does not match. Once all those tests are passed, the script can go on with the actual delete. The Delete entry module is an instance of sml:PerformUpdate that runs a SPARQL update call deleting all triples that have the given ?uri as their subject. Now that the query is gone the script forks, depending on how it was called. Assuming the script was called to return an HTML page, it continues with the right branch and the module Return delete query result will be ignored. The resulting HTML page will look like the following:

In order to produce this HTML page, the end result module uses a template string with the basic HTML outline. Only the URI string is different each time, the rest is static. The Return HTML module looks like the following.

The HTML page itself is encoded as a template into which {?uriString} and {?footer} will be inserted at execution time. An alternative way of creating HTML pages from a template is via the Semantic JSP support in TopBraid and SPARQLMotion (not shown here).

The outcome is then sent back to the client using text/html mime format. The footer is re-used in several HTML pages in our system, so we import it from a file:

To complete the script, a small detail is that we want to display the ?uri as a full string and therefore insert a string conversion module before we insert it into the template:

That's it! The script is now finished and ready to be used, assuming the deleteQuery.sms.n3 file has been uploaded to the TopBraid Liver server's workspace.

The other services are also implemented using SPARQLMotion scripts:

submitQuery takes a query string, comment, user name and password as arguments and uses SPARQL INSERT queries to insert those as QueryEntries into the database.
findQueries takes a namespace, resource URIs or a user name as arguments and then runs a SPARQL CONSTRUCT query to create an RDF response. This response is optionally rendered into a HTML table. To gain best performance, the SPARQL query string is assembled dynamically based on the input from various clause templates.
renderQuery takes a query URI and creates a pretty HTML page from it.

Summary

This example shows how the TopBraid Live platform and its SPARQLMotion support can be used to implement scalable public web services and HTML-based internet applications. SPARQLMotion can be used to define almost arbitrary REST-based web services. Deployment of those services together with the ontologies and triple stores they operate on is fairly simple. The scripts and the ontologies can be defined and tested using TopBraid Composer ME.

Please be aware that the approach shown here covers just one aspect of the TopBraid Live platform. Another approach for developing user interfaces is via TopBraid Ensemble. Version 3.0 (coming soon) is a complete framework for building rich Flex-based interfaces from configurable components. More on this some other day...

Understanding SPIN Templates

2009-01-08T15:34:00.000-08:00

The SPIN Modeling Vocabulary defines a mechanism to encapsulate SPARQL queries so that they can be reused in different contexts: SPIN Templates. A SPIN Template is basically a canned SPARQL query that is parameterized with arguments. In this blog entry, I will walk through an example SPIN template to explain how templates are defined and used. In the end of this article I will also boldly explain why I believe that SPIN is potentially one of the missing links in the Semantic Web puzzle, and a disruptive killer technology.

If you haven't done so yet, please read the previous posting about Understanding SPIN Functions. SPIN Functions and Templates basically use the same mechanisms - the main difference is that templates are more general.

We will again use the kennedysSPIN ontology that is shipped with TopBraid Composer 3.0. This example ontology defines various family relationships, including parent, grandFather and grandMother. The example uses a template called InferGrandParent to derive the values of the two grand parent properties from the values of parent and gender.

Let's first have a look at how these relationships could be inferred using plain SPARQL queries. In the following screenshots, we have attached two SPARQL CONSTRUCT queries as spin:rules to the class Person.

The upper rule computes the values of grandFather while the lower rule infers the values of grandMother. We can see that both rules are almost identical: both queries walk up the parent relationship two steps and then check the gender of the grand parent. The only difference are the pairs grandFather/male and grandMother/female, but the basic structure of the queries is the same. SPIN templates can be used to generalize such query patterns so that they can be reused in a more maintainable way. In our example, we will introduce a template that allows us to replace the two individual SPARQL queries, so that the rules look will like in the following screenshot.

In order to get there, we first introduce a template called InferGrandParent, the definition of which is shown below.

A SPIN template is a class that has the metaclass spin:Template as its type. I recommend creating a subclass of the system class spin:Templates to keep your templates organized in the class tree. A template should have a comment describing what the template does and can have any number of arguments. These arguments are similar to the arguments of SPIN Functions described earlier; the main difference is that the arguments of templates are unordered and can point to any property instead of being limited to sp:arg1, sp:arg2 etc.

In our example template, we need two arguments:

the gender that we are matching against (male or female)
the predicate that we want to infer (grandFather or grandMother)

The values of those arguments will be inserted into the body query of the template at execution time. Let's take the rule to infer the grandFather relationship as an example. The instantiated template is an instance of the template class in which the values for gender and predicate are filled in:

When executed as a SPIN rule, the template call above will be substituted with the body of the template, so that we get:

So instantiating the template has the effect that the variables ?predicate and ?gender have been replaced with the arguments; in this case: grandFather and male. The same template can now be reused for the pair grandMother and female.

To summarize, SPIN templates are a powerful mechanism to put SPARQL queries into a (black) box so that you or others can reuse a piece of behavior, even without having to worry about the specific details of the query. Instead, they can use a simple form-based interface to "fill in the blanks" and let the system do the low-level SPARQLing.

Taking this idea further, SPIN templates are a mechanism to create arbitrary new ontology modeling languages. For example, I have shown how you can create templates that encapsulate unit conversion expert knowledge. Another example was the re-definition of OWL 2 RL using SPARQL, although the latter did not really use template arguments because all rules were global. The SPIN Standard Modules Library is another example of very generic templates for tasks such as cardinality constraint checking. But these examples may be just the tip of the iceberg. There could be a marketplace for SPIN template libraries consisting of domain-specific modeling languages with executable semantics.

Instead of hard-coding a pre-defined collection of modeling constructs such as OWL with its system vocabulary owl:Restriction, owl:FunctionalProperty etc, SPIN is an extensible framework for building your own modeling language. There is no need for SPIN-aware tools to hard-code the semantics of any specific modeling language - all they need to understand is how to execute SPIN templates, and the rest is completely driven by whatever is encoded in the particular ontology. SPIN users are not restricted by the vocabulary that any particular W3C committee has selected for them. Instead the democracy of the Semantic Web user's community will select the ideal set of those constructs that are really needed in practice.

Understanding SPIN Functions

2009-01-08T13:57:00.000-08:00

We have recently introduced SPIN, a light-weight vocabulary that enables the use of SPARQL to define constraints and inference rules for semantic web models. One very powerful facet of SPIN is that it can be used to define new SPARQL FILTER and LET functions without writing a single line of programming code. The goal of this blog entry is to explain the mechanisms of user-defined SPIN functions using an example from the kennedysSPIN ontology that is shipped with TopBraid Composer 3.0.

Let's look at an example function called getFather that returns the father of a given person:

Each SPIN function is an instance of the metaclass spin:Function. The best way to create a new function is to create a subclass of spin:Functions, a system class that groups together all available functions in the class hierarchy. The name of the function defines the URI under which is will be accessible from SPARQL queries. The function getFather is in a file with the default namespace, i.e. no prefix is needed to call it. The following example shows a function call of :getFather() in the LET assignment.

You can see that the getFather function takes one argument. Each SPIN function must formally declare its arguments. For each argument, the function class must have a spin:constraint that points to an spl:Argument object. For each argument of the function, the spl:Argument specifies:

a comment describing the argument
the name and index (e.g., sp:arg1 for the first argument)
the value type (such as xsd:string)
whether it is optional or not
a default value in case it is optional

The easiest way of creating such an Argument definition in TopBraid is via drag and drop: locate sp:arg1 in the Properties View and then drag it over the spin:constraint label to instantiate a spl:Argument template. Here is the definition of the first (and only) argument of getFather in TopBraid:

The fact that those arguments are attached using spin:constraint might be confusing at first. You can ignore this and simply treat it as a convention that SPIN uses to look up which arguments are defined for a function. A technical explanation is that function calls are instances of the function class, and that those instances must fulfill the value type constraint encoded in the spl:Argument. spl:Argument is a SPIN template that encodes this value type check by means of another SPARQL query. But again, you can ignore this aspect.

But now let's have a look at the key aspect of a SPIN function definition: the function's body. The property spin:body links a function class with a SPARQL query. This is the query that will be executed whenever the function is called. The SPARQL query must be either an ASK query or a SELECT query.

If the body is an ASK query, then the function's result is true or false
If the body is a SELECT query, then the function's result is the first variable binding of the result variable in the SELECT clause. All other values will be ignored. Null will be returned if no binding exists.

In the getFather function, the body is a SELECT function, because we are interested in a specific instance of person, bound to the result variable ?father:

As you can see in the screenshot above, the body query refers to the variable ?arg1 and TopBraid Composer displays those variables in bold face. At execution time, ?arg1 will already have a value pre-assigned to it, namely the function's argument. In the following function call, ?arg1 is bound to the instance kennedys:JohnKennedy.

Now, when the SPIN engine executes the getFather function, it will pre-bind the ?arg1 argument so that it becomes:

So to summarize, SPIN functions are wrapped SPARQL queries that contain references to argument variables such as ?arg1 and ?arg2. These argument variables will be pre-assigned to the actual arguments at execution time.

Once you have defined your function, you can store them in a file ending with .spin., e.g. myFunctions.spin.n3 and you will be able to use them anywhere in TopBraid without even importing the function's file. This allows anyone to create and share libraries of functions, for example to do generic things such as unit conversion.

These very same mechanisms are also used to define SPIN templates, but this is another topic...

SPARQLpedia: Sharing Semantic Web queries on the Semantic Web

2009-01-07T13:11:00.000-08:00

Together with the TopBraid Composer 3.0 beta, TopQuadrant is also launching a new web service called SPARQLpedia. This is a free, public service that hosts SPARQL queries in a searchable repository. Anyone can submit new SPARQL queries. Anyone can search the repository for queries that mention a given namespace, certain resource URIs or have been submitted by certain users. All submitted queries that mention a SPARQL endpoint (FROM clause) can be conveniently executed online with a Run Query button.

The SPARQLpedia repository can be accessed in three ways:

as a (REST-based) web service from any application
via an HTML web interface at http://sparqlpedia.org
using submit and query buttons in TopBraid Composer

Here is a screenshot of the web interface displaying an example query from the repository:

Here is a screenshot of the new SPARQLpedia search view of TopBraid Composer. It shows how users can conveniently store a library of their favorite queries and use the queries from this library with a few mouse clicks.

There are several motivations for us to implement this service. First, I believe that many SPARQL queries are reusable in one way or another - they may demonstrate design patterns or ask common questions. There is a lot of useful knowledge encoded in queries. In the spirit of modern web applications, SPARQLpedia provides a mechanism to share and reuse queries on a global scale.

Second, SPARQLpedia serves as an example application to demonstrate how our TopBraid Live platform and SPARQLMotion can be used to implement RDF-based services. I will provide details on the implementation of SPARQLpedia and its use of SPIN and SPARQLMotion in a follow up posting. Since SPARQLpedia is implemented using SPARQLMotion, any company with a TopBraid Live license can also host their local SPARQL repositories inside of their firewalls to share frequently used queries among team members.

SPIN Box: A SPARQL-based Computer Game Engine

2009-01-06T12:15:00.000-08:00

Here is another example video (5 minutes) illustrating what you can do with SPIN. TopBraid Composer 3.0 beta comes with a SPARQL-based computer game engine called SPIN Box. This engine allows ontology developers to create new kinds of computer games without having to hard-code anything in a programming language like Java. Instead, the engine runs SPARQL CONSTRUCT rules (using SPIN) to determine the behavior of each field in the game.

The game may appear silly but the underlying message is a powerful one: the game demonstrates how Semantic Web standards and tools can be used for model-driven application development. The SPIN framework can be used to create domain models that have executable behavior attached to them. Being based on RDF and SPARQL, SPIN files can be shared online and re-purposed for different use cases. For example, anyone can extend the behavior of a computer game by introducing new types of objects. Theoretically such applications can even use other (Semantic) Web resources or background knowledge to drive their behavior.

While traditional applications operate on closed worlds, the Semantic Web has been designed to be open and linkable. Frameworks such as SPIN open an application's architecture and can provide an unprecedented level of dynamic behavior and flexibility. The box is now open.

Video: SPARQL-based Unit Conversion with SPIN

2009-01-05T17:32:00.001-08:00

Here is a video on another example application of SPIN. This time we are using SPARQL rules to convert values between units of measurement. In the example we convert from centimeters to feet and cubic meters to cubic feet. Two solutions are presented: The first is using SPARQL CONSTRUCTs that have the conversion factors encoded as part of the spin:rules. The second solution is much more generic and uses a NASA units ontology (developed by TopQuadrant) to dynamically find the appropriate conversion factors. For this second solution, the video also shows how to define new SPARQL functions using SPIN, and how to use SPIN templates to encapsulate reusable queries. The video is a step-by-step tutorial and takes about 16 minutes to complete. If you haven't done so yet, you should have a look at the TopBraid Composer SPIN page to get some background and screenshots about SPIN functions and templates.

I believe this is a fine example of what can be achieved if Semantic Web technology is applied the way that it should: to share data models and knowledge together with executable semantics. SPIN might be a missing link in the Semantic Web stack, because it allows users to create and share domain-specific modeling languages that encapsulate all the background needed to build useful applications out of them.

OWL 2 RL in SPARQL using SPIN

2009-01-05T15:43:00.000-08:00

The evolving OWL 2 standard comes with a profile called OWL RL. According to the OWL 2 RL W3C page "the OWL 2 RL profile is aimed at applications that require scalable reasoning without sacrificing too much expressive power. It is designed to accommodate both OWL 2 applications that can trade the full expressivity of the language for efficiency, and RDF(S) applications that need some added expressivity from OWL 2. This is achieved by defining a syntactic subset of OWL 2 which is amenable to implementation using rule-based technologies".

This means that there will soon be a well-defined specification that expresses the semantics of a good subset of OWL in a format that can be handled by rule engines. Many OWL implementations (such as OWLIM or Jena) have already used such a rule-based approach for ages and in many cases their performance is much better than with tableaux-based OWL implementations.

Based on its CONSTRUCT keyword, SPARQL can also be considered to be a rule language. SPIN is a new SPARQL-based vocabulary that we have recently introduced with TopBraid Composer 3.0 beta. SPIN can be used to encapsulate reusable SPARQL queries as templates. These templates can then be instantiated in any RDF or OWL ontology to add inference rules and constraint checks.

Mostly as an exercise and a proof-of-concept, I have converted the OWL RL rules into SPIN templates. The SPIN library at http://topbraid.org/spin/owlrl now contains the complete OWL 2 RL specification in executable form, formalized in SPARQL CONSTRUCT rules.

The following screen shows the example OWL RL rule cax-eq2 encoded as a SPIN template:

The example above implements the rule that if c1 has been declared to be owl:equivalentClass of c2 and x is an instance of c2, then x is also an instance of c1. In order to activate this type of inferencing in your model, you just need to instantiate this template as a spin:rule at owl:Thing. Or simply import the file http://topbraid.org/spin/owlrl-all, which activates all rules for all instances of owl:Thing.

This mechanism can not only be used to fine-tune the inferences for a specific model, but also to enhance the expressivity of other inference engines. TopBraid Composer allows users to combine inference engines, e.g. to run SPARQL rules on top of Jena inferencing. By activating a couple of rules from the OWL RL library, you can use some new OWL 2 keywords such as owl:key or owl:propertyChain in your model. And you can do all this yourself - just write your own SPIN rules to implement your own domain-specific modeling language and then share them with the others on the Semantic Web!

Note: the OWL RL SPIN library above is basically untested but I would appreciate bug reports or other suggestions on how to improve it. The conversion of most rules was straight-forward, but there were a handful of rules that were tricky to convert to SPARQL, especially those with "for-all" semantics. In one case (owl:key) I had to introduce a user-defined SPIN function to negate a list traversal. In some other cases I had to rely on built-in Jena functions such as list:member. For property chains, I only implemented chains with the length of two, e.g. the infamous "uncle" relationship. However, other lengths can easily be added if they ever become relevant in practice.

The following example uses the OWL RL property chain rule to infer the uncle relationship between two instances. The source code of the corresponding OWL file (in N3 is below, updated for the latest OWL 2 version from June 11, 2009).


@prefix spin:    <http://spinrdf.org/spin#> .
@prefix xsd:     <http://www.w3.org/2001/xmlschema#> .
@prefix owlrl:   <http://topbraid.org/spin/owlrl#> .
@prefix rdfs:    <http://www.w3.org/2000/01/rdf-schema#> .
@prefix rdf:     <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix owl:     <http://www.w3.org/2002/07/owl#> .
@prefix :        <http://topbraid.org/spin/owlrl-test#> .

<http://topbraid.org/spin/owlrl-test>
 a       owl:Ontology ;
 owl:imports  ;
 owl:versionInfo "Created with TopBraid Composer"^^xsd:string .

# Instantiate SPIN template with property chain semantics
owl:Thing
 spin:rule
         [ a       owlrl:prp-spo2-2
         ] .

:Person
 a       owl:Class ;
 rdfs:subClassOf owl:Thing .

:Darwin
 a       :Person ;
 :parent :Holger .

:Holger
 a       :Person ;
 :brother :Thorsten .

:Thorsten
 a       :Person .

:brother
 a       owl:ObjectProperty .

:parent
 a       owl:ObjectProperty .

:uncle
 a       owl:ObjectProperty ;
 owl:propertyChainAxiom (:parent :brother) .

# defines the new OWL 2 property if needed
owl:propertyChainAxiom
 a       rdf:Property ;
 rdfs:label "property chain axiom"^^xsd:string ;
 rdfs:range rdf:List .

Video: SPARQL-based inferencing and constraint checking with SPIN

2009-01-02T17:09:00.000-08:00

Here is a short and sweet 3 minutes video demonstrating how to use SPIN to execute inferences and to do constraint checking on the Kennedy example ontology. The content of the video is similar to what is explained (with screenshots) on the TopBraid SPIN page. The example files are part of the TopBraid Composer 3.0 download, i.e. you can replay the scenario on your own machine.

Introducing SPIN: the SPARQL Inferencing Notation

2009-01-02T09:06:00.000-08:00

With this week's release of TopBraid Composer 3.0 beta1, TopQuadrant is adding new lego bricks to the Semantic Web stack. SPIN is a collection of RDF vocabularies enabling the use of SPARQL to define constraints and inference rules on Semantic Web models. Let me give you some (technical) background on why I believe SPIN will be useful. Future postings will elaborate on use cases and example applications.

One of the main selling points of Semantic Web technology is the ability to publish domain models with executable semantics. Most Semantic Web models contain class and property definitions together with definitions of ranges, domains, OWL restrictions, OWL property types, SWRL rules etc. These formal definitions can be used by any tool that implements the underlying languages to operate on the model even if the tool does not have any hard-coded knowledge about the domain. So if I publish a Semantic Web model stating that all instances of Person can have string values for firstName then Semantic Web tools can build suitable input forms to collect instances. Or, if I include a rule that states that the age of a Person is the current date minus his or her birth day, then any Semantic Web tool can automatically compute the value of age just by executing the rules. Again, nothing needs to be hard-coded and the tool can dynamically discover what a given model is all about. This is also the foundation for various data integration and information discovery tasks.

RDF and RDF Schema only provide very limited expressivity for such definitions, and it has been (intentionally) left to higher-level languages such as OWL to provide richer modeling constructs. However, people quickly recognized that in practice OWL does not meet all requirements and use cases, so that additional languages like SWRL (and recently RIF) have been proposed. These are rule-based languages that contain constructs for IF-THEN conditions which infer new triples when a pre-condition is met in the current state of the model. These rule languages cover very important use cases and many practitioners find them quite natural to use.

Now let's get back to the use cases of rich Semantic Web languages. Typically, people use them for two different purposes:

constraint checking: test whether the model is in a consistent/expected state
deriving new values: compute implicit property values from what's stated in a model

The focus of OWL is on the latter aspect although many people seem to misunderstand its semantics or intentionally simply ignore the open-world assumption and the lack of unique name assumption to use it for constraint checking as well. But this is actually incorrect, and this misuse of OWL for these tasks indicates that other languages are required to fill in this gap.

But the quest for good modeling languages does not have to stop at OWL or SWRL - there is another well-known language in the Semantic Web space that can be used to formalize semantics: SPARQL. SPARQL is a firmly established W3C standard query language and implemented by all major Semantic Web stores on the market. SPARQL is very expressive as it provides means to define matches against almost arbitrary RDF graph patterns in the WHERE clauses. Also, many Semantic Web practitioners are already familiar with SPARQL and various query editing tools exist. Furthermore, SPARQL seems to meet the users' expectations very well with regards to things like the open-world assumption: SPARQL queries only operate on the triples mentioned in the WHERE clause - no other implicit assumptions are used at query execution time. You get what you see.

Most people know that SPARQL has the SELECT query form, but there is also the extremely useful CONSTRUCT keyword and the simple ASK keyword. The SPIN Modeling Vocabulary makes heavy use of the latter two keywords. To simplify a bit, SPIN suggests to use

ASK for constraint checking, and
CONSTRUCT for deriving new values

So, a SPIN-based ontology is a collection of classes and properties plus ASK and CONSTRUCT queries. The question then is: how can we connect those queries to the domain models? How can we store the queries together with the model in a seamless way?

In previous incarnations of SPIN (when it was not called SPIN yet), TopBraid had simply stored the SPARQL queries as strings as part of the domain model. We had used a dedicated property called sparql:query that would point from any RDF resource to a SPARQL string. This approach was of course fairly weak. Relying on a purely textual representation is error-prone, for example when someone renamed a resource the change must also be made to the query string. Also, what about the namespace prefixes used in SPARQL queries.

In order to provide a maintainable representation of SPARQL queries, SPIN defines an RDF vocabulary for storing SPARQL queries. Instead of storing an ASK query as a string, SPIN stores it as an instance of a dedicated RDF class sp:Ask etc. For example, the SPARQL query

    ASK WHERE {
   ?this my:age ?age .
   FILTER (?age < 18) .
  }

can be represented in SPIN RDF syntax in N3 format as

    [ a       sp:Ask ;
               sp:where ([ sp:object sp:_age ;
                           sp:predicate my:age ;
                           sp:subject spin:_this
                         ] [ a       sp:Filter ;
                           sp:expression
                                   [ sp:arg1 sp:_age ;
                                     sp:arg2 18 ;
                                     a sp:lt
                                   ]
                         ])
             ]

This may remind some of you of SWRL, where Semantic Web rules are also triplified or OWL class expressions that look similarly complex in the RDF. The RDF syntax is not necessarily pretty but it's intended to be used by software, not humans. In the case of SPIN, editing tools like TopBraid display these constructs in human-readable SPARQL syntax on the screen. Furthermore, there is a free public web service for converting between the two SPARQL syntaxes.

But the main achievement here is that we are now able to store SPARQL expressions as part of our Semantic Web models, and use SPARQL's rich expressivity to describe the concepts from our domain. The next question is: where do we put those SPARQL expressions? In SWRL, the inference rules have global scope that are simply placed anywhere in the model (as instances of swrl:Imp). In OWL, a frame-based approach is used where logical descriptions are attached to classes using rdfs:subClassOf or owl:equivalentClass. The latter has the advantage of providing some context and scope to the rules, i.e. the ontology designer consciously attaches the pieces of domain knowledge to the classes or properties where they belong to. Inheritance similar to that from object-oriented modeling is used to re-use and specialize those definitions.

SPIN supports both approaches, i.e. rules and constraints can be either global or scoped in the context of a given class. The recommended approach is to attach SPIN declarations to classes, following object-oriented design. Similar to object-oriented languages like Java, there is a special variable called ?this which refers to the current instance. For example, assume you attach a rule that computes the age of a person from its birth day to the class my:Person. If my:Parent is a subclass of my:Person, then the rule will also be applied to all instances of my:Parent. At execution time, the variable ?this will be bound to the current instance (of either my:Person or my:Parent). Global rules are simply attached to the root classes rdfs:Resource or owl:Thing and do not mention ?this.

The result of this mechanism is that SPIN users can exploit the whole range of SPARQL features to make their domain models executable, even on the scale of the Semantic Web. The following postings will provide some examples on how to use SPIN as a rule and constraint language. To summarize where we are so far, SPIN is a very light-weight mechanism that leverages SPARQL for new application areas that go far beyond querying. But there are additional capabilities in SPIN, for user-defined functions and query templates which I will introduce in future postings as well. Please stay tuned...

TopBraid Composer 3.0 goes into beta

2008-12-31T13:57:00.000-08:00

While this blog has been dormant for a couple of months, we have not been idle! This final day of the year marks the first beta release of TopBraid Composer 3.0. This release comes packed with numerous new capabilities and improvements or bug fixes to existing features.

One of the reasons for the long delay between the releases (2.6.2 came out in August 2008) is the fact that we have reorganized the development team. We have hired some more developers and in order to give each of the products of the TopBraid Suite sufficient attention, I am now sharing the team leadership with Jeremy Carroll. Jeremy, who is well-known as one of the original creators of HP Lab's Jena API, is spearheading a major redesign of our TopBraid Live platform and TopBraid Ensemble, and the outcome of these changes will be particularly visible in the second beta of TBC. Another side effect of having Jeremy on board is that I can focus on creating new features again.

One of the major new features in 3.0 is support for the SPARQL Inferencing Notation (SPIN). We have been working on SPIN for many months now and it is already in routine use in several of our own projects. The beta release is the first opportunity for me to expose this to the public. In my on humble opinion, SPIN is a potentially very significant addition to the Semantic Web stack. I will write more about it in follow-up blog entries in the next couple of days.

The work on SPIN also allowed me to revisit the core of the SPARQLMotion engine, and numerous improvements have been included, in particular for the development of Web Services using SPARQLMotion. It's now easier than ever to build complete web applications using SPARQLMotion. TopQuadrant's new online service SPARQLpedia is an example application that is hosted on a public TopBraid Live server that uses SPARQLMotion scripts to manage a repository of SPARQL queries on a Sesame database. I will write more about this in the next few days as well.

There are many other changes and new features in the TopBraid Composer 3.0 beta. Please download it and let us know what you think!

SPARQL Functions in Motion

2008-06-27T09:22:00.000-07:00

SPARQL is well established as the standard query language for the Semantic Web. Comparable to SQL, SPARQL provides the SELECT keyword to extract information out of an RDF/OWL repository. SPARQL also provides the CONSTRUCT keyword to construct new triples from existing ones, making SPARQL an attractive solution to defining ontology mappings or rule bases.

As so often with W3C standards, the official specifications take you 80% to where you really want to be, while the remaining 20% are often non-standard extensions that make the technology really useful in real-world applications. In the case of SPARQL, many implementations already support some form of SPARQL Update Language with keywords such as INSERT and DELETE, leading to de-facto standards that will hopefully be officially folded into the standard in the next iterations. Another extremely useful extension has recently been implemented by Andy Seaborne in the Jena ARQ SPARQL engine: LET Assignments. Here is an example derived from Andy's blog:

SELECT ?area
WHERE {
?x rdf:type :Rectangle ;
:height ?h ;
:width ?w .
LET (?area := ?h * ?w) .
}

The LET command can be used to create new values out of existing values. The syntax of the right hand side of the assignment provides the same expressivity as FILTER expressions, which are well covered by the standard. What makes this LET command so attractive is that it greatly extends the expressiveness of SPARQL, especially when using the CONSTRUCT or INSERT keywords. We can slightly modify the example above to define a rule that automatically infers an area triple from height and width:

CONSTRUCT {
?x :area ?area .
}
WHERE {
?x rdf:type :Rectangle ;
:height ?h ;
:width ?w .
LET (?area := ?h * ?w) .
}

In addition to simple arithmetical expressions such as above, SPARQL also defines a collection of built-in functions such as bound, isBlank, lang, and str. The Jena ARQ library adds many more, including string functions.

As of version 2.6.0, our RDF/OWL development platform TopBraid Suite, which is based on Jena, includes support for LET assignments and greatly extends this mechanism. We have added a comprehensive library of more SPARQL Functions - the SPARQLMotion Functions. Among others, these functions can be used to build URIs from other names, cast values between datatypes, analyze the class structure, extract sub-strings and convert resources into human-readable names. Many more such functions will be added in future versions, in response to the use cases that we encounter in practice. TopBraid Composer provides a convenient auto-complete and context help feature to use these functions as shown in the next screenshot.

Java programmers can use the Jena API to add new functions if the provided functions are insufficient. TopBraid Composer users can also use SPARQLMotion to define new functions and make them available to any SPARQL query. I have recently uploaded an example SPARQLMotion function definition. The following screenshot, taken from this example, shows that the new function takes some input string and extracts the text in parantheses:

Using the declarative visual scripting language SPARQLMotion, average RDF/OWL experts can custom-tailor SPARQL to their individual needs without having to work with Java.

Linking to DBpedia with TopBraid

2008-04-14T22:42:00.000-07:00

The semantic Web is coming. After at least a decade of preparation in its research community, the technology around RDF seems to be finally taking off. Re-branded as a web of linked data, the semantic Web is bootstrapping itself around a growing network of online databases, ontologies, SPARQL end-points, RDFa files and RDF-compliant web services.

A promising central hub in this linked data network is DBpedia, an RDF repository based on Wikipedia. DBpedia provides machine-readable RDF data for each of the pages in Wikipedia. Each Wikipedia page is represented by a corresponding RDF resource, and these resources are associated with RDF property values to provide descriptions, images, cross-references and tons of useful background knowledge. For example, the DBpedia pages for cities (e.g., Canberra) contain geographical information, the number of inhabitants, population density, links to famous inhabitants and average temperatures, all in machine-processable form. While these property values may not be totally stable and reliable, they are at least a good start.

However, the main benefit of DBpedia is that it provides relatively stable URIs for all relevant real-world concepts. This makes it a natural place to connect specific domain models with each other. If I publish my RDF files with links to DBpedia and you do the same, then we can automatically find cross-references and might more easily find mappings between our domain models. All I need to do is to add links such as { my:Canberra owl:sameAs dbpedia:Canberra }.

In order to support linking domain models with DBpedia and to encourage our users to link their domain models into the semantic Web, TopBraid Composer 2.5.3 contains some new features that semi-automatically suggest missing links. We have integrated a Wikipedia web service that takes a string (here, an rdfs:label or a local resource name) and tries to find a matching Wikipedia page for it. From the resulting page, TopBraid can derive the DBpedia page and display it in a Wizard as shown below. The wizard can then be used to preview and assign DBpedia links to one or more domain resources.

I have made a short video about all this.

Extending your tools with SPARQLMotion

2008-03-15T09:51:00.001-07:00

A few days ago I wrote about how to create Web Services with SPARQLMotion. The basic idea is that SPARQLMotion scripts can take parameters as external input and then process these parameters in SPARQL queries etc. With today's release of TopBraid Composer Maestro 2.5.2, we have applied this idea to create a new mechanism that can be used to extend the tool itself.

Here is an example. This simple SPARQLMotion script (SetCreatorService.sms.n3) takes the currently selected resource as input and adds the triple ?resource dc:creator "John Doe" using a SPARQL Update language call.

From top to bottom the steps are:

Take the selected resource as input and bind it to the variable ?resource.
Run the SPARQL call INSERT { ?resource dc:creator "John Doe" }
Done

Download the script and put it into your TopBraid workspace, then open the context menu (Resource menu) of any resource. This will insert a new menu item "Set creator" to execute the script on the selected resource:

When executed, the selected resource will be edited automatically, by the SPARQLMotion script. Needless to say, more complex SPARQLMotion services could be run as well.

The trick is that TopBraid is scanning your workspace for all files ending with .sms.xyz (where xyz might be n3). If these files contain a service that takes the selected resource as input (sml:BindWithSelectedResource), then TopBraid will add a menu item for the corresponding sml:ReturnXYZ module.

Here is another example, that is also explained in depth on our web site (click on the image for details).

This service can actually be used to send an email from TopBraid Composer. It demonstrates that these services can also be supplied with a pre-condition, so that the menu item only shows up for certain resources. This is implemented by means of an ASK query in the sml:BindWithSelectedResource module.

It is easy to see that this feature is potentially yet another disruptive move in the direction of model-driven applications. SPARQLMotion can be used to extend the TopBraid Composer tool itself to provide convenient short-cuts to frequently needed activities. Instead of having to rely on your IT department to write plug-ins in a programming language like Java, users of the tool can now do such things themselves. No more need to learn complex APIs, fiddle with the extension mechanisms, just plain modeling. The tricky parts of connecting the dots are already solved by the SPARQLMotion engine.

Creating Web Services with SPARQLMotion

2008-03-11T21:17:00.000-07:00

A week ago we have officially launched SPARQLMotion 1.0 as part of the latest TopBraid Suite release. SPARQLMotion is a visual scripting language based on Semantic Web standards. The language is particularly useful to automate all kinds of data integration tasks because SPARQLMotion has built-in facilities to merge, map and transform data from various sources and formats. Furthermore, being a visual language, little programming skills beyond SPARQL are required to use SPARQLMotion and its tools.

One of the new features of SPARQLMotion 1.0 is that it can be used to create customized Web Services. TopBraid users can visually define REST-style web services and execute them within Maestro or the TopBraid Live server. I am describing a small example SPARQLMotion web service on our web page, but here is a screenshot of the script for your convenience.

This small SPARQLMotion script takes a calling code such as "61" as input, sends it in a SELECT query to the DBPedia SPARQL end point, and then sends a string response such as "61 is the calling code of Australia." back to the client. (Thanks to Henry Story for a variation of this scenario!)

This particular example only highlights one aspect of the possibilities of SPARQLMotion, namely the ability to create wrappers of arbitrary SPARQL calls. However, imagine that you can also define any sequence of processing steps in the middle (between the green start and the red end module). You could mash up data from multiple newsfeeds, databases, spreadsheets or XML sources, include data from external web services, construct new triples, define iterations and if-then-else branches, apply inference engines, send emails, construct complex web pages using JSPs or other templates, etc. With SPARQLMotion, defining useful Web Services becomes a matter of drag and drop - at least that's our goal at TopQuadrant.

There are many other SPARQLMotion improvements in the latest TopBraid Composer release, and we are incrementally adding examples and documentation - finally also including a user's guide. The tool now also provides a visual script debugger. Realistically there are still some rough edges that demand for improvements, but there are also quite a lot of opportunities to discover in this new semantic programming paradigm. Just make a mark and see where it takes you.

Editing Oracle 11g RDF Rules in TopBraid

2008-03-03T12:15:00.000-08:00

Oracle has been supporting native RDF/OWL support since version 10g of their database. Now in its second release as 11g, Oracle has become a very serious option for projects that operate on large amounts of data. Many of our own customers already have Oracle installed in their enterprise and trust the infrastructure that Oracle provides. However, Oracle does not yet offer complete solutions that would also involve ontology editors, rule editors, semantic data browsers, semantic information integration tools, etc. This is where TopBraid Suite is well established, so that using TopBraid on top of Oracle is an attractive option. In fact, in the last couple of months, the majority of our customers has explicitly asked for this combination.

In response to this increasing interest in using Oracle in conjunction with TopBraid, we have added some native Oracle capabilities to our platform, based on the Oracle Jena API for the connection. In particular, TopBraid Composer 2.5.1 includes a feature to edit user-defined native Oracle rules, and to run server-side inferences. The following screenshot shows TopBraid's basic Oracle rules editor.

With this rule editing support, TopBraid users can select one or more user-defined rule bases and also combine them with the pre-defined Oracle rule sets RDFS, RDFS++, OWLSIF and OWLPRIME. The rule language supported by Oracle includes triple patterns. This is enough for the most common needs like the infamous "uncle" relationship which goes beyond the expressivity of OWL.

The major difference between using Oracle's native rule support and any of the other inference engines built into TopBraid is that Oracle rules are executed server-side. This makes execution significantly faster than executing them inside a TopBraid client. This is because if the rules were executed in TopBraid, then the rule engine would need to fetch lots of query results from the database, process the results and then continue with the next loop. Native Oracle avoids this communication overhead. Furthermore, Oracle rules are executed incrementally on each database commit, making it easier to maintain up-to-date inferences. There are several disadvantages though as well: TopBraid cannot distinguish inferred triples from asserted triples (at least not yet), making it for example unclear which triples can be deleted or not. A more critical limitation though is that the rules operating on Oracle would not "see" any imported triples, or triples that were inferred in other ways. This favors scenarios in which all RDF data can reside in a single Oracle model.

By the way, TopBraid Composer now also has features to execute arbitrary SQL commands and SELECT queries directly on the Oracle database, displaying the results in a console together with other status reports. With these capabilities, TopBraid Composer can be used to set up and maintain Oracle databases including rule bases, so that the Oracle database can be accessed in the desired way by end-user applications (such as TopBraid Live or Ensemble). In a sense, in addition to all its other Semantic Web features, TopBraid is becoming an admin tool for Oracle RDF...

Text Analysis with OpenCalais in TopBraid

2008-02-08T14:06:00.000-08:00

OpenCalais is an amazing Web Service that was recently made publicly available by Reuters. In a nutshell, OpenCalais takes arbitrary text or HTML documents as input and tries to extract semantic web entities from it. For example it can identify persons, companies and countries and returns them as machine-readable RDF data structures. Needless to say this extraction does not work perfectly well because understanding human languages requires (artificial) intelligence and a lot of implicit background knowledge. In any case it can create astonishing results. We ran it over the TopQuadrant management web site and it correctly identified all five people, their respective roles as well as parts of their former companies.

Such text-entity-extraction services have in the past been (very expensive) niche products. The recent announcement by Reuters (who have acquired the text mining company ClearForest last year) to make OpenCalais available for free came as a great surprise. After all, many customers of ours have requested features to import text into their ontologies in the past. Given all this, it was an obvious next step for us to include OpenCalais into TopBraid. TopBraid Composer 2.5 now includes several features that seamlessly integrate the Calais web service into data processing tasks. For example, you can extract RDF from arbitrary HTML files from the web and save the results into files. Or you can put .txt files into your workspace and directly import them into some other RDF/OWL project - Calais will be called automatically.

However, the real power of OpenCalais is exposed when used in data processing pipelines such as SPARQLMotion scripts. The following TopBraid screenshot shows a SPARQLMotion script that

loads the latest business news from a New York Times RSS feed
sends the text of the news items to OpenCalais (OpenCalais will identify all countries mentioned in the news)
iterates over all countries to request their geo coordinates from the geonames web service
displays all countries on a Google Map

This script is of course just one possibility of using information delivered by Calais. A more comprehensive solution would probably include a countries ontology that already has background information (including coordinates, capitals, financial details) about each country. Then SPARQLMotion could be used to create an intelligent agent that analyzes newsfeeds (or any other textual data source) against semantic query patterns such as "Alert me if there are any news about a company merger located in an oil-exporting country". If you want to play with all this, please download TopBraid Composer Maestro 2.5.0 but keep in mind that SPARQLMotion is work in progress and not complete (Matt Fischer recently wrote an independent review of an even older version of SPARQLMotion that illustrates some of the open issues).

Note that OpenCalais seems to be part of a larger roadmap at Reuters aiming at making "all the world's content more accessible and valuable". It is great to see a world-leading information company embrace the Semantic Web vision so directly! As a comprehensive information integration and ontology design tool, TopBraid Composer and its SPARQLMotion language seem to be ideal platforms to process, analyze and visualize the information that OpenCalais delivers.

Beyond Social Networking with FOAF and TopBraid

2008-02-08T10:02:00.000-08:00

The recent announcement that Google is now systematically scanning FOAF files is potentially a large leap toward building the Semantic Web as a linked network of distributed data sources. FOAF files contain personal and work-related information such as name, acquaintances, publications, projects and contact information in RDF/OWL format. In contrast to most of the well-known commercial social networking services, FOAF files are maintained in a decentralized network, in which each user can publish and edit his or her own profile without being locked into any vendor's private database.

I personally haven't followed the FOAF project well in the past, but Google's announcement triggered me to have a second look. While there are certainly several historical and questionable design decisions in the current FOAF ontology version, it is nevertheless a very important domain model because it is widely used. In my opinion, the best thing about FOAF is that it defines stable URIs for concepts such as Person, name, mbox and img. Even if we don't like all aspects of FOAF (e.g. the redundant and inconsistent naming of properties like surname/family_name and firstName/givenname), FOAF at least provides shared URIs that ensure that users to talk about the same things.

Semantic Web technology users can reuse those shared URIs for their own purpose. For example, we can import the FOAF namespace into a project management ontology so that we can reuse and leverage the facts from FOAF files for all team members. For that purpose we may only select to use certain FOAF properties, and we are not forced to use the official specification ontology in all its details.

In our case, I wanted to see how FOAF files look in TopBraid Composer, so that I could update my own FOAF profile that was sitting neglected on the web. I did a few adjustments to the original FOAF spec so that it is less confusing to average users. In particular, I removed a couple of subclass relationships that pointed to other namespaces, removed redundant rdf:types of many properties, and removed redundant domains and ranges that would otherwise clutter up forms with unhelpful widgets. The result is an editor-friendly FOAF which has been made part of TopBraid's standard ontology library as of TopBraid Composer version 2.5.0.

I also did a couple of extensions for TopBraid to better support typical usage patterns of FOAF files. In particular I added a "follow-your-nose" feature that allows users to dynamically import a namespace that is mentioned in a URI. This is needed to explore details of FOAF profiles that are linked via the foaf:knows property. I also added imaging support to the graph editor so that from now on any value of foaf:depiction (and its sub-properties) will be rendered as images. See the TopBraid Composer 2.5 screenshot below.

So in principle you can now use TopBraid to edit your personal profile. This of course isn't cool by itself, but your personal profile alone isn't cool by itself either. The added value is that TopBraid (or similar ontology editors) are generic and this means you can use the full range of components like graphs, forms, maps and query builders as well as the various inference and data processing engines (e.g. SPARQLMotion) on your model. Even more importantly, you can make FOAF models part of other domain models and do things that cross the borders of what could be done with a traditional social networking service. For example, I could link my FOAF profile with a music ontology that contains background knowledge about the style of music that I prefer to listen to.

If Google would notice that a growing number of FOAF files out there also reference related namespaces, then it may want to scan those other namespaces as well. Google may soon find out that instead of maintaining a dedicated internal social networking data structure and providing a specific social networking web service, it will be easier for them to store the original data in a flexible and self-describing format such as RDF/OWL and simply publish a SPARQL endpoint to their API users. FOAF files could therefore be the start of a long friendship...

SparqlMotion: A visual semantic web scripting language

2007-11-15T10:55:00.000-08:00

The open architecture of semantic web languages like RDF, OWL and SPARQL make them an excellent choice for data integration problems, aka mash-ups. Semantic technology tools can be used to bring together heterogeneous data sources, to post-process and filter them, and to query the resulting aggregated data models. One of those tools, TopBraid Composer, provides import capabilities for legacy data in XML, UML, relational databases, spreadsheets, news feeds, HTML pages etc. Users can edit ontologies to bridge the various data items, and run inference or query engines to get information out. However, going through these steps is typically a manual process that needs to be repeated for each new data source.

SparqlMotion is a new visual language that enables average users to define scripts to import, post-process, query and visualize data using semantic web technology. Users can define and share those scripts as OWL models, based on a dedicated SparqlMotion ontology and module library. The graph editor of Composer's Maestro Edition (or any other OWL editor) can be used to define the data and execution flow of these scripts using drag and drop:

Here is a screencam video (15 minutes) that shows how to create the above SparqlMotion script with TopBraid Composer. Here is the example script in N3 notation. The script loads data from a news feed, post-processes the resulting triples, ask the user to enter a keyword, and then displays all events that contain the keyword in a calendar. The output of the script could also be another file, a spreadsheet, a database or a dynamic model that can be imported into other ontologies.

Each of the nodes in the above diagram represents a data processing step, which must be an instance of a SparqlMotion module class such as sml:LoadNewsFeed. The sm:next relationship specifies the information flow between two modules. For example, the resulting output of the newsfeed loader (RDF triples) is used as input for the data type conversion module below it. The latter module can process/filter the RDF input and pass it on to the next node etc. Scripts can branch their data flow and merge RDF input of multiple modules into a single node at any time.

Two information formats are currently supported: RDF and XML. We provide translation modules based on our Semantic XML algorithm that can convert between RDF and XML at any time. In addition to these formats, modules can bind variables. For example, a user input module such as "Enter keyword" above can prompt the user to enter a string and then pass that string literal to the following modules in a variable such as "keyword". Succeeding modules can reference this variable, for example, in SPARQL queries.

SPARQL is the central language of SparqlMotion. Many modules (such as those that display data on a calendar or a Google map) use a SPARQL query to select which resources to display. There is also an iteration module that repeats other modules for each result row of a SPARQL select clause. Finally, SPARQL's CONSTRUCT keyword is used heavily to transform and filter RDF data.

The SparqlMotion modules library is growing rapidly since we started using it in customer projects. We are also working on a web-based graph editor in Flex based on TopBraid Ensemble's graphing capabilities. This will remind some people of Yahoo Pipes. The current version included in TopBraid Composer Maestro is rather alpha software as we better understand SparqlMotion design patterns and add support for best practices. We expect to incrementally roll out many more features over the next few months. In any case, the system is available for download if you want to give it a try. Make sure to watch the video before exploring this exciting space.

Creating documents with SPARQL and JSP

2007-11-14T16:51:00.000-08:00

My co-workers at TopQuadrant recently had a deadline to create some deliverables for a large customer project (for a national space agency :) ). A large fraction of the work in this project is actually ontology design, and the deliverables were Word documents that described and explained these ontologies. Tired of endless hours of manual work, my manager wished he had a "generate-document-from-ontology" button.

On that same day we implemented a prototypical document generator that allows users to embed SPARQL queries into HTML templates, and then let the system insert the resulting variable bindings into certain spots in the templates. The basic idea is that certain blocks of text (such as a row in an HTML table) are repeated for each result row of the SPARQL query. Furthermore, loops over SPARQL queries can be nested so that you can reference variables from the outside in an inner loop.

This turned out to be a really useful feature, so I generalized this to a new tool feature based on a generic document generation framework, Java Server Pages (JSP). Since I had never implemented anything on top of JSP, I did a quick search on the web, revealing David Powell's SPARQL JSP taglib. This small but fine open-source project turned out to be a very helpful starting point for the implementation of TopBraid Composer's new document generation facility. Since TBC's Maestro Edition comes with its own integrated Java web server, we can now execute Java Server Pages within the development tool.

In a nutshell, this feature can be used to create arbitrary text documents (such as HTML or XML files) from the RDF model that is currently open in Composer. The user selects a JSP page (which can be edited with Eclipse/TBC or tools such as DreamWeaver) and then the output file. The system compiles the JSP page internally into a Servlet and then runs it, writing the result into a new file in the Eclipse workspace. With the TopBraid Live server, these pages can also be put online to produce dynamic web pages.

Here is an example JSP document with embedded SPARQL code. When executed over this ontology, you get this output.

BIRT: Creating SPARQL-based charts and reports

2007-11-12T12:23:00.000-08:00

Being based on the Eclipse platform, TopBraid Composer can seamlessly integrate with other Eclipse-based tools and services. One of the most complex Eclipse plug-ins is BIRT, an open source Eclipse-based reporting system that can be used to generate charts and other reports from input data. BIRT is typically used to take its input from relational databases or spreadsheets, but it provides an open architecture that allows programmers to plug in arbitrary tabular data sources.

A simple way of generating tabular data from an OWL/RDF data model is via SPARQL Select queries, which deliver result sets in rows with one or multiple columns. TopBraid Composer's Maestro Edition now provides a powerful interface between any OWL/RDF data source and BIRT. The following screenshot illustrates the kind of output that BIRT can create.

Details on how to use BIRT are available in TopBraid's help pages. I also made a quick screencam demo (3 minutes) showing BIRT.

Semantic XML: Mapping arbitrary XML documents to OWL

2007-11-10T13:37:00.001-08:00

XML is everywhere. Many XML documents contain valuable information that should be lifted into Semantic Web technology. Furthermore, we often encounter situations in which we want to generate XML output from data stored in our ontologies, for example to interoperate with external tools and web services. The new Maestro Edition of TopBraid Composer introduces a new XML-to-OWL/RDF mapping capability, called Semantic XML.

I made a small screencam video (10 minutes) to illustrate some of Semantic XML's capabilities. In the video I am taking an XML document, load it into TopBraid and run a SPARQL query over it. Then I am fine-tuning the generated ontology and show how other files can map into the resulting OWL model. Finally I show how to load arbitrary HTML documents into a Semantic XML-based XHTML ontology.

Here is also a screenshot of what Semantic XML does (click on the image for a larger view):

TopBraid can automatically generate an OWL/RDF ontology from any XML file. Each distinct XML element name is mapped into a class, and the elements themselves become instances of those classes. A datatype property is generated for each attribute. The nesting of the XML elements is stored by means of the composite:child property described in a recent blog entry.

The key idea of Semantic XML is that each of the generated OWL classes and datatype properties is annotated with an annotation property (xmap:element and xmap:attribute, resp.) that points from the OWL concept to the XML serialization. These annotations are also used if an OWL model needs to be serialized back to XML format.

If you import an XML file into an ontology that already contains classes and properties with Semantic XML annotations, then the loader will reuse those. For example, we have defined an XHTML ontology that is automatically loaded when a user opens an HTML file with TopBraid Composer. We plan to provide further standard mapping ontologies for other popular formats such as SVG in the near future.

To summarize, TopBraid can be used to import arbitrary XML documents into OWL so that they can be queried and processed with semantic web tools. The mapping is bi-directional and lossless so that files can be loaded, manipulated and saved without losing structural information.

Note that this mapping approach is very generic, but may not provide the best possible mapping for every XML-based language. In some cases, if starting with an XML Schema file is possible, the XML Schema importer of TopBraid may be more appropriate. On the other hand, TopBraid's round-tripping capability makes it a better option when bidirectional interoperability with existing XML-based tools is needed.

Edited (2007-11-17): Note that earlier versions of this blog entry referred to this technology under the name xmap which is trademarked by another company, so we have changed to Semantic XML.

TopBraid Composer - Maestro Edition

2007-11-09T16:06:00.000-08:00

I am thrilled to announce TopBraid Composer 2.4.0. This release features significant usability improvements (graph editing and support for multiple editors/forms for the same file), and also Oracle 11g integration. Please check our web page for a detailed list of changes.

With version 2.4.0 we are also launching an extended version of TopBraid Composer, called the Maestro Edition. Maestro includes all features of Standard TBC, but also comes with many new and extremely powerful capabilities that are not present in the Standard Edition. A detailed list of new features of Maestro is available online, but here is a summary:

A built-in TopBraid Live test server for rapid application development.
Support for document generation with semantic Java Server Pages.
SparqlMotion: A new visual semantic web scripting language comparable with Yahoo Pipes.
Semantic XML: A new technology to import arbitrary XML files, and to query, edit and export them back to XML.
BIRT: A SPARQL-based visual report generation and charting tool kit.
EMail import to OWL for semantic analysis of emails.

I will write more about all those capabilities in the next few days. Stay tuned!

The Maestro Edition is available for download now and we have reset the evaluation period so that all users can get a fresh view on the tool.

TopBraid now supports the OWLIM Inference Engine

2007-09-18T20:06:00.000-07:00

I am pleased to announce that TopBraid Composer 2.3 now comes bundled with a new inference engine: SwiftOWLIM is a rule-based high-performance reasoner for RDFS and (partial) OWL semantics. OWLIM has been implemented by OntoText and uses the Sesame API as its main interface. It is an attractive foundation both for smaller projects (SwiftOWLIM is a free open-source engine) and for professional use (OntoText offers BigOWLIM, scaling to billions of triples).

In Composer, OWLIM can be configured to run in three modes (OWL-max, OWL Horst and RDFS), depending on what type of expressivity you need for your application. Our initial tests with OWLIM indicate that OWLIM may become a serious alternative to the better-known engines such as Pellet and Jena. My colleague Dean Allemang is using OWLIM to classify ontologies that contain tons of individuals. His models are essentially impossible to handle with Pellet which uses a different reasoning algorithm that is geared for complete OWL DL semantics.

While Composer 2.3 is the first release with OWLIM support, we anticipate that several improvements will be folded into the coming releases. For example we could expose OWLIM's own rule language similar to the Jena rules interface.

TopBraid Live: Rich Semantic Internet Applications with Flex

2007-08-31T17:31:00.000-07:00

Rich Internet Applications (RIAs) run inside web browsers and provide a much more dynamic user experience than conventional web pages. While traditional HTML-based pages require a full reload of the page when the user clicks on a link, many modern web pages only reload parts of the page and provide animations to dynamically navigate through an information space.

There are several platforms to implement such dynamic web pages, for example AJAX, Flash, Java FX, Microsoft Silverlight, OpenLaszlo, XUL. These platform all share a similar programming model that mixes declarative (XML) elements to describe the layout of user interfaces with some imperative scripting to define the application's behavior.

For our newest Semantic Web tooling platform TopBraid Live we had to pick one of these technologies. The goal of TopBraid Live is to serve as development platform for Rich Internet Applications based on server-side Semantic Web models. TopBraid Live provides the whole server-side infrastructure to store and query RDF/OWL data, and a comprehensive client-side API including a library of out-of-the-box components that make creating web applications easier.

After evaluating AJAX and OpenLaszlo for a while we have finally chosen Flash, in particular Flex 2 and ActionScript 3 as the primary foundation for the client-side API of TopBraid Live. In our experience, Flex is the best available technology that is widely deployed and comes with all the flexibility that one would expect for true Rich Internet Applications. In particular we appreciate the nice object-oriented model and strong typing that makes ActionScript 3 code as maintainable as Java or C# code - unlike JavaScript with all its browser incompatibilities and ad-hoc constructs.

Today we are thrilled to announce the first comprehensive Semantic Web API for Flash. The TopBraid Live API provides a comparable data model to Java libraries such as Jena and Sesame. The triples and nodes in the data model are automatically synchronized with the server in a highly scalable architecture. Data is loaded as the user browses through the Semantic Web model, and triples and nodes are cached on the client, avoiding to continuously reloading everything. We provide some abstraction layers to run queries to back user interface components. Many UI widgets such as trees, tables, forms and graphical browsers come out-of-the-box as part of TopBraid Live.

The screenshot below shows TopBraid Ensemble, a multi-user Semantic Web browser and editor/wiki based on TopBraid Live (click on the image for a larger view).

In order to use TopBraid Ensemble, you need a TopBraid Live server to manage your RDF/OWL data stores (in whatever format you chose). You can configure which models are visible (and editable) by which users. The users can then log in and either navigate through the model or edit parts of it. The tool also provides search capabilities and even a graphical browser.

I believe that these capabilities and the smooth integration with TopBraid Composer as a professional development tool will make TopBraid Live one of the primary Rich Semantic Web Application platforms on the market.

Explaining Inferences

2007-08-27T12:33:00.000-07:00

One of the nice capabilities of OWL is its rich built-in semantics. These semantics can be used to employ generic inference engines that can make implicit relationships in your OWL model explicit. However, once you start using an inference engine, you are often more or less working with a black box and may discover that the system makes inferences that are difficult to understand.

In the OWL tools spectrum, SWOOP was probably the first to provide some kind of explanation facility that would point users to a set of base axioms to explain inferences. This capability was based on an experimental version of the Pellet inference engine which has been folded into the regular Pellet distribution with its version 1.5. The new Pellet release made it straight-forward for us to add a similar capability to the new TopBraid Composer version 2.2. After running inferences via Pellet you can click on the menu next to inferred triples to open the Explanation view:

The Explanation view displays a clickable list of axioms. In the case below, it shows that the class Safari is inconsistent (i.e. a subclass of owl:Nothing) because it is the subclass of the two mutually disjoint classes Sightseeing and Adventure.

At TopQuadrant we have already had several use cases where this capability was a huge time-saver, in particular if ontologies get as large as those that we develop for NASA. Although the explanations may look weird or geeky at first, any pointers into the right direction can make a huge difference.

Composite Design Pattern in RDF/OWL

2007-07-16T14:48:00.000-07:00

One of the strengths of RDF is that it encourages reusing ontologies through globally unique URIs and imports from the Web. There is a small but growing library of de-facto standard ontologies such as foaf, geo and dublin core that serve as building blocks for much larger ontologies. Standard building blocks support the development of generic tools and algorithms. For example, if an ontology uses the latitude and longitude properties from the geo ontology, then geo-aware tools can render the subjects on geographical maps. Without such standard properties, coordinates would just be plain numbers, and tools would need to guess whether they are intended to represent locations or not.

Taking this idea further, some RDF ontologies may even implement design patterns similar to those known from object-oriented systems. There is a well-known pattern called the Composite Pattern which can be used to represent part-of relationships. In a nutshell, this pattern defines a class Component and a class Composite, and a relationship between them to express a child-of relationship. This pattern is frequently used in user-interface libraries to represent the nesting of UI widgets inside of each other, creating complex hierarchies. When this design pattern is used explicitly, then development tools can render the structure of the hierarchy appropriately, e.g. in a tree view.

In the ontology/Semantic Web world, this pattern is also used very frequently. Several papers were written about representing part-of relationships. For our own work, we also re-invented this pattern numerous times. In addition to representing part-of relationships, we also find a recurring need to represent the ordering of the children. RDF is by definition unordered, and using rdf:Lists is often a pain to use. In response to these recurring requirements, I created a tiny RDF/OWL implementation of the composite design pattern, which can be found at

http://www.topbraid.org/2007/05/composite.owl

This is a trivial ontology, defining three properties only:

composite:child is an object property pointing from a parent to a child
composite:parent is the optional inverse of composite:child
composite:index represents the index of each child among its siblings, typically starting at 0 for the first child

While this is a really tiny ontology, it can act as a quite powerful building block for larger models. In particular, TopBraid Composer 2.1 now has built-in support for this ontology. If you have imported the composite namespace into your model, then the form and the association tree will make use of the ordering using the composite:index property. In the following example, I have created a class Process with a property step which is a sub-property of composite:child. All children have an index attached to them, and the form will automatically order the children by their index. By default the ordering would have been alphabetically.

There are menu items to move children up or down. You can use the Associations View to display this relationship in arbitrary nesting:

We will add more support like that into future versions as we are exploring further design patterns. Feel free to reuse this composite ontology in your own projects and please let me know your comments (this blog does not allow comments since I received too much spam).

Default Namespaces considered harmful

2007-05-30T13:15:00.000-07:00

In RDF, default namespaces are typically used to minimize the length of resource names. For example, if you have a resource http://www.aldi.de/products.owl#Haferflocken and define everything up to the # as your default namespace, then you can abbreviate the whole URI with "Haferflocken" only.

While this is very convenient in simple ontologies, things become quickly very confusing if you are building modular ontologies that span multiple namespaces and files. For our NASA work, we are dealing with ontologies that consist of 80 or more modules, all of which have their own namespaces. In such settings, using default namespaces is problematic because the same resources will be called under different names when you switch between files. For example if you are working in SPARQL and store your queries as part of a string-based query library, then the queries may reference resources sometimes as :Haferflocken and sometimes as aldi:Haferflocken, depending on whether a namespace is used or not. Depending on which is the main file, the queries may not work. Another problem is that if only a default namespace exists, and another file imports your ontology, then it is unclear what prefix to use for the external namespace. This leads to situations in which the same namespace is sometimes abbreviated "aldi" and sometimes "products".

Our recommended practice is to not use any default namespaces, but instead explicitly declare a prefix for that namespace inside of the file so that when someone wants to import your model you all have consistent namespaces. This prepares your models to be imported in a modular fashion down the road.

Semantic Technology Conference Photos

2007-05-30T09:48:00.000-07:00

Back to normal life after last week's busy Semantic Technology Conference. For those who could not attend, Keefe has kindly uploaded some photos from the various TopQuadrant-related events:

http://www.flickr.com/photos/semanticconference/

TopBraid Suite presented at SemTech 2007

2007-05-23T07:16:00.000-07:00

Yesterday we presented the new TopBraid Live and TopBraid Ensemble products at the Semantic Technology conference. The room was packed full (and, more importantly, nobody left during the talk), so I guess we are going in the right direction. There were also excellent networking opportunities at the exhibition, which continues today.

Here are the slides of my presentation - thanks to all who attended:

http://www.topbraidcomposer.com/docs/STC-2007-HK.ppt

Some more information on the new TopBraid Suite products is available from the TopQuadrant web site. I'll write more about it over the next couple of weeks.

TopQuadrant is hiring!

2007-05-10T17:30:00.000-07:00

Most people at TopQuadrant are currently busy preparing for the upcoming Semantic Technology Conference in San Jose. This will be a great event for us, since we will introduce a new family of products under the umbrella of the TopBraid Suite. After we have quite successfully established TopBraid Composer as the leading professional semantic application development tool, we now take the next step.

We will introduce TopBraid Live as a client-server platform and library of reusable building blocks for all kinds of Semantic Web applications. This includes typical mash-up components (such as maps and calendars) but also more complex widgets such as trees, forms, reports and charts. Support for semantic search and other standard tasks will be integrated as well. We also introduce TopBraid Ensemble as an out-of-the-box solution for viewing and editing RDF/OWL instances with a thin AJAX-based Web client. Needless to say, all these Web applications are seamlessly integrated with each other. For example, you can use TopBraid Composer to define form layouts and get exactly the same layout in TopBraid Ensemble. Be sure to visit us at our booth to find out more.

In order to nurture and support all these new products and other customer projects, we are now looking for new developers. In addition to a good command of the underlying RDF-based technologies, candidates need to have excellent Java and Web development skills. If you are able to relocate, then sunny California is the place to be. TopQuadrant is in the middle of a solid phase of growth, and I think we are very well-positioned to become an important player in the semantic technology field. If you are interested in joining us (or know someone who might have a matching profile), please drop me a line.

New TopBraid Composer Users Mailing List

2007-03-23T08:44:00.000-07:00

I have received a number of request to launch a mailing list for our user community. I hesitated because many of our users work in restricted domains and prefer private emails with us. However, the rapidly increasing size of our user community has changed my mind. I have just launched a new Google Group at

http://groups.google.com/group/topbraid-composer-users

If you are interested in TopBraid Composer then please join this group and use the discussion forum to ask questions, provide usability feedback, suggest new features, report bugs, or to get in touch with other TopBraid users around the world.

Hybrid Inferencing with TopBraid Composer 2.0

2007-02-21T21:41:00.000-08:00

Many people who learn Semantic Web languages have a hard time understanding the concept of inference engines (aka reasoners). In a nutshell, an inference engine takes an "asserted" model as input and creates an "inferred" model as output. In the open world of the Semantic Web, inference engines can only add new triples, i.e. the output model is the input model plus additional inferred triples. In other words, inference engines derive new knowledge from existing knowledge.

Various types of inference engines exist, including OWL DL tableaux reasoners like Pellet, and Rule engines such as Jena. Even SPARQL can be regarded as an inference engine, because its CONSTRUCT keyword can be used to derive new triples from existing triples. Some applications may also require custom-tailored reasoning engines that apply temporal or geospatial knowledge to infer new knowledge.

However, implementing a complete inference engine that adds some functionality on top of OWL and RDF Schema is a lot of work, and few people want to reinvent the wheel. For example, vanilla implementations of SPARQL and the Jena rules language have no built-in knowledge of OWL DL semantics, yet in many use cases, you want to run SPARQL or Jena on top of an OWL-compliant reasoning service such as Pellet. Fortunately, since all inference engines are essentially black boxes that take some triples as input and generate more triples as output, it is possible to link various engines together so that the output of one engine becomes the input to the next.

Some support of such hybrid inference chaining had already been provided by previous versions of TopBraid Composer. However, this support was scattered across various places in the user interface and not as flexible as it should. This limited its use and made things complicating to explain and demonstrate. So we took the opportunity of the new TopBraid Composer 2.0 release to generalize and clean up our inferencing support. How this works is illustrated in the screenshot below (click on the image for a larger picture):

The configuration dialog above can be used to select and arrange reasoning engines for a given project. In this case,

we are taking an asserted model containing real-estate properties and execute Pellet over it. This would reveal OWL DL relationships between concepts, for example to infer that if House is a subclass of RealEstate, then any particular instance of House is also an instance of RealEstate. Supported by this additional knowledge,

we run a bunch of SWRL or Jena rules to infer relationships that cannot be expressed in OWL DL. For example, the Jena rules engine has a built-in function to do mathematical calculations, so that we can, for example, convert Australian dollar values into US dollars. After we have done these additional calculations,

we execute SPARQL CONSTRUCT queries to establish new triples and relationships. For example, now that we know the US dollar price of a certain House in Australia, we can evaluate whether it matches my SPARQL FILTER maximum of 900,000 $US. Since this step may create new triples, and these new triples may lead to new relationships from Pellet's point of view,

we repeat the steps above until no additional triples have been added by any step.

If you want to play with this scenario yourself, download TopBraid Composer 2.0, import the example real-estate ontology and configure the infererence engines as shown above. Use the new Run Inferences button to execute the process - it should come back with suggestions on which house to buy for some customers. Note that the file contains some constructs both in their Jena rules notation as well as the SPARQL notation, but you can disable all Jena rules (except for the currency conversion) to get the complete results - the Jena rules are there to illustrate the syntax of both approaches.

While the capability to chain multiple inference engines together is not radically new, the new architecture and its simple, intuitive user interface in TopBraid Composer will open the door for many new kinds of applications. For example, you can define your own reasoning engines with SPARQL or Jena and use them to provide on-the-fly translations and mappings from one data structure into another. Simply collect your mapping rules in a separate ontology and give them to others so that they can put these rules into their own inference delegation chain. If the expressivity of OWL, SPARQL, SWRL or Jena is not sufficient for your needs, you may use a new Eclipse extension point in Composer to add your own Java functions for domain-specific reasoners (ask me for details if you are interested in this). Based on this architecture, we will start to roll out several new features for ontology mapping, data integration and mash-up creation in the coming months.

OWL 1.1 Support in TopBraid Composer 2.0

2007-02-21T21:33:00.000-08:00

OWL 1.1 is an evolving extension of the Web Ontology Language (OWL). Driven by a world-wide community of researchers, this proposal aims at fixing some of the most frequently encountered limitations of OWL such as the lack of user-defined datatypes, and cumbersome disjointness axioms. The new 1.1 features include extra syntactic sugar, additional property and qualified cardinality constructors, extended datatype support, simple metamodelling, and extended annotations.

While OWL 1.1 is still at a rather early stage, and will not become an official recommendation before 2008, I believe that OWL 1.1 contains many invaluable extensions that are already useful for everyday work. We therefore decided to integrate 1.1 support into the new 2.0 release of TopBraid Composer. The Pellet version integrated in Composer already supports reasoning with many of the OWL 1.1 constructs, and more 1.1 tools and APIs will follow. At the same time, users need to be aware that they are adopting the language at a stage where it will almost certainly change. As a result, ontologies that use the new features now may contain deprecated constructs until the language has been officially sanctioned. Furthermore, tool support is on the bleeding edge and will not be as stable and reliable as support for OWL 1.0.

Here are some examples of how the new features are used in Composer. User-defined datatypes are subtypes of the built-in datatypes such as xsd:int and xsd:string. In the screenshot below, all instances of the class Adult must have their age between 18 and 65:

Internally, the new OWL 1.1 language constructs are represented as additional triples. In the example below, I have restricted the range of the age property to fall within the interval of 0 to 200. As shown in the nested form, the rdfs:range of the property is an instance of the system class owl:DataRange, with additional attributes to represent facets such as minInclusive.

A common problem with OWL DL ontologies is that a large number of disjointness axioms is needed to declare that certain classes do not overlap (i.e. share instances). OWL 1.1 introduces the disjointUnionOf property as a convenient shortcut for such cases. In the following TopBraid Composer screenshot, the class RealEstate is defined as the disjoint union of its subclasses Apartment, House and Townhouse using a compact xor notation. Reasoning engines that see such statements would insert pairwise disjointness axioms between those classes on the fly.

Qualified Cardinality Restrictions (QCRs) allow you to define the class of Persons that have some children, at least two of which must be Female:

There are several other extensions - please see TopBraid Composer's documentation (Help > Help Contents > TopBraid Composer) and the OWL 1.1 documents on the Web for details. Don't forget to activate OWL 1.1 support in the ontology start page - by default these features are switched off so that users can safely rely on OWL 1.0 if they prefer. Feel free to drop me a line to provide your feedback and questions.

Announcing TopBraid Composer 2.0

2007-02-21T21:21:00.000-08:00

We are very excited to announce the release of TopBraid Composer 2.0 today. This milestone release makes Composer the first professional Semantic Web development tool to support the evolving OWL 1.1 specification. We have also significantly improved the inferencing architecture, allowing TopBraid users to execute reasoners in a delegation chain, and to use SPARQL as a rule language. I will write detailed blog entries about these features shortly.

In addition to these and many other new features (listed on our web page), we have spent extra time debugging to make sure that Composer stays ahead of its competitors in terms of stability and performance. We have discovered and worked around some Eclipse incompatibilities on Mac computers. Doing a major release was also a good opportunity to review and update the help pages: I actually ended up adapting all pages and replacing all the screenshots. Finally, we have simplified the installation on Windows, and now provide a clean installer package including Java and Eclipse.

Reflecting on the rapid evolution of the tool since its first release early 2006, I would like to acknowledge all of our users who have provided invaluable feedback and otherwise supported our work. We very much appreciate to have you on board! Many of the features that you find in Composer these days are also a direct result of TopQuadrant's semantic web projects with customers around the world. We all hope you enjoy TopBraid Composer 2.0 as much as we do!

Real-time Data Browsing with SPARQL

2007-01-12T14:10:00.000-08:00

In many cases, good ideas for new features come directly from our end-users. A few days ago, a customer asked for the ability to automatically refresh the SPARQL view whenever the user selects a new resource. This is easy enough to implement, and now available in TopBraid Composer 1.5.1. Look at the following screenshot to see what it does.

Whenever the user selects a different Suburb, the SPARQL window will automatically run the given query and display the matches. This mechanism is triggered if the current query contains the variable ?this. The trick is to automatically bind the ?this variable with the current resource.

This useful mechanism can be used to quickly browse instances by arbitrary queries, e.g. to see how the system would behave during ontology design. Thanks a lot, Geoff, for this excellent idea! We appreciate more input like this...

More on Mashups: Linear Charts

2007-01-06T11:01:00.000-08:00

In my previous postings, I wrote about TopBraid's new Calendar mashup feature as well as our RDFa editor. Version 1.5 introduces yet another feature called the Linear Chart View. This view allows users to display numeric values along a linear chart with X and/or Y axes. If you have an ontology about real estate properties, and want to get a chart to compare the prices of all relevant properties, then you can display them along the X axis:

Alternatively, you can use the Y Axis:

To make things more interesting (also in the sense of mashing up data from multiple sources), select the price along the X axis and the number of bedrooms along the Y axis:

You can use the mouse to drag the chart in all directions or to change the zooming level. The chart is also clickable and has tool tip texts to display details. It is easy to imagine that we could have similar viewers for other types of diagrams such as pie charts.

TopBraid is increasingly becoming an information integration hub that puts a lot of power into the hands of ontology designers and users. For example, you could create ontologies that serve as a view on a relational database or lift Excel spreadsheets into the Semantic Web. Then combine these data sources with other ontologies and finally display them on chart. In the near future you will also be able to publish the very same models and views on the Web so that end users can analyze them with a thin Ajax-based client. There is quite a bit happening with TopBraid these days. I wonder where this journey is taking us...

Calendar Mashup in TopBraid Composer

2007-01-06T10:37:00.000-08:00

Mashups are applications that combine content from multiple sources into a single view. For example, if one web page lists the dates and addresses of certain conferences and another web service knows how to convert addresses into geographical coordinates, then it would be helpful to extract both pieces of information into an integrated experience to help with travel arrangements. The open architecture of Semantic Web languages like RDF Schema and OWL make it relatively easy to bring together information from various sources. Once the data structures are brought together and aligned, they can be displayed and analyzed.

We are incrementally extending our RDF/OWL development tool TopBraid Composer with new forms of visualization. This allows ontology designers and data architects to directly experience what they can actually do with the ontologies. TopBraid already has support for geographical information with its Google Maps interface, but version 1.5 now also introduces a Calendar view.

The Calendar View enables users to visualize xsd:date and xsd:dateTime values from an RDF/OWL model on a month-by-month basis. Users select which properties to show and the system then renders them into their corresponding day boxes. In the following screenshot, the Calendar is shown alongside with the geographical viewer (click for a larger image).

As you can see on these images, some temporal values are displayed to cover multiple days, i.e. they represent a "time span". Well, but there is no standard way to formalize a "time span" in RDF, so you may wonder how does the system actually know that some properties shall be interpreted as start and end points of a time interval? The flexibility of RDF Schema enables a quite elegant solution to this problem. The Calendar is connected to a small Calendar ontology which defines two properties calendar:startTime and calendar:endTime. If your domain has a pair of properties travel:departureDate and travel:returnDate then you just need to make these properties subproperties of the calendar properties, and run the classifier to tell the system that all departureDate values shall be interpreted as startTime and all associated returnDates shall become endTimes as well. Instead of having to change the code or configuration of the Calendar itself, we just change the model to drive the existing viewer.