Things XML

Colorising XPaths

2012-01-07T11:14:00.004Z

I've been working on Saxon-CE, a browser-based XSLT 2.0 processor, with the Saxonica team for a few months now. The main programming languages I've used JavaScript and GWT Java, with XSLT just being used for small well-defined test-cases.

Over the holidays I decided to whet my appetite to do more with client-side XSLT by developing a small Web App. I've started things off with an XPath Analyser, a simple app, but one that can be developed to be a more sophisticated XPath Tool in a number of separate stages.

Even with a small app it helps to write a tool that accomplishes something vaguely useful, so XPath Analyser lets you enter any XPath Expression (1.0 or 2.0) and colorises it when you press . If I want to include an XPath Expression in my blog (using Bloggers Rich Editor), instead of including something like:

if (count($splitToken) ne 2)

then dxt:main[attribute::place + 1.34e-27 = 28] (: Edit XPath and Refresh :)

else if ($splitToken[1] eq 'instance' and $splitToken[2] eq 'of') (:to review expressions (: castable as :) :)

then (count($pxt:pcz) ge 2200 or pxt:dema/pxt:semi[@test])

else if ($splitToken[1] = ('cast','castable','treat','(:literal:)') and $splitToken[2] eq 'as')

then true() else some $pxa in tree/branches satisfies $pxa > 22

I can make it prettier, by pressing Run and then copying and pasting from the tool, I then get this:

if (count($splitToken) ne 2)

then dxt:main[attribute::place + 1.34e-27 = 28] (: Edit XPath and Refresh :)

else if ($splitToken[1] eq 'instance' and $splitToken[2] eq 'of') (:to review expressions (: castable as :) :)

then (count($pxt:pcz) ge 2200 or pxt:dema/pxt:semi[@test])

else if ($splitToken[1] = ('cast','castable','treat','(:literal:)') and $splitToken[2] eq 'as')

then true() else some $pxa in tree/branches satisfies $pxa > 22

Much better, and just 500 or so lines of XSLT! One of the benefits of writing a lexer/parser with XSLT is that the declarative nature of the language helps enforce a certain-level of design discipline, especially helpful with XPath as there are some quite complex syntax rules.

Now, this isn't a true XPath Parser, such as Dimitre Novatchev's, but it does provides enough structure for colorising and will help with the next stage which is to dynamically generate selected parts as sub-expressions, by prepending the context path:

$a[position() lt $c][position() gt sum(25 + @books

when @books is selected, output is

$a[position() lt $c]/@books

Please help with this XMLQuire Poll:

2011-09-21T09:37:00.004+01:00

To recap on virtual formatting, in case you haven't yet tried XMLQuire or seen this video:

Many thanks, you should see the running tally as you vote. I will post back a summary in 3 days, when the poll closes.

Phil

Rise and Fall: Writing an XSLT Editor - Part 4

2011-09-20T19:05:00.000+01:00

CoherentWeb, the XSLT-based Integrated Batch-Processing Environment (IBPE) is now no longer being marketed, though it will continue in private development in the hope of resurrecting it some time in future. This was not a difficult decision:

Though this product was carefully placed to complement conventional IDEs rather than compete with them, it seems clear that the market-strength of the major IDEs make the IBPE concept non-viable as a commerical proposition, at least for a microISV startup.

I still believe earnestly that something is needed to fill the gap between the editing focussed IDEs and the processing focussed back-end systems. Few developers can realistically develop XSLT against a single XML instance and yet this is the way IDEs present the environment, one input, one output. Sure, they let you batch-process a folder but how many of them have these features?

Multi-threaded batch-processing
Seamless presentation of output alongside each each input
Summary reports of all outputs
Real-Time update of batch progress
Instant cancellation of a batch-job
Background clearing of previous batch-jobs
Single-Click reviewing
Management of linked resources such as graphics
Zip-Package creation

The answer is of course none of them. This doesn't make XSLT IDEs bad tools of course (far from it), it simply means that something else is required, something that can cope with a few dozen or a few hundred inputs and manage all the associated resource files.

So, along with the demise of CoherentWeb, the XSLT editor integrated into this tool is also very likely doomed. This introduced a range of features new to XSLT. Most popular of course was virtual formatting (mentioned in several other blogs), which would almost eliminate the need for xsl:text elements, if only other XSLT editors were to adopt this.

But there were many other new features also: the XPath manager with full predicate-by-predicate tracing, the literal result elements validation of static XSLT, the schema-scanner for schema-set based validation. I hope at least some of these features appear soon in the current (and otherwise excellent) crop of XSLT editors available. So even if CoherentWeb is dead, some ideas from it may live on.

There are some promising signs in the XSLT IDE sector, some products appear to be shedding their more peripheral XML authoring features, to provide a more focussed UI for the developer, this has got to be a good thing. This should give us simpler toolbars and less distraction from features we don't need, hopefully more will follow.

One last screenshot of CoherentWeb for the sake of nostalgia:

Abstract Syntax Trees: Writing an XSLT Editor - Part 3

2011-09-17T16:48:00.000+01:00

In the previous part to this series, I mentioned a 'frozen string' approach in the context of keeping an internal model of parsed the XML text, I now prefer using the term 'Abstract Syntax Tree' - though I'm not certain whether it strictly fits the WIKI defiintion or that used on the JetBrains MPS web page.

In CoherentWeb, the 'XML model' is maintained as a simple collection of 'Node Descriptor Groups' (NDG), with 1 NDG for each node, except attribute and text nodes as these don't contribute towards the XML hierarchy. Each CDATA section has its own NDG, and perhaps surprisingly, even all the 'XML' nodes within comments have one.

NDGs are slightly different depending on whether they represent element nodes or comment nodes for example, but to take an element node - you might have the following:

The above should be fairly self-explanatory, Start and End are the character positions of the node within the XML text. NodePointer is the count of how many preceding sibling nodes there are in the tree, this, combined with the ParentIndex and Level identify the exact postition of the NDG within the AST. This data is managed for each node and the whole collection makes up the Abstract Syntax Tree (AST).

The AST is used for a whole range of tasks including:
Knowing the context of the current selection
Building a lazy-loading element tree from the currently selected node
Showing the selected XPath location, complete with qualifying predicates
Displaying the currently selected element name
Showing XPath, name and value of all nodes that have value within the current element
Providing 'soft-edges' to tags and attributes to prevent accidental deletions
Determining whether a key-press requires a partial reparse, a full reparse, or just an offset within the model

The screenshot below shows the amount of information that needs to be displayed to the user when a node is selected, the AST is critical to doing this in a performant way:

As well as the AST, syntax coloring of XML text is also exploited as this contains useful information about such things as whether an attribute name or attribute value is pressed, so it isn't necessary to replicate this in the model. Even quotation marks are a slightly different color (is this cheating?) so it can be determined whether they're opening or closing.

All node data is stored in a generic List<nodedescriptorgroup>. NodeDescriptorGroup is a Class, it could have been a Struct, but then the AST would live on the stack instead of the heap - this could then cause stackoverflow issues for very large XML documents.

So, that's the Abstract Syntax Tree, an important part of an XML Editor design if you want to avoid excessive parsing of the XML text and provide user interface features where the exact selection context must be known.

XML to music

2011-09-15T17:27:00.000+01:00

Now for something a bit different.

This is a video demo showing virtual formatting at work - parts of the video have been sped up to take account of my slow typing, all narration is in the form of the typed XML - there's some background music just for effect:

XMLQuire Product Launch

2011-09-08T13:34:00.002+01:00

Yesterday was the launch day for XMLQuire. This free and lightweight XML Editor uses CoherentWeb's 'Virtual Formatting' technology, but is without CoherentWeb's extensive XSLT/XPath features. Instead, XMLQuire makes a feature out of UX simplicity, light-weight and ease of deployment.

The hope is that XMLQuire will complement CoherentWeb in promoting the use of Virtual Formatting to a wider set of developers and XML authors in general. If more XML tools could only adopt this concept, then interoperability between tools would be greatly improved.

As things are, there are still comments coming in that show I'm not explaining very well what Virtual Formatting does, both for the XML editing user-experience, but more importantly, for the integrity of the XML itself. I think the problem lies in that we've grown so used to inserting characters for indentation that we find it difficult to consider the problems indentation characters cause and what the alternatives are.

Perhaps these two product can do their bit to spread the word - with the introduction of Virtual Formatting, it's no longer a question of 'tabs or spaces?' but 'indentation characters or ... nothing?'.

Editing Virtually Formatted XSLT

2011-08-23T15:25:00.001+01:00

CoherentWeb's XSLT editor uses vitual formatting, where the XSLT is always properly indented, but without using tab or space characters. This absence of formatting characters allows us to write XSLT with much greater control of whitespace, without resorting to methods that affect the clarity and maintainability of the XSLT.

In the screenshots posted here, all characters except line-feeds are highlighted in yellow, so the space characters are clearly visible. (To highlight elements in this way in the XSLT editor, I just selected the document element in the outline treeview).

First, lets have a look at XSLT as edited in CoherentWeb:

Original XSLT - Clean of formatting characters

Before going on to review the output, lets switch the view to the left-justified, so we can check the alignment of text, no characters are modified for the view here, the left-margin is just collapsed. We can see clearly that 'Books' along with the '------' underline are going to be indented in the output, with the rest of the output text aligned to the left margin:

Now for the output which produces no surprises. The Books title is indented with space characters that were in the XSLT, all other lines are left-justified. The line-breaks are all output because they are in the XSLT as 'naked' text. The one exception is the line following ISBN, this is inserted using the value of $lf which has a line-feed \n as its string value. This is required because XSLT won't copy the whitespace-only node preceding the end tag into the result-tree.

Original Output

So, the above XSLT was relatively simple because there were no extra formatting characters that other XSLT editors use. To emulate their behaviour, I opened the orginal XSLT file in another XSLT editor and then just saved it again. When this did was insert formatting characters into content because the other editor needed to render it in a formatted way and didn't have virtual formatting. Now, to be fair, I could have modified the settings of this editor so that it didn't format the XSLT when opened, but then it wouldn't have been rendered properly.

Temporarily disabling CoherentWeb's normal trim behaviour (which would detect and selectively removed formatting characters), I then opened the file modified by the other editor and it looked like this:

The yellow highlighting shows us what the other XSLT editor did to our nice trim XSLT, look at all those space characters added for formatting!. Now, that other editor has been reasonably smart, because it knew the element must preserve whitespace, so thankfully it didn't insert formatting characters here. It did however ignore the whitespace preceding the Books line, choosing instead to pad the line for XML formatting purposes as if the leading whitespace wasn't there. This means we can't just trim the XML to get back what we started with, because the information about the extra whitespace has been lost by that editor.

Now we have the file from the other editor, let's see what output we would get:

So, the extra space characters in the input are now being copied into the output. The Books title is also no longer centered in the output formatting. This is expected behaviour for XSLT of course, all whitespace within a sequence contructor is significant unless it's within a whitespace-only node. The xsl:text element already mentioned is one way of remedying this, the extra effort required isn't too onerous, but for more complex XSLT it will affect readability:

The XSLT above has been modified to produce the required output, even when the XSLT has formatting whitespace characters inserted. The xsl:text element effectively turns the formatting characters into white-space only nodes which are not added to the result-tree. We also have to use instructions to insert line-feeds more frequently, I've used concat($lf, $lf) to this end.

So, the XSLT is no longer susceptible to formatting characters, but the cost is more complex XSLT. With CoherentWeb, I would strongly advise that your XSLT is ultimately hardened to work with editors that don't have Virtual formatting capabilities. But at least while doing initial design and development you have the option to use XSLT that more closely resembles the output you need.

2011-08-19T21:00:00.023+01:00

Virtual Formatting

Writing an XSLT Editor - Part 2

For more than 6 months now I've been developing the concept of virtual formatting for the XML Editor now integrated into CoherentWeb. This concept is simply where XML text is indented by adjusting the left-margin, rather than using tab or space characters for the same end. There's nothing new in this idea, web browsers use virtual formatting each time they're used to view an XML file, they don't go to the trouble of modifying the content, just the way it's rendered.

What works well as an XML viewing concept, however, is an entirely different beast when its used in an environment where, as in an XML editor, the text is potentially changed as quickly as someone can type. The rewards though are huge, bringing a whole new editing experience that is immediate and compelling, as this 5 minute video hopefully demonstrates:

Video: Imagine an XML Editor that could...

I should probably explain something about the formatting rules used by CoherentWeb:

First, XML formatting is not standardised in any way, but CoherentWeb follows a set of conventions in common use, its these that I've attempted to isolate and describe here. Below is some XML text with no indentation (a screenshot of the XML Editor in left-justified mode), the blocks where different formatting rules apply have been highlighted and annotated:

Let's just see how this XML is formatted when in indented mode before going on to look at each block:

The (background-highlighted) screenshot above shows the same XML as the first screenshot, but this time indented. The block coloring is to associate the indented blocks with their non-indented equivalents that are also annotated.

1. Namespace Declarations

These occur within the element start tag and the same indentation convention is applied as for attributes.

2a. Attribute Names - where the first attribute is on the same line as the element tag

In this case, if subsequent attributes start on new lines then they are aligned with the start of the first attribute name.

2b. Attribute Names - where the first attribute is on a new line

The first attribute is indented by the same fixed amount as if it were element content. Any subsequent attributes on new lines are indented likewise, so they are all left-aligned.

3. Element Tags and content

We're all very familiar with the basic formatting rules for elements and their content, but to reiterate: XML element open and close tags establish the hierarchy of an XML document. Each new open tag marks a new, higher level, of nesting; the entire contents between the open and close tags is therefore indented by a fixed amount to indicate each new level of nesting.

4. Attribute Values

When attribute values extend over several lines, the start of each new line is aligned with the very start of the attribute value. Attribute values may have their own indentation (as shown) outside of the XML context. Unless schema rules indicate other wise, the whitespace used for indentation of attribute values is normalized, it's therefore mainly used to assist in the readability of the XML text.

5. Mixed Content

If an element contains mixed content, then the contents of 'inline' elements is not indented any further than the container element warrants, allowing the text, complete with inline elements to be viewed as a single block.

6. Preformatted Element Content

An XML vocabulary may require that all whitespace content within specific elements is preserved, this is either indicated by the xml:space="preserve" attribute value setting, or declared in a schema defining the element. With virtual formatting, the content can still be rendered indented in the normal way because no characters have to be added to acheive this.

Virtual formatting content with padding characters

Currently, most hand-crafted XML has tab or space characters inserted for formatting, even XML processors such as XSLT (optionally) add whitespace for indentation. For virtual formatting, these characters are removed on loading a file or on clipboard operations.

Editing Benefits

The screenshot below shows the XML Editor with the same contents as the previous screenshot, but with all characters including whitespace highlighted in yellow:

This screenshot helps illustrate how, by dispensing with padding characters, virtual formatting gives absolute control to the user over all content of an XML file and how it's rendered.

CoherentWeb 4.0 Release - Writing an XSLT Editor - Part I

2011-08-16T11:42:00.000+01:00

CoherentWeb 4.0 has just been released. This marks the culmination in an effort lasting well over 6 months, to build an XSLT editor from scratch and blend it in seamlessly with CoherentWeb's XSLT-processing and XPath analysis tools.

I'm pleased now that I didn't go for the easy solution of incorporating a 3rd party editor into the tool, tempting though it was. Building an XML editor from scratch allowed me to rethink some of the very fundamentals of editor design.

The Initial State

CoherentWeb already had an XML text rendering system for viewing/analysis, it relied on .NET's XmlReader for parsing and a fallback parser for non well-formed XML (a modified version of MindTouch's SgmlReader). This approach though, was clearly unsuited for an editor, where each and every character found must be faithfully reproduced, no matter what (almost, more on that later). A fresh think was required:

Considering the XML source

There are 2 main parts to rendering XML text for viewing, syntax coloring and indentation. Files opened from other sources may already have padding characters for indentation, but they may not, so indentation was definitely needed on loading. The optimal solution would be of course to use the same code when loading a file as that used to reformat text when it was being edited. For optimal performance, this would require indentation and syntax-coloring to be performed in the same pass.

Using RTF codes for indentation

The base control for rendering text was .NET's RichTextBox (InkEdit is actually used as it's a later version and has better performance), this is effectively the same control used by WordPad in Windows, so has excellent RTF rendering capability. My natural though process went along the lines that, if you're generating RTF to colour XML text you can also use RTF to indent it, it didn't even occur to me that inserting padding characters was a viable alternative, adding characters to text in the middle of the parsing process would complicated things further (though line-feed characters for XML formatting must be added if not present in the source, this is less frequent)

Trimming the source

There was just one issue though, a big one, what if there were padding characters in the XML source? This would interfere with any RTF-based indentation, so there must be a capability to remove it, so we're not adding characters whilst parsing, but we are removing them, how can this be kept simple without having offset values all over the place? The compromise was for the parser to run a second time on the modified text, but with all RTF-related calls ignored. On the second pass, the only updates required were for the node positions.

The XML Model - A Frozen String

To be really effective, an XML Editor needs to know exactly where the text/mouse cursor is at all times, in the context of the XML. A model of the XML is therefore required that maps directly onto character positions. In this design, the internal model only stores XML node-context and associated character positions as integers, no text is stored, either for names or values. This is similar to the concept of the Frozen-Stream pattern, but we have a string instead of a stream. As well as tracking text position, the model also needs information on the node hierarchy to permit XML-aware operations, and to synchronise with a lazy-loading XML outline, in the form of a TreeView. This model needs to be efficient because it is updated on every keystroke.

Summary

A lot happens in an XML text editor, even before the XML text is rendered. But what happens here is critical, because it defines the functionality available and responsiveness when the XML text is actually edited. I've skipped over the way nodes are parsed because this is pretty standard stuff, the key is that rolling your own parser means that no character information is lost. It also made it possible to do non-standard processing, such as parsing XML comments independently, in case they contained XML (so XML is indented properly in comments, also without padding).

A taste of things to come

2011-06-29T18:56:00.003+01:00

Here at Qutoric, development has been progressing at a frantic pace in preparation for CoherentWeb V4.0.

The most significant change in the forthcoming release is that this XSLT batch-processing system will (for the first time) include XML editing capabilities. The fully featured XML editor means that CoherentWeb will provide an interesting alternative to conventional XSLT IDEs which include more limited batch-processing capabilities.

Some 'Coming Soon' screenshots of the product:

The Configure Panel: Desktop XML Processing starts here

Introducing Virtual Formatting in CoherentWeb's XML Editor

XML Editor: Fully integrated into a multi-threaded XSLT batch-processing system

Edit XML with full virtual indentation or in a flattened view (as shown)

Multi-Monitor support designed in from the very beginning

Print XML with full indented formatting, word-wrap and line-numbering options

Enhanced XPath Development Environment - with full trace/debug features

There's much more of course, but hopefully the above provides a flavour of what you can expect...very soon now.

Using XSLT/EXPath with Word 2010

2010-09-27T17:28:00.012+01:00

The EXPath Zip module shows great potential for publishers wishing to create documents that are XML based but rely on Zip-Compressed packages to bring related data together into a single file.

A number of well established standards such as ODF, XPS and ePub use Zip for this purpose, but in this example I chose the OOXML standard, as used by Microsoft Word 2010.

The presentation (above) describes the general approach used, a screencast (below) shows the XSLT in action. Developing the XSLT itself was relatively straightforwards because of the declarative approach EXPath ZIP uses for describing the ZIP file structure and contents. There was just one minor issue, the sample OData file I used had bmp files embedded using Base64 encoding, however the first 78 bytes of these were just header info - I therefore introduced a (non-standard) 'offset' attribute to the zip:entry element, to allow these first 78 bytes to be ignored.

Links for the files used are below. If you wish to use the XSLT, you will need to change the $word-in-href and $output variables that correspond to the locations of the resource and output files, to suit your own system:

Word Template $word-in-href
XSLT File
OData Input File

Non-Standard Extension to EXPath

As explained in the video presentatation, a non-standard 'offset' attribute for the zip:entry element was used to describe the Base64 output. This was because, after conversion from Base64, the first 78 bytes of the binary needed to be discarded as they contained header data that would otherwise have made the bmp file invalid.
Due to this non-standard attribute, this XSLT can only be run unaltered on the CoherentWeb XSLT test tool. If, however, this is found to be a common use case, it may be possible however for this to be added to the EXPath ZIP specification, which is still awaiting formal documentation.

ISO Schematron loader for Saxon.Net and C#

2010-06-04T16:43:00.006+01:00

I'm introducing an ISO Schematron 'compiler' into the CoherentWeb XSLT tool. This uses the open source XSLT implementation available from http://www.schematron.com/

To load a schema into CoherentWeb, first press: Schematron > [SVRL | MSG] (underneath the 'Recent XSLT' list) to generate the XSLT.

With this XSLT, you can now initiate a multi-threaded batch-validation of the set of files (zip-compressed or otherwise) by just pressing F5 .

After validating, you can review the SVRL output for any file by switching the XML viewer to 'Output' and browsing through the file list. Alternatively, with xsl:message output selected, you can see validation failures 'at a glance' from the file-list icons (see figure), and the adjacent message box.

CoherentWeb is a proprietary app written in C# and uses a bundled Saxon.NET 9.2 PE processor. The C# code used for compiling the schema into XSLT is shown below, error-checking and XSLT parameter handling code has been (mostly) removed.

          public static string ImportSchematron(string schemaPath, bool forMessage,
Processor sxnProc, List<Xslt2Parameter> paramsList)
     {
         string outPath = CreateDerivedXslPath(schemaPath, forMessage);

         string baseXsltPath = Settings.ResourceFilePath + "\\xslt\\";

         string[] schConverter = new string[] {
             baseXsltPath + "iso_dsdl_include.xsl",
             baseXsltPath + "iso_abstract_expand.xsl",
             baseXsltPath + "iso_svrl_for_xslt2.xsl"
         };

         if (forMessage)
         {
             schConverter[2] = baseXsltPath + "iso_schematron_message_xslt2.xsl";
         }

         Uri schemaUri = new Uri(schemaPath);

         XsltCompiler xslComp = sxnProc.NewXsltCompiler();
         //////transform-1/////
         Uri xslUri = new Uri(schConverter[0]);
    
         XsltExecutable xslExec = xslComp.Compile(xslUri);
    

         XsltTransformer xslTrans = xslExec.Load();
         LoadAllXslt2Params(xslTrans, paramsList);
         DomDestination domOut1 = new DomDestination(new XmlDocument());
         using(FileStream fs = File.Open(schemaPath, FileMode.Open))
         {
             xslTrans.SetInputStream(fs, schemaUri); // set baseUri
           xslTrans.Run(domOut1);
         }
         //////transform-2/////
         xslUri = new Uri(schConverter[1]);
         xslExec = xslComp.Compile(xslUri);

         xslTrans = xslExec.Load();
         LoadAllXslt2Params(xslTrans, paramsList);
         DomDestination domOut2 = new DomDestination(new XmlDocument());
         DocumentBuilder docBuilder = sxnProc.NewDocumentBuilder();
         docBuilder.BaseUri = schemaUri;
         XdmNode inputDoc2 = docBuilder.Wrap(domOut1.XmlDocument);

         xslTrans.InitialContextNode = inputDoc2;
         xslTrans.Run(domOut2);

         //////transform-3/////
         xslUri = new Uri(schConverter[2]);
         xslExec = xslComp.Compile(xslUri);
         xslTrans = xslExec.Load();
         LoadAllXslt2Params(xslTrans, paramsList);
         XdmNode inputDoc3 = docBuilder.Wrap(domOut2.XmlDocument);
         xslTrans.InitialContextNode = inputDoc3;
         Serializer serializer = new Serializer();
         using (TextWriter tw = new StreamWriter(outPath, false))
         {
             serializer.SetOutputWriter(tw);
             serializer.SetOutputProperty(Serializer.INDENT, "no");
             xslTrans.Run(serializer);
         }
         return outPath;


     }

Launch Day

2010-02-08T18:27:00.005Z

Its finally happened. After 6 months intense work, covering about 25,000 lines of c# code, CoherentWeb has been 'launched' in beta form.

Naturally, I'm very excited about this product launch. This tool attempts to introduce a significant number of innovations to the world of XML developers. My last post covered some of the unique user interface aspects, and these really reflect that CoherentWeb doesn't fit an existing tool category well at all.

There are, of course, risks inherent in assuming that there's a functionality gap that needs filling with a new kind of tool. Its quite a leap to think that, if there were truly a gap, then you just happen to be the first developer to realise.

Another problem, ironically, is the ubiquity of XML in todays world of IT solutions. Trying to build a tool generic enough to suit all XML developers, makes it quite possible to produce one that suits none.

So there we have it. The question that so many startups before Qutoric is now being asked. Is there a sufficient market for this product?
As for the answer, only time will tell...

XML Tools: User Interface Design

2009-10-23T14:44:00.002+01:00

Working on CoherentWeb has provided me with some useful experience on User Interface Design issues in the context of XML IDEs. What follows is a random list of miscellaneous thoughts on this subject.

Icon Design:
There is now a trend to minimise the cluttered appearance by minimising the complexity and colour-set of the icons used. Hopefully this reduces distraction and will allow the user to concentrate more on the data they're analysing or modifying rather that the aesthetics of the host IDE itself.

Intellisense:
Most IDEs provid a single column popup intellisense list. Unfortunately there are often too many entry options in a specific context for this to be really useful (for example, when providing auto-completion for XPath expressions).

An alternative approach that is being experimented with for CoherentWeb is to use multi-column intellisense that is also categorised so that, for example, function names are not randomly mixed within a list of node names.

Window Pane Management:
Managing multiple viewing panes within an IDE poses a number of challenges for the designer. By allowing the user to hide, collapse or re-orientate any pane, you quick reach a pretty enormous number of possible permutations for rendering these panes together.

The solution here appears to be to offer a number of preset IDE pane configurations, perhaps aimed towards specific tasks and then let the user fine-tune things from there - more work on this is required.

Screen resolution, aspect ratio and multi-monitor setups:
More LCD monitors with a 'widescreen' 16:9 aspect ratio are now being used, placing one or more montiors alongside the main monitor dramatically increases this ratio.

Aspect ratio therefore becomes quite an issue. The approach so far for CoherentWeb is to adopt a 'TweetDeck' approach where multiple vertical panes are used in most cases rather than the classic IDE look with the window split into both vertically and horizontally.

The IPhone Influence:
UI designers can't help but look at how the IPhone designers tackled usability issues for the smartphone. There are some useful tips to pick up here, designing applications for use with touch screens is another facet to consider. I currently use a Wacom tablet to test CoherentWeb usability, but this will need to advance if touch-screens become more popular.

Why do many IDEs look so much the same?
If you look at popular IDEs they all seem to share a family resemblance. Why is this? Visual Studio, Eclipse, Oxygen, XmlSpy, Stylus Studio...the list goes on.

Does this mean that we've reached the peak in evolution of UI design for an IDE? I hope not. In fact, new products like Expression Blend and Expression Web for example do have a new take on the IDE appearance which is quite refreshing, but have they gone far enough? Microsoft introduced the Ribbon for Office 2007/2010, but this doesn't appear to have been adopted yet for developers.

Ambitions

2009-08-03T07:01:00.003+01:00

What would you like to do with your life?

This is hopefully the question we ask ourselves each day, consciously or otherwise. Well, for me I've decided to listen to the answer that has been screaming at me for quite a while, and so its time to act, I'm starting my own business.

Now, I realise this is going to be tough, very tough, but, I need to give this a chance. I'm going to be busy of course, there's a few issues to resolve like creating the company, organising accounts and sorting out legal support (just in case).

The intention is to arrange things so I spend a minimal amount of time on business support, to allow me to maximise my time on what I enjoy most, designing and writing software applications for others.

More details in following blogs.

In Print?

2009-05-21T19:24:00.008+01:00

I've just come across this:

http://www.ptpress.com.cn/Book.aspx?id=12089

ISBN】 978-7-115-19834-1

I think this is the first published book on XML in which SketchPath gets a mention.

The thing is, this is a Chinese language book, and as my linguistic skills are limited, I don't quite know what the website says about it.

Anyway, thanks to the author who mentioned SketchPath, Google translator tells me that 使用SketchPath调试XPath means 'Common XPath Debugger'.

Some time ago I got some really useful comments on SketchPath, especially about non UTF-8 encoding such as GB2312, this was from a Chinese-Language technical author. So many thanks to him for his contribution, and perhaps its his book at this website?

SketchPath Word 2007 Addin

2009-05-18T22:45:00.011+01:00

A quick taster screencast of the SketchPath Word 2007 Addin. Works extremely well for analysing large XML files:

XMLPrague 2009 Presentations

2009-05-16T08:27:00.003+01:00

If you follow XML trends then you need to check out XMLPrague 2009 if you haven't already.

There are some really interesting, thought provoking video presentations here, my favourites:

XML Schema 1.1 Update by Michael Kay
High-Performance XML (Using Frozen Streams) by Alex Brown
Exploring XProc by Norman Walsh

There are many more excellent presentations, kind of makes me wish I was there.

XSLT 2.0 Upgrade: Success?

2009-04-22T20:29:00.003+01:00

The rather tricky question of where XSLT 2.0 has got to, comes up every now and again.

Well, given the incredible benefits that XSLT 2.0 brings over 1.0, the more recent version seems to have severely underperformed, in terms of takeup.

In my experience there has indeed been very good 2.0 adoption with XSLT-based custom solutions in the enterprise. This however is more than countered by the painfully slow adoption in the more popular and visible parts of the IT/Web industry, in areas such as javascript libraries, MSXML, Web Browsers, .NET Framework, Java/JEE, non-XML IDEs such as Eclipse, Netbeans, Visual Studio... the list goes on.

What caused this?

My own theory is that the failure of Microsoft to embrace XSLT 2.0 had a severe impact on adoption, both by its partners, but also by its competitors.

What's going to fix this?

Well, perhaps we just need a 'killer' application, one that would really pale XSLT 1.0, fine language as it is, into insignificance. The chances are it already exists on some dusty website with a small but enthusiastic following and a lowly page ranking, all that's needed is for one key industry player with enough imagination to find it and go for it...so come on!

SketchPath Word 2007 Addin Update

2009-03-08T11:42:00.004Z

While very pleased with the technical outcome of branching SketchPath to be a Word 2007 Addin, there has been a very low takeup on this product, less the 1% of the 200+ downloads per week for the SketchPath product.

Until I see any further public interest in this, I'll be concentrating on the standalone SketchPath version. Though I might first try a bit of additional 'marketing' for the Addin, one post to the XSL-List is hardly exhaustive and is probably the wrong demographic anyway.

XQuery and .NET

2009-03-08T11:18:00.004Z

In a previous article I speculated that a Microsoft could be developing a .NET XQuery engine. I was wrong.

There have however been some important developments with XQuery. One that I'm watching with interest is XQSharp.

[Update Sep 3 2009] As per the comment below, XQSharp is now available for download, it can be purchased, but is free for evaluation or non-commercial use. ~~This looks promising because a .NET API for this is to be published, though there's some question as to whether use of this API will be free.~~

[Update Sep 3 2009] Saxon .NET 9.2 has been available for a few weeks now. There is a new range or Saxon versions, so check the site for details.

In the mean time, Saxon remains, in my view, the best XQuery solution for .NET. The only issue issue I have with the Saxon approach is that this is a native Java app that uses IKVM files to port to .NET, this complicates deployment of dependent applications somewhat. I have though found the Saxon .NET API well documented and simple enough to use.

Handling Large XML Files

2009-02-15T15:02:00.006Z

The Issue

A persistent feature request from SketchPath users has been for this XPath Tool to support larger XML files. The realistic current limitation is about 500KB, and even for this file size 'load performance' is close to being unacceptable.

Word 2007

I'm still investigating options and will now trial a version of SketchPath that integrates with Word 2007 rather than using the .NET RichTextBox control. This should provide support for XML files up to about 20MB, which, bearing in mind that 4MB of XML is about 1600 pages, should answer some of the concerns.

Ribbons

Using Word 2007 with .NET is very straightforwards now thanks to VS2008. There's even a Ribbon designer, though for any production version, its likely that I'll choose to generate XML to define the Ribbon as this provide more flexibility - especially when overriding existing Word functions.

The Approach - RTF

To minimise code changes, SketchPath for Word generates RTF from the XML source which is then loaded into Word in the background. There are alternatives of course but this solution already provides good performance and takes advantage of the way Word does 'progressive loading' for files.

The Actions Pane
Word 2007 provides a very neat way of integrating with WinForms, this is via the ActionPanes control which works just like a .NET form and thus a container for any of the Winforms controls. The main limitation here is that it appears to only have one ActionsPane loaded at a time, it would have been nice to give users both a vertical and horizontal pane.
Managing the Release

If this 'proof of concept' has some degree of success and there is to be a sister product to SketchPath then this will be managed to minimise the impact on the development and popularity of SketchPath.

Where next for SketchPath?

2008-11-05T09:38:00.002Z

The SketchPath project has, in terms of download count, and Google PageRank been a qualified success. It does not however receive much feedback and needs your help to progress in the right direction, so please consider helping.

What features or enhancements would you like to see in SketchPath? Or what particular annoyances need to be fixed?

I've listed below some of the areas I'm considering, which of these would you like to see - what should the priorities be? Anything else?

Handle larger file sizes [up to 20 MB] with acceptable UI responsiveness
Tidy up User Interface
Improved Multi-File support for Collection() function
Support XInclude
Web Service Support (SOAP/REST)
XPath Update Support
Schema-Aware Enhancements

Getting the User Interface

2008-10-27T19:37:00.005Z

I found the following on StackOverflow a couple of days ago:

http://stackoverflow.com/questions/238177/worst-ui-youve-ever-used

The 'nomination' for SketchPath:

SketchPath is a tool I use often, and I despise its UI. Utterly confusing, needlessly complex, non-intuitive... But its the best xpath tool I've found so far. Using it speaks volumes for the need to hide "expert" ui components from the average user.

So, SketchPath comes in here for a fair degree of criticism. This is the first bit of really unfavourable feedback I've seen on SketchPath, so its important to see if I can learn something from this if possible without getting too defensive, or even too analytical.

The main point that comes accross is the apparently contradictory notion that though SketchPath has the 'worst ever' UI, it is still used frequently by the poster and is the best tool they've managed to find so far. My initial thoughts are that, perhaps in some kind of twisted way, the poor UI actually makes the tool good.

Now, obviously no tool's UI is going to appeal to all users, but to get such emotive terms as 'despise' means that something has gone wrong, so what did go wrong, and why?

Well, again the poster has been very helpful by explaining the problem; overly complex user interface and non-intuitive. My response? Guilty as charged I'm afraid - there are many controls on the main screen that could have been easily hidden away and saved for some 'expert mode'. Also, many controls are used in unconventional ways, let me list some idiosyncrasies just one control, the 'Assist Pane':

A grid control is used for the 'Assist Pane' instead of perhaps the more obvious menu bar.
A scrollable 'IntelliSense' list is embedded within the non-scrollable grid control
The 'IntelliSense' list is enclosed within 2 toolbar columns that also don't scroll
One column of the grid is 'paged' by clicking on the column header

You probably get the idea by now, there are some pretty strange things going on here, and they should be non-intuitive because it's unlikely that users will have come across this behaviour before. Was this wrong? Well, that's for you to decide, it certainly flouts Steve Krug's cardinal web site usability rule: 'Don't Make Me Think'.

To be honest, a screenshot can not do justice to how different this UI really is, its the dynamic behaviour that must settle it for most - one way or the other.

So, did I make a mistake in experimenting in this way? I don't yet know. Its important anyway that conventions used in tools are challenged and evolve, both to reflect new uses but also the new capabilities of GUI frameworks like WPF, and, perhaps just to keep things interesting.

The following facts may also shed some light on some design decisions:

Half of recent development effort was done on a laptop with a low-res 15.4 inch screen and no mouse, the other half on an overclocked workstation with a 24 inch screen, plus mouse.
I have only a limited set of free icons that I can use, and my graphic design skills are poor
I suffer from intermittent RSI, most likely causes: Mouse scroll-wheels and right-clicking
I find Mobile Phone UI's (pre-IPhone) surprisingly effective once you're accustomed to them
A high percentage of SketchPath users will never look at the online documentation.

Microsoft XML Tools: Have your say

2008-07-11T14:25:00.001+01:00

The Microsoft XML Team is conducting a survey for XML Tools at:

http://blogs.msdn.com/xmlteam/archive/2008/07/01/xml-tools-and-technology-survey.aspx

Curious? Why not have a look. The questions provide some insight into where Microsoft XML tools might be heading (Question 29 for example), and you might as well have your say at the same time.

[Edit: The Survey is now closed, sorry for not updating this earlier - Question 29 was asking whether you would like the capability for Visual Studio to be used with 2nd party XML/XSLT processors]

SketchPath Demo Video

2008-07-09T17:13:00.001+01:00

The move to WPF

2008-06-11T12:08:00.003+01:00

Now SketchPath has got XPath 2.0 support built-in, the next stage is to more to using the .NET WPF and XAML, to give this tool more flexibility.

Progress has been fairly slow with WPF as I try out different techniques. Initial impressions are that this is an extremely powerful environment for developing visually rich applications. The cost however is that there is some additional complexity, because the 'off the shelf' controls really need to be modified to get the most from WPF.

Now, the bridging between XAML and C# can be somewhat tricky. XAML dependency properties need to map to CLR properties. XAML ControlTemplates are a very powerful way of enhancing existing controls. However, if you need to add extra properties to your control you need to dip into c# to create a Custom Control class. This acts as a useful wrapper and registers dependency properties with equivalent CLR properties.

The biggest issue seems to be referencing XAML resources such as a ControlTemplate from within c#. As an interim measure I'm actually using a hidden control's instance (declared in XAML) to get a handle on that control's template dynamically in c# to create further controls (a custom TreeViewItem in this case) but there's got to be a better way. Whenever I come across a method starting with 'FindResource("name")' or even, 'TryFindResource' it makes me wary.

Still, even with these teething problems, there's a significant improvement over Windows Forms. Because geometry and trigonometry are not my favourite subjects, I'm relying heavily on Expression Blend 2 to do much of the visual design work and Visual Studio 2008 Professional for the C# side. The integration between these two products is much better than it was, but its still not exactly seamless.

Top 10 XPath Expression Errors

2008-06-11T11:03:00.004+01:00

My List of XPath expression errors I come across most frequently:

Attempted use of the default namespace in XPath 1.0
Incorrect loading of namespaces into the XPath evaluation environment
Use of a != b when not(a = b) is probably what was meant
Mis-Capitalization. Either in location-paths or for operators (use 'and' not 'AND')
Using a predicate to find the nth child node of each parent - wrap the expression in brackets - e.g. (/workbook/sheets)/sheet[n]
Assuming that nodes are zero-indexed - a[1] actually returns the first node
Using a reverse-axis such as ancestor:: without considering the nodes are accessed this way in reverse-document order - sheet/ancestor::*[1] returns the node closest to element sheet - in other words its parent
Misunderstanding of the real context node for a relative xpath expression (If the expression doesn't start with a '/' - then its relative to the context node which may be modified by the host language e.g. XSLT)
Overuse of the // expression for descendant-or-self - adversely affects performance
Errors due to an excessively lengthy expression that is hard to document. If in XSLT or XQuery, try using variables to make the expression clearer, and to help understand errors when they occur.

XPath isn't hard. Its just that some expressions that look really simple, actually require a bit of extra thought and testing.

A New XPath Tutorial

2008-06-11T10:42:00.003+01:00

If you're new to XPath you might want to try out the following tutorial by Brett McLaughLin:

http://www.ibm.com/developerworks/edu/x-dw-x-introxpath1.html?S_TACT=105AGX01&S_CMP=HP

This is just the first part and you need to register on the IBM developerworks host site first.

One thing to note is that this is an XPath 2.0 tutorial. Though this isn't mentioned explicitly, its clear from some small details in the contents of the first part. No doubt in subsequent parts things will become clearer, but it would have been nice to have seen this mentioned up front.

The XPath tools recommended by Brett are AquaPath for Mac OSX and Stylus Studio for Windows.

The tutorial and resources have a Java perspective on things but this should be useful to all developers new to XPath. No standalone XPath 2.0 processors are mentioned in the first part, but in my view, Java developers will probably go for Saxon.

Disappointed that SketchPath doesn't get a mention, but I'm sure there's a good reason, probably commercial.

LINQ to XML / XSD and XPath 2.0

2008-06-03T18:41:00.003+01:00

Much has been written comparing LINQ to XML with XPath 1.0.

This is not a valid comparison, so to do my small bit to even things up, I'll briefly show XPath 2.0 (though I could as easily have used XQuery or XSLT 2.0) on a simple test case that was first illustrated here.

Sample 1: LINQ to XML:

(from item in purchaseOrder.Elements("Item")
select (double)item.Element("Price")
* (int)item.Element("Quantity")
).Sum();

Sample 2: LINQ to XSD

(from item in purchaseOrder.Item
select item.Price * item.Quantity
).Sum()

Sample 3: XPath 2.0

sum(
for $item in purchaseOrder/item
return $item/price * $item/quantity
)

Now, LINQ to XSD is certainly an improvement on LINQ to XML (the latter just looks ugly to me), but is it neater than XPath 2.0. And given the similarities, why invent yet another XML expression language?

As to when LINQ to XSD is coming, there appears to be some doubt though progress is definitely being made. Scott Hanselman commented on this in a blog article some time ago.

A Microsoft .NET XQuery Implementation: When?

2008-05-01T17:38:00.002+01:00

To me at least, the question now seems to be 'When?' rather than 'If?'

My reasoning:

SQL Server 2008 will ship with what is still a cut down version of XQuery
LINQ has already been released to much acclaim/marketing with Visual Studio 2008 / .NET 3.5

With these two significant releases out of the way, there is now nothing further to hold up a full XQuery 1.0 implementation in .NET. An attempt at an earlier XQuery move that overlapped with these 2 products would have had potential to cause significant confusion, and a few headaches for the marketing guys.

There are of course a number of other factors that will weigh on the decision-makers minds:

Oracle's Out-Sourcing of XQuilla
The now significant number of serious 'Industrial-Strength' XQuery implementations. XQuery has achieved 'Critical-Mass'
Making full use of the LINQ to XML/XSD and SQL Server XQuery talent-pool while its still 'fresh'
.NET 3.5 is now significantly better placed for impementing incremental changes to standards through extension methods (as used by LINQ).

This raises of course a number of follow up questions, these are just a few:

Would a .NET implementation now make much difference to the uptake of XQuery?
Would a .NET XSLT 2.0 implementation come at the same time?
Would this be too too late for Microsoft anyway, have they missed the boat?

This is of course only a personal view, and I may well be wide of the mark. But to me the timing seems about right...

Upgrade Time

2008-04-01T19:30:00.003+01:00

Yes, its time. Time to make that move to XPath 2.0 - I've previously mentioned the need to upgrade SketchPath from XPath 1.0, and now seems to be about right. XPath 1.0 will of course continue to be supported - and features added as required.

I've decided to go for Saxon.Net as the XPath 2.0 processor for SketchPath. There are three main reasons:

1. Saxon is well established as a standards compliant processor, for XPath, XSLT and XQuery
2. A good repuation for processing Performance - especially the more highly optimised SA version
3. Very good documentation for the Saxon.Net API, and as I'm finding now, an API that is well suited to the SketchPath task: Evaluating XPath without the benefit of a 'host' language like XSLT and XQuery.

How far have I got? Well, its early days, and the first priority was to get the basic functionality in place before added more advanced features in future releases. So, the main changes to date:

1. There's a new item to choose when selecting the expression engine, so you can now choose Regex, XPath 1.0 and XPath 2.0 - Also the settings pane has been extended to allow the setting of the default XPath processor.

2. The results list now lists the atomic types as specified by the XPath 2.0 Data Model (XDM)

3. When XPath 2.0 is selected instead of XPath 1.0, the Assist Pane is modified to reflect changes to the XPath language.

3.1 The column used for the 4 pages of XPath 1.0 functions is instead used to show 4 pages of XPath operators that are grouped (loosely) according to category - A 5th page will probably be added later for XPath type operators such as 'cast as'.

3.2 The Intelli-List will now list all XPath 2.0 functions alphabetically, so they can be auto-selected when typing in the expression editor. Typing 'fn:' in the XPath editor or clicking the 'fn:' column heading populates the function list ready for selection

3.3 A new 'document-node' node-test has been added to the node-tests column

4. The most challenging (and interesting) task so far:

Modifying the built-in XPath parser that is used to colorize the expression, provide Intellisense and allow XPath expression tracing. There are many new operators to consider but the new sequence operations using for $, every $ and some $ proved especially challenging - effectively they create a localised variable whose context must be established and then maintained no matter how deeply nested within predicates or functions. Further testing on this required but early signs are good.

The 'To Do' List
Having completed an initial (successful but not published) early proof of concept for the forthcoming Schema-Aware version, I'm now reverting back to adding extra refinement to the basic features, so on the list I've got the following:

1. Provide 'mouse-hover' help for each of the XPath functions
2. Allow tabs and line-feeds to be inserted into the expression - for formatting purposes
3. Support XPath 2.0 comments
4. Add a 'smart' mode so that expressions are not evaluated continuously if an item is being selected from the Intelli-List
5. Provide an auto-format feature in the XPath editor to allow nested parts to be indented on new lines
6. Add a 'high-light matching bracket feature - expressions are getting longer and they need this feature now.

Well, thats a list that should help serve as reminder to me - other requirements are associated with improving the file load performance and increasing the file-size capacity of SketchPath, 100 KB files are the current useful maximum.

I will try to put in some Saxon.Net code samples here later if there's any interest shown.

If you want to give it a go, the XPath 2.0 technical-preview (pre-production) version of SketchPath is now available for download from a link from the main SketchPath page - note that because its pre-production it is not on the main downloads page.

XPath & Intellisense

2008-02-24T22:14:00.002Z

The very composability of XPath makes designing a smart Intellisense feature somewhat of a challenge.

How does the system know if the currently itemed type is going to be a node-name, an axis-specifier or a function? Well, the answer is, it doesn't - but I'm looking into this, so any ideas would be welcome.

Currently SketchPath only dynamically populates node-names and values where the context in an XPath expression makes them legal. Other XPath constructs are available through a static grid (called the Assist Pane).

Even node-names in intellisense are less than obvious. It wouldn't be that useful, for example, to wait for the full prefix to be typed before selecting the nearest name match.

Anyway, this is one area I'm concentrating on in the next release of SketchPath - this should help out those users who like to keep their hands on the keyboard, rather than straying to the mouse every minute - some extra conventions for shortcuts may need to be introduced to facilitiate this.

SketchPath Production Release

2007-12-20T14:14:00.000Z

SketchPath 1.0 is now available for free download.

Update Summary

Improved XPath Trace Logic for functions within predicates
Settings for XML and XPath Editor font sizes now adjustable
All configurable settings now persisted between sessions
Improved handling of CDATA in results highlighting and when generating XPath expressions
Expression Auto-Generation now supports Comment and Processing Instruction nodes
XPath expressions (including 'Unfiled' expressions) saved automatically on exiting SketchPath
'Unfiled expressions from previous session loaded automatically on launching SketchPath
Disabled XML Save function now re-enabled
Import XPath feature enhanced
Command Line parameter '/l' [lowercase 'L'] can be used to set the launch XPath library
Command Line parameter '/f' can be used to set the launch XML Instance File
Nodes pane now supports CDATA
Nodes pane now lists Processing Instruction and Comment Nodes
Nodes pane 'name' column now includes position predicates where required
TreeView Trackbar now indicates level of nesting of selected node
Performance of XPath Import feature improved

XPath Survives Alongside Linq

2007-12-03T21:09:00.000Z

With the understandable hype surrounding Linq to XML and the pushing back of XPath 2.0 it would be quite easy to believe that XPath was being sidelined within Microsoft solutions. Not so.

Check out Mike Taulty's screencast - Linq to XML: bringing in XPath This shows how XPath can usefully complement Linq using System.Xml.Xpath extension methods: XpathSelectElement, XpathSelectElements, XPathEvaluate and CreateNavigator.

XPath lives on...

Microphone Shy

2007-11-11T16:17:00.000Z

Hmm... Just done my first attempt at a voice commentary over a Flash demo - not brilliant, but didn't go too bad either. The 'urghh ummm...' count is rather high though, so will have to work on that.

There must be some tips out there on how to do a good podcast or screencast - more research required on my part, but in the mean time - my instincts are:

1. The demo needs to be kept short - both to keep listeners interested and to keep file size to a reasonable level.

2. Working to a script doesn't work unless you're a professional who knows how to make it sound natural

3. The mood needs to be 'light' without overdoing it

4. It would help to have another 'voice' to ask questions or to interrupt with the odd explanation

For some, this must all come naturally, but for the rest of us I guess its something that takes time. If you want to hear/see my first demo its at: http://pgfearo.googlepages.com/intro.html

Why All Tools May Suck

2007-11-10T18:52:00.000Z

Reading an old blog: All Tools Suck in 'Dr Macro's XML Rants' provides food for thought.

In a way I can agree: All tools have some failings and are not able to meet the standards they attempt to support. In the majority of tools, edge cases are not addressed in great detail. The key question to me is really about why we use tools at all, and how good they really need to be...

Using the (rather distant) analogy of our cave-dwelling ancestors and their first tools:
In order to create their first cutting tool I guess they'd just bang one rock against another until sufficient chips came off one of them to create a sharpish edge. But this would just be the beginning, they would then be able to use the product of this to create further tools with more efficiency, these tools in turn could be used to create more specialist flints for spear heads or hand-held blades.

So, the original tool (in my view) doesn't have to be perfect, so long as its better than banging two rocks together then its ok in my book. If a tool's not reliable enough or flexible enough for one purpose, then create another that is.

This, along with the fact that standards are always changing and new standards created, is why there will always be room for new tools. The Swiss Army Knife approach works well to a point but the specialist stand alone (perhaps disposable) tool will always have a place too.

SketchPath certainly has weaknesses, but many of these are deliberate compromises in the quest for a practical solution for a broad range or problems that could be delivered at reasonable cost (to keep the tool itself free).

On standards, I can think of 3 main cases where SketchPath deviates from the XPath 1.0 standard.

1. 'XPath variables' are supported.
In the XPath core standard, variables are not supported - XSLT provides the necessary extensions for this instead. The way SketchPath implements variables, by simple macro substitution, doesn't follow the XSLT 1.0 result tree fragment/value approach but is hopefully still better than nothing for XSLT developers.

2. There is some auto-conversion of XML character entities if they are used in an XPath expression - Non-standard, but a convenience feature for XSLT users

3. Default Namespaces. XPath doesn't support the default namespace, but SketchPath does. This was to save end users the trouble of deleting/modifying namespace declarations, the option is also provided to use the 'def:' prefix - which is probably a better choice. Again, a 'convenience' feature which may not sit well with purists.

Any chance in keeping standards fixed (before they superceded) for long enough so we can see more decent implementations?

Well, this may be happening incidentally due to the 'standards democracy' that currently prevails leading to delayed standards ratification etc. Unfortunately, just the knowlege that the next version of a standard is in the pipeline is often enough to make an implementor wait.

Where is XPath Used?

2007-11-07T20:07:00.000Z

The most popular use of XPath still has to be in systems exploiting XSLT. However, things are changing now that a number of additional formats/ standards are becoming established, here's my list:

XPath implementations:

1. XSLT (Transforms)
2. XQuery (Data Query)
3. XForms (User Interface - Data Binding)
4. XAML (User Interface - Data Binding)
5. XProc (Processing Rules - Content Management Systems)
6. Schematron (Validation / Security)
7. Vista Event Query Language (System Monitoring)
8. Open XML (Word Processing / Content Binding)
9. BizTalk Server (Orchestration Rules)
10. Sharepoint (Form views and filters)
11. InfoPath (Data Binding)
12. BPEL (Superset)
13. FPML (Financial Product Rules)
14. W3C Schema 1.1 (Validation Rules)

I'm certain there's more than this, but hopefully this illustrates how widely XPath is now used. The sector breakdown of SketchPath users reflect much of this diversity.

XPath++ ?

2007-11-01T19:22:00.000Z

I'm starting to wonder why XPath 2.0 is called XPath 2.0, why not XPath++ ? I guess the intention was that everyone moved from 1.0 to 2.0. This just hasn't happened yet, and with Microsoft pushing Linq for XML it doesn't look like its going to happen soon.

I know that eventually I need to provide a XPath 2.0 option for SketchPath. The trouble is, until I've actually used the new expression language, I've got no idea how to design a tool for it. I'm guessing that it wont just be enough to 'slot in' another evaluator. The UI is going to have to smarten up to deal with cool stuff like sequences (WPF might help me out here).

The best thing would be if my next work project used XPath 2.0. Then, whilst I struggle to learn the many new features and the altered approach, I can be thinking: 'how would I want a tool to help me do this?'

That's how SketchPath first happened. While I was learning the meaning of predicates, nodesets axes and mixed content models it was an ideal time to observe what I needed, and understand that if no one was going to do the job for me, I'd have to build the tool myself. Otherwise, I'd have built something, but it just wouldn't have ended up as SketchPath.

There's a steady list of improvements and fixes for SketchPath building up, so I've got plenty to keep me going - hopefully still on course for a production release before 2008.

Taking Stock

2007-09-11T19:10:00.000+01:00

Well, the initial download rush for SketchPath has abated somewhat, so this now provides an opportunity to take stock and consider priorities to move towards that first release version.

Things that need sorting:
1. Fill in feature gaps such as:
a. Auto-generation of XPath for comments and processing instructions
b. Tracing through parenthesized expression blocks
c. Population of Value Nodes pane with comments and processing instructions

2. While keeping the user interface simple, provide more intuitive flexibility with positioning and orientation of panes

3. Allow more control over various settings and permit these to be persisted between sessions.
4. Fix bugs

So, the list doesn't look to imposing, the main thing is to keep ambitions in rein for v1.0 this will also allow me to do a better job for the documentation and a few 'off the shelf' XPath libraries.

SketchPath Update

2007-09-09T11:21:00.000+01:00

The latest release of SketchPath, 0.5.5.0 (Beta) went out today.

This update is part of the continuous improvements that will happen until version 1.0 is ready, before the end of 2007.

Latest enhancements:
1. Unicode support for opening and saving XML - New Settings view allows configuration for this.
2. In the Regular Expression mode, its now possible to switch Case Sensitivity on or off
3. The XML text highlighter is now improved to enhance the readability of the highlighted section
4. Double-Clicking on any part of an XPath expression now takes you directly into Trace Mode for that part.
5. The Assist Pane has enhanced usability
6. A number of minor bugfixes

Please let me know what I can do to improve SketchPath - I've got a long list anyhow, but it would be good to prioritise this.

Anyone got any good (non copyright) XPath libraries they'd like to share?

Looking Good

2007-09-03T22:45:00.000+01:00

Spent the last couple of weeks giving the SketchPath Assist Pane a makeover, release 0.5.4.1 went out 2nd September with this enhancement.

Things have improved so much with the usability of the Assist Pane that I was confident enough to leave it visible as the default setting. Even when 'hidden' the 'Intelli-List' is still shown, this can be such a powerful aide and it takes up minimal room alongside the results pane.

This is probably my most preferred area of programming, looking at user interfaces and trying to make them more efficient whilst not being forced to conform to other adopted standards. Something isn't necessarily intuitive because lots of other tools use the same behaviour.

Had a bit of a breakthrough in performance issues with RichTextBox highlighting that has bothered me for some time, hopefully this will be evident in the next SketchPath release next Sunday.

Anyway for now, I've included a screenshot fragment of the latest release

On the up?

2007-08-23T08:51:00.000+01:00

In the last couple of days the SketchPath download count has seen phenomenal growth. Before Tuesday, the average daily number of downloads was about 3, on Tuesday this went up to 50 and for Wednesday this had reached the dizzy heights of 183! I imagine things will level off a bit now, but still thats not bad for a small project.

This increased interest seems to be mainly due to positive comments posted by a small handful of bloggers, so my thanks to them, hopefully one day I can contribute in a similar way.

There is now added impetus to improve SketchPath in this Beta stage, but also an additonal sense of responsibility, whilst not wanting to hold back some of the experimentation that has gone on with this project and helped differentiate it from other products (I'm not saying SketchPath is better here, just different).

Yesterday (last night), I put in an extremely minor change to hopefully improve support for the international character set. There are still many limitations, and most of these seem to be imposed by RTF. I am now keen to investigate the use of WPF as soon as possible, this seems to be the way to go

RTF, RichTextBox .NET and Encoding

2007-08-22T09:09:00.003+01:00

[Update: If you're dealing with large RTF files (say 4MB), the LoadFile() method of Rich TextBox is real slow, the InkEdit control inherits from RichTextBox and is sooooo much faster. This control only works on Vista, WS2008 x86 and Windows 7, not Windows XP (unless tablet enabled), it probably also works on WS2008 R2 but I haven't yet tested this. Note: To allow the InkEdit control to function on an x64 OS, you will need to set the target platform to be x86 in the Visual Studio build settings page]

A SketchPath user advised me recently that SketchPath has no foreign character set support. This came as a bit of a surprise because I thought I had done all the unicode / stream stuff as per the book (Obviously some more thorough testing on my part would have shown this up).

It comes to light that its the RTF that SketchPath generates for the .NET richtextbox that is at fault, so I'm investigating. Below is a summary of my progress:

I spent some time on this unicode/rtf issue last night, but had no real progress. This morning I found some threads on this, such as:
http://blogs.msdn.com/michkap/archive/2007/06/21/3431070.aspx
It seems that rtf and unicode don't go together that well. I'm tempted now to put this on hold until I start using .NET 3.0/3.5 WPF controls in SketchPath, there should be much better support for unicode in these.

If I start out by just replacing the RichTextBox with the WPF equivalent control but still embedded in a WinForms application that should save some time. A few months ago I did some proof of concept stuff in WPF with rich text, it seemed to work well but performance (especially scrolling) in the early betas was poor, now could be the time to look at this again.

I'll have one last push this evening with rtf (I have some more ideas to try out), but if I make no real progress I'm afraid it will be delayed until I can get WPF working in SketchPath, this could take a month or so (SketchPath is a 'spare-time' activity at the moment).

Now SketchPath seems to be gaining in popularity, I realise there's lot of people out there who need this, so will treat this as a priority.

[Belated Update]
The unicode problem with the RichTextBox was partially resolved some time ago. There are still RTF rendering issues though if Unicode from different character blocks in encountered.

C# code similar to the following [untested] was used:

public string ConvertUnicodeForRTF(string rtfUnicode)
{
// Regular expression to match non-ascii characters
Regex regNonAscii = new Regex(@"[^\x00-\x7F]");

MatchCollection mc = regNonAscii.Matches(rtfUnicode);
Match foundMatch;
string insert;
if (mc.Count == 0)
{
return rtfUnicode;
}
for (int i = mc.Count - 1; i > -1; i--)
{
//start with last match
foundMatch = mc[i];
char c = foundMatch.Value[0];
//remove unicode character
int dec = Convert.ToInt32(c);
rtfUnicode = rtfUnicode.Remove(foundMatch.Index, 1);
insert = @"\u" + dec.ToString() + "?";
rtfUnicode = rtfUnicode.Insert(foundMatch.Index, insert);
}
return rtfUnicode;
}

SketchPath begins to surface

2007-08-21T12:40:00.000+01:00

My thanks must go both to Mike Ormand and Derik Whittaker who have tried out SketchPath recently and have written positive comments in their respective blogs.

The number of SketchPath downloads has increased from roughly 2 per day to 42 in just the last day, so this is encouraging and gives me added motivation to continue to improve the product ready for a release 1.0 before Christmas.

The most recent release (0.5.3.5), had many 'unseen' improvements to the back-end XPath debugging and also results step-through, especially for complex mixed-content cases. However the most noticable change should be useful to those struggling with large XPath expressions: SketchPath expression editor has gone multi-line!

Initially I didn't see multi-line editing as a priority, wishing to promote the use of variables to break-up expressions. However, the use of increasingly high levels of nesting in XML schemas means that path locations can be extremely long, even on their own, so moving to multi-line was inevitable. There's still some way to go, because I'd like to support insertion of linefeed characters to help make expressions more readable (these currently get stripped out).

Currently I'm investigating issues with RTF and Unicode that appears to be the reason for non-ASCII characters being converted to '?' characters.

The 'To Do' List

2007-08-17T07:23:00.000+01:00

Improvements that I want/need to squeeze into SketchPath before Version 1.0 (Release before Christmas?)

Full resizeable multi-line expression editing window
XPath parser must remove line-feeds before processing, and re-insert them afterwards (in the colorized RTF output)
More efficient import of XPath from other XML documents
Improvement to Auto-Complete keyboard shortcuts - required for multi-line expression editing
XPath generator should provide relative paths if the context node is set
A quick 'proof of concept' look at improving the XML editor
Some minor tidying up of the user interface
Improved CDATA parsing for the element nodes pane
Create or locate a good source XML file and associated XPath library to be used as a 'getting started' aid.

So, the list doesn't look too long - I think I'll stick with this unless I get any more special requests.

What's in a name?

2007-08-13T07:55:00.000+01:00

The SketchPath project is progressing well, but fairly soon now I'm going to have to start thinking of how I label the 'post-Beta' version? Does this go out as a 'Community Technology Preview' or a 'Candidate Release' or something else?

If this were a commercial concern then one of the above would probably be appropriate, however, being a small private venture makes it very tempting to go straight for the 'Version 1.0'. Most users regard any software with a '1.0' label as being untried anyway and treat it with some justified caution.

The alternative is to keep SketchPath going as a Beta for some time to come, this however leads to the temptation to include new features before getting rid of all the significant bugs/annoyances, so as tempting as this may be, I shall resist and go for a 'shipped/shrink-wrapped' version as soon as possible. So the goal is 'Version 1.0' before the end of 2007!

Debugging XPath
The visual XPath debugging feature has been taking up quite a bit of time recently, I feel this has been worth it as it seems to be a fairly unique feature amongst XPath tools and so could be regarded as a differentiating factor. Also, now that this has been properly combined with auto-complete it provides yet another way to build/modify XPath expressions.

Comments and PIs
Whilst XPath comments and processing-instructions are fully supported for evaluating expressions and analyzing results, there is no proper XPath auto-generation for these yet, if you select such a node from the source XML. At one stage I regarded this as such an 'edge case' as to be worth ommitting. However, having looked at some Schematron scenarios where much use may be made of PIs and comments this is probably worth including.

Relative Paths
This must be resolved soon. The resolution of a relative path should be simple enough, given a context path and an absolute path, also there are no significant user interface issues with this as its logical, that if you have anything other that the root node set as the context, you expect paths relative to the context node to be auto-generated.

Namespace Manager
Finally, I've got around to upgrading the Namespace manager so that once you've overridden the declared prefixes in the source XML, you don't need to keep changing them each time you reload the same document or other XML documents with the same set of namespace URIs. Instead, the modified prefixes are preserved and a 'Reset' button appears which may be pressed to sync the prefixes back with the source XML. The real demand for this actually came from my own work, where I continously using prefixes that don't match those in the source XML, it started to become a real chore to keep on resetting the prefixes each time I refreshed the XML, hopefully this will help others too, in time.

SketchPath Version 1.0?

2007-07-05T21:07:00.000+01:00

Will there be a finished version of SketchPath?

The answer to this depends on a number of factors. In order to get a software product sufficiently tested for a public '1.0' roll out there has to be a significant test period.

The hope was that a public Beta test would assist in this, however, a month after the first Beta release, and over 100 downloads later, only a handful of responses have been received. This hardly constitutes a thorough test phase!

Mindful of this, there is now some doubt about the viability of a 'stand alone' freeware XPath tool such as SketchPath. The temptation is therefore growing to discontinue public Beta development of SketchPath and instead see if a specialized version of it can be included as part of a 'business package' for contracts with selected customers. We will see what the future holds...

XPath Debugging

2007-07-05T20:58:00.000+01:00

After having done quite a bit of work recently on the user interface parts of SketchPath, its time now to get back to basics and make sure that the XPath debugger is sufficiently enhanced.

The 'Predicate-Aware' step-through debugging, could be a unique 'selling' point for SketchPath so its important that it performs to spec. Most of the predicate handling now works but now this has to be combined with stepping through XPath functions within an expression in a more useful way.

The next release of SketchPath (0.4.2.0) will therefore not see significant changes to the user interface but there should be noticable improvements in the debugger, especially for expressions using funcions.

User Interface for Managing Grouped Items

2007-07-02T09:09:00.000+01:00

Whilst working on the latest release of SketchPath, I had the problem of trying to provide a user interface for moving items between different groups. A tree-view was one option I considered, but because these groups are not nested, there is probably no need for this.

Instead, I decided to go for the approach used by software, such as backup tools, and also interestingly, on mobile phones, where you first mark the items you want to perform an action on (in this case move them) and then you perform the action.

Whilst it takes a bit of time to get used to this, I found it extremely powerful. This was especially the case when synchronising two grids of the same data. The 'master' has all columns, whilst the secondary pane has a subset of columns and is also filtered, both by a selected group name and by whether an item is marked or not.

This combination of grids allows the user to view, at the same time, the other items in the group that they are about to transfer items too.

A few simple action buttons, such as 'Mark Group', 'Mark All' and 'Transfer Marked Items' provide very powerful functionality. This does take some learning, but it seems to make for a more powerful and flexible way of managing grouped items that are not in a complex hierarchy.

Anyway, time will tell whether this approach works, more details on this part of the user interface on the SketchPath web page for the XPath Expression Library.

First Feedback

2007-06-27T07:40:00.000+01:00

Well, its humble beginnings but I just got my first bit of feedback for my product: SketchPath.

Well, actually it was more like a support request, the 'SketchPath' install button on my website didn't work for a user. I suspect this is down to the fact that the Microsoft 'ClickOnce' setup requires Internet Explorer.

The fix? Well, I've now made the installation package available as a zip file so it can be downloaded and then installed locally, rather than online.

My delight at receiving an 'issue report' email, underlines how significant feedback is in the software development cycle. Its not just that its essential to ensure a product works correctly and meets user requirements, its also important to the development team (me) as a motivator to continue to improve the product as their are people out there who might actually be using it.

This is probably especially true for freeware such as 'SketchPath', because there is no revenue as a motivation factor.

To alleviate the feedback problem, I may actually set up a registration form, to get the bare minimum of details on users. However I'll probably have to go to a service provider for this, to ensure I'm complying with the 'Data Protection Act', so hopefully this will not incurr additional cost.

Its Sunday so its Release Day

2007-06-24T17:25:00.000+01:00

SketchPath 0.4.08 has just been published, check my website for further details.

XML Tools

2007-06-23T17:16:00.000+01:00

What features does the XML community really need in a tool that isn't readily available already?

This is the question that probably occupies the minds of a number of software vendors, but how easy is it to answer?

The thing is with XML is that its very flexibility means that no one tool can and should do everything. The key is to get the right balance, and also allow integration with other tools that specialize in other XML standards.

So what XML standards/formats can be described as 'Core', which most tools should provide support for?

Here's my view, in order of preference
1. XML itself - Support for XML editing features
2. XPath - As this is at the core of so many other standards, I'm including this second
3?

And here I struggle to complete the list. Yes, XSLT because it used both for XML data mapping purposes and for generating HTML output, but there must be many applications that use XML but don't use XSLT. So how about a Schema Definition Standard such as WXS or DTD or Relax NG or Schematron? Well, yes, but if you include one, perhaps you should include them all.

Perhaps XQuery would be next. This would appeal to those using XML in a database role, however, its unlikely to appeal to web designers.

How about support for some of those other key standards/formats like:
XSL:FO, ODF, OpenXML, XPS, XForms, XAML, XUL, DocBook, SOAP, WSDL?

It seems to me that this is quite a lot to take on for a single tool. What is probably needed is three types of XML tool:
1. A generic Eclipse-like environment that comes with some built in support but where lots of plug-ins can be added to support the specific standards required.
2. A 'Do It All' environment where everything is bundled, more or less, in a single integrated package.
3. A specialised tool that just tries to support one function or one standard/format and do it simply, efficiently and well.

All of the above approaches have merit, and there are also 'grey areas' in between these categories, but which philosophy would win through? The answer probably varies according to the individual taste/skills of those requiring a tool, as well as, the nature of the problem they're trying to solve.

My instinct is that the tool most likely to survive is one that can evolve and adapt but this doesn't necessarily rule out any of the above.

Some Progress

2007-06-19T07:50:00.000+01:00

The Element Nodes Pane in SketchPath is now more or less complete. The only remaining issue here is to ensure this is automatically updated as the source xml is modified.

Recently I spent time on the variable substitution logic. Discovered a pretty fundamental oversight. I wasn't containing the substituted expressions in parenthesis, thus ending up with a 'broken-out' expression from true = $name to true() = /book = 'winnie', instead of true() = (/book = 'winnie'), This is now fixed, but provides a good illustration on the potential pitfalls within XPath expressions.

Missed Deadline

2007-06-18T07:50:00.000+01:00

Well, I missed my own deadline on the next SketchPath update. Most functionality is completed for the Element Nodes pane, but there are some reliability issues associated with the xml parsing to populate this, I'd rather sort these out first, so next update not till Sunday now.

The good news is that the new features, especially the additional support for mixed-content are looking good. I've decided to use color instead of icons to help distinguish between different node types, there are possibly accessibility issues with this approach, but lots of tiny icons are hardly ideal either.

One UI decision on the Elements Pane that I'm still grappling with is how far to go with the 'look-ahead' in the nodes pane. At the moment I'm just showing all attributes, text nodes and sub-element and their text nodes. If I go much further than this, there is a performance hit, but also its a cause of possible confusion, and it duplicates the function of the element tree view anyway.

Still putting of a decision on 'medium-term' stuff. Do I go for integration with PowerShell and its XML export capability, or do I start the move towards support for XPath 2.0?

Next Beta

2007-06-17T17:39:00.000+01:00

I'm not sure what the protocol is for Beta releases. Currently the user base (if any) is so low that its not an issue with my weekly updates.

I'm working now to finish publishing the latest release this evening. If all goes well, there will be better support for mixed content within the Element Nodes pane. I've also added a splash of color to this, to help differentiate between text nodes, attributes and sub-elements (new).

If anyone's had a go with SketchPath it would be good to hear from you.

Another Beta Release

2007-06-15T18:51:00.000+01:00

Early this morning (before work) I uploaded the latest beta version of SketchPath onto my site. This is actually the 4th release of the 1st Beta, but I'm hoping I'll be forgiven for brushing over this on my website.

SketchPath is the culmination of a home-project spanning (on and off) about 18 months. I'm getting to the stage now where, even though I've still a few ideas left of my own, it would be nice to test the water and find out what people out there would like in an XML tool centred on XPath.