Promoting reoccuring elements

Found one useful article written by eliasen: Well, I suppose we have all been there – in order to get the business process running, a specific element from a schema needs to be promoted in order to route on it, correlate on it, and so on.

Unfortunately, elements that can occur more than once can not be promoted. This, off course, makes perfectly sense, since the property can only hold one value, and how would BizTalk know which one of the many occurring elements to take the value from at runtime? So we agree with the limitation, but hope for a nice solution. 

If you try to promote a reoccurring element, you get this error when adding it to the list of promoted properties

“This node can occur potentially multiple times in the instance document. Only nodes which are guaranteed to be unique can be promoted.”

Right. Now, some people have found the editor for the XPath describing the element that one wants to promote. If you have promoted some element, you can click on it like this

Then you can click on the dot at the right of the line, and get into the editor like this

Now, wouldn’t it be lovely, if you could just change this expression to include for instance an index on the reoccurring element? In my example from this screenshot, the “ReoccuringRecord” record can occur multiple times. So it would be nice, if I could just change the XPath to be like this:

/*[local-name()='ExampleRoot' and namespace-uri()='http://PromotingReoccuringElement.ExampleSchema']/*[local-name()='ReoccuringRecord' and namespace-uri()=''][1]/*[local-name()=’ElementWhereNumber1IsPromoted’ and namespace-uri()='']

By setting the “[1]” into the XPath, I state that I will be needing the first occurrence of the ReoccuringRecord and therefore, this XPath expression will always give me exactly one node. Unfortunately, the engine can not see this, so the error will be the same, only difference being that this error doesn’t occur until compile time:

Node "ElementWhereNumber1IsPromoted" - The promoted property field or one of its parents has Max Occurs greater than 1. Only nodes that are guaranteed to be unique can be promoted as property fields.

Bummer!

So how do we get this working? If I really need to promote a value that occurs in an element that might occur multiple times, I see four options:

1. Map to a schema on receive port
2. Custom pipeline component
3. Orchestration to do it and then publish to MessageBox
4. Call pipeline from orchestration

I will go these options in more detail here:

Option 1: Map to a schema on receive port.

When a map is executed on a receive port, some extra magic functionality is performed by BizTalk. After the map has been executed, the message is sent through some code that promotes properties that are specified inside the destination schema. If you execute a map inside an orchestration, this doesn’t happen.

So you can create a schema that has an extra field, in which you place the value that needs to be promoted. This element must not be able to occur multiple times. Promote this new field, and after the map on the receive port has been executed, you have your value promoted.

Option 2: Custom Pipeline Component.

It isn’t that difficult to create a custom pipeline component, that can promote a field for you. Your Execute method might look just like this:

public Microsoft.BizTalk.Message.Interop.IBaseMessage Execute(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)

{

    pInMsg.Context.Promote("MyProp", "http://ReoccuringElement.PropertySchema", "MyValue");

    return pInMsg;

 }

Of course, you will probably want to load the body stream of the IBaseMessage somehow, in order to find the value inside the body to promote and then replace "MyValue" with the value form within the XML.

Just use the pipeline component inside a custom receive pipeline, and you are all set.

Option 3: Orchestration to do it and then publish to MessageBox

Create a intermediate orchestration, that gets the input message. Then, it should create a new message of the same type in a message assignment shape like this:

NewMessage = InputMessage;

NewMessage(*) = InputMessage(*);

NewMessage(MyNewProperty) = xpath(InputMessage, xpathexpression);

Then, use a direct bound port to publish the message to the MessageBox. In order for the new property to follow the message, you need to initialize a correlation set on the send shape that is based on this new property.

Let other orchestrations and send ports subscribe to this message and let then do their work.

Option 4: Call pipeline from orchestration

The last option is to call a receive pipeline from within your orchestration. This requires a new schema, that has a field for the value to be promoted, just as in option 1. Inside your orchestration, map the input message to this new schema, and call a receive pipeline with this new message as a parameter. Remember to promote the field in this new schema. There is an article on MSDN about calling a pipeline from within an orchestration, which can be found at http://msdn2.microsoft.com/en-us/library/aa562035.aspx

Upsides and downsides

In order to choose which way to go in a specific solution, several things need to be considered.

Basically, I'd go for option 1 almost anytime. This is because it is best practices to map anything incoming into a canonical schema anyway. So instead of promoting values inside all your partners schemas - schemas they might change, you should promote from within your own canonical schema.

Reasons not to choose option 1 include: The canonical schema also has a reoccurring element, so it hasn't provided extra functionality with regards to getting this specific value promoted. Or perhaps, we aren't using canonical schemas, because there was no time for this when the project was started.

If we can't go for number 1, I'd go for number 3. Number 2 requires programming of a pipeline component, which can be a bottleneck, unless done correct. Also, the pipeline component is a whole new component to maintain, document and test. Number 4 requires a new schema and therefore also a map to be built. If I am ready to do this, I'd go for number 1 instead.

If I don't like number 3, for unknown reasons, I'd go for option 2 - the custom pipeline component. Allthough it is custom code, and must be done right, and testet and everything... I still feel that creating a new schema and map in order to call the pipeline in option 4 is overkill, since I'd go for option number 1 instead, which would also require the new schema and map.

I hope this explains some details about this issue, and that it helps someone in the future.

Flat File Messages with Positional Records

Positional records within a flat file instance message contain individual fields (items of data) that are each of a predefined length. The fields are parsed according to these lengths.

The character used to fill the unused portion of each field, known as the pad character.

Field Padding: Pad characters are used in fields within both delimited and positional records when the data contained within the field smaller than the number of characters or bytes reserved for the field. These characters occupy the portion of the field not required by the data, if any. Pad characters are specified on a field-by-field basis using the Pad Character and Pad Character Type properties of the corresponding Field Element and Field Attribute nodes. If no pad character is specified for a particular field, the default pad character, space (" "), is used for that field

For inbound instance messages, regardless of whether a particular record is positional or delimited, the flat file disassembler discards leading or trailing instances for the specified or default pad character for a particular field as the instance message is translated into its equivalent XML form. Whether it is leading or trailing instances of the relevant pad character that are discarded depends on whether the Justification property of corresponding Field Element and Field Attribute node is set to Right or Left, respectively.

For outbound instance messages, the flat file assembler will insert the appropriate number of the specified or default pad character into fields so that the length of the field is correct. The pad characters will be inserted before or after the data characters based on whether the Justification property of the corresponding Field Element and Field Attribute node is set to Right or Left, respectively.

When the field to be padded in an outbound instance message is contained within a positional record, the Positional Offset and Positional Length properties of the corresponding Field Element or Field Attribute node, combined with the number of data characters that the field must contain, determine whether any pad characters are required, and if so, how many. When the field to be padded in an outbound instance message is contained within a delimited record, pad characters are only inserted when the value of the Minimum

Length with Pad Character property of the corresponding Field Element or Field Attribute node exceeds the number of data characters.

Extract Digits nd Char out of string with Regular expression

Extract Digits out of string with Regular expression

public string ExtractNum(string xnumberx)
{
        string[] numbers = Regex.Split(xnumberx, @"\D+");
        string RetNum = "";
foreach (string value in numbers)
   {
      if (!string.IsNullOrEmpty(value))
             {
                int i = int.Parse(value);
                RetNum = RetNum + i;
             }
    }
       return RetNum;
}

Extract Char out of string with Regular expression

public string RetString(string strspcex)
   {
      return Regex.Replace(strspcex, @"\d", "");
   }

Clean Telephone Number with Regular expression

using System;
using System.Collections.Generic;
using System.Text;
using System.Text.RegularExpressions;
using System.Windows.Forms;
using System.Xml;

namespace GOSI.Motion
{
    public class StringUtils
    {
       public static string CleanTelephoneNumber(string inputString)
           {
               return Regex.Replace(inputString, @"[\(\)\-\.\s]", "");
           }
    }
}

Disassembler Component, directly invoking command lines

Pipeline Tools


DSDump.exe
Enables you to dump the document schema structure, which is an in-memory lightweight representation of the one or more XSD schemas, with or without flat file annotations. This tool can be helpful when you get parsing engine errors such as $Root$0$3$2 and you need to decode them. Numbers after $ mean 0-based index or records as they appear in the document schema

FFAsm.exe
Runs the flat file assembler component, directly invoking it by emulating a send pipeline to enable you to see how it serializes or assembles a user's XML document(s) into a flat file document.

FFDasm.exe
Runs the flat file disassembler component, directly invoking it by emulating a receive pipeline to enable you to see how it parses or disassembles a user's flat file document into one or more XML documents.

Pipeline.exe
Runs a send or receive pipeline; accepts one or more input documents and their parts, XSD schemas and related information; and produces an output document after the pipeline runs.

Pipeline.exe does not access BizTalk Server databases, so pipelines containing BizTalk Framework assembler and disassembler components that access BizTalk Server 2004 databases during execution may not be supported.
You should only use pipeline.exe to verify custom pipelines without third-party custom components. If you use pipeline.exe to verify a custom pipeline with third-party custom components, pipeline.exe will produce the desired output. However, if you deploy the same custom pipeline with third-party custom components, use the pipeline in a receive or send port, and then use pipeline.exe to submit a message to the pipeline, the pipeline will fail and BizTalk Server will return an error

XMLAsm.exe
Runs the XML assembler component, directly invoking it by emulating a send pipeline to enable you to see how it serializes, assembles, or envelopes a user's XML document(s) into an output XML document.

XMLDasm.exe
Runs the XML disassembler component, directly invoking it by emulating a receive pipeline to enable you to see how it parses, disassembles, or un-envelopes a user's XML document into one or more XML documents.

USAGE: FFDasm.exe
C:\xx\xx\xxinterfacedefinitionlocation>C:\"Program Files"\"Microsoft BizTalk Server 2009"\SDK\Utilities\PipelineTools\FFDasm xxx.txt -bs xx.xsd

FlatFile xs:Date field with an empty or NULL value

I have seen this issue a couple of times and thought of recording this extract from one of blog.

Exception:

After generating a flat file schema with optional elements(minoccurs =0) of type date,int,decimal etc., which does not accept a ‘ ‘ value, we might endup with an error as follows.

error BEC2004: The ‘JoinDate’ element is invalid – The value ” is invalid according to its datatype ‘http://www.w3.org/2001/XMLSchema:date’ – The string ” is not a valid XsdDateTime value.

Cause:

This exception is perfectly valid from the compiler’s perspective. Here comes the example.

Suppose my flatfile schema is generated using the following imput and is demilited by ‘|’ .

1|Name|2010-02-03|Description

As the date field(highnlighted in red) is optional in our scenario, the following input is also valid.

1|Name||Description

But when BizTalk validates this instance against the schema, it just picks the value between the two ‘|’ symbols for the date field which is ”. Thus it throws the error when it checks if the value ” is valid for a date field. This will occur for any datatype which does not accept ” value.

Solution:

The Solution is to logically let BizTalk know priorly that the datefield is optional and it has to ignore if any blank values occur.

This can be done by setting the “Suppress Empy Nodes” Property to “No” at the Schema Level(By Clicking on the word ”Schema” above the root node of the flatfile).

This will ensure that BizTalk knows that the field might have empty value and it should ignore it. This will get your validation successful.

------------------------------------------------------------------------------

public string ParseDate(string parameter1)

    {
          if (!(System.String.IsNullOrEmpty(parameter1)))
    {
          System.DateTime dt;
                if(DateTime.TryParse(parameter1, out dt)){
                return dt.ToString("s");
                }else{ return "";}
     }
else
{
return parameter1;
}
}

----------------------------------------------------------------------------------------------

xsl:if test="$paramDate != ''">

Debatching a flat file in BizTalk

below is extract of information which i found in one of blog, for debatching flat files in biztalk. Sometimes you receive a batch file containing multiple records and you have to split each record into a separate message and process them individually.

This kind of splitting of a batch file is also known as debatching.

Depending on your requirement, you can either debatch an incoming message inside the pipeline or inside the orchestration.

The technique used to debatch a message inside the pipeline depends on whether the input is an XML file or a flat file.

For an XML file, you will need to use an XML disassembler, an envelop schema and a document schema, which I will discuss some other time on separate blog post / article.

Here I will show you how you can debatch a flat file.

Let us assume that we need to debatch a flat file that contains the following structure.

Field1,Field2,Field3,Field4,Field5,Field6
Field1,Field2,Field3,Field4,Field5,Field6
Field1,Field2,Field3,Field4,Field5,Field6
Field1,Field2,Field3,Field4,Field5,Field6
Field1,Field2,Field3,Field4,Field5,Field6
Field1,Field2,Field3,Field4,Field5,Field6

First you need to create a flat file schema that looks similar to the one shown in the figure below.

Make sure you have specified the correct child delimiters for the Root (0x0D 0x0A) node as well as the Record (,) node.

Root Node:
Child Delimiter Type = Hexadecimal
Child Delimiter = 0x0d 0x0a
Child Order = Infix or Postfix

Record Node:
Child Delimiter Type = Character
Child Delimiter = ,
Child Order = Infix

Now select the Record node and set its MaxOccurs property to 1. This is actually the key to debatching. The flat file disassembler will automatically separate each record into individual message.

Important: If you have set the Child Order Property of the Root node to Infix, make sure you have also set the property 'Allow Message Breakup At Infix Root' to Yes

Next, as you would normally do, add a receive pipeline to the project and add a flat file disassembler and set the Document schema property to the schema we created.

That’s all there is to it.

Now suppose you have a flat file that also has a header record or a trailer (footer) record or both as shown below.

FirstName,LastName,RegNo,School,City
SubjectName1,Score1
SubjectName2,Score2
SubjectName3,Score3
SubjectName4,Score4
SubjectName5,Score5
SubjectName6,Score6
SubjectName7,Score7
SubjectName8,Score8
TotalScore

You would normally create a flat file schema as shown below to parse as one single message

But for the purpose of debatching the body records, you need to create 3 separate schemas as shown below. Make sure the MaxOccurs of the Body record is set to 1.

And in the flat file disassembler, you need to set the Header schema, Document schema and Trailer schema properties appropriately.

Note: You might need to set a unique Tag Identifier property for each of the schemas, and also prefix the same tag identifier on the records as shown below, so that the flat file parser can distinguish the header, body and footer records.

HFirstName,LastName,RegNo,School,City
BSubjectName1,Score1
BSubjectName2,Score2
BSubjectName3,Score3
BSubjectName4,Score4
BSubjectName5,Score5
BSubjectName6,Score6
BSubjectName7,Score7
BSubjectName8,Score8
FTotalScore

It is possible to preserve the header record into the message context, by setting the Preserve header property of the flat file disassembler to true, so that you can use it as a header record for each of the debatched message.

To use the preserved header, you need to set the Header schema property of the flat file assembler to the same header schema you used in the disassembler. Here’s how the output will look.

HFirstName,LastName,RegNo,School,City
BSubjectName1,Score1

HFirstName,LastName,RegNo,School,City
BSubjectName2,Score2

HFirstName,LastName,RegNo,School,City
BSubjectName3,Score3

HFirstName,LastName,RegNo,School,City
BSubjectName4,Score4

HFirstName,LastName,RegNo,School,City
BSubjectName5,Score5

HFirstName,LastName,RegNo,School,City
BSubjectName6,Score6

HFirstName,LastName,RegNo,School,City
BSubjectName7,Score7

HFirstName,LastName,RegNo,School,City
BSubjectName8,Score8