How hard can it be to integrate two systems?

Earlier this week I was working with a particular mapping for a client. They required to send invoices to one of their customers and we were already processing the same invoices for this client from a different (Older) system. The idea was that the client was migrating from their old system to the brand new shiny system.

Not a problem one might think, as a newer system should have better abilities to trade electronically than a system well over 10 years old. We live in the modern, integrated world.

Well it turned out not to be quite that simple. First we had an email to our mapping project system to say that the new system could not produce a particular piece of data, rather it provided a piece of data that was incorrect. We found a work around for this using our facility to treat message data as look-up data and so we were able to agree how we could process the data and produce the correct information for the recipient. All a standard part of the service.

We had the software vendors file specification and we had sample data so, on we go. Then we stopped. A quick check by the mapping engine highlighted that the sample file did not match the specification. Close but no cigar.

So we requested, and were sent, two files contains not one but two specifications. We were also sent two sample files. So we now have a quandary. If we choose, we have a 50% chance of getting the wrong specification. No problem, quick chat with the client and they point to the latest specification. Couple of hours later and we have a mapping of the input data, a mapping of the output data and we can test to make sure that the documents match business rules, add up etc.

Guess what, they don't. Well if everything worked first time there would be no need for testing. We changed some of the mapping, checked the arithmetic of the data and it just did not make sense, did not have tax totals and so was basically totally invalid.

After further discussion with the customer we find the root cause of the problem. You will remember we received two specifications and two sample files. It turns out that one of the sample files, sent by the software house, had nothing to do with the final specification and between us we had contrived to choose the wrong sample. Such is Sods Law.

This is the work of data mappers, and this is not a whinge about wrong specification etc. The work of data mapping involves detailed data analysis. Yes we at least have tools that speed that process up significantly but never the less it is detailed work and we love it. After completing the mapping, in less than 8 hours including dodgy files and specification, we had a laugh with the customer, a promise of a pint owed and a happy customer.

But here's the thing. Without the tools that we have built the data analysis and mapping process would take days, in fact we have heard competitors quote large sums of money for such work, if they do not already know the file standard, and I can understand why. Some people believe there cannot be much to sending a file of invoices from one company to another, but the devil is in the detail and that is why EDI or EIPP is becoming more and more a Software as a Service arena. Without the specialist skills and tools it is the Total Cost of Ownership of EDI or EIPP, the costs of skilled data analysts, mapping tools, testing teams and support, that more and more Financial Directors are saying is too expensive to do in house. The sending and receiving of physical files is easy, but the set-up of the variety of communication processes is not. What a lot of companies are now saying is get somebody else to cope with the messy bit, we just want to send and receive one file format and use one communications method.

After all, if you distribute building materials do you want to spend your money on making the distribution more efficient and selling more OR staffing up the IT team to have data analysts, EDI experts, XML experts, communications experts, oh and have them also run you IT systems as well?

A Simple Guide To Electronic Data Interchange

Over the holiday period I decided to create a "Simple" Guide to EDI for our Web Site. The idea was that, without needing to register or download, people could visit the Web Site and get a "Simple" FREE introduction to EDI, an answer to the question "What is EDI?".

I wanted to highlight to people what EDI is, what standards they can expect to find and how messages are exchanged. Six Web pages later and I have a thought. The concept of exchanging business documents electronically IS Simple.

The detail of it is convoluted, but for a good reason. Most businesses have a unique way of working. Most IT systems have a unique way of working. Certainly each industry has unique features, for example the way Paper is specified in an order is nothing like how a tin of baked beans is specified which is nothing like how a length of steel lintel for a building is specified. If a human is handling orders and invoices then their knowledge is used to "translate" between the buyers instructions and the suppliers computer system. With EDI or EIPP then the software must assume some or all of this intelligence.

The concept of EDI is easy, but as Ken Foster (One of Our Data Mappers) who has sat on many EDI committees and standards bodies says "The devil is in the detail". If people say that EDI is easy, then they are ignoring the detail or spinning a line (probably in sales ;-)). We chose to do this work because we a sad people who enjoy the detail, but it is not the standard IT software role and it is not to everyone's liking. We believe, and our customers believe, that Outsourcing EDI (SaaS) is best for most businesses because it is such a specialist area. Have a look at any "simple" Guide to EDI and if it is telling the truth you will see how complex it can be.

"Both Parties Must Both Gain Benefits" is the First Golden Rule

You may recall that in "Beware the "cheap" option!" we looked at some of the wasteful abuses that can be imposed on you if a "website" substitute for business-to-business, integrated EDI is presented to you as the means to provide your customer with electronic documents, such as invoices, despatch and remittance advices. The "cheap" option only leads to your staff having to double type all the information from the documents into your systems and into the “website”!

• Your work is doubled.
• Your risks from human error are doubled.
• Your costs go up.
• Your profits fall.
• Your prices need to go up...
• You become less competitive!

What can you do to avoid such abuses?

Well, the remedy is a bit like growing asparagus...

Asparagus?

Yes. To grow asparagus you dig a hole three meters long by three meters wide about one meter deep. You fill it with all sorts of good stuff from stables mixed with exquisite soil and you plant the asparagus – FIVE YEARS AGO!

You must avoid being pushed around by the eBusiness team. They are working to a very restricted agenda. They know nothing and care less about your business value to their employer. You need to maintain and cultivate your highest levels of contact within your customer. You need your relationship to be strong enough for your senior contact to be prepared to instruct the eBusiness team to co-operate with you. To make an exception in your case. You will still be a very willing eBusiness partner, but the job is going to be done properly and, to a certain extent, on your terms!

It is best if you avoid direct contact with the eBusiness team. Let your eBusiness provider do that whilst you maintain your good standing with your senior contact(s)!

What then do you offer to do for your ally in the senior echelons of your customer?

When the “website” was implemented it was configured to receive orders from your customer's purchasing systems. These orders are passed to the “website” in electronic form. They are files and they have a format. All that has to be done is for you to be supplied with this format and for your customer to agree to sending the order files to your eBusiness system by one of the accepted transport methods.

Similarly, the “website” passes the documents (such as invoices, despatch advices and remittance advices) that are inbound for the customer's systems as files and they too have formats. If the formats are made known to you and your eBusiness provider then they too can be passed directly to the customer's systems just as efficiently as the files that are currently passed to them from the “website”.

The benefits here are that you avoid doing the work twice.

All the risks listed above are eliminated.

You have reinstated the first GOLDEN RULE:

BOTH PARTIES MUST BOTH GAIN BENEFITS!

In that same WIN-WIN vein, you are perfectly entitled to follow the example of Oliver Twist and ask for more. You are being asked to make an investment in money, time and effort to help your customer. Are you getting as much business from your customer as you could? On a number of occasions I have seen the Sales Director of a supplier use the “request” for electronic trading as a very sound reason to visit the customer and literally “ask for more”. In one instance a customer of mine, a supplier of specialist tools and devices to the construction industry, went to her customer and said something along the lines of, “We would be happy to do as you ask, but currently you only give us about ten per cent of your business. You give ninety per cent to our competitor! Give us fifty per cent and we will do all you ask and do it immediately!”

She won the extra business, increasing her company's sales to that customer by 400 per cent!

In summary so far then:

You must be informed of exactly what is required, IN FULL. It can be too late to get a decent working relationship if you fail to cultivate your senior contacts in your customer and just become part of a target list drawn up by the customer's eBusiness team or their vendor. You and your customer must get worthwhile benefits out of the eBusiness relationship. Get these things right with your customer and you have both made a good start towards successful and beneficial electronic trading.

You may even find an early opportunity to multiply your sales to this customer!

In a later blog we will consider the second GOLDEN RULE for successful integrated EDI.

The use of catalogue data in EDI

One of the big issues that faces companies exchanging data electronically in any industry is the alignment of the data on all trading partners systems.

As a simple example we can look at a purchase order sent from a retailer to a supplier. The purchase order contains data that has been produced from the retailers back office system that is required by the suppliers back office system in order that the order is fulfilled efficiently. This data will include information such as the delivery point and the code for each product ordered.

When the order is processed by a human, the human interprets the data they see and translates this in to the data required by the suppliers ERP system. When the data is exchanged electronically this "translation" needs to be done automatically. For a lot of suppliers ERP systems this has already been accommodated, but we still get requests from customers to help with this process.

One of the advantages of a Software as a Service (SaaS) is in the nature of having a centralised hub. We have taken the approach of hosting catalogue data on the hub which means that ALL customers that need access to this data, to enhance the data they send or receive, can use it. If our software was deployed at our customers sites then each customer would need a copy of each catalogue they use, and they would have to be regularly updated. With the SaaS approach one catalogue is shared by all relevant users, and is regularly updated in one place.

I should say that to us a catalogue is a name given to ANY look-up data which we process. This could be a list of valid delivery points, a list of product groupings and, of course, products and prices. The important thing to do is to enhance the data received by the recipient so that they can process the incoming data automatically without human intervention. After all that is what EDI is about.

Whilst we have found this approach to be some use in the retail sector, the real advantages have arisen in the construction sector, where EDI is still a maturing technology and the back office systems of both sides of a trading relationship are not a well configured for electronic message exchange as those in the retail sector. But as we are adding construction trading relationships at the rate of 20 to 30 LIVE trading relationships per month, this technique is proving its worth.