How hard can it be to integrate two systems?

Earlier this week I was working with a particular mapping for a client. They required to send invoices to one of their customers and we were already processing the same invoices for this client from a different (Older) system. The idea was that the client was migrating from their old system to the brand new shiny system.

Not a problem one might think, as a newer system should have better abilities to trade electronically than a system well over 10 years old. We live in the modern, integrated world.

Well it turned out not to be quite that simple. First we had an email to our mapping project system to say that the new system could not produce a particular piece of data, rather it provided a piece of data that was incorrect. We found a work around for this using our facility to treat message data as look-up data and so we were able to agree how we could process the data and produce the correct information for the recipient. All a standard part of the service.

We had the software vendors file specification and we had sample data so, on we go. Then we stopped. A quick check by the mapping engine highlighted that the sample file did not match the specification. Close but no cigar.

So we requested, and were sent, two files contains not one but two specifications. We were also sent two sample files. So we now have a quandary. If we choose, we have a 50% chance of getting the wrong specification. No problem, quick chat with the client and they point to the latest specification. Couple of hours later and we have a mapping of the input data, a mapping of the output data and we can test to make sure that the documents match business rules, add up etc.

Guess what, they don't. Well if everything worked first time there would be no need for testing. We changed some of the mapping, checked the arithmetic of the data and it just did not make sense, did not have tax totals and so was basically totally invalid.

After further discussion with the customer we find the root cause of the problem. You will remember we received two specifications and two sample files. It turns out that one of the sample files, sent by the software house, had nothing to do with the final specification and between us we had contrived to choose the wrong sample. Such is Sods Law.

This is the work of data mappers, and this is not a whinge about wrong specification etc. The work of data mapping involves detailed data analysis. Yes we at least have tools that speed that process up significantly but never the less it is detailed work and we love it. After completing the mapping, in less than 8 hours including dodgy files and specification, we had a laugh with the customer, a promise of a pint owed and a happy customer.

But here's the thing. Without the tools that we have built the data analysis and mapping process would take days, in fact we have heard competitors quote large sums of money for such work, if they do not already know the file standard, and I can understand why. Some people believe there cannot be much to sending a file of invoices from one company to another, but the devil is in the detail and that is why EDI or EIPP is becoming more and more a Software as a Service arena. Without the specialist skills and tools it is the Total Cost of Ownership of EDI or EIPP, the costs of skilled data analysts, mapping tools, testing teams and support, that more and more Financial Directors are saying is too expensive to do in house. The sending and receiving of physical files is easy, but the set-up of the variety of communication processes is not. What a lot of companies are now saying is get somebody else to cope with the messy bit, we just want to send and receive one file format and use one communications method.

After all, if you distribute building materials do you want to spend your money on making the distribution more efficient and selling more OR staffing up the IT team to have data analysts, EDI experts, XML experts, communications experts, oh and have them also run you IT systems as well?