By Hiran de Silva
This part is an addendum to this article The History of Separating Data from the Spreadsheets.
At this point in the story, we need to introduce one more character: the Digital Librarian.
So far, when we have talked about the system of record, we have naturally imagined the large, authoritative systems maintained by IT: the ERP system, finance system, HR system, CRM, stock system or whatever other corporate database contains the organisation’s official records.
These systems are deliberately controlled.
And that is a good thing.
Ordinary spreadsheet users should not normally be allowed to write directly into the company’s ERP database. There are very good reasons for that. IT governance, security, validation, audit and data integrity all matter.
But that creates an important question.
Where does the business put the information that it owns and needs to manage before it becomes part of the corporate system of record?
Traditionally, the answer has often been: spreadsheets.
And that is where the problem begins.
The business needs its own system of record
Imagine a department with twenty spreadsheets being passed around by email or stored on a shared drive.
Different people are entering information. Others are consolidating it. Somebody is trying to determine which version is current. Somebody else has copied part of one spreadsheet into another.
The problem is not that these people are using Excel.
The problem is that the spreadsheets themselves have become the storage system.
We have mixed two completely different jobs.
The spreadsheet is an excellent working document. It is where people calculate, analyse, model, review, investigate, present and make decisions.
But the information being shared between those working documents needs somewhere else to live.
That somewhere is what I call the Digital Librarian.
It is our local system of record.
A departmental system of record
The Digital Librarian does not compete with the corporate system of record.
It serves a different purpose.
The corporate system of record belongs to the organisation and is governed accordingly.
The Digital Librarian belongs to the business process.
It might belong to Finance, Operations, Sales, a project team or even a relatively small group of people collaborating on a particular process.
Instead of storing the shared information across dozens or hundreds of spreadsheets, we identify the data that everybody is struggling to maintain and give it a proper home.
We decide what information we need.
We decide what the tables should contain.
We decide how those tables relate to one another.
And we put them into a relational database.
For many Microsoft Office users, the simplest starting point may be Microsoft Access on a shared drive.
It does not have to begin as a grand IT project. Indeed, it can begin as little more than a prototype — a structured whiteboard on which the department works out what its information actually looks like.
The important change is architectural.
The data is no longer owned by individual spreadsheets.
The spreadsheets become clients of the Digital Librarian.
They GET the information they need.
They do their work.
And when appropriate, they PUT information back.
Now everybody is working with the same centrally held information.
The Digital Librarian knows what has been stored. It can know who supplied it and when. It can provide the same information to another spreadsheet immediately.
That is exactly what a good librarian does.
But what about the real corporate system?
Eventually, some of the information held by our Digital Librarian may need to become part of the organisation’s official corporate record.
That does not mean our spreadsheets suddenly need permission to write directly into the ERP database.
There is usually already a controlled doorway.
In finance, one of the oldest and simplest examples is the journal import.
Accounting systems have supported mechanisms of this kind for decades. The business prepares a journal in an agreed format — very often something as simple as a CSV file — and the accounting system validates it before accepting it.
The account must exist.
The journal must balance.
Mandatory fields cannot be blank.
References may have to conform to prescribed formats.
Dates, currencies, cost centres and other values may have to satisfy predefined rules.
Only valid data gets through.
That is precisely how it should be.
Our Digital Librarian can therefore prepare information for the corporate system through the governed interface that the corporate system already provides.
In a modern environment that interface might be an API.
In another environment it might be an automated integration.
And sometimes it may simply be a carefully specified CSV journal import.
The principle is the same.
Two systems of record — for two different purposes
This distinction is crucial.
We are not proposing a shadow ERP system.
We are recognising that there are two different stages in the life of business information.
There is the information that people are still working with — collecting it, reviewing it, correcting it, approving it and transforming it.
And there is the information that has crossed the boundary into the organisation’s formally governed corporate record.
The Digital Librarian manages the first.
The corporate systems manage the second.
Between them sits a controlled interface.
And surrounding the Digital Librarian are the working documents — including Excel spreadsheets — through which human beings actually perform the work.
So our architecture now becomes very simple:
Working Documents ↔ Digital Librarian → Governed Interface → Corporate System of Record
Once we see that architecture, an apparent conflict between spreadsheets and databases disappears.
They were never supposed to be alternatives.
The spreadsheet is where people work with information.
The Digital Librarian is where their shared working information is organised and remembered.
And the corporate system of record is where the organisation’s official information is ultimately governed and preserved.
That is the missing layer in the history.
And, arguably, losing sight of that layer is one of the reasons organisations ended up trying to make hundreds of individual spreadsheets perform the job that a librarian should have been doing all along.
Add comment