I have been involved through my work, and attended the recent open partnership meeting, and it struck me that the recent move into an “operational phase” seems to have kickstarted a great leap forward. There were many more organisations present at the partnership meeting than I think has previously been there. Crucially, there is a mix of public bodies (including NDPB’s) and private companies now, and this is important as MEDIN is aiming to embed itself early into contractual agreements for the collection of Marine Data.

By embedding itself, I mean that MEDIN supplied a number of functions and standards that are useful for partners to include in data collection contracts. By including these, the contractor takes on an obligation to include metadata in relevant format, and to stick to standards for certain types of data.

The process for including and utilising MEDIN is that there are several Data Archive Centres (DAC’s) which store data long term and make it available, initially through MEDIN’s discovery portal, but also with view and download services planned for many areas. While there may be costs associated with committing large data sets to an archive centre, there are actually a number of great benefits, which I think are not fully realised by partners or data suppliers:

• Data are stored securely and handled by experts.
• Data are made public (if so agreed)
• The partner does not incur the costs of storing, backing up and maintaining Master Data Management aspects of these data sets
• Partners bandwidth costs are not escalated by starting to publish data.
• Data becomes part of a UK wide network of reporting data

As part of a data resuce project, I’m currently reading the documentation for a plankton database system from 1994. The documentation pretty much covers everything you would expect to find in a more modern documentation, e.g. a data model, work flows, methods and background code.  It was designed to run on an old VAX system, and of course there are some technical limitations that I am glad that we are over.

There are very clear limits on what could be put into tables, and the number of tables that were created. This is obviously a result of having to be careful about referencing. E.g. you did not reference long species names when an identifier was available as it would  slow your system to a crawl

However, looking at how very neatly data are structured in this old system, and looking to some of the more “modern” systems, it is actually far easier to re-extract and structure this data. All of the referential structure is intact even though the software itself is long gone, and I am left with nothing but the raw data. Looking at the raw data dump of many modern databases leaves something to be desired (IMO). The strict control of field formatting often goes out the window, and the “logic” of a system is frequently moved almost entirely to the front end, meaning that the DB itself sometimes doesn’t even contain all the referential information required.

Obviously you get what you pay for, and this is far from the case in many modern systems, but with the increased speed and power of computers, it has become ever so easier to throw clock cycles at a workflow problem rather than going back to the root causes.

Given the increase in computing power, I’m not sure if we have seen an equivalent increase in performance of databases. Granted we can store much, much more in there, and in a much wider range of formats, but does it encourage the cutting of corners rather than improvement of data models?

I’m happy to be proved wrong, and I suspect there isn’t a right or wrong, but there is certainly an observation that when you are forced to be economical with your clock cycles, there seems to be a higher attention to your data model.

Creating metadata is a great way to get an oerview internally, but there are increasing demands on public authorities to also share the metadata externally. Apart from the legal requirements for serving spatial data and metdata, there is a plethora of ways in which this can be done, and a lot of it comes down to how deep your pockets are, and how many staff hours/days/months/years you are willing to spend on it.

One of the characteristic of most metadata is that it adheres to a known standard. This standard defines the information fields used to describe data holdings, and also details if a fields is mandatory to fill in, optional or sometimes conditional, depending on the settings on other fields.

Common to a lost of these standards are some basic fields, like title, description/abstract, originator, publication and modificaiton dates for example. But some standards goes much further, and start providing much more detail on the source, formats and extent of the data sources.

There is generally considerable overlap between a lot of metadata stanards, and this is one of the sources of frustraiton for people beginning the process of working up metadata in my experience. With a bewildering array of standards, with sometimes only subtle differences, it can be difficult to get an overview of which one to start with. However, my strong recommendation is to spend some time on this aspect. Yes it involves reading long, tedious documentation and comparing across standards. But by getting this knowledge early on, you can avoid setting yourself up for disappointment. Some standards will be supersets of others, and as such require more information. Therefore, it is important to be clear on what your obligations and aims are in terms of capturing and serving information. There’s no need to capture more that needed- the only thing to come out of that is that either you or your staff spend more time on creating the records, or in worst case, abandon creating metadata, because it takes too long.

Next thing is to ensure that you have captured all the required fields for the stanadrds that you are going to deliver in. For example, in my organisaiton, we have a database which was natively in an eGMS metadata format. However, we actually need to serve data externally in a UK GEMINI and MEDIN metadata formats.  But rather than completely changing the existing 750 records, we could add some fields, and slightly modify existing ones. This allowed us to capture all the necessary information to comply with new formats even though we strictly speak still worked with an eGMS database.

In this post i’m not going into the creation and user engagement of metadata, but rather the “behind the scenes” element of getting things out. With a datbase, we got the ability to write out the data in XML format.

And this is where its currently at. We can export XML files form a SQL database. These are then validated in two ways. First they are validated for the standards that the XML format (in this case the MEDIN format) lists in its header, and the file’s conformity to those standards as well and generic XML formatting rules.

Once that hurdle is crossed we’re moving into actual publishing of the metadata, and as mentioned, there are many ways to do this, depending on the depth of pockets etc.

Commonly to comply with harvesters from other portals, we must be able to deliver the metadata in either a OAI-PMH format and act as a catalogue server. To comply with these abilities, we’re deploying geonetwork which will accept the individual XML records created locally, and serve them up up the web, both through a browsable and searchable interface, and runnign a CWS at the same time.

So in future posts, i will share about more about experiences about actually releasing and harvesting metadata, but to summarise the recommendations so far:

1. Get to know your standards. The documentation is dry, but necessary. You can save yourself a lot of work by investigating these early on

2. Don’t be afraid to mix’n match internally. If a combination of one or more standards suit your organisaiton better, then it’s not really a problem. As long as you can populate the standards that you want to share externally, you can have more or less fields (or even translate you internal formats to the standard) on the inside.

3. Consider the scale of distribution. How central is this to your business and who is going to use it? If metadata is at the core of your MDM, you prbably want full integration so the metadata is always close to the data, and perhaps even linekd directly to it in yur applications (e.g. GIS). But for others it may be a low number of records, and they can be hand edited and published.

4. More than one standard – or more than one harvester coming to visit? If you know who are going to be your main customers in terms of ingesting your accessible metadata you can set up to target collections for these. It also means that its quite easy to serve the same metadata to multiple groups, simply by “tagging” them – at least in Geonetwork.

5. Ask the devleopers. Wen you read metadata and methodology documentation, it can be easy to think of it all as terribly complex and “i probably don’t understand it”. But if something does not make sense- ask the writer or developers! In a couple of cases my own questions have turned out to be errors in the documentation and they have been updated and corrected as a consequence – thus makes it easier for the next person reading it.

When it comes to moving files around, it is a task that at face value seems incredibly easy, but when you scale it to many servers in terrabyte volumes, and add permissions for hundreds of groups to the mix, then it start becoming a bit of a job.

First part was the straight forward bit of getting things copied, and then synched – Robocopy is the workhorse here – but ultimately, you will probably want to make sure everything got included in those scripts, and this is where WinMerge comes in. The product has several modes of comparison for very fast size comparisons between two sets of folders to full binary comparison of all content. This is a major strength when working large scale as oyu can run fast scans, and home in on any issues this way.

It also allow you to fluently merge across from one set of fodlers to the other if you determine problems, but I mainly use it as a diagnostic tool – and so far, I’ve not found anything near it in user friendliness or performance.

The fact that it is an open source tool (GPL) tool only makes me happier – it is great to see collaborative projects that really nail some of the vaccums in the market!

It is instead the title of a short but interesting blog post by Jim Harris over at the OCDQ blog.

In this post, Jim covers how the word “Qualia” is used to describe subjective quality of conscious experience, and how very, very, relevant it is in terms of data quality management.

Anything above and beyond the most basic data quality checking is really in the eye of the beholder. One of the common memes of DQ is that it is “fit for purpose”. Although fitness of data is important, it is the more qulia than quality related element of the purpose that raise issues, especially when it comes to sharing data.  Trying to predict the purpose of someone accessing data and ensuring that the data are fit for this purpose is virtually impossible. Yet it is an issue that becomes important when organisations and public authorities may otherwise have to include lengthy licenses to remove doubt about liability.

So perhaps the term Data Qualia is one we will see more often in the future

Managers and Meetings are the prime source of interruptions and articifial divisions that means you never make it through a full “work-cycle” he explains in the talk. There is a lot of truth in this, in that meetings very often tend to generate two things:  more meetings and changed objectives.

So why am i even including this on a data management blog? Well for starters, most data managers work in some kind of office with some colleagues, and quite likelyt can be the source of thew two major evils to other workers: management and meetings. As someone with the word “management” directly in the title, I guess we are perhaps jsut native interruptions to other workers in an organisation, and the need to call meetings to get combined strategies and systems in place, pretty much makes us walking interruption bombs if we’re not careful.

So what can we do in terms of getting good management of data, but without being a constant source of interruption to others? Wll i think that the job of a data managemer is working as a conduit, translating needs, and bridging the technology and business goals of the organisations we work for. Jsut like everyone else, we’ll need periods of uninterrupted work to be creative, and to come up with good solutions – but we also need to respect others need for the same. SO it’s a two-way street – reduce your own interruptions, and be careful about interrupting others. Perhpas use email or IM for something that does not require a room full of people for two hours to reach consensus on.  Perhaps consider that the door shouldn’t always be open. It’s good to be approachable – but you need time to concentrate as well.

I think preparation is key as well. When it finally comes to having some of the meetings that simply are necessary – make sure you are prepared. Identify what you need to do at the meeting before you sit in the conference room! Don’t let anyone run away with the agenda or hijack the meeting – make a plan and stick to it. There are meetings that are productive – but those are the ones where people walk out afterwards and have a clear idea of what was achieved, and who is doing what. Those kinds of meetings takes time to prepare for – uninterrupted time

There are of course many, many places on the web where you can start discussing, but I often find that locating an active community can be a bit of a challenge. LinkedIn is a thriving community with many active groups, mainly focussed on the professional aspects of our lives. The benefit of some of these groups is that they include real life professionals with clear credentials, and have groups where people actively debate issues that are close to the hearts of data managers, ranging from how best to implement quality management through to provacy and ownership issues.

In this post, I have gathered a short list of som e of the groups i enjoy reading and participating in. Note that most of them you will have to request to join – but it helps reduce the amount of spam and keeps the discussion focussed amongst people with relevant interests. I hope you might find it useful, and if you have other sites/communities or groups, please do share

Data Ownership in the Cloud

The Data Ownership in the Cloud group on LinkedIn is a global venue for multi-disciplinary networking between technologists and non-technologists interested in providing thought leadership on this critical issue. 

IAIDQ Information/Data Quality Professional Open Community

LinkedIn.iaidq.org is an open community for Information Quality, Data Quality and Data Governance professionals (practitioners, consultants, academics, vendors etc.) to support collaboration, learning networking and interaction in a vendor neutral format. 

Obsessive-Compulsive Data Quality (OCDQ)

This is a networking group for Obsessive-Compulsive Data Quality (OCDQ), which is an independent blog offering a vendor-neutral perspective on data quality. 

Vendor-neutral does not mean no vendor related content. When the products and services of vendors are presented or discussed, it will be done in an objective manner. 

The goal is to foster an environment in which a diversity of viewpoints is freely shared without bias. Everyone is invited to get involved in the discussion and have an opportunity to hear what others have to offer.

Talend Open Source Data Integration

Talend is the first provider of open source data integration solutions, used primarily for ETL for business intelligence/data warehousing, data synchronization, data migration, operational data integration, data quality and MDM. 

This group enables Talend users (and friends) to share information, news, discussions, and ideas about their Talend projects – and anything else of interest for the Talend community! 

Data Management Utopia

The purpose of this group is to bring together a professional group of individuals to collaborate about issues, problems, situations within the data space. 

We all know that the world of enterprise data is sometimes hard to control and understand but together we can make a difference and learn from one another’s experiences. Whether you’re dealing with Oracle, SAP, IBM, etc. it doesn’t matter we still share some of the same pain points. 

This group encourages everyone to share their lessons learned, thought leadership ideas, best practices, etc. that will educate and promote the development of world class solutions.

Habitats – (Social Validation of INSPIRE Annex III Data…

Habitats (Social Validation of INSPIRE Annex III Data Structures in EU Habitats) related spatial data is critical in the management of Europe’s bio-diversity. INSPIRE needs work here, particularly in its Annex III data themes: 16-18 (Sea regions, Bio-geographical regions, Habitats & biotopes, Species distribution).

Google trends provides an option for combining multiple search terms and look at their trends, either on a global, or national basis (or even sub regions  if it is very popular search terms).  This tool can be quite useful for understanding a little about the field you work in. Enter some common search terms that you think would be logical  variations on a topic, and see how they stack up against each other.

No doubt people can argue long and hard about the selection of search terms that you want to combine to examine trends. In this case, I have taken the “data management term” to see how much it is used in comparison with two related subject “GIS” and “databases”. While in many ways they cover slightly different aspects, there is also a good degree of common ground. People who look for databases terms are involved in a form of data management or development, and GIS users are perhaps some of the most prominent users of data that require careful management of the source data.

I limited the search to UK, as I was mainly interested in looking at the development over time. I was half expecting a steady increase in the GIS term, given many more users, and increasing legal requirements surrounding spatial data and representation. However, when the three terms are combined, GIS is quite stable, while the data management term has developed in the early years, and then stayed relatively stable over the past 2-3 years. It is interesting to see that data management was not on the minds of many people prior to 2005, so it is indeed still a young field.

Google Trends for 3 search terms in the UK: "data management", "GIS", and "databases£

The great thing is that data is available in several recognised formats, NetCDF, ODV and ASCII files – so virtually everyone in the field can access this data in a preferred format.

There are some limitations in terms of the way you can refine searches, but most of them makes sense from the perspective of optimising searches and not hanging up the server in searches that return virtually everything.

By the time you have narrowed your search criteria to return 1,000 series or less, you can retrieve results. There’s the option of downloading a KML file of coverage, and you can retrieve data in your preferred format.

It is important to note that we’re talking data series, not individual points here, so even a single series can contain thousands of data points, giving you access to a seriously large amount of oceanographic data with a wide geographic coverage.

The initial map on the start page show waters around Britain, but make sure you either zoom out or pan around as there is data from a much wider region – virtually all of thw world - than what is shown on the map.

You do have to register an account with BODC in order to checkout your “data shopping”, but there is a huge amount of data freely available. The map tells you up front which data series are freely available.

BODC has truly made their data a lot more accessible with this exercise.