Data quality assurance requires real users

-
A hard lesson that I've learned over the past few months is that data quality assurance requires real data users.

I wrote a handful of data processing scripts two years ago to batch parse literature citations. The workflow was functioning well, and I even ran multiple test runs on different kinds of references to find bugs.
I've returned to these scripts and started processing new sets of data. I've found that the resulting datasets are riddled with problems. Some references are skipped entirely by the workflow, with no flagging system, and other features I excitedly added to the scripts are now broken.

This lesson has come to bear on my work a number of times, with software and workflows written by other, highly competent folks. You just can't predict what the problems with your workflow will be, until you have real users. What is the solution here? I hope I will find some clarity soon. 

Comments

Post a Comment

Popular posts from this blog

What I learned at Data Science Bootcamp

-
Image
Ottawa is where I attended the bootcamp. Photo by  Shanta Rohse  via  Wikipedia About a week ago I was fortunate enough to attend a Data Science Bootcamp hosted by a gracious and affable local Data Scientist . It turns out that many people within her network of family and friends were desperate to learn more about Data Science. What better way to educate all the interested people than to bring them all into one room? Here in Ottawa, Ontario, the federal government is the largest employer, with two major universities supplying many graduates for the public servant workforce. This resulted in a crowd composition of mostly government workers; incidentally many had a Biology background (one biologist friend brings another biologist friend, etc. etc). Our host went around the room to assess our goals for attending her workshop, and it appeared as though we were all keen learners who simply wanted to learn more! She gave three wonderful presentations, and invited a handful of us up
Read more

Existing in fast and slow moving spaces

-
Image
Ottawa has a thriving technology meet-up community. In fact, I am fortunate to live about a 5 minute bike ride away from the hosting venue of the Ottawa Machine Learning Meetup group. Tonight's lecture was "An Introduction to Natural Language Processing (NLP)". It struck me through a combination of comments made during the talk, and during my discussion with a young Computer Systems Engineering student, that working in the technology space means adjusting your frame of reference for what is a new and old idea. Botany. a discipline as old as time Image via  www.twenty20.com "Classical" NLP techniques date back to a decade ago. An "old" and developed Data Science company means it has existed for 5 years. These frames of reference are in stark contrast to my regular existence in the Biology, Systematics and Taxonomy world, disciplines pioneered over 100 years ago. "New" techniques in systematics include DNA sequencing, first used over 40 y
Read more