Talk:Data Quality Issues

Views

Watchers

[show all]

Topics

1 Data Quality Issue List [21 March 2023]
2 Coordination of Issue Resolution [21 March 2023]
- 2.1 Pre-1500 Genealogy [21 March 2023]
- 2.2 Statistics (across WeRelate) [21 March 2023]
3 Usability of the Data Quality Issues page [21 March 2023]
4 Quick Wins [15 October 2023]
5 Either / or Dates - how to correct [3 August 2023]

[add comment] [edit]

Data Quality Issue List [21 March 2023]

Click here (not the Article link) for a list of data quality issues in WeRelate.

[add comment] [edit]

Coordination of Issue Resolution [21 March 2023]

If two or more volunteers are prepared to start working through issues, I encourage them to coordinate efforts so they are starting in different places. However I will likely continue to spot-check and fix isolated issues, especially issues in my trees, as I have the time and inclination.--DataAnalyst 17:05, 16 May 2022 (UTC)

Perhaps a column might be added to report showing the timestamp of last edit? If an item on report uses data from multiple pages; report the timestamp of page most recently changed. This feature could reduce interaction between volunteers. For users resolving issues on their own pages; this will indicate which pages are presumably fixed.--fbax.ca

That's an idea. I'll have to see what kind of performance impact there might be. I won't get around to this for at least a couple of months.--DataAnalyst 17:35, 14 June 2022 (UTC)

I added a "Fixed" note after any issue that has been fixed.--DataAnalyst 20:51, 14 December 2022 (UTC)

For now, I'm going to focus on errors for people born in the 1800's, starting at A.--DataAnalyst 20:01, 15 June 2022 (UTC)

[add comment] [edit]

Pre-1500 Genealogy [21 March 2023]

Estimated dates in trees that extend prior to 1500 can be inconsistent from one page to another, leading to many issues being reported. This is an area that is probably best tackled as a significant project (or several projects, for different countries/lineages). I've written code to extrapolate reasonable birth year ranges where dates are sparse, which would be helpful for this project, but am not ready to tackle cleaning up the data yet - maybe by about 2024.--DataAnalyst 22:59, 16 June 2022 (UTC)

[add comment] [edit]

Statistics (across WeRelate) [21 March 2023]

To see the progress we are making on addressing the backlog of issues, see the Data Quality Statistics page.

Note that between 19 Jul 2022 and 19 Aug 2022, several new issue types were added to the list, increasing the total count of issues. In particular, invalid dates (many of which are text that should be in the description field) account for over 36,000 new issues on the list. Accounting for these changes, it appears that over 1,700 issues were resolved or verified between 15 May 2022 (the inception of the list) and 19 Aug 2022, about 1,100 of which were in the last month (19 Jul to 19 Aug). Thanks to everyone who is tackling the backlog of issues!--DataAnalyst 17:45, 20 August 2022 (UTC)

[add comment] [edit]

Usability of the Data Quality Issues page [21 March 2023]

This is a place to discuss the usability of this feature, keeping in mind that performance considerations and coding effort may preclude some desired enhancements.--DataAnalyst 02:30, 16 May 2022 (UTC)

It seems the "Person" pages are listed by surname then given name with surnames starting with an upper case letter before those starting with a lower case letter. This is fine by me except that surnames starting with a lower case letter are not being sorted properly; several surnames starting with 'van' appear to be in almost random sequence.--fbax.ca 23:46, 13 June 2022 (UTC)

Thanks - I hadn't noticed. I'll look at fixing this sometime in the future.--DataAnalyst 17:35, 14 June 2022 (UTC)

It seems that any surname with a space in it is affected by this issue. Surnames beginning with 'De' (first letter capitalized) are also presented somewhat randomly.--fbax.ca 20:56, 15 June 2022 (UTC)

This was fixed August 2022.--DataAnalyst 20:49, 14 December 2022 (UTC)

Data Quality Issues last updated 17:10 -- What timezone? --fbax.ca 21:10, 15 June 2022 (UTC)

Your local time, as far as I can tell. It is 12:10 Central Daylight Time, if that helps.--DataAnalyst 22:04, 15 June 2022 (UTC)

Nope. Report now has 17:10, 17 June 2022 but current local time is 16:07, 17 June 2022 --fbax.ca

Sounds like it is displaying in UTC/GMT. That's what I thought it was supposed to do, but it doesn't for me, so I guess the code behind the scenes is converting to local time - at least for me. If you go to Settings > Date and Time, is your local time set correctly to your local time, or is it the same as server time? My offset is set to -6 hours and my local time is 6 hours earlier than the server time.--DataAnalyst 22:31, 17 June 2022 (UTC)

Copied from the Watercooler:

Would you consider adding some type of "Explanation or Summary" field to the Verified template that gets added to the Talk page? I feel like an explanation would be helpful and good record keeping as to what was done or why nothing was done, yet the issue was still "verified" - si--Amelia 01:30, 27 November 2022 (UTC)milar to the Defer template? I find myself wanting to go back to the Talk page to add brief explanations. Thanks, --cos1776 15:25, 17 June 2022 (UTC)

Sure - I'll look into it. I wasn't sure if anyone would want to add comments, but if you do, I'll add the popup to capture them. Maybe next week.--DataAnalyst 17:30, 15 July 2022 (UTC)

This is done.--DataAnalyst 16:26, 19 July 2022 (UTC)

Copied from the Watercooler:

I've done a cursory review of the Data Quality Issues (DQI) for pages that I'm watching. It seems to me that if I correct an entry; e.g., an error in marriage date that, when corrected, resolves the issue, the cleanest way to proceed is to mark that DQI verified, the history for that page reflecting the change. Any necessary additional explanation can be documented either on the Talk page or on the Summary associated with that change. If I don't hear any objections to that method, I will proceed with cleaning up what I can on my watched pages based on the above. --jaques1724 01:22, 11 July 2022 (UTC)

Hi. I'm glad to hear you're planning to use the list to find and fix errors. However, please DON"T use the Verified by me button if you fixed an error. The button will put a template on the Talk page that will be nonsensical and confusing the next time someone looks at a page - e.g., it will say that you verified that the person was born before their parents were married, when in fact they were born after their parents were married.

I realize that it can be annoying to wait a whole day (or more) for a fixed item to no longer show up on the list, but I don't have a good way to remove the item from the list before the next time the batch job is run. The Verified by me button is NOT intended for nor should be used for this purpose. You can keep track of what you looked at (and might have fixed) by the color of the link, and can use the Defer button to keep track of what you looked at but chose not to fix (yet) - you can always remove the template if you fix the issue later. If you fix one date that addresses multiple errors (e.g., a bunch of children born before an incorrect marriage date), you might be able to keep track by clicking on all the affected pages right away - or you can go on to a different family and follow up again after the DQI page has been updated.

The next time I take a look at the code, I'll see how feasible it is to indicate that an issue has already been fixed - I might be able to do this for at least some of the issues without impacting performance too much. Thanks for the feedback.--DataAnalyst 17:30, 15 July 2022 (UTC)

The report now says "Fixed" for any issue fixed since the last run. (You have to refresh the page to see this if you just fixed an issue.) Note that there are a few cases incorrectly marked "Fixed", due to a couple of defects that will be corrected soon.--DataAnalyst 20:49, 14 December 2022 (UTC)

It looks like I have a bunch of stuff hitting because it's an error to use more than one qualifier - i.e., "by about 1600" isn't allowed. Is that true, and if so, is there anything we can do about it? It's both frustrating and not a good use of our time to flag as an "error" to fix things to be less precise. Most of these come from Great Migration, and thus reflect academic estimates, fwiw. --Amelia 01:30, 27 November 2022 (UTC)

As far as I can tell, the GEDCOM standard doesn't support combining modifiers such as "BEF ABT". Since WeRelate both accepts and produces GEDCOM files, it is important to stay consistent with the GEDCOM standard.

I've struggled myself with what to do about estimates expressed as "Bef abt" and have a few ideas:

Update the Date Conventions to say that since "Bef abt" is not supported, it is wise for the reader to assume that the date might not be precise - that is, they shouldn't assume that "Bef 1600" precludes, say, 1601. This is not particularly satisfying, since I assume few people read the Help.
Add a parenthetical phrase after the date - e.g., "Bef 1600 (about)". I've done this a few times. The GEDCOM standard allows such a phrase, but generally intends it to indicate the original source - e.g., "Bef 1600 (by about 1600)", which you could also do.
As our Date Conventions point out, the source is always supposed to be cited in a source reference. So you could say "Bef 1600" in the date and "by about 1600" in the source citation.
Or you could combine the last 2 approaches if you assume that people don't always bother reading the source citations. That might be overkill, since more experienced readers won't take "Bef 1600" as literal truth without checking the sources, but it might feel like it communicates the sense of the date more appropriately.

We could, of course, do all 3

Update the Date Conventions to caution users that "Bef, Aft, Bet/and, and From/to" might express inexact dates.
Add "(about)" or the exact source phrase, e.g., "(by about 1600)" to the date.
Cite the exact source phrase in the source citation.

BTW: GEDCOM doesn't support "by" as a modifier. WeRelate has defined "Bef" as "on or before" so that it means essentially the same thing as "by". I can't think of a similar word that means "on or after" - WeRelate defines "Aft as "on or after". This is necessary since it is quite common to say that a person was born before or died after the date they were mentioned in a document, when, of course, they could have been born earlier in the day or died later in the day.

[add comment] [edit]

Quick Wins [15 October 2023]

A lot of errors are date errors, and some can be fixed without research. The pages below simply need the "0" or "00" removed from one or more dates, if anyone is game to tackle them.--DataAnalyst 18:00, 29 March 2023 (UTC)

list removed (done)

These are done (hope I didn't miss one) --Paula 10:12, 27 July 2023 (UTC)

Thanks so much, Paula. I'll post another list soon, but I see you've been fixing other issues as well. I appreciate the help!--DataAnalyst 16:14, 28 July 2023 (UTC)

Here is another set of easy fixes. Remove the word "dead", "deceased", "died", or "yes" or the letter "y" from the death date field.--DataAnalyst 17:13, 28 July 2023 (UTC)

Going by my browser links showing as visited, the following can be removed from the list as they are done

Person:Samuel_Fish_(10) through to Person:Isabel_Cassedy_(1)

and

Person:Claude_Bateman_(1) through to Person:Marjory_Van_Benschoten_(1)

--Paula 11:24, 14 October 2023 (UTC)

I removed those pages. Thanks so much for tackling this!--DataAnalyst 12:24, 15 October 2023 (UTC)

[add comment] [edit]

Either / or Dates - how to correct [3 August 2023]

The majority of my identified issues are due to the use of either / or for dates, where sources point to both the dates.

What is the preferred resolution of these issues. For now, I am not updating them. However, if I want to make other changes to the page, I have to correct this issue before the data can be saved.--RGMoffat 20:47, 3 August 2023 (UTC)

If the date is a birth, christening, death, burial or marriage date, the preferred method is to add an Alt Birth, Alt Christening, Alt Death, Alt Burial, or Alt Marriage fact for one of the dates. Alternatively, you can pick one date and put "or xxx" in parentheses after it - e.g., 5 Jun 1884 (or 6 Jun 1884). Anything can be put in parentheses after the date, and it is ignored. This approach works for any type of event.--DataAnalyst 21:02, 3 August 2023 (UTC)

Retrieved from "https://www.werelate.org/wiki/Talk:Data_Quality_Issues"

Don't want ads?

Menu

Personal tools

Talk:Data Quality Issues

Views

Watchers

Topics

Data Quality Issue List [21 March 2023]

Coordination of Issue Resolution [21 March 2023]

Pre-1500 Genealogy [21 March 2023]

Statistics (across WeRelate) [21 March 2023]

Usability of the Data Quality Issues page [21 March 2023]

Quick Wins [15 October 2023]

Either / or Dates - how to correct [3 August 2023]