Difference between revisions of "Data Catalog"

From wiki
Jump to navigation Jump to search
 
(26 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
== Data Catalog ==
+
== Data Catalogue ==
−
Data catalog is an organized service that allows users to centralize metadata and learn more about their data sources which help organizations achieve more values from their assets.
+
Data Catalogue is an organized service that allows users to centralize [[Metadata | Metadata]] and learn more about their data sources which help organizations achieve more values from their assets.
−
Following are some advantages to centralizing metadata:<br>
+
Following are some advantages to centralizing [[Metadata | Metadata]]:<br>
  
 
* Consistency and accuracy across the department/government
 
* Consistency and accuracy across the department/government
Line 8: Line 8:
 
* Allow users to self-serve
 
* Allow users to self-serve
  
−
== Why Data Catalog ==
+
== Why Data Catalogue ==
 
[[File:USGS DataLifeCycle.png|thumb|188x188px|
 
[[File:USGS DataLifeCycle.png|thumb|188x188px|
 
USGS data Life Cycle
 
USGS data Life Cycle
Line 15: Line 15:
 
Data-driven culture empowers users with getting access to their data source. With this, the growing numbers of cloud applications, privacy regulations and security rules are making it more difficult to effectively secure and govern data<br>
 
Data-driven culture empowers users with getting access to their data source. With this, the growing numbers of cloud applications, privacy regulations and security rules are making it more difficult to effectively secure and govern data<br>
  
−
Therefor, a Data Catalog is needed to:
+
Therefore, a Data Catalogue is needed to:
  
 
* Spend less time searching for data and more time using it to gain insight
 
* Spend less time searching for data and more time using it to gain insight
 
* Have a better/safer access to data through governance. Departments have a hard time finding data and controlling who has access to it.
 
* Have a better/safer access to data through governance. Departments have a hard time finding data and controlling who has access to it.
 
* Reduce the cost of data redundancy and hoarding.
 
* Reduce the cost of data redundancy and hoarding.
−
* Better linkage between the technical value of metadata and the business value.
+
* Better linkage between the technical value of [[Metadata | Metadata]] and the business value.
 +
 
 +
== Data Catalogue Architecture ==
 +
'''A work in progress Architecture'''
 +
 
 +
As explained in the following Diagram, the Catalogue will be the entry point for [[TADAP| TADAP]]
 +
 
 +
When a user requires to add a data set to [[TADAP| TADAP]], the Catalogue will be able to process this request and help the user create the metadata that goes with it, data is uploaded.
 +
 
 +
1- How the Catalogue will be connected to [[TADAP| TADAP]]
 +
 
 +
[[File:Catalogue_TADAP.png|545x545px]]
 +
 
 +
 +
2- How data is published to FGP and Open Data Portal
 +
{| class="wikitable"
 +
|-
 +
! Catalogue Conceptual Architecure !! How data is published to FGP and Open Data Portal
 +
|-
 +
|  [[File:ConceptualArchitecture2.png|545x545px]] || [[File:Catalog_FGP_OpenData_Publish.png|545x545px]]
 +
|}
  
−
== Data Catalog Architecture ==
 
−
<gallery>
 
−
File:CatalogArchitecture.png|A work in progress DFO-MPO Data Catalog Conceptual Architecture
 
−
</gallery>
 
 
== List of Stakeholders ==
 
== List of Stakeholders ==
 +
'''Please note this is an initial list.'''
  
−
{| class="wikitable sortable mw-collapsible"
+
'''Please feel free to contact us if you think there's room for improvements.'''
 +
{| class="wikitable mw-collapsible"
 
|+
 
|+
−
!
 
−
!
 
−
!
 
−
!
 
−
!
 
−
!
 
 
|-
 
|-
 
|Sector
 
|Sector
 
|Contact Person
 
|Contact Person
 
|Project / Program
 
|Project / Program
−
|Meeting date
 
−
|Relation to TADAP
 
−
|Other Comments
 
 
|-
 
|-
 
|Conservation and Protection
 
|Conservation and Protection
 
|Lise Melanson
 
|Lise Melanson
 
|Department Violation System (DVS)
 
|Department Violation System (DVS)
−
|Augut 24th, 2018
 
−
|These projects have to be  supported by TADAP
 
−
|We are contacting  André  Bélanger to get more info about data sources
 
 
|-
 
|-
 
|
 
|
 
|Pauline Lalonde
 
|Pauline Lalonde
 
|C-STAT
 
|C-STAT
−
|September 25th, 2018
 
−
|This project has to be supported by TADAP
 
−
|
 
 
|-
 
|-
 
|Canada Coast Guard
 
|Canada Coast Guard
 
|Shawn Legault and Nicholas O’Hara
 
|Shawn Legault and Nicholas O’Hara
 
|SIPA
 
|SIPA
−
|September 11th, 2018
 
−
|CCG has its own opertional network. However, we need to at  least catalouge their data
 
−
|
 
 
|-
 
|-
 
|
 
|
 
|Jean-François Coutu  
 
|Jean-François Coutu  
 
|INNAV
 
|INNAV
−
|September 24th, 2018
 
−
|CCG has its own opertional network. However, we need to at  least catalouge their data
 
−
|
 
 
|-
 
|-
 
|
 
|
 
|Chris Burnie-Gardiner and Patrick Marion
 
|Chris Burnie-Gardiner and Patrick Marion
 
|SISAR
 
|SISAR
−
|September 13th, 2018
 
−
|CCG has its own opertional network. However, we need to at  least catalouge their data
 
−
|The goal of this meeting was to explore how AI can be applied  in Search and rescue. However, we may at least want to catalouge their SISAR  database
 
 
|-
 
|-
 
|
 
|
 
|Bert Paulin
 
|Bert Paulin
 
|Reporting for CCG programs
 
|Reporting for CCG programs
−
|To be determined
 
−
|Further discussion required
 
−
|
 
 
|-
 
|-
 
|Canadian Hydrographic Services
 
|Canadian Hydrographic Services
 
|Terry Fanning and Matthew McGowan
 
|Terry Fanning and Matthew McGowan
 
|Main work is to move science, ocean, and species data to the  Government of Canada Open Portal
 
|Main work is to move science, ocean, and species data to the  Government of Canada Open Portal
−
|August 27th, 2018
 
−
|This program needs to be supported by TADAP
 
−
|George is contacting David Bradley, new manager of science data  sub-committee, to arrange for future meetings
 
 
|-
 
|-
 
|
 
|
 
|Claude Guay
 
|Claude Guay
 
|Metadata management and data management for whole science  sector
 
|Metadata management and data management for whole science  sector
−
|To be determined
 
−
|
 
−
|
 
 
|-
 
|-
 
|SRS, Science Branch
 
|SRS, Science Branch
 
|Theraesa Coyle and Johannie Duhaime
 
|Theraesa Coyle and Johannie Duhaime
 
|Aquaculture Monitoring Program
 
|Aquaculture Monitoring Program
−
|August 27th and September 4th, 2018
+
|-
−
|This program needs to be supported by TADAP
+
|Ocean Science Division
−
|
+
|Di Wan
 +
|Scientific oceanographic survey data management
 
|-
 
|-
 
|Fishery & Assessment Data  Section, Science Branch
 
|Fishery & Assessment Data  Section, Science Branch
−
|Bruce A. Patten and Di Wan
+
|Bruce A. Patten
 
|Scientific survey documentation for fish population  assessment, fish harvest tracking and reporting
 
|Scientific survey documentation for fish population  assessment, fish harvest tracking and reporting
−
|August 31st, 2018
 
−
|Stakeholders are very interested in collobration. Their  programs need to be supported by TADAP
 
−
|
 
 
|-
 
|-
 
|Aquaculture Management  Directorate, Aquatic Ecosystems Sector
 
|Aquaculture Management  Directorate, Aquatic Ecosystems Sector
 
|Tyree Lush and Arsenault Shane
 
|Tyree Lush and Arsenault Shane
 
|Canadian Shellfish Sanitation Program (CSSP) mapping system
 
|Canadian Shellfish Sanitation Program (CSSP) mapping system
−
|September 7th, 2018
 
−
|This program needs to be supported by TADAP
 
−
|
 
 
|-
 
|-
 
|Fisheries & Licence Policy  / Fisheries and Harbour Management
 
|Fisheries & Licence Policy  / Fisheries and Harbour Management
 
|Mark Ledwell  
 
|Mark Ledwell  
 
|FHM
 
|FHM
−
|To be determined.
 
−
|Further discussion required
 
−
|
 
 
|-
 
|-
 
|Ecosystems and Fisheries  Management: System Integration
 
|Ecosystems and Fisheries  Management: System Integration
 
|Aaron Gillis and Andrew Frost
 
|Aaron Gillis and Andrew Frost
 
|EFM-SI
 
|EFM-SI
−
|To be determined
 
−
|This program has to be supported by TADAP
 
−
|
 
 
|-
 
|-
 
|Information Management and  Policy Strategies
 
|Information Management and  Policy Strategies
−
|Lyn Warner
+
|Annette Anthony/Sylvie Boucher
 
|Open Data Portal
 
|Open Data Portal
−
|To be determined
 
−
|Has to be supported by TADAP
 
−
|
 
 
|-
 
|-
 
|Ocean Data and Information  Section
 
|Ocean Data and Information  Section
 
|Tobias Spears
 
|Tobias Spears
 
|To be determined
 
|To be determined
−
|To be determined
+
|}
 +
 
 +
== Data Catalogue Team ==
 +
please feel free to contact us to discuss the project and how you can participate
 +
 
 +
{|
 +
|'''Name'''
 +
!
 +
|-
 +
|George Esper
 +
|
 +
|-
 +
|Riham Elhabyan
 +
|
 +
|-
 +
|David Cornwell
 
|
 
|
 +
|-
 +
|Yask Shelat
 +
|
 +
|-
 +
|Abdul K Hamdo
 
|
 
|
 
|}
 
|}
−
 
−
== Data Catalog Team ==
 

Latest revision as of 10:01, 19 March 2020

Data Catalogue

Data Catalogue is an organized service that allows users to centralize Metadata and learn more about their data sources which help organizations achieve more values from their assets. Following are some advantages to centralizing Metadata:

  • Consistency and accuracy across the department/government
  • Better data congruency, quality and structure
  • Makes data easily accessible
  • Allow users to self-serve

Why Data Catalogue

USGS data Life Cycle

In an enterprise vision, we would like to minimize the number of data silos, get a faster access to what matters most, and function as a single source (during the Data Life Cycle) for better doing.
Data-driven culture empowers users with getting access to their data source. With this, the growing numbers of cloud applications, privacy regulations and security rules are making it more difficult to effectively secure and govern data

Therefore, a Data Catalogue is needed to:

  • Spend less time searching for data and more time using it to gain insight
  • Have a better/safer access to data through governance. Departments have a hard time finding data and controlling who has access to it.
  • Reduce the cost of data redundancy and hoarding.
  • Better linkage between the technical value of Metadata and the business value.

Data Catalogue Architecture

A work in progress Architecture

As explained in the following Diagram, the Catalogue will be the entry point for TADAP

When a user requires to add a data set to TADAP, the Catalogue will be able to process this request and help the user create the metadata that goes with it, data is uploaded.

1- How the Catalogue will be connected to TADAP

545x545px


2- How data is published to FGP and Open Data Portal

Catalogue Conceptual Architecure How data is published to FGP and Open Data Portal
ConceptualArchitecture2.png Catalog FGP OpenData Publish.png

List of Stakeholders

Please note this is an initial list.

Please feel free to contact us if you think there's room for improvements.

Sector Contact Person Project / Program
Conservation and Protection Lise Melanson Department Violation System (DVS)
Pauline Lalonde C-STAT
Canada Coast Guard Shawn Legault and Nicholas O’Hara SIPA
Jean-François Coutu INNAV
Chris Burnie-Gardiner and Patrick Marion SISAR
Bert Paulin Reporting for CCG programs
Canadian Hydrographic Services Terry Fanning and Matthew McGowan Main work is to move science, ocean, and species data to the Government of Canada Open Portal
Claude Guay Metadata management and data management for whole science sector
SRS, Science Branch Theraesa Coyle and Johannie Duhaime Aquaculture Monitoring Program
Ocean Science Division Di Wan Scientific oceanographic survey data management
Fishery & Assessment Data Section, Science Branch Bruce A. Patten Scientific survey documentation for fish population assessment, fish harvest tracking and reporting
Aquaculture Management Directorate, Aquatic Ecosystems Sector Tyree Lush and Arsenault Shane Canadian Shellfish Sanitation Program (CSSP) mapping system
Fisheries & Licence Policy / Fisheries and Harbour Management Mark Ledwell FHM
Ecosystems and Fisheries Management: System Integration Aaron Gillis and Andrew Frost EFM-SI
Information Management and Policy Strategies Annette Anthony/Sylvie Boucher Open Data Portal
Ocean Data and Information Section Tobias Spears To be determined

Data Catalogue Team

please feel free to contact us to discuss the project and how you can participate

Name
George Esper
Riham Elhabyan
David Cornwell
Yask Shelat
Abdul K Hamdo
Retrieved from "https://wiki.gccollab.ca/index.php?title=Data_Catalog&oldid=19275"