https://wiki.gccollab.ca/api.php?action=feedcontributions&feedformat=atom&user=Kpere060wiki - User contributions [en]2026-04-30T13:13:19ZUser contributionsMediaWiki 1.35.2https://wiki.gccollab.ca/index.php?title=Technology_Trends&diff=13855Technology Trends2019-11-18T13:59:40Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:{{FULLPAGENAME}}}}<br />
[[fr:Tendances_Technologiques]]<br />
<br />
<div class="sideTable"><br />
<div class="navigation-section"><br />
<div class="navigation-table"><br />
<table class="wikitable nav-tab" style="margin-top: 0px;"><br />
<tr><th class="breadcrumb" colspan="2"><b>Technology Trends</b><br><b>[[Tendances_Technologiques|Tendances Technologiques]]</b></th></tr><br />
<tr><td colspan="2" class="logo">[[File:Technology_brief_logo.png|200px]]</td></tr><br />
<tr><br />
<th>Trend Template</th><br />
<td>[[Media:EN_-_Technology_Trends_Template.docx|Technology Trends.docx]]</td><br />
</tr><br />
<tr><br />
<th>Other Related Publications</th><br />
<td><b>[[Architecture_Framework_Advisory_Committee|Architecture Framework Advisory Committee]]</b><hr/><b>[[Technology_Trends/User_Guide|User Guide]]</b></td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">Recent additions include [[Technology_Trends/Blockchain|Blockchain]], [[Technology_Trends/Drones|Drones]], [[Technology_Trends/Internet_of_Things|Internet of Things]], [[Technology_Trends/Low_Code_Application_Development|Low Code Application Development]], [[Technology_Trends/Open_by_Default|Open by Default]], [[Technology_Trends/Zero_Trust_Network|Zero Trust Network]].</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
</div><br />
</div><br />
<br />
<div><br />
<br />
<p>In an effort to better support its mandate of providing innovative Information Technology (IT) services to Government of Canada (GC) departments and agencies, Shared Services Canada (SSC) is actively researching and evaluating emerging technology trends. This research aims to Inform and prepare SSC, and the broader GC community, to handle new disruptive technologies that are just over the horizon by outlining potential benefits, impacts and use cases.</p><br />
<br />
<p>Technology trend briefing papers are the vehicle used to convey the research conducted on a given subject. These briefing papers are short, to the point, and are targeted towards both a business and technical oriented readership.</p><br />
<br />
<p>SSC is looking to the broader GC community to contribute and collaborate on existing and future topics. By fostering a more collaborative and inclusive development approach, we hope to reduce GC research duplication and accelerate the development of technology roadmaps across government by referencing common GC documentation.</p><br />
<br />
<h2>Format</h2><br />
<br />
<p>Each technology trend briefing paper follows a common template:</p><br />
<br />
<ul><br />
<li><b>Business brief </b>describes the business aspect of the technology by offering a high level understanding, the origin and the uses.</li><br />
<li><b>Technical brief </b>gives an in-depth description of the technology. It focuses on the most important technical aspects of the technology without going into too much detail.</li><br />
<li><b>Industry usage </b>offers an understanding of how the technology is currently used or thought to be used in the industry when commercially available. This section clarifies the benefits of the technology through examples.</li><br />
<li><b>Canadian Government use </b>details how the technology is currently used or thought to be used in the context of the Federal Canadian Government when commercially available. This intent is to provide an understanding of how the technology could ultimately benefit the federal government in regards to the delivery of services to Canadians through short examples.</li><br />
<li><b>Implications for Shared Services Canada (SSC) </b>demonstrates the clear and measurable benefits that pertains to Shared Service Canada when adopting the use of the technology. This section is comprised of these following subsections :<br />
<ul><br />
<li><b>Value proposition </b>illustrates the potential strengths and opportunities that SSC can encounter by leveraging the technology.</li><br />
<li><b>Challenges </b>outlines the risks and difficulties associated with the technology that includes both risks of implementation such as the development of the product or the risks of solution by adopting the technology.</li><br />
<li><b>Considerations </b>presents any recommendations SSC may have for another department of the Government of Canada that wishes to adopt and leverage the technology.</li><br />
</ul><br />
</li><br />
</ul><br />
<br />
<h2>Development Process</h2><br />
<br />
<p>External market trends, private-public sector reports, online trends and Government of Canada strategic plans are all factors considered when selecting a topic for development. Once a topic is selected for development, it undergoes a five phase development process prior to publishing.</p><br />
<br />
<div>[[File:Technology_Trends_-_Development_Process.png|center]]</div><br />
<br />
<h2>How to Contribute</h2><br />
<br />
<p>We want your help! Our goal is to ensure that the content generated is developed in the open, routinely kept up to date, representative of the wide ranging GC departmental mandates and widely distributed to the broader Canadian Government community. To that end, we strongly encourage you to share, edit and provide feedback on the published content.</p><br />
<br />
<p>You can edit by making use of the “Discussion” tab located at the top-left of the page.</p> <br />
<br />
<p>Should you wish to sponsor and or author a topic not listed below, please reach out to [mailto:jean-francois.michaud@canada.ca Jean-François Michaud]</p><br />
<br />
</div><br />
<br />
<div class="container"><br />
<div class="row"><br />
<div class="col-sm-12"><br />
<h2>Technology Trend Briefing Papers</h2><br />
<br />
<table class="wikitable sortable technology-table" style="border: 0px solid;"><br />
<tr><br />
<th colspan="3">Name</th><br />
<th>Status</th><br />
<th style="white-space: nowrap;">Release Date</th><br />
<th>Official Publications</th><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Blockchain|Blockchain]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Chaîne_de_Blocs|Chaîne de Blocs]]</td><br />
<td>Published</td><br />
<td>2019-05-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Blockchain.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Blockchain.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Drones|Drones]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Drones|Drones]]</td><br />
<td>Published</td><br />
<td>2019-07-04</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Drones.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Drones.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Internet_of_Things|Internet of Things]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Internet_des_Objets|Internet des Objets]]</td><br />
<td>Published</td><br />
<td>2019-07-04</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Internet_of_Things.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Internet_des_Objets.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Low_Code_Application_Development|Low Code Application Development]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Développement_d%27Application_LowCode|Développement d'Application Programmation Schématisée]]</td><br />
<td>Published</td><br />
<td>2019-05-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Low_Code_Application_Development.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Développement_d%27Application_LowCode.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Open_by_Default|Open by Default]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Ouvert_par_Défaut|Ouvert par Défaut]]</td><br />
<td>Published</td><br />
<td>2019-05-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Open_by_Default.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Ouvert_par_Défaut.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Zero_Trust_Network|Zero Trust Network]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Réseaux_Zéro_Confiance|Réseaux Zéro Confiance]]</td><br />
<td>Published</td><br />
<td>2019-05-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Zero_Trust_Network.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Réseaux_Zéro_Confiance.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/5G_Networks|5G Networks]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Réseaux_5G|Réseaux 5G]]</td><br />
<td>Published</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_5G_Networks.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Réseaux_5G.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Big_Data|Big Data]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Mégadonnées|Mégadonnées]]</td><br />
<td>Published</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Big_Data.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Mégadonnées.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Edge_Computing|Edge Computing (Serverless Computing)]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Informatique_en_Périphérie|Informatique en Périphérie (Informatique sans Serveurs)]]</td><br />
<td>Published</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Edge_Computing.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Informatique_en_Périphérie.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Face_Recognition|Face Recognition]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Reconnaissance_Faciale|Reconnaissance Faciale]]</td><br />
<td>Published</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Face_Recognition.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Reconnaissance_Faciale.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Datalakes|Datalakes]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Lac de Données</td><br />
<td>Translation</td><br />
<td>2019-08-06</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Datalakes.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Digital_Twins|Digital Twins]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Jumeaux Numériques</td><br />
<td>Translation</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Digital_Twins.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Kubernetes|Kubernetes]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Kubernetes</td><br />
<td>Translation</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Kubernetes.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Data_Leak_Prevention|Data Leak Prevention]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Prévention des Fuites de Données</td><br />
<td>Translation</td><br />
<td>2019-08-06</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Data_Leak_Prevention.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Cloud_Management_Platform|Cloud Management Platform]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Platforme de Gestion d'Infonuagique</td><br />
<td>Translation</td><br />
<td>2019-09-18</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Cloud_Management_Platform.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Digital_Assistants|Digital Assistants (Conversational UI)]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Assistants Numériques (Interface Conversationnelle)</td><br />
<td>Translation</td><br />
<td>2019-09-25</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Digital_Assistants.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Enterprise_Application_Stores|Enterprise Application Stores ]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Magasins d'Applications d'Entreprise </td><br />
<td>Translation</td><br />
<td>2019-10-30</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Enterprise_Application_Stores.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Software_Defined_Anything|Software Defined Anything]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Tout Défini par Logiciel</td><br />
<td>Translation</td><br />
<td>2019-10-30</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Software_Defined_Anything.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Digital_Services|Digital Services]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Services Numériques</td><br />
<td>Translation</td><br />
<td>2019-11-15</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Digital_Services.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Interoperability (Tell Us Once)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Interopérabilité</td><br />
<td>Development</td><br />
<td>2019-07-23</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">3D XPoint (Optane)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">3D XPoint (Optane)</td><br />
<td>Development</td><br />
<td>FY 19-20 Q2</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Application_Containers|Application Containers]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Conteneurs Logiciels</td><br />
<td>Development</td><br />
<td>FY 19-20 Q2</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">AI (Machine Learning)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">IA (Apprentissage Automatique)</td><br />
<td>Development</td><br />
<td>FY 19-20 Q2</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Crowdsourcing</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Production participative</td><br />
<td>Development</td><br />
<td>FY 19-20 Q2</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Autonomous Vehicles</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Véhicules Autonomes</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Infrastructure as Code</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Infrastructure en tant que Code</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">SCADA Systems</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Systèmes SCADA</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Cloud Workplace (Office)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Bureau Infonuagique</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Low Earth Orbit System</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Système en Orbite Terrestre Basse</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Quantum Computing</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Informatique Quantique</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Robotics</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Robotiques</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Data Centre Infrastructure Monitoring</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Surveillance de l'Infrastructure du Centre de Données</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Digitial / Smart Offices</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Bureaux Digitales / Smart</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Enterprise Collaboration</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Collaboration d'Entreprise</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Gamification (Innovation Framework)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Gamification (Cadre d'Innovation)</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Intent Based Networking</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Réseaux basé sur l'Intention</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">IT Talent</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Talent Informatique</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Pay as you go</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Pay as you go</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Li-Fi|Li-Fi]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Li-Fi</td><br />
<td>Unassigned</td><br />
<td></td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Augmented_and_Virtual_Reality|Augmented and Virtual Reality]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Li-Fi</td><br />
<td>Unassigned</td><br />
<td></td><br />
<td></td><br />
</tr><br />
<br />
</table><br />
</div><br />
</div><br />
</div><br />
<br />
[[Category:Architecture]]<br />
[[Category:Technology]]<br />
<br />
{{#css:<br />
.technology-table{<br />
width: 100%;<br />
margin: auto;<br />
}<br />
<br />
th, td{<br />
text-align: center;<br />
}<br />
<br />
.logo{<br />
text-align: center;<br />
}<br />
<br />
.disclaimer{<br />
border: 1px solid red;<br />
background-color: rgba(255, 0, 0, 0.3);<br />
}<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
padding: 5px 10px 5px 10px;<br />
}<br />
<br />
.disclaimerText{<br />
width: 100%;<br />
text-align: left;<br />
}<br />
<br />
.news{<br />
border: 1px solid green;<br />
background-color: rgba(0, 255, 0, 0.3);<br />
}<br />
<br />
.toggle{<br />
color: rgb(6, 69, 173);<br />
float:left;<br />
<br />
}<br />
<br />
.toggle:hover{<br />
text-decoration: underline;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { float: left; }<br />
<br />
.col-sm-1 { width: 8.333333333%; }<br />
.col-sm-2 { width: 16.66666666%%; }<br />
.col-sm-3 { width: 25%; }<br />
.col-sm-4 { width: 33.333333333333%; }<br />
.col-sm-5 { width: 41.6666666666%; }<br />
.col-sm-6 { width: 50%; }<br />
.col-sm-7 { width: 58.333333333%; }<br />
.col-sm-8 { width: 66.66666666666%; }<br />
.col-sm-9 { width: 75%; }<br />
.col-sm-10 { width: 83.33333333333%; }<br />
.col-sm-11 { width: 91.6666666666%; }<br />
.col-sm-12 { width: 100%; }<br />
<br />
.sideTable {<br />
width: 27.5%;<br />
float: right;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 95%; <br />
margin-left: 2.5%; <br />
margin-right: 2.5%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
}<br />
<br />
.inverse-direction {<br />
float: right;<br />
}<br />
<br />
.td-left { border-right: 0px solid; }<br />
.td-middle { border-right: 0px solid; border-left: 0px solid; }<br />
.td-right { border-left: 0px solid; }<br />
<br />
.nav-tab {<br />
width: 100%;<br />
margin-top: 0px;<br />
padding-top: 0px;<br />
}<br />
<br />
<br />
@media (max-width: 1050px){<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.sideTable {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%; <br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%; <br />
margin-right: 0%;<br />
}<br />
<br />
/*<br />
.navigation-table {<br />
width: 30%;<br />
margin-left: 1.6666666%; <br />
margin-right: 1.6666666%;<br />
float: left;<br />
}<br />
*/<br />
<br />
.content-section {<br />
float: left;<br />
}<br />
<br />
.inverse-direction {<br />
clear: both;<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { clear: both; width: 100%; margin: auto; }<br />
}<br />
<br />
div>a>img {<br />
width: 600px;<br />
height: auto;<br />
margin: auto;<br />
}<br />
<br />
@media (max-width: 850px){<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%; <br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
margin-left: 0%; <br />
margin-right: 0%;<br />
clear: both;<br />
}<br />
}<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Augmented_and_Virtual_Reality&diff=13690Technology Trends/Augmented and Virtual Reality2019-11-14T03:09:24Z<p>Kpere060: Created page with "{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}} fr:Tendances_Technologiques/Chaîne_d..."</p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Augmented and Virtual Reality</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques/|Tendances_Technologiques/]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[File:|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Published</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 23, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>May 23, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:AR_VR.pdf|Augmented and Virtual Reality.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Augmented reality (AR)</b> provides a live direct, or indirect, view of the real-world with aspects that are modified or “augmented” by aid of a computer.</p><br />
<br><p><b>Virtual Reality (VR)</b> on the other hand, occludes (obstructs) the entire field of view of the user (i.e. a headset and other sensory gear), and replaces the entire surrounding environment with a completely computer-generated one.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>Augmented reality (AR) provides a live direct, or indirect, view of the real-world with aspects that are modified or “augmented” by aid of a computer. These computer-generated modifications can be applied across multiple sensory modes, including visual, audio, haptic (touch), and olfactory (smell). AR technology provides an enhancement to the surrounding environment and can be either constructive (add elements to the environment) or destructive (mask elements through localized occlusion without blocking the entire real world view of the user).</p><br />
<p>AR and VR concepts are found on the virtuality continuum, otherwise known as the mixed reality spectrum (the spectrum). One end of the spectrum is unmodified reality and the opposite end is digital reality, with anything in between falling under Mixed Reality (MR). The spectrum includes all possible variations and combinations of real and virtual objects, such as augmented virtuality, where elements of reality augment the virtual world. Virtual and augmented technologies are essentially interactive mediums that enable the combination of the virtual and real world where physical and digital objects may co-exist in real time.</p><br />
<p class="inline">Devices used to overlay digital elements onto reality (AR) include, AR enabled smartphones, tablets, and see-through hands-free head mounted displays (HMD). Most consumer AR applications are created for smartphones due to their proliferation and availability. At this point, HMDs do not have many consumer AR applications due to their prohibitive price point, however, many business use cases have been developed for professional use. This is due to the fact that the hands free nature of the HMD devices allows for more creativity in the development of applications, where other smaller but necessary sensors and cameras can be used in these purpose built devices. For VR, HMDs completely obstruct the user’s field of view and replaces it with a digital environment are usually paired with any combination of controllers and wearable body trackers that let the user physically interact with the virtual world.</p> <p class="expand mw-collapsible-content"> Another concept for enabling VR is the Cave Automatic Virtual Environment (CAVE) where a user is completely immersed within a physical room with walls made up of rear-projection screens (or flat panel screens) that displays a virtual environment that can be interacted with via controller or camera sensors. HMDs are used to immerse the senses in VR, whereas the CAVE system is used to immerse the whole person within a virtual space. </p><br />
<br />
<h2>Technology Brief</h2><br />
<p>AR and VR make use of several different technologies. Both use sensors to gain a mapping of the surrounding environment, however VR technology also makes use of sensors outside a headset to provide a more immersive experience. VR headsets like the HTC Vive and Oculus Rift track different body movement in order to allow the user to be able to look around and interact with virtual items. The image processing needs to be done at around a 60 frames per second for the user to be able to move their head naturally. Several companies currently working on VR technology are now also attempting to track eye movements within their headsets. This will allow the images displayed on screen to appear even more realistic as the image will automatically focus based on where the user is looking. </p><br />
<p>Currently on the market there are three categories of VR headsets available:</p><br />
<ol><br />
<li><p><b>Headsets for Gaming Consoles and PC:</b>this grouping of headsets relies on the extra computing power of gaming consoles and personal computers. Since they are tethered to a computer system, more peripherals can also be connected and integrated into the VR experience like handheld controllers, motion trackers, eye tracking glasses, glove controllers and treadmills.</p></li><br />
<li><p><b>Standalone Headsets:</b>these are wireless headsets that do not need to be plugged into a computer or smartphone to work. All of the necessary technology is packed into the device; the computing hardware, motion tracking, controller connectivity, and the operating system itself.</p></li><br />
<li><p><b>Mobile Headsets:</b>these headsets are powered by smartphones that are VR enabled, where the smartphone acts as the screen. These are essentially just headgear that keep the phone in place on the wearer’s head and they optimize the wearer’s field of view for VR. Some may add more tracking capabilities to the smartphone and be made of high quality materials, and others like Google Cardboard are the bare essentials to make the concept work. </p></li><br />
</ol><br />
<p>AR technology deploys a virtual image over real-world objects. The system receives input from the camera or other input devices. There are generally three approaches which AR technology makes use of. Simultaneous Localization and Mapping (SLAM) is a set of algorithms tasked with solving complex localization and mapping problems. This technology localizes sensors with respect to their surroundings. It uses this data to map the structure of the immediate environment and allows applications to incorporate real world information into the graphics generated on screen (i.e. the Ikea Place app that generates proportional examples of furniture onto a user’s living space). </p><br />
<p>Recognition based or marker based AR is another approach to the design of AR systems. The camera of the device identifies visual markers using either, QR/2D code or natural feature tracking markers. When the device senses a marker it places the virtual image at that position. It uses these markers to determine orientation and position of the marker image.</p><br />
<p>Lastly, location based AR is another technology that utilizes GPS, digital compass, velocity meter, and accelerometer as inputs to determine orientation and position of the virtual image. This is a popular approach to AR technology as smartphones already have the necessary infrastructure to provide all these inputs. Simply put, there are two ways in which augmented reality can be viewed: </p><br />
<ol><br />
<li><p><b>Augmented Reality Glasses: </b>all of the processing power and projecting technology is packed into the glasses. 3D augmentations are superimposed into the wearer’s direct line of sight (i.e. Google Glass). </p></li><br />
<li><p><b>Smartphones: </b>the user must view reality through the screen of their phone where the computer graphics are then applied. </p></li><br />
</ol><br />
<h2>Industry Usage</h2><br />
<br />
<p>AR and VR technology has a wide variety of applications from both commercial and business standpoints and more possibilities are being explored constantly. The global AR market is expected to grow to $60.55 billion (USD) <ref>Markets and Markets. (July, 2017). Augmented Reality Market by Offering, Device Type, Application, and Geography. MarketsandMarkets Research Private Ltd. Retrieved on 13-February-2019 from: <i>[www.marketsandmarkets.com/Market-Reports/augmented-reality-market-82758548.html?gclid=EAIaIQobChMI_J60r6_Q4AIVCZ7ACh3BuQ4tEAAYASAAEgLvq_D_BwE. ]</i></ref> by 2023 and the VR market is expected to be $44.7 billion by 2024<ref> Markets and Markets. (January, 2019). Virtual Reality Market by Offering, Device Type, Application, and Geography. MarketsandMarkets Research Private Ltd. Retrieved on 13-February-2019 from: <i>[www.marketsandmarkets.com/Market-Reports/augmented-reality-market-82758548.html?gclid=EAIaIQobChMI_J60r6_Q4AIVCZ7ACh3BuQ4tEAAYASAAEgLvq_D_BwE. ]</i></ref> . Within the workplace, several companies are experimenting with virtual workstations. The Virtual Desktop app capable of being used on the Oculus Rift and HTC Vive is designed to provide users with a virtual Windows Desktop that can be physically interacted with in virtual reality. <ref>Heaney, David. (November 29th, 2018). Virtual Desktop (Mobile) Review: The Best Way to Access Your PC From Occulus Go. UploadVR. San Francisco, USA. Retrieved on 13-February-2019 from: <i>[https://uploadvr.com/virtual-desktop-oculus-go-review/ ]</i></ref> On the AR side, a company called Meta is testing the use of an interactive holographic workstation to replace desktop computers. Users can instead use a 3D environment to store, display, and interact with virtual objects and other documents. <ref> Wang, Selina. (July 18th, 2017). This Startup Wants to Replace Your Office With 3d Holograms. Bloomberg L.P. New York, USA. Retrieved on 11-February-2019 from: <i>[https://www.bloomberg.com/news/articles/2017-07-18/this-startup-wants-to-replace-your-office-with-3d-holograms ]</i></ref> </p><br />
<p>For general use, Google Translate running on a smartphone can be used in AR mode to translate speech and text. When using the camera on a smartphone the Google Translate application can process an image with text written in one language and translate it to another. Another sensory modality of AR is provided by Google in the form of natural language translation. A recent update has enabled all Google Assistant optimized headphones and Android phones with the ability to do live audio translations. <ref>Google. (2019). What are Google Pixel Buds?. Google. Retrieved 11-February-2019 from: <i>[https://support.google.com/googlepixelbuds/answer/7545575?hl=en&ref_topic=7544331 ]</i></ref> </p><br />
<p>The process of prototyping is undergoing a major shift towards using AR and VR to explore and test new prototypes in a cost-effective way. The benefit of these technologies is that it allows all interested parties (engineers, assembly line workers, customers, management, marketers, safety regulators, etc.) of a project to come together and interact with a model before it goes into production. For example, the tech firm Finger Food created an AR solution for the truck manufacturer Paccar, which could be seamlessly integrated into their design workflow. <ref>Finger Food. (2019). Paccar: Fundamentally changing big industry with big technology. Finger Food Studios. Port Coquitlam, British Columbia. Retrieved 11-February-2019 from: <i>[https://www.fingerfoodstudios.com/case-study/paccar/ ]</i></ref> It allows the user to view and interact with full scale models of trucks and to test various physics simulations on the models in real time. Another example is the global architecture firm IA (Interior Architecture) who is using InsiteVR to build models of their designs. <ref>InsiteVR. (2019). VR Meetings for Architecture, Engineering, and Construction. InsiteVR. 2019. Retrieved 13-February-2019 from: <i>[https://www.insitevr.com/ ]</i></ref> This also allows clients of the company to “tour” the designs and give input before designs are finalized. Ford is another company using VR technology to help its engineers when designing elements of their vehicles. Audi has taken this technology to the consumer allowing customers to view, configure, and customize certain elements of their vehicles. </p><br />
<p>Education and training are also starting to change thanks to the adoption of AR and VR. Through VR, trainees can be immersed in a training environment where they can get hands-on practice without the possibility of real life mistakes. The aviation industry has been using simulation technologies for decades, but now with the relative ease of access to those technologies more sectors are adopting AR and VR for training purposes. For example, the forklift training company certify.me created a VR app for certifying new forklift operators where trainees are presented with a variety of virtual scenarios that demonstrate what to do and what not to do. <ref>CertifyMe.net (September, 2016) Virtual Reality Training: New Forklift Training Video in VR Certifyme.net. Retrieved 15-February-2019 from: <i>[https://www.certifyme.net/virtual-reality-training/ ]</i></ref> NASA has a mixed reality space where astronauts in training are immersed in a VR simulation while also being in a 1:1 physical space where they can walk around and touch real objects presented to them in the VR environment. <ref> Kolodny, Lora. (2017). Nasa is using a mixed reality space station to train astronauts. TechCrunch. Retrieved 15-February-2019 from: <i>[https://techcrunch.com/2017/03/26/nasa-is-using-a-mixed-reality-space-station-to-train-astronauts/ ]</i></ref> </p><br />
<p class="inline">Repair and construction work have also been augmented. Recently, some AR solutions have been created to help workers make effective repairs. For example, Boeing engineers are now equipped with glasses that overlay instructions and the proper placement of wires over a physical object that is being repaired. <ref>Upskill. (2019) Upskill and Boeing: Reinventing aerospace manufacturing and supply chain management. Upskill. Retrieved 15-February-2019 from: <i>[https://upskill.io/landing/upskill-and-boeing/]</i></ref> The glasses are also equipped with a camera that lets a supervisor view the work being done and give live feedback. The adoption of AR has reduced production time for Boeing by 25% and lowered error rates to practically zero. The construction industry is also adopting a mixed reality approach for completing construction projects. A recent development is an augmented reality app for construction workers that super-imposes the 3D mockup of a construction project over the real environment being built. <ref>Lorek, Sarah. (January 30th, 2018) Mixed Reality in Construction Gets Real with Trimble’s Hololens Hard Hat. Constructible. Retrieved 13-February-2019 from: <i>[https://constructible.trimble.com/construction-industry/mixed-reality-in-construction-gets-real-with-trimble-hololens-hard-hat-video ]</i></ref> This has helped with accurately finishing projects with fewer interpretation errors, as opposed to the traditional method of comparing the worksite to a two dimensional architectural plan.</p> <p class="expand mw-collapsible-content"> Augmented reality is also well suited for the itemization of tasks, in the sense that all of the steps of a process or a task are neatly overlaid on a wearer’s field of view, until the item is completed properly. For example, AR has been used to help workers navigate large and complicated warehouse layouts to locate packages they need to find to complete an order. <ref>Porter, Michael, et al. (November-December 2017) A Manager’s guide to Augmented Reality. Harvard Business Publishing. Retrieved 11-February-2019 from: <i>[https://hbr.org/2017/11/a-managers-guide-to-augmented-reality ]</i></ref> </p><br />
<p class="inline">AR and VR are providing marketers with novel spaces and new formats for creating campaigns. Many business have developed AR solutions that let consumers “test drive” products before they are purchased, like trying on clothes, glasses, and hairstyles. Ikea, for example created the Ikea Place app that lets users see how a virtual piece of furniture would look inside their home at full scale. <ref> Dasey, Daniel. Try Before you Buy: Ikea Place Augmented Reality. Ikea. Retrieved 15-February-2019 from: <i>[https://highlights.ikea.com/2017/ikea-place/ ]</i></ref> There is also a shift towards making advertisements more interactive and immersive as they will leave a longer lasting impression on the audience, as opposed to traditional advertising efforts.</p><p class="expand mw-collapsible-content"> To promote the horror franchise “Saw” a VR mini-game was created where users needed solve a puzzle within an interactive room. <ref>D’Angleo, Matt. (December 5th, 2017). How Virtual Reality is Impacting the Ad Industry. Business.com. Retrieved 27-February-2019 from: <i>[: https://www.business.com/articles/virtual-reality-advertising-augmented/ ]</i></ref> Biometric data from that group was compared with a control group that only saw a trailer in VR, and it revealed that the VR room left a longer lasting impression. Within a VR environment, biometric data can be measured in real time which lets marketers know how effective a campaign really is. </p><br />
<p>Entertainment is still one of the largest uses for AR and VR. The explosive and sustained popularity of Pokémon Go (a mobile game AR application with 147 million monthly users <ref>Philips, Tom. (June 27th, 2018). Pokemon Go Active Player Count Highest Since 2016 Summer Launch. EuroGamer. Retrieved 27-February-2019 from: <i>[https://www.eurogamer.net/articles/2018-06-27-pokemon-go-player-count-at-highest-since-2016-summer-launch ]</i></ref>) proves there is a large base of users who are open to using the technology. VR games come in all shapes and sizes and the market is projected to hit 22.9 billion by 2020 <ref>Statista. (2015). Virtual Reality video gaming sales revenue worldwide from 2015-2020. Statista. Retrieved 29-Fabruary-2019 from: <i>[https://www.statista.com/statistics/499714/global-virtual-reality-gaming-sales-revenue/ ]</i></ref>. VR as a medium is also being explored for the film industry and it is said to improve empathy and it can be used as a teaching tool. Even in the world of law, Bloomberg Law speculates that VR might be incorporated into the courtroom, so that evidence can be better understood in an immersive environment. <ref>Kaufman, Bruce. (November 16th, 2017). The Next Frontier for Virtual Reality: Courtrooms. Bloomberg BNA. Retrieved 20-February-2019 from: <i>[https://www.bna.com/next-frontier-virtual-n73014472186/ ]</i></ref> </p><br />
<p>Use cases have also been explored in the medical field. Since medical practitioners rely heavily on visual information, AR and VR is well placed to help them visualize tasks. One such creation is a handheld device developed by Accuvein that reads the heat signature of a patients veins and projects an AR map of the underlying vasculature onto their skin. <ref>AccuVein. (2015). AccuVein Vein Visualization: The Future of Healthcare is Here. AccuVein. Retrieved 18-February-2019 from: <i>[https://www.accuvein.com/why-accuvein/ar/ ]</i></ref> This has increased the likelihood of finding a vein on the first try by 3.5 times for those using the machine. A patient’s information can also be used to pre-plan surgeries in VR, their CT and MRI scans can be used to recreate a digital body that can be explored in detail in VR. <ref>mVR Division. Surgical Navigation Advanced Platform. Surgical Theater LLC. Los Angeles, California. Retrieved 18-February-2019 from: <i>[https://www.surgicaltheater.net/plan-navigate/ ]</i></ref> Similarly, those same scans can be super-imposed onto the patient in real-time to guide doctors to the problem area before a surgery and have a better idea of what they are targeting under the patient’s skin.<ref>Microsoft Reporter. (February 8th, 2018). Surgeons use Microsoft HoloLens to see inside patients before they operate. Microsoft. Retrieved 20-February-2019 from: <i>[https://news.microsoft.com/en-gb/2018/02/08/surgeons-use-microsoft-hololens-to-see-inside-patients-before-they-operate-on-them/ ]</i></ref> </p><br />
<h2>Canadian Government Use</h2><br />
<p>The GC has invested $9.5 million in AR technology developed by Vancouver-based software company NGRAIN. <ref>Market Wired. (June 26th, 2013). Harper Government Supports Market-Leading Canadian SME. Yahoo Finance. Retrieved 22-February-2019 from: <i>[https://finance.yahoo.com/news/harper-government-supports-market-leading-180734827.html ]</i></ref> The software company has developed an interactive 3D AR platform for aerospace company Lockheed Martin. Lockheed Martin uses this interactive platform as a tool to perform maintenance on fighter jets like the F-35 and F-22. The company reports reduced maintenance time and less errors. The Canadian Military is also investing in VR, they have recently purchased a mixed reality simulation to train helicopter operators in the use of a hoist system.<ref>Burke, David. (April 10th, 2018) Fear makes virtual training lessons stick in Nova Scotia’s thriving VR industry. Canadian Broadcasting Corporation. Retrieved 20-February-2019 from: <i>[https://www.cbc.ca/news/canada/nova-scotia/virtual-reality-business-technology-nova-scotia-1.4609172 ]</i></ref> This and other projects like it reduce the training costs associated with hands-on jobs, where no costly accidents can happen and learners are allowed to learn from their mistakes in a risk-free environment.</p><br />
<p>The GC has also been investing in AR and VR as tools of education and entertainment. Through the Build in Canada Innovation Program, $482,000 was awarded to Motive.io to develop a location based AR experience that takes users through Ottawa’s cultural heritage. <ref>Public Services and Procurement Canada. (November 20th, 2017). Government of Canada supports BC augmented reality startup through Build in Canada Innovation Program. Published by Cision Canada. Government of Canada. Retrieved 22-February-2019 from: <i>[https://www.newswire.ca/news-releases/government-of-canada-supports-british-columbia-augmented-reality-startup-through-build-in-canada-innovation-program-658845663.html ]</i></ref> Although there is no substantial research yet to support the claim that immersive VR experiences increase empathetic responses, many projects are being developed with that goal in mind. The Social Sciences and Humanities Research Council (SSHRC) helped fund a VR Residential School that takes viewers into a world crafted by survivors and researchers, it’s hoped that the project will eventually be adopted as a teaching tool for high schools. <ref>University of Manitoba. (September 12th, 2018). Launch of a virtual reality Canadian Residential School. University of Manitoba. Winnipeg, Manitoba. Retrieved 20-February-2019 from: <i>[http://news.umanitoba.ca/launch-of-a-virtual-reality-canadian-residential-school/ ]</i></ref> Even the Parliament of Canada has been turned into an interactive augmented virtuality environment, this time by partnering with Carleton University to create a virtual tour of the Senate where users can walk around the building and click on points of interest to learn of their historical significance.<ref>Senate of Canada. (March 1st, 2017). Senate Virtual Tour. Government of Canada. Retrieved 6-February-2019 from: http://tour-ext.sencanada.ca/tour/index.html and <i>[https://sencanada.ca/en/sencaplus/how-why/the-senate-virtual-tour-your-digital-admission-to-the-red-chamber/ ]</i></ref> </p><br />
<p>The Canadian Military stands to benefit the most from the use of AR and VR technology, given their needs for logistics management, training solutions, and personnel management. At present, the military mainly uses those technologies for training purposes. Some simulations are entirely based in VR, and some have real physical equipment that can be “used” in a VR environment. Bluedrop, a performance learning and IT consulting and development firm, has designed both types of training programs, and one of their mixed reality simulations in particular integrates a real CH-47 Chinook helicopter fuselage into a virtual training program. <ref>Arellano, Nestor. (August 26th, 2019). VR, gaming technology and training the future soldier. Vanguard Canada. Retrieved 6-February-2019 from: <i>[https://vanguardcanada.com/2016/08/26/vr-gaming-technology-training-future-soldier/ ]</i></ref> The Canadian Armed Forces has more than 230 simulators in 125 locations across Canada, and not included in this count is the number of Navy and Airforce simulators<ref>Boutilier, Alex. (January 1st, 2015). Canadian Forces expand virtual reality training capabilities. Toronto Star Papers. Toronto, Ontario. Retrieved 8-February-2019 from: <i>[https://www.thestar.com/news/canada/2015/01/01/canadian_forces_expand_virtual_reality_training_capabilities.html ]</i></ref>. Military therapists have also explored the use of VR to treat post-traumatic stress disorder in soldiers, as a way for them to safely explore the source of their trauma and move past it. <ref>Madigan, Jennifer. (December 19th, 2017). Virtual reality simulation and neurofeedback to help PTSD. Blueline, Annex Business Media. Toronto, Ontario. Retrieved 8-February-2019 from: <i>[https://www.blueline.ca/news/features/virtual-reality-simulation-and-neurofeedback-to-help-ptsd-5076 ]</i></ref> Through the Innovation for Defense Excellence and Security program, a few projects involving AR and VR have received funding from the Department of National Defense. Categories include cognitive performance enhancement, Resilient Non-GPS Based Positioning Navigation and Timing, and understanding and addressing PTSD.<ref>Department of National Defense. IDEas Recipients. Government of Canada. Retrieved 8-February-2019 from: <i>[https://www.canada.ca/en/department-national-defence/programs/defence-ideas/past-opportunities/ideas-recipients.html ]</i></ref> </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<br />
<h4>Value Proposition</h4><br />
<p>SSC should consider the positive impacts that AR and VR can have on the Department. For the Department to benefit from AR and VR technologies, very specific projects and use cases may be considered before applications are developed. Projects should not be launched simply for the sake of innovation but because they fulfill business objectives and requirements. The Department could lead by example for other departments by adopting AR and VR solutions that have already been developed and promote those solutions.</p><br />
<p>The Government of Canada (GC) can provide better services to citizens and other departments through the use of AR and VR technology. The government can provide a clearer vision of its planning and documentation. For architectural projects, for example, virtual tours of proposed sites and renovations can be given to citizens and other stakeholders. The same concept can also be used in courtrooms to help explain evidence in a convincing and immersive way. Policy and documentation can now become more interactive in the world of AR and VR technology. </p><br />
<p>Additionally, AR and VR technology can accommodate the accessibility needs of GC employees and citizens interacting with the GC. AR and VR are part of a suite of technologies known as Adaptive Computer Technology (ACT), which aids in providing assistance to those with sensory, mobility, dexterity and learning disabilities who work on or communicate through computers. AR and VR can reduce the barriers presented by the standard computer interface and can offer new and adaptive ways for GC employees and citizens to interact with GC information and data. <ref>Shared Services Canada. (April 3rd, 2019). Accessibility, Accommodation and Adaptive Computer Technology (AAACT). Government of Canada. Aaact-aatia@ssc-spc.gc.ca. (819) 994-4835. Retrieved 03- May-2019 from: <i>[https://www.canada.ca/en/shared-services/corporate/aaact-program.html ]</i></ref> </p><br />
<p class="expand mw-collapsible">The use of AR and VR technology can also aid and support public safety and emergency services. For example, if emergency vehicles are fitted with AR Head-Up windshield displays this could help with route navigation, allowing responders to find the quickest route and obtain traffic information in real-time. Location based AR technology could also allow first responders to gain insight on dangers and hazardous conditions around them. Through the Innovative Solutions Canada program, a current challenge open to the public is the creation of a heads up and hands free solution for supporting firefighters in the field. <ref> Innovation, Science and Economic Development Canada. (2018). Head Up Hands Free Firefighting. Government of Canada. Retrieved 8-February-2019 from: <i>[http://www.ic.gc.ca/eic/site/101.nsf/eng/00056.html]</i></ref> </p><br />
<p>In general, AR and VR technologies are well suited for tasks that rely heavily on visual elements. The industry is not yet mature (see Appendix A) so not all potential use cases have been explored, or even discovered. At present, AR business use cases include the itemization of tasks, remote work support (a supervisor can see and comment on what a worker is doing if a HDM has a camera), performance dashboards for Internet of Things (IoT) enabled devices, guiding precision manual labor (i.e. repair, assembly, construction), prototyping, and for overlaying medical information on a patient.</p><br />
<p>Similarly, not all use cases have been uncovered for VR but there are many existing business solutions. A strength of VR is that it lends itself well for immersive experiences since the entire field of view is replaced with a virtual world. Video games and 360̊ videos are very popular for consumer use and can be easily transformed for business use. Mixed reality practice simulations are being adopted to replace high-risk training scenarios since there is less risk of injury and equipment damage. Undergoing training in VR helps learners retain information since they are being completely immersed into a practice scenario where they can physically interact with virtual objects (as opposed to learning in a classroom). Prototyping and virtual tours can also be done in VR</p><br />
<p>As a communications service provider, SSC is well suited to become a provider of AR and VR equipment for other partnered departments. The Department could also lead the development of AR and VR solutions on an on demand basis for other departments.</p><br />
<h4>Challenges</h4><br />
<p>Prolonged use of AR and VR gear has not been extensively studied, and there could be unknown risks that have yet to be uncovered, such as health risks to employees etc. The use of AR and VR in the form of wearable technology in the workplace could also provide further challenges. Although the technology can increase connectivity among employees and simplicity of user interfaces, AR and VR technology could also cause employees to lose focus on the task at hand when working using the AR and VR technology. This challenge could be expected to diminish as AR and VR technologies become more commonplace.</p><br />
<p>AR and VR technologies are great for the immersion factor, however being disconnected from the physical world while still being physically present within it can make users “blind” to the environment around them. According to a study from Purdue University done in Tippecanoe County (US), the popular AR game Pokemon Go has been linked to two deaths and has caused an increase in car accidents near “PokeStops” (real world locations with in-game components) since users are playing the game while driving or walking. <ref>Sharwood, Simon. (November 27th, 2019). Pokemon Go Caused Hundreds of Deaths, Increased Crashes. The Register. International. Retrieved 27-February-2019 from: <i>[https://www.theregister.co.uk/2017/11/27/pokemon_go_caused_car_accidents_and_deaths/ ]</i></ref> There is also anecdotal evidence of VR headset wearers forgetting their surroundings and tripping over physical objects within a room, falling out of chairs, and even hitting other people around them. Developers will need to consider elements of special awareness when designing solutions and how liability laws may affect them. </p><br />
<p>Another issue for AR and VR is that of interoperability between devices. For mobile devices, there are a number of platforms from which to access VR applications but not all platforms are available on all devices. iPhone and Android users are limited by what they are able to download from their respective application stores. AR headsets are not limited by their hardware like smartphones, but applications must live within a platform for them to function and there are many platforms to choose from on the market. An AR platform in this context is similar to an operating system for the headset, where developers can create solutions that run on the platform. </p><br />
<p>Lastly, standard AR and VR solutions may not be suited for those with sensory, mobility, dexterity, and learning disabilities. Individuals who may benefit from ACT would need to be properly assessed and outfitted with tools and technology that appropriately addresses their unique needs. A standard off-the-shelf corporate AR and VR solution may improve the ease of work and productivity of able-bodied employees, but it may not adequately address the needs of those requiring sensory, mobility, or learning aids. The AR or VR solution may require adaptation and modification to fit each end user, which will require additional resources and expertise in resolving.</p><br />
<br />
<h4>Considerations</h4><br />
<p>In the realm of AR and VR technology, the question of data ownership will be a very important issue to consider, given that data created from the use of AR and VR (user profiles, biometric data, sentiment response, eye tracking, etc.) presents a number of privacy and security concerns. If applications are created by outside companies, there is the potential for information to be collected by those providers and SSC will need to be very clear about who can access that information and for what purposes. Clarification will also be needed as to whether or not SSC can collect that data in the first place, SSC may need to anonymize all AR / VR data or simply not collect it. Moving forward, SSC has the potential to use AR and VR solutions but the Department needs to carefully look at the business problem that needs to be solved and if the technology is an appropriate choice to solve that problem.</p><br />
<p>The adoption of AR and VR systems present several challenges. Due to the current lack of low-cost hardware capable of running AR based applications, most AR solutions are designed as mobile apps. This is because a mobile device can track all the necessary information for a properly function application, but it comes with computational limitations since it is all contained within a small device. Location-based AR applications are currently the only type of AR application available for use on mobile phones. This is because Recognition-based and SLAM technology is much too computationally intensive for a smartphone. Since Location-based software requires only GPS it is a viable approach. As the need for more complex AR and VR applications grows so will the computational demands. Smartphones alone will not be able to handle these new complex demands. SSC will need to carefully consider project goals when deciding on whether the Department should create a VR or AR application, and on what kind of hardware it should run on. Scalability is another consideration for SSC, that is, the ability to grow a project’s infrastructure and capabilities to let more people use it at the same time. A project’s eventual growth in terms of scalability and deployment will need to be considered right at the start its development.</p><br />
<p>SSC should consider working through its Accessibility, Accommodation and Adaptive Computer Technology (AAACT) team to pilot AR and VR solutions to those with sensory, mobility, and learning needs. <ref>Shared Services Canada. (April 3rd, 2019). Accessibility, Accommodation and Adaptive Computer Technology (AAACT). Government of Canada. Aaact-aatia@ssc-spc.gc.ca. (819) 994-4835. Retrieved 03- May-2019 from: <i>[https://www.canada.ca/en/shared-services/corporate/aaact-program.html]</i></ref> SSC could engage with employees who have requested assistance from AAACT and collaborate with them to test various technologies in order to assess whether or not they reduce the barriers to work for those with impairments.</p><br />
<p>Given that the AR and VR markets are not yet mature and that more advancements are projected, SSC senior management should proceed with caution as it relates to the procurement of AR and VR. Additionally, caution is advised regarding the hype surrounding the promises of immersive technologies. What is depicted in popular films such as “Ready Player One” has already inflated the expectations of consumers and does not accurately reflect the business reality and utility of AR and VR today.</p><br />
<p>Given that these technologies generate a lot of data while they are being used, as the IT infrastructure provider for the GC, SSC should ensure that it is well equipped to handle the additional bandwidth usage of large scale deployments. </p><br />
<p>Since AR and VR are technologies that are still being refined, SSC will need to make investment/procurement decisions within a relatively short timeframe. Depending on the project scope, availability of funds, and availability of personnel the Department will need to consider how it will develop AR and VR solutions. These solutions will require specialized knowledge which is highly sought after in the tech world and hiring personnel could be a competitive endeavor. Taking this into account, having an in-house team for creating AR and VR applications may not be feasible unless there is a lot of demand from other departments. The Department at this time should consider contracting in the short term (the next 5-10 years), since the talent pool, technologies, and business use cases are not at an adequate level of maturity. </p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Blockchain";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends&diff=13689Technology Trends2019-11-14T03:08:51Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:{{FULLPAGENAME}}}}<br />
[[fr:Tendances_Technologiques]]<br />
<br />
<div class="sideTable"><br />
<div class="navigation-section"><br />
<div class="navigation-table"><br />
<table class="wikitable nav-tab" style="margin-top: 0px;"><br />
<tr><th class="breadcrumb" colspan="2"><b>Technology Trends</b><br><b>[[Tendances_Technologiques|Tendances Technologiques]]</b></th></tr><br />
<tr><td colspan="2" class="logo">[[File:Technology_brief_logo.png|200px]]</td></tr><br />
<tr><br />
<th>Trend Template</th><br />
<td>[[Media:EN_-_Technology_Trends_Template.docx|Technology Trends.docx]]</td><br />
</tr><br />
<tr><br />
<th>Other Related Publications</th><br />
<td><b>[[Architecture_Framework_Advisory_Committee|Architecture Framework Advisory Committee]]</b><hr/><b>[[Technology_Trends/User_Guide|User Guide]]</b></td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">Recent additions include [[Technology_Trends/Blockchain|Blockchain]], [[Technology_Trends/Drones|Drones]], [[Technology_Trends/Internet_of_Things|Internet of Things]], [[Technology_Trends/Low_Code_Application_Development|Low Code Application Development]], [[Technology_Trends/Open_by_Default|Open by Default]], [[Technology_Trends/Zero_Trust_Network|Zero Trust Network]].</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
</div><br />
</div><br />
<br />
<div><br />
<br />
<p>In an effort to better support its mandate of providing innovative Information Technology (IT) services to Government of Canada (GC) departments and agencies, Shared Services Canada (SSC) is actively researching and evaluating emerging technology trends. This research aims to Inform and prepare SSC, and the broader GC community, to handle new disruptive technologies that are just over the horizon by outlining potential benefits, impacts and use cases.</p><br />
<br />
<p>Technology trend briefing papers are the vehicle used to convey the research conducted on a given subject. These briefing papers are short, to the point, and are targeted towards both a business and technical oriented readership.</p><br />
<br />
<p>SSC is looking to the broader GC community to contribute and collaborate on existing and future topics. By fostering a more collaborative and inclusive development approach, we hope to reduce GC research duplication and accelerate the development of technology roadmaps across government by referencing common GC documentation.</p><br />
<br />
<h2>Format</h2><br />
<br />
<p>Each technology trend briefing paper follows a common template:</p><br />
<br />
<ul><br />
<li><b>Business brief </b>describes the business aspect of the technology by offering a high level understanding, the origin and the uses.</li><br />
<li><b>Technical brief </b>gives an in-depth description of the technology. It focuses on the most important technical aspects of the technology without going into too much detail.</li><br />
<li><b>Industry usage </b>offers an understanding of how the technology is currently used or thought to be used in the industry when commercially available. This section clarifies the benefits of the technology through examples.</li><br />
<li><b>Canadian Government use </b>details how the technology is currently used or thought to be used in the context of the Federal Canadian Government when commercially available. This intent is to provide an understanding of how the technology could ultimately benefit the federal government in regards to the delivery of services to Canadians through short examples.</li><br />
<li><b>Implications for Shared Services Canada (SSC) </b>demonstrates the clear and measurable benefits that pertains to Shared Service Canada when adopting the use of the technology. This section is comprised of these following subsections :<br />
<ul><br />
<li><b>Value proposition </b>illustrates the potential strengths and opportunities that SSC can encounter by leveraging the technology.</li><br />
<li><b>Challenges </b>outlines the risks and difficulties associated with the technology that includes both risks of implementation such as the development of the product or the risks of solution by adopting the technology.</li><br />
<li><b>Considerations </b>presents any recommendations SSC may have for another department of the Government of Canada that wishes to adopt and leverage the technology.</li><br />
</ul><br />
</li><br />
</ul><br />
<br />
<h2>Development Process</h2><br />
<br />
<p>External market trends, private-public sector reports, online trends and Government of Canada strategic plans are all factors considered when selecting a topic for development. Once a topic is selected for development, it undergoes a five phase development process prior to publishing.</p><br />
<br />
<div>[[File:Technology_Trends_-_Development_Process.png|center]]</div><br />
<br />
<h2>How to Contribute</h2><br />
<br />
<p>We want your help! Our goal is to ensure that the content generated is developed in the open, routinely kept up to date, representative of the wide ranging GC departmental mandates and widely distributed to the broader Canadian Government community. To that end, we strongly encourage you to share, edit and provide feedback on the published content.</p><br />
<br />
<p>You can edit by making use of the “Discussion” tab located at the top-left of the page.</p> <br />
<br />
<p>Should you wish to sponsor and or author a topic not listed below, please reach out to [mailto:jean-francois.michaud@canada.ca Jean-François Michaud]</p><br />
<br />
</div><br />
<br />
<div class="container"><br />
<div class="row"><br />
<div class="col-sm-12"><br />
<h2>Technology Trend Briefing Papers</h2><br />
<br />
<table class="wikitable sortable technology-table" style="border: 0px solid;"><br />
<tr><br />
<th colspan="3">Name</th><br />
<th>Status</th><br />
<th style="white-space: nowrap;">Release Date</th><br />
<th>Official Publications</th><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Blockchain|Blockchain]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Chaîne_de_Blocs|Chaîne de Blocs]]</td><br />
<td>Published</td><br />
<td>2019-05-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Blockchain.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Blockchain.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Drones|Drones]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Drones|Drones]]</td><br />
<td>Published</td><br />
<td>2019-07-04</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Drones.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Drones.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Internet_of_Things|Internet of Things]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Internet_des_Objets|Internet des Objets]]</td><br />
<td>Published</td><br />
<td>2019-07-04</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Internet_of_Things.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Internet_des_Objets.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Low_Code_Application_Development|Low Code Application Development]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Développement_d%27Application_LowCode|Développement d'Application Programmation Schématisée]]</td><br />
<td>Published</td><br />
<td>2019-05-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Low_Code_Application_Development.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Développement_d%27Application_LowCode.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Open_by_Default|Open by Default]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Ouvert_par_Défaut|Ouvert par Défaut]]</td><br />
<td>Published</td><br />
<td>2019-05-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Open_by_Default.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Ouvert_par_Défaut.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Zero_Trust_Network|Zero Trust Network]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Réseaux_Zéro_Confiance|Réseaux Zéro Confiance]]</td><br />
<td>Published</td><br />
<td>2019-05-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Zero_Trust_Network.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Réseaux_Zéro_Confiance.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/5G_Networks|5G Networks]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Réseaux_5G|Réseaux 5G]]</td><br />
<td>Published</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_5G_Networks.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Réseaux_5G.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Big_Data|Big Data]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Mégadonnées|Mégadonnées]]</td><br />
<td>Published</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Big_Data.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Mégadonnées.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Edge_Computing|Edge Computing (Serverless Computing)]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Informatique_en_Périphérie|Informatique en Périphérie (Informatique sans Serveurs)]]</td><br />
<td>Published</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Edge_Computing.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Informatique_en_Périphérie.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Face_Recognition|Face Recognition]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Reconnaissance_Faciale|Reconnaissance Faciale]]</td><br />
<td>Published</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Face_Recognition.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Reconnaissance_Faciale.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Datalakes|Datalakes]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Lac de Données</td><br />
<td>Translation</td><br />
<td>2019-08-06</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Datalakes.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Digital_Twins|Digital Twins]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Jumeaux Numériques</td><br />
<td>Translation</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Digital_Twins.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Kubernetes|Kubernetes]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Kubernetes</td><br />
<td>Translation</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Kubernetes.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Data_Leak_Prevention|Data Leak Prevention]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Prévention des Fuites de Données</td><br />
<td>Translation</td><br />
<td>2019-08-06</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Data_Leak_Prevention.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Cloud_Management_Platform|Cloud Management Platform]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Platforme de Gestion d'Infonuagique</td><br />
<td>Translation</td><br />
<td>2019-09-18</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Cloud_Management_Platform.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Digital_Assistants|Digital Assistants (Conversational UI)]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Assistants Numériques (Interface Conversationnelle)</td><br />
<td>Translation</td><br />
<td>2019-09-25</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Digital_Assistants.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Enterprise_Application_Stores|Enterprise Application Stores (API Stores)]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Magasins d'Applications d'Entreprise (Magasins d'API)</td><br />
<td>Translation</td><br />
<td>2019-10-30</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Enterprise_Application_Stores.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Software_Defined_Anything|Software Defined Anything]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Tout Défini par Logiciel</td><br />
<td>Translation</td><br />
<td>2019-10-30</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Software_Defined_Anything.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Digital Services</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Services Numériques</td><br />
<td>Development</td><br />
<td>2019-07-23</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Interoperability (Tell Us Once)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Interopérabilité</td><br />
<td>Development</td><br />
<td>2019-07-23</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">3D XPoint (Optane)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">3D XPoint (Optane)</td><br />
<td>Development</td><br />
<td>FY 19-20 Q2</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Application_Containers|Application Containers]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Conteneurs Logiciels</td><br />
<td>Development</td><br />
<td>FY 19-20 Q2</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">AI (Machine Learning)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">IA (Apprentissage Automatique)</td><br />
<td>Development</td><br />
<td>FY 19-20 Q2</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Crowdsourcing</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Production participative</td><br />
<td>Development</td><br />
<td>FY 19-20 Q2</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Autonomous Vehicles</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Véhicules Autonomes</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Infrastructure as Code</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Infrastructure en tant que Code</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">SCADA Systems</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Systèmes SCADA</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Cloud Workplace (Office)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Bureau Infonuagique</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Low Earth Orbit System</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Système en Orbite Terrestre Basse</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Quantum Computing</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Informatique Quantique</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Robotics</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Robotiques</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Data Centre Infrastructure Monitoring</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Surveillance de l'Infrastructure du Centre de Données</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Digitial / Smart Offices</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Bureaux Digitales / Smart</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Enterprise Collaboration</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Collaboration d'Entreprise</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Gamification (Innovation Framework)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Gamification (Cadre d'Innovation)</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Intent Based Networking</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Réseaux basé sur l'Intention</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">IT Talent</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Talent Informatique</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Pay as you go</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Pay as you go</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Li-Fi|Li-Fi]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Li-Fi</td><br />
<td>Unassigned</td><br />
<td></td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Augmented_and_Virtual_Reality|Augmented and Virtual Reality]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Li-Fi</td><br />
<td>Unassigned</td><br />
<td></td><br />
<td></td><br />
</tr><br />
<br />
</table><br />
</div><br />
</div><br />
</div><br />
<br />
[[Category:Architecture]]<br />
[[Category:Technology]]<br />
<br />
{{#css:<br />
.technology-table{<br />
width: 100%;<br />
margin: auto;<br />
}<br />
<br />
th, td{<br />
text-align: center;<br />
}<br />
<br />
.logo{<br />
text-align: center;<br />
}<br />
<br />
.disclaimer{<br />
border: 1px solid red;<br />
background-color: rgba(255, 0, 0, 0.3);<br />
}<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
padding: 5px 10px 5px 10px;<br />
}<br />
<br />
.disclaimerText{<br />
width: 100%;<br />
text-align: left;<br />
}<br />
<br />
.news{<br />
border: 1px solid green;<br />
background-color: rgba(0, 255, 0, 0.3);<br />
}<br />
<br />
.toggle{<br />
color: rgb(6, 69, 173);<br />
float:left;<br />
<br />
}<br />
<br />
.toggle:hover{<br />
text-decoration: underline;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { float: left; }<br />
<br />
.col-sm-1 { width: 8.333333333%; }<br />
.col-sm-2 { width: 16.66666666%%; }<br />
.col-sm-3 { width: 25%; }<br />
.col-sm-4 { width: 33.333333333333%; }<br />
.col-sm-5 { width: 41.6666666666%; }<br />
.col-sm-6 { width: 50%; }<br />
.col-sm-7 { width: 58.333333333%; }<br />
.col-sm-8 { width: 66.66666666666%; }<br />
.col-sm-9 { width: 75%; }<br />
.col-sm-10 { width: 83.33333333333%; }<br />
.col-sm-11 { width: 91.6666666666%; }<br />
.col-sm-12 { width: 100%; }<br />
<br />
.sideTable {<br />
width: 27.5%;<br />
float: right;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 95%; <br />
margin-left: 2.5%; <br />
margin-right: 2.5%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
}<br />
<br />
.inverse-direction {<br />
float: right;<br />
}<br />
<br />
.td-left { border-right: 0px solid; }<br />
.td-middle { border-right: 0px solid; border-left: 0px solid; }<br />
.td-right { border-left: 0px solid; }<br />
<br />
.nav-tab {<br />
width: 100%;<br />
margin-top: 0px;<br />
padding-top: 0px;<br />
}<br />
<br />
<br />
@media (max-width: 1050px){<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.sideTable {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%; <br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%; <br />
margin-right: 0%;<br />
}<br />
<br />
/*<br />
.navigation-table {<br />
width: 30%;<br />
margin-left: 1.6666666%; <br />
margin-right: 1.6666666%;<br />
float: left;<br />
}<br />
*/<br />
<br />
.content-section {<br />
float: left;<br />
}<br />
<br />
.inverse-direction {<br />
clear: both;<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { clear: both; width: 100%; margin: auto; }<br />
}<br />
<br />
div>a>img {<br />
width: 600px;<br />
height: auto;<br />
margin: auto;<br />
}<br />
<br />
@media (max-width: 850px){<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%; <br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
margin-left: 0%; <br />
margin-right: 0%;<br />
clear: both;<br />
}<br />
}<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Application_Containers&diff=13688Technology Trends/Application Containers2019-11-14T02:59:37Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Application Containers</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques/|Tendances_Technologiques/]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[File:|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Published</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 23, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>May 23, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:Application_Containers.pdf|Application Containers.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Application Containers</b> are a form of OS-Virtualization in which distributed applications are run on a single virtual machine or physical host rather than one for each application</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>A single host OS can support multiple containers. Containers can be used on multiple systems including cloud instances and virtual machines, as well as across different OS such as Linux, Microsoft, and MAC OS. Application containers have huge benefits with their ability to be deployed in cloud environments. The main advantage to containers is their architecture. Because applications can now be placed on different virtual and physical machines this offers greater availability of those applications. These machines can be within a cloud or not. They offer high flexibility with regard to workload management and allow the developer to make vault-tolerant systems. </p><br />
<p>Application containers transition data centres from being machine-oriented to application-oriented. Containers encapsulate the application environment, abstracting details from the machine, operating system, the application developer, and deployment infrastructure. Because well-designed containers and container images are scoped to a single application, managing containers means managing applications rather than machines. This shift with management application programming interfaces (API) from machine-oriented to application-oriented dramatically improves application deployment.</p><br />
<h2>Technology Brief</h2><br />
<br />
<p>It is useful to look at the concept of containers as images when trying to understand how containers work. Images can be stacked hierarchically. Applications can be attached to this hierarchy and each node of the tree allows the user to share images among applications. This allows for the configuration of binary states, meaning that you do not have to shift around the entire application’s stack in single files. The relationship between images and containers is analogous to that of a class and object. In object oriented programming an object is an instance of a class. For example, an apple would be an object of class fruit. The class defines the data stored about the object and what functions can be performed on it. With application containers, the image, like a class, provides the blueprint to the container. The container is like an instance or object of this class. </p><br />
<p>Another advantage of this is when a vulnerability is discovered within a particular node, the hierarchical tree allows the user to verify all other nodes that are impacted by this vulnerability. In order to create images, it is important to look at an example. Docker is a container engine. Docker uses a Docker file to create images. Containers are used to isolate processes within the OS. </p><br />
<p>Application containers can be thought of as a form of virtualization similar to Virtual Machines (VM). However instead of virtualizing the entire stack, containers virtualize at an operating system level <ref>What are Containers and their benefits | Google Cloud. (n.d.). Retrieved from <i>[https://cloud.google.com/containers/.]</i></ref>. Because of this they are extremely lightweight, enabling them to use only a fraction of the memory and processing as only a single OS Kernel is needed to host multiple containers.</p><br />
<p class="expand mw-collapsible-content">Application containers facilitate a service oriented architecture (SOA). In an SOA the application is decomposed in into several services. With containers, individual services can be launched as different containers. With clustering this enables the ability to scale up a solution. The proper orchestration tools are required to properly enable scaling. Container orchestration tools manage the lifecycle of containers, a software development team will use them to provision and deploy containers, check redundancy and availability, and to move a container from one host to another. </p><br />
<p class="expand mw-collapsible-content">Monolithic applications can also be deployed as containers. As a monolithic container is essentially a single–tiered application a single instance will be deployed as a single container. This differs from the microservice and SOA approach as a container will represent a single service of that application. With containers, in order to scale up a monolithic application multiple instances of the application and thus several containers need to be deployed. When using Docker, updates to the application can be deployed as docker images which is faster and network efficient. Also since containers can be spun up faster, creating new application instances a quicker process.</p><br />
<p>Two major vendors of application container technology are Docker and CoreOS. Docker was introduced first, and as a result is used more widely in the industry. Application containers are of most benefit to large companies. It has been four years since Docker’s initial foray into the market, and a survey conducted by Datadog in 2016 revealed that adoption of Docker had increased 30%+ over the previous year. What is important to note about this statistic is that these companies are using the software in production rather than development, which suggests that companies are using containers as a long-term approach and not simply on a trial basis</p><br />
<p>Docker can be characterized as the container management engine which creates the containers. There are also several management platforms that give users a number of different tools to manage their containers. Three of these tools are OpenShift, Kubernetes, and Rancher. </p><br />
<p>OpenShift uses Docker and Kubernetes as its underlying engine to manage the containers. The software integrates with Kubernetes to enable the automation of functions such as scaling, health management, and deployment. Kubernetes, which was created by Google, is a well-designed orchestration tool used in the management of containers. It offers the client a high level API that allows the user to logically group containers as well as load balancing and container pools. Rancher is a container management tool built on top of the Kubernetes container engine platform.</p><br />
<br />
<h2>Industry Usage</h2><br />
<br />
<p>Several Large organizations have remarked the benefits of containers within their development lifecycle. ADP is a large vendor of cloud-based human capital management solutions. The company has 630,000 clients with around 35 million users of their systems. They began using application containers to speed up their development lifecycle. As of April 2017, ADP had 469 Docker engines, with 3771 containers running on virtual machines (VM). Paypal is another major user of application containers. The company hosts the largest online payment systems in the world. They have over 210 million users, in 200 international markets, and handle close to $100 billion in transactions per quarter. Paypal used Docker to perform datacenter modernization. In 2016, they launched their first container in QA and production. Since their applications were now decoupled form the operating system they were running the company could now modernize their infrastructure without needed to update the application. With more modern and faster infrastructure they remarked a 10-20% increase in efficiency with no downtime. Paypal now has 700 applications migrated to Docker and has in production 45000 VM hosts with over 150,000 containers running on them. </p><br />
<br />
<h2>Canadian Government Use</h2><br />
<p>There is a lack of documented Government of Canada (GC) initiatives and programs promoting the current and future use of application containerization and application container technology. As a GC strategic IT item, application containers is absent from both the GC’s Digital Operations Strategic Plan: 2018-2022 and the GC Strategic Plan for Information Management and Information Technology 2017 to 2021. This may be due to the fact that the SSC is currently grappling with consolidation and digital transformations in parallel to the implementation of Cloud Services. The majority of resources and efforts are occupied with implementation challenges, as well as security concerns related to the protection of the information of Canadians.</p><br />
<p>However, the inception of containers into the market has shown that large-scale organizations, who are involved in cloud-native application development as well as networking, can benefit greatly from the use of containers <ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes. <i>[https://www.cengn.ca/docker-kubernetes-training-jan18/]</i></ref>. Although the infrastructure applications providing cloud services can be based solely on Virtual Machines (VMs), the maintenance costs associated with running different operating systems on individual VMs outweighs the benefit<ref>Heron, P. (2018). Experimenting with containerised infrastructure for GOV.UK - Inside GOV.UK. [online] Insidegovuk.blog.gov.uk. Available at: <i>[https://insidegovuk.blog.gov.uk/2017/09/15/experimenting-with-containerised-infrastructure-for-gov-uk/]</i></ref>. Containers and Containerization is a replacement and/or complimentary architecture for VMs. As the GC moves toward cloud services and development of cloud-native applications, the use of containers and orchestrating them with Kubernetes can become an integral part the GC IT architecture. </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<br />
<h4>Value Proposition</h4><br />
<p>Application containers have a multitude of uses and SSC can benefit from such a flexible tool. Application containers especially shine if they are used for developing and testing new applications, since they can be easily deployed to create a safe testing environment (like a sandbox) for running code. Containers can be easily controlled from a container file, and can be scaled to fit any resource requirements. Since they are so easily customized, application containers are often used within the Services Oriented Architecture (SOA) software development framework to house microservices. An application developed under the SOA framework is usually made up of many small pieces (microservices) that communicate and work together. <ref>Abeywardhana, Sajith. (February 9th, 2019). SOA vs Microservices with Docker and Kubernetes-1. Medium. Retrieved 18-July-2019 <i>[https://medium.com/@sajithswa/soa-vs-microservices-with-docker-and-kubernetes-1-291686200a0f ]</i></ref> Should SSC adopt this or other similar software development models (like Microservices Architecture), application containers will be a necessary development tool. </p><br />
<p class="inline">Containers can facilitate easier and more efficient deployment of applications. There is the potential for increased efficiencies since containers create a smaller footprint from a memory perspective on a physical host. Since multiple containers can be created for a single application (like in SOA), in practical terms this means that version control is simplified. Instead of patching an application’s code, a container housing a small portion of the application can be easily pulled out and updated without disrupting the rest of the application. </p><br />
<p class="expand mw-collapsible-content">If an orchestration tool is used to manage all of an applications containers, there is less opportunity for disruption since the orchestrator can monitor the “health” of all of its containers <ref>Wagner, Bill, et al. (June 1st, 2019). Health Monitoring - .NET Microservices: Architecture for Containerized .NET Applications. Microsoft Developer Division. Redmon, Washington. Retrieved 19-July-2019. <i>[https://docs.microsoft.com/en-us/dotnet/standard/microservices-architecture/implement-resilient-applications/monitor-app-health ]</i></ref>. If the orchestrator determines that a container is malfunctioning, it can shut it down automatically and launch other containers to pick up where it left off. </p><br />
<p class="inline">In addition containers are portable, meaning that they will run whatever program they contain in the exact same way every time it is launched, and wherever it is launched. In other words applications will run consistently in a standardized environment, even if the container is launched from the cloud or from a physical server. This practically eliminates the need to do lengthy custom installations on every single machine in a network, an application can simply be run from a container. Portability will also help with running legacy applications, in the sense that if an older app needs a specific environment for it to run (for example, older versions of Java or Python), that environment can be recreated in a container.</p><br />
<p class="expand mw-collapsible-content">However, there are some limitations still present, containers from one platform (like Docker) can’t be run on another platform (like Linux Containers) and there are also backwards compatibility issues, containers designed for the newest version of a platform may not work on older versions. <ref>Tozzi, Chris. (2017). The pros and cos of container platforms for portability. Search Microservices. Retrieved 22-July-2019 <i>[https://searchmicroservices.techtarget.com/tip/The-pros-and-cons-of-container-platforms-for-portability ]</i></ref> </p><br />
<h4>Challenges</h4><br />
<p>There are four major challenges with the adoption of application containers:</p><br />
<p><b>Staff education and training </b>-- The proper utilization of containers requires a whole new skill set. It also requires IT teams to be familiar with the container platform and container orchestration tools they will be using. In addition, IT teams may need to learn new design standards (like SOA) to reap the full benefit of application containers. </p><br />
<p><b>Choosing the proper container platform and orchestration program</b>-- This is a crucial decision which will depend on the needs of the department. Depending on the Operating System in which the containers will operate, there will be different tools available. Although the GC is looking to adopt open source tools, there is no shortage of options, each with their own pros and cons. </p><br />
<p><b>Re-designing applications </b>– Legacy applications developed under a monolithic model may not always be suitable for containers. Containers may run into trouble when developers attempt to update a single feature of the older application, but this holds true for monolithic applications in general. Monolithic applications can instead be broken into smaller pieces and container-ized, so they can continue to operate on newer systems.</p><br />
<p><b>Security</b>-- Security is crucial since the containers can be based on a combination of cloud and local networks. The key factor for security is designing a solution that grants its users freedom and speed but is not vulnerable to data breaches. </p><br />
<h4>Considerations</h4><br />
<p>In general, containers still have their own limitations (just like any technology) and should not be adopted for the sake of innovation. Application containers come with their own unique requirements and challenges, like building skill sets and choosing the right software suites. If SSC were to move in a direction where application containers are a necessity, then overcoming those challenges and related costs will be a worthwhile investment.</p><br />
<p>In terms of security, applications running within a container should not be assumed to be secure. Security should be a consideration throughout the entire lifecycle of an application and containers should be no exception. If SSC were to adopt application containers, different security processes would need to be developed to handle the new technology. </p><br />
<p class="expand mw-collapsible-content">In the case that SSC develops applications under the SOA framework or others like it, there will need to be a shift in how these applications are monitored. Rather than monitoring the machines on which they are operating, the containers themselves would need to be monitored for performance. Key Performance Indicators (KPIs) would need to be collected from the orchestration tool that controls an application’s containers.</p><br />
<p class="expand mw-collapsible-content">As a tool, application containers present many potential benefits to SSC if they are to be properly managed and integrated into the workplace.</p><br />
<br />
<h2>References</h2><br />
<ref>6 Best Practices for Creating a Container Platform Strategy. (n.d.). Retrieved from <i>[https://www.gartner.com/smarterwithgartner/6-best-practices-for-creating-a-container-platform-strategy/.]</i></ref><br />
<ref>Harvey, C., Proch, D., Davis, J., Froehlich, A., MacVittie, L., Wallace, B., … Windels, J. (2019, January 10). 5 Container Deployment Challenges. Retrieved from <i>[https://www.networkcomputing.com/data-centers/5-container-deployment-challenges/1532194322.]</i></ref><br />
<ref>Rouse, M., McKenzie, C., Rouse, M., & Rouse, M. (n.d.). What is application containerization (app containerization)? - Definition from WhatIs.com. Retrieved from <i>[https://searchitoperations.techtarget.com/definition/application-containerization-app-containerization.]</i></ref><br />
<ref>Run Kubernetes Everywhere. (n.d.). Retrieved from <i>[https://rancher.com/]</i></ref><br />
<ref>Team, S.-S., & Written by Shadow-Soft Team View more posts written by this author ». (2018, February 8). 3 Open Source Container Management Tools (Comparison & Review). Retrieved from<i>[https://shadow-soft.com/open-source-container-management-tools/]</i></ref><br />
<ref>Who's Using Docker? (2017, August 16). Retrieved from <i>[https://www.contino.io/insights/whos-using-docker]</i></ref><br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Blockchain";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Application_Containers&diff=13687Technology Trends/Application Containers2019-11-14T02:58:24Z<p>Kpere060: Created page with "{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}} fr:Tendances_Technologiques/Chaîne_d..."</p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Augmented and Virtual Reality</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques/|Tendances_Technologiques/]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[File:|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Published</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 23, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>May 23, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:AR_VR.pdf|Augmented and Virtual Reality.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Augmented reality (AR)</b> provides a live direct, or indirect, view of the real-world with aspects that are modified or “augmented” by aid of a computer.</p><br />
<br><p><b>Virtual Reality (VR)</b> on the other hand, occludes (obstructs) the entire field of view of the user (i.e. a headset and other sensory gear), and replaces the entire surrounding environment with a completely computer-generated one.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>Augmented reality (AR) provides a live direct, or indirect, view of the real-world with aspects that are modified or “augmented” by aid of a computer. These computer-generated modifications can be applied across multiple sensory modes, including visual, audio, haptic (touch), and olfactory (smell). AR technology provides an enhancement to the surrounding environment and can be either constructive (add elements to the environment) or destructive (mask elements through localized occlusion without blocking the entire real world view of the user).</p><br />
<p>AR and VR concepts are found on the virtuality continuum, otherwise known as the mixed reality spectrum (the spectrum). One end of the spectrum is unmodified reality and the opposite end is digital reality, with anything in between falling under Mixed Reality (MR). The spectrum includes all possible variations and combinations of real and virtual objects, such as augmented virtuality, where elements of reality augment the virtual world. Virtual and augmented technologies are essentially interactive mediums that enable the combination of the virtual and real world where physical and digital objects may co-exist in real time.</p><br />
<p class="inline">Devices used to overlay digital elements onto reality (AR) include, AR enabled smartphones, tablets, and see-through hands-free head mounted displays (HMD). Most consumer AR applications are created for smartphones due to their proliferation and availability. At this point, HMDs do not have many consumer AR applications due to their prohibitive price point, however, many business use cases have been developed for professional use. This is due to the fact that the hands free nature of the HMD devices allows for more creativity in the development of applications, where other smaller but necessary sensors and cameras can be used in these purpose built devices. For VR, HMDs completely obstruct the user’s field of view and replaces it with a digital environment are usually paired with any combination of controllers and wearable body trackers that let the user physically interact with the virtual world.</p> <p class="expand mw-collapsible-content"> Another concept for enabling VR is the Cave Automatic Virtual Environment (CAVE) where a user is completely immersed within a physical room with walls made up of rear-projection screens (or flat panel screens) that displays a virtual environment that can be interacted with via controller or camera sensors. HMDs are used to immerse the senses in VR, whereas the CAVE system is used to immerse the whole person within a virtual space. </p><br />
<br />
<h2>Technology Brief</h2><br />
<p>AR and VR make use of several different technologies. Both use sensors to gain a mapping of the surrounding environment, however VR technology also makes use of sensors outside a headset to provide a more immersive experience. VR headsets like the HTC Vive and Oculus Rift track different body movement in order to allow the user to be able to look around and interact with virtual items. The image processing needs to be done at around a 60 frames per second for the user to be able to move their head naturally. Several companies currently working on VR technology are now also attempting to track eye movements within their headsets. This will allow the images displayed on screen to appear even more realistic as the image will automatically focus based on where the user is looking. </p><br />
<p>Currently on the market there are three categories of VR headsets available:</p><br />
<ol><br />
<li><p><b>Headsets for Gaming Consoles and PC:</b>this grouping of headsets relies on the extra computing power of gaming consoles and personal computers. Since they are tethered to a computer system, more peripherals can also be connected and integrated into the VR experience like handheld controllers, motion trackers, eye tracking glasses, glove controllers and treadmills.</p></li><br />
<li><p><b>Standalone Headsets:</b>these are wireless headsets that do not need to be plugged into a computer or smartphone to work. All of the necessary technology is packed into the device; the computing hardware, motion tracking, controller connectivity, and the operating system itself.</p></li><br />
<li><p><b>Mobile Headsets:</b>these headsets are powered by smartphones that are VR enabled, where the smartphone acts as the screen. These are essentially just headgear that keep the phone in place on the wearer’s head and they optimize the wearer’s field of view for VR. Some may add more tracking capabilities to the smartphone and be made of high quality materials, and others like Google Cardboard are the bare essentials to make the concept work. </p></li><br />
</ol><br />
<p>AR technology deploys a virtual image over real-world objects. The system receives input from the camera or other input devices. There are generally three approaches which AR technology makes use of. Simultaneous Localization and Mapping (SLAM) is a set of algorithms tasked with solving complex localization and mapping problems. This technology localizes sensors with respect to their surroundings. It uses this data to map the structure of the immediate environment and allows applications to incorporate real world information into the graphics generated on screen (i.e. the Ikea Place app that generates proportional examples of furniture onto a user’s living space). </p><br />
<p>Recognition based or marker based AR is another approach to the design of AR systems. The camera of the device identifies visual markers using either, QR/2D code or natural feature tracking markers. When the device senses a marker it places the virtual image at that position. It uses these markers to determine orientation and position of the marker image.</p><br />
<p>Lastly, location based AR is another technology that utilizes GPS, digital compass, velocity meter, and accelerometer as inputs to determine orientation and position of the virtual image. This is a popular approach to AR technology as smartphones already have the necessary infrastructure to provide all these inputs. Simply put, there are two ways in which augmented reality can be viewed: </p><br />
<ol><br />
<li><p><b>Augmented Reality Glasses: </b>all of the processing power and projecting technology is packed into the glasses. 3D augmentations are superimposed into the wearer’s direct line of sight (i.e. Google Glass). </p></li><br />
<li><p><b>Smartphones: </b>the user must view reality through the screen of their phone where the computer graphics are then applied. </p></li><br />
</ol><br />
<h2>Industry Usage</h2><br />
<br />
<p>AR and VR technology has a wide variety of applications from both commercial and business standpoints and more possibilities are being explored constantly. The global AR market is expected to grow to $60.55 billion (USD) <ref>Markets and Markets. (July, 2017). Augmented Reality Market by Offering, Device Type, Application, and Geography. MarketsandMarkets Research Private Ltd. Retrieved on 13-February-2019 from: <i>[www.marketsandmarkets.com/Market-Reports/augmented-reality-market-82758548.html?gclid=EAIaIQobChMI_J60r6_Q4AIVCZ7ACh3BuQ4tEAAYASAAEgLvq_D_BwE. ]</i></ref> by 2023 and the VR market is expected to be $44.7 billion by 2024<ref> Markets and Markets. (January, 2019). Virtual Reality Market by Offering, Device Type, Application, and Geography. MarketsandMarkets Research Private Ltd. Retrieved on 13-February-2019 from: <i>[www.marketsandmarkets.com/Market-Reports/augmented-reality-market-82758548.html?gclid=EAIaIQobChMI_J60r6_Q4AIVCZ7ACh3BuQ4tEAAYASAAEgLvq_D_BwE. ]</i></ref> . Within the workplace, several companies are experimenting with virtual workstations. The Virtual Desktop app capable of being used on the Oculus Rift and HTC Vive is designed to provide users with a virtual Windows Desktop that can be physically interacted with in virtual reality. <ref>Heaney, David. (November 29th, 2018). Virtual Desktop (Mobile) Review: The Best Way to Access Your PC From Occulus Go. UploadVR. San Francisco, USA. Retrieved on 13-February-2019 from: <i>[https://uploadvr.com/virtual-desktop-oculus-go-review/ ]</i></ref> On the AR side, a company called Meta is testing the use of an interactive holographic workstation to replace desktop computers. Users can instead use a 3D environment to store, display, and interact with virtual objects and other documents. <ref> Wang, Selina. (July 18th, 2017). This Startup Wants to Replace Your Office With 3d Holograms. Bloomberg L.P. New York, USA. Retrieved on 11-February-2019 from: <i>[https://www.bloomberg.com/news/articles/2017-07-18/this-startup-wants-to-replace-your-office-with-3d-holograms ]</i></ref> </p><br />
<p>For general use, Google Translate running on a smartphone can be used in AR mode to translate speech and text. When using the camera on a smartphone the Google Translate application can process an image with text written in one language and translate it to another. Another sensory modality of AR is provided by Google in the form of natural language translation. A recent update has enabled all Google Assistant optimized headphones and Android phones with the ability to do live audio translations. <ref>Google. (2019). What are Google Pixel Buds?. Google. Retrieved 11-February-2019 from: <i>[https://support.google.com/googlepixelbuds/answer/7545575?hl=en&ref_topic=7544331 ]</i></ref> </p><br />
<p>The process of prototyping is undergoing a major shift towards using AR and VR to explore and test new prototypes in a cost-effective way. The benefit of these technologies is that it allows all interested parties (engineers, assembly line workers, customers, management, marketers, safety regulators, etc.) of a project to come together and interact with a model before it goes into production. For example, the tech firm Finger Food created an AR solution for the truck manufacturer Paccar, which could be seamlessly integrated into their design workflow. <ref>Finger Food. (2019). Paccar: Fundamentally changing big industry with big technology. Finger Food Studios. Port Coquitlam, British Columbia. Retrieved 11-February-2019 from: <i>[https://www.fingerfoodstudios.com/case-study/paccar/ ]</i></ref> It allows the user to view and interact with full scale models of trucks and to test various physics simulations on the models in real time. Another example is the global architecture firm IA (Interior Architecture) who is using InsiteVR to build models of their designs. <ref>InsiteVR. (2019). VR Meetings for Architecture, Engineering, and Construction. InsiteVR. 2019. Retrieved 13-February-2019 from: <i>[https://www.insitevr.com/ ]</i></ref> This also allows clients of the company to “tour” the designs and give input before designs are finalized. Ford is another company using VR technology to help its engineers when designing elements of their vehicles. Audi has taken this technology to the consumer allowing customers to view, configure, and customize certain elements of their vehicles. </p><br />
<p>Education and training are also starting to change thanks to the adoption of AR and VR. Through VR, trainees can be immersed in a training environment where they can get hands-on practice without the possibility of real life mistakes. The aviation industry has been using simulation technologies for decades, but now with the relative ease of access to those technologies more sectors are adopting AR and VR for training purposes. For example, the forklift training company certify.me created a VR app for certifying new forklift operators where trainees are presented with a variety of virtual scenarios that demonstrate what to do and what not to do. <ref>CertifyMe.net (September, 2016) Virtual Reality Training: New Forklift Training Video in VR Certifyme.net. Retrieved 15-February-2019 from: <i>[https://www.certifyme.net/virtual-reality-training/ ]</i></ref> NASA has a mixed reality space where astronauts in training are immersed in a VR simulation while also being in a 1:1 physical space where they can walk around and touch real objects presented to them in the VR environment. <ref> Kolodny, Lora. (2017). Nasa is using a mixed reality space station to train astronauts. TechCrunch. Retrieved 15-February-2019 from: <i>[https://techcrunch.com/2017/03/26/nasa-is-using-a-mixed-reality-space-station-to-train-astronauts/ ]</i></ref> </p><br />
<p class="inline">Repair and construction work have also been augmented. Recently, some AR solutions have been created to help workers make effective repairs. For example, Boeing engineers are now equipped with glasses that overlay instructions and the proper placement of wires over a physical object that is being repaired. <ref>Upskill. (2019) Upskill and Boeing: Reinventing aerospace manufacturing and supply chain management. Upskill. Retrieved 15-February-2019 from: <i>[https://upskill.io/landing/upskill-and-boeing/]</i></ref> The glasses are also equipped with a camera that lets a supervisor view the work being done and give live feedback. The adoption of AR has reduced production time for Boeing by 25% and lowered error rates to practically zero. The construction industry is also adopting a mixed reality approach for completing construction projects. A recent development is an augmented reality app for construction workers that super-imposes the 3D mockup of a construction project over the real environment being built. <ref>Lorek, Sarah. (January 30th, 2018) Mixed Reality in Construction Gets Real with Trimble’s Hololens Hard Hat. Constructible. Retrieved 13-February-2019 from: <i>[https://constructible.trimble.com/construction-industry/mixed-reality-in-construction-gets-real-with-trimble-hololens-hard-hat-video ]</i></ref> This has helped with accurately finishing projects with fewer interpretation errors, as opposed to the traditional method of comparing the worksite to a two dimensional architectural plan.</p> <p class="expand mw-collapsible-content"> Augmented reality is also well suited for the itemization of tasks, in the sense that all of the steps of a process or a task are neatly overlaid on a wearer’s field of view, until the item is completed properly. For example, AR has been used to help workers navigate large and complicated warehouse layouts to locate packages they need to find to complete an order. <ref>Porter, Michael, et al. (November-December 2017) A Manager’s guide to Augmented Reality. Harvard Business Publishing. Retrieved 11-February-2019 from: <i>[https://hbr.org/2017/11/a-managers-guide-to-augmented-reality ]</i></ref> </p><br />
<p class="inline">AR and VR are providing marketers with novel spaces and new formats for creating campaigns. Many business have developed AR solutions that let consumers “test drive” products before they are purchased, like trying on clothes, glasses, and hairstyles. Ikea, for example created the Ikea Place app that lets users see how a virtual piece of furniture would look inside their home at full scale. <ref> Dasey, Daniel. Try Before you Buy: Ikea Place Augmented Reality. Ikea. Retrieved 15-February-2019 from: <i>[https://highlights.ikea.com/2017/ikea-place/ ]</i></ref> There is also a shift towards making advertisements more interactive and immersive as they will leave a longer lasting impression on the audience, as opposed to traditional advertising efforts.</p><p class="expand mw-collapsible-content"> To promote the horror franchise “Saw” a VR mini-game was created where users needed solve a puzzle within an interactive room. <ref>D’Angleo, Matt. (December 5th, 2017). How Virtual Reality is Impacting the Ad Industry. Business.com. Retrieved 27-February-2019 from: <i>[: https://www.business.com/articles/virtual-reality-advertising-augmented/ ]</i></ref> Biometric data from that group was compared with a control group that only saw a trailer in VR, and it revealed that the VR room left a longer lasting impression. Within a VR environment, biometric data can be measured in real time which lets marketers know how effective a campaign really is. </p><br />
<p>Entertainment is still one of the largest uses for AR and VR. The explosive and sustained popularity of Pokémon Go (a mobile game AR application with 147 million monthly users <ref>Philips, Tom. (June 27th, 2018). Pokemon Go Active Player Count Highest Since 2016 Summer Launch. EuroGamer. Retrieved 27-February-2019 from: <i>[https://www.eurogamer.net/articles/2018-06-27-pokemon-go-player-count-at-highest-since-2016-summer-launch ]</i></ref>) proves there is a large base of users who are open to using the technology. VR games come in all shapes and sizes and the market is projected to hit 22.9 billion by 2020 <ref>Statista. (2015). Virtual Reality video gaming sales revenue worldwide from 2015-2020. Statista. Retrieved 29-Fabruary-2019 from: <i>[https://www.statista.com/statistics/499714/global-virtual-reality-gaming-sales-revenue/ ]</i></ref>. VR as a medium is also being explored for the film industry and it is said to improve empathy and it can be used as a teaching tool. Even in the world of law, Bloomberg Law speculates that VR might be incorporated into the courtroom, so that evidence can be better understood in an immersive environment. <ref>Kaufman, Bruce. (November 16th, 2017). The Next Frontier for Virtual Reality: Courtrooms. Bloomberg BNA. Retrieved 20-February-2019 from: <i>[https://www.bna.com/next-frontier-virtual-n73014472186/ ]</i></ref> </p><br />
<p>Use cases have also been explored in the medical field. Since medical practitioners rely heavily on visual information, AR and VR is well placed to help them visualize tasks. One such creation is a handheld device developed by Accuvein that reads the heat signature of a patients veins and projects an AR map of the underlying vasculature onto their skin. <ref>AccuVein. (2015). AccuVein Vein Visualization: The Future of Healthcare is Here. AccuVein. Retrieved 18-February-2019 from: <i>[https://www.accuvein.com/why-accuvein/ar/ ]</i></ref> This has increased the likelihood of finding a vein on the first try by 3.5 times for those using the machine. A patient’s information can also be used to pre-plan surgeries in VR, their CT and MRI scans can be used to recreate a digital body that can be explored in detail in VR. <ref>mVR Division. Surgical Navigation Advanced Platform. Surgical Theater LLC. Los Angeles, California. Retrieved 18-February-2019 from: <i>[https://www.surgicaltheater.net/plan-navigate/ ]</i></ref> Similarly, those same scans can be super-imposed onto the patient in real-time to guide doctors to the problem area before a surgery and have a better idea of what they are targeting under the patient’s skin.<ref>Microsoft Reporter. (February 8th, 2018). Surgeons use Microsoft HoloLens to see inside patients before they operate. Microsoft. Retrieved 20-February-2019 from: <i>[https://news.microsoft.com/en-gb/2018/02/08/surgeons-use-microsoft-hololens-to-see-inside-patients-before-they-operate-on-them/ ]</i></ref> </p><br />
<h2>Canadian Government Use</h2><br />
<p>The GC has invested $9.5 million in AR technology developed by Vancouver-based software company NGRAIN. <ref>Market Wired. (June 26th, 2013). Harper Government Supports Market-Leading Canadian SME. Yahoo Finance. Retrieved 22-February-2019 from: <i>[https://finance.yahoo.com/news/harper-government-supports-market-leading-180734827.html ]</i></ref> The software company has developed an interactive 3D AR platform for aerospace company Lockheed Martin. Lockheed Martin uses this interactive platform as a tool to perform maintenance on fighter jets like the F-35 and F-22. The company reports reduced maintenance time and less errors. The Canadian Military is also investing in VR, they have recently purchased a mixed reality simulation to train helicopter operators in the use of a hoist system.<ref>Burke, David. (April 10th, 2018) Fear makes virtual training lessons stick in Nova Scotia’s thriving VR industry. Canadian Broadcasting Corporation. Retrieved 20-February-2019 from: <i>[https://www.cbc.ca/news/canada/nova-scotia/virtual-reality-business-technology-nova-scotia-1.4609172 ]</i></ref> This and other projects like it reduce the training costs associated with hands-on jobs, where no costly accidents can happen and learners are allowed to learn from their mistakes in a risk-free environment.</p><br />
<p>The GC has also been investing in AR and VR as tools of education and entertainment. Through the Build in Canada Innovation Program, $482,000 was awarded to Motive.io to develop a location based AR experience that takes users through Ottawa’s cultural heritage. <ref>Public Services and Procurement Canada. (November 20th, 2017). Government of Canada supports BC augmented reality startup through Build in Canada Innovation Program. Published by Cision Canada. Government of Canada. Retrieved 22-February-2019 from: <i>[https://www.newswire.ca/news-releases/government-of-canada-supports-british-columbia-augmented-reality-startup-through-build-in-canada-innovation-program-658845663.html ]</i></ref> Although there is no substantial research yet to support the claim that immersive VR experiences increase empathetic responses, many projects are being developed with that goal in mind. The Social Sciences and Humanities Research Council (SSHRC) helped fund a VR Residential School that takes viewers into a world crafted by survivors and researchers, it’s hoped that the project will eventually be adopted as a teaching tool for high schools. <ref>University of Manitoba. (September 12th, 2018). Launch of a virtual reality Canadian Residential School. University of Manitoba. Winnipeg, Manitoba. Retrieved 20-February-2019 from: <i>[http://news.umanitoba.ca/launch-of-a-virtual-reality-canadian-residential-school/ ]</i></ref> Even the Parliament of Canada has been turned into an interactive augmented virtuality environment, this time by partnering with Carleton University to create a virtual tour of the Senate where users can walk around the building and click on points of interest to learn of their historical significance.<ref>Senate of Canada. (March 1st, 2017). Senate Virtual Tour. Government of Canada. Retrieved 6-February-2019 from: http://tour-ext.sencanada.ca/tour/index.html and <i>[https://sencanada.ca/en/sencaplus/how-why/the-senate-virtual-tour-your-digital-admission-to-the-red-chamber/ ]</i></ref> </p><br />
<p>The Canadian Military stands to benefit the most from the use of AR and VR technology, given their needs for logistics management, training solutions, and personnel management. At present, the military mainly uses those technologies for training purposes. Some simulations are entirely based in VR, and some have real physical equipment that can be “used” in a VR environment. Bluedrop, a performance learning and IT consulting and development firm, has designed both types of training programs, and one of their mixed reality simulations in particular integrates a real CH-47 Chinook helicopter fuselage into a virtual training program. <ref>Arellano, Nestor. (August 26th, 2019). VR, gaming technology and training the future soldier. Vanguard Canada. Retrieved 6-February-2019 from: <i>[https://vanguardcanada.com/2016/08/26/vr-gaming-technology-training-future-soldier/ ]</i></ref> The Canadian Armed Forces has more than 230 simulators in 125 locations across Canada, and not included in this count is the number of Navy and Airforce simulators<ref>Boutilier, Alex. (January 1st, 2015). Canadian Forces expand virtual reality training capabilities. Toronto Star Papers. Toronto, Ontario. Retrieved 8-February-2019 from: <i>[https://www.thestar.com/news/canada/2015/01/01/canadian_forces_expand_virtual_reality_training_capabilities.html ]</i></ref>. Military therapists have also explored the use of VR to treat post-traumatic stress disorder in soldiers, as a way for them to safely explore the source of their trauma and move past it. <ref>Madigan, Jennifer. (December 19th, 2017). Virtual reality simulation and neurofeedback to help PTSD. Blueline, Annex Business Media. Toronto, Ontario. Retrieved 8-February-2019 from: <i>[https://www.blueline.ca/news/features/virtual-reality-simulation-and-neurofeedback-to-help-ptsd-5076 ]</i></ref> Through the Innovation for Defense Excellence and Security program, a few projects involving AR and VR have received funding from the Department of National Defense. Categories include cognitive performance enhancement, Resilient Non-GPS Based Positioning Navigation and Timing, and understanding and addressing PTSD.<ref>Department of National Defense. IDEas Recipients. Government of Canada. Retrieved 8-February-2019 from: <i>[https://www.canada.ca/en/department-national-defence/programs/defence-ideas/past-opportunities/ideas-recipients.html ]</i></ref> </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<br />
<h4>Value Proposition</h4><br />
<p>SSC should consider the positive impacts that AR and VR can have on the Department. For the Department to benefit from AR and VR technologies, very specific projects and use cases may be considered before applications are developed. Projects should not be launched simply for the sake of innovation but because they fulfill business objectives and requirements. The Department could lead by example for other departments by adopting AR and VR solutions that have already been developed and promote those solutions.</p><br />
<p>The Government of Canada (GC) can provide better services to citizens and other departments through the use of AR and VR technology. The government can provide a clearer vision of its planning and documentation. For architectural projects, for example, virtual tours of proposed sites and renovations can be given to citizens and other stakeholders. The same concept can also be used in courtrooms to help explain evidence in a convincing and immersive way. Policy and documentation can now become more interactive in the world of AR and VR technology. </p><br />
<p>Additionally, AR and VR technology can accommodate the accessibility needs of GC employees and citizens interacting with the GC. AR and VR are part of a suite of technologies known as Adaptive Computer Technology (ACT), which aids in providing assistance to those with sensory, mobility, dexterity and learning disabilities who work on or communicate through computers. AR and VR can reduce the barriers presented by the standard computer interface and can offer new and adaptive ways for GC employees and citizens to interact with GC information and data. <ref>Shared Services Canada. (April 3rd, 2019). Accessibility, Accommodation and Adaptive Computer Technology (AAACT). Government of Canada. Aaact-aatia@ssc-spc.gc.ca. (819) 994-4835. Retrieved 03- May-2019 from: <i>[https://www.canada.ca/en/shared-services/corporate/aaact-program.html ]</i></ref> </p><br />
<p class="expand mw-collapsible">The use of AR and VR technology can also aid and support public safety and emergency services. For example, if emergency vehicles are fitted with AR Head-Up windshield displays this could help with route navigation, allowing responders to find the quickest route and obtain traffic information in real-time. Location based AR technology could also allow first responders to gain insight on dangers and hazardous conditions around them. Through the Innovative Solutions Canada program, a current challenge open to the public is the creation of a heads up and hands free solution for supporting firefighters in the field. <ref> Innovation, Science and Economic Development Canada. (2018). Head Up Hands Free Firefighting. Government of Canada. Retrieved 8-February-2019 from: <i>[http://www.ic.gc.ca/eic/site/101.nsf/eng/00056.html]</i></ref> </p><br />
<p>In general, AR and VR technologies are well suited for tasks that rely heavily on visual elements. The industry is not yet mature (see Appendix A) so not all potential use cases have been explored, or even discovered. At present, AR business use cases include the itemization of tasks, remote work support (a supervisor can see and comment on what a worker is doing if a HDM has a camera), performance dashboards for Internet of Things (IoT) enabled devices, guiding precision manual labor (i.e. repair, assembly, construction), prototyping, and for overlaying medical information on a patient.</p><br />
<p>Similarly, not all use cases have been uncovered for VR but there are many existing business solutions. A strength of VR is that it lends itself well for immersive experiences since the entire field of view is replaced with a virtual world. Video games and 360̊ videos are very popular for consumer use and can be easily transformed for business use. Mixed reality practice simulations are being adopted to replace high-risk training scenarios since there is less risk of injury and equipment damage. Undergoing training in VR helps learners retain information since they are being completely immersed into a practice scenario where they can physically interact with virtual objects (as opposed to learning in a classroom). Prototyping and virtual tours can also be done in VR</p><br />
<p>As a communications service provider, SSC is well suited to become a provider of AR and VR equipment for other partnered departments. The Department could also lead the development of AR and VR solutions on an on demand basis for other departments.</p><br />
<h4>Challenges</h4><br />
<p>Prolonged use of AR and VR gear has not been extensively studied, and there could be unknown risks that have yet to be uncovered, such as health risks to employees etc. The use of AR and VR in the form of wearable technology in the workplace could also provide further challenges. Although the technology can increase connectivity among employees and simplicity of user interfaces, AR and VR technology could also cause employees to lose focus on the task at hand when working using the AR and VR technology. This challenge could be expected to diminish as AR and VR technologies become more commonplace.</p><br />
<p>AR and VR technologies are great for the immersion factor, however being disconnected from the physical world while still being physically present within it can make users “blind” to the environment around them. According to a study from Purdue University done in Tippecanoe County (US), the popular AR game Pokemon Go has been linked to two deaths and has caused an increase in car accidents near “PokeStops” (real world locations with in-game components) since users are playing the game while driving or walking. <ref>Sharwood, Simon. (November 27th, 2019). Pokemon Go Caused Hundreds of Deaths, Increased Crashes. The Register. International. Retrieved 27-February-2019 from: <i>[https://www.theregister.co.uk/2017/11/27/pokemon_go_caused_car_accidents_and_deaths/ ]</i></ref> There is also anecdotal evidence of VR headset wearers forgetting their surroundings and tripping over physical objects within a room, falling out of chairs, and even hitting other people around them. Developers will need to consider elements of special awareness when designing solutions and how liability laws may affect them. </p><br />
<p>Another issue for AR and VR is that of interoperability between devices. For mobile devices, there are a number of platforms from which to access VR applications but not all platforms are available on all devices. iPhone and Android users are limited by what they are able to download from their respective application stores. AR headsets are not limited by their hardware like smartphones, but applications must live within a platform for them to function and there are many platforms to choose from on the market. An AR platform in this context is similar to an operating system for the headset, where developers can create solutions that run on the platform. </p><br />
<p>Lastly, standard AR and VR solutions may not be suited for those with sensory, mobility, dexterity, and learning disabilities. Individuals who may benefit from ACT would need to be properly assessed and outfitted with tools and technology that appropriately addresses their unique needs. A standard off-the-shelf corporate AR and VR solution may improve the ease of work and productivity of able-bodied employees, but it may not adequately address the needs of those requiring sensory, mobility, or learning aids. The AR or VR solution may require adaptation and modification to fit each end user, which will require additional resources and expertise in resolving.</p><br />
<br />
<h4>Considerations</h4><br />
<p>In the realm of AR and VR technology, the question of data ownership will be a very important issue to consider, given that data created from the use of AR and VR (user profiles, biometric data, sentiment response, eye tracking, etc.) presents a number of privacy and security concerns. If applications are created by outside companies, there is the potential for information to be collected by those providers and SSC will need to be very clear about who can access that information and for what purposes. Clarification will also be needed as to whether or not SSC can collect that data in the first place, SSC may need to anonymize all AR / VR data or simply not collect it. Moving forward, SSC has the potential to use AR and VR solutions but the Department needs to carefully look at the business problem that needs to be solved and if the technology is an appropriate choice to solve that problem.</p><br />
<p>The adoption of AR and VR systems present several challenges. Due to the current lack of low-cost hardware capable of running AR based applications, most AR solutions are designed as mobile apps. This is because a mobile device can track all the necessary information for a properly function application, but it comes with computational limitations since it is all contained within a small device. Location-based AR applications are currently the only type of AR application available for use on mobile phones. This is because Recognition-based and SLAM technology is much too computationally intensive for a smartphone. Since Location-based software requires only GPS it is a viable approach. As the need for more complex AR and VR applications grows so will the computational demands. Smartphones alone will not be able to handle these new complex demands. SSC will need to carefully consider project goals when deciding on whether the Department should create a VR or AR application, and on what kind of hardware it should run on. Scalability is another consideration for SSC, that is, the ability to grow a project’s infrastructure and capabilities to let more people use it at the same time. A project’s eventual growth in terms of scalability and deployment will need to be considered right at the start its development.</p><br />
<p>SSC should consider working through its Accessibility, Accommodation and Adaptive Computer Technology (AAACT) team to pilot AR and VR solutions to those with sensory, mobility, and learning needs. <ref>Shared Services Canada. (April 3rd, 2019). Accessibility, Accommodation and Adaptive Computer Technology (AAACT). Government of Canada. Aaact-aatia@ssc-spc.gc.ca. (819) 994-4835. Retrieved 03- May-2019 from: <i>[https://www.canada.ca/en/shared-services/corporate/aaact-program.html]</i></ref> SSC could engage with employees who have requested assistance from AAACT and collaborate with them to test various technologies in order to assess whether or not they reduce the barriers to work for those with impairments.</p><br />
<p>Given that the AR and VR markets are not yet mature and that more advancements are projected, SSC senior management should proceed with caution as it relates to the procurement of AR and VR. Additionally, caution is advised regarding the hype surrounding the promises of immersive technologies. What is depicted in popular films such as “Ready Player One” has already inflated the expectations of consumers and does not accurately reflect the business reality and utility of AR and VR today.</p><br />
<p>Given that these technologies generate a lot of data while they are being used, as the IT infrastructure provider for the GC, SSC should ensure that it is well equipped to handle the additional bandwidth usage of large scale deployments. </p><br />
<p>Since AR and VR are technologies that are still being refined, SSC will need to make investment/procurement decisions within a relatively short timeframe. Depending on the project scope, availability of funds, and availability of personnel the Department will need to consider how it will develop AR and VR solutions. These solutions will require specialized knowledge which is highly sought after in the tech world and hiring personnel could be a competitive endeavor. Taking this into account, having an in-house team for creating AR and VR applications may not be feasible unless there is a lot of demand from other departments. The Department at this time should consider contracting in the short term (the next 5-10 years), since the talent pool, technologies, and business use cases are not at an adequate level of maturity. </p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Blockchain";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends&diff=13686Technology Trends2019-11-14T02:57:59Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:{{FULLPAGENAME}}}}<br />
[[fr:Tendances_Technologiques]]<br />
<br />
<div class="sideTable"><br />
<div class="navigation-section"><br />
<div class="navigation-table"><br />
<table class="wikitable nav-tab" style="margin-top: 0px;"><br />
<tr><th class="breadcrumb" colspan="2"><b>Technology Trends</b><br><b>[[Tendances_Technologiques|Tendances Technologiques]]</b></th></tr><br />
<tr><td colspan="2" class="logo">[[File:Technology_brief_logo.png|200px]]</td></tr><br />
<tr><br />
<th>Trend Template</th><br />
<td>[[Media:EN_-_Technology_Trends_Template.docx|Technology Trends.docx]]</td><br />
</tr><br />
<tr><br />
<th>Other Related Publications</th><br />
<td><b>[[Architecture_Framework_Advisory_Committee|Architecture Framework Advisory Committee]]</b><hr/><b>[[Technology_Trends/User_Guide|User Guide]]</b></td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">Recent additions include [[Technology_Trends/Blockchain|Blockchain]], [[Technology_Trends/Drones|Drones]], [[Technology_Trends/Internet_of_Things|Internet of Things]], [[Technology_Trends/Low_Code_Application_Development|Low Code Application Development]], [[Technology_Trends/Open_by_Default|Open by Default]], [[Technology_Trends/Zero_Trust_Network|Zero Trust Network]].</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
</div><br />
</div><br />
<br />
<div><br />
<br />
<p>In an effort to better support its mandate of providing innovative Information Technology (IT) services to Government of Canada (GC) departments and agencies, Shared Services Canada (SSC) is actively researching and evaluating emerging technology trends. This research aims to Inform and prepare SSC, and the broader GC community, to handle new disruptive technologies that are just over the horizon by outlining potential benefits, impacts and use cases.</p><br />
<br />
<p>Technology trend briefing papers are the vehicle used to convey the research conducted on a given subject. These briefing papers are short, to the point, and are targeted towards both a business and technical oriented readership.</p><br />
<br />
<p>SSC is looking to the broader GC community to contribute and collaborate on existing and future topics. By fostering a more collaborative and inclusive development approach, we hope to reduce GC research duplication and accelerate the development of technology roadmaps across government by referencing common GC documentation.</p><br />
<br />
<h2>Format</h2><br />
<br />
<p>Each technology trend briefing paper follows a common template:</p><br />
<br />
<ul><br />
<li><b>Business brief </b>describes the business aspect of the technology by offering a high level understanding, the origin and the uses.</li><br />
<li><b>Technical brief </b>gives an in-depth description of the technology. It focuses on the most important technical aspects of the technology without going into too much detail.</li><br />
<li><b>Industry usage </b>offers an understanding of how the technology is currently used or thought to be used in the industry when commercially available. This section clarifies the benefits of the technology through examples.</li><br />
<li><b>Canadian Government use </b>details how the technology is currently used or thought to be used in the context of the Federal Canadian Government when commercially available. This intent is to provide an understanding of how the technology could ultimately benefit the federal government in regards to the delivery of services to Canadians through short examples.</li><br />
<li><b>Implications for Shared Services Canada (SSC) </b>demonstrates the clear and measurable benefits that pertains to Shared Service Canada when adopting the use of the technology. This section is comprised of these following subsections :<br />
<ul><br />
<li><b>Value proposition </b>illustrates the potential strengths and opportunities that SSC can encounter by leveraging the technology.</li><br />
<li><b>Challenges </b>outlines the risks and difficulties associated with the technology that includes both risks of implementation such as the development of the product or the risks of solution by adopting the technology.</li><br />
<li><b>Considerations </b>presents any recommendations SSC may have for another department of the Government of Canada that wishes to adopt and leverage the technology.</li><br />
</ul><br />
</li><br />
</ul><br />
<br />
<h2>Development Process</h2><br />
<br />
<p>External market trends, private-public sector reports, online trends and Government of Canada strategic plans are all factors considered when selecting a topic for development. Once a topic is selected for development, it undergoes a five phase development process prior to publishing.</p><br />
<br />
<div>[[File:Technology_Trends_-_Development_Process.png|center]]</div><br />
<br />
<h2>How to Contribute</h2><br />
<br />
<p>We want your help! Our goal is to ensure that the content generated is developed in the open, routinely kept up to date, representative of the wide ranging GC departmental mandates and widely distributed to the broader Canadian Government community. To that end, we strongly encourage you to share, edit and provide feedback on the published content.</p><br />
<br />
<p>You can edit by making use of the “Discussion” tab located at the top-left of the page.</p> <br />
<br />
<p>Should you wish to sponsor and or author a topic not listed below, please reach out to [mailto:jean-francois.michaud@canada.ca Jean-François Michaud]</p><br />
<br />
</div><br />
<br />
<div class="container"><br />
<div class="row"><br />
<div class="col-sm-12"><br />
<h2>Technology Trend Briefing Papers</h2><br />
<br />
<table class="wikitable sortable technology-table" style="border: 0px solid;"><br />
<tr><br />
<th colspan="3">Name</th><br />
<th>Status</th><br />
<th style="white-space: nowrap;">Release Date</th><br />
<th>Official Publications</th><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Blockchain|Blockchain]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Chaîne_de_Blocs|Chaîne de Blocs]]</td><br />
<td>Published</td><br />
<td>2019-05-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Blockchain.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Blockchain.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Drones|Drones]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Drones|Drones]]</td><br />
<td>Published</td><br />
<td>2019-07-04</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Drones.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Drones.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Internet_of_Things|Internet of Things]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Internet_des_Objets|Internet des Objets]]</td><br />
<td>Published</td><br />
<td>2019-07-04</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Internet_of_Things.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Internet_des_Objets.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Low_Code_Application_Development|Low Code Application Development]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Développement_d%27Application_LowCode|Développement d'Application Programmation Schématisée]]</td><br />
<td>Published</td><br />
<td>2019-05-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Low_Code_Application_Development.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Développement_d%27Application_LowCode.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Open_by_Default|Open by Default]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Ouvert_par_Défaut|Ouvert par Défaut]]</td><br />
<td>Published</td><br />
<td>2019-05-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Open_by_Default.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Ouvert_par_Défaut.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Zero_Trust_Network|Zero Trust Network]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Réseaux_Zéro_Confiance|Réseaux Zéro Confiance]]</td><br />
<td>Published</td><br />
<td>2019-05-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Zero_Trust_Network.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Réseaux_Zéro_Confiance.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/5G_Networks|5G Networks]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Réseaux_5G|Réseaux 5G]]</td><br />
<td>Published</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_5G_Networks.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Réseaux_5G.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Big_Data|Big Data]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Mégadonnées|Mégadonnées]]</td><br />
<td>Published</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Big_Data.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Mégadonnées.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Edge_Computing|Edge Computing (Serverless Computing)]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Informatique_en_Périphérie|Informatique en Périphérie (Informatique sans Serveurs)]]</td><br />
<td>Published</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Edge_Computing.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Informatique_en_Périphérie.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Face_Recognition|Face Recognition]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">[[Tendances_Technologiques/Reconnaissance_Faciale|Reconnaissance Faciale]]</td><br />
<td>Published</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Face_Recognition.pdf|EN]] - [[Media:FR_-_Tendances_Technologiques_-_Reconnaissance_Faciale.pdf|FR]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Datalakes|Datalakes]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Lac de Données</td><br />
<td>Translation</td><br />
<td>2019-08-06</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Datalakes.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Digital_Twins|Digital Twins]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Jumeaux Numériques</td><br />
<td>Translation</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Digital_Twins.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Kubernetes|Kubernetes]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Kubernetes</td><br />
<td>Translation</td><br />
<td>2019-07-23</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Kubernetes.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Data_Leak_Prevention|Data Leak Prevention]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Prévention des Fuites de Données</td><br />
<td>Translation</td><br />
<td>2019-08-06</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Data_Leak_Prevention.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Cloud_Management_Platform|Cloud Management Platform]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Platforme de Gestion d'Infonuagique</td><br />
<td>Translation</td><br />
<td>2019-09-18</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Cloud_Management_Platform.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Digital_Assistants|Digital Assistants (Conversational UI)]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Assistants Numériques (Interface Conversationnelle)</td><br />
<td>Translation</td><br />
<td>2019-09-25</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Digital_Assistants.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Enterprise_Application_Stores|Enterprise Application Stores (API Stores)]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Magasins d'Applications d'Entreprise (Magasins d'API)</td><br />
<td>Translation</td><br />
<td>2019-10-30</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Enterprise_Application_Stores.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Software_Defined_Anything|Software Defined Anything]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Tout Défini par Logiciel</td><br />
<td>Translation</td><br />
<td>2019-10-30</td><br />
<td>[[Media:EN_-_Technology_Trends_-_Software_Defined_Anything.pdf|EN]]</td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Digital Services</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Services Numériques</td><br />
<td>Development</td><br />
<td>2019-07-23</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Interoperability (Tell Us Once)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Interopérabilité</td><br />
<td>Development</td><br />
<td>2019-07-23</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">3D XPoint (Optane)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">3D XPoint (Optane)</td><br />
<td>Development</td><br />
<td>FY 19-20 Q2</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Application_Containers|Application Containers]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Conteneurs Logiciels</td><br />
<td>Development</td><br />
<td>FY 19-20 Q2</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">AI (Machine Learning)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">IA (Apprentissage Automatique)</td><br />
<td>Development</td><br />
<td>FY 19-20 Q2</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Crowdsourcing</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Production participative</td><br />
<td>Development</td><br />
<td>FY 19-20 Q2</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Autonomous Vehicles</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Véhicules Autonomes</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Infrastructure as Code</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Infrastructure en tant que Code</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">SCADA Systems</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Systèmes SCADA</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Cloud Workplace (Office)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Bureau Infonuagique</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Low Earth Orbit System</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Système en Orbite Terrestre Basse</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Quantum Computing</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Informatique Quantique</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Robotics</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Robotiques</td><br />
<td>Development</td><br />
<td>FY 19-20 Q3</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Data Centre Infrastructure Monitoring</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Surveillance de l'Infrastructure du Centre de Données</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Digitial / Smart Offices</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Bureaux Digitales / Smart</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Enterprise Collaboration</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Collaboration d'Entreprise</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Gamification (Innovation Framework)</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Gamification (Cadre d'Innovation)</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Intent Based Networking</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Réseaux basé sur l'Intention</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">IT Talent</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Talent Informatique</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">Pay as you go</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Pay as you go</td><br />
<td>Unassigned</td><br />
<td>FY 19-20 Q4</td><br />
<td></td><br />
</tr><br />
<tr><br />
<td style="border-right: 0px solid; text-align:right;">[[Technology_Trends/Li-Fi|Li-Fi]]</td><br />
<td style="border-left: 0px solid; border-right: 0px solid;">-</td><br />
<td style="border-left: 0px solid; text-align:left">Li-Fi</td><br />
<td>Unassigned</td><br />
<td></td><br />
<td></td><br />
</tr><br />
</table><br />
</div><br />
</div><br />
</div><br />
<br />
[[Category:Architecture]]<br />
[[Category:Technology]]<br />
<br />
{{#css:<br />
.technology-table{<br />
width: 100%;<br />
margin: auto;<br />
}<br />
<br />
th, td{<br />
text-align: center;<br />
}<br />
<br />
.logo{<br />
text-align: center;<br />
}<br />
<br />
.disclaimer{<br />
border: 1px solid red;<br />
background-color: rgba(255, 0, 0, 0.3);<br />
}<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
padding: 5px 10px 5px 10px;<br />
}<br />
<br />
.disclaimerText{<br />
width: 100%;<br />
text-align: left;<br />
}<br />
<br />
.news{<br />
border: 1px solid green;<br />
background-color: rgba(0, 255, 0, 0.3);<br />
}<br />
<br />
.toggle{<br />
color: rgb(6, 69, 173);<br />
float:left;<br />
<br />
}<br />
<br />
.toggle:hover{<br />
text-decoration: underline;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { float: left; }<br />
<br />
.col-sm-1 { width: 8.333333333%; }<br />
.col-sm-2 { width: 16.66666666%%; }<br />
.col-sm-3 { width: 25%; }<br />
.col-sm-4 { width: 33.333333333333%; }<br />
.col-sm-5 { width: 41.6666666666%; }<br />
.col-sm-6 { width: 50%; }<br />
.col-sm-7 { width: 58.333333333%; }<br />
.col-sm-8 { width: 66.66666666666%; }<br />
.col-sm-9 { width: 75%; }<br />
.col-sm-10 { width: 83.33333333333%; }<br />
.col-sm-11 { width: 91.6666666666%; }<br />
.col-sm-12 { width: 100%; }<br />
<br />
.sideTable {<br />
width: 27.5%;<br />
float: right;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 95%; <br />
margin-left: 2.5%; <br />
margin-right: 2.5%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
}<br />
<br />
.inverse-direction {<br />
float: right;<br />
}<br />
<br />
.td-left { border-right: 0px solid; }<br />
.td-middle { border-right: 0px solid; border-left: 0px solid; }<br />
.td-right { border-left: 0px solid; }<br />
<br />
.nav-tab {<br />
width: 100%;<br />
margin-top: 0px;<br />
padding-top: 0px;<br />
}<br />
<br />
<br />
@media (max-width: 1050px){<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.sideTable {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%; <br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%; <br />
margin-right: 0%;<br />
}<br />
<br />
/*<br />
.navigation-table {<br />
width: 30%;<br />
margin-left: 1.6666666%; <br />
margin-right: 1.6666666%;<br />
float: left;<br />
}<br />
*/<br />
<br />
.content-section {<br />
float: left;<br />
}<br />
<br />
.inverse-direction {<br />
clear: both;<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { clear: both; width: 100%; margin: auto; }<br />
}<br />
<br />
div>a>img {<br />
width: 600px;<br />
height: auto;<br />
margin: auto;<br />
}<br />
<br />
@media (max-width: 850px){<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%; <br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
margin-left: 0%; <br />
margin-right: 0%;<br />
clear: both;<br />
}<br />
}<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Comit%C3%A9_Consultatif_sur_le_Cadre_d%27Architecture&diff=11321Comité Consultatif sur le Cadre d'Architecture2019-07-22T18:41:22Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:{{FULLPAGENAME}}}}<br />
[[fr:Tendances_Technologiques]]<br />
<br />
<div class="sideTable"><br />
<div class="navigation-section"><br />
<div class="navigation-table"><br />
<table class="wikitable nav-tab"><br />
<tr><th class="breadcrumb" colspan="2"><b>[[Comité_Consultatif_sur_le_Cadre_d%27Architecture|Comité Consultatif sur le Cadre d'Architecture]]</b><br><b>[[Architecture_Framework_Advisory_Committee|Architecture Framework Advisory Committee]]</b></th></tr><br />
<tr><td colspan="2" class="logo">[[File:AFAC_EN_Logo.png|200px]]</td></tr><br />
<tr><br />
<th>Contactez Nous</th><br />
<td>[mailto:ssc.afac-ccca.spc@canada.ca ssc.afac-ccca.spc@canada.ca]</td><br />
</tr><br />
<tr><br />
<th>Autres Publications Connexes</th><br />
<td><b>[[Tendances_Technologiques|Tendances Technologiques]]</b></td><br />
</tr><br />
<tr><td colspan="2" class="news"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="newsText">Cette nouvelle section est maintenant disponible!</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
</div><br />
</div><br />
<br />
<div><br />
<br />
<p>Le Comité Consultatif sur le Cadre d’Architecture (CCCA) de Services partagés Canada (SPC) fournit des conseils et des recommandations stratégiques à l’appui du mandat de SPC et des objectifs associés au gouvernement du Canada.</p><br />
<br />
<h2>Mandat</h2><br />
<br />
<p>CCCA soutient le mandat de SPC et les initiatives de transformation numérique du GC en servant de forum de consultations des secteurs publics et privés sur l’architecture d’entreprise. SPC identifiera ou analysera les approches commerciales sous une optique stratégique avec des spécialistes du domaine de l'industry afin d’élaborer ou d’affiner les directions proposées pour la planification à moyen et long terme (p. Ex. Cycles de planification de trois à dix ans).</p><br />
<br />
<h2>Communauté</h2><br />
<br />
<p>La communauté est composée d'individus et d'organisations des secteurs publics et privés ayant une expertise dans les technologies de l'information et de la communication (TIC).</p><br />
<br />
<p>Les réunions de CCCA sont animées par :</p><br />
<br />
<ul><br />
<li>Président – Dirigeant principal de la technologie (DPT), SPC</li><br />
<li>Coprésident – DPT du gouvernement du Canada, Secrétariat du Conseil du Trésor du Canada</li><br />
</ul><br />
<br />
<p>La communauté de CCCA comprend les associations ci-dessous du secteur des TIC (et leurs affiliations) :</p><br />
<br />
<ul><br />
<li>Réseau Canadien de Technologie de l'Information d'Affaires (CABiNET)</li><br />
<li>Alliance Canadienne de Technologie de Pointe (CATA)</li><br />
<li>Association Canadienne de la Technologie de l'Information (ACTI)</li><br />
<li>Investir Ottawa</li><br />
</ul><br />
<br />
<p>Les membres d'office sont : </p><br />
<br />
<ul><br />
<li>Dirigeants principaux de l'information de d'autres ministères du gouvernement du Canada</li><br />
</ul><br />
<br />
<p>CCCA demandera aux associations du secteur des TIC la désignation de sociétés possédant une expertise pertinente dans des domaines intéressant le comité. Chaque membre de l'association aura l'occasion de nommer un à deux experts en la matière pour participer aux discussions relatives à leurs domaines de compétence.</p><br />
<br />
<p>SPC se réserve le droit d'adresser des invitations à des organisations du secteur privé et non gouvernementales dotées d'expertise pertinente, ainsi que la possibilité d'ajuster ou de compléter la liste des personnes et d'organisations participant aux réunions afin de garantir le niveau d'expertise requis. Les membres du personnel de SPC ayant des responsabilités ou une expertise en rapport avec les sujets traités dans le CCCA ou les réunions de groupes de travail connexes seront invités à participer en fonction des besoins.</p><br />
<br />
<h2>Réunions et Agenda</h2><br />
<br />
<p>CCCA se réunit tous les trimestres ou, au besoin, sur convocation du président. Les ordres du jour varient en fonction des sujets de discussion. CCCA veillera à ce que le niveau d'expertise nécessaire soit partagé lors des consultations, tout en limitant le nombre de participants à une taille raisonnable, afin d'encourager la participation active et de maximiser la valeur du temps alloué à chaque point de l'ordre du jour.</p><br />
<br />
<h2>Soutient du Secrétariat</h2><br />
<br />
<p>Le Secrétariat de l'engagement au sein de PGET, PVR, gérera la logistique et fournira des services de secrétariat. Le matériel de réunion sera fourni au moins une semaine avant les réunions prévues. Des services de téléconférence seront fournis aux participants participant à distance.</p><br />
<br />
<p>Le secrétariat préparera les procès-verbaux de chaque réunion et les diffusera en interne au sein de SPC. Les commentaires formulés lors des réunions ne seront pas attribués à des individus ou à des organisations.</p><br />
<br />
</div><br />
<br />
<div class="container"><br />
<div class="row"><br />
<div class="col-sm-12"><br />
<h2>Sujets Antérieur</h2><br />
<br />
<table class="wikitable sortable technology-table"><br />
<tr><br />
<th>Sujet</th><br />
<th>Date</th><br />
<th>Présentation </th><br />
<th>Summaire de Discussion</th><br />
<br />
</tr><br />
<tr><br />
<td>Adoption de la conteneurisation des applications</td><br />
<td>2019-04-08</td><br />
<td>[[Media:EN_-_AFAC_Presentation_-_2019-04-08.pdf|EN]] / [[Media:FR_-_Exposé_(Adoption_de_la_conteneurisation_des_applications).pdf|FR]]</td><br />
<td>[[Media:EN_-_AFAC_Containers_-_Discussion_Summary_v0.1.pdf|EN]]</td><br />
<br />
</tr><br />
<tr><br />
<td>Mise en oeuvre de la stratégie de données d’entreprise du GC</td><br />
<td>2019-06-14</td><br />
<td>[[Media:EN_-_AFAC_(EDS)_Presentation_-_2019-06-14_vFinal_EN.pdf|EN]] / [[Media:FR_-_AFAC_(EDS)_Presentation_-_2019-06-14_vFinal_FR.pdf|FR]]</td><br />
<td>[[Media:EN_-_AFAC_-_EDS_Discussion_Summary_v0.2.pdf|EN]]</td><br />
<br />
</tr><br />
</div><br />
</div><br />
</div><br />
<br />
<br />
[[Category:Architecture]]<br />
[[Category:Technology]]<br />
<br />
{{#css:<br />
.technology-table{<br />
width: 100%;<br />
margin: auto;<br />
}<br />
<br />
th, td{<br />
text-align: center;<br />
}<br />
<br />
.logo{<br />
text-align: center;<br />
}<br />
<br />
.disclaimer{<br />
border: 1px solid red;<br />
background-color: rgba(255, 0, 0, 0.3);<br />
}<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
padding: 5px 10px 5px 10px;<br />
}<br />
<br />
.disclaimerText{<br />
width: 100%;<br />
text-align: left;<br />
}<br />
<br />
.news{<br />
border: 1px solid green;<br />
background-color: rgba(0, 255, 0, 0.3);<br />
}<br />
<br />
.toggle{<br />
color: rgb(6, 69, 173);<br />
float:left;<br />
<br />
}<br />
<br />
.toggle:hover{<br />
text-decoration: underline;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { float: left; }<br />
<br />
.col-sm-1 { width: 8.333333333%; }<br />
.col-sm-2 { width: 16.66666666%%; }<br />
.col-sm-3 { width: 25%; }<br />
.col-sm-4 { width: 33.333333333333%; }<br />
.col-sm-5 { width: 41.6666666666%; }<br />
.col-sm-6 { width: 50%; }<br />
.col-sm-7 { width: 58.333333333%; }<br />
.col-sm-8 { width: 66.66666666666%; }<br />
.col-sm-9 { width: 75%; }<br />
.col-sm-10 { width: 83.33333333333%; }<br />
.col-sm-11 { width: 91.6666666666%; }<br />
.col-sm-12 { width: 100%; }<br />
<br />
.sideTable {<br />
width: 27.5%;<br />
float: right;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 95%;<br />
margin-left: 2.5%;<br />
margin-right: 2.5%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
}<br />
<br />
.inverse-direction {<br />
float: right;<br />
}<br />
<br />
.nav-tab {<br />
width: 100%;<br />
}<br />
<br />
@media (max-width: 1050px){<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.sideTable {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-table {<br />
width: 30%;<br />
margin-left: 1.6666666%;<br />
margin-right: 1.6666666%;<br />
float: left;<br />
}<br />
<br />
.content-section {<br />
float: left;<br />
}<br />
<br />
.inverse-direction {<br />
clear: both;<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { clear: both; width: 100%; margin: auto; }<br />
}<br />
<br />
div>a>img {<br />
width: 75%;<br />
height: auto;<br />
margin: auto;<br />
}<br />
<br />
@media (max-width: 850px){<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
clear: both;<br />
}<br />
}<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=File:EN_-_AFAC_-_EDS_Discussion_Summary_v0.2.pdf&diff=11319File:EN - AFAC - EDS Discussion Summary v0.2.pdf2019-07-22T17:57:16Z<p>Kpere060: </p>
<hr />
<div></div>Kpere060https://wiki.gccollab.ca/index.php?title=Architecture_Framework_Advisory_Committee&diff=11318Architecture Framework Advisory Committee2019-07-22T17:56:52Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:{{FULLPAGENAME}}}}<br />
[[fr:Tendances_Technologiques]]<br />
<br />
<div class="sideTable"><br />
<div class="navigation-section"><br />
<div class="navigation-table"><br />
<table class="wikitable nav-tab"><br />
<tr><th class="breadcrumb" colspan="2"><b>[[Architecture_Framework_Advisory_Committee|Architecture Framework Advisory Committee]]</b><br><b>[[Comité_Consultatif_sur_le_Cadre_d%27Architecture|Comité Consultatif sur le Cadre d'Architecture]]</b></th></tr><br />
<tr><td colspan="2" class="logo">[[File:AFAC_EN_Logo.png|200px]]</td></tr><br />
<tr><br />
<th>Contact Us</th><br />
<td>[mailto:ssc.afac-ccca.spc@canada.ca ssc.afac-ccca.spc@canada.ca]</td><br />
</tr><br />
<tr><br />
<th>Other Related Publications</th><br />
<td><b>[[Technology_Trends|Technology Trends]]</b></td><br />
</tr><br />
<tr><td colspan="2" class="news"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="newsText">This new section is now available!</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
</div><br />
</div><br />
<br />
<div><br />
<br />
<p>The Shared Services Canada (SSC) Architecture Framework Advisory Committee (AFAC) provides strategic advice and recommendations in support of SSC’s mandate and the Government of Canada’s (GC) associated objectives.</p><br />
<br />
<h2>Mandate</h2><br />
<br />
<p>AFAC supports SSC’s mandate and the GC’s digital transformation initiatives by serving as a forum for public-private sector consultation on enterprise architecture. SSC will identify or review business approaches through a strategic lens with industry subject matter experts (SMEs) to develop or refine proposed directions for mid to long-term planning (e.g., three- to ten-year planning cycles).</p><br />
<br />
<h2>Membership</h2><br />
<br />
<p>The membership is composed of individuals and organizations from the public and private sectors with expertise in information and communications technology (ICT).</p><br />
<br />
<p>AFAC meetings are led by the following:</p><br />
<br />
<ul><br />
<li>Chair – Chief Technology Officer (CTO), SSC</li><br />
<li>Co-chair – CTO for the Government of Canada, Treasury Board of Canada Secretariat</li><br />
</ul><br />
<br />
<p>AFAC membership includes the below ICT-sector associations (and memberships thereof):</p><br />
<br />
<ul><br />
<li>Canadian Business Information Technology Network (CABiNET)</li><br />
<li>Canadian Advanced Technology Alliance (CATA)</li><br />
<li>Information Technology Association of Canada (ITAC)</li><br />
<li>Invest Ottawa</li><br />
</ul><br />
<br />
<p>The ex-officio members are:</p><br />
<br />
<ul><br />
<li>Chief Information Officers of other Government of Canada departments</li><br />
</ul><br />
<br />
<p>AFAC will request from the ICT-sector associations nominations of companies with relevant expertise in topics of interest to the committee. Each member association will have the opportunity to nominate one to two subject matter experts to participate in discussions pertaining to their areas of expertise.</p><br />
<br />
<p>SSC reserves the right to extend invitations to private-sector and non-governmental organizations with relevant expertise, as well as the option to adjust or supplement the list of individuals and organizations participating in meetings to ensure that the required level of expertise is available. SSC staff with responsibilities or expertise related to topics being addressed in AFAC or related working-group meetings will be invited to participate on an as-needed basis.</p><br />
<br />
<h2>Meetings and Agenda</h2><br />
<br />
<p>AFAC meets quarterly or, as required, at the call of the Chair. Agendas vary depending on the topics of discussion. AFAC will ensure that the necessary degree of expertise will be shared during consultations, while also limiting the number of participants to a manageable size, to encourage active participation and to maximize the value of the time allocated to each agenda item.</p><br />
<br />
<h2>Secretariat Support</h2><br />
<br />
<p>The Engagement Secretariat within PGET, PVR will manage logistics and provide secretariat services. Meeting materials will be provided a minimum of one week prior to any scheduled meetings. Teleconference services will be provided for attendees participating remotely.</p><br />
<br />
<p>The secretariat will prepare minutes for each meeting and circulate them internally within SSC. Comments made during meetings will not be attributed to individuals or organizations.</p><br />
</div><br />
<div class="container"><br />
<div class="row"><br />
<div class="col-sm-12"><br />
<h2>Past Topics</h2><br />
<br />
<table class="wikitable sortable technology-table"><br />
<tr><br />
<th>Subject</th><br />
<th>Date</th><br />
<th>Presentation</th><br />
<th>Discussion Summary </th><br />
<br />
</tr><br />
<tr><br />
<td>Adoption of Application Containers</td><br />
<td>2019-04-08</td><br />
<td>[[Media:EN_-_AFAC_Presentation_-_2019-04-08.pdf|EN]] / [[Media:FR_-_Exposé_(Adoption_de_la_conteneurisation_des_applications).pdf|FR]]</td><br />
<td>[[Media:EN_-_AFAC Containers - Discussion Summary v0.1.pdf|EN]]</td><br />
<br />
</tr><br />
<tr><br />
<td>Enabling the GC Enterprise Data Strategy</td><br />
<td>2019-06-14</td><br />
<td>[[Media:EN_-_AFAC_(EDS)_Presentation_-_2019-06-14_vFinal_EN.pdf|EN]] / [[Media:FR_-_AFAC_(EDS)_Presentation_-_2019-06-14_vFinal_FR.pdf|FR]]</td><br />
<td>[[Media:EN_-_AFAC_-_EDS_Discussion_Summary_v0.2.pdf|EN]]</td><br />
<br />
</tr><br />
</div><br />
</div><br />
</div><br />
[[Category:Architecture]]<br />
[[Category:Technology]]<br />
<br />
{{#css:<br />
.technology-table{<br />
width: 100%;<br />
margin: auto;<br />
}<br />
<br />
th, td{<br />
text-align: center;<br />
}<br />
<br />
.logo{<br />
text-align: center;<br />
}<br />
<br />
.disclaimer{<br />
border: 1px solid red;<br />
background-color: rgba(255, 0, 0, 0.3);<br />
}<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
padding: 5px 10px 5px 10px;<br />
}<br />
<br />
.disclaimerText{<br />
width: 100%;<br />
text-align: left;<br />
}<br />
<br />
.news{<br />
border: 1px solid green;<br />
background-color: rgba(0, 255, 0, 0.3);<br />
}<br />
<br />
.toggle{<br />
color: rgb(6, 69, 173);<br />
float:left;<br />
<br />
}<br />
<br />
.toggle:hover{<br />
text-decoration: underline;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { float: left; }<br />
<br />
.col-sm-1 { width: 8.333333333%; }<br />
.col-sm-2 { width: 16.66666666%%; }<br />
.col-sm-3 { width: 25%; }<br />
.col-sm-4 { width: 33.333333333333%; }<br />
.col-sm-5 { width: 41.6666666666%; }<br />
.col-sm-6 { width: 50%; }<br />
.col-sm-7 { width: 58.333333333%; }<br />
.col-sm-8 { width: 66.66666666666%; }<br />
.col-sm-9 { width: 75%; }<br />
.col-sm-10 { width: 83.33333333333%; }<br />
.col-sm-11 { width: 91.6666666666%; }<br />
.col-sm-12 { width: 100%; }<br />
<br />
.sideTable {<br />
width: 27.5%;<br />
float: right;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 95%;<br />
margin-left: 2.5%;<br />
margin-right: 2.5%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
}<br />
<br />
.inverse-direction {<br />
float: right;<br />
}<br />
<br />
.nav-tab {<br />
width: 100%;<br />
}<br />
<br />
@media (max-width: 1050px){<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.sideTable {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-table {<br />
width: 30%;<br />
margin-left: 1.6666666%;<br />
margin-right: 1.6666666%;<br />
float: left;<br />
}<br />
<br />
.content-section {<br />
float: left;<br />
}<br />
<br />
.inverse-direction {<br />
clear: both;<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { clear: both; width: 100%; margin: auto; }<br />
}<br />
<br />
div>a>img {<br />
width: 75%;<br />
height: auto;<br />
margin: auto;<br />
}<br />
<br />
@media (max-width: 850px){<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
clear: both;<br />
}<br />
}<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Architecture_Framework_Advisory_Committee&diff=11317Architecture Framework Advisory Committee2019-07-22T17:56:19Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:{{FULLPAGENAME}}}}<br />
[[fr:Tendances_Technologiques]]<br />
<br />
<div class="sideTable"><br />
<div class="navigation-section"><br />
<div class="navigation-table"><br />
<table class="wikitable nav-tab"><br />
<tr><th class="breadcrumb" colspan="2"><b>[[Architecture_Framework_Advisory_Committee|Architecture Framework Advisory Committee]]</b><br><b>[[Comité_Consultatif_sur_le_Cadre_d%27Architecture|Comité Consultatif sur le Cadre d'Architecture]]</b></th></tr><br />
<tr><td colspan="2" class="logo">[[File:AFAC_EN_Logo.png|200px]]</td></tr><br />
<tr><br />
<th>Contact Us</th><br />
<td>[mailto:ssc.afac-ccca.spc@canada.ca ssc.afac-ccca.spc@canada.ca]</td><br />
</tr><br />
<tr><br />
<th>Other Related Publications</th><br />
<td><b>[[Technology_Trends|Technology Trends]]</b></td><br />
</tr><br />
<tr><td colspan="2" class="news"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="newsText">This new section is now available!</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
</div><br />
</div><br />
<br />
<div><br />
<br />
<p>The Shared Services Canada (SSC) Architecture Framework Advisory Committee (AFAC) provides strategic advice and recommendations in support of SSC’s mandate and the Government of Canada’s (GC) associated objectives.</p><br />
<br />
<h2>Mandate</h2><br />
<br />
<p>AFAC supports SSC’s mandate and the GC’s digital transformation initiatives by serving as a forum for public-private sector consultation on enterprise architecture. SSC will identify or review business approaches through a strategic lens with industry subject matter experts (SMEs) to develop or refine proposed directions for mid to long-term planning (e.g., three- to ten-year planning cycles).</p><br />
<br />
<h2>Membership</h2><br />
<br />
<p>The membership is composed of individuals and organizations from the public and private sectors with expertise in information and communications technology (ICT).</p><br />
<br />
<p>AFAC meetings are led by the following:</p><br />
<br />
<ul><br />
<li>Chair – Chief Technology Officer (CTO), SSC</li><br />
<li>Co-chair – CTO for the Government of Canada, Treasury Board of Canada Secretariat</li><br />
</ul><br />
<br />
<p>AFAC membership includes the below ICT-sector associations (and memberships thereof):</p><br />
<br />
<ul><br />
<li>Canadian Business Information Technology Network (CABiNET)</li><br />
<li>Canadian Advanced Technology Alliance (CATA)</li><br />
<li>Information Technology Association of Canada (ITAC)</li><br />
<li>Invest Ottawa</li><br />
</ul><br />
<br />
<p>The ex-officio members are:</p><br />
<br />
<ul><br />
<li>Chief Information Officers of other Government of Canada departments</li><br />
</ul><br />
<br />
<p>AFAC will request from the ICT-sector associations nominations of companies with relevant expertise in topics of interest to the committee. Each member association will have the opportunity to nominate one to two subject matter experts to participate in discussions pertaining to their areas of expertise.</p><br />
<br />
<p>SSC reserves the right to extend invitations to private-sector and non-governmental organizations with relevant expertise, as well as the option to adjust or supplement the list of individuals and organizations participating in meetings to ensure that the required level of expertise is available. SSC staff with responsibilities or expertise related to topics being addressed in AFAC or related working-group meetings will be invited to participate on an as-needed basis.</p><br />
<br />
<h2>Meetings and Agenda</h2><br />
<br />
<p>AFAC meets quarterly or, as required, at the call of the Chair. Agendas vary depending on the topics of discussion. AFAC will ensure that the necessary degree of expertise will be shared during consultations, while also limiting the number of participants to a manageable size, to encourage active participation and to maximize the value of the time allocated to each agenda item.</p><br />
<br />
<h2>Secretariat Support</h2><br />
<br />
<p>The Engagement Secretariat within PGET, PVR will manage logistics and provide secretariat services. Meeting materials will be provided a minimum of one week prior to any scheduled meetings. Teleconference services will be provided for attendees participating remotely.</p><br />
<br />
<p>The secretariat will prepare minutes for each meeting and circulate them internally within SSC. Comments made during meetings will not be attributed to individuals or organizations.</p><br />
</div><br />
<div class="container"><br />
<div class="row"><br />
<div class="col-sm-12"><br />
<h2>Past Topics</h2><br />
<br />
<table class="wikitable sortable technology-table"><br />
<tr><br />
<th>Subject</th><br />
<th>Date</th><br />
<th>Presentation</th><br />
<th>Discussion Summary </th><br />
<br />
</tr><br />
<tr><br />
<td>Adoption of Application Containers</td><br />
<td>2019-04-08</td><br />
<td>[[Media:EN_-_AFAC_Presentation_-_2019-04-08.pdf|EN]] / [[Media:FR_-_Exposé_(Adoption_de_la_conteneurisation_des_applications).pdf|FR]]</td><br />
<td>[[Media:EN_-_AFAC Containers - Discussion Summary v0.1.pdf|EN]]</td><br />
<br />
</tr><br />
<tr><br />
<td>Enabling the GC Enterprise Data Strategy</td><br />
<td>2019-06-14</td><br />
<td>[[Media:EN_-_AFAC_(EDS)_Presentation_-_2019-06-14_vFinal_EN.pdf|EN]] / [[Media:FR_-_AFAC_(EDS)_Presentation_-_2019-06-14_vFinal_FR.pdf|FR]]</td><br />
<td>[[Media:EN_-_AFAC_-_EDS_Discussion_Summary_v0.2.pdf|EN]] / [[Media:FR_-_|FR]]</td><br />
<br />
</tr><br />
</div><br />
</div><br />
</div><br />
[[Category:Architecture]]<br />
[[Category:Technology]]<br />
<br />
{{#css:<br />
.technology-table{<br />
width: 100%;<br />
margin: auto;<br />
}<br />
<br />
th, td{<br />
text-align: center;<br />
}<br />
<br />
.logo{<br />
text-align: center;<br />
}<br />
<br />
.disclaimer{<br />
border: 1px solid red;<br />
background-color: rgba(255, 0, 0, 0.3);<br />
}<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
padding: 5px 10px 5px 10px;<br />
}<br />
<br />
.disclaimerText{<br />
width: 100%;<br />
text-align: left;<br />
}<br />
<br />
.news{<br />
border: 1px solid green;<br />
background-color: rgba(0, 255, 0, 0.3);<br />
}<br />
<br />
.toggle{<br />
color: rgb(6, 69, 173);<br />
float:left;<br />
<br />
}<br />
<br />
.toggle:hover{<br />
text-decoration: underline;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { float: left; }<br />
<br />
.col-sm-1 { width: 8.333333333%; }<br />
.col-sm-2 { width: 16.66666666%%; }<br />
.col-sm-3 { width: 25%; }<br />
.col-sm-4 { width: 33.333333333333%; }<br />
.col-sm-5 { width: 41.6666666666%; }<br />
.col-sm-6 { width: 50%; }<br />
.col-sm-7 { width: 58.333333333%; }<br />
.col-sm-8 { width: 66.66666666666%; }<br />
.col-sm-9 { width: 75%; }<br />
.col-sm-10 { width: 83.33333333333%; }<br />
.col-sm-11 { width: 91.6666666666%; }<br />
.col-sm-12 { width: 100%; }<br />
<br />
.sideTable {<br />
width: 27.5%;<br />
float: right;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 95%;<br />
margin-left: 2.5%;<br />
margin-right: 2.5%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
}<br />
<br />
.inverse-direction {<br />
float: right;<br />
}<br />
<br />
.nav-tab {<br />
width: 100%;<br />
}<br />
<br />
@media (max-width: 1050px){<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.sideTable {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-table {<br />
width: 30%;<br />
margin-left: 1.6666666%;<br />
margin-right: 1.6666666%;<br />
float: left;<br />
}<br />
<br />
.content-section {<br />
float: left;<br />
}<br />
<br />
.inverse-direction {<br />
clear: both;<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { clear: both; width: 100%; margin: auto; }<br />
}<br />
<br />
div>a>img {<br />
width: 75%;<br />
height: auto;<br />
margin: auto;<br />
}<br />
<br />
@media (max-width: 850px){<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
clear: both;<br />
}<br />
}<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=File:FR_-_FR_5G_Networks_v0.1.pdf&diff=11315File:FR - FR 5G Networks v0.1.pdf2019-07-22T17:53:15Z<p>Kpere060: </p>
<hr />
<div></div>Kpere060https://wiki.gccollab.ca/index.php?title=Tendances_Technologiques/R%C3%A9seaux_5G&diff=11314Tendances Technologiques/Réseaux 5G2019-07-22T17:47:55Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[en:Technology_Trends/Zero_Trust_Network]]<br />
<br />
<div class="mw-collapsible" data-highlightedtext="Afficher la Vue Détaillée" data-collapsetext="Cacher la Vue Détaillée"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>Reseaux 5G</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>[[Technology_Trends/Face_Recognition|5G Networks]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Publié</td><br />
</tr><br />
<tr><br />
<th>Version original</th><br />
<td>25 juin 2019</td><br />
</tr><br />
<tr><br />
<th>Mise à jour</th><br />
<td>22 juillet 2019</td><br />
</tr><br />
<tr><br />
<th>Publication officiel</th><br />
<td>[[Media:FR_-_FR_5G Networks v0.1.pdf|Reseaux_5G.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">Cette page est sujet à des mises à jours. On encourage toutes rétroactions. Veuillez utiliser la page de discussion pour apporter des commentaires et des suggestions. Une fois que cette version soit approuvée et finalisée, elle va être traduit.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>5G Networks</b>également appelée 5G NR (pour l’anglais new radio), représente la technologie cellulaire sans fil de 5e génération.<ref>Sega, Sashca, “What is 5G”, PCMag, 20 March 2018.<i> [https://www.pcmag.com/article/345387/what-is-5g/ What is 5G]</i></ref> Dans l’univers mobile, une génération (G) indique généralement une rupture de compatibilité, autrement dit que les utilisateurs auront besoin de renouveler leurs appareils<ref>Segan, S., & Segan, S. (2019, January 07). 5G vs. 5G E vs. 5GHz: What's the Difference? <i>[https://www.pcmag.com/article/365754/5g-vs-5g-e-vs-5ghz-whats-the-difference/ 5G vs. 5G E vs. 5GHz: What's the Difference?]</i></ref>. Bien que les générations de technologies sans fil aient été techniquement définies par leur vitesse de transmission de données, elles ont toutes été marquées par une rupture dans les méthodes de chiffrement, ou « interfaces radio », qui les rendent incompatibles avec la génération précédente.<ref>Sega, Sashca, “What is 5G”, PCMag, 20 March 2018.<i> [https://www.pcmag.com/article/345387/what-is-5g/ What is 5G]</i></ref></p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Cacher la Vue Détaillée</span></div><br />
</div><br><br><br />
<br />
<h2>Sommaire opérationnel</h2><br />
<p>Les fournisseurs d’infrastructures Internet se rendent compte du fait que le réseau de quatrième génération (4G) actuel n’est pas en mesure de faire face à l’augmentation du trafic de données mobiles. D’ici 2020, le trafic projeté de données sera trop important pour que les réseaux 4G puissent les prendre en charge. Afin de résoudre ce problème, les fournisseurs et les consommateurs devront passer aux réseaux 5G.</p><br />
<p class="highlighted mw-collapsible-content">1G – Voix analogique : Date de la fin des années 1970; les premiers téléphones cellulaires ne fournissaient que des appels vocaux. Des années plus tard, certains téléphones cellulaires 1G fournissaient à l’occasion des services de données sans fil à un ordinateur portable si on les branchait au modem commuté de l’ordinateur, mais les connexions étaient précaires et, si elles fonctionnaient, la vitesse de transfert des données était très basse <ref>Encyclopedia. (n.d.). Retrieved from <i>[https://www.pcmag.com/encyclopedia/term/55406/cellular-generations/ cellular generations]</i></ref>.</p><br />
<p class="highlighted mw-collapsible-content">2G – Réseaux numériques : Arrivée d’une nouvelle technologie numérique pour la transmission sans fil également connue sous le nom de Global System for Mobile communication (GSM). La technologie GSM est devenue la norme de base pour l’élaboration ultérieure des normes applicables au sans-fil. La 2G pouvait prendre en charge un débit de données de 14,4 kb/s à 64 kb/s (maximum), ce qui est suffisant pour les services de SMS et de messagerie. Des réseaux de données (GPRS, EDGE, IS-95B) ont été ajoutés et communément appelés technologies 2.5G et 2.75G.<ref>Rajiv, & Noman, S. (2018, December 14). Evolution of wireless technologies 1G to 5G in mobile communication. Retrieved from <i>[https://www.rfpage.com/evolution-of-wireless-technologies-1g-to-5g-in-mobile-communication/]<i></ref></p><br />
<p class="highlighted mw-collapsible-content">3G – Réseaux de données IP haute vitesse : La troisième génération offre un accès plus rapide à Internet avec des débits en aval allant jusqu’à 1 Mb/s et même plus, selon la version de la 3G<ref>Encyclopedia. (n.d.). Retrieved from <i>[https://www.pcmag.com/encyclopedia/term/55406/cellular-generations/ cellular generations]</i></ref>. La troisième génération de communications mobiles a commencé avec les Universal Mobile Terrestrial / Telecommunication Systems (UMTS). Après l’arrivée des systèmes de communication mobile 3G, les téléphones intelligents ont gagné en popularité dans le monde entier. Des applications ont été développées expressément pour les téléphones intelligents, pour le clavardage multimédia, la messagerie électronique, les appels vidéo, les jeux, les médias sociaux et les soins de santé.<ref>Rajiv, & Noman, S. (2018, December 14). Evolution of wireless technologies 1G to 5G in mobile communication. Retrieved from <i>[https://www.rfpage.com/evolution-of-wireless-technologies-1g-to-5g-in-mobile-communication/]<i></ref>.<br />
<p class="highlighted mw-collapsible-content">Afin d’améliorer le débit de données dans les réseaux 3G existants, deux autres améliorations technologiques ont été apportées au réseau : l’accès par paquets en liaison descendante haut débit ou HSDPA pour High Speed Downlink Packet Access et l’accès par paquets en liaison montante haut débit ou HSUPA pour High Speed Uplink Packet Access, mis au point et déployés sur les réseaux 3G, appelés 3.5G. L’évolution suivante de la 3G, appelée 3.75, est une version améliorée du réseau 3G avec accès par paquets haut débit évolué ou HSPA+ pour High Speed Packet Access Plus. Plus tard, ce système deviendra le puissant 3.9G qu’on appellera technologie d’évolution à long terme ou LTE pour Long Term Evolution.</p><br />
<p class="highlighted mw-collapsible-content">4G – Augmentation de la bande passante mobile : Les systèmes 4G sont des versions améliorées des réseaux 3G développés par IEEE, qui offrent un débit de données plus élevé et sont capables de gérer des services multimédias avancés. Les technologies sans fil LTE et LTE évoluée sont utilisées dans les systèmes de 4e génération. De plus, la 4G est compatible avec les versions précédentes, ce qui facilite le déploiement et la mise à niveau des réseaux LTE et LTE évoluée.<ref>Rajiv, & Noman, S. (2018, December 14). Evolution of wireless technologies 1G to 5G in mobile communication. Retrieved from <i>[https://www.rfpage.com/evolution-of-wireless-technologies-1g-to-5g-in-mobile-communication/]<i></ref>. Il s’agit essentiellement d’une extension de la technologie 3G avec plus de bande passante et de services. L’une des principales différences technologiques de la 4G par rapport à la 3G réside dans l’élimination de la commutation de circuits et l’utilisation d’un réseau IP de bout en bout. Ainsi, la 4G a permis de traiter les appels vocaux comme n’importe quel autre type de média audio en continu au moyen de la commutation de paquets sur Internet, d’un réseau local ou d’un réseau étendu, par voix sur IP.<ref>NA. (2008, 08 23). 1G, 2G, 3G, 4G - The Evolution of Wireless Generations. Retrieved from Support.Chinavision:<i> https://support.chinavasion.com/index.php?/Knowledgebase/Article/View/284/42/1g-2g-3g-4g---the-evolution-of-wireless-generations</i></ref></p><br />
<br />
<p>5G – Spectre sans licence : Le réseau 5G présente trois avantages principaux par rapport à son prédécesseur :<br />
<ul><br />
<li>il peut offrir une vitesse de téléchargement de données de 10 à 20 Gb/s;</li><br />
<li>il offre une faible latence, inférieure à une milliseconde, un point crucial pour les applications qui doivent être mises à jour en temps réel;</li><br />
<li>comme cette technologie utilise des ondes radio millimétriques pour la transmission, elle peut fournir une bande passante supérieure sur les réseaux LTE actuels ainsi que des débits de données beaucoup plus élevés.</li><br />
</ul><br />
Concrètement, les réseaux 5G seront ainsi en mesure de fournir un accès à l’entreposage en nuage et d’exécuter des applications opérationnelles et auront le pouvoir d’exécuter virtuellement des tâches plus complexes. Le réseau 5G offre la possibilité de connecter 100 fois plus d’appareils que les ondes 4G LTE. Il peut permettre également de réduire la consommation d’énergie de 90 % par rapport à la 4G tout en assurant des vitesses Internet uniquement atteintes actuellement grâce à une connexion réseau directe par l’intermédiaire d’un câble à fibres optiques.<br />
</p><br />
<p>La 5G est de plus prête à transformer le monde des appareils de l’Internet des objets. L’utilisation des ondes radio millimétriques et du réseau central 5G non seulement accélère la transmission de données, mais accroît aussi la fiabilité de connexion. On améliore ainsi la connectivité des nouveaux types d’applications mobiles, l’automatisation industrielle et les véhicules autonomes, entre autres. Pour l’essentiel, toute application de l’Internet des objets qui utilise actuellement la technologie LPWA (Low Power Wide Area) connaîtra des améliorations progressives. De nombreux fournisseurs d’appareils cellulaires sont prêts à lancer des téléphones intelligents et d’autres appareils capables de se connecter aux réseaux 5G d’ici la fin de 2019. Actuellement, des entreprises comme AT&T ont lancé la 5G Evolution, une évolution de la 4G LTE qui n’offre cependant pas toutes les capacités de la 5G.</p><br />
<h2>Sommaire technique</h2><br />
<p>Tout comme les réseaux cellulaires actuels, la 5G divise le territoire en petits secteurs dans lesquels les appareils se connectent aux sites cellulaires. Ces sites cellulaires sont alors en mesure de transmettre des données chiffrées en utilisant des ondes radioélectriques. La différence entre la 5G et son prédécesseur réside dans sa capacité à transmettre ces ondes radio à des fréquences beaucoup plus élevées, ce qui se traduit par des vitesses de transmission de données supérieures, même à celle des réseaux de fibres optiques actuels, soit 1 Gb/s. Cette évolution, aussi petite soit-elle, a déjà eu son application dans le monde réel lorsque Sprint a lancé une fonctionnalité similaire avec sa technologie LAA. Dans le spectre des ondes millimétriques, ces fréquences se situent entre 30 et 300 GHz.</p><br />
<p>Deux séries de fréquences sont en cours d’approbation par la Federal Communications Commission des États-Unis. Les réseaux d’ondes 5G de bande basse et 5G centrales utilisent des fréquences de 600 MHz à 6 GHz, surtout 3,5 à 4,2 GHz. Les ondes centrales n’affecteront probablement pas beaucoup le matériel existant utilisé pour le sans-fil. Bien qu’il soit nécessaire d’utiliser des amplificateurs de puissance pour éviter une forte atténuation du signal, les ondes millimétriques constitueront une réelle évolution des technologies sans fil au point de nécessiter un tout nouveau système d’antennes, de câbles et d’amplificateurs.</p><br />
<br />
<p>Les réseaux 5G seront utilisés avec des sites cellulaires beaucoup plus petits. Les ondes radio à haute fréquence ne peuvent parcourir que de courtes distances en comparaison des ondes 4G LTE, à plus basse fréquence. Comme le signal 5G ne peut être transmis que sur la longueur d’un pâté de maisons ou à peu près et ne peut pas pénétrer dans les bâtiments, il n’y aura plus besoin de grandes tours, au profit de petites tours de téléphonie cellulaire installées sur chaque pâté de maisons et à l’intérieur des bâtiments. Cela signifie également que la vitesse des réseaux individuels sera plus importante qu’auparavant.</p><p class="inline">Un article rédigé par des professeurs des universités de Waterloo, Carleton et Ozyegin explique que les réseaux 5G pourraient transformer complètement l’architecture cellulaire actuelle. Ils expliquent que pour que la 5G puisse fonctionner avec une telle demande de bande passante sur le réseau par les appareils de l’Internet des objets, l’architecture cellulaire traditionnelle peut être divisée en une architecture à deux niveaux : 1) un niveau macrocellulaire, pour la communication entre la station et l’appareil et 2) un niveau équipements, pour la communication entre appareils.</p> <p class="highlighted inline mw-collapsible-content"> Cependant, la situation n’est pas sans risques pour la sécurité. La communication entre appareils nécessite une sécurité réseau plus complexe que l’actuelle. La communication est possible grâce à l’utilisation de relais entre appareils; les appareils connectés se servent les uns des autres pour retransmettre des données, ce qui crée un réseau maillé ad hoc. De cette façon, les appareils peuvent communiquer entre eux dans une largeur de bande cellulaire autorisée sans l’utilisation d’une station de base. Il s’agit d’un changement radical par rapport à l’architecture cellulaire traditionnelle où les téléphones cellulaires se connectent à une tour de téléphonie cellulaire.</p><br />
<br />
<p class="highlighted mw-collapsible-content">Auparavant, la communication entre appareils n’était utilisée qu’à de rares occasions. La demande à cet effet a récemment augmenté au gré de la commercialisation d’applications contextuelles. Ces applications nécessitent généralement à la fois des services de localisation et la capacité de communiquer avec d’autres appareils. En accédant à cette capacité par la technologie nécessaire, on réaliserait des économies puisque ce ne serait pas tous les appareils du réseau qui auraient besoin d’être connectés à la station de base. La communication entre appareils pourrait également jouer un rôle dans l’infonuagique mobile et rendre le partage des ressources plus efficace. Si un dispositif se trouve en périphérie d’un site cellulaire ou dans une zone encombrée, la communication entre appareils pourrait libérer la station de base d’une importante consommation de ressources.</p><br />
<br />
<br />
<h2>Utilisation par l'industrie</h2><br />
<p>Plusieurs fournisseurs de télécommunications aux États-Unis ont commencé à développer et à mettre à l’essai des réseaux 5G. Des fournisseurs de télécommunications comme Verizon, AT&T et Sprint ont tous fait des progrès dans ce domaine et mènent chacun des projets de recherche pour tester les réseaux. Verizon, AT&T, Sprint et T-Mobile ont tous commencé à déployer la 5G sur différents marchés et continueront à le faire tout au long de 2019. Verizon dispose de la 5G fixe et mobile dans quelques régions. AT&T dispose de la 5G mobile pour certaines entreprises dans certaines villes alors que Sprint déploie la 5G dans certaines zones. T-Mobile lancera la 5G commerciale au second semestre 2019 et devrait avoir une couverture nationale en 2020.</p><br />
<br />
<br />
<p class="highlighted mw-collapsible-content">Sprint et T-Mobile ont investi dans la 5G basse fréquence, qui offre des vitesses plus lentes, mais une autonomie accrue. Les deux entreprises pourront ainsi offrir la 5G dans des zones moins denses à moindre coût. Sprint a investi dans la bande centrale, la 5G 2,5 GHz, tandis que T-Mobile compte utiliser la 4G de bande basse 600 MHz en plus de la 5G haute fréquence dans les zones plus denses. En comparaison, Verizon et AT&T utiliseront principalement des bandes de fréquences beaucoup plus élevées, comme la bande des 28 GHz.</p><p class="highlighted mw-collapsible-content"><br />
Au Canada, la 5G ne sera pas offerte à grande échelle avant 2020. Bien que la 5G ait le potentiel d’atteindre des vitesses de 20 Gb/s, elle ne sera probablement qu’à 6 Gb/s à son premier déploiement. Comme pour les technologies similaires, il faudra jusqu’à 10 ans pour que la nouvelle technologie atteigne sa pleine maturité.</p><br />
<p>L’une des utilités de la 5G est la gestion des sources d’énergies renouvelables comme l’énergie solaire et éolienne en compensation de la consommation d’énergie électrique. Étant donné que la 5G permettra la collecte de données, il sera possible de l’analyser pour déterminer les moments de forte et de faible consommation d’électricité. L’information servira ensuite à planifier un réseau électrique plus cohérent et plus fiable.</p><br />
<br />
<p>La rapidité des réseaux 5G et leur faible latence inhérente rendront également possible la chirurgie à distance. Les habitants des milieux ruraux auront ainsi accès à des chirurgiens et à des spécialistes qui ne pratiquent habituellement que dans les grandes villes. La première chirurgie à distance réussie a déjà été réalisée en Chine. Le réseau 5G est la pièce manquante au casse-tête de la chirurgie à distance. Pour réussir, une chirurgie à distance a besoin d’un patient, d’un chirurgien, d’un robot et d’une connexion Internet stable et ultrarapide.</p><br />
<p>Que se passerait-il si les voitures autonomes pouvaient signaler leurs intentions ou diffuser leur itinéraire à d’autres voitures autonomes? La 5G pourrait en faire une réalité et contribuerait ainsi à rendre les routes plus sûres. Les conducteurs pourraient aussi montrer aux autres à proximité dans quelle direction ils vont. La chose pourrait se faire lorsque nous utilisons nos téléphones pour trouver le chemin à emprunter pour parvenir à notre destination. Le téléphone pourrait également diffuser ces informations par la 5G jusqu’aux téléphones et aux voitures autonomes à proximité.</p><br />
<br />
<h2>Utilisation par le gouvernement du Canada</h2><br />
<p class="highlighted mw-collapsible-content">Les réseaux mobiles 5G (ou de 5e génération) ne sont pas encore accessibles au Canada ni dans la plupart des pays du monde d’ailleurs. Malgré tout, le gouvernement du Canada s’est préparé à son arrivée. Le Canada est au même point que les autres pays développés dans la préparation à la 5G.</p><br />
<b>Innovation, Sciences et Développement économique Canada (ISDE) et gestion du spectre mobile</b><br />
<br />
<p>La demande d’applications et de contenu numériques continue d’augmenter, tant au Canada qu’à l’étranger, ce qui constitue le principal moteur de l’essor de la technologie 5G. Les téléphones intelligents et autres appareils cellulaires de même que les tablettes, les appareils informatiques personnels (Internet des objets) et la connectivité entre machines jouent un rôle de plus en plus crucial dans la vie quotidienne des entreprises et citoyens canadiens. Dans un contexte de hausse de l’utilisation de ces appareils, le taux de croissance composé du trafic de données mobiles a été calculé à 54 % par an. En tant que telle, la création de spectres ou la conversion de spectres existants (ou de radiofréquences qu’empruntent les données mobiles) par les organismes de réglementation nationaux est cruciale si on veut répondre à la demande et éviter tout effet nuisible sur l’économie.</p><br />
<p class="highlighted mw-collapsible-content">Tout le spectre radioélectrique mondial est attribué par l’Union internationale des télécommunications (UIT). Au Canada, les téléphones cellulaires et les radiofréquences sont réglementés par Innovation, Sciences et Développement économique Canada (ISDE), qui fait partie de l’UIT. Ce ministère supervise également la délivrance des licences et l’installation des tours de téléphonie cellulaire, effectue des évaluations de l’impact environnemental et de l’utilisation du sol pour les projets d’installation de telles tours ou d’autres infrastructures de téléphonie cellulaire et veille à ce que l’équipement satisfasse à toutes les exigences réglementaires. Il est également responsable de l’attribution de licences d’utilisation du spectre aux entreprises de télécommunications sans fil au Canada. En 2015, après avoir consulté les entreprises de télécommunications et les télédiffuseurs, il a été décidé que le Canada utiliserait à nouveau la portion de 600 MHz du spectre de télévision pour les services mobiles. La mise aux enchères de ce spectre aux entreprises de télécommunications mobiles s’est achevée en avril 2019 et montre que le gouvernement du Canada est conscient de l’importance sans cesse croissante de la technologie mobile et de la nécessité d’élargir les bandes de fréquences.<br />
</p><br />
<p>Toutefois, avec l’arrivée de la 5G à l’horizon 2023, l’année où la plupart des opérateurs en Amérique du Nord ont l’intention de lancer la 5G à grande échelle, il faudra encore plus d’espace sur le spectre:</p><br />
<p><i>“Il est essentiel d’avoir un nouveau spectre si on veut assurer le bon fonctionnement des services mobiles terrestres de la cinquième génération (5G). À l’échelle mondiale, on s’affaire sans relâche à déterminer les spectres qui conviendraient, notamment les bandes qui peuvent être utilisées dans le plus grand nombre de pays possible pour permettre l’itinérance mondiale et favoriser des économies d’échelle. Des travaux sont en cours dans le monde entier pour viser l’harmonisation du spectre à utiliser pour la 5G. Les services 5G devraient couvrir un large éventail d’applications”- 5G Americas<br />
</i></p><br />
<p class="highlighted mw-collapsible-content">En juin 2017, ISDE a lancé des consultations concernant la libération future de fréquences supplémentaires, au-delà des 648 MHz actuellement utilisés. ISDE souhaitait se renseigner sur les quantités qui devraient être nécessaires ainsi que sur la nécessité d’éventuelles considérations politiques et réglementaires, à mesure que de nouveaux modèles commerciaux et de nouvelles applications réseau feraient leur arrivée. Divers intervenants ont pris part aux consultations et ont manifesté leur appui à la proposition du gouvernement du Canada concernant la libération des bandes de fréquences de 28 GHz, 37 à 40 GHz et 64 à 71 GHz. Le ministre d’ISDE, l’honorable Navdeep Bains, a déclaré qu’aucune décision définitive ne sera prise avant la conférence mondiale des radiocommunications à l’automne 2019 et que les consultations sur ces questions durent généralement deux ans. Toutefois, certains acteurs de premier plan aimeraient que ce processus s’accélère. Un représentant de Telus a eu ces mots : « Il faut prendre des mesures réglementaires immédiates et définitives pour que le Canada récolte les fruits de sa rapidité d’action dans la nouvelle économie numérique mondiale. »</p><br />
<b>Sécurité publique et inquiétudes par rapport à l’espionnage</b><br />
<p class="highlighted mw-collapsible-content">En mai 2019, le gouvernement du Canada a entrepris une étude de la cybersécurité de la technologie 5G et des fournisseurs potentiels d’équipement. Actuellement, les principaux fournisseurs mondiaux sont Nokia, Ericsson, Samsung, Qualcomm et Huawei, cette dernière société ayant la situation la plus nébuleuse de toutes. En 2018, l’Australie, la Nouvelle-Zélande et les États-Unis ont tous interdit l’utilisation d’équipements de télécommunications Huawei dans leurs réseaux 5G par crainte que l’entreprise n’ait des liens avec le gouvernement chinois, qui pourrait potentiellement utiliser Huawei pour réaliser ses activités d’espionnage ou attaquer des infrastructures publiques essentielles au moyen d’un programme malveillant. Huawei a toujours nié avec véhémence ces allégations. Le Royaume-Uni a ordonné l’interdiction partielle de Huawei dans les composantes centrales de son réseau 5G. Les autres pays européens se sont jusqu’à présent abstenus de lui emboîter le pas.</p><br />
<p>Bien que ce soit normalement la responsabilité des entreprises canadiennes comme Bell, Rogers et Telus d’assurer la sécurité de leurs réseaux, le gouvernement du Canada a l’obligation de veiller à la sécurité publique, dont la cybersécurité est une composante. Le 1er mai 2019, selon le ministre de la Sécurité publique Ralph Goodale, le ministre responsable de la sécurité nationale et la [https://www.publicsafety.gc.ca/cnt/rsrcs/pblctns/ntnl-cbr-scrt-strtg/index-en.aspx/ Stratégie nationale de cybersécurité du Canada], l’examen sur la sûreté de la 5G, qui comprend le rôle potentiel de Huawei, est en cours, et une décision finale est attendue d’ici l’automne 2019. Indépendamment de cette décision, les opérateurs et le gouvernement du Canada devront déployer des efforts pour assurer la sécurité des réseaux, comme c’est le cas pour la 4G LTE actuellement.</p><br />
<b>Autres investissements et initiatives</b><br />
<p>Le 19 mars 2018, le gouvernement du Canada a annoncé l’octroi de fonds pour le projet du corridor d’essai 5G entre le Québec et l’Ontario. Cet octroi représente une étape dans l’adoption de la prochaine génération de technologie sans fil. Le gouvernement du Canada établit des partenariats avec divers acteurs privés en guise d’exemple de collaboration entre toutes les parties prenantes. La 5G exigera une énorme et coûteuse refonte de l’infrastructure et il faut en tenir compte. </p><br />
<br />
<p>L’Association canadienne des télécommunications sans fil a lancé le Conseil 5G Canada pour favoriser la collaboration dans les travaux de mise en place de la 5G. La date de sortie de la technologie est toujours prévue d’ici 2020. Le gouvernement du Canada devra encore se pencher sur la manière dont il prendra en charge les fréquences radioélectriques entre 600 et 3 500 MHz, qui sont nécessaires pour les réseaux 5G. Cette gamme de fréquences est cruciale, car la fréquence de 600 MHz est l’une des fréquences les plus élevées qui peuvent s’utiliser dans les régions rurales et éloignées du pays.</p><br />
<p class="highlighted mw-collapsible-content">Le gouvernement canadien a annoncé un montant pouvant atteindre 40 millions de dollars pour financer la recherche de Nokia sur la technologie 5G au Canada. Nokia a lancé de multiples projets sur le routage des données dans les réseaux optiques ainsi que la création d’outils de cybersécurité qui protégeront les réseaux de télécommunications dans leur transition vers la 5G.</p><br />
<h2>Répercussions pour les agences gouvernementales</h2><br />
<br />
<h3>Services partagés Canada (SPC)</h3><br />
<p>SPC aura un rôle important à jouer pour que les ministères canadiens disposent des outils, de l’infrastructure et de l’architecture nécessaires au lancement à grande échelle de la 5G au cours des prochaines années. Ainsi, le déploiement de la 5G aura des répercussions majeures sur SPC. </p><br />
<h4>Proposition de valeur</h4><br />
<p class="highlighted mw-collapsible-content">Comme il est mentionné dans le sommaire opérationnel, la 5G offre trois avantages principaux par rapport au réseau 4G actuel : une vitesse accrue, une latence moindre et la possibilité de connecter beaucoup plus d’appareils à la fois. Concrètement, les réseaux 5G seront ainsi en mesure de fournir un meilleur accès à l’entreposage en nuage (et à l’informatique en périphérie) et d’exécuter des applications opérationnelles avec un délai de réponse en temps réel amélioré et auront le pouvoir d’exécuter virtuellement des tâches plus complexes. Ces avantages vont de pair avec la volonté d’avoir un gouvernement du Canada ouvert et d’augmenter le partage des données et la collaboration depuis n’importe quel appareil (y compris les appareils mobiles), comme l’indique le [https://www.canada.ca/en/government/system/digital-government/digital-operations-strategic-plan-2018-2022.html#ToC9_2/ Plan stratégique des opérations numériques de 2018 à 2022. ] </p><br />
<p>SPC a apporté des changements importants dans la modernisation des centres de données du gouvernement du Canada ainsi que dans l’offre de services infonuagiques, sur le plan du traitement et du stockage des données. Cependant, au fil de l’évolution technologique, l’informatique en périphérie viendra compléter ces deux modèles. D’ici 2022, plus de 50 % des données générées par l’entreprise seront créées et traitées à l’extérieur du centre de données ou du nuage, selon une recherche effectuée par Gartner. L’informatique en périphérie s’impose de plus en plus comme solution aux problèmes de latence entre machines. La 5G améliorera par sa nature même la bande passante et donc les problèmes de latence, ce qui permettra de prendre en charge une plus grande densité d’appareils, dont ceux en périphérie. Grâce à la 5G, les données atteindront leurs points terminaux (qu’il s’agisse d’un nuage ou d’un centre de données) plus rapidement et accéléreront ainsi le traitement et le stockage. </p><br />
<br />
<h4>Difficultés</h4><br />
<p class="highlighted mw-collapsible-content">Premièrement, les appareils compatibles devront être redistribués aux fonctionnaires de partout au Canada. Les appareils actuels ne seront en effet pas compatibles avec les réseaux 5G, et les mises à jour automatiques ne seront pas possibles. Seuls les appareils compatibles avec la 5G peuvent être utilisés sur les réseaux 5G (ils peuvent également être utilisés sur les réseaux 4G). Toutefois, il ne sera probablement pas nécessaire de mettre les appareils à niveau tout de suite. Les premières instances de la 5G utiliseront les réseaux et l’équipement 4G et non des réseaux ou de l’équipement particuliers. Quoi qu’il en soit, le renouvellement complet de tous les appareils du gouvernement du Canada sur à peu près la même période sera un projet d’envergure sur le plan logistique et financier. Néanmoins, il faudra aller de l’avant si le gouvernement veut pouvoir continuer d’exploiter le numérique au profit de la population canadienne, l’un des principaux thèmes stratégiques énoncés dans le [https://www.canada.ca/en/government/system/digital-government/digital-operations-strategic-plan-2018-2022.html/ Plan stratégique des opérations numériques de 2018 à 2022] aussi décrit dans [http://www.clerk.gc.ca/local_grfx/bp2020/bp2020-eng.pdf/ Objectif 2020].</p><br />
<p class="highlighted mw-collapsible-content">Deuxièmement, la 5G fonctionnera en partie sur les ondes millimétriques, qui ont des fréquences comprises entre 30 et 300 GHz. Les ondes millimétriques ont généralement une faible portée et sont sensibles aux interférences et aux blocages causés par des objets tels que les bâtiments, les arbres, voire la pluie et les nuages. Il devient donc difficile d’avoir une bonne couverture pour les appareils. De plus, les interférences et les problèmes de portée peuvent nuire à la tenue de certaines activités, notamment les prévisions météorologiques, les initiatives SmartCities, les procédures médicales et les opérations militaires et policières. Pour contourner le problème, il faudra déployer de petites cellules, terme forgé décrivant l’utilisation de nombreuses petites antennes, petites tours et petits émetteurs dans une zone dense. La situation entraînera une modification considérable des structures de réseau actuelles et pourrait compliquer passablement les choses dans certaines régions éloignées ou rurales.</p><br />
<p>Enfin, pour terminer, l’arrivée de la 5G pourrait susciter des inquiétudes en matière de santé et de sécurité au travail chez les employés et les citoyens en général, par rapport à l’exposition accrue aux rayonnements en raison du nombre élevé d’antennes et de tours situées plus près du lieu de résidence et de travail des gens. Même avec la technologie 4G actuelle et les équipements de télécommunications existants, certains pensent que les appareils et leurs équipements représentent une menace pour la santé humaine. Cependant, aucune recherche n’a actuellement démontré de risque réel. Santé Canada a rédigé des lignes directrices sur l’exposition humaine sécuritaire aux radiofréquences. Les radiofréquences de la 5G sont bien en dessous de la gamme de sécurité de 300 GHz. </p><br />
<br />
<h4>Considérations</h4><br />
<br />
<p>C’est sur ses centres de données que les réseaux 5G auront le plus important effet pour SPC. La 5G promet de prendre en charge un trafic plus élevé sur le réseau à des vitesses plus élevées avec une faible latence. Il faut aussi en comprendre que les applications commenceront à être conçues pour les réseaux 5G. Le passage à la 5G ne changera rien aux tâches à accomplir pour les centres de données, sinon leur échelle. Les centres de données devront se décentraliser et suivre un modèle à deux ou trois niveaux afin de fournir un traitement à faible latence à la périphérie du réseau tout en maintenant un rôle central d’agrégation et de coordination des données. Ils devront aussi augmenter la bande passante, le pouvoir de traitement et le stockage. </p><br />
<p class="highlighted mw-collapsible-content">La 5G transformera également l’architecture cellulaire traditionnelle. Pour avoir une architecture à deux niveaux, comme l’explique le sommaire technologique, il faut accroître la sécurité. En effet, l’utilisation d’appareils pour se connecter les uns aux autres afin d’établir une connexion réseau fait en sorte qu’il n’y a pas qu’une seule station de base centrale à protéger. Dorénavant, le fournisseur devra également se protéger contre les appareils qui établissent eux-mêmes les connexions. La sécurité sera une question de première importance pour SPC à l’arrivée de la 5G, car tous les appareils de l’Internet des objets seront connectés. Le nombre accru d’appareils fait augmenter le trafic sur le réseau.</p><br />
<p class="highlighted mw-collapsible-content">Des entreprises comme Cisco et Ericsson ont commencé à utiliser des réseaux SDN et la virtualisation des fonctions réseau parce que ce sont des solutions flexibles qui peuvent prendre en charge dynamiquement un nombre élevé d’appareils. Les réseaux SDN découplent le matériel du logiciel, ce qui signifie que les tâches peuvent être effectuées dans le nuage ou dans des grappes de serveurs. La virtualisation des fonctions réseau, généralement utilisée en combinaison avec les réseaux SDN, transforme les fonctions réseau de manière à pouvoir être exécutée dans des machines virtuelles au lieu de dépendre du matériel. Il s’agit là d’options viables pour SPC à mesure que le Ministère migre vers le nuage à l’aube de l’ère 5G.</p><br />
<p>Les réseaux 5G ne sont pas qu’une simple évolution technologique : c’est une révolution. La 5G a le potentiel de modifier de fond en comble la façon dont les données sont transmises et traitées, et par qui (ou quoi). SPC devra s’informer sur le potentiel des réseaux 5G d’entreprise, à l’instar de nombreuses entreprises qui ont commencé des démarches en ce sens. En raison de certaines activités essentielles à la mission de divers ministères, comme la Défense nationale et la Gendarmerie royale du Canada, l’arrivée des réseaux 5G privés peut se révéler nécessaire si on veut transmettre des renseignements sensibles ou lorsque l’infrastructure et les réseaux publics ne sont pas fiables ou jugés sûrs. </p><br />
<p class="highlighted mw-collapsible-content">Malgré les avantages de la 5G, il y aura des coûts initiaux et financiers et humains. Non seulement la mise à jour et le déploiement de l’infrastructure et des appareils actuels seront nécessaires, mais la densification de l’infrastructure sera également inévitable avec la technologie 5G. En raison des difficultés liées aux distances de transmission et aux interférences, il peut devenir nécessaire de déployer de petites cellules (tours et antennes radio), éventuellement sur chaque bâtiment gouvernemental à la grandeur du pays. Les répercussions sur les budgets et la main-d’œuvre ne font aucun doute. </p><br />
<p class="highlighted mw-collapsible-content">Enfin, on pourra tirer des leçons de l’expérience des premiers qui adopteront la 5G. À l’heure actuelle, la technologie 5G est loin du stade de maturité et n’est pas encore déployée à grande échelle ailleurs dans le monde. Cependant, en avril 2019, la Corée du Sud est devenue le premier pays à adopter pleinement la 5G et devrait compter près d’un million d’utilisateurs d’ici la fin juin 2019. Dans les premiers mois qui ont suivi le lancement, les utilisateurs se sont plaints de la couverture et de la vitesse, principalement en raison du manque de stations de base (tours et antennes) en dehors des zones urbaines à forte densité de population. Les opérateurs ont réagi en installant de 3 000 à 4 000 nouvelles stations par semaine afin de répondre à la demande et de résoudre les problèmes. On voit ainsi l’importance d’avoir une bonne infrastructure en place avant l’arrivée de la technologie afin d’éviter la frustration des clients qui pourraient ainsi vouloir lui tourner le dos.</p><br />
<br />
<h2>References</h2><br />
<br />
<div style= "display:none"><ref>Cheng, Roger "What is 5G? Here are the basics", cnet, 9 February 2018.<i>https://www.cnet.com/how-to/5g-network-technology-here-are-the-basics/</i></ref><br />
<ref>M. N. Tehrani, M. Uysal and H. Yanikomeroglu, "Device-to-device communication in 5G cellular networks: challenges, solutions, and future directions," in IEEE Communications Magazine, vol. 52, no. 5, pp. 86-92, May <i>http://ieeexplore.ieee.org/abstract/document/6815897/</i></ref><br />
<ref>Oliveira, Michael, “Canadian wireless providers mum as U.S. companies announce 5G launch dates”, The Canadian Press, April 2018. <i>https://www.theglobeandmail.com/business/article-canadian-wireless-providers-mum-as-us-companies-announce-5g-launch/</i></ref><br />
<ref>Pretz, Kathy, “5G: The Future of Communications Networks”, IEEE, 1 March 2017. <i>http://theinstitute.ieee.org/technology-topics/communications/5g-the-future-of-communications-networks</i></ref><br />
<ref>“5G spectrum guide – everything you need to know”, GSMA, 16 February 2018 <i>https://www.gsma.com/spectrum/5g-spectrum-guide/</i></ref><br />
<ref>Lavallee, Brian, “Data Centers to 5G: Bring It On!” Data Center Knowledge, 6 February 2018. <i>http://www.datacenterknowledge.com/industry-perspectives/data-centers-5g-bring-it</i></ref><br />
<ref>Williams, Kevin, “How 5G Will Usher In The Internet Of Power Delivery”, 21 June 2016 <i>https://www.forbes.com/sites/huawei/2016/06/21/how-5g-internet-will-change-the-way-we-generate-and-consume-power/#62c0aa608031</i></ref><br />
<ref>Cheng, Roger, “5G is finally starting to feel real”, 18 December 2018 <i>https://www.cnet.com/news/5g-is-finally-starting-to-feel-real/</i></ref><br />
<ref>Humphries, Matthew, “China Performs First 5G Remote Surgery”, 15 January 2019, <i>https://www.pcmag.com/news/365992/china-performs-first-5g-remote-surgery</i></ref><br />
<ref>Behar, Rose, “Everything you need to know about 5G in Canada”, 1 March 2019, <i>https://mobilesyrup.com/2018/03/01/everything-you-need-to-know-about-5g-canada/</i></ref><br />
<ref>CableFree “Introducing 5G Frequency Bands”<i>https://www.cablefree.net/wirelesstechnology/4glte/5g-frequency-bands-lte/</i></ref><br />
<ref>New York Intelligencer “5G is Going to Transform Smartphones – Eventually”, February 2019<i>http://nymag.com/intelligencer/2019/02/5g-is-going-to-transform-smartphones-eventually.html</i></ref><br />
<ref>ISED, Government of Canada, “Decision on Repurposing the 600 MHz Band”, August 2015<i>https://www.ic.gc.ca/eic/site/smt-gst.nsf/eng/sf11049.html</i></ref><br />
<br />
<ref>5G Americas, “5G Spectrum Recommendations”, April 2017.</ref><br />
<ref>ISED, Government of Canada, “Consultation on Releasing Millimetre Wave Spectrum to Support 5G”, June 2017.<i>https://www.ic.gc.ca/eic/site/smt-gst.nsf/eng/sf11298.html</i></ref><br />
<ref>Behar, Rose, “Here’s Why mmWave is Crucial to Canada’s 5G Future”, MobileSyrup, Oct 2017.<i>https://mobilesyrup.com/2017/10/16/mmwave-spectrum-explainer/</i></ref><br />
<ref>Bryan-Low, Cassell,Colin Packham, David Lague, Steve Stecklow & Jack Stubbs, Reuters, “Hobbling Huawei: Inside the U.S. war on China’s tech giant”, May 2019.<i>https://www.reuters.com/investigates/special-report/huawei-usa-campaign/</i></ref><br />
<ref>Fife, Robert & Steven Chase, The Globe and Mail, “Goodale says decision on Huawei 5G network to come before election”, May 2019.<i>https://www.theglobeandmail.com/politics/article-goodale-says-decision-on-huawei-to-come-before-election/</i></ref><br />
<ref>Business Wire, “Ixia, a Keysight Business, Delivers Network Performance Monitoring and Visibility to Remote Sites and Edge Computing”, Feb 2019.<i>https://finance.yahoo.com/news/ixia-keysight-business-delivers-network-161500291.html</i></ref><br />
<ref>Nordrum, Amy, Kristen Clark and IEEE Spectrum Staff, “Everything You Need to Know About 5G”, Jan 2017.<i>https://spectrum.ieee.org/video/telecom/wireless/everything-you-need-to-know-about-5g</i></ref><br />
<ref>Health Canada, Government of Canada, “Limits of Human Exposure to Radiofrequency Electromagnetic Energy in the Frequency Range from 3 kHz to 300 GHz - Safety Code 6 (2015)”, 2015<i>https://www.canada.ca/en/health-canada/services/environmental-workplace-health/consultations/limits-human-exposure-radiofrequency-electromagnetic-energy-frequency-range-3-300.html15</i></ref><br />
<br />
</div><br />
</div><br />
<br />
{{#css:<br />
<br />
#firstHeading::after{<br />
content:"Reconnaissance Faciale";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.highlighted{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Tendances_Technologiques/R%C3%A9seaux_5G&diff=11313Tendances Technologiques/Réseaux 5G2019-07-22T17:47:04Z<p>Kpere060: Created page with "{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}} en:Technology_Trends/Zero_Trust_Netwo..."</p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[en:Technology_Trends/Zero_Trust_Network]]<br />
<br />
<div class="mw-collapsible" data-highlightedtext="Afficher la Vue Détaillée" data-collapsetext="Cacher la Vue Détaillée"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>Reseaux 5G</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>[[Technology_Trends/Face_Recognition|5G Networks]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Publié</td><br />
</tr><br />
<tr><br />
<th>Version original</th><br />
<td>25 juin 2019</td><br />
</tr><br />
<tr><br />
<th>Mise à jour</th><br />
<td>22 juillet 2019</td><br />
</tr><br />
<tr><br />
<th>Publication officiel</th><br />
<td>[[Media:FR_-_FR_5G Networks v0.1.pdf|Reseaux_5G.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">Cette page est sujet à des mises à jours. On encourage toutes rétroactions. Veuillez utiliser la page de discussion pour apporter des commentaires et des suggestions. Une fois que cette version soit approuvée et finalisée, elle va être traduit.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>5G Networks</b>également appelée 5G NR (pour l’anglais new radio), représente la technologie cellulaire sans fil de 5e génération.<ref>Sega, Sashca, “What is 5G”, PCMag, 20 March 2018.<i> [https://www.pcmag.com/article/345387/what-is-5g/ What is 5G]</i></ref> Dans l’univers mobile, une génération (G) indique généralement une rupture de compatibilité, autrement dit que les utilisateurs auront besoin de renouveler leurs appareils<ref>Segan, S., & Segan, S. (2019, January 07). 5G vs. 5G E vs. 5GHz: What's the Difference? <i>[https://www.pcmag.com/article/365754/5g-vs-5g-e-vs-5ghz-whats-the-difference/ 5G vs. 5G E vs. 5GHz: What's the Difference?]</i></ref>. Bien que les générations de technologies sans fil aient été techniquement définies par leur vitesse de transmission de données, elles ont toutes été marquées par une rupture dans les méthodes de chiffrement, ou « interfaces radio », qui les rendent incompatibles avec la génération précédente.<ref>Sega, Sashca, “What is 5G”, PCMag, 20 March 2018.<i> [https://www.pcmag.com/article/345387/what-is-5g/ What is 5G]</i></ref></p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Cacher la Vue Détaillée</span></div><br />
</div><br><br><br />
<br />
<h2>Sommaire opérationnel</h2><br />
<p>Les fournisseurs d’infrastructures Internet se rendent compte du fait que le réseau de quatrième génération (4G) actuel n’est pas en mesure de faire face à l’augmentation du trafic de données mobiles. D’ici 2020, le trafic projeté de données sera trop important pour que les réseaux 4G puissent les prendre en charge. Afin de résoudre ce problème, les fournisseurs et les consommateurs devront passer aux réseaux 5G.</p><br />
<p class="highlighted mw-collapsible-content">1G – Voix analogique : Date de la fin des années 1970; les premiers téléphones cellulaires ne fournissaient que des appels vocaux. Des années plus tard, certains téléphones cellulaires 1G fournissaient à l’occasion des services de données sans fil à un ordinateur portable si on les branchait au modem commuté de l’ordinateur, mais les connexions étaient précaires et, si elles fonctionnaient, la vitesse de transfert des données était très basse <ref>Encyclopedia. (n.d.). Retrieved from <i>[https://www.pcmag.com/encyclopedia/term/55406/cellular-generations/ cellular generations]</i></ref>.</p><br />
<p class="highlighted mw-collapsible-content">2G – Réseaux numériques : Arrivée d’une nouvelle technologie numérique pour la transmission sans fil également connue sous le nom de Global System for Mobile communication (GSM). La technologie GSM est devenue la norme de base pour l’élaboration ultérieure des normes applicables au sans-fil. La 2G pouvait prendre en charge un débit de données de 14,4 kb/s à 64 kb/s (maximum), ce qui est suffisant pour les services de SMS et de messagerie. Des réseaux de données (GPRS, EDGE, IS-95B) ont été ajoutés et communément appelés technologies 2.5G et 2.75G.<ref>Rajiv, & Noman, S. (2018, December 14). Evolution of wireless technologies 1G to 5G in mobile communication. Retrieved from <i>[https://www.rfpage.com/evolution-of-wireless-technologies-1g-to-5g-in-mobile-communication/]<i></ref></p><br />
<p class="highlighted mw-collapsible-content">3G – Réseaux de données IP haute vitesse : La troisième génération offre un accès plus rapide à Internet avec des débits en aval allant jusqu’à 1 Mb/s et même plus, selon la version de la 3G<ref>Encyclopedia. (n.d.). Retrieved from <i>[https://www.pcmag.com/encyclopedia/term/55406/cellular-generations/ cellular generations]</i></ref>. La troisième génération de communications mobiles a commencé avec les Universal Mobile Terrestrial / Telecommunication Systems (UMTS). Après l’arrivée des systèmes de communication mobile 3G, les téléphones intelligents ont gagné en popularité dans le monde entier. Des applications ont été développées expressément pour les téléphones intelligents, pour le clavardage multimédia, la messagerie électronique, les appels vidéo, les jeux, les médias sociaux et les soins de santé.<ref>Rajiv, & Noman, S. (2018, December 14). Evolution of wireless technologies 1G to 5G in mobile communication. Retrieved from <i>[https://www.rfpage.com/evolution-of-wireless-technologies-1g-to-5g-in-mobile-communication/]<i></ref>.<br />
<p class="highlighted mw-collapsible-content">Afin d’améliorer le débit de données dans les réseaux 3G existants, deux autres améliorations technologiques ont été apportées au réseau : l’accès par paquets en liaison descendante haut débit ou HSDPA pour High Speed Downlink Packet Access et l’accès par paquets en liaison montante haut débit ou HSUPA pour High Speed Uplink Packet Access, mis au point et déployés sur les réseaux 3G, appelés 3.5G. L’évolution suivante de la 3G, appelée 3.75, est une version améliorée du réseau 3G avec accès par paquets haut débit évolué ou HSPA+ pour High Speed Packet Access Plus. Plus tard, ce système deviendra le puissant 3.9G qu’on appellera technologie d’évolution à long terme ou LTE pour Long Term Evolution.</p><br />
<p class="highlighted mw-collapsible-content">4G – Augmentation de la bande passante mobile : Les systèmes 4G sont des versions améliorées des réseaux 3G développés par IEEE, qui offrent un débit de données plus élevé et sont capables de gérer des services multimédias avancés. Les technologies sans fil LTE et LTE évoluée sont utilisées dans les systèmes de 4e génération. De plus, la 4G est compatible avec les versions précédentes, ce qui facilite le déploiement et la mise à niveau des réseaux LTE et LTE évoluée.<ref>Rajiv, & Noman, S. (2018, December 14). Evolution of wireless technologies 1G to 5G in mobile communication. Retrieved from <i>[https://www.rfpage.com/evolution-of-wireless-technologies-1g-to-5g-in-mobile-communication/]<i></ref>. Il s’agit essentiellement d’une extension de la technologie 3G avec plus de bande passante et de services. L’une des principales différences technologiques de la 4G par rapport à la 3G réside dans l’élimination de la commutation de circuits et l’utilisation d’un réseau IP de bout en bout. Ainsi, la 4G a permis de traiter les appels vocaux comme n’importe quel autre type de média audio en continu au moyen de la commutation de paquets sur Internet, d’un réseau local ou d’un réseau étendu, par voix sur IP.<ref>NA. (2008, 08 23). 1G, 2G, 3G, 4G - The Evolution of Wireless Generations. Retrieved from Support.Chinavision:<i> https://support.chinavasion.com/index.php?/Knowledgebase/Article/View/284/42/1g-2g-3g-4g---the-evolution-of-wireless-generations</i></ref></p><br />
<br />
<p>5G – Spectre sans licence : Le réseau 5G présente trois avantages principaux par rapport à son prédécesseur :<br />
<ul><br />
<li>• il peut offrir une vitesse de téléchargement de données de 10 à 20 Gb/s;</li><br />
<li>• il offre une faible latence, inférieure à une milliseconde, un point crucial pour les applications qui doivent être mises à jour en temps réel;</li><br />
<li>• comme cette technologie utilise des ondes radio millimétriques pour la transmission, elle peut fournir une bande passante supérieure sur les réseaux LTE actuels ainsi que des débits de données beaucoup plus élevés.</li><br />
</ul><br />
Concrètement, les réseaux 5G seront ainsi en mesure de fournir un accès à l’entreposage en nuage et d’exécuter des applications opérationnelles et auront le pouvoir d’exécuter virtuellement des tâches plus complexes. Le réseau 5G offre la possibilité de connecter 100 fois plus d’appareils que les ondes 4G LTE. Il peut permettre également de réduire la consommation d’énergie de 90 % par rapport à la 4G tout en assurant des vitesses Internet uniquement atteintes actuellement grâce à une connexion réseau directe par l’intermédiaire d’un câble à fibres optiques.<br />
</p><br />
<p>La 5G est de plus prête à transformer le monde des appareils de l’Internet des objets. L’utilisation des ondes radio millimétriques et du réseau central 5G non seulement accélère la transmission de données, mais accroît aussi la fiabilité de connexion. On améliore ainsi la connectivité des nouveaux types d’applications mobiles, l’automatisation industrielle et les véhicules autonomes, entre autres. Pour l’essentiel, toute application de l’Internet des objets qui utilise actuellement la technologie LPWA (Low Power Wide Area) connaîtra des améliorations progressives. De nombreux fournisseurs d’appareils cellulaires sont prêts à lancer des téléphones intelligents et d’autres appareils capables de se connecter aux réseaux 5G d’ici la fin de 2019. Actuellement, des entreprises comme AT&T ont lancé la 5G Evolution, une évolution de la 4G LTE qui n’offre cependant pas toutes les capacités de la 5G.</p><br />
<h2>Sommaire technique</h2><br />
<p>Tout comme les réseaux cellulaires actuels, la 5G divise le territoire en petits secteurs dans lesquels les appareils se connectent aux sites cellulaires. Ces sites cellulaires sont alors en mesure de transmettre des données chiffrées en utilisant des ondes radioélectriques. La différence entre la 5G et son prédécesseur réside dans sa capacité à transmettre ces ondes radio à des fréquences beaucoup plus élevées, ce qui se traduit par des vitesses de transmission de données supérieures, même à celle des réseaux de fibres optiques actuels, soit 1 Gb/s. Cette évolution, aussi petite soit-elle, a déjà eu son application dans le monde réel lorsque Sprint a lancé une fonctionnalité similaire avec sa technologie LAA. Dans le spectre des ondes millimétriques, ces fréquences se situent entre 30 et 300 GHz.</p><br />
<p>Deux séries de fréquences sont en cours d’approbation par la Federal Communications Commission des États-Unis. Les réseaux d’ondes 5G de bande basse et 5G centrales utilisent des fréquences de 600 MHz à 6 GHz, surtout 3,5 à 4,2 GHz. Les ondes centrales n’affecteront probablement pas beaucoup le matériel existant utilisé pour le sans-fil. Bien qu’il soit nécessaire d’utiliser des amplificateurs de puissance pour éviter une forte atténuation du signal, les ondes millimétriques constitueront une réelle évolution des technologies sans fil au point de nécessiter un tout nouveau système d’antennes, de câbles et d’amplificateurs.</p><br />
<br />
<p>Les réseaux 5G seront utilisés avec des sites cellulaires beaucoup plus petits. Les ondes radio à haute fréquence ne peuvent parcourir que de courtes distances en comparaison des ondes 4G LTE, à plus basse fréquence. Comme le signal 5G ne peut être transmis que sur la longueur d’un pâté de maisons ou à peu près et ne peut pas pénétrer dans les bâtiments, il n’y aura plus besoin de grandes tours, au profit de petites tours de téléphonie cellulaire installées sur chaque pâté de maisons et à l’intérieur des bâtiments. Cela signifie également que la vitesse des réseaux individuels sera plus importante qu’auparavant.</p><p class="inline">Un article rédigé par des professeurs des universités de Waterloo, Carleton et Ozyegin explique que les réseaux 5G pourraient transformer complètement l’architecture cellulaire actuelle. Ils expliquent que pour que la 5G puisse fonctionner avec une telle demande de bande passante sur le réseau par les appareils de l’Internet des objets, l’architecture cellulaire traditionnelle peut être divisée en une architecture à deux niveaux : 1) un niveau macrocellulaire, pour la communication entre la station et l’appareil et 2) un niveau équipements, pour la communication entre appareils.</p> <p class="highlighted inline mw-collapsible-content"> Cependant, la situation n’est pas sans risques pour la sécurité. La communication entre appareils nécessite une sécurité réseau plus complexe que l’actuelle. La communication est possible grâce à l’utilisation de relais entre appareils; les appareils connectés se servent les uns des autres pour retransmettre des données, ce qui crée un réseau maillé ad hoc. De cette façon, les appareils peuvent communiquer entre eux dans une largeur de bande cellulaire autorisée sans l’utilisation d’une station de base. Il s’agit d’un changement radical par rapport à l’architecture cellulaire traditionnelle où les téléphones cellulaires se connectent à une tour de téléphonie cellulaire.</p><br />
<br />
<p class="highlighted mw-collapsible-content">Auparavant, la communication entre appareils n’était utilisée qu’à de rares occasions. La demande à cet effet a récemment augmenté au gré de la commercialisation d’applications contextuelles. Ces applications nécessitent généralement à la fois des services de localisation et la capacité de communiquer avec d’autres appareils. En accédant à cette capacité par la technologie nécessaire, on réaliserait des économies puisque ce ne serait pas tous les appareils du réseau qui auraient besoin d’être connectés à la station de base. La communication entre appareils pourrait également jouer un rôle dans l’infonuagique mobile et rendre le partage des ressources plus efficace. Si un dispositif se trouve en périphérie d’un site cellulaire ou dans une zone encombrée, la communication entre appareils pourrait libérer la station de base d’une importante consommation de ressources.</p><br />
<br />
<br />
<h2>Utilisation par l'industrie</h2><br />
<p>Plusieurs fournisseurs de télécommunications aux États-Unis ont commencé à développer et à mettre à l’essai des réseaux 5G. Des fournisseurs de télécommunications comme Verizon, AT&T et Sprint ont tous fait des progrès dans ce domaine et mènent chacun des projets de recherche pour tester les réseaux. Verizon, AT&T, Sprint et T-Mobile ont tous commencé à déployer la 5G sur différents marchés et continueront à le faire tout au long de 2019. Verizon dispose de la 5G fixe et mobile dans quelques régions. AT&T dispose de la 5G mobile pour certaines entreprises dans certaines villes alors que Sprint déploie la 5G dans certaines zones. T-Mobile lancera la 5G commerciale au second semestre 2019 et devrait avoir une couverture nationale en 2020.</p><br />
<br />
<br />
<p class="highlighted mw-collapsible-content">Sprint et T-Mobile ont investi dans la 5G basse fréquence, qui offre des vitesses plus lentes, mais une autonomie accrue. Les deux entreprises pourront ainsi offrir la 5G dans des zones moins denses à moindre coût. Sprint a investi dans la bande centrale, la 5G 2,5 GHz, tandis que T-Mobile compte utiliser la 4G de bande basse 600 MHz en plus de la 5G haute fréquence dans les zones plus denses. En comparaison, Verizon et AT&T utiliseront principalement des bandes de fréquences beaucoup plus élevées, comme la bande des 28 GHz.</p><p class="highlighted mw-collapsible-content"><br />
Au Canada, la 5G ne sera pas offerte à grande échelle avant 2020. Bien que la 5G ait le potentiel d’atteindre des vitesses de 20 Gb/s, elle ne sera probablement qu’à 6 Gb/s à son premier déploiement. Comme pour les technologies similaires, il faudra jusqu’à 10 ans pour que la nouvelle technologie atteigne sa pleine maturité.</p><br />
<p>L’une des utilités de la 5G est la gestion des sources d’énergies renouvelables comme l’énergie solaire et éolienne en compensation de la consommation d’énergie électrique. Étant donné que la 5G permettra la collecte de données, il sera possible de l’analyser pour déterminer les moments de forte et de faible consommation d’électricité. L’information servira ensuite à planifier un réseau électrique plus cohérent et plus fiable.</p><br />
<br />
<p>La rapidité des réseaux 5G et leur faible latence inhérente rendront également possible la chirurgie à distance. Les habitants des milieux ruraux auront ainsi accès à des chirurgiens et à des spécialistes qui ne pratiquent habituellement que dans les grandes villes. La première chirurgie à distance réussie a déjà été réalisée en Chine. Le réseau 5G est la pièce manquante au casse-tête de la chirurgie à distance. Pour réussir, une chirurgie à distance a besoin d’un patient, d’un chirurgien, d’un robot et d’une connexion Internet stable et ultrarapide.</p><br />
<p>Que se passerait-il si les voitures autonomes pouvaient signaler leurs intentions ou diffuser leur itinéraire à d’autres voitures autonomes? La 5G pourrait en faire une réalité et contribuerait ainsi à rendre les routes plus sûres. Les conducteurs pourraient aussi montrer aux autres à proximité dans quelle direction ils vont. La chose pourrait se faire lorsque nous utilisons nos téléphones pour trouver le chemin à emprunter pour parvenir à notre destination. Le téléphone pourrait également diffuser ces informations par la 5G jusqu’aux téléphones et aux voitures autonomes à proximité.</p><br />
<br />
<h2>Utilisation par le gouvernement du Canada</h2><br />
<p class="highlighted mw-collapsible-content">Les réseaux mobiles 5G (ou de 5e génération) ne sont pas encore accessibles au Canada ni dans la plupart des pays du monde d’ailleurs. Malgré tout, le gouvernement du Canada s’est préparé à son arrivée. Le Canada est au même point que les autres pays développés dans la préparation à la 5G.</p><br />
<b>Innovation, Sciences et Développement économique Canada (ISDE) et gestion du spectre mobile</b><br />
<br />
<p>La demande d’applications et de contenu numériques continue d’augmenter, tant au Canada qu’à l’étranger, ce qui constitue le principal moteur de l’essor de la technologie 5G. Les téléphones intelligents et autres appareils cellulaires de même que les tablettes, les appareils informatiques personnels (Internet des objets) et la connectivité entre machines jouent un rôle de plus en plus crucial dans la vie quotidienne des entreprises et citoyens canadiens. Dans un contexte de hausse de l’utilisation de ces appareils, le taux de croissance composé du trafic de données mobiles a été calculé à 54 % par an. En tant que telle, la création de spectres ou la conversion de spectres existants (ou de radiofréquences qu’empruntent les données mobiles) par les organismes de réglementation nationaux est cruciale si on veut répondre à la demande et éviter tout effet nuisible sur l’économie.</p><br />
<p class="highlighted mw-collapsible-content">Tout le spectre radioélectrique mondial est attribué par l’Union internationale des télécommunications (UIT). Au Canada, les téléphones cellulaires et les radiofréquences sont réglementés par Innovation, Sciences et Développement économique Canada (ISDE), qui fait partie de l’UIT. Ce ministère supervise également la délivrance des licences et l’installation des tours de téléphonie cellulaire, effectue des évaluations de l’impact environnemental et de l’utilisation du sol pour les projets d’installation de telles tours ou d’autres infrastructures de téléphonie cellulaire et veille à ce que l’équipement satisfasse à toutes les exigences réglementaires. Il est également responsable de l’attribution de licences d’utilisation du spectre aux entreprises de télécommunications sans fil au Canada. En 2015, après avoir consulté les entreprises de télécommunications et les télédiffuseurs, il a été décidé que le Canada utiliserait à nouveau la portion de 600 MHz du spectre de télévision pour les services mobiles. La mise aux enchères de ce spectre aux entreprises de télécommunications mobiles s’est achevée en avril 2019 et montre que le gouvernement du Canada est conscient de l’importance sans cesse croissante de la technologie mobile et de la nécessité d’élargir les bandes de fréquences.<br />
</p><br />
<p>Toutefois, avec l’arrivée de la 5G à l’horizon 2023, l’année où la plupart des opérateurs en Amérique du Nord ont l’intention de lancer la 5G à grande échelle, il faudra encore plus d’espace sur le spectre:</p><br />
<p><i>“Il est essentiel d’avoir un nouveau spectre si on veut assurer le bon fonctionnement des services mobiles terrestres de la cinquième génération (5G). À l’échelle mondiale, on s’affaire sans relâche à déterminer les spectres qui conviendraient, notamment les bandes qui peuvent être utilisées dans le plus grand nombre de pays possible pour permettre l’itinérance mondiale et favoriser des économies d’échelle. Des travaux sont en cours dans le monde entier pour viser l’harmonisation du spectre à utiliser pour la 5G. Les services 5G devraient couvrir un large éventail d’applications”- 5G Americas<br />
</i></p><br />
<p class="highlighted mw-collapsible-content">En juin 2017, ISDE a lancé des consultations concernant la libération future de fréquences supplémentaires, au-delà des 648 MHz actuellement utilisés. ISDE souhaitait se renseigner sur les quantités qui devraient être nécessaires ainsi que sur la nécessité d’éventuelles considérations politiques et réglementaires, à mesure que de nouveaux modèles commerciaux et de nouvelles applications réseau feraient leur arrivée. Divers intervenants ont pris part aux consultations et ont manifesté leur appui à la proposition du gouvernement du Canada concernant la libération des bandes de fréquences de 28 GHz, 37 à 40 GHz et 64 à 71 GHz. Le ministre d’ISDE, l’honorable Navdeep Bains, a déclaré qu’aucune décision définitive ne sera prise avant la conférence mondiale des radiocommunications à l’automne 2019 et que les consultations sur ces questions durent généralement deux ans. Toutefois, certains acteurs de premier plan aimeraient que ce processus s’accélère. Un représentant de Telus a eu ces mots : « Il faut prendre des mesures réglementaires immédiates et définitives pour que le Canada récolte les fruits de sa rapidité d’action dans la nouvelle économie numérique mondiale. »</p><br />
<b>Sécurité publique et inquiétudes par rapport à l’espionnage</b><br />
<p class="highlighted mw-collapsible-content">En mai 2019, le gouvernement du Canada a entrepris une étude de la cybersécurité de la technologie 5G et des fournisseurs potentiels d’équipement. Actuellement, les principaux fournisseurs mondiaux sont Nokia, Ericsson, Samsung, Qualcomm et Huawei, cette dernière société ayant la situation la plus nébuleuse de toutes. En 2018, l’Australie, la Nouvelle-Zélande et les États-Unis ont tous interdit l’utilisation d’équipements de télécommunications Huawei dans leurs réseaux 5G par crainte que l’entreprise n’ait des liens avec le gouvernement chinois, qui pourrait potentiellement utiliser Huawei pour réaliser ses activités d’espionnage ou attaquer des infrastructures publiques essentielles au moyen d’un programme malveillant. Huawei a toujours nié avec véhémence ces allégations. Le Royaume-Uni a ordonné l’interdiction partielle de Huawei dans les composantes centrales de son réseau 5G. Les autres pays européens se sont jusqu’à présent abstenus de lui emboîter le pas.</p><br />
<p>Bien que ce soit normalement la responsabilité des entreprises canadiennes comme Bell, Rogers et Telus d’assurer la sécurité de leurs réseaux, le gouvernement du Canada a l’obligation de veiller à la sécurité publique, dont la cybersécurité est une composante. Le 1er mai 2019, selon le ministre de la Sécurité publique Ralph Goodale, le ministre responsable de la sécurité nationale et la [https://www.publicsafety.gc.ca/cnt/rsrcs/pblctns/ntnl-cbr-scrt-strtg/index-en.aspx/ Stratégie nationale de cybersécurité du Canada], l’examen sur la sûreté de la 5G, qui comprend le rôle potentiel de Huawei, est en cours, et une décision finale est attendue d’ici l’automne 2019. Indépendamment de cette décision, les opérateurs et le gouvernement du Canada devront déployer des efforts pour assurer la sécurité des réseaux, comme c’est le cas pour la 4G LTE actuellement.</p><br />
<b>Autres investissements et initiatives</b><br />
<p>Le 19 mars 2018, le gouvernement du Canada a annoncé l’octroi de fonds pour le projet du corridor d’essai 5G entre le Québec et l’Ontario. Cet octroi représente une étape dans l’adoption de la prochaine génération de technologie sans fil. Le gouvernement du Canada établit des partenariats avec divers acteurs privés en guise d’exemple de collaboration entre toutes les parties prenantes. La 5G exigera une énorme et coûteuse refonte de l’infrastructure et il faut en tenir compte. </p><br />
<br />
<p>L’Association canadienne des télécommunications sans fil a lancé le Conseil 5G Canada pour favoriser la collaboration dans les travaux de mise en place de la 5G. La date de sortie de la technologie est toujours prévue d’ici 2020. Le gouvernement du Canada devra encore se pencher sur la manière dont il prendra en charge les fréquences radioélectriques entre 600 et 3 500 MHz, qui sont nécessaires pour les réseaux 5G. Cette gamme de fréquences est cruciale, car la fréquence de 600 MHz est l’une des fréquences les plus élevées qui peuvent s’utiliser dans les régions rurales et éloignées du pays.</p><br />
<p class="highlighted mw-collapsible-content">Le gouvernement canadien a annoncé un montant pouvant atteindre 40 millions de dollars pour financer la recherche de Nokia sur la technologie 5G au Canada. Nokia a lancé de multiples projets sur le routage des données dans les réseaux optiques ainsi que la création d’outils de cybersécurité qui protégeront les réseaux de télécommunications dans leur transition vers la 5G.</p><br />
<h2>Répercussions pour les agences gouvernementales</h2><br />
<br />
<h3>Services partagés Canada (SPC)</h3><br />
<p>SPC aura un rôle important à jouer pour que les ministères canadiens disposent des outils, de l’infrastructure et de l’architecture nécessaires au lancement à grande échelle de la 5G au cours des prochaines années. Ainsi, le déploiement de la 5G aura des répercussions majeures sur SPC. </p><br />
<h4>Proposition de valeur</h4><br />
<p class="highlighted mw-collapsible-content">Comme il est mentionné dans le sommaire opérationnel, la 5G offre trois avantages principaux par rapport au réseau 4G actuel : une vitesse accrue, une latence moindre et la possibilité de connecter beaucoup plus d’appareils à la fois. Concrètement, les réseaux 5G seront ainsi en mesure de fournir un meilleur accès à l’entreposage en nuage (et à l’informatique en périphérie) et d’exécuter des applications opérationnelles avec un délai de réponse en temps réel amélioré et auront le pouvoir d’exécuter virtuellement des tâches plus complexes. Ces avantages vont de pair avec la volonté d’avoir un gouvernement du Canada ouvert et d’augmenter le partage des données et la collaboration depuis n’importe quel appareil (y compris les appareils mobiles), comme l’indique le [https://www.canada.ca/en/government/system/digital-government/digital-operations-strategic-plan-2018-2022.html#ToC9_2/ Plan stratégique des opérations numériques de 2018 à 2022. ] </p><br />
<p>SPC a apporté des changements importants dans la modernisation des centres de données du gouvernement du Canada ainsi que dans l’offre de services infonuagiques, sur le plan du traitement et du stockage des données. Cependant, au fil de l’évolution technologique, l’informatique en périphérie viendra compléter ces deux modèles. D’ici 2022, plus de 50 % des données générées par l’entreprise seront créées et traitées à l’extérieur du centre de données ou du nuage, selon une recherche effectuée par Gartner. L’informatique en périphérie s’impose de plus en plus comme solution aux problèmes de latence entre machines. La 5G améliorera par sa nature même la bande passante et donc les problèmes de latence, ce qui permettra de prendre en charge une plus grande densité d’appareils, dont ceux en périphérie. Grâce à la 5G, les données atteindront leurs points terminaux (qu’il s’agisse d’un nuage ou d’un centre de données) plus rapidement et accéléreront ainsi le traitement et le stockage. </p><br />
<br />
<h4>Difficultés</h4><br />
<p class="highlighted mw-collapsible-content">Premièrement, les appareils compatibles devront être redistribués aux fonctionnaires de partout au Canada. Les appareils actuels ne seront en effet pas compatibles avec les réseaux 5G, et les mises à jour automatiques ne seront pas possibles. Seuls les appareils compatibles avec la 5G peuvent être utilisés sur les réseaux 5G (ils peuvent également être utilisés sur les réseaux 4G). Toutefois, il ne sera probablement pas nécessaire de mettre les appareils à niveau tout de suite. Les premières instances de la 5G utiliseront les réseaux et l’équipement 4G et non des réseaux ou de l’équipement particuliers. Quoi qu’il en soit, le renouvellement complet de tous les appareils du gouvernement du Canada sur à peu près la même période sera un projet d’envergure sur le plan logistique et financier. Néanmoins, il faudra aller de l’avant si le gouvernement veut pouvoir continuer d’exploiter le numérique au profit de la population canadienne, l’un des principaux thèmes stratégiques énoncés dans le [https://www.canada.ca/en/government/system/digital-government/digital-operations-strategic-plan-2018-2022.html/ Plan stratégique des opérations numériques de 2018 à 2022] aussi décrit dans [http://www.clerk.gc.ca/local_grfx/bp2020/bp2020-eng.pdf/ Objectif 2020].</p><br />
<p class="highlighted mw-collapsible-content">Deuxièmement, la 5G fonctionnera en partie sur les ondes millimétriques, qui ont des fréquences comprises entre 30 et 300 GHz. Les ondes millimétriques ont généralement une faible portée et sont sensibles aux interférences et aux blocages causés par des objets tels que les bâtiments, les arbres, voire la pluie et les nuages. Il devient donc difficile d’avoir une bonne couverture pour les appareils. De plus, les interférences et les problèmes de portée peuvent nuire à la tenue de certaines activités, notamment les prévisions météorologiques, les initiatives SmartCities, les procédures médicales et les opérations militaires et policières. Pour contourner le problème, il faudra déployer de petites cellules, terme forgé décrivant l’utilisation de nombreuses petites antennes, petites tours et petits émetteurs dans une zone dense. La situation entraînera une modification considérable des structures de réseau actuelles et pourrait compliquer passablement les choses dans certaines régions éloignées ou rurales.</p><br />
<p>Enfin, pour terminer, l’arrivée de la 5G pourrait susciter des inquiétudes en matière de santé et de sécurité au travail chez les employés et les citoyens en général, par rapport à l’exposition accrue aux rayonnements en raison du nombre élevé d’antennes et de tours situées plus près du lieu de résidence et de travail des gens. Même avec la technologie 4G actuelle et les équipements de télécommunications existants, certains pensent que les appareils et leurs équipements représentent une menace pour la santé humaine. Cependant, aucune recherche n’a actuellement démontré de risque réel. Santé Canada a rédigé des lignes directrices sur l’exposition humaine sécuritaire aux radiofréquences. Les radiofréquences de la 5G sont bien en dessous de la gamme de sécurité de 300 GHz. </p><br />
<br />
<h4>Considérations</h4><br />
<br />
<p>C’est sur ses centres de données que les réseaux 5G auront le plus important effet pour SPC. La 5G promet de prendre en charge un trafic plus élevé sur le réseau à des vitesses plus élevées avec une faible latence. Il faut aussi en comprendre que les applications commenceront à être conçues pour les réseaux 5G. Le passage à la 5G ne changera rien aux tâches à accomplir pour les centres de données, sinon leur échelle. Les centres de données devront se décentraliser et suivre un modèle à deux ou trois niveaux afin de fournir un traitement à faible latence à la périphérie du réseau tout en maintenant un rôle central d’agrégation et de coordination des données. Ils devront aussi augmenter la bande passante, le pouvoir de traitement et le stockage. </p><br />
<p class="highlighted mw-collapsible-content">La 5G transformera également l’architecture cellulaire traditionnelle. Pour avoir une architecture à deux niveaux, comme l’explique le sommaire technologique, il faut accroître la sécurité. En effet, l’utilisation d’appareils pour se connecter les uns aux autres afin d’établir une connexion réseau fait en sorte qu’il n’y a pas qu’une seule station de base centrale à protéger. Dorénavant, le fournisseur devra également se protéger contre les appareils qui établissent eux-mêmes les connexions. La sécurité sera une question de première importance pour SPC à l’arrivée de la 5G, car tous les appareils de l’Internet des objets seront connectés. Le nombre accru d’appareils fait augmenter le trafic sur le réseau.</p><br />
<p class="highlighted mw-collapsible-content">Des entreprises comme Cisco et Ericsson ont commencé à utiliser des réseaux SDN et la virtualisation des fonctions réseau parce que ce sont des solutions flexibles qui peuvent prendre en charge dynamiquement un nombre élevé d’appareils. Les réseaux SDN découplent le matériel du logiciel, ce qui signifie que les tâches peuvent être effectuées dans le nuage ou dans des grappes de serveurs. La virtualisation des fonctions réseau, généralement utilisée en combinaison avec les réseaux SDN, transforme les fonctions réseau de manière à pouvoir être exécutée dans des machines virtuelles au lieu de dépendre du matériel. Il s’agit là d’options viables pour SPC à mesure que le Ministère migre vers le nuage à l’aube de l’ère 5G.</p><br />
<p>Les réseaux 5G ne sont pas qu’une simple évolution technologique : c’est une révolution. La 5G a le potentiel de modifier de fond en comble la façon dont les données sont transmises et traitées, et par qui (ou quoi). SPC devra s’informer sur le potentiel des réseaux 5G d’entreprise, à l’instar de nombreuses entreprises qui ont commencé des démarches en ce sens. En raison de certaines activités essentielles à la mission de divers ministères, comme la Défense nationale et la Gendarmerie royale du Canada, l’arrivée des réseaux 5G privés peut se révéler nécessaire si on veut transmettre des renseignements sensibles ou lorsque l’infrastructure et les réseaux publics ne sont pas fiables ou jugés sûrs. </p><br />
<p class="highlighted mw-collapsible-content">Malgré les avantages de la 5G, il y aura des coûts initiaux et financiers et humains. Non seulement la mise à jour et le déploiement de l’infrastructure et des appareils actuels seront nécessaires, mais la densification de l’infrastructure sera également inévitable avec la technologie 5G. En raison des difficultés liées aux distances de transmission et aux interférences, il peut devenir nécessaire de déployer de petites cellules (tours et antennes radio), éventuellement sur chaque bâtiment gouvernemental à la grandeur du pays. Les répercussions sur les budgets et la main-d’œuvre ne font aucun doute. </p><br />
<p class="highlighted mw-collapsible-content">Enfin, on pourra tirer des leçons de l’expérience des premiers qui adopteront la 5G. À l’heure actuelle, la technologie 5G est loin du stade de maturité et n’est pas encore déployée à grande échelle ailleurs dans le monde. Cependant, en avril 2019, la Corée du Sud est devenue le premier pays à adopter pleinement la 5G et devrait compter près d’un million d’utilisateurs d’ici la fin juin 2019. Dans les premiers mois qui ont suivi le lancement, les utilisateurs se sont plaints de la couverture et de la vitesse, principalement en raison du manque de stations de base (tours et antennes) en dehors des zones urbaines à forte densité de population. Les opérateurs ont réagi en installant de 3 000 à 4 000 nouvelles stations par semaine afin de répondre à la demande et de résoudre les problèmes. On voit ainsi l’importance d’avoir une bonne infrastructure en place avant l’arrivée de la technologie afin d’éviter la frustration des clients qui pourraient ainsi vouloir lui tourner le dos.</p><br />
<br />
<h2>References</h2><br />
<br />
<div style= "display:none"><ref>Cheng, Roger "What is 5G? Here are the basics", cnet, 9 February 2018.<i>https://www.cnet.com/how-to/5g-network-technology-here-are-the-basics/</i></ref><br />
<ref>M. N. Tehrani, M. Uysal and H. Yanikomeroglu, "Device-to-device communication in 5G cellular networks: challenges, solutions, and future directions," in IEEE Communications Magazine, vol. 52, no. 5, pp. 86-92, May <i>http://ieeexplore.ieee.org/abstract/document/6815897/</i></ref><br />
<ref>Oliveira, Michael, “Canadian wireless providers mum as U.S. companies announce 5G launch dates”, The Canadian Press, April 2018. <i>https://www.theglobeandmail.com/business/article-canadian-wireless-providers-mum-as-us-companies-announce-5g-launch/</i></ref><br />
<ref>Pretz, Kathy, “5G: The Future of Communications Networks”, IEEE, 1 March 2017. <i>http://theinstitute.ieee.org/technology-topics/communications/5g-the-future-of-communications-networks</i></ref><br />
<ref>“5G spectrum guide – everything you need to know”, GSMA, 16 February 2018 <i>https://www.gsma.com/spectrum/5g-spectrum-guide/</i></ref><br />
<ref>Lavallee, Brian, “Data Centers to 5G: Bring It On!” Data Center Knowledge, 6 February 2018. <i>http://www.datacenterknowledge.com/industry-perspectives/data-centers-5g-bring-it</i></ref><br />
<ref>Williams, Kevin, “How 5G Will Usher In The Internet Of Power Delivery”, 21 June 2016 <i>https://www.forbes.com/sites/huawei/2016/06/21/how-5g-internet-will-change-the-way-we-generate-and-consume-power/#62c0aa608031</i></ref><br />
<ref>Cheng, Roger, “5G is finally starting to feel real”, 18 December 2018 <i>https://www.cnet.com/news/5g-is-finally-starting-to-feel-real/</i></ref><br />
<ref>Humphries, Matthew, “China Performs First 5G Remote Surgery”, 15 January 2019, <i>https://www.pcmag.com/news/365992/china-performs-first-5g-remote-surgery</i></ref><br />
<ref>Behar, Rose, “Everything you need to know about 5G in Canada”, 1 March 2019, <i>https://mobilesyrup.com/2018/03/01/everything-you-need-to-know-about-5g-canada/</i></ref><br />
<ref>CableFree “Introducing 5G Frequency Bands”<i>https://www.cablefree.net/wirelesstechnology/4glte/5g-frequency-bands-lte/</i></ref><br />
<ref>New York Intelligencer “5G is Going to Transform Smartphones – Eventually”, February 2019<i>http://nymag.com/intelligencer/2019/02/5g-is-going-to-transform-smartphones-eventually.html</i></ref><br />
<ref>ISED, Government of Canada, “Decision on Repurposing the 600 MHz Band”, August 2015<i>https://www.ic.gc.ca/eic/site/smt-gst.nsf/eng/sf11049.html</i></ref><br />
<br />
<ref>5G Americas, “5G Spectrum Recommendations”, April 2017.</ref><br />
<ref>ISED, Government of Canada, “Consultation on Releasing Millimetre Wave Spectrum to Support 5G”, June 2017.<i>https://www.ic.gc.ca/eic/site/smt-gst.nsf/eng/sf11298.html</i></ref><br />
<ref>Behar, Rose, “Here’s Why mmWave is Crucial to Canada’s 5G Future”, MobileSyrup, Oct 2017.<i>https://mobilesyrup.com/2017/10/16/mmwave-spectrum-explainer/</i></ref><br />
<ref>Bryan-Low, Cassell,Colin Packham, David Lague, Steve Stecklow & Jack Stubbs, Reuters, “Hobbling Huawei: Inside the U.S. war on China’s tech giant”, May 2019.<i>https://www.reuters.com/investigates/special-report/huawei-usa-campaign/</i></ref><br />
<ref>Fife, Robert & Steven Chase, The Globe and Mail, “Goodale says decision on Huawei 5G network to come before election”, May 2019.<i>https://www.theglobeandmail.com/politics/article-goodale-says-decision-on-huawei-to-come-before-election/</i></ref><br />
<ref>Business Wire, “Ixia, a Keysight Business, Delivers Network Performance Monitoring and Visibility to Remote Sites and Edge Computing”, Feb 2019.<i>https://finance.yahoo.com/news/ixia-keysight-business-delivers-network-161500291.html</i></ref><br />
<ref>Nordrum, Amy, Kristen Clark and IEEE Spectrum Staff, “Everything You Need to Know About 5G”, Jan 2017.<i>https://spectrum.ieee.org/video/telecom/wireless/everything-you-need-to-know-about-5g</i></ref><br />
<ref>Health Canada, Government of Canada, “Limits of Human Exposure to Radiofrequency Electromagnetic Energy in the Frequency Range from 3 kHz to 300 GHz - Safety Code 6 (2015)”, 2015<i>https://www.canada.ca/en/health-canada/services/environmental-workplace-health/consultations/limits-human-exposure-radiofrequency-electromagnetic-energy-frequency-range-3-300.html15</i></ref><br />
<br />
</div><br />
</div><br />
<br />
{{#css:<br />
<br />
#firstHeading::after{<br />
content:"Reconnaissance Faciale";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.highlighted{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Tendances_Technologiques/Reconnaissance_Faciale&diff=11304Tendances Technologiques/Reconnaissance Faciale2019-07-22T15:46:39Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[en:Technology_Trends/Zero_Trust_Network]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Afficher la Vue Détaillée" data-collapsetext="Cacher la Vue Détaillée"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>Reconnaissance de Visage</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>[[Technology_Trends/Zero_Trust_Network|Face Recognition]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Publié</td><br />
</tr><br />
<tr><br />
<th>Version original</th><br />
<td>25 juin 2019</td><br />
</tr><br />
<tr><br />
<th>Mise à jour</th><br />
<td>22 juillet 2019</td><br />
</tr><br />
<tr><br />
<th>Publication officiel</th><br />
<td>[[Media:FR_-_FR_Face Recognition v0.1.pdf|Réconnaisance_de_Visage.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">Cette page est sujet à des mises à jours. On encourage toutes rétroactions. Veuillez utiliser la page de discussion pour apporter des commentaires et des suggestions. Une fois que cette version soit approuvée et finalisée, elle va être traduit.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p>La <b>reconnaissance faciale </b> est une technologie biométrique servant à établir et à valider l’identité d’une personne. Elle fait appel à une application informatique, ou système de reconnaissance faciale, qui extrait une image numérique d’une photo, d’un cadre vidéo ou d’une numérisation tridimensionnelle et crée une empreinte faciale, soit un ensemble de mesures caractéristiques d’un visage, qui identifie le visage particulier d’une personne aux fins d’authentification.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Cacher la Vue Détaillée</span></div><br />
</div><br><br><br />
<br />
<h2>Sommaire opérationnel</h2><br />
<br />
<p class="expand inline mw-collapsible-content">Généralement, l’authentification et l’identification se font par correspondance entre les caractéristiques faciales et les empreintes faciales d’un profil dans une base de données. </p><p class="inline">La technologie de reconnaissance faciale se décline en de nombreuses applications (contrôle d’accès, surveillance et enquêtes criminelles). Elle s’utilise également en combinaison avec d’autres technologies biométriques pour renforcer les mesures de sécurité.</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Les grandes sociétés technologiques comme Apple, Google, Samsung, Facebook et Amazon ont commencé à prendre conscience de l’importance que revêt la reconnaissance faciale pour leurs infrastructures de sécurité. </p><p class="inline">Apple a essayé de perfectionner la technologie en y ajoutant la reconnaissance de mouvements. La personne dont le visage est numérisé peut maintenant parler ou bouger pendant la numérisation, et ainsi la reconnaissance faciale peut se combiner avec d’autres mesures de sécurité biométriques comme la reconnaissance vocale. Étant donné que les visages en mouvement peuvent être numérisés, les systèmes de reconnaissance faciale peuvent identifier les gens même dans une foule, et ce, sans intrusion.</p><br />
<br />
<br />
<h2>Sommaire technique</h2><br />
<p class="inline">Les systèmes de reconnaissance faciale peuvent utiliser une image bidimensionnelle ou tridimensionnelle ou une alimentation vidéo pour créer une image numérique, établir l’empreinte faciale et identifier un visage en comparant l’image numérique avec les empreintes faciales dans la base de données. Chaque visage a ses caractéristiques déterminantes que le système marquera comme des points nodaux. Un visage humain peut compter jusqu’à 80 de ces points. Ils constituent des zones d’intérêt sur le visage mesurées par le système. </p><p class="expand inline mw-collapsible-content">SLa distance entre les yeux, la largeur du nez et la profondeur de l’orbite en sont des exemples. Ces mesures seront stockées dans une base de données sous forme d’empreinte faciale. </p><p class="inline">Lorsque le système numérise un visage, il compare toutes ces mesures aux profils, soit les empreintes faciales, dans la base de données. </p><p class="expand inline mw-collapsible-content">Les systèmes de reconnaissance faciale utilisent un algorithme, comme le Facial Recognition Vendor Test, qui permet de prédire s’il y a une correspondance en fonction des points nodaux du visage d’une personne. </p><p class="inline">Habituellement, la technologie suit quatre étapes : <ref><i>[http://www.ex-sight.com/technology.html]</i></ref>: </p><br />
<p>Capture –un échantillon physique ou comportemental est capté par le système lors d’une analyse;</p><br />
<p>extraction – des données uniques sont extraites de l’échantillon, et un modèle est créé;</p><br />
<p>comparaison – le modèle est ensuite comparé à un autre échantillon;</p><br />
<p>correspondance – le système décide alors si la caractéristique extraite du nouvel échantillon a une correspondance.</p><br />
<br />
<br />
<h2>Utilisation par l'industrie</h2><br />
<p class="inline">La reconnaissance faciale offre une autre forme d’identification et d’authentification biométriques. Plusieurs fournisseurs utilisent la reconnaissance faciale comme outil de contrôle d’accès et d’authentification pour leurs clients ou pour un usage interne. Son application ne se limite pas à la sécurité; la technologie sert également aux soins de santé et à la vente au détail.</p><p class="expand inline mw-collapsible-content"> Bien que la reconnaissance faciale bidimensionnelle ne soit pas aussi précise que d’autres formes de technologies biométriques comme les lecteurs d’empreintes digitales, elle a ses avantages. </p><p class="inline">La personne dont le visage est numérisé ne sait pas nécessairement quand il y a numérisation; ainsi, la technologie peut s’utiliser dans de grandes foules et mettre rapidement en évidence les menaces.</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Plusieurs fournisseurs sur le marché actuel emploient la reconnaissance faciale pour leurs nombreuses applications. </p><p class="inline">Par exemple, Amazon a mis au point un système qui permet aux utilisateurs de payer leurs articles à l’aide d’une image exploitable (égoportrait). L’utilisateur peut en effet utiliser un égoportrait dans lequel il se déplace ou prononce une phrase particulière comme mot de passe pour valider son identité lors du paiement d’un article. Le fait que le client doive prononcer une phrase ou faire un mouvement se veut un moyen d’éliminer la possibilité d’utiliser une image bidimensionnelle numérisée de la personne de manière frauduleuse. </p><p class="expand inline mw-collapsible-content">Amazon Rekognition est un autre produit qui fournit deux ensembles d’API : Amazon Rekognition Image pour les images et Amazon Rekognition Video pour les vidéos. Les deux API effectuent des analyses de détection et de reconnaissance d’images et de vidéos qui procurent de l’information utile à vos applications.</p><br />
<p>Après avoir acheté Face.com en 2012, Facebook a commencé à utiliser la technologie de reconnaissance faciale pour connecter les utilisateurs par leurs photos. Lorsqu’un utilisateur téléverse une photo, le logiciel suggère automatiquement d’autres personnes à identifier. Lorsqu’on est identifié dans une photo, on peut voir plus de contenu regroupé sur les personnes qui y sont identifiées. </p><br />
<p class="inline">Face ID est une technologie développée par Apple et accessible sur l’iPhone X. Elle offre une authentification intuitive et sécurisée grâce au système de caméra TrueDepth à la fine pointe de la technologie et aux technologies perfectionnées permettant de cartographier avec précision la géométrie du visage. D’un simple coup d’œil, Face ID déverrouille en toute sécurité votre iPhone ou iPad Pro. </p><p class="expand inline mw-collapsible-content">Vous pouvez l’utiliser pour autoriser des achats sur iTunes Store, App Store et Apple Books, et effectuer des paiements avec Apple Pay. Les iPhone XR, XS et XS Max sont tous équipés de la deuxième génération de Face ID, une version mise à jour du système d’authentification biométrique qui est censée être plus rapide que la version de l’iPhone X.</p><br />
<h2>Utilisation par le gouvernement du Canada</h2><br />
<p class="expand mw-collapsible-content">Contrairement au secteur privé, l’utilité des applications d’identification faciale pour le gouvernement est principalement liée à la sécurité, en particulier pour la vérification d’identité et la prévention des fraudes. Par exemple, l’Agence des services frontaliers du Canada a récemment lancé le programme des bornes d’inspection primaire, dans le cadre duquel les passagers qui entrent au pays depuis des aéroports doivent s’enregistrer dans des kiosques libre-service.<ref>Braga, Matthew. (March 2nd, 2017). Facial Recognition Technology is coming to Canadian Airports this spring. Canadian Broadcasting Corporation. Retrieved 17-05-2019 from: <i>[https://www.cbc.ca/news/technology/cbsa-canada-airports-facial-recognition-kiosk-biometrics-1.4007344]</i></ref>. Ces bornes vérifient l’identité des passagers par la reconnaissance faciale. L’adoption progressive des bornes sans personnel depuis 2015 a augmenté la sécurité tout en réduisant la congestion dans les aéroports. L’entreprise portugaise Vision-Box a installé 130 bornes à l’aéroport international Pearson de Toronto. Les bornes ont été conçues pour effectuer une reconnaissance biométrique en deux étapes : une étape de reconnaissance faciale et une autre de reconnaissance des empreintes digitales. Ces bornes seront également en mesure d’effectuer une reconnaissance de l’iris, une fonction réservée aux personnes inscrites au programme NEXUS.</p><br />
<p class="inline">Les systèmes de reconnaissance faciale sont également utilisés dans les casinos provinciaux pour identifier les visiteurs ayant une dépendance au jeu qui se sont volontairement inscrits sur des listes d’auto-exclusion et les empêcher d’entrer .<ref>Elash, Anita, and Luk, Vivian. (July 25th, 2011). Canadian Casinos, Banks, Police use Facial-Recognition Technology. The Globe and Mail. Toronto, Ontario. Retrieved 21-05-2019 from: <i>[https://www.theglobeandmail.com/news/national/time-to-lead/canadian-casinos-banks-police-use-facial-recognition-technology/article590998/ ]</i></ref>. Il convient de noter que le système a été élaboré conjointement avec le Commissaire à la protection de la vie privée de l’Ontario afin que soient prises en compte dans la conception par défaut les questions de protection de la vie privée. </p><p class="expand inline mw-collapsible-content">En temps réel, le système fait une lecture du visage des clients qui entrent dans le casino et compare leurs images à celles des joueurs de la liste d’auto-exclusion. S’il y a correspondance, le système avertit la sécurité, sinon le système supprime automatiquement l’image. L’accès à la base de données est restreint, et les renseignements d’une personne ne sont accessibles que si la personne sur la photo est physiquement sur place. </p><br />
<p class="inline">Passeport Canada utilise un logiciel de reconnaissance faciale depuis dix ans pour comparer les nouvelles photos de passeport à sa base de données afin de prévenir les fraudes. On effectue des comparaisons un à un pour confirmer l’identité d’une personne, ce qui signifie qu’une image récente est comparée à une image déjà dans la base de données qui est associée à l’identité de la personne en question. Les comparaisons un à plusieurs servent à comparer une image à l’ensemble de la base de données de photos d’identité afin de vérifier s’il y a des demandeurs en double ou des personnes ayant plusieurs identités<ref>Mackrael, Kim, and Ha, Tu Thanh. (May 15th, 2014) Facial Recognition Program Allows RCMP to Identify Alleged Passport Fraud. The Globe and Mail. Toronto, Ontario. Retrieved 27-05-2019 from: <i>[https://www.theglobeandmail.com/news/national/facial-recognition-program-allows-rcmp-to-nab-alleged-passport-fraudster/article18703608/]</i></ref>. </p><p class="expand inline mw-collapsible-content">Le projet a réussi à mettre au jour des cas de personnes qui tentaient d’obtenir plusieurs passeports. Ce même concept est également utilisé pour les permis de conduire délivrés par les provinces<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303/]</i></ref>. </p><br />
<p>Le projet de loi C-309, la Loi modifiant le Code criminel, a rendu illégale la dissimulation d’identité (utilisation de masques ou de déguisements) dans les émeutes ou les assemblées illégales<ref> Parliament of Canada. (June 19th, 2013). Bill C-309 An Act to Amend the Criminal Code (concealment of identity). Government of Canada. Retrieved 03-06-2019 from: <i>[https://www.parl.ca/LegisInfo/BillDetails.aspx?Bill=C309&Language=E&Mode=1&Parl=41&Ses=1 ]</i></ref>. Bien que la Loi sur la protection des renseignements personnels et la Loi sur la protection des renseignements personnels et les documents électroniques mentionnent que le consentement doit être obtenu avant la collecte de renseignements personnels, le projet de loi C-309 permet aux organismes d’application de la loi d’utiliser des logiciels de reconnaissance faciale dans les grandes foules pour mettre au jour l’identité des participants.</p><br />
<br />
<br />
<h2>Répercussions pour les agences gouvernementales</h2><br />
<br />
<h3>Services partagés Canada (SPC)</h3><br />
<br />
<h4>Proposition de valeur</h4><br />
<p class="inline">SPC pourrait tirer parti de cette technologie pour offrir la reconnaissance faciale comme service. Cette technologie remplacerait l’actuelle carte de sécurité des employés du gouvernement. Une caméra intelligente saisit instantanément les données biométriques des personnes et les analyse localement, puis ouvre le portail pour accéder au bâtiment. Ce service pourrait réduire les coûts de sécurité récurrents qu’entraîne la présence d’une équipe de sécurité sur place, mais il n’y aura probablement pas d’économies à court terme en raison du coût de développement des applications et d’installation du matériel connexe. </p><p class="expand inline mw-collapsible-content">La technologie de reconnaissance faciale est une forme non intrusive de vérification d’identité qui ne se perd pas. Dans la réalité de SPC, on élimine l’obligation pour les employés de transporter des cartes de sécurité. De plus, on empêcherait les personnes non autorisées d’accéder à des installations protégées. </p><p class="inline">L’authentification à deux facteurs avec le visage de l’utilisateur pourrait également s’employer pour accéder à des fichiers protégés ayant des classifications de sécurité supérieures (tels que des documents secrets).</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Les fabricants de téléphones intelligents ont de plus en plus tendance à créer des appareils qui peuvent être déverrouillés grâce à la technologie de reconnaissance faciale. </p><p class="inline">Une société d’études de marché de Hong Kong estime que près de 64 % (soit un milliard) de tous les téléphones intelligents expédiés dans le monde entier seront dotés de fonctions de reconnaissance faciale en 2020<ref>Naiya, Pavel. (February 7th, 2018) More than one billion smartphones to feature facial recognition in 2020. Counterpoint technology Market Research. Hong Kong, China. Retrieved 27-05-2019 from: <i>[https://www.counterpointresearch.com/one-billion-smartphones-feature-face-recognition-2020/]</i></ref>.Prenant appui sur cette recherche, SPC pourrait ne distribuer aux employés que des téléphones dotés de fonctions de reconnaissance faciale. Les informations du visage peuvent être associées à un autre type de méthode d’authentification pour créer un processus de vérification en deux étapes pour tous les téléphones intelligents. </p><p class="expand inline mw-collapsible-content">SPC n’aurait pas besoin d’acquérir de licences logicielles supplémentaires, car ses téléphones seraient déjà dotés de la capacité de reconnaissance faciale. Comme la vérification est effectuée localement (les images de référence sont stockées sur l’appareil en dehors du nuage), les risques de sécurité associés à la technologie de reconnaissance faciale sont minimisés.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">La reconnaissance faciale nécessite beaucoup de puissance pour le traitement des images en temps réel, un problème que pourrait résoudre l’informatique en périphérie. Les tâches de prétraitement de l’image peuvent être effectuées par l’appareil qui a pris la photo, ou beaucoup plus près de l’appareil que le centre de données. L’appareil capterait l’image, la balaierait à la recherche de visages puis extrairait l’information sous forme d’empreinte faciale. Une fois que l’empreinte faciale a été créée, elle est envoyée au serveur principal pour l’authentification, et l’image d’origine est éliminée. </p><p class="expand inline mw-collapsible-content">Comme le prétraitement de l’empreinte faciale a été effectué à l’extérieur du serveur, ce dernier n’a qu’à vérifier s’il y a correspondance interne avec l’empreinte faciale récente.</p><br />
<br />
<br />
<h4>Difficultés</h4><br />
<p class="expand mw-collapsible-content">La plus grande embûche sur le chemin de la technologie de reconnaissance faciale est la protection de la vie privée. L’une des solutions au problème de confidentialité est l’informatique en périphérie, grâce à laquelle on peut stocker les données biométriques localement et éviter la perte de données et la mauvaise mise en correspondance de données de deux systèmes différents. Il est préférable de stocker les données sous forme d’éléments biométriques (une empreinte faciale par exemple) plutôt que sous forme d’images de visages, qui devraient être détruites une fois l’empreinte collectée; les données d’empreinte faciale ne devraient être stockées, chiffrées et rendues accessibles que si on passe des contrôles de sécurité. De cette façon, on empêche l’utilisation de l’image pour des raisons non autorisées. </p><br />
<p>Si cette technologie devait être utilisée pour authentifier l’identité de personnes à plusieurs endroits, la puissance nécessaire exigerait la contribution de beaucoup de matériel. Si la technologie de reconnaissance faciale, avec ses millions d’empreintes faciales numérisées, devait être adoptée par au moins un ministère, la puissance nécessaire aux traitements et aux mises en correspondance avec les empreintes faciales dans la base de données serait considérable. De ce point de vue, si SPC devait prendre en charge un tel projet, la puissance requise pourrait provenir d’un nuage privé ou public.</p><br />
<p>Certains facteurs peuvent également limiter l’efficacité des systèmes de reconnaissance faciale. Si la photo a été prise de profil ou si la qualité de l’image est trop basse, il se peut que le système ne dispose pas d’informations suffisantes pour extraire de l’information et trouver une correspondance. Les coupes de cheveux, la couleur de la peau, le maquillage, les lunettes et les protections faciales comme les masques chirurgicaux peuvent également nuire à la reconnaissance. Compte tenu du fait que ces systèmes font appel à l’intelligence artificielle, il y a aussi la possibilité de leur faire apprendre les mauvaises choses.</p><br />
<p>Il devrait y avoir un quelconque mécanisme pour récompenser les systèmes de reconnaissance faciale qui établissent de bonnes correspondances, mais si les exemples utilisés pour l’apprentissage ne comptent qu’un groupe démographique bien circonscrit, les systèmes ne pourront pas détecter les autres types de visages. L’absence de diversité dans l’apprentissage crée des biais de reconnaissance à tel point que les systèmes n’auront de facilité que pour identifier des personnes ayant des traits particuliers.</p><br />
<br />
<br />
<p class="expand mw-collapsible">Dans une étude menée par Joy Buolamwini, où trois systèmes de reconnaissance faciale ont été testés pour déterminer le sexe, le taux d’erreur se situait entre 21 et 35 % chez les femmes dont la peau était plus foncée, alors que le taux d’erreur était inférieur à 1 % chez les hommes à la peau claire<ref>Lohr, Steve. (February 9th,2018). Facial Recognition is Accurate, if You’re a White Guy. New York Times. New York, USA. Retrieved 29-05-2019 from: <i>[https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html ]</i></ref>. On peut donc se poser des questions sur la fiabilité de ces systèmes. Pour éviter toute discrimination à l’endroit de groupes minoritaires, il faut mettre à l’essai et perfectionner les systèmes de manière à éviter les biais.</p><br />
<p class="inline">Ces systèmes ont également des correspondances partielles, autrement dit qui n’atteignent jamais la pleine certitude, lorsqu’ils effectuent la recherche dans une base de données d’images, par exemple. La possibilité de faux positifs est bien réelle (une correspondance est trouvée, mais ce n’est pas la bonne personne), comme celle de faux négatifs (il y a réellement une correspondance dans la base de données, mais le système ne la trouve pas). </p><p class="expand inline mw-collapsible-content">De tels risques d’erreur montrent que les systèmes ne devraient être utilisés que par des personnes formées qui en comprennent le fonctionnement et que des procédures devraient être établies lorsqu’il y a correspondance.</p><br />
<p class="expand mw-collapsible">En voici un bon exemple concret : le service de police de Toronto utilise le système, mais seulement six agents formés par le FBI peuvent s’en prévaloir, et le système ne génère qu’une liste de candidats. Le système ne peut pas en lui-même servir à arrêter des gens; il doit être utilisé de pair avec d’autres méthodes traditionnelles de collecte de preuves<ref>Burt, Chris. (May 28th, 2019). Toronto police using facial recognition as Canadian government ponders rules. Biometrics Research Group Inc. Retrieved 29-05-2019 from: <i>[https://www.biometricupdate.com/201905/toronto-police-using-facial-recognition-as-canadian-government-ponders-rules ]</i></ref>. Les systèmes d’intelligence artificielle, lorsqu’ils contribuent à la prise de décisions importantes, ne devraient jamais être utilisés seuls sans regard critique ni constituer l’argument central d’une décision.</p><br />
<p class="inline">Pour régler les problèmes de mauvais éclairage ou d’angles trop prononcés, certains systèmes modifient les images pour qu’elles soient plus faciles à lire. </p><p class="expand inline mw-collapsible-content">Panasonic a créé un logiciel de reconnaissance faciale qui analyse les mouvements, la vitesse et l’éclairage présents dans les vidéos et corrige automatiquement les images fixes qui seraient autrement floues<ref>Panasonic. (February 20th, 2018) Panasonic to Launch Face Recognition Server Software Using Deep Learning Technology. Panasonic Corporation. Kadoma, Japan. Retrieved 15-05-2019 from: <i>[https://security.panasonic.com/news/archives/686 ]</i></ref>. </p><p class="inline">Comme le logiciel modifie l’image avant de l’analyser, il augmente le risque de faux positifs. Le fait de retoucher une image avant de l’intégrer à un système de reconnaissance faciale peut modifier l’empreinte faciale analysée et créer un biais dans les résultats de la recherche.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Autre limite, ces systèmes ne peuvent reconnaître que les personnes dont les images sont déjà contenues dans leur base de données. </p><p class="expand inline mw-collapsible-content">Les systèmes doivent également être capables de déterminer si la personne est réellement devant eux, car les visages ne peuvent être cachés comme les mots de passe. En effet, les systèmes ne sont efficaces que parce qu’il est trop difficile de se faire passer pour un autre. Le système doit ainsi être capable de faire la différence entre une personne réelle et une simple photo.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">La technologie de reconnaissance faciale n’a pas encore été réglementée au Canada, et les organisations qui l’utilisent actuellement doivent respecter un cadre juridique précis. Conformément à la Loi sur la protection des renseignements personnels du Canada, les institutions fédérales ne peuvent utiliser les renseignements personnels qu’aux fins auxquelles ils ont été recueillis, et il faut le consentement de la personne concernée avant que ces renseignements puissent être utilisés à une autre fin. Comme l’exige la Loi sur la protection des renseignements personnels et les documents électroniques, une organisation doit informer les personnes et obtenir leur consentement concernant l’utilisation de leurs renseignements personnels<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303]/</i></ref>. </p><p class="expand inline mw-collapsible-content">Il s’agit là d’un obstacle juridique potentiel pour toute organisation qui voudrait faire une lecture de foule, car chaque personne devrait alors consentir à la collecte et à l’utilisation de son visage (renseignements personnels). </p><p class="inline">Les lois font en sorte que les bases de données contenant des renseignements personnels appartenant à différents ministères du gouvernement du Canada ne peuvent être partagées entre eux à des fins autres que celles qui ont fait l’objet du consentement.</p><br />
<br />
<h4>Considérations</h4><br />
<br />
<p>Toute utilisation à grande échelle envisagée par SPC devra être soumise à l’évaluation du Commissariat à la protection de la vie privée. SPC devra également dans tous les cas se conformer à la Loi sur la protection des renseignements personnels et à la Loi sur la protection des renseignements personnels et les documents électroniques. Il est aussi obligatoire de justifier les utilisations étant donné les intrusions possibles dans la vie privée; le Commissariat à la protection de la vie privée propose à cet effet un test en quatre parties :<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303/]</i></ref> </p><br />
<ul><br />
<li>Est-il démontré que la mesure est nécessaire pour répondre à un besoin précis?</li><br />
<li>Cette mesure est-elle susceptible de répondre efficacement à ce besoin?</li><br />
<li>La perte au chapitre de la vie privée serait-elle proportionnelle à l’avantage obtenu?</li><br />
<li>Existe-t-il un autre moyen moins envahissant qui pourrait permettre d’atteindre le même but?</li><br />
</ul><br />
<p class="expand mw-collapsible-content">De plus, les utilisations à grande échelle au sein de SPC, comme dans le cas des systèmes de reconnaissance faciale pour accéder aux bâtiments sécurisés, devraient être soumises au consentement de tous les participants. Des photos de haute qualité avec des expressions faciales neutres devraient également être prises ou tirées de bases de données existantes (comme celle des cartes d’accès). Les répertoires centraux de photos du personnel pourraient également devenir la cible d’attaques de cybersécurité. Une protection maximale devra leur être accordée. Les utilisations de moindre ampleur, comme pour le déverrouillage d’un appareil avec capture faciale, poseraient moins de risques puisque l’information est stockée localement.</p><br />
<p>Si SPC devait adopter ou créer un logiciel de reconnaissance faciale, le logiciel devrait être testé afin de mettre au jour les biais potentiels. Autrement dit, le logiciel devrait pouvoir reconnaître tout le monde, peu importe le sexe, le type de peau ou l’âge. S’il y a des biais, le logiciel devrait retourner en développement, ce qui pourrait entraîner d’autres coûts. Les applications de reconnaissance faciale achetées d’un fournisseur devraient être étudiées avec soin étant donné que SPC ne saurait pas exactement comment l’application a été conçue. De plus, il n’existe aucune norme applicable aux produits de reconnaissance faciale, ce qui signifie qu’il n’y a pas de niveau de performance à atteindre avant qu’un produit ne soit mis en marché. Si SPC créait lui-même son logiciel, il saurait intégralement comment il fonctionne et aurait la mainmise sur lui.</p><br />
<p class="expand mw-collapsible-content">Enfin, il est important de tenir compte du contexte juridique applicable à tout système de reconnaissance faciale déployé au sein de SPC. À l’heure actuelle, la technologie n’est pas réglementée et aucune norme de performance n’a été établie pour les développeurs. Le paysage juridique pourrait avoir changé au moment où SPC sera prêt à adopter la technologie. Il pourrait être utile d’évaluer à long terme comment les services actuellement offerts respectent les exigences de protection de la vie privée si on veut bien comprendre les effets potentiels de la technologie de reconnaissance faciale.</p><br />
<br />
<h2>References</h2><br />
<br />
<div style="display: none"><br />
<br />
<br />
<ref>Facial recognition system. (2019, July 16). Retrieved from <i>[https://en.wikipedia.org/wiki/Facial_recognition_system]</i></ref><br />
<ref>Gartner_Inc. (n.d.). Face Recognition Software: Anti-Terrorism Tool and More. Retrieved from <i>[https://www.gartner.com/doc/341020/face-recognition-software-antiterrorism-tool]</i></ref><br />
<ref>Solutions, T. (2018, September 25). Pros and Cons of Facial Recognition Technology For Your Business. Retrieved from <i>[https://www.upwork.com/hiring/for-clients/pros-cons-facial-recognition-technology-business/]</i></ref><br />
<ref>5 Applications of Facial Recognition Technology. (2017, July 09). Retrieved from <i>[https://disruptionhub.com/5-applications-facial-recognition-technology/]</i></ref><br />
<ref>Biometric Facial Recognition. (n.d.). Retrieved from <i>[https://findbiometrics.com/solutions/facial-recognition/]</i></ref><br />
<ref>How Facial Recognition Systems Work. (2001, September 04). Retrieved from <i>[https://electronics.howstuffworks.com/gadgets/high-tech-gadgets/facial-recognition2.htm]</i></ref><br />
<ref>Virdee-Chapman, B. (2016, May 26). 5 Companies Using Facial Recognition to Change The World. Retrieved from <i>[https://www.kairos.com/blog/5-companies-using-facial-recognition-to-change-the-world]</i></ref><br />
<ref>Sennaar, K. (2019, May 20). Facial Recognition Applications – Security, Retail, and Beyond. Retrieved from <i>[https://www.techemergence.com/facial-recognition-applications/]</i></ref><br />
<ref>Facial recognition technology is coming to Canadian airports this spring | CBC News. (2017, March 06). Retrieved from <i>[http://www.cbc.ca/news/technology/cbsa-canada-airports-facial-recognition-kiosk-biometrics-1.4007344]</i></ref><br />
<ref>Lee, J. (2016, June 09). Canadian government used facial recognition to detect passport fraudsters. Retrieved from <i>[https://www.biometricupdate.com/201606/canadian-government-used-facial-recognition-to-detect-passport-fraudsters]</i></ref><br />
<ref>Solutions, T. (2018, September 25). Pros and Cons of Facial Recognition Technology For Your Business. Retrieved from <i>[https://www.upwork.com/hiring/for-clients/pros-cons-facial-recognition-technology-business/]</i></ref><br />
<ref>Face ID Security. (2017, November). Retrieved from apple.com:<i>[https://www.apple.com/ca/business-docs/FaceID_Security_Guide.pdf]</i></ref><br />
</div><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
#firstHeading::after{<br />
content:"Réseaux Zéro Confiance";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.highlighted{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Tendances_Technologiques/Reconnaissance_Faciale&diff=11303Tendances Technologiques/Reconnaissance Faciale2019-07-22T15:45:28Z<p>Kpere060: Created page with "{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}} en:Technology_Trends/Zero_Trust_Netwo..."</p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[en:Technology_Trends/Zero_Trust_Network]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Afficher la Vue Détaillée" data-collapsetext="Cacher la Vue Détaillée"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>Reconnaissance de Visage</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>[[Technology_Trends/Zero_Trust_Network|Face Recognition]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Publié</td><br />
</tr><br />
<tr><br />
<th>Version original</th><br />
<td>25 juin 2019</td><br />
</tr><br />
<tr><br />
<th>Mise à jour</th><br />
<td>22 juillet 2019</td><br />
</tr><br />
<tr><br />
<th>Publication officiel</th><br />
<td>[[Media:FR_-_FR_Face Recognition v0.1.pdf|Réconnaisance_de_Visage.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">Cette page est sujet à des mises à jours. On encourage toutes rétroactions. Veuillez utiliser la page de discussion pour apporter des commentaires et des suggestions. Une fois que cette version soit approuvée et finalisée, elle va être traduit.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p>La <b>reconnaissance faciale </b> est une technologie biométrique servant à établir et à valider l’identité d’une personne. Elle fait appel à une application informatique, ou système de reconnaissance faciale, qui extrait une image numérique d’une photo, d’un cadre vidéo ou d’une numérisation tridimensionnelle et crée une empreinte faciale, soit un ensemble de mesures caractéristiques d’un visage, qui identifie le visage particulier d’une personne aux fins d’authentification.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Cacher la Vue Détaillée</span></div><br />
</div><br><br><br />
<br />
<h2>Sommaire opérationnel</h2><br />
<br />
<p class="expand inline mw-collapsible-content">Généralement, l’authentification et l’identification se font par correspondance entre les caractéristiques faciales et les empreintes faciales d’un profil dans une base de données. </p><p class="inline">La technologie de reconnaissance faciale se décline en de nombreuses applications (contrôle d’accès, surveillance et enquêtes criminelles). Elle s’utilise également en combinaison avec d’autres technologies biométriques pour renforcer les mesures de sécurité.</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Les grandes sociétés technologiques comme Apple, Google, Samsung, Facebook et Amazon ont commencé à prendre conscience de l’importance que revêt la reconnaissance faciale pour leurs infrastructures de sécurité. </p><p class="inline">Apple a essayé de perfectionner la technologie en y ajoutant la reconnaissance de mouvements. La personne dont le visage est numérisé peut maintenant parler ou bouger pendant la numérisation, et ainsi la reconnaissance faciale peut se combiner avec d’autres mesures de sécurité biométriques comme la reconnaissance vocale. Étant donné que les visages en mouvement peuvent être numérisés, les systèmes de reconnaissance faciale peuvent identifier les gens même dans une foule, et ce, sans intrusion.</p><br />
<br />
<br />
<h2>Sommaire technique</h2><br />
<p class="inline">Les systèmes de reconnaissance faciale peuvent utiliser une image bidimensionnelle ou tridimensionnelle ou une alimentation vidéo pour créer une image numérique, établir l’empreinte faciale et identifier un visage en comparant l’image numérique avec les empreintes faciales dans la base de données. Chaque visage a ses caractéristiques déterminantes que le système marquera comme des points nodaux. Un visage humain peut compter jusqu’à 80 de ces points. Ils constituent des zones d’intérêt sur le visage mesurées par le système. </p><p class="expand inline mw-collapsible-content">SLa distance entre les yeux, la largeur du nez et la profondeur de l’orbite en sont des exemples. Ces mesures seront stockées dans une base de données sous forme d’empreinte faciale. </p><p class="inline">Lorsque le système numérise un visage, il compare toutes ces mesures aux profils, soit les empreintes faciales, dans la base de données. </p><p class="expand inline mw-collapsible-content">Les systèmes de reconnaissance faciale utilisent un algorithme, comme le Facial Recognition Vendor Test, qui permet de prédire s’il y a une correspondance en fonction des points nodaux du visage d’une personne. </p><p class="inline">Habituellement, la technologie suit quatre étapes : <ref><i>[http://www.ex-sight.com/technology.html]</i></ref>: </p><br />
<p>Capture –un échantillon physique ou comportemental est capté par le système lors d’une analyse;</p><br />
<p>extraction – des données uniques sont extraites de l’échantillon, et un modèle est créé;</p><br />
<p>comparaison – le modèle est ensuite comparé à un autre échantillon;</p><br />
<p>correspondance – le système décide alors si la caractéristique extraite du nouvel échantillon a une correspondance.</p><br />
<br />
<br />
<h2>Utilisation par l'industrie</h2><br />
<p class="inline">La reconnaissance faciale offre une autre forme d’identification et d’authentification biométriques. Plusieurs fournisseurs utilisent la reconnaissance faciale comme outil de contrôle d’accès et d’authentification pour leurs clients ou pour un usage interne. Son application ne se limite pas à la sécurité; la technologie sert également aux soins de santé et à la vente au détail.</p><p class="expand inline mw-collapsible-content"> Bien que la reconnaissance faciale bidimensionnelle ne soit pas aussi précise que d’autres formes de technologies biométriques comme les lecteurs d’empreintes digitales, elle a ses avantages. </p><p class="inline">La personne dont le visage est numérisé ne sait pas nécessairement quand il y a numérisation; ainsi, la technologie peut s’utiliser dans de grandes foules et mettre rapidement en évidence les menaces.</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Plusieurs fournisseurs sur le marché actuel emploient la reconnaissance faciale pour leurs nombreuses applications. </p><p class="inline">Par exemple, Amazon a mis au point un système qui permet aux utilisateurs de payer leurs articles à l’aide d’une image exploitable (égoportrait). L’utilisateur peut en effet utiliser un égoportrait dans lequel il se déplace ou prononce une phrase particulière comme mot de passe pour valider son identité lors du paiement d’un article. Le fait que le client doive prononcer une phrase ou faire un mouvement se veut un moyen d’éliminer la possibilité d’utiliser une image bidimensionnelle numérisée de la personne de manière frauduleuse. </p><p class="expand inline mw-collapsible-content">Amazon Rekognition est un autre produit qui fournit deux ensembles d’API : Amazon Rekognition Image pour les images et Amazon Rekognition Video pour les vidéos. Les deux API effectuent des analyses de détection et de reconnaissance d’images et de vidéos qui procurent de l’information utile à vos applications.</p><br />
<p>Après avoir acheté Face.com en 2012, Facebook a commencé à utiliser la technologie de reconnaissance faciale pour connecter les utilisateurs par leurs photos. Lorsqu’un utilisateur téléverse une photo, le logiciel suggère automatiquement d’autres personnes à identifier. Lorsqu’on est identifié dans une photo, on peut voir plus de contenu regroupé sur les personnes qui y sont identifiées. </p><br />
<p class="inline">Face ID est une technologie développée par Apple et accessible sur l’iPhone X. Elle offre une authentification intuitive et sécurisée grâce au système de caméra TrueDepth à la fine pointe de la technologie et aux technologies perfectionnées permettant de cartographier avec précision la géométrie du visage. D’un simple coup d’œil, Face ID déverrouille en toute sécurité votre iPhone ou iPad Pro. </p><p class="expand inline mw-collapsible-content">Vous pouvez l’utiliser pour autoriser des achats sur iTunes Store, App Store et Apple Books, et effectuer des paiements avec Apple Pay. Les iPhone XR, XS et XS Max sont tous équipés de la deuxième génération de Face ID, une version mise à jour du système d’authentification biométrique qui est censée être plus rapide que la version de l’iPhone X.</p><br />
<h2>Utilisation par le gouvernement du Canada</h2><br />
<p class="expand mw-collapsible-content">Contrairement au secteur privé, l’utilité des applications d’identification faciale pour le gouvernement est principalement liée à la sécurité, en particulier pour la vérification d’identité et la prévention des fraudes. Par exemple, l’Agence des services frontaliers du Canada a récemment lancé le programme des bornes d’inspection primaire, dans le cadre duquel les passagers qui entrent au pays depuis des aéroports doivent s’enregistrer dans des kiosques libre-service.<ref>Braga, Matthew. (March 2nd, 2017). Facial Recognition Technology is coming to Canadian Airports this spring. Canadian Broadcasting Corporation. Retrieved 17-05-2019 from: <i>[https://www.cbc.ca/news/technology/cbsa-canada-airports-facial-recognition-kiosk-biometrics-1.4007344]</i></ref>. Ces bornes vérifient l’identité des passagers par la reconnaissance faciale. L’adoption progressive des bornes sans personnel depuis 2015 a augmenté la sécurité tout en réduisant la congestion dans les aéroports. L’entreprise portugaise Vision-Box a installé 130 bornes à l’aéroport international Pearson de Toronto. Les bornes ont été conçues pour effectuer une reconnaissance biométrique en deux étapes : une étape de reconnaissance faciale et une autre de reconnaissance des empreintes digitales. Ces bornes seront également en mesure d’effectuer une reconnaissance de l’iris, une fonction réservée aux personnes inscrites au programme NEXUS.</p><br />
<p class="inline">Les systèmes de reconnaissance faciale sont également utilisés dans les casinos provinciaux pour identifier les visiteurs ayant une dépendance au jeu qui se sont volontairement inscrits sur des listes d’auto-exclusion et les empêcher d’entrer .<ref>Elash, Anita, and Luk, Vivian. (July 25th, 2011). Canadian Casinos, Banks, Police use Facial-Recognition Technology. The Globe and Mail. Toronto, Ontario. Retrieved 21-05-2019 from: <i>[https://www.theglobeandmail.com/news/national/time-to-lead/canadian-casinos-banks-police-use-facial-recognition-technology/article590998/ ]</i></ref>. Il convient de noter que le système a été élaboré conjointement avec le Commissaire à la protection de la vie privée de l’Ontario afin que soient prises en compte dans la conception par défaut les questions de protection de la vie privée. </p><p class="expand inline mw-collapsible-content">En temps réel, le système fait une lecture du visage des clients qui entrent dans le casino et compare leurs images à celles des joueurs de la liste d’auto-exclusion. S’il y a correspondance, le système avertit la sécurité, sinon le système supprime automatiquement l’image. L’accès à la base de données est restreint, et les renseignements d’une personne ne sont accessibles que si la personne sur la photo est physiquement sur place. </p><br />
<p class="inline">Passeport Canada utilise un logiciel de reconnaissance faciale depuis dix ans pour comparer les nouvelles photos de passeport à sa base de données afin de prévenir les fraudes. On effectue des comparaisons un à un pour confirmer l’identité d’une personne, ce qui signifie qu’une image récente est comparée à une image déjà dans la base de données qui est associée à l’identité de la personne en question. Les comparaisons un à plusieurs servent à comparer une image à l’ensemble de la base de données de photos d’identité afin de vérifier s’il y a des demandeurs en double ou des personnes ayant plusieurs identités<ref>Mackrael, Kim, and Ha, Tu Thanh. (May 15th, 2014) Facial Recognition Program Allows RCMP to Identify Alleged Passport Fraud. The Globe and Mail. Toronto, Ontario. Retrieved 27-05-2019 from: <i>[https://www.theglobeandmail.com/news/national/facial-recognition-program-allows-rcmp-to-nab-alleged-passport-fraudster/article18703608/]</i></ref>. </p><p class="expand inline mw-collapsible-content">Le projet a réussi à mettre au jour des cas de personnes qui tentaient d’obtenir plusieurs passeports. Ce même concept est également utilisé pour les permis de conduire délivrés par les provinces<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303/]</i></ref>. </p><br />
<p>Le projet de loi C-309, la Loi modifiant le Code criminel, a rendu illégale la dissimulation d’identité (utilisation de masques ou de déguisements) dans les émeutes ou les assemblées illégales<ref> Parliament of Canada. (June 19th, 2013). Bill C-309 An Act to Amend the Criminal Code (concealment of identity). Government of Canada. Retrieved 03-06-2019 from: <i>[https://www.parl.ca/LegisInfo/BillDetails.aspx?Bill=C309&Language=E&Mode=1&Parl=41&Ses=1 ]</i></ref>. Bien que la Loi sur la protection des renseignements personnels et la Loi sur la protection des renseignements personnels et les documents électroniques mentionnent que le consentement doit être obtenu avant la collecte de renseignements personnels, le projet de loi C-309 permet aux organismes d’application de la loi d’utiliser des logiciels de reconnaissance faciale dans les grandes foules pour mettre au jour l’identité des participants.</p><br />
<br />
<br />
<h2>Répercussions pour les agences gouvernementales</h2><br />
<br />
<h3>Services partagés Canada (SPC)</h3><br />
<br />
<h4>Proposition de valeur</h4><br />
<p class="inline">SPC pourrait tirer parti de cette technologie pour offrir la reconnaissance faciale comme service. Cette technologie remplacerait l’actuelle carte de sécurité des employés du gouvernement. Une caméra intelligente saisit instantanément les données biométriques des personnes et les analyse localement, puis ouvre le portail pour accéder au bâtiment. Ce service pourrait réduire les coûts de sécurité récurrents qu’entraîne la présence d’une équipe de sécurité sur place, mais il n’y aura probablement pas d’économies à court terme en raison du coût de développement des applications et d’installation du matériel connexe. </p><p class="expand inline mw-collapsible-content">La technologie de reconnaissance faciale est une forme non intrusive de vérification d’identité qui ne se perd pas. Dans la réalité de SPC, on élimine l’obligation pour les employés de transporter des cartes de sécurité. De plus, on empêcherait les personnes non autorisées d’accéder à des installations protégées. </p><p class="inline">L’authentification à deux facteurs avec le visage de l’utilisateur pourrait également s’employer pour accéder à des fichiers protégés ayant des classifications de sécurité supérieures (tels que des documents secrets).</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Les fabricants de téléphones intelligents ont de plus en plus tendance à créer des appareils qui peuvent être déverrouillés grâce à la technologie de reconnaissance faciale. </p><p class="inline">Une société d’études de marché de Hong Kong estime que près de 64 % (soit un milliard) de tous les téléphones intelligents expédiés dans le monde entier seront dotés de fonctions de reconnaissance faciale en 2020<ref>Naiya, Pavel. (February 7th, 2018) More than one billion smartphones to feature facial recognition in 2020. Counterpoint technology Market Research. Hong Kong, China. Retrieved 27-05-2019 from: <i>[https://www.counterpointresearch.com/one-billion-smartphones-feature-face-recognition-2020/]</i></ref>.Prenant appui sur cette recherche, SPC pourrait ne distribuer aux employés que des téléphones dotés de fonctions de reconnaissance faciale. Les informations du visage peuvent être associées à un autre type de méthode d’authentification pour créer un processus de vérification en deux étapes pour tous les téléphones intelligents. </p><p class="expand inline mw-collapsible-content">SPC n’aurait pas besoin d’acquérir de licences logicielles supplémentaires, car ses téléphones seraient déjà dotés de la capacité de reconnaissance faciale. Comme la vérification est effectuée localement (les images de référence sont stockées sur l’appareil en dehors du nuage), les risques de sécurité associés à la technologie de reconnaissance faciale sont minimisés.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">La reconnaissance faciale nécessite beaucoup de puissance pour le traitement des images en temps réel, un problème que pourrait résoudre l’informatique en périphérie. Les tâches de prétraitement de l’image peuvent être effectuées par l’appareil qui a pris la photo, ou beaucoup plus près de l’appareil que le centre de données. L’appareil capterait l’image, la balaierait à la recherche de visages puis extrairait l’information sous forme d’empreinte faciale. Une fois que l’empreinte faciale a été créée, elle est envoyée au serveur principal pour l’authentification, et l’image d’origine est éliminée. </p><p class="expand inline mw-collapsible-content">Comme le prétraitement de l’empreinte faciale a été effectué à l’extérieur du serveur, ce dernier n’a qu’à vérifier s’il y a correspondance interne avec l’empreinte faciale récente.</p><br />
<br />
<br />
<h4>Difficultés</h4><br />
<p class="expand mw-collapsible-content">La plus grande embûche sur le chemin de la technologie de reconnaissance faciale est la protection de la vie privée. L’une des solutions au problème de confidentialité est l’informatique en périphérie, grâce à laquelle on peut stocker les données biométriques localement et éviter la perte de données et la mauvaise mise en correspondance de données de deux systèmes différents. Il est préférable de stocker les données sous forme d’éléments biométriques (une empreinte faciale par exemple) plutôt que sous forme d’images de visages, qui devraient être détruites une fois l’empreinte collectée; les données d’empreinte faciale ne devraient être stockées, chiffrées et rendues accessibles que si on passe des contrôles de sécurité. De cette façon, on empêche l’utilisation de l’image pour des raisons non autorisées. </p><br />
<p>Si cette technologie devait être utilisée pour authentifier l’identité de personnes à plusieurs endroits, la puissance nécessaire exigerait la contribution de beaucoup de matériel. Si la technologie de reconnaissance faciale, avec ses millions d’empreintes faciales numérisées, devait être adoptée par au moins un ministère, la puissance nécessaire aux traitements et aux mises en correspondance avec les empreintes faciales dans la base de données serait considérable. De ce point de vue, si SPC devait prendre en charge un tel projet, la puissance requise pourrait provenir d’un nuage privé ou public.</p><br />
<p>Certains facteurs peuvent également limiter l’efficacité des systèmes de reconnaissance faciale. Si la photo a été prise de profil ou si la qualité de l’image est trop basse, il se peut que le système ne dispose pas d’informations suffisantes pour extraire de l’information et trouver une correspondance. Les coupes de cheveux, la couleur de la peau, le maquillage, les lunettes et les protections faciales comme les masques chirurgicaux peuvent également nuire à la reconnaissance. Compte tenu du fait que ces systèmes font appel à l’intelligence artificielle, il y a aussi la possibilité de leur faire apprendre les mauvaises choses.</p><br />
<p>Il devrait y avoir un quelconque mécanisme pour récompenser les systèmes de reconnaissance faciale qui établissent de bonnes correspondances, mais si les exemples utilisés pour l’apprentissage ne comptent qu’un groupe démographique bien circonscrit, les systèmes ne pourront pas détecter les autres types de visages. L’absence de diversité dans l’apprentissage crée des biais de reconnaissance à tel point que les systèmes n’auront de facilité que pour identifier des personnes ayant des traits particuliers.</p><br />
<br />
<br />
<p class="expand mw-collapsible">Dans une étude menée par Joy Buolamwini, où trois systèmes de reconnaissance faciale ont été testés pour déterminer le sexe, le taux d’erreur se situait entre 21 et 35 % chez les femmes dont la peau était plus foncée, alors que le taux d’erreur était inférieur à 1 % chez les hommes à la peau claire<ref>Lohr, Steve. (February 9th,2018). Facial Recognition is Accurate, if You’re a White Guy. New York Times. New York, USA. Retrieved 29-05-2019 from: <i>[https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html ]</i></ref>. On peut donc se poser des questions sur la fiabilité de ces systèmes. Pour éviter toute discrimination à l’endroit de groupes minoritaires, il faut mettre à l’essai et perfectionner les systèmes de manière à éviter les biais.</p><br />
<p class="inline">Ces systèmes ont également des correspondances partielles, autrement dit qui n’atteignent jamais la pleine certitude, lorsqu’ils effectuent la recherche dans une base de données d’images, par exemple. La possibilité de faux positifs est bien réelle (une correspondance est trouvée, mais ce n’est pas la bonne personne), comme celle de faux négatifs (il y a réellement une correspondance dans la base de données, mais le système ne la trouve pas). </p><p class="expand inline mw-collapsible-content">De tels risques d’erreur montrent que les systèmes ne devraient être utilisés que par des personnes formées qui en comprennent le fonctionnement et que des procédures devraient être établies lorsqu’il y a correspondance.</p><br />
<p class="expand mw-collapsible">En voici un bon exemple concret : le service de police de Toronto utilise le système, mais seulement six agents formés par le FBI peuvent s’en prévaloir, et le système ne génère qu’une liste de candidats. Le système ne peut pas en lui-même servir à arrêter des gens; il doit être utilisé de pair avec d’autres méthodes traditionnelles de collecte de preuves<ref>Burt, Chris. (May 28th, 2019). Toronto police using facial recognition as Canadian government ponders rules. Biometrics Research Group Inc. Retrieved 29-05-2019 from: <i>[https://www.biometricupdate.com/201905/toronto-police-using-facial-recognition-as-canadian-government-ponders-rules ]</i></ref>. Les systèmes d’intelligence artificielle, lorsqu’ils contribuent à la prise de décisions importantes, ne devraient jamais être utilisés seuls sans regard critique ni constituer l’argument central d’une décision.</p><br />
<p class="inline">Pour régler les problèmes de mauvais éclairage ou d’angles trop prononcés, certains systèmes modifient les images pour qu’elles soient plus faciles à lire. </p><p class="expand inline mw-collapsible-content">Panasonic a créé un logiciel de reconnaissance faciale qui analyse les mouvements, la vitesse et l’éclairage présents dans les vidéos et corrige automatiquement les images fixes qui seraient autrement floues<ref>Panasonic. (February 20th, 2018) Panasonic to Launch Face Recognition Server Software Using Deep Learning Technology. Panasonic Corporation. Kadoma, Japan. Retrieved 15-05-2019 from: <i>[https://security.panasonic.com/news/archives/686 ]</i></ref>. </p><p class="inline">Comme le logiciel modifie l’image avant de l’analyser, il augmente le risque de faux positifs. Le fait de retoucher une image avant de l’intégrer à un système de reconnaissance faciale peut modifier l’empreinte faciale analysée et créer un biais dans les résultats de la recherche.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Autre limite, ces systèmes ne peuvent reconnaître que les personnes dont les images sont déjà contenues dans leur base de données. </p><p class="expand inline mw-collapsible-content">Les systèmes doivent également être capables de déterminer si la personne est réellement devant eux, car les visages ne peuvent être cachés comme les mots de passe. En effet, les systèmes ne sont efficaces que parce qu’il est trop difficile de se faire passer pour un autre. Le système doit ainsi être capable de faire la différence entre une personne réelle et une simple photo.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">La technologie de reconnaissance faciale n’a pas encore été réglementée au Canada, et les organisations qui l’utilisent actuellement doivent respecter un cadre juridique précis. Conformément à la Loi sur la protection des renseignements personnels du Canada, les institutions fédérales ne peuvent utiliser les renseignements personnels qu’aux fins auxquelles ils ont été recueillis, et il faut le consentement de la personne concernée avant que ces renseignements puissent être utilisés à une autre fin. Comme l’exige la Loi sur la protection des renseignements personnels et les documents électroniques, une organisation doit informer les personnes et obtenir leur consentement concernant l’utilisation de leurs renseignements personnels<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303]/</i></ref>. </p><p class="expand inline mw-collapsible-content">Il s’agit là d’un obstacle juridique potentiel pour toute organisation qui voudrait faire une lecture de foule, car chaque personne devrait alors consentir à la collecte et à l’utilisation de son visage (renseignements personnels). </p><p class="inline">Les lois font en sorte que les bases de données contenant des renseignements personnels appartenant à différents ministères du gouvernement du Canada ne peuvent être partagées entre eux à des fins autres que celles qui ont fait l’objet du consentement.</p><br />
<br />
<h4>Considérations</h4><br />
<br />
<p>Toute utilisation à grande échelle envisagée par SPC devra être soumise à l’évaluation du Commissariat à la protection de la vie privée. SPC devra également dans tous les cas se conformer à la Loi sur la protection des renseignements personnels et à la Loi sur la protection des renseignements personnels et les documents électroniques. Il est aussi obligatoire de justifier les utilisations étant donné les intrusions possibles dans la vie privée; le Commissariat à la protection de la vie privée propose à cet effet un test en quatre parties :<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303/]</i></ref> </p><br />
<ul><br />
<li>• Est-il démontré que la mesure est nécessaire pour répondre à un besoin précis?</li><br />
<li>• Cette mesure est-elle susceptible de répondre efficacement à ce besoin?</li><br />
<li>La perte au chapitre de la vie privée serait-elle proportionnelle à l’avantage obtenu?</li><br />
<li>• Existe-t-il un autre moyen moins envahissant qui pourrait permettre d’atteindre le même but?</li><br />
</ul><br />
<p class="expand mw-collapsible-content">De plus, les utilisations à grande échelle au sein de SPC, comme dans le cas des systèmes de reconnaissance faciale pour accéder aux bâtiments sécurisés, devraient être soumises au consentement de tous les participants. Des photos de haute qualité avec des expressions faciales neutres devraient également être prises ou tirées de bases de données existantes (comme celle des cartes d’accès). Les répertoires centraux de photos du personnel pourraient également devenir la cible d’attaques de cybersécurité. Une protection maximale devra leur être accordée. Les utilisations de moindre ampleur, comme pour le déverrouillage d’un appareil avec capture faciale, poseraient moins de risques puisque l’information est stockée localement.</p><br />
<p>Si SPC devait adopter ou créer un logiciel de reconnaissance faciale, le logiciel devrait être testé afin de mettre au jour les biais potentiels. Autrement dit, le logiciel devrait pouvoir reconnaître tout le monde, peu importe le sexe, le type de peau ou l’âge. S’il y a des biais, le logiciel devrait retourner en développement, ce qui pourrait entraîner d’autres coûts. Les applications de reconnaissance faciale achetées d’un fournisseur devraient être étudiées avec soin étant donné que SPC ne saurait pas exactement comment l’application a été conçue. De plus, il n’existe aucune norme applicable aux produits de reconnaissance faciale, ce qui signifie qu’il n’y a pas de niveau de performance à atteindre avant qu’un produit ne soit mis en marché. Si SPC créait lui-même son logiciel, il saurait intégralement comment il fonctionne et aurait la mainmise sur lui.</p><br />
<p class="expand mw-collapsible-content">Enfin, il est important de tenir compte du contexte juridique applicable à tout système de reconnaissance faciale déployé au sein de SPC. À l’heure actuelle, la technologie n’est pas réglementée et aucune norme de performance n’a été établie pour les développeurs. Le paysage juridique pourrait avoir changé au moment où SPC sera prêt à adopter la technologie. Il pourrait être utile d’évaluer à long terme comment les services actuellement offerts respectent les exigences de protection de la vie privée si on veut bien comprendre les effets potentiels de la technologie de reconnaissance faciale.</p><br />
<br />
<h2>References</h2><br />
<br />
<div style="display: none"><br />
<br />
<br />
<ref>Facial recognition system. (2019, July 16). Retrieved from <i>[https://en.wikipedia.org/wiki/Facial_recognition_system]</i></ref><br />
<ref>Gartner_Inc. (n.d.). Face Recognition Software: Anti-Terrorism Tool and More. Retrieved from <i>[https://www.gartner.com/doc/341020/face-recognition-software-antiterrorism-tool]</i></ref><br />
<ref>Solutions, T. (2018, September 25). Pros and Cons of Facial Recognition Technology For Your Business. Retrieved from <i>[https://www.upwork.com/hiring/for-clients/pros-cons-facial-recognition-technology-business/]</i></ref><br />
<ref>5 Applications of Facial Recognition Technology. (2017, July 09). Retrieved from <i>[https://disruptionhub.com/5-applications-facial-recognition-technology/]</i></ref><br />
<ref>Biometric Facial Recognition. (n.d.). Retrieved from <i>[https://findbiometrics.com/solutions/facial-recognition/]</i></ref><br />
<ref>How Facial Recognition Systems Work. (2001, September 04). Retrieved from <i>[https://electronics.howstuffworks.com/gadgets/high-tech-gadgets/facial-recognition2.htm]</i></ref><br />
<ref>Virdee-Chapman, B. (2016, May 26). 5 Companies Using Facial Recognition to Change The World. Retrieved from <i>[https://www.kairos.com/blog/5-companies-using-facial-recognition-to-change-the-world]</i></ref><br />
<ref>Sennaar, K. (2019, May 20). Facial Recognition Applications – Security, Retail, and Beyond. Retrieved from <i>[https://www.techemergence.com/facial-recognition-applications/]</i></ref><br />
<ref>Facial recognition technology is coming to Canadian airports this spring | CBC News. (2017, March 06). Retrieved from <i>[http://www.cbc.ca/news/technology/cbsa-canada-airports-facial-recognition-kiosk-biometrics-1.4007344]</i></ref><br />
<ref>Lee, J. (2016, June 09). Canadian government used facial recognition to detect passport fraudsters. Retrieved from <i>[https://www.biometricupdate.com/201606/canadian-government-used-facial-recognition-to-detect-passport-fraudsters]</i></ref><br />
<ref>Solutions, T. (2018, September 25). Pros and Cons of Facial Recognition Technology For Your Business. Retrieved from <i>[https://www.upwork.com/hiring/for-clients/pros-cons-facial-recognition-technology-business/]</i></ref><br />
<ref>Face ID Security. (2017, November). Retrieved from apple.com:<i>[https://www.apple.com/ca/business-docs/FaceID_Security_Guide.pdf]</i></ref><br />
</div><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
#firstHeading::after{<br />
content:"Réseaux Zéro Confiance";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.highlighted{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Face_Recognition&diff=11270Technology Trends/Face Recognition2019-07-22T13:48:19Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Face Recognition</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Reconnaissance de Visage]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[200px]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>June 25, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 22, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Face_Recognition_v0.1_EN.pdf|Face Recognition.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Face Recognition</b> is a biometric technology that is used to establish an individual identity and capable of uniquely identifying and verifying a person. It uses a computer application, which is also known as a facial recognition system, to extract a digital image from a picture, video frame or 3D scan and create a faceprint, a set of characteristic measurements of a face structure, that uniquely identifies one person’s particular face for identity and authentication purposes. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p class="expand inline mw-collapsible-content"> Generally, the authentication and identification is done by matching the corresponding facial features to the faceprints stored in a database record. </p><p class="inline">The facial recognition technology has a wide variety of applications in access control, surveillance, and criminal investigations. It is also used in combination with other biometric technology to improve and enhance security measures.</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Major technology companies like Apple, Google, Samsung, Facebook, and Amazon have begun to realize the impact that facial recognition can have on their existing security infrastructure. </p><p class="inline">Apple has been attempting to add movement capabilities to the system. The subject of a scanned face can now be talking or moving during the scan, which allows facial recognition to combine with other biometric security measures like voice recognition. Since moving subjects can be scanned, individuals can be identified in a crowd without intrusion by using facial recognition systems.</p><br />
<h2>Technology Brief</h2><br />
<p class="inline">Facial recognition systems can use either a 2D/3D image or video feed to create a digital image, establish the faceprint, and identify a face by comparing its digital image with the faceprints in a database. Every face has several “landmarks” and the system will flag these as “nodal points”. A human face can have up to 80 of these points. They represent areas of interest on the face that the system measures. </p><p class="expand inline mw-collapsible-content">Some examples of these measurements would be, distance between the eyes, width of the nose, depth of the eye socket, and more. These measurements will be stored in a database as a faceprint. </p><p class="inline">When the system scans a face, it will compare all these measurements to the records, faceprints, in the database. </p><p class="expand inline mw-collapsible-content">Facial recognition systems employ an algorithm, such as the Facial Recognition Vendor Test, that can predict whether there’s a match based on the “nodal points” on an individual’s face. </p><p class="inline">Usually, there is a 4-stage process involved in the operation of this technology <ref><i>[http://www.ex-sight.com/technology.html]</i></ref>: </p><br />
<p>Capture – a physical or behavioral sample is captured by the system during enrolment</p><br />
<p>Extraction – Unique data is extracted from the sample and a template is created</p><br />
<p>Comparison – the template is then compared with a new sample</p><br />
<p>Matching – the system then decides if the feature extracted from the new sample is a match or not</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Facial recognition offers another form of biometric identification and authentication. Several vendors have been leveraging face recognition as an access control and authentication tool either for their clients or for internal use. Its application is not limited to a form of security measure, but it can also be used for healthcare and retail.</p><p class="expand inline mw-collapsible-content"> Although 2D face recognition may not be as accurate as other forms of biometric technology like fingerprint readers, it does have its advantages. </p><p class="inline">The subject of a scanned face does not have to know when they were scanned, which demonstrates that the technology can be used in large crowds to identify threats quickly.</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Several vendors in today’s market are leveraging face recognition for their many applications. </p><p class="inline">For example, Amazon has been developing a system, which allows users to pay for their items using an actionable image (selfie). A user can use a selfie in which he is moving or speaking a particular phrase as a password to verify their identity when paying for an item. The logic behind having the customer speak a phrase or make a movement, is to try to eliminate the possibility of fooling the system with a scanned 2D image of the person. </p><p class="expand inline mw-collapsible-content">Amazon Rekognition is another product that provides two API sets - Amazon Rekognition Image for analyzing images and Amazon Rekognition Video for analyzing videos. Both APIs perform detection and recognition analysis of images and videos to provide insights for use in your applications.</p><br />
<p>After purchasing Face.com in 2012, Facebook began using facial recognition technology to connect users with their photos. When a user uploads a photo, the software will automatically suggest other people to tag. When you are tagged in a photo, more content is shown in a grouped format about other tagged individuals. </p><br />
<p class="inline">Face ID is a technology developed by Apple and introduced in iPhone X. It provides intuitive and secure authentication enabled by the state-of-the-art TrueDepth camera system with advanced technologies to accurately map the geometry of your face. With a simple glance, Face ID securely unlocks your iPhone or iPad Pro. </p><p class="expand inline mw-collapsible-content">You can use it to authorize purchases from the iTunes Store, App Store, and Apple Books, and make payments with Apple Pay. The iPhone XR, XS, and XS Max are all packing the second-generation of Face ID, which is an updated version of the biometric authentication system that is supposed to be faster than the version introduced with the iPhone X.</p><br />
<h2>Canadian Government Use</h2><br />
<p class="expand mw-collapsible-content">Unlike the private sector, Government use cases for facial recognition applications are primarily related to security, specifically for identity verification and fraud prevention. For example, the Canada Border Services Agency (CBSA) has recently launched the Primary Inspection Kiosk (PIK) program where passengers entering the country from airports must check-in using self-serve kiosks<ref>Braga, Matthew. (March 2nd, 2017). Facial Recognition Technology is coming to Canadian Airports this spring. Canadian Broadcasting Corporation. Retrieved 17-05-2019 from: <i>[https://www.cbc.ca/news/technology/cbsa-canada-airports-facial-recognition-kiosk-biometrics-1.4007344]</i></ref>. These kiosks use facial recognition in order to clear passengers. The overall shift to un-maned kiosks has bolstered security while reducing congestion at airports and has been in development since 2015. Portuguese company Vision-Box has installed 130 Kiosks in the Toronto’s Pearson International Airport. The Kiosks are designed to take biometric data in two phases - facial recognition and fingerprint biometrics. The kiosks will also be able to obtain iris data, a feature reserved for people travelling under the NEXUS program.</p><br />
<p class="inline">Facial recognition systems are also used in provincial casinos for identifying and locking out visitors with gambling addictions who have voluntarily entered themselves into self-exclusion lists<ref>Elash, Anita, and Luk, Vivian. (July 25th, 2011). Canadian Casinos, Banks, Police use Facial-Recognition Technology. The Globe and Mail. Toronto, Ontario. Retrieved 21-05-2019 from: <i>[https://www.theglobeandmail.com/news/national/time-to-lead/canadian-casinos-banks-police-use-facial-recognition-technology/article590998/ ]</i></ref>. It is worth noting that the system was developed jointly with the Ontario Privacy Commissioner to ensure a privacy-by-default design. </p><p class="expand inline mw-collapsible-content">In real time, the system scans customers entering the casino and compares their images with gamblers on the self-exclusion list. If there is a match, the system notifies security and if not, the system deletes the image automatically. Access to the database is restricted and information about an individual is only accessible if the person in the picture is physically present.</p><br />
<p class="inline">Passport Canada has been using facial recognition software for the past decade to compare new passport photos against its database to prevent passport fraud. One to one (1:1) comparisons are done to confirm a person’s identity, meaning that a recently taken image is compared to one already in the database that is associated with that person’s identity. One to many (1:N) comparisons are done to compare an image against the entire database of passport photos to make sure there are no duplicate applicants or individuals with multiple identities<ref>Mackrael, Kim, and Ha, Tu Thanh. (May 15th, 2014) Facial Recognition Program Allows RCMP to Identify Alleged Passport Fraud. The Globe and Mail. Toronto, Ontario. Retrieved 27-05-2019 from: <i>[https://www.theglobeandmail.com/news/national/facial-recognition-program-allows-rcmp-to-nab-alleged-passport-fraudster/article18703608/]</i></ref>. </p><p class="expand inline mw-collapsible-content">This initiative has been successfully used to catch individuals attempting to obtain multiple passports. This same concept is also used for driver’s licences being issued at the provincial level<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303/]</i></ref>. </p><br />
<p>Bill C-309, An Act to Amend the Criminal Code has made the concealment of identity (using masks or disguises) unlawful while participating in riots or unlawful assemblies<ref> Parliament of Canada. (June 19th, 2013). Bill C-309 An Act to Amend the Criminal Code (concealment of identity). Government of Canada. Retrieved 03-06-2019 from: <i>[https://www.parl.ca/LegisInfo/BillDetails.aspx?Bill=C309&Language=E&Mode=1&Parl=41&Ses=1 ]</i></ref>. Although the Privacy Act and PIPEDA (Personal Information Protection and Electronic Documents Act) state that consent must be obtained before private information is collected, Bill C-309 paves the way for law enforcement to scan large crowds using facial recognition software and uncover the identities of participants.</p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="inline">SSC could leverage this technology by offering facial recognition as a service. SSC could utilise this technology to replace the current government employee security ID. A smart camera will instantly capture the biometric data of the individuals for local analysis and then open the gate to access the building. This service could reduce ongoing security costs associated with having a security team on site, but there probably will not be any savings in the short term due to the cost of developing the applications and installing the related equipment. </p><p class="expand inline mw-collapsible-content">Facial recognition technology is a non-intrusive form of identity verification that cannot be lost by the individual. Within the SSC context, it would eliminate the need for employees to carry security passes. Additionally, this would help prevent unauthorized individuals from gaining access to secure facilities. </p><p class="inline">Two-factor authentication with a user’s face could also be used for accessing secure files with higher security classifications (such as secret documents).</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">There is a growing trend among smartphone manufacturers to create devices that can be unlocked with facial recognition technology. </p><p class="inline">A market research firm based in Hong Kong estimates that nearly 64% (or 1 billion) of all smartphones shipped worldwide will have facial recognition capabilities in 2020<ref>Naiya, Pavel. (February 7th, 2018) More than one billion smartphones to feature facial recognition in 2020. Counterpoint technology Market Research. Hong Kong, China. Retrieved 27-05-2019 from: <i>[https://www.counterpointresearch.com/one-billion-smartphones-feature-face-recognition-2020/]</i></ref>. SSC can take advantage of this research and only issue phones with facial recognition capabilities to employees. This biometric information can be paired with any other type of authentication method to create a two-step verification process for all smartphones. </p><p class="expand inline mw-collapsible-content">SSC would not need to acquire any additional software licences, as these phones would already have the capacity for facial recognition. Since verification is done locally, (reference images are stored on the device outside of a cloud environment) this minimizes the security risks associated with facial recognition technology.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Although facial recognition requires a lot of computing power to process images in real time, Edge Computing can mitigate this concern. Image pre-processing tasks can be completed by the device that took the picture, or much closer to the device than the data center. The device would capture the image, scan it for faces, and then extract information like a faceprint from the image. Once the faceprint has been created, it is sent to the main server for authentication and the original image is discarded. </p><p class="expand inline mw-collapsible-content">Since pre-processing of the faceprint has been done outside of the server, the server can focus on verifying the recent faceprint against an internal match.</p><br />
<h4>Challenges</h4><br />
<p class="expand mw-collapsible-content">The primary challenge with facial recognition technology is privacy. One way to mitigate the privacy concern is to leverage Edge Computing to store biometric data locally, which helps prevent data loss and inappropriate cross-linking of data across systems. The data is better stored as biometric features (a faceprint for example) rather than images of people’s faces, which should be destroyed once the faceprint is collected; faceprint data should be stored, encrypted, and be made available through secured measures. In this way, it will inhibit the misuse of the image for unauthorized reasons.</p><br />
<p>If this technology were to be used to authenticate identities in multiple areas, there may be large amounts of hardware required for computing power. If facial recognition technology, with millions of scanned faceprints were to be adopted by one or multiple departments, the computing capacity being used to process the requirements and match the image information with the faceprint records in the database would be extensive. From this perspective, if SSC were to support such a project, the demanded computing power to support this technology could be provided either in the form of a private or public cloud to meet the computing demands.</p><br />
<p>There are also certain factors that can limit the accuracy of facial recognition systems. If the photo was taken in profile or if the image quality is too low, there may not be enough information available for the system to extract and generate a match. Haircuts, skin color, makeup, glasses, and face coverings such as surgical masks can also lower recognition accuracy. Considering that these systems are based on Artificial Intelligence (AI), there is also the possibility of improperly “training” them.</p><br />
<p>An AI for a facial recognition system should be manually supervised to “reward” correct matches, but if the training set only consists of a very specific demographic of people it will have difficulty with detecting other types of faces. The lack of a diverse training set creates recognition biases in the programs, which makes them better at detecting and correctly identifying individuals with specific attributes over others.</p><br />
<p class="expand mw-collapsible">In a study conducted by Joy Buolamwini where three different facial recognition systems were tested for accuracy in determining genders, they had error rates between 21% and 35% for women with darker skin tones whereas the error rate for light-skinned males was less than 1%<ref>Lohr, Steve. (February 9th,2018). Facial Recognition is Accurate, if You’re a White Guy. New York Times. New York, USA. Retrieved 29-05-2019 from: <i>[https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html ]</i></ref>. This brings into question the reliability of these systems. To avoid discrimination against specific minority groups, they must be developed and tested to make sure they don’t have any recognition biases.</p><br />
<p class="inline">These systems also have fuzzy accuracy rates, meaning that matches are never 100% accurate when doing searches, for example, in a database of images. There is the real possibility of false positives, where matches are found but they aren’t for the right person, and of false negatives, where the real match exists within the database but the system is unable to create a match. </p><p class="expand inline mw-collapsible-content">This accuracy gap means that the systems should only be used by trained individuals who understand how the technology works and specific guidelines should be followed when matches are generated by the system.</p><br />
<p class="expand mw-collapsible">A good example in the field is how the Toronto police force is using the system: only six FBI-trained Toronto Police officers can use the system and it can only generate a list of candidates. They do not use the system as a sole basis for arrests but in tandem with other traditional evidence gathering methods<ref>Burt, Chris. (May 28th, 2019). Toronto police using facial recognition as Canadian government ponders rules. Biometrics Research Group Inc. Retrieved 29-05-2019 from: <i>[https://www.biometricupdate.com/201905/toronto-police-using-facial-recognition-as-canadian-government-ponders-rules ]</i></ref>. AI systems, if they are to help inform important decisions should never be solely trusted and should be used as tools to inform decisions, not guide them.</p><br />
<p class="inline">To deal with the issue of bad lighting or faces at unrecognizable angles, some systems are altering images so that they are more “readable”. </p><p class="expand inline mw-collapsible-content">Panasonic has developed facial recognition software that analyses movement, speed and lighting present in videos to automatically correct still images that would otherwise be blurry<ref>Panasonic. (February 20th, 2018) Panasonic to Launch Face Recognition Server Software Using Deep Learning Technology. Panasonic Corporation. Kadoma, Japan. Retrieved 15-05-2019 from: <i>[https://security.panasonic.com/news/archives/686 ]</i></ref>. </p><p class="inline">Since the software adjusts the image before being analysed, it creates a new concern that additional false positives will be created. If an image has been touched up and edited before it was “plugged-in” to a facial recognition system, this can alter the faceprint being analysed and the search results may be biased or incorrect.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Another limitation of these systems is that it can only recognize individuals whose images are already contained within its database. </p><p class="expand inline mw-collapsible-content">Systems must also be able to perform “liveness” testing, or in other words, they must be able to determine if the subject in question is actually there in person since faces are not secret, in the same sense that passwords are secret, and faces cannot be hidden. Facial recognition systems rely on the difficulty of impersonating a real person to keep the system secure. Given this fact, the system needs to be able to determine the “liveness” of the image it’s analysing and determine if the person it just photographed is real or if it’s a picture.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Facial recognition system technology has not yet been regulated in Canada and organizations that currently use it must operate within a specific legal framework. Under the Canadian Privacy Act, federal government institutions can only use personal information for the specific purpose for which it was collected and consent of the individual must be obtained before that information can be used for another purpose. Under PIPEDA, an organization must inform individuals and receive consent to any use of their personal information<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303]/</i></ref>. </p><p class="expand inline mw-collapsible-content">This is a potential legal barrier to any organization planning to conduct live analysis of public crowds since each individual would need to consent to the collection and use of their faces (private information). </p><p class="inline">These regulations ensure that databases containing personal information belonging to different GC departments cannot be shared between departments for purposes other than the specific use for which consent was obtained.</p><br />
<br />
<h4>Considerations</h4><br />
<p>Any wide-scale use case developed by SSC will need to be reported to the Office of the Privacy Commissioner (OPC) for assessment. They would also need to comply with the Privacy Act and PIPEDA. Use cases must also be justified with regards to possible privacy intrusions and the OPC has suggested a four-part test to determine this:<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303/]</i></ref> </p><br />
<ul><br />
<li>Is the measure demonstrably necessary to meet a specific need?</li><br />
<li>Is it likely to be effective in meeting that need?</li><br />
<li>Would the loss of privacy be proportionate to the benefit gained?</li><br />
<li>Is there a less privacy-invasive way of achieving the same end?</li><br />
</ul><br />
<p class="expand mw-collapsible-content">Additionally, use cases that are to be deployed on a wide scale within SSC, such as facial recognition systems to enter secured buildings, would rely on the consent of all participants. High quality images with neutral facial expressions would also need to be collected, or compiled from existing databases (i.e. building passes). These new centralized repositories of staff pictures could also be new targets of cybersecurity attacks. They would need to have the utmost protections afforded to them. Smaller scale use cases, such as unlocking a device with a face capture, would pose smaller risks since the information is stored locally.</p><br />
<p>If SSC were to adopt or develop facial recognition software, the software would need to be tested for recognition biases. Meaning, that it should have uniform recognition performance across all genders, skin types, and ages. If there are any biases present, the software would need to undergo more development and could garner more associated costs. Facial recognition applications purchased from a vendor would need to be closely scrutinized and evaluated since SSC would not know exactly how the application was trained and developed. Additionally, there are no industry standards for facial recognition products, meaning that there is no benchmarked performance level that needs to be achieved before a product goes to market. If SSC were to develop the software itself, it would have complete control and knowledge over the performance of the system.</p><br />
<p class="expand mw-collapsible-content">Lastly, it is important to consider the legal context of any facial recognition systems deployed within SSC. Presently, the technology is not regulated and performance standards have not been set for developers. The legal landscape could change by the time SSC is ready to adopt the technology. An ongoing privacy assessment of current service offerings may help provide a better understanding of the impact that facial recognition technology would have.</p><br />
<h2>References</h2><br />
<div style="display: none"><br />
<br />
<br />
<ref>Facial recognition system. (2019, July 16). Retrieved from <i>[https://en.wikipedia.org/wiki/Facial_recognition_system]</i></ref><br />
<ref>Gartner_Inc. (n.d.). Face Recognition Software: Anti-Terrorism Tool and More. Retrieved from <i>[https://www.gartner.com/doc/341020/face-recognition-software-antiterrorism-tool]</i></ref><br />
<ref>Solutions, T. (2018, September 25). Pros and Cons of Facial Recognition Technology For Your Business. Retrieved from <i>[https://www.upwork.com/hiring/for-clients/pros-cons-facial-recognition-technology-business/]</i></ref><br />
<ref>5 Applications of Facial Recognition Technology. (2017, July 09). Retrieved from <i>[https://disruptionhub.com/5-applications-facial-recognition-technology/]</i></ref><br />
<ref>Biometric Facial Recognition. (n.d.). Retrieved from <i>[https://findbiometrics.com/solutions/facial-recognition/]</i></ref><br />
<ref>How Facial Recognition Systems Work. (2001, September 04). Retrieved from <i>[https://electronics.howstuffworks.com/gadgets/high-tech-gadgets/facial-recognition2.htm]</i></ref><br />
<ref>Virdee-Chapman, B. (2016, May 26). 5 Companies Using Facial Recognition to Change The World. Retrieved from <i>[https://www.kairos.com/blog/5-companies-using-facial-recognition-to-change-the-world]</i></ref><br />
<ref>Sennaar, K. (2019, May 20). Facial Recognition Applications – Security, Retail, and Beyond. Retrieved from <i>[https://www.techemergence.com/facial-recognition-applications/]</i></ref><br />
<ref>Facial recognition technology is coming to Canadian airports this spring | CBC News. (2017, March 06). Retrieved from <i>[http://www.cbc.ca/news/technology/cbsa-canada-airports-facial-recognition-kiosk-biometrics-1.4007344]</i></ref><br />
<ref>Lee, J. (2016, June 09). Canadian government used facial recognition to detect passport fraudsters. Retrieved from <i>[https://www.biometricupdate.com/201606/canadian-government-used-facial-recognition-to-detect-passport-fraudsters]</i></ref><br />
<ref>Solutions, T. (2018, September 25). Pros and Cons of Facial Recognition Technology For Your Business. Retrieved from <i>[https://www.upwork.com/hiring/for-clients/pros-cons-facial-recognition-technology-business/]</i></ref><br />
<ref>Face ID Security. (2017, November). Retrieved from apple.com:<i>[https://www.apple.com/ca/business-docs/FaceID_Security_Guide.pdf]</i></ref><br />
</div><br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Face Recognition";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Face_Recognition&diff=11269Technology Trends/Face Recognition2019-07-22T13:31:28Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Face Recognition</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Reconnaissance de Visage]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[200px]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>June 25, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 22, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Face_Recognition_v0.1_EN.pdf|Face Recognition.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Face Recognition</b> is a biometric technology that is used to establish an individual identity and capable of uniquely identifying and verifying a person. It uses a computer application, which is also known as a facial recognition system, to extract a digital image from a picture, video frame or 3D scan and create a faceprint, a set of characteristic measurements of a face structure, that uniquely identifies one person’s particular face for identity and authentication purposes. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p class="inline">Face recognition is a biometric technology that is used to establish an individual identity and capable of uniquely identifying and verifying a person. It uses a computer application, which is also known as a facial recognition system, to extract a digital image from a picture, video frame or 3D scan and create a faceprint, a set of characteristic measurements of a face structure, that uniquely identifies one person’s particular face for identity and authentication purposes.</p><p class="expand inline mw-collapsible-content"> Generally, the authentication and identification is done by matching the corresponding facial features to the faceprints stored in a database record. </p><p class="inline">The facial recognition technology has a wide variety of applications in access control, surveillance, and criminal investigations. It is also used in combination with other biometric technology to improve and enhance security measures.</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Major technology companies like Apple, Google, Samsung, Facebook, and Amazon have begun to realize the impact that facial recognition can have on their existing security infrastructure. </p><p class="inline">Apple has been attempting to add movement capabilities to the system. The subject of a scanned face can now be talking or moving during the scan, which allows facial recognition to combine with other biometric security measures like voice recognition. Since moving subjects can be scanned, individuals can be identified in a crowd without intrusion by using facial recognition systems.</p><br />
<h2>Technology Brief</h2><br />
<p class="inline">Facial recognition systems can use either a 2D/3D image or video feed to create a digital image, establish the faceprint, and identify a face by comparing its digital image with the faceprints in a database. Every face has several “landmarks” and the system will flag these as “nodal points”. A human face can have up to 80 of these points. They represent areas of interest on the face that the system measures. </p><p class="expand inline mw-collapsible-content">Some examples of these measurements would be, distance between the eyes, width of the nose, depth of the eye socket, and more. These measurements will be stored in a database as a faceprint. </p><p class="inline">When the system scans a face, it will compare all these measurements to the records, faceprints, in the database. </p><p class="expand inline mw-collapsible-content">Facial recognition systems employ an algorithm, such as the Facial Recognition Vendor Test, that can predict whether there’s a match based on the “nodal points” on an individual’s face. </p><p class="inline">Usually, there is a 4-stage process involved in the operation of this technology <ref><i>[http://www.ex-sight.com/technology.html]</i></ref>: </p><br />
<p>Capture – a physical or behavioral sample is captured by the system during enrolment</p><br />
<p>Extraction – Unique data is extracted from the sample and a template is created</p><br />
<p>Comparison – the template is then compared with a new sample</p><br />
<p>Matching – the system then decides if the feature extracted from the new sample is a match or not</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Facial recognition offers another form of biometric identification and authentication. Several vendors have been leveraging face recognition as an access control and authentication tool either for their clients or for internal use. Its application is not limited to a form of security measure, but it can also be used for healthcare and retail.</p><p class="expand inline mw-collapsible-content"> Although 2D face recognition may not be as accurate as other forms of biometric technology like fingerprint readers, it does have its advantages. </p><p class="inline">The subject of a scanned face does not have to know when they were scanned, which demonstrates that the technology can be used in large crowds to identify threats quickly.</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Several vendors in today’s market are leveraging face recognition for their many applications. </p><p class="inline">For example, Amazon has been developing a system, which allows users to pay for their items using an actionable image (selfie). A user can use a selfie in which he is moving or speaking a particular phrase as a password to verify their identity when paying for an item. The logic behind having the customer speak a phrase or make a movement, is to try to eliminate the possibility of fooling the system with a scanned 2D image of the person. </p><p class="expand inline mw-collapsible-content">Amazon Rekognition is another product that provides two API sets - Amazon Rekognition Image for analyzing images and Amazon Rekognition Video for analyzing videos. Both APIs perform detection and recognition analysis of images and videos to provide insights for use in your applications.</p><br />
<p>After purchasing Face.com in 2012, Facebook began using facial recognition technology to connect users with their photos. When a user uploads a photo, the software will automatically suggest other people to tag. When you are tagged in a photo, more content is shown in a grouped format about other tagged individuals. </p><br />
<p class="inline">Face ID is a technology developed by Apple and introduced in iPhone X. It provides intuitive and secure authentication enabled by the state-of-the-art TrueDepth camera system with advanced technologies to accurately map the geometry of your face. With a simple glance, Face ID securely unlocks your iPhone or iPad Pro. </p><p class="expand inline mw-collapsible-content">You can use it to authorize purchases from the iTunes Store, App Store, and Apple Books, and make payments with Apple Pay. The iPhone XR, XS, and XS Max are all packing the second-generation of Face ID, which is an updated version of the biometric authentication system that is supposed to be faster than the version introduced with the iPhone X.</p><br />
<h2>Canadian Government Use</h2><br />
<p class="expand mw-collapsible-content">Unlike the private sector, Government use cases for facial recognition applications are primarily related to security, specifically for identity verification and fraud prevention. For example, the Canada Border Services Agency (CBSA) has recently launched the Primary Inspection Kiosk (PIK) program where passengers entering the country from airports must check-in using self-serve kiosks<ref>Braga, Matthew. (March 2nd, 2017). Facial Recognition Technology is coming to Canadian Airports this spring. Canadian Broadcasting Corporation. Retrieved 17-05-2019 from: <i>[https://www.cbc.ca/news/technology/cbsa-canada-airports-facial-recognition-kiosk-biometrics-1.4007344]</i></ref>. These kiosks use facial recognition in order to clear passengers. The overall shift to un-maned kiosks has bolstered security while reducing congestion at airports and has been in development since 2015. Portuguese company Vision-Box has installed 130 Kiosks in the Toronto’s Pearson International Airport. The Kiosks are designed to take biometric data in two phases - facial recognition and fingerprint biometrics. The kiosks will also be able to obtain iris data, a feature reserved for people travelling under the NEXUS program.</p><br />
<p class="inline">Facial recognition systems are also used in provincial casinos for identifying and locking out visitors with gambling addictions who have voluntarily entered themselves into self-exclusion lists<ref>Elash, Anita, and Luk, Vivian. (July 25th, 2011). Canadian Casinos, Banks, Police use Facial-Recognition Technology. The Globe and Mail. Toronto, Ontario. Retrieved 21-05-2019 from: <i>[https://www.theglobeandmail.com/news/national/time-to-lead/canadian-casinos-banks-police-use-facial-recognition-technology/article590998/ ]</i></ref>. It is worth noting that the system was developed jointly with the Ontario Privacy Commissioner to ensure a privacy-by-default design. </p><p class="expand inline mw-collapsible-content">In real time, the system scans customers entering the casino and compares their images with gamblers on the self-exclusion list. If there is a match, the system notifies security and if not, the system deletes the image automatically. Access to the database is restricted and information about an individual is only accessible if the person in the picture is physically present.</p><br />
<p class="inline">Passport Canada has been using facial recognition software for the past decade to compare new passport photos against its database to prevent passport fraud. One to one (1:1) comparisons are done to confirm a person’s identity, meaning that a recently taken image is compared to one already in the database that is associated with that person’s identity. One to many (1:N) comparisons are done to compare an image against the entire database of passport photos to make sure there are no duplicate applicants or individuals with multiple identities<ref>Mackrael, Kim, and Ha, Tu Thanh. (May 15th, 2014) Facial Recognition Program Allows RCMP to Identify Alleged Passport Fraud. The Globe and Mail. Toronto, Ontario. Retrieved 27-05-2019 from: <i>[https://www.theglobeandmail.com/news/national/facial-recognition-program-allows-rcmp-to-nab-alleged-passport-fraudster/article18703608/]</i></ref>. </p><p class="expand inline mw-collapsible-content">This initiative has been successfully used to catch individuals attempting to obtain multiple passports. This same concept is also used for driver’s licences being issued at the provincial level<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303/]</i></ref>. </p><br />
<p>Bill C-309, An Act to Amend the Criminal Code has made the concealment of identity (using masks or disguises) unlawful while participating in riots or unlawful assemblies<ref> Parliament of Canada. (June 19th, 2013). Bill C-309 An Act to Amend the Criminal Code (concealment of identity). Government of Canada. Retrieved 03-06-2019 from: <i>[https://www.parl.ca/LegisInfo/BillDetails.aspx?Bill=C309&Language=E&Mode=1&Parl=41&Ses=1 ]</i></ref>. Although the Privacy Act and PIPEDA (Personal Information Protection and Electronic Documents Act) state that consent must be obtained before private information is collected, Bill C-309 paves the way for law enforcement to scan large crowds using facial recognition software and uncover the identities of participants.</p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="inline">SSC could leverage this technology by offering facial recognition as a service. SSC could utilise this technology to replace the current government employee security ID. A smart camera will instantly capture the biometric data of the individuals for local analysis and then open the gate to access the building. This service could reduce ongoing security costs associated with having a security team on site, but there probably will not be any savings in the short term due to the cost of developing the applications and installing the related equipment. </p><p class="expand inline mw-collapsible-content">Facial recognition technology is a non-intrusive form of identity verification that cannot be lost by the individual. Within the SSC context, it would eliminate the need for employees to carry security passes. Additionally, this would help prevent unauthorized individuals from gaining access to secure facilities. </p><p class="inline">Two-factor authentication with a user’s face could also be used for accessing secure files with higher security classifications (such as secret documents).</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">There is a growing trend among smartphone manufacturers to create devices that can be unlocked with facial recognition technology. </p><p class="inline">A market research firm based in Hong Kong estimates that nearly 64% (or 1 billion) of all smartphones shipped worldwide will have facial recognition capabilities in 2020<ref>Naiya, Pavel. (February 7th, 2018) More than one billion smartphones to feature facial recognition in 2020. Counterpoint technology Market Research. Hong Kong, China. Retrieved 27-05-2019 from: <i>[https://www.counterpointresearch.com/one-billion-smartphones-feature-face-recognition-2020/]</i></ref>. SSC can take advantage of this research and only issue phones with facial recognition capabilities to employees. This biometric information can be paired with any other type of authentication method to create a two-step verification process for all smartphones. </p><p class="expand inline mw-collapsible-content">SSC would not need to acquire any additional software licences, as these phones would already have the capacity for facial recognition. Since verification is done locally, (reference images are stored on the device outside of a cloud environment) this minimizes the security risks associated with facial recognition technology.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Although facial recognition requires a lot of computing power to process images in real time, Edge Computing can mitigate this concern. Image pre-processing tasks can be completed by the device that took the picture, or much closer to the device than the data center. The device would capture the image, scan it for faces, and then extract information like a faceprint from the image. Once the faceprint has been created, it is sent to the main server for authentication and the original image is discarded. </p><p class="expand inline mw-collapsible-content">Since pre-processing of the faceprint has been done outside of the server, the server can focus on verifying the recent faceprint against an internal match.</p><br />
<h4>Challenges</h4><br />
<p class="expand mw-collapsible-content">The primary challenge with facial recognition technology is privacy. One way to mitigate the privacy concern is to leverage Edge Computing to store biometric data locally, which helps prevent data loss and inappropriate cross-linking of data across systems. The data is better stored as biometric features (a faceprint for example) rather than images of people’s faces, which should be destroyed once the faceprint is collected; faceprint data should be stored, encrypted, and be made available through secured measures. In this way, it will inhibit the misuse of the image for unauthorized reasons.</p><br />
<p>If this technology were to be used to authenticate identities in multiple areas, there may be large amounts of hardware required for computing power. If facial recognition technology, with millions of scanned faceprints were to be adopted by one or multiple departments, the computing capacity being used to process the requirements and match the image information with the faceprint records in the database would be extensive. From this perspective, if SSC were to support such a project, the demanded computing power to support this technology could be provided either in the form of a private or public cloud to meet the computing demands.</p><br />
<p>There are also certain factors that can limit the accuracy of facial recognition systems. If the photo was taken in profile or if the image quality is too low, there may not be enough information available for the system to extract and generate a match. Haircuts, skin color, makeup, glasses, and face coverings such as surgical masks can also lower recognition accuracy. Considering that these systems are based on Artificial Intelligence (AI), there is also the possibility of improperly “training” them.</p><br />
<p>An AI for a facial recognition system should be manually supervised to “reward” correct matches, but if the training set only consists of a very specific demographic of people it will have difficulty with detecting other types of faces. The lack of a diverse training set creates recognition biases in the programs, which makes them better at detecting and correctly identifying individuals with specific attributes over others.</p><br />
<p class="expand mw-collapsible">In a study conducted by Joy Buolamwini where three different facial recognition systems were tested for accuracy in determining genders, they had error rates between 21% and 35% for women with darker skin tones whereas the error rate for light-skinned males was less than 1%<ref>Lohr, Steve. (February 9th,2018). Facial Recognition is Accurate, if You’re a White Guy. New York Times. New York, USA. Retrieved 29-05-2019 from: <i>[https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html ]</i></ref>. This brings into question the reliability of these systems. To avoid discrimination against specific minority groups, they must be developed and tested to make sure they don’t have any recognition biases.</p><br />
<p class="inline">These systems also have fuzzy accuracy rates, meaning that matches are never 100% accurate when doing searches, for example, in a database of images. There is the real possibility of false positives, where matches are found but they aren’t for the right person, and of false negatives, where the real match exists within the database but the system is unable to create a match. </p><p class="expand inline mw-collapsible-content">This accuracy gap means that the systems should only be used by trained individuals who understand how the technology works and specific guidelines should be followed when matches are generated by the system.</p><br />
<p class="expand mw-collapsible">A good example in the field is how the Toronto police force is using the system: only six FBI-trained Toronto Police officers can use the system and it can only generate a list of candidates. They do not use the system as a sole basis for arrests but in tandem with other traditional evidence gathering methods<ref>Burt, Chris. (May 28th, 2019). Toronto police using facial recognition as Canadian government ponders rules. Biometrics Research Group Inc. Retrieved 29-05-2019 from: <i>[https://www.biometricupdate.com/201905/toronto-police-using-facial-recognition-as-canadian-government-ponders-rules ]</i></ref>. AI systems, if they are to help inform important decisions should never be solely trusted and should be used as tools to inform decisions, not guide them.</p><br />
<p class="inline">To deal with the issue of bad lighting or faces at unrecognizable angles, some systems are altering images so that they are more “readable”. </p><p class="expand inline mw-collapsible-content">Panasonic has developed facial recognition software that analyses movement, speed and lighting present in videos to automatically correct still images that would otherwise be blurry<ref>Panasonic. (February 20th, 2018) Panasonic to Launch Face Recognition Server Software Using Deep Learning Technology. Panasonic Corporation. Kadoma, Japan. Retrieved 15-05-2019 from: <i>[https://security.panasonic.com/news/archives/686 ]</i></ref>. </p><p class="inline">Since the software adjusts the image before being analysed, it creates a new concern that additional false positives will be created. If an image has been touched up and edited before it was “plugged-in” to a facial recognition system, this can alter the faceprint being analysed and the search results may be biased or incorrect.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Another limitation of these systems is that it can only recognize individuals whose images are already contained within its database. </p><p class="expand inline mw-collapsible-content">Systems must also be able to perform “liveness” testing, or in other words, they must be able to determine if the subject in question is actually there in person since faces are not secret, in the same sense that passwords are secret, and faces cannot be hidden. Facial recognition systems rely on the difficulty of impersonating a real person to keep the system secure. Given this fact, the system needs to be able to determine the “liveness” of the image it’s analysing and determine if the person it just photographed is real or if it’s a picture.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Facial recognition system technology has not yet been regulated in Canada and organizations that currently use it must operate within a specific legal framework. Under the Canadian Privacy Act, federal government institutions can only use personal information for the specific purpose for which it was collected and consent of the individual must be obtained before that information can be used for another purpose. Under PIPEDA, an organization must inform individuals and receive consent to any use of their personal information<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303]/</i></ref>. </p><p class="expand inline mw-collapsible-content">This is a potential legal barrier to any organization planning to conduct live analysis of public crowds since each individual would need to consent to the collection and use of their faces (private information). </p><p class="inline">These regulations ensure that databases containing personal information belonging to different GC departments cannot be shared between departments for purposes other than the specific use for which consent was obtained.</p><br />
<br />
<h4>Considerations</h4><br />
<p>Any wide-scale use case developed by SSC will need to be reported to the Office of the Privacy Commissioner (OPC) for assessment. They would also need to comply with the Privacy Act and PIPEDA. Use cases must also be justified with regards to possible privacy intrusions and the OPC has suggested a four-part test to determine this:<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303/]</i></ref> </p><br />
<ul><br />
<li>Is the measure demonstrably necessary to meet a specific need?</li><br />
<li>Is it likely to be effective in meeting that need?</li><br />
<li>Would the loss of privacy be proportionate to the benefit gained?</li><br />
<li>Is there a less privacy-invasive way of achieving the same end?</li><br />
</ul><br />
<p class="expand mw-collapsible-content">Additionally, use cases that are to be deployed on a wide scale within SSC, such as facial recognition systems to enter secured buildings, would rely on the consent of all participants. High quality images with neutral facial expressions would also need to be collected, or compiled from existing databases (i.e. building passes). These new centralized repositories of staff pictures could also be new targets of cybersecurity attacks. They would need to have the utmost protections afforded to them. Smaller scale use cases, such as unlocking a device with a face capture, would pose smaller risks since the information is stored locally.</p><br />
<p>If SSC were to adopt or develop facial recognition software, the software would need to be tested for recognition biases. Meaning, that it should have uniform recognition performance across all genders, skin types, and ages. If there are any biases present, the software would need to undergo more development and could garner more associated costs. Facial recognition applications purchased from a vendor would need to be closely scrutinized and evaluated since SSC would not know exactly how the application was trained and developed. Additionally, there are no industry standards for facial recognition products, meaning that there is no benchmarked performance level that needs to be achieved before a product goes to market. If SSC were to develop the software itself, it would have complete control and knowledge over the performance of the system.</p><br />
<p class="expand mw-collapsible-content">Lastly, it is important to consider the legal context of any facial recognition systems deployed within SSC. Presently, the technology is not regulated and performance standards have not been set for developers. The legal landscape could change by the time SSC is ready to adopt the technology. An ongoing privacy assessment of current service offerings may help provide a better understanding of the impact that facial recognition technology would have.</p><br />
<h2>References</h2><br />
<div style="display: none"><br />
<br />
<br />
<ref>Facial recognition system. (2019, July 16). Retrieved from <i>[https://en.wikipedia.org/wiki/Facial_recognition_system]</i></ref><br />
<ref>Gartner_Inc. (n.d.). Face Recognition Software: Anti-Terrorism Tool and More. Retrieved from <i>[https://www.gartner.com/doc/341020/face-recognition-software-antiterrorism-tool]</i></ref><br />
<ref>Solutions, T. (2018, September 25). Pros and Cons of Facial Recognition Technology For Your Business. Retrieved from <i>[https://www.upwork.com/hiring/for-clients/pros-cons-facial-recognition-technology-business/]</i></ref><br />
<ref>5 Applications of Facial Recognition Technology. (2017, July 09). Retrieved from <i>[https://disruptionhub.com/5-applications-facial-recognition-technology/]</i></ref><br />
<ref>Biometric Facial Recognition. (n.d.). Retrieved from <i>[https://findbiometrics.com/solutions/facial-recognition/]</i></ref><br />
<ref>How Facial Recognition Systems Work. (2001, September 04). Retrieved from <i>[https://electronics.howstuffworks.com/gadgets/high-tech-gadgets/facial-recognition2.htm]</i></ref><br />
<ref>Virdee-Chapman, B. (2016, May 26). 5 Companies Using Facial Recognition to Change The World. Retrieved from <i>[https://www.kairos.com/blog/5-companies-using-facial-recognition-to-change-the-world]</i></ref><br />
<ref>Sennaar, K. (2019, May 20). Facial Recognition Applications – Security, Retail, and Beyond. Retrieved from <i>[https://www.techemergence.com/facial-recognition-applications/]</i></ref><br />
<ref>Facial recognition technology is coming to Canadian airports this spring | CBC News. (2017, March 06). Retrieved from <i>[http://www.cbc.ca/news/technology/cbsa-canada-airports-facial-recognition-kiosk-biometrics-1.4007344]</i></ref><br />
<ref>Lee, J. (2016, June 09). Canadian government used facial recognition to detect passport fraudsters. Retrieved from <i>[https://www.biometricupdate.com/201606/canadian-government-used-facial-recognition-to-detect-passport-fraudsters]</i></ref><br />
<ref>Solutions, T. (2018, September 25). Pros and Cons of Facial Recognition Technology For Your Business. Retrieved from <i>[https://www.upwork.com/hiring/for-clients/pros-cons-facial-recognition-technology-business/]</i></ref><br />
<ref>Face ID Security. (2017, November). Retrieved from apple.com:<i>[https://www.apple.com/ca/business-docs/FaceID_Security_Guide.pdf]</i></ref><br />
</div><br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Face Recognition";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=File:EN_-_Face_Recognition_v0.1_EN.pdf&diff=11268File:EN - Face Recognition v0.1 EN.pdf2019-07-22T13:29:42Z<p>Kpere060: </p>
<hr />
<div></div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Face_Recognition&diff=11267Technology Trends/Face Recognition2019-07-22T13:29:03Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Face Recognition</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Reconnaissance de Visage]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[200px]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>June 25, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 22, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Face_Recognition_v0.1_EN.pdf|Face Recognition.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Face Recognition</b> is a biometric technology that is used to establish an individual identity and capable of uniquely identifying and verifying a person. It uses a computer application, which is also known as a facial recognition system, to extract a digital image from a picture, video frame or 3D scan and create a faceprint, a set of characteristic measurements of a face structure, that uniquely identifies one person’s particular face for identity and authentication purposes. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p class="inline">Face recognition is a biometric technology that is used to establish an individual identity and capable of uniquely identifying and verifying a person. It uses a computer application, which is also known as a facial recognition system, to extract a digital image from a picture, video frame or 3D scan and create a faceprint, a set of characteristic measurements of a face structure, that uniquely identifies one person’s particular face for identity and authentication purposes.</p><p class="expand inline mw-collapsible-content"> Generally, the authentication and identification is done by matching the corresponding facial features to the faceprints stored in a database record. </p><p class="inline">The facial recognition technology has a wide variety of applications in access control, surveillance, and criminal investigations. It is also used in combination with other biometric technology to improve and enhance security measures.</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Major technology companies like Apple, Google, Samsung, Facebook, and Amazon have begun to realize the impact that facial recognition can have on their existing security infrastructure. </p><p class="inline">Apple has been attempting to add movement capabilities to the system. The subject of a scanned face can now be talking or moving during the scan, which allows facial recognition to combine with other biometric security measures like voice recognition. Since moving subjects can be scanned, individuals can be identified in a crowd without intrusion by using facial recognition systems.</p><br />
<h2>Technology Brief</h2><br />
<p class="inline">Facial recognition systems can use either a 2D/3D image or video feed to create a digital image, establish the faceprint, and identify a face by comparing its digital image with the faceprints in a database. Every face has several “landmarks” and the system will flag these as “nodal points”. A human face can have up to 80 of these points. They represent areas of interest on the face that the system measures. </p><p class="expand inline mw-collapsible-content">Some examples of these measurements would be, distance between the eyes, width of the nose, depth of the eye socket, and more. These measurements will be stored in a database as a faceprint. </p><p class="inline">When the system scans a face, it will compare all these measurements to the records, faceprints, in the database. </p><p class="expand inline mw-collapsible-content">Facial recognition systems employ an algorithm, such as the Facial Recognition Vendor Test, that can predict whether there’s a match based on the “nodal points” on an individual’s face. </p><p class="inline">Usually, there is a 4-stage process involved in the operation of this technology <ref><i>[http://www.ex-sight.com/technology.html]</i></ref>: </p><br />
<p>Capture – a physical or behavioral sample is captured by the system during enrolment</p><br />
<p>Extraction – Unique data is extracted from the sample and a template is created</p><br />
<p>Comparison – the template is then compared with a new sample</p><br />
<p>Matching – the system then decides if the feature extracted from the new sample is a match or not</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Facial recognition offers another form of biometric identification and authentication. Several vendors have been leveraging face recognition as an access control and authentication tool either for their clients or for internal use. Its application is not limited to a form of security measure, but it can also be used for healthcare and retail.</p><p class="expand inline mw-collapsible-content"> Although 2D face recognition may not be as accurate as other forms of biometric technology like fingerprint readers, it does have its advantages. </p><p class="inline">The subject of a scanned face does not have to know when they were scanned, which demonstrates that the technology can be used in large crowds to identify threats quickly.</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Several vendors in today’s market are leveraging face recognition for their many applications. </p><p class="inline">For example, Amazon has been developing a system, which allows users to pay for their items using an actionable image (selfie). A user can use a selfie in which he is moving or speaking a particular phrase as a password to verify their identity when paying for an item. The logic behind having the customer speak a phrase or make a movement, is to try to eliminate the possibility of fooling the system with a scanned 2D image of the person. </p><p class="expand inline mw-collapsible-content">Amazon Rekognition is another product that provides two API sets - Amazon Rekognition Image for analyzing images and Amazon Rekognition Video for analyzing videos. Both APIs perform detection and recognition analysis of images and videos to provide insights for use in your applications.</p><br />
<p>After purchasing Face.com in 2012, Facebook began using facial recognition technology to connect users with their photos. When a user uploads a photo, the software will automatically suggest other people to tag. When you are tagged in a photo, more content is shown in a grouped format about other tagged individuals. </p><br />
<p class="inline">Face ID is a technology developed by Apple and introduced in iPhone X. It provides intuitive and secure authentication enabled by the state-of-the-art TrueDepth camera system with advanced technologies to accurately map the geometry of your face. With a simple glance, Face ID securely unlocks your iPhone or iPad Pro. </p><p class="expand inline mw-collapsible-content">You can use it to authorize purchases from the iTunes Store, App Store, and Apple Books, and make payments with Apple Pay. The iPhone XR, XS, and XS Max are all packing the second-generation of Face ID, which is an updated version of the biometric authentication system that is supposed to be faster than the version introduced with the iPhone X.</p><br />
<h2>Canadian Government Use</h2><br />
<p class="expand mw-collapsible-content">Unlike the private sector, Government use cases for facial recognition applications are primarily related to security, specifically for identity verification and fraud prevention. For example, the Canada Border Services Agency (CBSA) has recently launched the Primary Inspection Kiosk (PIK) program where passengers entering the country from airports must check-in using self-serve kiosks<ref>Braga, Matthew. (March 2nd, 2017). Facial Recognition Technology is coming to Canadian Airports this spring. Canadian Broadcasting Corporation. Retrieved 17-05-2019 from: <i>[https://www.cbc.ca/news/technology/cbsa-canada-airports-facial-recognition-kiosk-biometrics-1.4007344]</i></ref>. These kiosks use facial recognition in order to clear passengers. The overall shift to un-maned kiosks has bolstered security while reducing congestion at airports and has been in development since 2015. Portuguese company Vision-Box has installed 130 Kiosks in the Toronto’s Pearson International Airport. The Kiosks are designed to take biometric data in two phases - facial recognition and fingerprint biometrics. The kiosks will also be able to obtain iris data, a feature reserved for people travelling under the NEXUS program.</p><br />
<p class="inline">Facial recognition systems are also used in provincial casinos for identifying and locking out visitors with gambling addictions who have voluntarily entered themselves into self-exclusion lists<ref>Elash, Anita, and Luk, Vivian. (July 25th, 2011). Canadian Casinos, Banks, Police use Facial-Recognition Technology. The Globe and Mail. Toronto, Ontario. Retrieved 21-05-2019 from: <i>[https://www.theglobeandmail.com/news/national/time-to-lead/canadian-casinos-banks-police-use-facial-recognition-technology/article590998/ ]</i></ref>. It is worth noting that the system was developed jointly with the Ontario Privacy Commissioner to ensure a privacy-by-default design. </p><p class="expand inline mw-collapsible-content">In real time, the system scans customers entering the casino and compares their images with gamblers on the self-exclusion list. If there is a match, the system notifies security and if not, the system deletes the image automatically. Access to the database is restricted and information about an individual is only accessible if the person in the picture is physically present.</p><br />
<p class="inline">Passport Canada has been using facial recognition software for the past decade to compare new passport photos against its database to prevent passport fraud. One to one (1:1) comparisons are done to confirm a person’s identity, meaning that a recently taken image is compared to one already in the database that is associated with that person’s identity. One to many (1:N) comparisons are done to compare an image against the entire database of passport photos to make sure there are no duplicate applicants or individuals with multiple identities<ref>Mackrael, Kim, and Ha, Tu Thanh. (May 15th, 2014) Facial Recognition Program Allows RCMP to Identify Alleged Passport Fraud. The Globe and Mail. Toronto, Ontario. Retrieved 27-05-2019 from: <i>[https://www.theglobeandmail.com/news/national/facial-recognition-program-allows-rcmp-to-nab-alleged-passport-fraudster/article18703608/]</i></ref>. </p><p class="expand inline mw-collapsible-content">This initiative has been successfully used to catch individuals attempting to obtain multiple passports. This same concept is also used for driver’s licences being issued at the provincial level<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303/]</i></ref>. </p><br />
<p>Bill C-309, An Act to Amend the Criminal Code has made the concealment of identity (using masks or disguises) unlawful while participating in riots or unlawful assemblies<ref> Parliament of Canada. (June 19th, 2013). Bill C-309 An Act to Amend the Criminal Code (concealment of identity). Government of Canada. Retrieved 03-06-2019 from: <i>[https://www.parl.ca/LegisInfo/BillDetails.aspx?Bill=C309&Language=E&Mode=1&Parl=41&Ses=1 ]</i></ref>. Although the Privacy Act and PIPEDA (Personal Information Protection and Electronic Documents Act) state that consent must be obtained before private information is collected, Bill C-309 paves the way for law enforcement to scan large crowds using facial recognition software and uncover the identities of participants.</p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="inline">SSC could leverage this technology by offering facial recognition as a service. SSC could utilise this technology to replace the current government employee security ID. A smart camera will instantly capture the biometric data of the individuals for local analysis and then open the gate to access the building. This service could reduce ongoing security costs associated with having a security team on site, but there probably will not be any savings in the short term due to the cost of developing the applications and installing the related equipment. </p><p class="expand inline mw-collapsible-content">Facial recognition technology is a non-intrusive form of identity verification that cannot be lost by the individual. Within the SSC context, it would eliminate the need for employees to carry security passes. Additionally, this would help prevent unauthorized individuals from gaining access to secure facilities. </p><p class="inline">Two-factor authentication with a user’s face could also be used for accessing secure files with higher security classifications (such as secret documents).</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">There is a growing trend among smartphone manufacturers to create devices that can be unlocked with facial recognition technology. </p><p class="inline">A market research firm based in Hong Kong estimates that nearly 64% (or 1 billion) of all smartphones shipped worldwide will have facial recognition capabilities in 2020<ref>Naiya, Pavel. (February 7th, 2018) More than one billion smartphones to feature facial recognition in 2020. Counterpoint technology Market Research. Hong Kong, China. Retrieved 27-05-2019 from: <i>[https://www.counterpointresearch.com/one-billion-smartphones-feature-face-recognition-2020/]</i></ref>. SSC can take advantage of this research and only issue phones with facial recognition capabilities to employees. This biometric information can be paired with any other type of authentication method to create a two-step verification process for all smartphones. </p><p class="expand inline mw-collapsible-content">SSC would not need to acquire any additional software licences, as these phones would already have the capacity for facial recognition. Since verification is done locally, (reference images are stored on the device outside of a cloud environment) this minimizes the security risks associated with facial recognition technology.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Although facial recognition requires a lot of computing power to process images in real time, Edge Computing can mitigate this concern. Image pre-processing tasks can be completed by the device that took the picture, or much closer to the device than the data center. The device would capture the image, scan it for faces, and then extract information like a faceprint from the image. Once the faceprint has been created, it is sent to the main server for authentication and the original image is discarded. </p><p class="expand inline mw-collapsible-content">Since pre-processing of the faceprint has been done outside of the server, the server can focus on verifying the recent faceprint against an internal match.</p><br />
<h4>Challenges</h4><br />
<p class="expand mw-collapsible-content">The primary challenge with facial recognition technology is privacy. One way to mitigate the privacy concern is to leverage Edge Computing to store biometric data locally, which helps prevent data loss and inappropriate cross-linking of data across systems. The data is better stored as biometric features (a faceprint for example) rather than images of people’s faces, which should be destroyed once the faceprint is collected; faceprint data should be stored, encrypted, and be made available through secured measures. In this way, it will inhibit the misuse of the image for unauthorized reasons.</p><br />
<p>If this technology were to be used to authenticate identities in multiple areas, there may be large amounts of hardware required for computing power. If facial recognition technology, with millions of scanned faceprints were to be adopted by one or multiple departments, the computing capacity being used to process the requirements and match the image information with the faceprint records in the database would be extensive. From this perspective, if SSC were to support such a project, the demanded computing power to support this technology could be provided either in the form of a private or public cloud to meet the computing demands.</p><br />
<p>There are also certain factors that can limit the accuracy of facial recognition systems. If the photo was taken in profile or if the image quality is too low, there may not be enough information available for the system to extract and generate a match. Haircuts, skin color, makeup, glasses, and face coverings such as surgical masks can also lower recognition accuracy. Considering that these systems are based on Artificial Intelligence (AI), there is also the possibility of improperly “training” them.</p><br />
<p>An AI for a facial recognition system should be manually supervised to “reward” correct matches, but if the training set only consists of a very specific demographic of people it will have difficulty with detecting other types of faces. The lack of a diverse training set creates recognition biases in the programs, which makes them better at detecting and correctly identifying individuals with specific attributes over others.</p><br />
<p class="expand mw-collapsible">In a study conducted by Joy Buolamwini where three different facial recognition systems were tested for accuracy in determining genders, they had error rates between 21% and 35% for women with darker skin tones whereas the error rate for light-skinned males was less than 1%<ref>Lohr, Steve. (February 9th,2018). Facial Recognition is Accurate, if You’re a White Guy. New York Times. New York, USA. Retrieved 29-05-2019 from: <i>[https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html ]</i></ref>. This brings into question the reliability of these systems. To avoid discrimination against specific minority groups, they must be developed and tested to make sure they don’t have any recognition biases.</p><br />
<p class="inline">These systems also have fuzzy accuracy rates, meaning that matches are never 100% accurate when doing searches, for example, in a database of images. There is the real possibility of false positives, where matches are found but they aren’t for the right person, and of false negatives, where the real match exists within the database but the system is unable to create a match. </p><p class="expand inline mw-collapsible-content">This accuracy gap means that the systems should only be used by trained individuals who understand how the technology works and specific guidelines should be followed when matches are generated by the system.</p><br />
<p class="expand mw-collapsible">A good example in the field is how the Toronto police force is using the system: only six FBI-trained Toronto Police officers can use the system and it can only generate a list of candidates. They do not use the system as a sole basis for arrests but in tandem with other traditional evidence gathering methods<ref>Burt, Chris. (May 28th, 2019). Toronto police using facial recognition as Canadian government ponders rules. Biometrics Research Group Inc. Retrieved 29-05-2019 from: <i>[https://www.biometricupdate.com/201905/toronto-police-using-facial-recognition-as-canadian-government-ponders-rules ]</i></ref>. AI systems, if they are to help inform important decisions should never be solely trusted and should be used as tools to inform decisions, not guide them.</p><br />
<p class="inline">To deal with the issue of bad lighting or faces at unrecognizable angles, some systems are altering images so that they are more “readable”. </p><p class="expand inline mw-collapsible-content">Panasonic has developed facial recognition software that analyses movement, speed and lighting present in videos to automatically correct still images that would otherwise be blurry<ref>Panasonic. (February 20th, 2018) Panasonic to Launch Face Recognition Server Software Using Deep Learning Technology. Panasonic Corporation. Kadoma, Japan. Retrieved 15-05-2019 from: <i>[https://security.panasonic.com/news/archives/686 ]</i></ref>. </p><p class="inline">Since the software adjusts the image before being analysed, it creates a new concern that additional false positives will be created. If an image has been touched up and edited before it was “plugged-in” to a facial recognition system, this can alter the faceprint being analysed and the search results may be biased or incorrect.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Another limitation of these systems is that it can only recognize individuals whose images are already contained within its database. </p><p class="expand inline mw-collapsible-content">Systems must also be able to perform “liveness” testing, or in other words, they must be able to determine if the subject in question is actually there in person since faces are not secret, in the same sense that passwords are secret, and faces cannot be hidden. Facial recognition systems rely on the difficulty of impersonating a real person to keep the system secure. Given this fact, the system needs to be able to determine the “liveness” of the image it’s analysing and determine if the person it just photographed is real or if it’s a picture.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Facial recognition system technology has not yet been regulated in Canada and organizations that currently use it must operate within a specific legal framework. Under the Canadian Privacy Act, federal government institutions can only use personal information for the specific purpose for which it was collected and consent of the individual must be obtained before that information can be used for another purpose. Under PIPEDA, an organization must inform individuals and receive consent to any use of their personal information<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303]/</i></ref>. </p><p class="expand inline mw-collapsible-content">This is a potential legal barrier to any organization planning to conduct live analysis of public crowds since each individual would need to consent to the collection and use of their faces (private information). </p><p class="inline">These regulations ensure that databases containing personal information belonging to different GC departments cannot be shared between departments for purposes other than the specific use for which consent was obtained.</p><br />
<br />
<h4>Considerations</h4><br />
<p>Any wide-scale use case developed by SSC will need to be reported to the Office of the Privacy Commissioner (OPC) for assessment. They would also need to comply with the Privacy Act and PIPEDA. Use cases must also be justified with regards to possible privacy intrusions and the OPC has suggested a four-part test to determine this:<ref>Office of the Privacy Commissioner of Canada. (March 2013). Automated Facial Recognition in the Public and Private Sectors. Government of Canada. Retrieved 23-05-2019 from: <i>[https://www.priv.gc.ca/en/opc-actions-and-decisions/research/explore-privacy-research/2013/fr_201303/]</i></ref> </p><br />
<ul><br />
<li>Is the measure demonstrably necessary to meet a specific need?</li><br />
<li>Is it likely to be effective in meeting that need?</li><br />
<li>Would the loss of privacy be proportionate to the benefit gained?</li><br />
<li>Is there a less privacy-invasive way of achieving the same end?</li><br />
</ul><br />
<p class="expand mw-collapsible-content">Additionally, use cases that are to be deployed on a wide scale within SSC, such as facial recognition systems to enter secured buildings, would rely on the consent of all participants. High quality images with neutral facial expressions would also need to be collected, or compiled from existing databases (i.e. building passes). These new centralized repositories of staff pictures could also be new targets of cybersecurity attacks. They would need to have the utmost protections afforded to them. Smaller scale use cases, such as unlocking a device with a face capture, would pose smaller risks since the information is stored locally.</p><br />
<p>If SSC were to adopt or develop facial recognition software, the software would need to be tested for recognition biases. Meaning, that it should have uniform recognition performance across all genders, skin types, and ages. If there are any biases present, the software would need to undergo more development and could garner more associated costs. Facial recognition applications purchased from a vendor would need to be closely scrutinized and evaluated since SSC would not know exactly how the application was trained and developed. Additionally, there are no industry standards for facial recognition products, meaning that there is no benchmarked performance level that needs to be achieved before a product goes to market. If SSC were to develop the software itself, it would have complete control and knowledge over the performance of the system.</p><br />
<p class="expand mw-collapsible-content">Lastly, it is important to consider the legal context of any facial recognition systems deployed within SSC. Presently, the technology is not regulated and performance standards have not been set for developers. The legal landscape could change by the time SSC is ready to adopt the technology. An ongoing privacy assessment of current service offerings may help provide a better understanding of the impact that facial recognition technology would have.</p><br />
<h2>References</h2><br />
<div style="display: none"><br />
<br />
</div><br />
<ref>Facial recognition system. (2019, July 16). Retrieved from <i>[https://en.wikipedia.org/wiki/Facial_recognition_system]</i></ref><br />
<ref>Gartner_Inc. (n.d.). Face Recognition Software: Anti-Terrorism Tool and More. Retrieved from <i>[https://www.gartner.com/doc/341020/face-recognition-software-antiterrorism-tool]</i></ref><br />
<ref>Solutions, T. (2018, September 25). Pros and Cons of Facial Recognition Technology For Your Business. Retrieved from <i>[https://www.upwork.com/hiring/for-clients/pros-cons-facial-recognition-technology-business/]</i></ref><br />
<ref>5 Applications of Facial Recognition Technology. (2017, July 09). Retrieved from <i>[https://disruptionhub.com/5-applications-facial-recognition-technology/]</i></ref><br />
<ref>Biometric Facial Recognition. (n.d.). Retrieved from <i>[https://findbiometrics.com/solutions/facial-recognition/]</i></ref><br />
<ref>How Facial Recognition Systems Work. (2001, September 04). Retrieved from <i>[https://electronics.howstuffworks.com/gadgets/high-tech-gadgets/facial-recognition2.htm]</i></ref><br />
<ref>Virdee-Chapman, B. (2016, May 26). 5 Companies Using Facial Recognition to Change The World. Retrieved from <i>[https://www.kairos.com/blog/5-companies-using-facial-recognition-to-change-the-world]</i></ref><br />
<ref>Sennaar, K. (2019, May 20). Facial Recognition Applications – Security, Retail, and Beyond. Retrieved from <i>[https://www.techemergence.com/facial-recognition-applications/]</i></ref><br />
<ref>Facial recognition technology is coming to Canadian airports this spring | CBC News. (2017, March 06). Retrieved from <i>[http://www.cbc.ca/news/technology/cbsa-canada-airports-facial-recognition-kiosk-biometrics-1.4007344]</i></ref><br />
<ref>Lee, J. (2016, June 09). Canadian government used facial recognition to detect passport fraudsters. Retrieved from <i>[https://www.biometricupdate.com/201606/canadian-government-used-facial-recognition-to-detect-passport-fraudsters]</i></ref><br />
<ref>Solutions, T. (2018, September 25). Pros and Cons of Facial Recognition Technology For Your Business. Retrieved from <i>[https://www.upwork.com/hiring/for-clients/pros-cons-facial-recognition-technology-business/]</i></ref><br />
<ref>Face ID Security. (2017, November). Retrieved from apple.com:<i>[https://www.apple.com/ca/business-docs/FaceID_Security_Guide.pdf]</i></ref><br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Face Recognition";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Face_Recognition&diff=11198Technology Trends/Face Recognition2019-07-19T15:39:04Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Face Recognition</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Reconnaissance de Visage]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[200px]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>July 12, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Datalakes_v0.1_EN_Published.pdf|Datalakes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Face Recognition</b> is a biometric technology that is used to establish an individual identity and capable of uniquely identifying and verifying a person. It uses a computer application, which is also known as a facial recognition system, to extract a digital image from a picture, video frame or 3D scan and create a faceprint, a set of characteristic measurements of a face structure, that uniquely identifies one person’s particular face for identity and authentication purposes. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p class="inline">Face recognition is a biometric technology that is used to establish an individual identity and capable of uniquely identifying and verifying a person. It uses a computer application, which is also known as a facial recognition system, to extract a digital image from a picture, video frame or 3D scan and create a faceprint, a set of characteristic measurements of a face structure, that uniquely identifies one person’s particular face for identity and authentication purposes.</p><p class="expand inline mw-collapsible-content"> Generally, the authentication and identification is done by matching the corresponding facial features to the faceprints stored in a database record. </p><p class="inline">The facial recognition technology has a wide variety of applications in access control, surveillance, and criminal investigations. It is also used in combination with other biometric technology to improve and enhance security measures.</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Major technology companies like Apple, Google, Samsung, Facebook, and Amazon have begun to realize the impact that facial recognition can have on their existing security infrastructure. </p><p class="inline">Apple has been attempting to add movement capabilities to the system. The subject of a scanned face can now be talking or moving during the scan, which allows facial recognition to combine with other biometric security measures like voice recognition. Since moving subjects can be scanned, individuals can be identified in a crowd without intrusion by using facial recognition systems.</p><br />
<h2>Technology Brief</h2><br />
<p class="inline">Facial recognition systems can use either a 2D/3D image or video feed to create a digital image, establish the faceprint, and identify a face by comparing its digital image with the faceprints in a database. Every face has several “landmarks” and the system will flag these as “nodal points”. A human face can have up to 80 of these points. They represent areas of interest on the face that the system measures. </p><p class="expand inline mw-collapsible-content">Some examples of these measurements would be, distance between the eyes, width of the nose, depth of the eye socket, and more. These measurements will be stored in a database as a faceprint. </p><p class="inline">When the system scans a face, it will compare all these measurements to the records, faceprints, in the database. </p><p class="expand inline mw-collapsible-content">Facial recognition systems employ an algorithm, such as the Facial Recognition Vendor Test, that can predict whether there’s a match based on the “nodal points” on an individual’s face. </p><p class="inline">Usually, there is a 4-stage process involved in the operation of this technology [*]: </p><br />
<p>Capture – a physical or behavioral sample is captured by the system during enrolment</p><br />
<p>Extraction – Unique data is extracted from the sample and a template is created</p><br />
<p>Comparison – the template is then compared with a new sample</p><br />
<p>Matching – the system then decides if the feature extracted from the new sample is a match or not</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Facial recognition offers another form of biometric identification and authentication. Several vendors have been leveraging face recognition as an access control and authentication tool either for their clients or for internal use. Its application is not limited to a form of security measure, but it can also be used for healthcare and retail.</p><p class="expand inline mw-collapsible-content"> Although 2D face recognition may not be as accurate as other forms of biometric technology like fingerprint readers, it does have its advantages. </p><p class="inline">The subject of a scanned face does not have to know when they were scanned, which demonstrates that the technology can be used in large crowds to identify threats quickly.</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">Several vendors in today’s market are leveraging face recognition for their many applications. </p><p class="inline">For example, Amazon has been developing a system, which allows users to pay for their items using an actionable image (selfie). A user can use a selfie in which he is moving or speaking a particular phrase as a password to verify their identity when paying for an item. The logic behind having the customer speak a phrase or make a movement, is to try to eliminate the possibility of fooling the system with a scanned 2D image of the person. </p><p class="expand inline mw-collapsible-content">Amazon Rekognition is another product that provides two API sets - Amazon Rekognition Image for analyzing images and Amazon Rekognition Video for analyzing videos. Both APIs perform detection and recognition analysis of images and videos to provide insights for use in your applications.</p><br />
<p>After purchasing Face.com in 2012, Facebook began using facial recognition technology to connect users with their photos. When a user uploads a photo, the software will automatically suggest other people to tag. When you are tagged in a photo, more content is shown in a grouped format about other tagged individuals. </p><br />
<p class="inline">Face ID is a technology developed by Apple and introduced in iPhone X. It provides intuitive and secure authentication enabled by the state-of-the-art TrueDepth camera system with advanced technologies to accurately map the geometry of your face. With a simple glance, Face ID securely unlocks your iPhone or iPad Pro. </p><p class="expand inline mw-collapsible-content">You can use it to authorize purchases from the iTunes Store, App Store, and Apple Books, and make payments with Apple Pay. The iPhone XR, XS, and XS Max are all packing the second-generation of Face ID, which is an updated version of the biometric authentication system that is supposed to be faster than the version introduced with the iPhone X.</p><br />
<h2>Canadian Government Use</h2><br />
<p class="expand mw-collapsible-content">Unlike the private sector, Government use cases for facial recognition applications are primarily related to security, specifically for identity verification and fraud prevention. For example, the Canada Border Services Agency (CBSA) has recently launched the Primary Inspection Kiosk (PIK) program where passengers entering the country from airports must check-in using self-serve kiosks. These kiosks use facial recognition in order to clear passengers. The overall shift to un-maned kiosks has bolstered security while reducing congestion at airports and has been in development since 2015. Portuguese company Vision-Box has installed 130 Kiosks in the Toronto’s Pearson International Airport. The Kiosks are designed to take biometric data in two phases - facial recognition and fingerprint biometrics. The kiosks will also be able to obtain iris data, a feature reserved for people travelling under the NEXUS program.</p><br />
<p class="inline">Facial recognition systems are also used in provincial casinos for identifying and locking out visitors with gambling addictions who have voluntarily entered themselves into self-exclusion lists. It is worth noting that the system was developed jointly with the Ontario Privacy Commissioner to ensure a privacy-by-default design. </p><p class="expand inline mw-collapsible-content">In real time, the system scans customers entering the casino and compares their images with gamblers on the self-exclusion list. If there is a match, the system notifies security and if not, the system deletes the image automatically. Access to the database is restricted and information about an individual is only accessible if the person in the picture is physically present.</p><br />
<p class="inline">Passport Canada has been using facial recognition software for the past decade to compare new passport photos against its database to prevent passport fraud. One to one (1:1) comparisons are done to confirm a person’s identity, meaning that a recently taken image is compared to one already in the database that is associated with that person’s identity. One to many (1:N) comparisons are done to compare an image against the entire database of passport photos to make sure there are no duplicate applicants or individuals with multiple identities. </p><p class="expand inline mw-collapsible-content">This initiative has been successfully used to catch individuals attempting to obtain multiple passports. This same concept is also used for driver’s licences being issued at the provincial level. </p><br />
<p>Bill C-309, An Act to Amend the Criminal Code has made the concealment of identity (using masks or disguises) unlawful while participating in riots or unlawful assemblies. Although the Privacy Act and PIPEDA (Personal Information Protection and Electronic Documents Act) state that consent must be obtained before private information is collected, Bill C-309 paves the way for law enforcement to scan large crowds using facial recognition software and uncover the identities of participants.</p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="inline">SSC could leverage this technology by offering facial recognition as a service. SSC could utilise this technology to replace the current government employee security ID. A smart camera will instantly capture the biometric data of the individuals for local analysis and then open the gate to access the building. This service could reduce ongoing security costs associated with having a security team on site, but there probably will not be any savings in the short term due to the cost of developing the applications and installing the related equipment. </p><p class="expand inline mw-collapsible-content">Facial recognition technology is a non-intrusive form of identity verification that cannot be lost by the individual. Within the SSC context, it would eliminate the need for employees to carry security passes. Additionally, this would help prevent unauthorized individuals from gaining access to secure facilities. </p><p class="inline">Two-factor authentication with a user’s face could also be used for accessing secure files with higher security classifications (such as secret documents).</p><br />
<p class="inline-spacer"></p><br />
<p class="expand inline mw-collapsible-content">There is a growing trend among smartphone manufacturers to create devices that can be unlocked with facial recognition technology. </p><p class="inline">A market research firm based in Hong Kong estimates that nearly 64% (or 1 billion) of all smartphones shipped worldwide will have facial recognition capabilities in 2020. SSC can take advantage of this research and only issue phones with facial recognition capabilities to employees. This biometric information can be paired with any other type of authentication method to create a two-step verification process for all smartphones. </p><p class="expand inline mw-collapsible-content">SSC would not need to acquire any additional software licences, as these phones would already have the capacity for facial recognition. Since verification is done locally, (reference images are stored on the device outside of a cloud environment) this minimizes the security risks associated with facial recognition technology.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Although facial recognition requires a lot of computing power to process images in real time, Edge Computing can mitigate this concern. Image pre-processing tasks can be completed by the device that took the picture, or much closer to the device than the data center. The device would capture the image, scan it for faces, and then extract information like a faceprint from the image. Once the faceprint has been created, it is sent to the main server for authentication and the original image is discarded. </p><p class="expand inline mw-collapsible-content">Since pre-processing of the faceprint has been done outside of the server, the server can focus on verifying the recent faceprint against an internal match.</p><br />
<h4>Challenges</h4><br />
<p class="expand mw-collapsible-content">The primary challenge with facial recognition technology is privacy. One way to mitigate the privacy concern is to leverage Edge Computing to store biometric data locally, which helps prevent data loss and inappropriate cross-linking of data across systems. The data is better stored as biometric features (a faceprint for example) rather than images of people’s faces, which should be destroyed once the faceprint is collected; faceprint data should be stored, encrypted, and be made available through secured measures. In this way, it will inhibit the misuse of the image for unauthorized reasons.</p><br />
<p>If this technology were to be used to authenticate identities in multiple areas, there may be large amounts of hardware required for computing power. If facial recognition technology, with millions of scanned faceprints were to be adopted by one or multiple departments, the computing capacity being used to process the requirements and match the image information with the faceprint records in the database would be extensive. From this perspective, if SSC were to support such a project, the demanded computing power to support this technology could be provided either in the form of a private or public cloud to meet the computing demands.</p><br />
<p>There are also certain factors that can limit the accuracy of facial recognition systems. If the photo was taken in profile or if the image quality is too low, there may not be enough information available for the system to extract and generate a match. Haircuts, skin color, makeup, glasses, and face coverings such as surgical masks can also lower recognition accuracy. Considering that these systems are based on Artificial Intelligence (AI), there is also the possibility of improperly “training” them.</p><br />
<p>An AI for a facial recognition system should be manually supervised to “reward” correct matches, but if the training set only consists of a very specific demographic of people it will have difficulty with detecting other types of faces. The lack of a diverse training set creates recognition biases in the programs, which makes them better at detecting and correctly identifying individuals with specific attributes over others.</p><br />
<p class="expand mw-collapsible">In a study conducted by Joy Buolamwini where three different facial recognition systems were tested for accuracy in determining genders, they had error rates between 21% and 35% for women with darker skin tones whereas the error rate for light-skinned males was less than 1%. This brings into question the reliability of these systems. To avoid discrimination against specific minority groups, they must be developed and tested to make sure they don’t have any recognition biases.</p><br />
<p class="inline">These systems also have fuzzy accuracy rates, meaning that matches are never 100% accurate when doing searches, for example, in a database of images. There is the real possibility of false positives, where matches are found but they aren’t for the right person, and of false negatives, where the real match exists within the database but the system is unable to create a match. </p><p class="expand inline mw-collapsible-content">This accuracy gap means that the systems should only be used by trained individuals who understand how the technology works and specific guidelines should be followed when matches are generated by the system.</p><br />
<p class="expand mw-collapsible">A good example in the field is how the Toronto police force is using the system: only six FBI-trained Toronto Police officers can use the system and it can only generate a list of candidates. They do not use the system as a sole basis for arrests but in tandem with other traditional evidence gathering methods. AI systems, if they are to help inform important decisions should never be solely trusted and should be used as tools to inform decisions, not guide them.</p><br />
<p class="inline">To deal with the issue of bad lighting or faces at unrecognizable angles, some systems are altering images so that they are more “readable”. </p><p class="expand inline mw-collapsible-content">Panasonic has developed facial recognition software that analyses movement, speed and lighting present in videos to automatically correct still images that would otherwise be blurry. </p><p class="inline">Since the software adjusts the image before being analysed, it creates a new concern that additional false positives will be created. If an image has been touched up and edited before it was “plugged-in” to a facial recognition system, this can alter the faceprint being analysed and the search results may be biased or incorrect.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Another limitation of these systems is that it can only recognize individuals whose images are already contained within its database. </p><p class="expand inline mw-collapsible-content">Systems must also be able to perform “liveness” testing, or in other words, they must be able to determine if the subject in question is actually there in person since faces are not secret, in the same sense that passwords are secret, and faces cannot be hidden. Facial recognition systems rely on the difficulty of impersonating a real person to keep the system secure. Given this fact, the system needs to be able to determine the “liveness” of the image it’s analysing and determine if the person it just photographed is real or if it’s a picture.</p><br />
<p class="inline-spacer"></p><br />
<p class="inline">Facial recognition system technology has not yet been regulated in Canada and organizations that currently use it must operate within a specific legal framework. Under the Canadian Privacy Act, federal government institutions can only use personal information for the specific purpose for which it was collected and consent of the individual must be obtained before that information can be used for another purpose. Under PIPEDA, an organization must inform individuals and receive consent to any use of their personal information. </p><p class="expand inline mw-collapsible-content">This is a potential legal barrier to any organization planning to conduct live analysis of public crowds since each individual would need to consent to the collection and use of their faces (private information). </p><p class="inline">These regulations ensure that databases containing personal information belonging to different GC departments cannot be shared between departments for purposes other than the specific use for which consent was obtained.</p><br />
<br />
<h4>Considerations</h4><br />
<p>Any wide-scale use case developed by SSC will need to be reported to the Office of the Privacy Commissioner (OPC) for assessment. They would also need to comply with the Privacy Act and PIPEDA. Use cases must also be justified with regards to possible privacy intrusions and the OPC has suggested a four-part test to determine this: </p><br />
<ul><br />
<li>Is the measure demonstrably necessary to meet a specific need?</li><br />
<li>Is it likely to be effective in meeting that need?</li><br />
<li>Would the loss of privacy be proportionate to the benefit gained?</li><br />
<li>Is there a less privacy-invasive way of achieving the same end?</li><br />
</ul><br />
<p class="expand mw-collapsible-content">Additionally, use cases that are to be deployed on a wide scale within SSC, such as facial recognition systems to enter secured buildings, would rely on the consent of all participants. High quality images with neutral facial expressions would also need to be collected, or compiled from existing databases (i.e. building passes). These new centralized repositories of staff pictures could also be new targets of cybersecurity attacks. They would need to have the utmost protections afforded to them. Smaller scale use cases, such as unlocking a device with a face capture, would pose smaller risks since the information is stored locally.</p><br />
<p>If SSC were to adopt or develop facial recognition software, the software would need to be tested for recognition biases. Meaning, that it should have uniform recognition performance across all genders, skin types, and ages. If there are any biases present, the software would need to undergo more development and could garner more associated costs. Facial recognition applications purchased from a vendor would need to be closely scrutinized and evaluated since SSC would not know exactly how the application was trained and developed. Additionally, there are no industry standards for facial recognition products, meaning that there is no benchmarked performance level that needs to be achieved before a product goes to market. If SSC were to develop the software itself, it would have complete control and knowledge over the performance of the system.</p><br />
<p class="expand mw-collapsible-content">Lastly, it is important to consider the legal context of any facial recognition systems deployed within SSC. Presently, the technology is not regulated and performance standards have not been set for developers. The legal landscape could change by the time SSC is ready to adopt the technology. An ongoing privacy assessment of current service offerings may help provide a better understanding of the impact that facial recognition technology would have.</p><br />
<h2>References</h2><br />
<div style="display: none"><br />
<ref>Dennis, A. L. (2018, October 15). Data Lakes 101: An Overview. Retrieved from <i>[https://www.dataversity.net/data-lakes-101-overview/#]</i></ref><br />
<ref>Marvin, R., Marvin, R., & Marvin, R. (2016, August 22). Data Lakes, Explained. Retrieved from <i>[ https://www.pcmag.com/article/347020/data-lakes-explained]</i></ref><br />
<ref>The Data Lake journey. (2014, March 15). Retrieved from <i>[https://hortonworks.com/blog/enterprise-hadoop-journey-data-lake/]</i></ref><br />
<ref>Google File System. (2019, July 14). Retrieved from <i>[https://en.wikipedia.org/wiki/Google_File_System]</i></ref><br />
<ref>Coates, M. (2016, October 02). Data Lake Use Cases and Planning Considerations. Retrieved from <i>[https://www.sqlchick.com/entries/2016/7/31/data-lake-use-cases-and-planning]</i></ref><br />
<ref>Bhalchandra, V. (2018, July 23). Six reasons to think twice about your data lake strategy. Retrieved from <i>[https://dataconomy.com/2018/07/six-reasons-to-think-twice-about-your-data-lake-strategy/]</i></ref><br />
<ref>Data Lake Expectations: Why Data Lakes Fail. (2018, September 20). Retrieved from <i>[https://www.arcadiadata.com/blog/the-top-six-reasons-data-lakes-have-failed-to-live-up-to-expectations/]</i></ref><br />
<ref>Data Lake: AWS Solutions. (n.d.). Retrieved from <i>[https://aws.amazon.com/solutions/data-lake-solution/]</i></ref><br />
</div><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Face Recognition";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Comit%C3%A9_Consultatif_sur_le_Cadre_d%27Architecture&diff=11183Comité Consultatif sur le Cadre d'Architecture2019-07-19T14:54:15Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:{{FULLPAGENAME}}}}<br />
[[fr:Tendances_Technologiques]]<br />
<br />
<div class="sideTable"><br />
<div class="navigation-section"><br />
<div class="navigation-table"><br />
<table class="wikitable nav-tab"><br />
<tr><th class="breadcrumb" colspan="2"><b>[[Comité_Consultatif_sur_le_Cadre_d%27Architecture|Comité Consultatif sur le Cadre d'Architecture]]</b><br><b>[[Architecture_Framework_Advisory_Committee|Architecture Framework Advisory Committee]]</b></th></tr><br />
<tr><td colspan="2" class="logo">[[File:AFAC_EN_Logo.png|200px]]</td></tr><br />
<tr><br />
<th>Contactez Nous</th><br />
<td>[mailto:ssc.afac-ccca.spc@canada.ca ssc.afac-ccca.spc@canada.ca]</td><br />
</tr><br />
<tr><br />
<th>Autres Publications Connexes</th><br />
<td><b>[[Tendances_Technologiques|Tendances Technologiques]]</b></td><br />
</tr><br />
<tr><td colspan="2" class="news"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="newsText">Cette nouvelle section est maintenant disponible!</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
</div><br />
</div><br />
<br />
<div><br />
<br />
<p>Le Comité Consultatif sur le Cadre d’Architecture (CCCA) de Services partagés Canada (SPC) fournit des conseils et des recommandations stratégiques à l’appui du mandat de SPC et des objectifs associés au gouvernement du Canada.</p><br />
<br />
<h2>Mandat</h2><br />
<br />
<p>CCCA soutient le mandat de SPC et les initiatives de transformation numérique du GC en servant de forum de consultations des secteurs publics et privés sur l’architecture d’entreprise. SPC identifiera ou analysera les approches commerciales sous une optique stratégique avec des spécialistes du domaine de l'industry afin d’élaborer ou d’affiner les directions proposées pour la planification à moyen et long terme (p. Ex. Cycles de planification de trois à dix ans).</p><br />
<br />
<h2>Communauté</h2><br />
<br />
<p>La communauté est composée d'individus et d'organisations des secteurs publics et privés ayant une expertise dans les technologies de l'information et de la communication (TIC).</p><br />
<br />
<p>Les réunions de CCCA sont animées par :</p><br />
<br />
<ul><br />
<li>Président – Dirigeant principal de la technologie (DPT), SPC</li><br />
<li>Coprésident – DPT du gouvernement du Canada, Secrétariat du Conseil du Trésor du Canada</li><br />
</ul><br />
<br />
<p>La communauté de CCCA comprend les associations ci-dessous du secteur des TIC (et leurs affiliations) :</p><br />
<br />
<ul><br />
<li>Réseau Canadien de Technologie de l'Information d'Affaires (CABiNET)</li><br />
<li>Alliance Canadienne de Technologie de Pointe (CATA)</li><br />
<li>Association Canadienne de la Technologie de l'Information (ACTI)</li><br />
<li>Investir Ottawa</li><br />
</ul><br />
<br />
<p>Les membres d'office sont : </p><br />
<br />
<ul><br />
<li>Dirigeants principaux de l'information de d'autres ministères du gouvernement du Canada</li><br />
</ul><br />
<br />
<p>CCCA demandera aux associations du secteur des TIC la désignation de sociétés possédant une expertise pertinente dans des domaines intéressant le comité. Chaque membre de l'association aura l'occasion de nommer un à deux experts en la matière pour participer aux discussions relatives à leurs domaines de compétence.</p><br />
<br />
<p>SPC se réserve le droit d'adresser des invitations à des organisations du secteur privé et non gouvernementales dotées d'expertise pertinente, ainsi que la possibilité d'ajuster ou de compléter la liste des personnes et d'organisations participant aux réunions afin de garantir le niveau d'expertise requis. Les membres du personnel de SPC ayant des responsabilités ou une expertise en rapport avec les sujets traités dans le CCCA ou les réunions de groupes de travail connexes seront invités à participer en fonction des besoins.</p><br />
<br />
<h2>Réunions et Agenda</h2><br />
<br />
<p>CCCA se réunit tous les trimestres ou, au besoin, sur convocation du président. Les ordres du jour varient en fonction des sujets de discussion. CCCA veillera à ce que le niveau d'expertise nécessaire soit partagé lors des consultations, tout en limitant le nombre de participants à une taille raisonnable, afin d'encourager la participation active et de maximiser la valeur du temps alloué à chaque point de l'ordre du jour.</p><br />
<br />
<h2>Soutient du Secrétariat</h2><br />
<br />
<p>Le Secrétariat de l'engagement au sein de PGET, PVR, gérera la logistique et fournira des services de secrétariat. Le matériel de réunion sera fourni au moins une semaine avant les réunions prévues. Des services de téléconférence seront fournis aux participants participant à distance.</p><br />
<br />
<p>Le secrétariat préparera les procès-verbaux de chaque réunion et les diffusera en interne au sein de SPC. Les commentaires formulés lors des réunions ne seront pas attribués à des individus ou à des organisations.</p><br />
<br />
</div><br />
<br />
<div class="container"><br />
<div class="row"><br />
<div class="col-sm-12"><br />
<h2>Sujets Antérieur</h2><br />
<br />
<table class="wikitable sortable technology-table"><br />
<tr><br />
<th>Sujet</th><br />
<th>Date</th><br />
<th>Présentation </th><br />
<th>Summaire de Discussion</th><br />
<br />
</tr><br />
<tr><br />
<td>Adoption de la conteneurisation des applications</td><br />
<td>2019-04-08</td><br />
<td>[[Media:EN_-_AFAC_Presentation_-_2019-04-08.pdf|EN]] / [[Media:FR_-_Exposé_(Adoption_de_la_conteneurisation_des_applications).pdf|FR]]</td><br />
<td>[[Media:EN_-_AFAC_Containers_-_Discussion_Summary_v0.1.pdf|EN]]</td><br />
<br />
</tr><br />
<tr><br />
<td>Mise en oeuvre de la stratégie de données d’entreprise du GC</td><br />
<td>2019-06-14</td><br />
<td>[[Media:EN_-_AFAC_(EDS)_Presentation_-_2019-06-14_vFinal_EN.pdf|EN]] / [[Media:FR_-_AFAC_(EDS)_Presentation_-_2019-06-14_vFinal_FR.pdf|FR]]</td><br />
<td></td><br />
<br />
</tr><br />
</div><br />
</div><br />
</div><br />
<br />
<br />
[[Category:Architecture]]<br />
[[Category:Technology]]<br />
<br />
{{#css:<br />
.technology-table{<br />
width: 100%;<br />
margin: auto;<br />
}<br />
<br />
th, td{<br />
text-align: center;<br />
}<br />
<br />
.logo{<br />
text-align: center;<br />
}<br />
<br />
.disclaimer{<br />
border: 1px solid red;<br />
background-color: rgba(255, 0, 0, 0.3);<br />
}<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
padding: 5px 10px 5px 10px;<br />
}<br />
<br />
.disclaimerText{<br />
width: 100%;<br />
text-align: left;<br />
}<br />
<br />
.news{<br />
border: 1px solid green;<br />
background-color: rgba(0, 255, 0, 0.3);<br />
}<br />
<br />
.toggle{<br />
color: rgb(6, 69, 173);<br />
float:left;<br />
<br />
}<br />
<br />
.toggle:hover{<br />
text-decoration: underline;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { float: left; }<br />
<br />
.col-sm-1 { width: 8.333333333%; }<br />
.col-sm-2 { width: 16.66666666%%; }<br />
.col-sm-3 { width: 25%; }<br />
.col-sm-4 { width: 33.333333333333%; }<br />
.col-sm-5 { width: 41.6666666666%; }<br />
.col-sm-6 { width: 50%; }<br />
.col-sm-7 { width: 58.333333333%; }<br />
.col-sm-8 { width: 66.66666666666%; }<br />
.col-sm-9 { width: 75%; }<br />
.col-sm-10 { width: 83.33333333333%; }<br />
.col-sm-11 { width: 91.6666666666%; }<br />
.col-sm-12 { width: 100%; }<br />
<br />
.sideTable {<br />
width: 27.5%;<br />
float: right;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 95%;<br />
margin-left: 2.5%;<br />
margin-right: 2.5%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
}<br />
<br />
.inverse-direction {<br />
float: right;<br />
}<br />
<br />
.nav-tab {<br />
width: 100%;<br />
}<br />
<br />
@media (max-width: 1050px){<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.sideTable {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-table {<br />
width: 30%;<br />
margin-left: 1.6666666%;<br />
margin-right: 1.6666666%;<br />
float: left;<br />
}<br />
<br />
.content-section {<br />
float: left;<br />
}<br />
<br />
.inverse-direction {<br />
clear: both;<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { clear: both; width: 100%; margin: auto; }<br />
}<br />
<br />
div>a>img {<br />
width: 75%;<br />
height: auto;<br />
margin: auto;<br />
}<br />
<br />
@media (max-width: 850px){<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
clear: both;<br />
}<br />
}<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Architecture_Framework_Advisory_Committee&diff=11182Architecture Framework Advisory Committee2019-07-19T14:53:05Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:{{FULLPAGENAME}}}}<br />
[[fr:Tendances_Technologiques]]<br />
<br />
<div class="sideTable"><br />
<div class="navigation-section"><br />
<div class="navigation-table"><br />
<table class="wikitable nav-tab"><br />
<tr><th class="breadcrumb" colspan="2"><b>[[Architecture_Framework_Advisory_Committee|Architecture Framework Advisory Committee]]</b><br><b>[[Comité_Consultatif_sur_le_Cadre_d%27Architecture|Comité Consultatif sur le Cadre d'Architecture]]</b></th></tr><br />
<tr><td colspan="2" class="logo">[[File:AFAC_EN_Logo.png|200px]]</td></tr><br />
<tr><br />
<th>Contact Us</th><br />
<td>[mailto:ssc.afac-ccca.spc@canada.ca ssc.afac-ccca.spc@canada.ca]</td><br />
</tr><br />
<tr><br />
<th>Other Related Publications</th><br />
<td><b>[[Technology_Trends|Technology Trends]]</b></td><br />
</tr><br />
<tr><td colspan="2" class="news"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="newsText">This new section is now available!</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
</div><br />
</div><br />
<br />
<div><br />
<br />
<p>The Shared Services Canada (SSC) Architecture Framework Advisory Committee (AFAC) provides strategic advice and recommendations in support of SSC’s mandate and the Government of Canada’s (GC) associated objectives.</p><br />
<br />
<h2>Mandate</h2><br />
<br />
<p>AFAC supports SSC’s mandate and the GC’s digital transformation initiatives by serving as a forum for public-private sector consultation on enterprise architecture. SSC will identify or review business approaches through a strategic lens with industry subject matter experts (SMEs) to develop or refine proposed directions for mid to long-term planning (e.g., three- to ten-year planning cycles).</p><br />
<br />
<h2>Membership</h2><br />
<br />
<p>The membership is composed of individuals and organizations from the public and private sectors with expertise in information and communications technology (ICT).</p><br />
<br />
<p>AFAC meetings are led by the following:</p><br />
<br />
<ul><br />
<li>Chair – Chief Technology Officer (CTO), SSC</li><br />
<li>Co-chair – CTO for the Government of Canada, Treasury Board of Canada Secretariat</li><br />
</ul><br />
<br />
<p>AFAC membership includes the below ICT-sector associations (and memberships thereof):</p><br />
<br />
<ul><br />
<li>Canadian Business Information Technology Network (CABiNET)</li><br />
<li>Canadian Advanced Technology Alliance (CATA)</li><br />
<li>Information Technology Association of Canada (ITAC)</li><br />
<li>Invest Ottawa</li><br />
</ul><br />
<br />
<p>The ex-officio members are:</p><br />
<br />
<ul><br />
<li>Chief Information Officers of other Government of Canada departments</li><br />
</ul><br />
<br />
<p>AFAC will request from the ICT-sector associations nominations of companies with relevant expertise in topics of interest to the committee. Each member association will have the opportunity to nominate one to two subject matter experts to participate in discussions pertaining to their areas of expertise.</p><br />
<br />
<p>SSC reserves the right to extend invitations to private-sector and non-governmental organizations with relevant expertise, as well as the option to adjust or supplement the list of individuals and organizations participating in meetings to ensure that the required level of expertise is available. SSC staff with responsibilities or expertise related to topics being addressed in AFAC or related working-group meetings will be invited to participate on an as-needed basis.</p><br />
<br />
<h2>Meetings and Agenda</h2><br />
<br />
<p>AFAC meets quarterly or, as required, at the call of the Chair. Agendas vary depending on the topics of discussion. AFAC will ensure that the necessary degree of expertise will be shared during consultations, while also limiting the number of participants to a manageable size, to encourage active participation and to maximize the value of the time allocated to each agenda item.</p><br />
<br />
<h2>Secretariat Support</h2><br />
<br />
<p>The Engagement Secretariat within PGET, PVR will manage logistics and provide secretariat services. Meeting materials will be provided a minimum of one week prior to any scheduled meetings. Teleconference services will be provided for attendees participating remotely.</p><br />
<br />
<p>The secretariat will prepare minutes for each meeting and circulate them internally within SSC. Comments made during meetings will not be attributed to individuals or organizations.</p><br />
</div><br />
<div class="container"><br />
<div class="row"><br />
<div class="col-sm-12"><br />
<h2>Past Topics</h2><br />
<br />
<table class="wikitable sortable technology-table"><br />
<tr><br />
<th>Subject</th><br />
<th>Date</th><br />
<th>Presentation</th><br />
<th>Discussion Summary </th><br />
<br />
</tr><br />
<tr><br />
<td>Adoption of Application Containers</td><br />
<td>2019-04-08</td><br />
<td>[[Media:EN_-_AFAC_Presentation_-_2019-04-08.pdf|EN]] / [[Media:FR_-_Exposé_(Adoption_de_la_conteneurisation_des_applications).pdf|FR]]</td><br />
<td>[[Media:EN_-_AFAC Containers - Discussion Summary v0.1.pdf|EN]]</td><br />
<br />
</tr><br />
<tr><br />
<td>Enabling the GC Enterprise Data Strategy</td><br />
<td>2019-06-14</td><br />
<td>[[Media:EN_-_AFAC_(EDS)_Presentation_-_2019-06-14_vFinal_EN.pdf|EN]] / [[Media:FR_-_AFAC_(EDS)_Presentation_-_2019-06-14_vFinal_FR.pdf|FR]]</td><br />
<td></td><br />
<br />
</tr><br />
</div><br />
</div><br />
</div><br />
[[Category:Architecture]]<br />
[[Category:Technology]]<br />
<br />
{{#css:<br />
.technology-table{<br />
width: 100%;<br />
margin: auto;<br />
}<br />
<br />
th, td{<br />
text-align: center;<br />
}<br />
<br />
.logo{<br />
text-align: center;<br />
}<br />
<br />
.disclaimer{<br />
border: 1px solid red;<br />
background-color: rgba(255, 0, 0, 0.3);<br />
}<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
padding: 5px 10px 5px 10px;<br />
}<br />
<br />
.disclaimerText{<br />
width: 100%;<br />
text-align: left;<br />
}<br />
<br />
.news{<br />
border: 1px solid green;<br />
background-color: rgba(0, 255, 0, 0.3);<br />
}<br />
<br />
.toggle{<br />
color: rgb(6, 69, 173);<br />
float:left;<br />
<br />
}<br />
<br />
.toggle:hover{<br />
text-decoration: underline;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { float: left; }<br />
<br />
.col-sm-1 { width: 8.333333333%; }<br />
.col-sm-2 { width: 16.66666666%%; }<br />
.col-sm-3 { width: 25%; }<br />
.col-sm-4 { width: 33.333333333333%; }<br />
.col-sm-5 { width: 41.6666666666%; }<br />
.col-sm-6 { width: 50%; }<br />
.col-sm-7 { width: 58.333333333%; }<br />
.col-sm-8 { width: 66.66666666666%; }<br />
.col-sm-9 { width: 75%; }<br />
.col-sm-10 { width: 83.33333333333%; }<br />
.col-sm-11 { width: 91.6666666666%; }<br />
.col-sm-12 { width: 100%; }<br />
<br />
.sideTable {<br />
width: 27.5%;<br />
float: right;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 95%;<br />
margin-left: 2.5%;<br />
margin-right: 2.5%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
}<br />
<br />
.inverse-direction {<br />
float: right;<br />
}<br />
<br />
.nav-tab {<br />
width: 100%;<br />
}<br />
<br />
@media (max-width: 1050px){<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.sideTable {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-table {<br />
width: 30%;<br />
margin-left: 1.6666666%;<br />
margin-right: 1.6666666%;<br />
float: left;<br />
}<br />
<br />
.content-section {<br />
float: left;<br />
}<br />
<br />
.inverse-direction {<br />
clear: both;<br />
}<br />
<br />
.col-sm-1,<br />
.col-sm-2,<br />
.col-sm-3,<br />
.col-sm-4,<br />
.col-sm-5,<br />
.col-sm-6,<br />
.col-sm-7,<br />
.col-sm-8,<br />
.col-sm-9,<br />
.col-sm-10,<br />
.col-sm-11,<br />
.col-sm-12 { clear: both; width: 100%; margin: auto; }<br />
}<br />
<br />
div>a>img {<br />
width: 75%;<br />
height: auto;<br />
margin: auto;<br />
}<br />
<br />
@media (max-width: 850px){<br />
<br />
.navigation-section {<br />
clear: both;<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
}<br />
<br />
.navigation-table {<br />
width: 100%;<br />
margin-left: 0%;<br />
margin-right: 0%;<br />
clear: both;<br />
}<br />
}<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Face_Recognition&diff=11165Technology Trends/Face Recognition2019-07-19T13:08:50Z<p>Kpere060: Created page with "test"</p>
<hr />
<div>test</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Datalakes&diff=11154Technology Trends/Datalakes2019-07-18T19:55:15Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Datalakes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Lac de Donnés]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[200px]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>July 12, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Datalakes_v0.1_EN_Published.pdf|Datalakes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Datalakes</b> is a central system or repository of data that is stored in its natural/raw format. A datalake acts as a single store for all enterprise data. Data is transformed using machine learning, advanced,analytics, and visualization. Several forms of data can be hosed in a datalake. These include structured data from relational databases, unstructured data, semi-structured data, and binary data. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>In an ever-increasing hyperconnected world, corporations and businesses are struggling to deal with the responsibilities of storage, management and quick availability of raw data. To break these data challenges down further:</p><br />
<ul><br />
<li>Data comes in many different structures.<br />
<ul><br />
<li>Unstructured</li><br />
<li>Semi-Sturctured</li><br />
<li>Structured</li><br />
</ul></li><br />
<li>Data comes from many disparate sources.<br />
<ul><br />
<li>Enterprise Applications</li><br />
<li>Raw Files</li><br />
<li>Operation and Security Logs</li><br />
<li>Financial Transactions</li><br />
<li>Internet of Things (IoT) Devices and Network Sensors</li><br />
<li>Websites</li><br />
<li>Scientific Research</li><br />
</ul></li><br />
<li>Data sources are often geographically distributed to multiple locations<br />
<ul><br />
<li>Datacenters</li><br />
<li>Remote Offices</li><br />
<li>Mobile Devices</li><br />
</ul></li><br />
</ul><br />
<br />
<p>In an effort to resolve these data challenges, a new way of managing data was created which drove data oriented companies to invent a new data storage mechanism called a Data Lake.</p><br />
<p class="expand mw-collapsible-content">Data Lakes are characterized as: </p><br />
<ul class="expand mw-collapsible-content"><br />
<li>Collect Everything<ul><br />
<li>A Data Lake contains all data; raw sources over extended periods of time as well as any processed data.</li><br />
</ul></li><br />
<li>Dive in anywhere<ul><br />
<li>A Data Lake enables users across multiple business units to refine, explore and enrich data on their terms.</li><br />
</ul></li><br />
<li>Flexible Access<ul><br />
<li>o A Data Lake enables multiple data access patterns across a shared infrastructure: batch, interactive, online, search, in-memory and other processing engines.</li><br />
</ul></li><br />
</ul><br />
<br />
<p>Data Lakes are essentially a technology platform for holding data. Their value to the business is only realized when applying data science skills to the lake. </p><br />
<br />
<p>To summarize, usecases for Data Lakes are still being discovered. Cloud providers are making it easier to procure Data Lakes and today Data Lakes are primarily used by Research Institutions, Financial Services, Telecom, Media, Retail, Manufacturing, Healthcare, Pharma, Oi l& Gas and Governments.<br />
</p><br />
<br />
<br />
<br />
<h2>Technology Brief</h2><br />
<p class="expand mw-collapsible-content">The most popular implementation of a Data Lake is through the open source platform called Apache Hadoop. Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. Hadoop was originally created by researchers at Google as a storage method to handle the indexing of websites on the Internet; At that time it was called the Google File System. </p><br />
<p>A Data Lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.”</p><br />
<p class="expand mw-collapsible-content">Data can flow into the Data Lake by either batch processing or real-time processing of streaming data. Additionally, data itself is no longer restrained by initial schema decisions and can be exploited more freely by the enterprise. Rising above this repository is a set of capabilities that allow IT to provide Data and Analytics as a Service (DAaaS), in a supply-demand model. IT takes the role of the data provider (supplier), while business users (data scientists, business analysts) are consumers.</p><br />
<p>The DAaaS model enables users to self-serve their data and analytic needs. Users browse the lake’s data catalog (a Datapedia) to find and select the available data and fill a metaphorical “shopping cart” (effectively an analytics sandbox) with data to work with. Once access is provisioned, users can use the analytics tools of their choice to develop models and gain insights. Subsequently, users can publish analytical models or push refined or transformed data back into the Data Lake to share with the larger community.</p><br />
<p class="expand mw-collapsible-content">Although provisioning an analytic sandbox is a primary use, the Data Lake also has other applications. For example, the Data Lake can also be used to ingest raw data, curate the data, and apply Export-Transform-Load (ETL). This data can then be loaded to an Enterprise Data Warehouse. To take advantage of the flexibility provided by the Data Lake, organizations need to customize and configure the Data Lake to their specific requirements and domains.</p><br />
<h2>Industry Usage</h2><br />
<p>There are a variety of ways Data Lakes are being used in the industry:</p><br />
<ul><br />
<li><p><b>Ingestion of semi-structured and unstructured data sources (aka big data)</b>such as equipment readings, telemetry data, logs, streaming data, and so forth. A Data Lake is a great solution for storing IoT (Internet of Things) type of data which has traditionally been more difficult to store, and can support near real-time analysis. Optionally, you can also add structured data (i.e., extracted from a relational data source) to a Data Lake if your objective is a single repository of all data to be available via the lake.</p></li><br />
<li><p><b>Experimental analysis </b>of data before its value or purpose has been fully defined. Agility is important for every business these days, so a Data Lake can play an important role in "proof of value" type of situations because of the "ELT" approach discussed above.</p></li><br />
<li><p><b>Advanced analytics support. </b>A Data Lake is useful for data scientists and analysts to provision and experiment with data.</p></li><br />
<li><p><b>Archival and historical data storage. </b>Sometimes data is used infrequently, but does need to be available for analysis. A Data Lake strategy can be very valuable to support an active archive strategy.</p></li><br />
<li><p><b>Distributed processing </b>capabilities associated with a logical data warehouse.</p></li><br />
</ul><br />
<p class="expand mw-collapsible-content"><b>How TD Bank Made Its Data Lake More Usa</b></p><br />
<p class="expand mw-collapsible-content">[[https://www.datanami.com/2017/10/03/td-bank-made-data-lake-usable]]<br>Toronto-Dominion Bank (TD Bank) is one of the largest banks in North America, with 85,000 employees, more than 2,400 locations between Canada and the United States, and assets nearing $1 trillion. In 2014, the company decided to standardize how it warehouses data for various business intelligence and regulatory reporting functions. The company purchased a Hadoop distribution and set off to build a large cluster that could function as a centralized lake to store data originating from a variety of departments.</p><br />
<br />
<h2>Canadian Government Use</h2><br />
<p>In 2019, the Treasury Board of Canada Secretariat (TBS), partnered with Shared Services Canada and other departments, to identify a business lead to develop a Data Lake (a repository of raw data) service strategy so that the GC can take advantage of big data and market innovation to foster better analytics and promote horizontal data-sharing<ref>Treasury Board of Canada Secretariat. (March 29th, 2019). Digital Operations Strategic Plan: 2018-2022. Government of Canada. Treasury Board of Canada Secretariat. Retrieved 26-May-2019 from: <i>[https://www.canada.ca/en/government/system/digital-government/digital-operations-strategic-plan-2018-2022.html] </i></ref>. </p><br />
<p class="expand mw-collapsible-content">Big data is the technology that stores and processes data and information in datasets that are so large or complex that traditional data processing applications can’t analyze them. Big data can make available almost limitless amounts of information, improving data-driven decision-making and expanding open data initiatives. Business intelligence involves creating, aggregating, analyzing and visualizing data to inform and facilitate business management and strategy. TBS, working with departments, will lead the development of requirements for an enterprise analytics platform<ref>Ibid.<i></i></ref>.</p><br />
<p>Data Lake development in the GC is a more recent initiative. This is mainly due to the GC focussing resources on the implementation of cloud initiatives. However, there are some GC departments engaged in developing Data Lake environments in tandem to cloud initiatives.</p><br />
<p class="expand mw-collapsible-content">Notably, the Employment and Social Development Canada (ESDC) is preparing the installment of multiple Data Lakes in order to enable a Data Lake Ecosystem and Data Analytics and Machine Learning toolset. This will enable ESDC to share information horizontally both effectively and safely, while enabling a wide variety of data analytics capabilities. ESDC aims to maintain current data and analytics capabilities up-to-date while exploring new ones to mitigate gaps and continuously evolve our services to meet client’s needs<ref>Brisson, Yannick, and Craig, Sheila. (November, 2018). ESDC Data Lake – Implementation Strategy and Roadmap Update. Government of Canada. Employment and Social Development Canada – Data and Analytics Services. Presentation. Last Modified on 2019-04-26 15:45. Retrieved 07-May-2019 from GCDocs<i>[https://gcdocs.gc.ca/ssc-spc/llisapi.dll?func=ll&objaction=overview&objid=36624914 ]</i></ref>. </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="expand mw-collapsible-content">There are three common value propositions for pursuing Data Lakes. 1) It can provide an easy and accessible way to obtain data faster; 2) It can create a singular inflow point of data to help connect and merge information silos in an organization; and 3) It can provide an experimental environment for experienced data scientists to enable new analytical insights.</p><br />
<p class="inline-spacer"> </p><br />
<p class="inline">Data Lakes can provide data to consumers more quickly by offering data in a more raw and easily accessible form. Data is stored in its native form with little to no processing, it is optimized to store vast amounts of data in their native formats. By allowing the data to remain in its native format, a much timelier stream of data is available for unlimited queries and analysis. A Data Lake can help data consumers bypass strict data retrieval and data structured applications such as a data warehouse and/or data mart. This has the effect of improving a business’ data flexibility.</p><p class="expand inline mw-collapsible-content">Some companies have in fact used Data Lakes to replace existing warehousing environments where implementing a new data warehouse is more cost prohibitive. A Data Lake can contain unrefined data, this is helpful when either a business data structure is unknown, or when a data consumer requires access to the data quickly. </p><br />
<p class="inline-spacer"> </p><br />
<p class="inline">A Data Lake is not a single source of truth. A Data Lake is a central location in which data converges from all data sources and is stored, regardless of the data formatting. </p><p class="expand inline mw-collapsible-content">As a singular point for the inflow of data, sections of a business can pool their information together in the Data Lake and increase the sharing of information with other parts of the organization. In this way everyone in the organization has access to the data. A Data Lake can increase the horizontal data sharing within an organization by creating this singular data inflow point. Using a variety of storage and processing tools analysts can extract data value quickly in order to inform key business decisions.</p><br />
<p class="inline-spacer"> </p><br />
<p class="expand inline mw-collapsible-content">A Data Lake is optimized for exploration and provides an experimental environment for experienced data scientists to uncover new insights from data. Analysts can overlay context on the data to extract value. All organizations want to increase analytics and operational agility.</p><p class="inline">The Data Lake architectural approach can store large volumes of data, this can be a way in which cross-cutting teams can pool their data in a central location and by complementing their systems of record with systems of insight. </p><p class="expand inline mw-collapsible-content">Data Lakes present the most potential benefits for experienced and competant data scientists. </p><p class="inline">Having structured, unstructured and semistructured data, usually in the same data set, can contain business, predictive, and prescriptive insights previously not possible from a structured platform as observed in data warehouses and data marts.</p><br />
<br />
<h4>Challenges</h4><br />
<p>Although Data Lake technology has many benefits for organizations dealing with big data it has its own challenges. For example:</p><br />
<p><b>Data Governance and Semantic Issues</b></p><br />
<p class="expand inline mw-collapsible-content">The biggest challenge for Data Lakes is to resolve assorted data governance requirements in a single centralized data platform. Data Lakes fail mostly when they lack governance, self-disciplined users, and a rational data flow.</p><p class="inline">Often, Data Lake implementations are focused on storing data instead of managing the data. Data Lakes are not optimized for semantic enforcement or consistency. They are made for semantic flexibility, to allow anyone to provide context to data if they have the skills to do so. </p><br />
<p>Putting data in the same place does not remove it’s ambiguity or meaning. Data Lakes provide unconstrained, “no compromises” storage model environment without the data governance assurances common to data warehouses or data marts. Proper meta data is essential for a Data Lake, without appropriate meta data the Data Lake will not work as intended. It is beneficial to think of meta data as the fish finder in the Data Lake.</p><br />
<p><b class="expand mw-collapsible-content">Lack of Quality and Trust in Data</b></p><br />
<p class="expand mw-collapsible-content">Data quality and trust in the data is a perennial issue for many organizations. Although data discovery tools can apply Machine Learning across related datasets from multiple data sources to identify anomalies (incorrect values, missing values, duplicates and outdated data), quality and trustworthiness of data continue to be an issue for Data Lakes who can easily become data dumping grounds. Some data is more accurate than others. This can present a real problem for anyone using multiple data sets and making decisions based upon analysis conducted with data of varying degrees of quality.</p><br />
<p><b>Data Swamps, Performance, and Flexibility Challenges</b></p><br />
<p class="expand inline mw-collapsible-content">Data stored in Data Lakes can sometimes become muddy when good data is mixed with bad data. Data Lake infrastructure is meant to store and process large amounts of data, usually in massive data files. </p><p class="inline">A Data Lake is not optimized for a high number of users or diverse and simultaneous workloads due to intensive query tasks. This can result in performance degradation and failures are common when running extractions, transformations, and loading tasks all at the same time. On-premises Data Lakes face other performance challenges in that they have a static configuration. </p><br />
<p><b class="expand mw-collapsible-content">Data Hoarding and Storage Capacity</b></p><br />
<p class="expand mw-collapsible-content">Data stored in Data Lakes may actually never be used in production and stay unused indefinitely in the Data Lake. By storing massive amounts of historical data, the infinite Data Lake may skew analysis with data that is no longer relevant to the priorities of the business. In keeping the historical data the metadata describing it must be understood as well. This decreases the performance of the Data Lake by increasing the overall workload of employees to clean the datasets no longer in use for analysis.</p><br />
<p class="expand mw-collapsible-content">Storing increasingly massive amounts of data for an unlimited time will also lead to scalability and cost challenges. Scalability challenges are less of a risk in public cloud environments, but cost remains a factor. On-premises Data Lakes are more susceptible to cost challenges. This is because their cluster nodes require all three dimensions of computing (storage, memory and processing). Organizations of all kinds generate massive amounts of data (including meta data) and it is increasing exponentially.</p><br />
<p>The storage capacity of all this data (and future data) will be an ongoing challenge and one that will require constant management. While Data Lakes can and will be stored on the cloud, SSC as cloud broker for the GC will need to provide the appropriate infrastructure and scalability to clients.</p><br />
<p><b class="expand mw-collapsible-content">Advanced Users Required</b></p><br />
<p class="expand mw-collapsible-content">Data Lakes are not a platform to be explored by everyone. Data Lakes present an unrefined view of data that usually only the most highly skilled analysts are able to explore and engage in data refinement independent of any other formal system-of-record such as a data warehouse. </p><br />
<p class="expand mw-collapsible-content">Not just anyone in an organization is data-literate enough to derive value from large amounts of raw or uncurated data. The reality is only a handful of staff are skilled enough to navigate a Data Lake. Since Data Lakes store raw data their business value is entirely determined by the skills of Data Lake users. These skills are often lacking in an organization.</p><br />
<p><b>Data Security</b></p><br />
<p class="inline"></p>Data in a Data Lake lacks standard security protection with a relational database management system or an enterprise database. In practice, this means that the data is unencrypted and lacks access control.<p class="expand inline mw-collapsible-content">. Security is not just a binary solution. We have varying degrees of security (unclassified, secret, top secret, etc.) and all of which require different approaches. This will inevitably present challenges with the successful use of data from Data Lakes.To combat this, organizations will have to embrace a new security framework to be compatable with Data Lakes and Data Scientists.</p><br />
<h4>Considerations</h4><br />
<p class="expand mw-collapsible-content">Shared Services Canada (SSC) has an excellent opportunity to capitalize on its mandate of providing data storage service to GC’s other departments. SSC, as the GC’s Service Provider, could potentially a centralized GC Data Lake and allow GC Data Scientists access to this central data using a single unified Data Lake interface. However, this is a project which should be implemented after cloud has been adopted and enterprise data centers have been migrated to in order to provide adequate infrastructure and scaling.</p><br />
<p class="inline">Data Lakes should not be confused for conventional databases although they both store information. A Data Lake will always underperform when tasked with the jobs of a conventional database. </p><p class="expand inline mw-collapsible-content">To combat this, SSC must create data architectures that define the proper application of Data Lakes. Too often, Data Lakes suffer from lack of foresight on what they're supposed to achieve. </p><p class="inline">Creating a Data Lake becomes the goal rather than achieving a strategic objective. </p><p class="expand inline mw-collapsible-content"></p><br />
<p class="expand mw-collapsible-content">Shared Services Canada (SSC) should consider designing Data Lake infrastructure around Service-Level Agreements (SLA) to keep Data Lake efforts on track. This includes ensuring that SSC has established clear goals for Data Lakes prior to deployment. </p><br />
<p class="expand mw-collapsible-content">SSC should also consider building an expert special group focussed on advanced analytics and experimental data trend discovery in Data Lakes. While the fundamental assumption behind the Data Lake concept is that everyone accessing a Data Lake is moderately to highly skilled at data manipulation and analysis, the reality is most are not. SSC should consider significant investment in training employees necessary skills, such as Data Science, Artificial Intelligence, Machine Learning, or Data Engineering.</p><br />
<p>SSC should be cognisant that there are significant overinflated expectations revolving around Data Lakes. Inflated expectations lead to vague and ambiguous use cases and increased chances of catastrophic failures. As a Service Provider, SSC must be strict in establishing clear goals for Data Lake provision efforts before deployment. SSC, should be wary of attempts to replace strategy development with infrastructure. A Data Lake can be a technology component that supports a data and analytics strategy, but it cannot replace that strategy.</p><br />
<p class="expand mw-collapsible-content">SSC should be concerned with the provision and running of the infrastructure, the departments themselves are responsible for the data they put in the Data Lake. However, as a Service Provider, SSC should monitor the Data Lake with regards to data governance, data lifecycle for data hygiene, and what is happening in the Data Lake overall. Depending on technology, SSC will need to be very clear on how to monitor activities in the Data Lakes it provides to the GC. </p><br />
<p class="expand mw-collapsible-content">SSC should consider a Data Lake implementation project as a way to introduce or reinvigorate a data management program by positioning data management capabilities as a prerequisite for a<br />
successful Data Lake. Data will need to be qualified before it hits the data lake, this can and should be done in a system of record first. In this way the data can be organizedto fit into the Data Lake implementation.<br />
</p><br />
<p class="expand mw-collapsible-content">SSC should create policies on how data is managed and cleaned in the Data Lake. Automated data governance technologies should be added to support advanced analytics. Standardizing on a specific type of governance tool is an issue which must be resolved. Additionally, planning for effective metadata management, considering metadata discovery, cataloguing and enterprise metadata management applied to Data Lake implementation is vital. Rigorous application of data discipline and data hygiene is needed. To combat this, SSC should use data management tools and create policies on how data is managed and cleaned in the Data Lake. The majority of Data Lake analysts will prefer to work with clean, enriched, and trusted data. However, data quality is relative to the task at hand. Lowquality data may be acceptable for low-impact analysis or distant forecasting, but unacceptable for tactical or high-impact analysis. SSC assessments should take this into account.</p><br />
<p>Design Data Lakes with the elements necessary to deliver reliable analytical results to a variety of data consumers. The goal is to increase cross-business usage in order to deliver advanced analytical insights. Build Data Lakes for specific business units or analytics applications, rather than try to implement some vague notion of a single enterprise Data Lake. However, alternative architectures, like data hubs, are often better fits for sharing data within an organization.</p><br />
<h2>References</h2><br />
<div style="display: none"><br />
<ref>Dennis, A. L. (2018, October 15). Data Lakes 101: An Overview. Retrieved from <i>[https://www.dataversity.net/data-lakes-101-overview/#]</i></ref><br />
<ref>Marvin, R., Marvin, R., & Marvin, R. (2016, August 22). Data Lakes, Explained. Retrieved from <i>[ https://www.pcmag.com/article/347020/data-lakes-explained]</i></ref><br />
<ref>The Data Lake journey. (2014, March 15). Retrieved from <i>[https://hortonworks.com/blog/enterprise-hadoop-journey-data-lake/]</i></ref><br />
<ref>Google File System. (2019, July 14). Retrieved from <i>[https://en.wikipedia.org/wiki/Google_File_System]</i></ref><br />
<ref>Coates, M. (2016, October 02). Data Lake Use Cases and Planning Considerations. Retrieved from <i>[https://www.sqlchick.com/entries/2016/7/31/data-lake-use-cases-and-planning]</i></ref><br />
<ref>Bhalchandra, V. (2018, July 23). Six reasons to think twice about your data lake strategy. Retrieved from <i>[https://dataconomy.com/2018/07/six-reasons-to-think-twice-about-your-data-lake-strategy/]</i></ref><br />
<ref>Data Lake Expectations: Why Data Lakes Fail. (2018, September 20). Retrieved from <i>[https://www.arcadiadata.com/blog/the-top-six-reasons-data-lakes-have-failed-to-live-up-to-expectations/]</i></ref><br />
<ref>Data Lake: AWS Solutions. (n.d.). Retrieved from <i>[https://aws.amazon.com/solutions/data-lake-solution/]</i></ref><br />
</div><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Datalakes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Datalakes&diff=11153Technology Trends/Datalakes2019-07-18T19:54:05Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Datalakes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Lac de Donnés]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[200px]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>July 12, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Datalakes_v0.1_EN_Published.pdf|Datalakes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Datalakes</b> is a central system or repository of data that is stored in its natural/raw format. A datalake acts as a single store for all enterprise data. Data is transformed using machine learning, advanced,analytics, and visualization. Several forms of data can be hosed in a datalake. These include structured data from relational databases, unstructured data, semi-structured data, and binary data. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>In an ever-increasing hyperconnected world, corporations and businesses are struggling to deal with the responsibilities of storage, management and quick availability of raw data. To break these data challenges down further:</p><br />
<ul><br />
<li>Data comes in many different structures.<br />
<ul><br />
<li>Unstructured</li><br />
<li>Semi-Sturctured</li><br />
<li>Structured</li><br />
</ul></li><br />
<li>Data comes from many disparate sources.<br />
<ul><br />
<li>Enterprise Applications</li><br />
<li>Raw Files</li><br />
<li>Operation and Security Logs</li><br />
<li>Financial Transactions</li><br />
<li>Internet of Things (IoT) Devices and Network Sensors</li><br />
<li>Websites</li><br />
<li>Scientific Research</li><br />
</ul></li><br />
<li>Data sources are often geographically distributed to multiple locations<br />
<ul><br />
<li>Datacenters</li><br />
<li>Remote Offices</li><br />
<li>Mobile Devices</li><br />
</ul></li><br />
</ul><br />
<br />
<p>In an effort to resolve these data challenges, a new way of managing data was created which drove data oriented companies to invent a new data storage mechanism called a Data Lake.</p><br />
<p class="expand mw-collapsible-content">Data Lakes are characterized as: </p><br />
<ul class="expand mw-collapsible-content"><br />
<li>Collect Everything<ul><br />
<li>A Data Lake contains all data; raw sources over extended periods of time as well as any processed data.</li><br />
</ul></li><br />
<li>Dive in anywhere<ul><br />
<li>A Data Lake enables users across multiple business units to refine, explore and enrich data on their terms.</li><br />
</ul></li><br />
<li>Flexible Access<ul><br />
<li>o A Data Lake enables multiple data access patterns across a shared infrastructure: batch, interactive, online, search, in-memory and other processing engines.</li><br />
</ul></li><br />
</ul><br />
<br />
<p>Data Lakes are essentially a technology platform for holding data. Their value to the business is only realized when applying data science skills to the lake. </p><br />
<br />
<p>To summarize, usecases for Data Lakes are still being discovered. Cloud providers are making it easier to procure Data Lakes and today Data Lakes are primarily used by Research Institutions, Financial Services, Telecom, Media, Retail, Manufacturing, Healthcare, Pharma, Oi l& Gas and Governments.<br />
</p><br />
<br />
<br />
<br />
<h2>Technology Brief</h2><br />
<p class="expand mw-collapsible-content">The most popular implementation of a Data Lake is through the open source platform called Apache Hadoop. Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. Hadoop was originally created by researchers at Google as a storage method to handle the indexing of websites on the Internet; At that time it was called the Google File System. </p><br />
<p>A Data Lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.”</p><br />
<p class="expand mw-collapsible-content">Data can flow into the Data Lake by either batch processing or real-time processing of streaming data. Additionally, data itself is no longer restrained by initial schema decisions and can be exploited more freely by the enterprise. Rising above this repository is a set of capabilities that allow IT to provide Data and Analytics as a Service (DAaaS), in a supply-demand model. IT takes the role of the data provider (supplier), while business users (data scientists, business analysts) are consumers.</p><br />
<p>The DAaaS model enables users to self-serve their data and analytic needs. Users browse the lake’s data catalog (a Datapedia) to find and select the available data and fill a metaphorical “shopping cart” (effectively an analytics sandbox) with data to work with. Once access is provisioned, users can use the analytics tools of their choice to develop models and gain insights. Subsequently, users can publish analytical models or push refined or transformed data back into the Data Lake to share with the larger community.</p><br />
<p class="expand mw-collapsible-content">Although provisioning an analytic sandbox is a primary use, the Data Lake also has other applications. For example, the Data Lake can also be used to ingest raw data, curate the data, and apply Export-Transform-Load (ETL). This data can then be loaded to an Enterprise Data Warehouse. To take advantage of the flexibility provided by the Data Lake, organizations need to customize and configure the Data Lake to their specific requirements and domains.</p><br />
<h2>Industry Usage</h2><br />
<p>There are a variety of ways Data Lakes are being used in the industry:</p><br />
<ul><br />
<li><p><b>Ingestion of semi-structured and unstructured data sources (aka big data)</b>such as equipment readings, telemetry data, logs, streaming data, and so forth. A Data Lake is a great solution for storing IoT (Internet of Things) type of data which has traditionally been more difficult to store, and can support near real-time analysis. Optionally, you can also add structured data (i.e., extracted from a relational data source) to a Data Lake if your objective is a single repository of all data to be available via the lake.</p></li><br />
<li><p><b>Experimental analysis </b>of data before its value or purpose has been fully defined. Agility is important for every business these days, so a Data Lake can play an important role in "proof of value" type of situations because of the "ELT" approach discussed above.</p></li><br />
<li><p><b>Advanced analytics support. </b>A Data Lake is useful for data scientists and analysts to provision and experiment with data.</p></li><br />
<li><p><b>Archival and historical data storage. </b>Sometimes data is used infrequently, but does need to be available for analysis. A Data Lake strategy can be very valuable to support an active archive strategy.</p></li><br />
<li><p><b>Distributed processing </b>capabilities associated with a logical data warehouse.</p></li><br />
</ul><br />
<p class="expand mw-collapsible-content"><b>How TD Bank Made Its Data Lake More Usa</b></p><br />
<p class="expand mw-collapsible-content">[[https://www.datanami.com/2017/10/03/td-bank-made-data-lake-usable]]<br>Toronto-Dominion Bank (TD Bank) is one of the largest banks in North America, with 85,000 employees, more than 2,400 locations between Canada and the United States, and assets nearing $1 trillion. In 2014, the company decided to standardize how it warehouses data for various business intelligence and regulatory reporting functions. The company purchased a Hadoop distribution and set off to build a large cluster that could function as a centralized lake to store data originating from a variety of departments.</p><br />
<br />
<h2>Canadian Government Use</h2><br />
<p>In 2019, the Treasury Board of Canada Secretariat (TBS), partnered with Shared Services Canada and other departments, to identify a business lead to develop a Data Lake (a repository of raw data) service strategy so that the GC can take advantage of big data and market innovation to foster better analytics and promote horizontal data-sharing<ref>Treasury Board of Canada Secretariat. (March 29th, 2019). Digital Operations Strategic Plan: 2018-2022. Government of Canada. Treasury Board of Canada Secretariat. Retrieved 26-May-2019 from: <i>[https://www.canada.ca/en/government/system/digital-government/digital-operations-strategic-plan-2018-2022.html] </i></ref>. </p><br />
<p class="expand mw-collapsible-content">Big data is the technology that stores and processes data and information in datasets that are so large or complex that traditional data processing applications can’t analyze them. Big data can make available almost limitless amounts of information, improving data-driven decision-making and expanding open data initiatives. Business intelligence involves creating, aggregating, analyzing and visualizing data to inform and facilitate business management and strategy. TBS, working with departments, will lead the development of requirements for an enterprise analytics platform<ref>Ibid.<i></i></ref>.</p><br />
<p>Data Lake development in the GC is a more recent initiative. This is mainly due to the GC focussing resources on the implementation of cloud initiatives. However, there are some GC departments engaged in developing Data Lake environments in tandem to cloud initiatives.</p><br />
<p class="expand mw-collapsible-content">Notably, the Employment and Social Development Canada (ESDC) is preparing the installment of multiple Data Lakes in order to enable a Data Lake Ecosystem and Data Analytics and Machine Learning toolset. This will enable ESDC to share information horizontally both effectively and safely, while enabling a wide variety of data analytics capabilities. ESDC aims to maintain current data and analytics capabilities up-to-date while exploring new ones to mitigate gaps and continuously evolve our services to meet client’s needs<ref>Brisson, Yannick, and Craig, Sheila. (November, 2018). ESDC Data Lake – Implementation Strategy and Roadmap Update. Government of Canada. Employment and Social Development Canada – Data and Analytics Services. Presentation. Last Modified on 2019-04-26 15:45. Retrieved 07-May-2019 from GCDocs<i>[https://gcdocs.gc.ca/ssc-spc/llisapi.dll?func=ll&objaction=overview&objid=36624914 ]</i></ref>. </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="expand mw-collapsible-content">There are three common value propositions for pursuing Data Lakes. 1) It can provide an easy and accessible way to obtain data faster; 2) It can create a singular inflow point of data to help connect and merge information silos in an organization; and 3) It can provide an experimental environment for experienced data scientists to enable new analytical insights.</p><br />
<p class="inline-spacer"> </p><br />
<p class="inline">Data Lakes can provide data to consumers more quickly by offering data in a more raw and easily accessible form. Data is stored in its native form with little to no processing, it is optimized to store vast amounts of data in their native formats. By allowing the data to remain in its native format, a much timelier stream of data is available for unlimited queries and analysis. A Data Lake can help data consumers bypass strict data retrieval and data structured applications such as a data warehouse and/or data mart. This has the effect of improving a business’ data flexibility.</p><p class="expand inline mw-collapsible-content">Some companies have in fact used Data Lakes to replace existing warehousing environments where implementing a new data warehouse is more cost prohibitive. A Data Lake can contain unrefined data, this is helpful when either a business data structure is unknown, or when a data consumer requires access to the data quickly. </p><br />
<p class="inline-spacer"> </p><br />
<p class="inline">A Data Lake is not a single source of truth. A Data Lake is a central location in which data converges from all data sources and is stored, regardless of the data formatting. </p><p class="expand inline mw-collapsible-content">As a singular point for the inflow of data, sections of a business can pool their information together in the Data Lake and increase the sharing of information with other parts of the organization. In this way everyone in the organization has access to the data. A Data Lake can increase the horizontal data sharing within an organization by creating this singular data inflow point. Using a variety of storage and processing tools analysts can extract data value quickly in order to inform key business decisions.</p><br />
<p class="inline-spacer"> </p><br />
<p class="expand inline mw-collapsible-content">A Data Lake is optimized for exploration and provides an experimental environment for experienced data scientists to uncover new insights from data. Analysts can overlay context on the data to extract value. All organizations want to increase analytics and operational agility.</p><p class="inline">The Data Lake architectural approach can store large volumes of data, this can be a way in which cross-cutting teams can pool their data in a central location and by complementing their systems of record with systems of insight. </p><p class="expand inline mw-collapsible-content">Data Lakes present the most potential benefits for experienced and competant data scientists. </p><p class="inline">Having structured, unstructured and semistructured data, usually in the same data set, can contain business, predictive, and prescriptive insights previously not possible from a structured platform as observed in data warehouses and data marts.</p><br />
<br />
<h4>Challenges</h4><br />
<p>Although Data Lake technology has many benefits for organizations dealing with big data it has its own challenges. For example:</p><br />
<p><b>Data Governance and Semantic Issues</b></p><br />
<p class="expand inline mw-collapsible-content">The biggest challenge for Data Lakes is to resolve assorted data governance requirements in a single centralized data platform. Data Lakes fail mostly when they lack governance, self-disciplined users, and a rational data flow.</p><p class="inline">Often, Data Lake implementations are focused on storing data instead of managing the data. Data Lakes are not optimized for semantic enforcement or consistency. They are made for semantic flexibility, to allow anyone to provide context to data if they have the skills to do so. </p><br />
<p>Putting data in the same place does not remove it’s ambiguity or meaning. Data Lakes provide unconstrained, “no compromises” storage model environment without the data governance assurances common to data warehouses or data marts. Proper meta data is essential for a Data Lake, without appropriate meta data the Data Lake will not work as intended. It is beneficial to think of meta data as the fish finder in the Data Lake.</p><br />
<p><b class="expand mw-collapsible-content">Lack of Quality and Trust in Data</b></p><br />
<p class="expand mw-collapsible-content">Data quality and trust in the data is a perennial issue for many organizations. Although data discovery tools can apply Machine Learning across related datasets from multiple data sources to identify anomalies (incorrect values, missing values, duplicates and outdated data), quality and trustworthiness of data continue to be an issue for Data Lakes who can easily become data dumping grounds. Some data is more accurate than others. This can present a real problem for anyone using multiple data sets and making decisions based upon analysis conducted with data of varying degrees of quality.</p><br />
<p><b>Data Swamps, Performance, and Flexibility Challenges</b></p><br />
<p class="expand inline mw-collapsible-content">Data stored in Data Lakes can sometimes become muddy when good data is mixed with bad data. Data Lake infrastructure is meant to store and process large amounts of data, usually in massive data files. </p><p class="inline">A Data Lake is not optimized for a high number of users or diverse and simultaneous workloads due to intensive query tasks. This can result in performance degradation and failures are common when running extractions, transformations, and loading tasks all at the same time. On-premises Data Lakes face other performance challenges in that they have a static configuration. </p><br />
<p><b class="expand mw-collapsible-content">Data Hoarding and Storage Capacity</b></p><br />
<p class="expand mw-collapsible-content">Data stored in Data Lakes may actually never be used in production and stay unused indefinitely in the Data Lake. By storing massive amounts of historical data, the infinite Data Lake may skew analysis with data that is no longer relevant to the priorities of the business. In keeping the historical data the metadata describing it must be understood as well. This decreases the performance of the Data Lake by increasing the overall workload of employees to clean the datasets no longer in use for analysis.</p><br />
<p class="expand mw-collapsible-content">Storing increasingly massive amounts of data for an unlimited time will also lead to scalability and cost challenges. Scalability challenges are less of a risk in public cloud environments, but cost remains a factor. On-premises Data Lakes are more susceptible to cost challenges. This is because their cluster nodes require all three dimensions of computing (storage, memory and processing). Organizations of all kinds generate massive amounts of data (including meta data) and it is increasing exponentially.</p><br />
<p>The storage capacity of all this data (and future data) will be an ongoing challenge and one that will require constant management. While Data Lakes can and will be stored on the cloud, SSC as cloud broker for the GC will need to provide the appropriate infrastructure and scalability to clients.</p><br />
<p><b class="expand mw-collapsible-content">Advanced Users Required</b></p><br />
<p class="expand mw-collapsible-content">Data Lakes are not a platform to be explored by everyone. Data Lakes present an unrefined view of data that usually only the most highly skilled analysts are able to explore and engage in data refinement independent of any other formal system-of-record such as a data warehouse. </p><br />
<p class="expand mw-collapsible-content">Not just anyone in an organization is data-literate enough to derive value from large amounts of raw or uncurated data. The reality is only a handful of staff are skilled enough to navigate a Data Lake. Since Data Lakes store raw data their business value is entirely determined by the skills of Data Lake users. These skills are often lacking in an organization.</p><br />
<p><b>Data Security</b></p><br />
<p class="inline"></p>Data in a Data Lake lacks standard security protection with a relational database management system or an enterprise database. In practice, this means that the data is unencrypted and lacks access control.<p class="expand inline mw-collapsible-content">. Security is not just a binary solution. We have varying degrees of security (unclassified, secret, top secret, etc.) and all of which require different approaches. This will inevitably present challenges with the successful use of data from Data Lakes.To combat this, organizations will have to embrace a new security framework to be compatable with Data Lakes and Data Scientists.</p><br />
<h4>Considerations</h4><br />
<p class="expand mw-collapsible-content">Shared Services Canada (SSC) has an excellent opportunity to capitalize on its mandate of providing data storage service to GC’s other departments. SSC, as the GC’s Service Provider, could potentially a centralized GC Data Lake and allow GC Data Scientists access to this central data using a single unified Data Lake interface. However, this is a project which should be implemented after cloud has been adopted and enterprise data centers have been migrated to in order to provide adequate infrastructure and scaling.</p><br />
<p class="inline">Data Lakes should not be confused for conventional databases although they both store information. A Data Lake will always underperform when tasked with the jobs of a conventional database. </p><p class="expand inline mw-collapsible-content">To combat this, SSC must create data architectures that define the proper application of Data Lakes. Too often, Data Lakes suffer from lack of foresight on what they're supposed to achieve. </p><p class="inline">Creating a Data Lake becomes the goal rather than achieving a strategic objective. </p><p class="expand inline mw-collapsible-content"></p><br />
<p class="expand mw-collapsible-content">Shared Services Canada (SSC) should consider designing Data Lake infrastructure around Service-Level Agreements (SLA) to keep Data Lake efforts on track. This includes ensuring that SSC has established clear goals for Data Lakes prior to deployment. </p><br />
<p class="expand mw-collapsible-content">SSC should also consider building an expert special group focussed on advanced analytics and experimental data trend discovery in Data Lakes. While the fundamental assumption behind the Data Lake concept is that everyone accessing a Data Lake is moderately to highly skilled at data manipulation and analysis, the reality is most are not. SSC should consider significant investment in training employees necessary skills, such as Data Science, Artificial Intelligence, Machine Learning, or Data Engineering.</p><br />
<p>SSC should be cognisant that there are significant overinflated expectations revolving around Data Lakes. Inflated expectations lead to vague and ambiguous use cases and increased chances of catastrophic failures. As a Service Provider, SSC must be strict in establishing clear goals for Data Lake provision efforts before deployment. SSC, should be wary of attempts to replace strategy development with infrastructure. A Data Lake can be a technology component that supports a data and analytics strategy, but it cannot replace that strategy.</p><br />
<p class="expand mw-collapsible-content">SSC should be concerned with the provision and running of the infrastructure, the departments themselves are responsible for the data they put in the Data Lake. However, as a Service Provider, SSC should monitor the Data Lake with regards to data governance, data lifecycle for data hygiene, and what is happening in the Data Lake overall. Depending on technology, SSC will need to be very clear on how to monitor activities in the Data Lakes it provides to the GC. </p><br />
<p class="expand mw-collapsible-content">SSC should consider a Data Lake implementation project as a way to introduce or reinvigorate a data management program by positioning data management capabilities as a prerequisite for a<br />
successful Data Lake. Data will need to be qualified before it hits the data lake, this can and should be done in a system of record first. In this way the data can be organizedto fit into the Data Lake implementation.<br />
</p><br />
<p class="expand mw-collapsible-content">SSC should create policies on how data is managed and cleaned in the Data Lake. Automated data governance technologies should be added to support advanced analytics. Standardizing on a specific type of governance tool is an issue which must be resolved. Additionally, planning for effective metadata management, considering metadata discovery, cataloguing and enterprise metadata management applied to Data Lake implementation is vital. Rigorous application of data discipline and data hygiene is needed. To combat this, SSC should use data management tools and create policies on how data is managed and cleaned in the Data Lake. The majority of Data Lake analysts will prefer to work with clean, enriched, and trusted data. However, data quality is relative to the task at hand. Lowquality data may be acceptable for low-impact analysis or distant forecasting, but unacceptable for tactical or high-impact analysis. SSC assessments should take this into account.</p><br />
<p>Design Data Lakes with the elements necessary to deliver reliable analytical results to a variety of data consumers. The goal is to increase cross-business usage in order to deliver advanced analytical insights. Build Data Lakes for specific business units or analytics applications, rather than try to implement some vague notion of a single enterprise Data Lake. However, alternative architectures, like data hubs, are often better fits for sharing data within an organization.</p><br />
<h2>References</h2><br />
<div style="display: none"><br />
<ref>Dennis, A. L. (2018, October 15). Data Lakes 101: An Overview. Retrieved from <i>[https://www.dataversity.net/data-lakes-101-overview/#]</i></ref><br />
<ref>Marvin, R., Marvin, R., & Marvin, R. (2016, August 22). Data Lakes, Explained. Retrieved from <i>[ https://www.pcmag.com/article/347020/data-lakes-explained]</i></ref><br />
<ref>The Data Lake journey. (2014, March 15). Retrieved from <i>[https://hortonworks.com/blog/enterprise-hadoop-journey-data-lake/]</i></ref><br />
<ref>Google File System. (2019, July 14). Retrieved from <i>[https://en.wikipedia.org/wiki/Google_File_System]</i></ref><br />
<ref>Coates, M. (2016, October 02). Data Lake Use Cases and Planning Considerations. Retrieved from <i>[https://www.sqlchick.com/entries/2016/7/31/data-lake-use-cases-and-planning]</i></ref><br />
<ref>Bhalchandra, V. (2018, July 23). Six reasons to think twice about your data lake strategy. Retrieved from <i>[https://dataconomy.com/2018/07/six-reasons-to-think-twice-about-your-data-lake-strategy/]</i></ref><br />
<ref>Data Lake Expectations: Why Data Lakes Fail. (2018, September 20). Retrieved from <i>[https://www.arcadiadata.com/blog/the-top-six-reasons-data-lakes-have-failed-to-live-up-to-expectations/]</i></ref><br />
<ref>Data Lake: AWS Solutions. (n.d.). Retrieved from <i>[https://aws.amazon.com/solutions/data-lake-solution/]</i></ref><br />
</div><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=File:EN_-_Datalakes_v0.1_EN_Published.pdf&diff=11152File:EN - Datalakes v0.1 EN Published.pdf2019-07-18T19:51:45Z<p>Kpere060: </p>
<hr />
<div></div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Datalakes&diff=11151Technology Trends/Datalakes2019-07-18T19:43:26Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Datalakes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Lac de Donnés]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[200px]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>July 12, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Datalakes_v0.1_EN_Published.pdf|Datalakes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Datalakes</b> is a central system or repository of data that is stored in its natural/raw format. A datalake acts as a single store for all enterprise data. Data is transformed using machine learning, advanced,analytics, and visualization. Several forms of data can be hosed in a datalake. These include structured data from relational databases, unstructured data, semi-structured data, and binary data. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>In an ever-increasing hyperconnected world, corporations and businesses are struggling to deal with the responsibilities of storage, management and quick availability of raw data. To break these data challenges down further:</p><br />
<ul><br />
<li>Data comes in many different structures.<br />
<ul><br />
<li>Unstructured</li><br />
<li>Semi-Sturctured</li><br />
<li>Structured</li><br />
</ul></li><br />
<li>Data comes from many disparate sources.<br />
<ul><br />
<li>Enterprise Applications</li><br />
<li>Raw Files</li><br />
<li>Operation and Security Logs</li><br />
<li>Financial Transactions</li><br />
<li>Internet of Things (IoT) Devices and Network Sensors</li><br />
<li>Websites</li><br />
<li>Scientific Research</li><br />
</ul></li><br />
<li>Data sources are often geographically distributed to multiple locations<br />
<ul><br />
<li>Datacenters</li><br />
<li>Remote Offices</li><br />
<li>Mobile Devices</li><br />
</ul></li><br />
</ul><br />
<br />
<p>In an effort to resolve these data challenges, a new way of managing data was created which drove data oriented companies to invent a new data storage mechanism called a Data Lake.</p><br />
<p class="expand mw-collapsible-content">Data Lakes are characterized as: </p><br />
<ul class="expand mw-collapsible-content"><br />
<li>Collect Everything<ul><br />
<li>A Data Lake contains all data; raw sources over extended periods of time as well as any processed data.</li><br />
</ul></li><br />
<li>Dive in anywhere<ul><br />
<li>A Data Lake enables users across multiple business units to refine, explore and enrich data on their terms.</li><br />
</ul></li><br />
<li>Flexible Access<ul><br />
<li>o A Data Lake enables multiple data access patterns across a shared infrastructure: batch, interactive, online, search, in-memory and other processing engines.</li><br />
</ul></li><br />
</ul><br />
<br />
<p>Data Lakes are essentially a technology platform for holding data. Their value to the business is only realized when applying data science skills to the lake. </p><br />
<br />
<p>To summarize, usecases for Data Lakes are still being discovered. Cloud providers are making it easier to procure Data Lakes and today Data Lakes are primarily used by Research Institutions, Financial Services, Telecom, Media, Retail, Manufacturing, Healthcare, Pharma, Oi l& Gas and Governments.<br />
</p><br />
<br />
<br />
<br />
<h2>Technology Brief</h2><br />
<p class="expand mw-collapsible-content">The most popular implementation of a Data Lake is through the open source platform called Apache Hadoop. Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. Hadoop was originally created by researchers at Google as a storage method to handle the indexing of websites on the Internet; At that time it was called the Google File System. </p><br />
<p>A Data Lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.”</p><br />
<p class="expand mw-collapsible-content">Data can flow into the Data Lake by either batch processing or real-time processing of streaming data. Additionally, data itself is no longer restrained by initial schema decisions and can be exploited more freely by the enterprise. Rising above this repository is a set of capabilities that allow IT to provide Data and Analytics as a Service (DAaaS), in a supply-demand model. IT takes the role of the data provider (supplier), while business users (data scientists, business analysts) are consumers.</p><br />
<p>The DAaaS model enables users to self-serve their data and analytic needs. Users browse the lake’s data catalog (a Datapedia) to find and select the available data and fill a metaphorical “shopping cart” (effectively an analytics sandbox) with data to work with. Once access is provisioned, users can use the analytics tools of their choice to develop models and gain insights. Subsequently, users can publish analytical models or push refined or transformed data back into the Data Lake to share with the larger community.</p><br />
<p class="expand mw-collapsible-content">Although provisioning an analytic sandbox is a primary use, the Data Lake also has other applications. For example, the Data Lake can also be used to ingest raw data, curate the data, and apply Export-Transform-Load (ETL). This data can then be loaded to an Enterprise Data Warehouse. To take advantage of the flexibility provided by the Data Lake, organizations need to customize and configure the Data Lake to their specific requirements and domains.</p><br />
<h2>Industry Usage</h2><br />
<p>There are a variety of ways Data Lakes are being used in the industry:</p><br />
<ul><br />
<li><p><b>Ingestion of semi-structured and unstructured data sources (aka big data)</b>such as equipment readings, telemetry data, logs, streaming data, and so forth. A Data Lake is a great solution for storing IoT (Internet of Things) type of data which has traditionally been more difficult to store, and can support near real-time analysis. Optionally, you can also add structured data (i.e., extracted from a relational data source) to a Data Lake if your objective is a single repository of all data to be available via the lake.</p></li><br />
<li><p><b>Experimental analysis </b>of data before its value or purpose has been fully defined. Agility is important for every business these days, so a Data Lake can play an important role in "proof of value" type of situations because of the "ELT" approach discussed above.</p></li><br />
<li><p><b>Advanced analytics support. </b>A Data Lake is useful for data scientists and analysts to provision and experiment with data.</p></li><br />
<li><p><b>Archival and historical data storage. </b>Sometimes data is used infrequently, but does need to be available for analysis. A Data Lake strategy can be very valuable to support an active archive strategy.</p></li><br />
<li><p><b>Distributed processing </b>capabilities associated with a logical data warehouse.</p></li><br />
</ul><br />
<p class="expand mw-collapsible-content"><b>How TD Bank Made Its Data Lake More Usa</b></p><br />
<p class="expand mw-collapsible-content">[[https://www.datanami.com/2017/10/03/td-bank-made-data-lake-usable]]<br>Toronto-Dominion Bank (TD Bank) is one of the largest banks in North America, with 85,000 employees, more than 2,400 locations between Canada and the United States, and assets nearing $1 trillion. In 2014, the company decided to standardize how it warehouses data for various business intelligence and regulatory reporting functions. The company purchased a Hadoop distribution and set off to build a large cluster that could function as a centralized lake to store data originating from a variety of departments.</p><br />
<br />
<h2>Canadian Government Use</h2><br />
<p>In 2019, the Treasury Board of Canada Secretariat (TBS), partnered with Shared Services Canada and other departments, to identify a business lead to develop a Data Lake (a repository of raw data) service strategy so that the GC can take advantage of big data and market innovation to foster better analytics and promote horizontal data-sharing<ref>Treasury Board of Canada Secretariat. (March 29th, 2019). Digital Operations Strategic Plan: 2018-2022. Government of Canada. Treasury Board of Canada Secretariat. Retrieved 26-May-2019 from: <i>[https://www.canada.ca/en/government/system/digital-government/digital-operations-strategic-plan-2018-2022.html] </i></ref>. </p><br />
<p class="expand mw-collapsible-content">Big data is the technology that stores and processes data and information in datasets that are so large or complex that traditional data processing applications can’t analyze them. Big data can make available almost limitless amounts of information, improving data-driven decision-making and expanding open data initiatives. Business intelligence involves creating, aggregating, analyzing and visualizing data to inform and facilitate business management and strategy. TBS, working with departments, will lead the development of requirements for an enterprise analytics platform<ref>Ibid.<i></i></ref>.</p><br />
<p>Data Lake development in the GC is a more recent initiative. This is mainly due to the GC focussing resources on the implementation of cloud initiatives. However, there are some GC departments engaged in developing Data Lake environments in tandem to cloud initiatives.</p><br />
<p class="expand mw-collapsible-content">Notably, the Employment and Social Development Canada (ESDC) is preparing the installment of multiple Data Lakes in order to enable a Data Lake Ecosystem and Data Analytics and Machine Learning toolset. This will enable ESDC to share information horizontally both effectively and safely, while enabling a wide variety of data analytics capabilities. ESDC aims to maintain current data and analytics capabilities up-to-date while exploring new ones to mitigate gaps and continuously evolve our services to meet client’s needs<ref>Brisson, Yannick, and Craig, Sheila. (November, 2018). ESDC Data Lake – Implementation Strategy and Roadmap Update. Government of Canada. Employment and Social Development Canada – Data and Analytics Services. Presentation. Last Modified on 2019-04-26 15:45. Retrieved 07-May-2019 from GCDocs<i>[https://gcdocs.gc.ca/ssc-spc/llisapi.dll?func=ll&objaction=overview&objid=36624914 ]</i></ref>. </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="expand mw-collapsible-content">There are three common value propositions for pursuing Data Lakes. 1) It can provide an easy and accessible way to obtain data faster; 2) It can create a singular inflow point of data to help connect and merge information silos in an organization; and 3) It can provide an experimental environment for experienced data scientists to enable new analytical insights.</p><br />
<p> </p><br />
<p class="inline">Data Lakes can provide data to consumers more quickly by offering data in a more raw and easily accessible form. Data is stored in its native form with little to no processing, it is optimized to store vast amounts of data in their native formats. By allowing the data to remain in its native format, a much timelier stream of data is available for unlimited queries and analysis. A Data Lake can help data consumers bypass strict data retrieval and data structured applications such as a data warehouse and/or data mart. This has the effect of improving a business’ data flexibility.</p><p class="expand inline mw-collapsible-content">Some companies have in fact used Data Lakes to replace existing warehousing environments where implementing a new data warehouse is more cost prohibitive. A Data Lake can contain unrefined data, this is helpful when either a business data structure is unknown, or when a data consumer requires access to the data quickly. </p><br />
<p> </p><br />
<p class="inline">A Data Lake is not a single source of truth. A Data Lake is a central location in which data converges from all data sources and is stored, regardless of the data formatting. </p><p class="expand inline mw-collapsible-content">As a singular point for the inflow of data, sections of a business can pool their information together in the Data Lake and increase the sharing of information with other parts of the organization. In this way everyone in the organization has access to the data. A Data Lake can increase the horizontal data sharing within an organization by creating this singular data inflow point. Using a variety of storage and processing tools analysts can extract data value quickly in order to inform key business decisions.</p><br />
<p> </p><br />
<p class="expand inline mw-collapsible-content">A Data Lake is optimized for exploration and provides an experimental environment for experienced data scientists to uncover new insights from data. Analysts can overlay context on the data to extract value. All organizations want to increase analytics and operational agility.</p><p class="inline">The Data Lake architectural approach can store large volumes of data, this can be a way in which cross-cutting teams can pool their data in a central location and by complementing their systems of record with systems of insight. </p><p class="expand inline mw-collapsible-content">Data Lakes present the most potential benefits for experienced and competant data scientists. </p><p class="inline">Having structured, unstructured and semistructured data, usually in the same data set, can contain business, predictive, and prescriptive insights previously not possible from a structured platform as observed in data warehouses and data marts.</p><br />
<br />
<h4>Challenges</h4><br />
<p>Although Data Lake technology has many benefits for organizations dealing with big data it has its own challenges. For example:</p><br />
<p><b>Data Governance and Semantic Issues</b></p><br />
<p class="expand inline mw-collapsible-content">The biggest challenge for Data Lakes is to resolve assorted data governance requirements in a single centralized data platform. Data Lakes fail mostly when they lack governance, self-disciplined users, and a rational data flow.</p><p class="inline">Often, Data Lake implementations are focused on storing data instead of managing the data. Data Lakes are not optimized for semantic enforcement or consistency. They are made for semantic flexibility, to allow anyone to provide context to data if they have the skills to do so. </p><br />
<p>Putting data in the same place does not remove it’s ambiguity or meaning. Data Lakes provide unconstrained, “no compromises” storage model environment without the data governance assurances common to data warehouses or data marts. Proper meta data is essential for a Data Lake, without appropriate meta data the Data Lake will not work as intended. It is beneficial to think of meta data as the fish finder in the Data Lake.</p><br />
<p><b class="expand mw-collapsible-content">Lack of Quality and Trust in Data</b></p><br />
<p class="expand mw-collapsible-content">Data quality and trust in the data is a perennial issue for many organizations. Although data discovery tools can apply Machine Learning across related datasets from multiple data sources to identify anomalies (incorrect values, missing values, duplicates and outdated data), quality and trustworthiness of data continue to be an issue for Data Lakes who can easily become data dumping grounds. Some data is more accurate than others. This can present a real problem for anyone using multiple data sets and making decisions based upon analysis conducted with data of varying degrees of quality.</p><br />
<p><b>Data Swamps, Performance, and Flexibility Challenges</b></p><br />
<p class="expand inline mw-collapsible-content">Data stored in Data Lakes can sometimes become muddy when good data is mixed with bad data. Data Lake infrastructure is meant to store and process large amounts of data, usually in massive data files. </p><p class="inline">A Data Lake is not optimized for a high number of users or diverse and simultaneous workloads due to intensive query tasks. This can result in performance degradation and failures are common when running extractions, transformations, and loading tasks all at the same time. On-premises Data Lakes face other performance challenges in that they have a static configuration. </p><br />
<p><b class="expand mw-collapsible-content">Data Hoarding and Storage Capacity</b></p><br />
<p class="expand mw-collapsible-content">Data stored in Data Lakes may actually never be used in production and stay unused indefinitely in the Data Lake. By storing massive amounts of historical data, the infinite Data Lake may skew analysis with data that is no longer relevant to the priorities of the business. In keeping the historical data the metadata describing it must be understood as well. This decreases the performance of the Data Lake by increasing the overall workload of employees to clean the datasets no longer in use for analysis.</p><br />
<p class="expand mw-collapsible-content">Storing increasingly massive amounts of data for an unlimited time will also lead to scalability and cost challenges. Scalability challenges are less of a risk in public cloud environments, but cost remains a factor. On-premises Data Lakes are more susceptible to cost challenges. This is because their cluster nodes require all three dimensions of computing (storage, memory and processing). Organizations of all kinds generate massive amounts of data (including meta data) and it is increasing exponentially.</p><br />
<p>The storage capacity of all this data (and future data) will be an ongoing challenge and one that will require constant management. While Data Lakes can and will be stored on the cloud, SSC as cloud broker for the GC will need to provide the appropriate infrastructure and scalability to clients.</p><br />
<p><b class="expand mw-collapsible-content">Advanced Users Required</b></p><br />
<p class="expand mw-collapsible-content">Data Lakes are not a platform to be explored by everyone. Data Lakes present an unrefined view of data that usually only the most highly skilled analysts are able to explore and engage in data refinement independent of any other formal system-of-record such as a data warehouse. </p><br />
<p class="expand mw-collapsible-content">Not just anyone in an organization is data-literate enough to derive value from large amounts of raw or uncurated data. The reality is only a handful of staff are skilled enough to navigate a Data Lake. Since Data Lakes store raw data their business value is entirely determined by the skills of Data Lake users. These skills are often lacking in an organization.</p><br />
<p><b>Data Security</b></p><br />
<p class="inline"></p>Data in a Data Lake lacks standard security protection with a relational database management system or an enterprise database. In practice, this means that the data is unencrypted and lacks access control.<p class="expand inline mw-collapsible-content">. Security is not just a binary solution. We have varying degrees of security (unclassified, secret, top secret, etc.) and all of which require different approaches. This will inevitably present challenges with the successful use of data from Data Lakes.To combat this, organizations will have to embrace a new security framework to be compatable with Data Lakes and Data Scientists.</p><br />
<h4>Considerations</h4><br />
<p class="expand mw-collapsible-content">Shared Services Canada (SSC) has an excellent opportunity to capitalize on its mandate of providing data storage service to GC’s other departments. SSC, as the GC’s Service Provider, could potentially a centralized GC Data Lake and allow GC Data Scientists access to this central data using a single unified Data Lake interface. However, this is a project which should be implemented after cloud has been adopted and enterprise data centers have been migrated to in order to provide adequate infrastructure and scaling.</p><br />
<p class="inline">Data Lakes should not be confused for conventional databases although they both store information. A Data Lake will always underperform when tasked with the jobs of a conventional database. </p><p class="expand inline mw-collapsible-content">To combat this, SSC must create data architectures that define the proper application of Data Lakes. Too often, Data Lakes suffer from lack of foresight on what they're supposed to achieve. </p><p class="inline">Creating a Data Lake becomes the goal rather than achieving a strategic objective. </p><p class="expand inline mw-collapsible-content"></p><br />
<p class="expand mw-collapsible-content">Shared Services Canada (SSC) should consider designing Data Lake infrastructure around Service-Level Agreements (SLA) to keep Data Lake efforts on track. This includes ensuring that SSC has established clear goals for Data Lakes prior to deployment. </p><br />
<p class="expand mw-collapsible-content">SSC should also consider building an expert special group focussed on advanced analytics and experimental data trend discovery in Data Lakes. While the fundamental assumption behind the Data Lake concept is that everyone accessing a Data Lake is moderately to highly skilled at data manipulation and analysis, the reality is most are not. SSC should consider significant investment in training employees necessary skills, such as Data Science, Artificial Intelligence, Machine Learning, or Data Engineering.</p><br />
<p>SSC should be cognisant that there are significant overinflated expectations revolving around Data Lakes. Inflated expectations lead to vague and ambiguous use cases and increased chances of catastrophic failures. As a Service Provider, SSC must be strict in establishing clear goals for Data Lake provision efforts before deployment. SSC, should be wary of attempts to replace strategy development with infrastructure. A Data Lake can be a technology component that supports a data and analytics strategy, but it cannot replace that strategy.</p><br />
<p class="expand mw-collapsible-content">SSC should be concerned with the provision and running of the infrastructure, the departments themselves are responsible for the data they put in the Data Lake. However, as a Service Provider, SSC should monitor the Data Lake with regards to data governance, data lifecycle for data hygiene, and what is happening in the Data Lake overall. Depending on technology, SSC will need to be very clear on how to monitor activities in the Data Lakes it provides to the GC. </p><br />
<p class="expand mw-collapsible-content">SSC should consider a Data Lake implementation project as a way to introduce or reinvigorate a data management program by positioning data management capabilities as a prerequisite for a<br />
successful Data Lake. Data will need to be qualified before it hits the data lake, this can and should be done in a system of record first. In this way the data can be organizedto fit into the Data Lake implementation.<br />
</p><br />
<p class="expand mw-collapsible-content">SSC should create policies on how data is managed and cleaned in the Data Lake. Automated data governance technologies should be added to support advanced analytics. Standardizing on a specific type of governance tool is an issue which must be resolved. Additionally, planning for effective metadata management, considering metadata discovery, cataloguing and enterprise metadata management applied to Data Lake implementation is vital. Rigorous application of data discipline and data hygiene is needed. To combat this, SSC should use data management tools and create policies on how data is managed and cleaned in the Data Lake. The majority of Data Lake analysts will prefer to work with clean, enriched, and trusted data. However, data quality is relative to the task at hand. Lowquality data may be acceptable for low-impact analysis or distant forecasting, but unacceptable for tactical or high-impact analysis. SSC assessments should take this into account.</p><br />
<p>Design Data Lakes with the elements necessary to deliver reliable analytical results to a variety of data consumers. The goal is to increase cross-business usage in order to deliver advanced analytical insights. Build Data Lakes for specific business units or analytics applications, rather than try to implement some vague notion of a single enterprise Data Lake. However, alternative architectures, like data hubs, are often better fits for sharing data within an organization.</p><br />
<h2>References</h2><br />
<div style="display: none"><br />
<ref>Dennis, A. L. (2018, October 15). Data Lakes 101: An Overview. Retrieved from <i>[https://www.dataversity.net/data-lakes-101-overview/#]</i></ref><br />
<ref>Marvin, R., Marvin, R., & Marvin, R. (2016, August 22). Data Lakes, Explained. Retrieved from <i>[ https://www.pcmag.com/article/347020/data-lakes-explained]</i></ref><br />
<ref>The Data Lake journey. (2014, March 15). Retrieved from <i>[https://hortonworks.com/blog/enterprise-hadoop-journey-data-lake/]</i></ref><br />
<ref>Google File System. (2019, July 14). Retrieved from <i>[https://en.wikipedia.org/wiki/Google_File_System]</i></ref><br />
<ref>Coates, M. (2016, October 02). Data Lake Use Cases and Planning Considerations. Retrieved from <i>[https://www.sqlchick.com/entries/2016/7/31/data-lake-use-cases-and-planning]</i></ref><br />
<ref>Bhalchandra, V. (2018, July 23). Six reasons to think twice about your data lake strategy. Retrieved from <i>[https://dataconomy.com/2018/07/six-reasons-to-think-twice-about-your-data-lake-strategy/]</i></ref><br />
<ref>Data Lake Expectations: Why Data Lakes Fail. (2018, September 20). Retrieved from <i>[https://www.arcadiadata.com/blog/the-top-six-reasons-data-lakes-have-failed-to-live-up-to-expectations/]</i></ref><br />
<ref>Data Lake: AWS Solutions. (n.d.). Retrieved from <i>[https://aws.amazon.com/solutions/data-lake-solution/]</i></ref><br />
</div><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Datalakes&diff=11150Technology Trends/Datalakes2019-07-18T19:39:30Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Datalakes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Lac de Donnés]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[200px]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>July 12, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Datalakes_v0.1_EN_Published.pdf|Datalakes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Datalakes</b> is a central system or repository of data that is stored in its natural/raw format. A datalake acts as a single store for all enterprise data. Data is transformed using machine learning, advanced,analytics, and visualization. Several forms of data can be hosed in a datalake. These include structured data from relational databases, unstructured data, semi-structured data, and binary data. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>In an ever-increasing hyperconnected world, corporations and businesses are struggling to deal with the responsibilities of storage, management and quick availability of raw data. To break these data challenges down further:</p><br />
<ul><br />
<li>Data comes in many different structures.<br />
<ul><br />
<li>Unstructured</li><br />
<li>Semi-Sturctured</li><br />
<li>Structured</li><br />
</ul></li><br />
<li>Data comes from many disparate sources.<br />
<ul><br />
<li>Enterprise Applications</li><br />
<li>Raw Files</li><br />
<li>Operation and Security Logs</li><br />
<li>Financial Transactions</li><br />
<li>Internet of Things (IoT) Devices and Network Sensors</li><br />
<li>Websites</li><br />
<li>Scientific Research</li><br />
</ul></li><br />
<li>Data sources are often geographically distributed to multiple locations<br />
<ul><br />
<li>Datacenters</li><br />
<li>Remote Offices</li><br />
<li>Mobile Devices</li><br />
</ul></li><br />
</ul><br />
<br />
<p>In an effort to resolve these data challenges, a new way of managing data was created which drove data oriented companies to invent a new data storage mechanism called a Data Lake.</p><br />
<p class="expand mw-collapsible-content">Data Lakes are characterized as: </p><br />
<ul class="expand mw-collapsible-content"><br />
<li>Collect Everything<ul><br />
<li>A Data Lake contains all data; raw sources over extended periods of time as well as any processed data.</li><br />
</ul></li><br />
<li>Dive in anywhere<ul><br />
<li>A Data Lake enables users across multiple business units to refine, explore and enrich data on their terms.</li><br />
</ul></li><br />
<li>Flexible Access<ul><br />
<li>o A Data Lake enables multiple data access patterns across a shared infrastructure: batch, interactive, online, search, in-memory and other processing engines.</li><br />
</ul></li><br />
</ul><br />
<br />
<p>Data Lakes are essentially a technology platform for holding data. Their value to the business is only realized when applying data science skills to the lake. </p><br />
<br />
<p>To summarize, usecases for Data Lakes are still being discovered. Cloud providers are making it easier to procure Data Lakes and today Data Lakes are primarily used by Research Institutions, Financial Services, Telecom, Media, Retail, Manufacturing, Healthcare, Pharma, Oi l& Gas and Governments.<br />
</p><br />
<br />
<br />
<br />
<h2>Technology Brief</h2><br />
<p class="expand mw-collapsible-content">The most popular implementation of a Data Lake is through the open source platform called Apache Hadoop. Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. Hadoop was originally created by researchers at Google as a storage method to handle the indexing of websites on the Internet; At that time it was called the Google File System. </p><br />
<p>A Data Lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.”</p><br />
<p class="expand mw-collapsible-content">Data can flow into the Data Lake by either batch processing or real-time processing of streaming data. Additionally, data itself is no longer restrained by initial schema decisions and can be exploited more freely by the enterprise. Rising above this repository is a set of capabilities that allow IT to provide Data and Analytics as a Service (DAaaS), in a supply-demand model. IT takes the role of the data provider (supplier), while business users (data scientists, business analysts) are consumers.</p><br />
<p>The DAaaS model enables users to self-serve their data and analytic needs. Users browse the lake’s data catalog (a Datapedia) to find and select the available data and fill a metaphorical “shopping cart” (effectively an analytics sandbox) with data to work with. Once access is provisioned, users can use the analytics tools of their choice to develop models and gain insights. Subsequently, users can publish analytical models or push refined or transformed data back into the Data Lake to share with the larger community.</p><br />
<p class="expand mw-collapsible-content">Although provisioning an analytic sandbox is a primary use, the Data Lake also has other applications. For example, the Data Lake can also be used to ingest raw data, curate the data, and apply Export-Transform-Load (ETL). This data can then be loaded to an Enterprise Data Warehouse. To take advantage of the flexibility provided by the Data Lake, organizations need to customize and configure the Data Lake to their specific requirements and domains.</p><br />
<h2>Industry Usage</h2><br />
<p>There are a variety of ways Data Lakes are being used in the industry:</p><br />
<ul><br />
<li><p><b>Ingestion of semi-structured and unstructured data sources (aka big data)</b>such as equipment readings, telemetry data, logs, streaming data, and so forth. A Data Lake is a great solution for storing IoT (Internet of Things) type of data which has traditionally been more difficult to store, and can support near real-time analysis. Optionally, you can also add structured data (i.e., extracted from a relational data source) to a Data Lake if your objective is a single repository of all data to be available via the lake.</p></li><br />
<li><p><b>Experimental analysis </b>of data before its value or purpose has been fully defined. Agility is important for every business these days, so a Data Lake can play an important role in "proof of value" type of situations because of the "ELT" approach discussed above.</p></li><br />
<li><p><b>Advanced analytics support. </b>A Data Lake is useful for data scientists and analysts to provision and experiment with data.</p></li><br />
<li><p><b>Archival and historical data storage. </b>Sometimes data is used infrequently, but does need to be available for analysis. A Data Lake strategy can be very valuable to support an active archive strategy.</p></li><br />
<li><p><b>Distributed processing </b>capabilities associated with a logical data warehouse.</p></li><br />
</ul><br />
<p class="expand mw-collapsible-content"><b>How TD Bank Made Its Data Lake More Usa</b></p><br />
<p class="expand mw-collapsible-content">[[https://www.datanami.com/2017/10/03/td-bank-made-data-lake-usable]]<br>Toronto-Dominion Bank (TD Bank) is one of the largest banks in North America, with 85,000 employees, more than 2,400 locations between Canada and the United States, and assets nearing $1 trillion. In 2014, the company decided to standardize how it warehouses data for various business intelligence and regulatory reporting functions. The company purchased a Hadoop distribution and set off to build a large cluster that could function as a centralized lake to store data originating from a variety of departments.</p><br />
<br />
<h2>Canadian Government Use</h2><br />
<p>In 2019, the Treasury Board of Canada Secretariat (TBS), partnered with Shared Services Canada and other departments, to identify a business lead to develop a Data Lake (a repository of raw data) service strategy so that the GC can take advantage of big data and market innovation to foster better analytics and promote horizontal data-sharing<ref>Treasury Board of Canada Secretariat. (March 29th, 2019). Digital Operations Strategic Plan: 2018-2022. Government of Canada. Treasury Board of Canada Secretariat. Retrieved 26-May-2019 from: <i>[https://www.canada.ca/en/government/system/digital-government/digital-operations-strategic-plan-2018-2022.html] </i></ref>. </p><br />
<p class="expand mw-collapsible-content">Big data is the technology that stores and processes data and information in datasets that are so large or complex that traditional data processing applications can’t analyze them. Big data can make available almost limitless amounts of information, improving data-driven decision-making and expanding open data initiatives. Business intelligence involves creating, aggregating, analyzing and visualizing data to inform and facilitate business management and strategy. TBS, working with departments, will lead the development of requirements for an enterprise analytics platform<ref>Ibid.<i></i></ref>.</p><br />
<p>Data Lake development in the GC is a more recent initiative. This is mainly due to the GC focussing resources on the implementation of cloud initiatives. However, there are some GC departments engaged in developing Data Lake environments in tandem to cloud initiatives.</p><br />
<p class="expand mw-collapsible-content">Notably, the Employment and Social Development Canada (ESDC) is preparing the installment of multiple Data Lakes in order to enable a Data Lake Ecosystem and Data Analytics and Machine Learning toolset. This will enable ESDC to share information horizontally both effectively and safely, while enabling a wide variety of data analytics capabilities. ESDC aims to maintain current data and analytics capabilities up-to-date while exploring new ones to mitigate gaps and continuously evolve our services to meet client’s needs<ref>Brisson, Yannick, and Craig, Sheila. (November, 2018). ESDC Data Lake – Implementation Strategy and Roadmap Update. Government of Canada. Employment and Social Development Canada – Data and Analytics Services. Presentation. Last Modified on 2019-04-26 15:45. Retrieved 07-May-2019 from GCDocs<i>[https://gcdocs.gc.ca/ssc-spc/llisapi.dll?func=ll&objaction=overview&objid=36624914 ]</i></ref>. </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="expand mw-collapsible-content">There are three common value propositions for pursuing Data Lakes. 1) It can provide an easy and accessible way to obtain data faster; 2) It can create a singular inflow point of data to help connect and merge information silos in an organization; and 3) It can provide an experimental environment for experienced data scientists to enable new analytical insights.</p><br />
<p> </p><br />
<p class="inline">Data Lakes can provide data to consumers more quickly by offering data in a more raw and easily accessible form. Data is stored in its native form with little to no processing, it is optimized to store vast amounts of data in their native formats. By allowing the data to remain in its native format, a much timelier stream of data is available for unlimited queries and analysis. A Data Lake can help data consumers bypass strict data retrieval and data structured applications such as a data warehouse and/or data mart. This has the effect of improving a business’ data flexibility.</p><p class="expand inline mw-collapsible-content">Some companies have in fact used Data Lakes to replace existing warehousing environments where implementing a new data warehouse is more cost prohibitive. A Data Lake can contain unrefined data, this is helpful when either a business data structure is unknown, or when a data consumer requires access to the data quickly. </p><br />
<p> </p><br />
<p class="inline">A Data Lake is not a single source of truth. A Data Lake is a central location in which data converges from all data sources and is stored, regardless of the data formatting. </p><p class="expand inline mw-collapsible-content">As a singular point for the inflow of data, sections of a business can pool their information together in the Data Lake and increase the sharing of information with other parts of the organization. In this way everyone in the organization has access to the data. A Data Lake can increase the horizontal data sharing within an organization by creating this singular data inflow point. Using a variety of storage and processing tools analysts can extract data value quickly in order to inform key business decisions.</p><br />
<p> </p><br />
<p class="expand inline mw-collapsible-content">A Data Lake is optimized for exploration and provides an experimental environment for experienced data scientists to uncover new insights from data. Analysts can overlay context on the data to extract value. All organizations want to increase analytics and operational agility.</p><p class="inline">The Data Lake architectural approach can store large volumes of data, this can be a way in which cross-cutting teams can pool their data in a central location and by complementing their systems of record with systems of insight. </p><p class="expand inline mw-collapsible-content">Data Lakes present the most potential benefits for experienced and competant data scientists. </p><p class="inline">Having structured, unstructured and semistructured data, usually in the same data set, can contain business, predictive, and prescriptive insights previously not possible from a structured platform as observed in data warehouses and data marts.</p><br />
<br />
<h4>Challenges</h4><br />
<p>Although Data Lake technology has many benefits for organizations dealing with big data it has its own challenges. For example:</p><br />
<p><b>Data Governance and Semantic Issues</b></p><br />
<p class="expand inline mw-collapsible-content">The biggest challenge for Data Lakes is to resolve assorted data governance requirements in a single centralized data platform. Data Lakes fail mostly when they lack governance, self-disciplined users, and a rational data flow.</p><p class="inline">Often, Data Lake implementations are focused on storing data instead of managing the data. Data Lakes are not optimized for semantic enforcement or consistency. They are made for semantic flexibility, to allow anyone to provide context to data if they have the skills to do so. </p><br />
<p>Putting data in the same place does not remove it’s ambiguity or meaning. Data Lakes provide unconstrained, “no compromises” storage model environment without the data governance assurances common to data warehouses or data marts. Proper meta data is essential for a Data Lake, without appropriate meta data the Data Lake will not work as intended. It is beneficial to think of meta data as the fish finder in the Data Lake.</p><br />
<p><b class="expand mw-collapsible-content">Lack of Quality and Trust in Data</b></p><br />
<p class="expand mw-collapsible-content">Data quality and trust in the data is a perennial issue for many organizations. Although data discovery tools can apply Machine Learning across related datasets from multiple data sources to identify anomalies (incorrect values, missing values, duplicates and outdated data), quality and trustworthiness of data continue to be an issue for Data Lakes who can easily become data dumping grounds. Some data is more accurate than others. This can present a real problem for anyone using multiple data sets and making decisions based upon analysis conducted with data of varying degrees of quality.</p><br />
<p><b>Data Swamps, Performance, and Flexibility Challenges</b></p><br />
<p class="expand inline mw-collapsible-content">Data stored in Data Lakes can sometimes become muddy when good data is mixed with bad data. Data Lake infrastructure is meant to store and process large amounts of data, usually in massive data files. </p><p class="inline">A Data Lake is not optimized for a high number of users or diverse and simultaneous workloads due to intensive query tasks. This can result in performance degradation and failures are common when running extractions, transformations, and loading tasks all at the same time. On-premises Data Lakes face other performance challenges in that they have a static configuration. </p><br />
<p><b class="expand mw-collapsible-content">Data Hoarding and Storage Capacity</b></p><br />
<p class="expand mw-collapsible-content">Data stored in Data Lakes may actually never be used in production and stay unused indefinitely in the Data Lake. By storing massive amounts of historical data, the infinite Data Lake may skew analysis with data that is no longer relevant to the priorities of the business. In keeping the historical data the metadata describing it must be understood as well. This decreases the performance of the Data Lake by increasing the overall workload of employees to clean the datasets no longer in use for analysis.</p><br />
<p class="expand mw-collapsible-content">Storing increasingly massive amounts of data for an unlimited time will also lead to scalability and cost challenges. Scalability challenges are less of a risk in public cloud environments, but cost remains a factor. On-premises Data Lakes are more susceptible to cost challenges. This is because their cluster nodes require all three dimensions of computing (storage, memory and processing). Organizations of all kinds generate massive amounts of data (including meta data) and it is increasing exponentially.</p><br />
<p>The storage capacity of all this data (and future data) will be an ongoing challenge and one that will require constant management. While Data Lakes can and will be stored on the cloud, SSC as cloud broker for the GC will need to provide the appropriate infrastructure and scalability to clients.</p><br />
<p><b class="expand mw-collapsible-content">Advanced Users Required</b></p><br />
<p class="expand mw-collapsible-content">Data Lakes are not a platform to be explored by everyone. Data Lakes present an unrefined view of data that usually only the most highly skilled analysts are able to explore and engage in data refinement independent of any other formal system-of-record such as a data warehouse. </p><br />
<p class="expand mw-collapsible-content">Not just anyone in an organization is data-literate enough to derive value from large amounts of raw or uncurated data. The reality is only a handful of staff are skilled enough to navigate a Data Lake. Since Data Lakes store raw data their business value is entirely determined by the skills of Data Lake users. These skills are often lacking in an organization.</p><br />
<p><b>Data Security</b></p><br />
<p class="inline"></p>Data in a Data Lake lacks standard security protection with a relational database management system or an enterprise database. In practice, this means that the data is unencrypted and lacks access control.<p class="expand inline mw-collapsible-content">. Security is not just a binary solution. We have varying degrees of security (unclassified, secret, top secret, etc.) and all of which require different approaches. This will inevitably present challenges with the successful use of data from Data Lakes.To combat this, organizations will have to embrace a new security framework to be compatable with Data Lakes and Data Scientists.</p><br />
<h4>Considerations</h4><br />
<p class="expand mw-collapsible-content">Shared Services Canada (SSC) has an excellent opportunity to capitalize on its mandate of providing data storage service to GC’s other departments. SSC, as the GC’s Service Provider, could potentially a centralized GC Data Lake and allow GC Data Scientists access to this central data using a single unified Data Lake interface. However, this is a project which should be implemented after cloud has been adopted and enterprise data centers have been migrated to in order to provide adequate infrastructure and scaling.</p><br />
<p class="inline">Data Lakes should not be confused for conventional databases although they both store information. A Data Lake will always underperform when tasked with the jobs of a conventional database. </p><p class="expand inline mw-collapsible-content">To combat this, SSC must create data architectures that define the proper application of Data Lakes. Too often, Data Lakes suffer from lack of foresight on what they're supposed to achieve. </p><p class="inline">Creating a Data Lake becomes the goal rather than achieving a strategic objective. </p><p class="expand inline mw-collapsible-content"></p><br />
<p class="expand mw-collapsible-content">Shared Services Canada (SSC) should consider designing Data Lake infrastructure around Service-Level Agreements (SLA) to keep Data Lake efforts on track. This includes ensuring that SSC has established clear goals for Data Lakes prior to deployment. </p><br />
<p class="expand mw-collapsible-content">SSC should also consider building an expert special group focussed on advanced analytics and experimental data trend discovery in Data Lakes. While the fundamental assumption behind the Data Lake concept is that everyone accessing a Data Lake is moderately to highly skilled at data manipulation and analysis, the reality is most are not. SSC should consider significant investment in training employees necessary skills, such as Data Science, Artificial Intelligence, Machine Learning, or Data Engineering.</p><br />
<p>SSC should be cognisant that there are significant overinflated expectations revolving around Data Lakes. Inflated expectations lead to vague and ambiguous use cases and increased chances of catastrophic failures. As a Service Provider, SSC must be strict in establishing clear goals for Data Lake provision efforts before deployment. SSC, should be wary of attempts to replace strategy development with infrastructure. A Data Lake can be a technology component that supports a data and analytics strategy, but it cannot replace that strategy.</p><br />
<p class="expand mw-collapsible-content">SSC should be concerned with the provision and running of the infrastructure, the departments themselves are responsible for the data they put in the Data Lake. However, as a Service Provider, SSC should monitor the Data Lake with regards to data governance, data lifecycle for data hygiene, and what is happening in the Data Lake overall. Depending on technology, SSC will need to be very clear on how to monitor activities in the Data Lakes it provides to the GC. </p><br />
<p class="expand mw-collapsible-content">SSC should consider a Data Lake implementation project as a way to introduce or reinvigorate a data management program by positioning data management capabilities as a prerequisite for a<br />
successful Data Lake. Data will need to be qualified before it hits the data lake, this can and should be done in a system of record first. In this way the data can be organizedto fit into the Data Lake implementation.<br />
</p><br />
<p class="expand mw-collapsible-content">SSC should create policies on how data is managed and cleaned in the Data Lake. Automated data governance technologies should be added to support advanced analytics. Standardizing on a specific type of governance tool is an issue which must be resolved. Additionally, planning for effective metadata management, considering metadata discovery, cataloguing and enterprise metadata management applied to Data Lake implementation is vital. Rigorous application of data discipline and data hygiene is needed. To combat this, SSC should use data management tools and create policies on how data is managed and cleaned in the Data Lake. The majority of Data Lake analysts will prefer to work with clean, enriched, and trusted data. However, data quality is relative to the task at hand. Lowquality data may be acceptable for low-impact analysis or distant forecasting, but unacceptable for tactical or high-impact analysis. SSC assessments should take this into account.</p><br />
<p>Design Data Lakes with the elements necessary to deliver reliable analytical results to a variety of data consumers. The goal is to increase cross-business usage in order to deliver advanced analytical insights. Build Data Lakes for specific business units or analytics applications, rather than try to implement some vague notion of a single enterprise Data Lake. However, alternative architectures, like data hubs, are often better fits for sharing data within an organization.</p><br />
<h2>References</h2><br />
<ref>Dennis, A. L. (2018, October 15). Data Lakes 101: An Overview. Retrieved from <i>[https://www.dataversity.net/data-lakes-101-overview/#]</i></ref><br />
<ref>Marvin, R., Marvin, R., & Marvin, R. (2016, August 22). Data Lakes, Explained. Retrieved from <i>[ https://www.pcmag.com/article/347020/data-lakes-explained]</i></ref><br />
<ref>The Data Lake journey. (2014, March 15). Retrieved from <i>[https://hortonworks.com/blog/enterprise-hadoop-journey-data-lake/]</i></ref><br />
<ref>Google File System. (2019, July 14). Retrieved from <i>[https://en.wikipedia.org/wiki/Google_File_System]</i></ref><br />
<ref>Coates, M. (2016, October 02). Data Lake Use Cases and Planning Considerations. Retrieved from <i>[https://www.sqlchick.com/entries/2016/7/31/data-lake-use-cases-and-planning]</i></ref><br />
<ref>Bhalchandra, V. (2018, July 23). Six reasons to think twice about your data lake strategy. Retrieved from <i>[https://dataconomy.com/2018/07/six-reasons-to-think-twice-about-your-data-lake-strategy/]</i></ref><br />
<ref>Data Lake Expectations: Why Data Lakes Fail. (2018, September 20). Retrieved from <i>[https://www.arcadiadata.com/blog/the-top-six-reasons-data-lakes-have-failed-to-live-up-to-expectations/]</i></ref><br />
<ref>Data Lake: AWS Solutions. (n.d.). Retrieved from <i>[https://aws.amazon.com/solutions/data-lake-solution/]</i></ref><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Datalakes&diff=11145Technology Trends/Datalakes2019-07-18T19:01:40Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Datalakes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Lac de Donnés]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[KubernetesIMG.png|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Datalakes</b> is a central system or repository of data that is stored in its natural/raw format. A datalake acts as a single store for all enterprise data. Data is transformed using machine learning, advanced,analytics, and visualization. Several forms of data can be hosed in a datalake. These include structured data from relational databases, unstructured data, semi-structured data, and binary data. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>In an ever-increasing hyperconnected world, corporations and businesses are struggling to deal with the responsibilities of storage, management and quick availability of raw data. To break these data challenges down further:</p><br />
<ul><br />
<li>Data comes in many different structures.<br />
<ul><br />
<li>Unstructured</li><br />
<li>Semi-Sturctured</li><br />
<li>Structured</li><br />
</ul></li><br />
<li>Data comes from many disparate sources.<br />
<ul><br />
<li>Enterprise Applications</li><br />
<li>Raw Files</li><br />
<li>Operation and Security Logs</li><br />
<li>Financial Transactions</li><br />
<li>Internet of Things (IoT) Devices and Network Sensors</li><br />
<li>Websites</li><br />
<li>Scientific Research</li><br />
</ul></li><br />
<li>Data sources are often geographically distributed to multiple locations<br />
<ul><br />
<li>Datacenters</li><br />
<li>Remote Offices</li><br />
<li>Mobile Devices</li><br />
</ul></li><br />
</ul><br />
<br />
<p>In an effort to resolve these data challenges, a new way of managing data was created which drove data oriented companies to invent a new data storage mechanism called a Data Lake.</p><br />
<p class="expand mw-collapsible-content">Data Lakes are characterized as: </p><br />
<ul class="expand mw-collapsible-content"><br />
<li>Collect Everything<ul><br />
<li>A Data Lake contains all data; raw sources over extended periods of time as well as any processed data.</li><br />
</ul></li><br />
<li>Dive in anywhere<ul><br />
<li>A Data Lake enables users across multiple business units to refine, explore and enrich data on their terms.</li><br />
</ul></li><br />
<li>Flexible Access<ul><br />
<li>o A Data Lake enables multiple data access patterns across a shared infrastructure: batch, interactive, online, search, in-memory and other processing engines.</li><br />
</ul></li><br />
</ul><br />
<br />
<p>Data Lakes are essentially a technology platform for holding data. Their value to the business is only realized when applying data science skills to the lake. </p><br />
<br />
<p>To summarize, usecases for Data Lakes are still being discovered. Cloud providers are making it easier to procure Data Lakes and today Data Lakes are primarily used by Research Institutions, Financial Services, Telecom, Media, Retail, Manufacturing, Healthcare, Pharma, Oi l& Gas and Governments.<br />
</p><br />
<br />
<br />
<br />
<h2>Technology Brief</h2><br />
<p class="expand mw-collapsible-content">The most popular implementation of a Data Lake is through the open source platform called Apache Hadoop. Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. Hadoop was originally created by researchers at Google as a storage method to handle the indexing of websites on the Internet; At that time it was called the Google File System. </p><br />
<p>A Data Lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.”</p><br />
<p class="expand mw-collapsible-content">Data can flow into the Data Lake by either batch processing or real-time processing of streaming data. Additionally, data itself is no longer restrained by initial schema decisions and can be exploited more freely by the enterprise. Rising above this repository is a set of capabilities that allow IT to provide Data and Analytics as a Service (DAaaS), in a supply-demand model. IT takes the role of the data provider (supplier), while business users (data scientists, business analysts) are consumers.</p><br />
<p>The DAaaS model enables users to self-serve their data and analytic needs. Users browse the lake’s data catalog (a Datapedia) to find and select the available data and fill a metaphorical “shopping cart” (effectively an analytics sandbox) with data to work with. Once access is provisioned, users can use the analytics tools of their choice to develop models and gain insights. Subsequently, users can publish analytical models or push refined or transformed data back into the Data Lake to share with the larger community.</p><br />
<p class="expand mw-collapsible-content">Although provisioning an analytic sandbox is a primary use, the Data Lake also has other applications. For example, the Data Lake can also be used to ingest raw data, curate the data, and apply Export-Transform-Load (ETL). This data can then be loaded to an Enterprise Data Warehouse. To take advantage of the flexibility provided by the Data Lake, organizations need to customize and configure the Data Lake to their specific requirements and domains.</p><br />
<h2>Industry Usage</h2><br />
<p>There are a variety of ways Data Lakes are being used in the industry:</p><br />
<ul><br />
<li><p><b>Ingestion of semi-structured and unstructured data sources (aka big data)</b>such as equipment readings, telemetry data, logs, streaming data, and so forth. A Data Lake is a great solution for storing IoT (Internet of Things) type of data which has traditionally been more difficult to store, and can support near real-time analysis. Optionally, you can also add structured data (i.e., extracted from a relational data source) to a Data Lake if your objective is a single repository of all data to be available via the lake.</p></li><br />
<li><p><b>Experimental analysis </b>of data before its value or purpose has been fully defined. Agility is important for every business these days, so a Data Lake can play an important role in "proof of value" type of situations because of the "ELT" approach discussed above.</p></li><br />
<li><p><b>Advanced analytics support. </b>A Data Lake is useful for data scientists and analysts to provision and experiment with data.</p></li><br />
<li><p><b>Archival and historical data storage. </b>Sometimes data is used infrequently, but does need to be available for analysis. A Data Lake strategy can be very valuable to support an active archive strategy.</p></li><br />
<li><p><b>Distributed processing </b>capabilities associated with a logical data warehouse.</p></li><br />
</ul><br />
<p class="expand mw-collapsible-content"><b>How TD Bank Made Its Data Lake More Usa</b></p><br />
<p class="expand mw-collapsible-content">[[https://www.datanami.com/2017/10/03/td-bank-made-data-lake-usable]]<br>Toronto-Dominion Bank (TD Bank) is one of the largest banks in North America, with 85,000 employees, more than 2,400 locations between Canada and the United States, and assets nearing $1 trillion. In 2014, the company decided to standardize how it warehouses data for various business intelligence and regulatory reporting functions. The company purchased a Hadoop distribution and set off to build a large cluster that could function as a centralized lake to store data originating from a variety of departments.</p><br />
<br />
<h2>Canadian Government Use</h2><br />
<p>In 2019, the Treasury Board of Canada Secretariat (TBS), partnered with Shared Services Canada and other departments, to identify a business lead to develop a Data Lake (a repository of raw data) service strategy so that the GC can take advantage of big data and market innovation to foster better analytics and promote horizontal data-sharing<ref>Treasury Board of Canada Secretariat. (March 29th, 2019). Digital Operations Strategic Plan: 2018-2022. Government of Canada. Treasury Board of Canada Secretariat. Retrieved 26-May-2019 from: <i>[https://www.canada.ca/en/government/system/digital-government/digital-operations-strategic-plan-2018-2022.html] </i></ref>. </p><br />
<p class="expand mw-collapsible-content">Big data is the technology that stores and processes data and information in datasets that are so large or complex that traditional data processing applications can’t analyze them. Big data can make available almost limitless amounts of information, improving data-driven decision-making and expanding open data initiatives. Business intelligence involves creating, aggregating, analyzing and visualizing data to inform and facilitate business management and strategy. TBS, working with departments, will lead the development of requirements for an enterprise analytics platform<ref>Ibid.<i></i></ref>.</p><br />
<p>Data Lake development in the GC is a more recent initiative. This is mainly due to the GC focussing resources on the implementation of cloud initiatives. However, there are some GC departments engaged in developing Data Lake environments in tandem to cloud initiatives.</p><br />
<p class="expand mw-collapsible-content">Notably, the Employment and Social Development Canada (ESDC) is preparing the installment of multiple Data Lakes in order to enable a Data Lake Ecosystem and Data Analytics and Machine Learning toolset. This will enable ESDC to share information horizontally both effectively and safely, while enabling a wide variety of data analytics capabilities. ESDC aims to maintain current data and analytics capabilities up-to-date while exploring new ones to mitigate gaps and continuously evolve our services to meet client’s needs<ref>Brisson, Yannick, and Craig, Sheila. (November, 2018). ESDC Data Lake – Implementation Strategy and Roadmap Update. Government of Canada. Employment and Social Development Canada – Data and Analytics Services. Presentation. Last Modified on 2019-04-26 15:45. Retrieved 07-May-2019 from GCDocs<i>[https://gcdocs.gc.ca/ssc-spc/llisapi.dll?func=ll&objaction=overview&objid=36624914 ]</i></ref>. </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="expand mw-collapsible-content">There are three common value propositions for pursuing Data Lakes. 1) It can provide an easy and accessible way to obtain data faster; 2) It can create a singular inflow point of data to help connect and merge information silos in an organization; and 3) It can provide an experimental environment for experienced data scientists to enable new analytical insights.</p><br />
<p> </p><br />
<p class="inline">Data Lakes can provide data to consumers more quickly by offering data in a more raw and easily accessible form. Data is stored in its native form with little to no processing, it is optimized to store vast amounts of data in their native formats. By allowing the data to remain in its native format, a much timelier stream of data is available for unlimited queries and analysis. A Data Lake can help data consumers bypass strict data retrieval and data structured applications such as a data warehouse and/or data mart. This has the effect of improving a business’ data flexibility.</p><p class="expand inline mw-collapsible-content">Some companies have in fact used Data Lakes to replace existing warehousing environments where implementing a new data warehouse is more cost prohibitive. A Data Lake can contain unrefined data, this is helpful when either a business data structure is unknown, or when a data consumer requires access to the data quickly. </p><br />
<p> </p><br />
<p class="inline">A Data Lake is not a single source of truth. A Data Lake is a central location in which data converges from all data sources and is stored, regardless of the data formatting. </p><p class="expand inline mw-collapsible-content">As a singular point for the inflow of data, sections of a business can pool their information together in the Data Lake and increase the sharing of information with other parts of the organization. In this way everyone in the organization has access to the data. A Data Lake can increase the horizontal data sharing within an organization by creating this singular data inflow point. Using a variety of storage and processing tools analysts can extract data value quickly in order to inform key business decisions.</p><br />
<p> </p><br />
<p class="expand inline mw-collapsible-content">A Data Lake is optimized for exploration and provides an experimental environment for experienced data scientists to uncover new insights from data. Analysts can overlay context on the data to extract value. All organizations want to increase analytics and operational agility.</p><p class="inline">The Data Lake architectural approach can store large volumes of data, this can be a way in which cross-cutting teams can pool their data in a central location and by complementing their systems of record with systems of insight. </p><p class="expand inline mw-collapsible-content">Data Lakes present the most potential benefits for experienced and competant data scientists. </p><p class="inline">Having structured, unstructured and semistructured data, usually in the same data set, can contain business, predictive, and prescriptive insights previously not possible from a structured platform as observed in data warehouses and data marts.</p><br />
<br />
<h4>Challenges</h4><br />
<p>Although Data Lake technology has many benefits for organizations dealing with big data it has its own challenges. For example:</p><br />
<p><b>Data Governance and Semantic Issues</b></p><br />
<p class="expand inline mw-collapsible-content">The biggest challenge for Data Lakes is to resolve assorted data governance requirements in a single centralized data platform. Data Lakes fail mostly when they lack governance, self-disciplined users, and a rational data flow.</p><p class="inline">Often, Data Lake implementations are focused on storing data instead of managing the data. Data Lakes are not optimized for semantic enforcement or consistency. They are made for semantic flexibility, to allow anyone to provide context to data if they have the skills to do so. </p><br />
<p>Putting data in the same place does not remove it’s ambiguity or meaning. Data Lakes provide unconstrained, “no compromises” storage model environment without the data governance assurances common to data warehouses or data marts. Proper meta data is essential for a Data Lake, without appropriate meta data the Data Lake will not work as intended. It is beneficial to think of meta data as the fish finder in the Data Lake.</p><br />
<p><b class="expand mw-collapsible-content">Lack of Quality and Trust in Data</b></p><br />
<p class="expand mw-collapsible-content">Data quality and trust in the data is a perennial issue for many organizations. Although data discovery tools can apply Machine Learning across related datasets from multiple data sources to identify anomalies (incorrect values, missing values, duplicates and outdated data), quality and trustworthiness of data continue to be an issue for Data Lakes who can easily become data dumping grounds. Some data is more accurate than others. This can present a real problem for anyone using multiple data sets and making decisions based upon analysis conducted with data of varying degrees of quality.</p><br />
<p><b>Data Swamps, Performance, and Flexibility Challenges</b></p><br />
<p class="expand inline mw-collapsible-content">Data stored in Data Lakes can sometimes become muddy when good data is mixed with bad data. Data Lake infrastructure is meant to store and process large amounts of data, usually in massive data files. </p><p class="inline">A Data Lake is not optimized for a high number of users or diverse and simultaneous workloads due to intensive query tasks. This can result in performance degradation and failures are common when running extractions, transformations, and loading tasks all at the same time. On-premises Data Lakes face other performance challenges in that they have a static configuration. </p><br />
<p><b class="expand mw-collapsible-content">Data Hoarding and Storage Capacity</b></p><br />
<p class="expand mw-collapsible-content">Data stored in Data Lakes may actually never be used in production and stay unused indefinitely in the Data Lake. By storing massive amounts of historical data, the infinite Data Lake may skew analysis with data that is no longer relevant to the priorities of the business. In keeping the historical data the metadata describing it must be understood as well. This decreases the performance of the Data Lake by increasing the overall workload of employees to clean the datasets no longer in use for analysis.</p><br />
<p class="expand mw-collapsible-content">Storing increasingly massive amounts of data for an unlimited time will also lead to scalability and cost challenges. Scalability challenges are less of a risk in public cloud environments, but cost remains a factor. On-premises Data Lakes are more susceptible to cost challenges. This is because their cluster nodes require all three dimensions of computing (storage, memory and processing). Organizations of all kinds generate massive amounts of data (including meta data) and it is increasing exponentially.</p><br />
<p>The storage capacity of all this data (and future data) will be an ongoing challenge and one that will require constant management. While Data Lakes can and will be stored on the cloud, SSC as cloud broker for the GC will need to provide the appropriate infrastructure and scalability to clients.</p><br />
<p><b class="expand mw-collapsible-content">Advanced Users Required</b></p><br />
<p class="expand mw-collapsible-content">Data Lakes are not a platform to be explored by everyone. Data Lakes present an unrefined view of data that usually only the most highly skilled analysts are able to explore and engage in data refinement independent of any other formal system-of-record such as a data warehouse. </p><br />
<p class="expand mw-collapsible-content">Not just anyone in an organization is data-literate enough to derive value from large amounts of raw or uncurated data. The reality is only a handful of staff are skilled enough to navigate a Data Lake. Since Data Lakes store raw data their business value is entirely determined by the skills of Data Lake users. These skills are often lacking in an organization.</p><br />
<p><b>Data Security</b></p><br />
<p class="inline"></p>Data in a Data Lake lacks standard security protection with a relational database management system or an enterprise database. In practice, this means that the data is unencrypted and lacks access control.<p class="expand inline mw-collapsible-content">. Security is not just a binary solution. We have varying degrees of security (unclassified, secret, top secret, etc.) and all of which require different approaches. This will inevitably present challenges with the successful use of data from Data Lakes.To combat this, organizations will have to embrace a new security framework to be compatable with Data Lakes and Data Scientists.</p><br />
<h4>Considerations</h4><br />
<p class="expand mw-collapsible-content">Shared Services Canada (SSC) has an excellent opportunity to capitalize on its mandate of providing data storage service to GC’s other departments. SSC, as the GC’s Service Provider, could potentially a centralized GC Data Lake and allow GC Data Scientists access to this central data using a single unified Data Lake interface. However, this is a project which should be implemented after cloud has been adopted and enterprise data centers have been migrated to in order to provide adequate infrastructure and scaling.</p><br />
<p class="inline">Data Lakes should not be confused for conventional databases although they both store information. A Data Lake will always underperform when tasked with the jobs of a conventional database. </p><p class="expand inline mw-collapsible-content">To combat this, SSC must create data architectures that define the proper application of Data Lakes. Too often, Data Lakes suffer from lack of foresight on what they're supposed to achieve. </p><p class="inline">Creating a Data Lake becomes the goal rather than achieving a strategic objective. </p><p class="expand inline mw-collapsible-content"></p><br />
<p class="expand mw-collapsible-content">Shared Services Canada (SSC) should consider designing Data Lake infrastructure around Service-Level Agreements (SLA) to keep Data Lake efforts on track. This includes ensuring that SSC has established clear goals for Data Lakes prior to deployment. </p><br />
<p class="expand mw-collapsible-content">SSC should also consider building an expert special group focussed on advanced analytics and experimental data trend discovery in Data Lakes. While the fundamental assumption behind the Data Lake concept is that everyone accessing a Data Lake is moderately to highly skilled at data manipulation and analysis, the reality is most are not. SSC should consider significant investment in training employees necessary skills, such as Data Science, Artificial Intelligence, Machine Learning, or Data Engineering.</p><br />
<p>SSC should be cognisant that there are significant overinflated expectations revolving around Data Lakes. Inflated expectations lead to vague and ambiguous use cases and increased chances of catastrophic failures. As a Service Provider, SSC must be strict in establishing clear goals for Data Lake provision efforts before deployment. SSC, should be wary of attempts to replace strategy development with infrastructure. A Data Lake can be a technology component that supports a data and analytics strategy, but it cannot replace that strategy.</p><br />
<p class="expand mw-collapsible-content">SSC should be concerned with the provision and running of the infrastructure, the departments themselves are responsible for the data they put in the Data Lake. However, as a Service Provider, SSC should monitor the Data Lake with regards to data governance, data lifecycle for data hygiene, and what is happening in the Data Lake overall. Depending on technology, SSC will need to be very clear on how to monitor activities in the Data Lakes it provides to the GC. </p><br />
<p class="expand mw-collapsible-content">SSC should consider a Data Lake implementation project as a way to introduce or reinvigorate a data management program by positioning data management capabilities as a prerequisite for a<br />
successful Data Lake. Data will need to be qualified before it hits the data lake, this can and should be done in a system of record first. In this way the data can be organizedto fit into the Data Lake implementation.<br />
</p><br />
<p class="expand mw-collapsible-content">SSC should create policies on how data is managed and cleaned in the Data Lake. Automated data governance technologies should be added to support advanced analytics. Standardizing on a specific type of governance tool is an issue which must be resolved. Additionally, planning for effective metadata management, considering metadata discovery, cataloguing and enterprise metadata management applied to Data Lake implementation is vital. Rigorous application of data discipline and data hygiene is needed. To combat this, SSC should use data management tools and create policies on how data is managed and cleaned in the Data Lake. The majority of Data Lake analysts will prefer to work with clean, enriched, and trusted data. However, data quality is relative to the task at hand. Lowquality data may be acceptable for low-impact analysis or distant forecasting, but unacceptable for tactical or high-impact analysis. SSC assessments should take this into account.</p><br />
<p>Design Data Lakes with the elements necessary to deliver reliable analytical results to a variety of data consumers. The goal is to increase cross-business usage in order to deliver advanced analytical insights. Build Data Lakes for specific business units or analytics applications, rather than try to implement some vague notion of a single enterprise Data Lake. However, alternative architectures, like data hubs, are often better fits for sharing data within an organization.</p><br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Datalakes&diff=11144Technology Trends/Datalakes2019-07-18T18:50:32Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Datalakes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Lac de Donnés]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[KubernetesIMG.png|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Datalakes</b> is a central system or repository of data that is stored in its natural/raw format. A datalake acts as a single store for all enterprise data. Data is transformed using machine learning, advanced,analytics, and visualization. Several forms of data can be hosed in a datalake. These include structured data from relational databases, unstructured data, semi-structured data, and binary data. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>In an ever-increasing hyperconnected world, corporations and businesses are struggling to deal with the responsibilities of storage, management and quick availability of raw data. To break these data challenges down further:</p><br />
<ul><br />
<li>Data comes in many different structures.<br />
<ul><br />
<li>Unstructured</li><br />
<li>Semi-Sturctured</li><br />
<li>Structured</li><br />
</ul></li><br />
<li>Data comes from many disparate sources.<br />
<ul><br />
<li>Enterprise Applications</li><br />
<li>Raw Files</li><br />
<li>Operation and Security Logs</li><br />
<li>Financial Transactions</li><br />
<li>Internet of Things (IoT) Devices and Network Sensors</li><br />
<li>Websites</li><br />
<li>Scientific Research</li><br />
</ul></li><br />
<li>Data sources are often geographically distributed to multiple locations<br />
<ul><br />
<li>Datacenters</li><br />
<li>Remote Offices</li><br />
<li>Mobile Devices</li><br />
</ul></li><br />
</ul><br />
<br />
<p>In an effort to resolve these data challenges, a new way of managing data was created which drove data oriented companies to invent a new data storage mechanism called a Data Lake.</p><br />
<p class="expand mw-collapsible-content">Data Lakes are characterized as: </p><br />
<ul class="expand mw-collapsible-content"><br />
<li>Collect Everything<ul><br />
<li>A Data Lake contains all data; raw sources over extended periods of time as well as any processed data.</li><br />
</ul></li><br />
<li>Dive in anywhere<ul><br />
<li>A Data Lake enables users across multiple business units to refine, explore and enrich data on their terms.</li><br />
</ul></li><br />
<li>Flexible Access<ul><br />
<li>o A Data Lake enables multiple data access patterns across a shared infrastructure: batch, interactive, online, search, in-memory and other processing engines.</li><br />
</ul></li><br />
</ul><br />
<br />
<p>Data Lakes are essentially a technology platform for holding data. Their value to the business is only realized when applying data science skills to the lake. </p><br />
<br />
<p>To summarize, usecases for Data Lakes are still being discovered. Cloud providers are making it easier to procure Data Lakes and today Data Lakes are primarily used by Research Institutions, Financial Services, Telecom, Media, Retail, Manufacturing, Healthcare, Pharma, Oi l& Gas and Governments.<br />
</p><br />
<br />
<br />
<br />
<h2>Technology Brief</h2><br />
<p class="expand mw-collapsible-content">The most popular implementation of a Data Lake is through the open source platform called Apache Hadoop. Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. Hadoop was originally created by researchers at Google as a storage method to handle the indexing of websites on the Internet; At that time it was called the Google File System. </p><br />
<p>A Data Lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.”</p><br />
<p class="expand mw-collapsible-content">Data can flow into the Data Lake by either batch processing or real-time processing of streaming data. Additionally, data itself is no longer restrained by initial schema decisions and can be exploited more freely by the enterprise. Rising above this repository is a set of capabilities that allow IT to provide Data and Analytics as a Service (DAaaS), in a supply-demand model. IT takes the role of the data provider (supplier), while business users (data scientists, business analysts) are consumers.</p><br />
<p>The DAaaS model enables users to self-serve their data and analytic needs. Users browse the lake’s data catalog (a Datapedia) to find and select the available data and fill a metaphorical “shopping cart” (effectively an analytics sandbox) with data to work with. Once access is provisioned, users can use the analytics tools of their choice to develop models and gain insights. Subsequently, users can publish analytical models or push refined or transformed data back into the Data Lake to share with the larger community.</p><br />
<p class="expand mw-collapsible-content">Although provisioning an analytic sandbox is a primary use, the Data Lake also has other applications. For example, the Data Lake can also be used to ingest raw data, curate the data, and apply Export-Transform-Load (ETL). This data can then be loaded to an Enterprise Data Warehouse. To take advantage of the flexibility provided by the Data Lake, organizations need to customize and configure the Data Lake to their specific requirements and domains.</p><br />
<h2>Industry Usage</h2><br />
<p>There are a variety of ways Data Lakes are being used in the industry:</p><br />
<ul><br />
<li><p><b>Ingestion of semi-structured and unstructured data sources (aka big data)</b>such as equipment readings, telemetry data, logs, streaming data, and so forth. A Data Lake is a great solution for storing IoT (Internet of Things) type of data which has traditionally been more difficult to store, and can support near real-time analysis. Optionally, you can also add structured data (i.e., extracted from a relational data source) to a Data Lake if your objective is a single repository of all data to be available via the lake.</p></li><br />
<li><p><b>Experimental analysis </b>of data before its value or purpose has been fully defined. Agility is important for every business these days, so a Data Lake can play an important role in "proof of value" type of situations because of the "ELT" approach discussed above.</p></li><br />
<li><p><b>Advanced analytics support. </b>A Data Lake is useful for data scientists and analysts to provision and experiment with data.</p></li><br />
<li><p><b>Archival and historical data storage. </b>Sometimes data is used infrequently, but does need to be available for analysis. A Data Lake strategy can be very valuable to support an active archive strategy.</p></li><br />
<li><p><b>Distributed processing </b>capabilities associated with a logical data warehouse.</p></li><br />
</ul><br />
<p class="expand mw-collapsible-content"><b>How TD Bank Made Its Data Lake More Usa</b></p><br />
<p class="expand mw-collapsible-content">[[https://www.datanami.com/2017/10/03/td-bank-made-data-lake-usable]]<br>Toronto-Dominion Bank (TD Bank) is one of the largest banks in North America, with 85,000 employees, more than 2,400 locations between Canada and the United States, and assets nearing $1 trillion. In 2014, the company decided to standardize how it warehouses data for various business intelligence and regulatory reporting functions. The company purchased a Hadoop distribution and set off to build a large cluster that could function as a centralized lake to store data originating from a variety of departments.</p><br />
<br />
<h2>Canadian Government Use</h2><br />
<p>In 2019, the Treasury Board of Canada Secretariat (TBS), partnered with Shared Services Canada and other departments, to identify a business lead to develop a Data Lake (a repository of raw data) service strategy so that the GC can take advantage of big data and market innovation to foster better analytics and promote horizontal data-sharing. </p><br />
<p class="expand mw-collapsible-content">Big data is the technology that stores and processes data and information in datasets that are so large or complex that traditional data processing applications can’t analyze them. Big data can make available almost limitless amounts of information, improving data-driven decision-making and expanding open data initiatives. Business intelligence involves creating, aggregating, analyzing and visualizing data to inform and facilitate business management and strategy. TBS, working with departments, will lead the development of requirements for an enterprise analytics platform.</p><br />
<p>Data Lake development in the GC is a more recent initiative. This is mainly due to the GC focussing resources on the implementation of cloud initiatives. However, there are some GC departments engaged in developing Data Lake environments in tandem to cloud initiatives.</p><br />
<p class="expand mw-collapsible-content">Notably, the Employment and Social Development Canada (ESDC) is preparing the installment of multiple Data Lakes in order to enable a Data Lake Ecosystem and Data Analytics and Machine Learning toolset. This will enable ESDC to share information horizontally both effectively and safely, while enabling a wide variety of data analytics capabilities. ESDC aims to maintain current data and analytics capabilities up-to-date while exploring new ones to mitigate gaps and continuously evolve our services to meet client’s needs. </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="expand mw-collapsible-content">There are three common value propositions for pursuing Data Lakes. 1) It can provide an easy and accessible way to obtain data faster; 2) It can create a singular inflow point of data to help connect and merge information silos in an organization; and 3) It can provide an experimental environment for experienced data scientists to enable new analytical insights.</p><br />
<p> </p><br />
<p class="inline">Data Lakes can provide data to consumers more quickly by offering data in a more raw and easily accessible form. Data is stored in its native form with little to no processing, it is optimized to store vast amounts of data in their native formats. By allowing the data to remain in its native format, a much timelier stream of data is available for unlimited queries and analysis. A Data Lake can help data consumers bypass strict data retrieval and data structured applications such as a data warehouse and/or data mart. This has the effect of improving a business’ data flexibility.</p><p class="expand inline mw-collapsible-content">Some companies have in fact used Data Lakes to replace existing warehousing environments where implementing a new data warehouse is more cost prohibitive. A Data Lake can contain unrefined data, this is helpful when either a business data structure is unknown, or when a data consumer requires access to the data quickly. </p><br />
<p> </p><br />
<p class="inline">A Data Lake is not a single source of truth. A Data Lake is a central location in which data converges from all data sources and is stored, regardless of the data formatting. </p><p class="expand inline mw-collapsible-content">As a singular point for the inflow of data, sections of a business can pool their information together in the Data Lake and increase the sharing of information with other parts of the organization. In this way everyone in the organization has access to the data. A Data Lake can increase the horizontal data sharing within an organization by creating this singular data inflow point. Using a variety of storage and processing tools analysts can extract data value quickly in order to inform key business decisions.</p><br />
<p> </p><br />
<p class="expand inline mw-collapsible-content">A Data Lake is optimized for exploration and provides an experimental environment for experienced data scientists to uncover new insights from data. Analysts can overlay context on the data to extract value. All organizations want to increase analytics and operational agility.</p><p class="inline">The Data Lake architectural approach can store large volumes of data, this can be a way in which cross-cutting teams can pool their data in a central location and by complementing their systems of record with systems of insight. </p><p class="expand inline mw-collapsible-content">Data Lakes present the most potential benefits for experienced and competant data scientists. </p><p class="inline">Having structured, unstructured and semistructured data, usually in the same data set, can contain business, predictive, and prescriptive insights previously not possible from a structured platform as observed in data warehouses and data marts.</p><br />
<br />
<h4>Challenges</h4><br />
<p>Although Data Lake technology has many benefits for organizations dealing with big data it has its own challenges. For example:</p><br />
<b>Data Governance and Semantic Issues</b><br />
<p class="expand inline mw-collapsible-content">The biggest challenge for Data Lakes is to resolve assorted data governance requirements in a single centralized data platform. Data Lakes fail mostly when they lack governance, self-disciplined users, and a rational data flow.</p><p class="inline">Often, Data Lake implementations are focused on storing data instead of managing the data. Data Lakes are not optimized for semantic enforcement or consistency. They are made for semantic flexibility, to allow anyone to provide context to data if they have the skills to do so. </p><br />
<p>Putting data in the same place does not remove it’s ambiguity or meaning. Data Lakes provide unconstrained, “no compromises” storage model environment without the data governance assurances common to data warehouses or data marts. Proper meta data is essential for a Data Lake, without appropriate meta data the Data Lake will not work as intended. It is beneficial to think of meta data as the fish finder in the Data Lake.</p><br />
<b class="expand mw-collapsible-content">Lack of Quality and Trust in Data</b><br />
<p class="expand mw-collapsible-content">Data quality and trust in the data is a perennial issue for many organizations. Although data discovery tools can apply Machine Learning across related datasets from multiple data sources to identify anomalies (incorrect values, missing values, duplicates and outdated data), quality and trustworthiness of data continue to be an issue for Data Lakes who can easily become data dumping grounds. Some data is more accurate than others. This can present a real problem for anyone using multiple data sets and making decisions based upon analysis conducted with data of varying degrees of quality.</p><br />
<b>Data Swamps, Performance, and Flexibility Challenges</b><br />
<p class="expand inline mw-collapsible-content">Data stored in Data Lakes can sometimes become muddy when good data is mixed with bad data. Data Lake infrastructure is meant to store and process large amounts of data, usually in massive data files. </p><p class="inline">A Data Lake is not optimized for a high number of users or diverse and simultaneous workloads due to intensive query tasks. This can result in performance degradation and failures are common when running extractions, transformations, and loading tasks all at the same time. On-premises Data Lakes face other performance challenges in that they have a static configuration. </p><br />
<b class="expand mw-collapsible-content">Data Hoarding and Storage Capacity</b><br />
<p class="expand mw-collapsible-content">Data stored in Data Lakes may actually never be used in production and stay unused indefinitely in the Data Lake. By storing massive amounts of historical data, the infinite Data Lake may skew analysis with data that is no longer relevant to the priorities of the business. In keeping the historical data the metadata describing it must be understood as well. This decreases the performance of the Data Lake by increasing the overall workload of employees to clean the datasets no longer in use for analysis.</p><br />
<p class="expand mw-collapsible-content">Storing increasingly massive amounts of data for an unlimited time will also lead to scalability and cost challenges. Scalability challenges are less of a risk in public cloud environments, but cost remains a factor. On-premises Data Lakes are more susceptible to cost challenges. This is because their cluster nodes require all three dimensions of computing (storage, memory and processing). Organizations of all kinds generate massive amounts of data (including meta data) and it is increasing exponentially.</p><br />
<p>The storage capacity of all this data (and future data) will be an ongoing challenge and one that will require constant management. While Data Lakes can and will be stored on the cloud, SSC as cloud broker for the GC will need to provide the appropriate infrastructure and scalability to clients.</p><br />
<b class="expand mw-collapsible-content">Advanced Users Required</b><br />
<p class="expand mw-collapsible-content">Data Lakes are not a platform to be explored by everyone. Data Lakes present an unrefined view of data that usually only the most highly skilled analysts are able to explore and engage in data refinement independent of any other formal system-of-record such as a data warehouse. </p><br />
<p class="expand mw-collapsible-content">Not just anyone in an organization is data-literate enough to derive value from large amounts of raw or uncurated data. The reality is only a handful of staff are skilled enough to navigate a Data Lake. Since Data Lakes store raw data their business value is entirely determined by the skills of Data Lake users. These skills are often lacking in an organization.</p><br />
<b>Data Security</b><br />
<p class="inline"></p>Data in a Data Lake lacks standard security protection with a relational database management system or an enterprise database. In practice, this means that the data is unencrypted and lacks access control.<p class="expand inline mw-collapsible-content">. Security is not just a binary solution. We have varying degrees of security (unclassified, secret, top secret, etc.) and all of which require different approaches. This will inevitably present challenges with the successful use of data from Data Lakes.To combat this, organizations will have to embrace a new security framework to be compatable with Data Lakes and Data Scientists.</p><br />
<h4>Considerations</h4><br />
<p class="expand mw-collapsible-content">Shared Services Canada (SSC) has an excellent opportunity to capitalize on its mandate of providing data storage service to GC’s other departments. SSC, as the GC’s Service Provider, could potentially a centralized GC Data Lake and allow GC Data Scientists access to this central data using a single unified Data Lake interface. However, this is a project which should be implemented after cloud has been adopted and enterprise data centers have been migrated to in order to provide adequate infrastructure and scaling.</p><br />
<p class="inline">Data Lakes should not be confused for conventional databases although they both store information. A Data Lake will always underperform when tasked with the jobs of a conventional database. </p><p class="expand inline mw-collapsible-content">To combat this, SSC must create data architectures that define the proper application of Data Lakes. Too often, Data Lakes suffer from lack of foresight on what they're supposed to achieve. </p><p class="inline">Creating a Data Lake becomes the goal rather than achieving a strategic objective. </p><p class="expand inline mw-collapsible-content"></p><br />
<p class="expand mw-collapsible-content">Shared Services Canada (SSC) should consider designing Data Lake infrastructure around Service-Level Agreements (SLA) to keep Data Lake efforts on track. This includes ensuring that SSC has established clear goals for Data Lakes prior to deployment. </p><br />
<p class="expand mw-collapsible-content">SSC should also consider building an expert special group focussed on advanced analytics and experimental data trend discovery in Data Lakes. While the fundamental assumption behind the Data Lake concept is that everyone accessing a Data Lake is moderately to highly skilled at data manipulation and analysis, the reality is most are not. SSC should consider significant investment in training employees necessary skills, such as Data Science, Artificial Intelligence, Machine Learning, or Data Engineering.</p><br />
<p>SSC should be cognisant that there are significant overinflated expectations revolving around Data Lakes. Inflated expectations lead to vague and ambiguous use cases and increased chances of catastrophic failures. As a Service Provider, SSC must be strict in establishing clear goals for Data Lake provision efforts before deployment. SSC, should be wary of attempts to replace strategy development with infrastructure. A Data Lake can be a technology component that supports a data and analytics strategy, but it cannot replace that strategy.</p><br />
<p class="expand mw-collapsible-content">SSC should be concerned with the provision and running of the infrastructure, the departments themselves are responsible for the data they put in the Data Lake. However, as a Service Provider, SSC should monitor the Data Lake with regards to data governance, data lifecycle for data hygiene, and what is happening in the Data Lake overall. Depending on technology, SSC will need to be very clear on how to monitor activities in the Data Lakes it provides to the GC. </p><br />
<p class="expand mw-collapsible-content">SSC should consider a Data Lake implementation project as a way to introduce or reinvigorate a data management program by positioning data management capabilities as a prerequisite for a<br />
successful Data Lake. Data will need to be qualified before it hits the data lake, this can and should be done in a system of record first. In this way the data can be organizedto fit into the Data Lake implementation.<br />
</p><br />
<p class="expand mw-collapsible-content">SSC should create policies on how data is managed and cleaned in the Data Lake. Automated data governance technologies should be added to support advanced analytics. Standardizing on a specific type of governance tool is an issue which must be resolved. Additionally, planning for effective metadata management, considering metadata discovery, cataloguing and enterprise metadata management applied to Data Lake implementation is vital. Rigorous application of data discipline and data hygiene is needed. To combat this, SSC should use data management tools and create policies on how data is managed and cleaned in the Data Lake. The majority of Data Lake analysts will prefer to work with clean, enriched, and trusted data. However, data quality is relative to the task at hand. Lowquality data may be acceptable for low-impact analysis or distant forecasting, but unacceptable for tactical or high-impact analysis. SSC assessments should take this into account.</p><br />
<p>Design Data Lakes with the elements necessary to deliver reliable analytical results to a variety of data consumers. The goal is to increase cross-business usage in order to deliver advanced analytical insights. Build Data Lakes for specific business units or analytics applications, rather than try to implement some vague notion of a single enterprise Data Lake. However, alternative architectures, like data hubs, are often better fits for sharing data within an organization.</p><br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Datalakes&diff=11143Technology Trends/Datalakes2019-07-18T18:35:04Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Datalakes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Lac de Donnés]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[KubernetesIMG.png|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Datalakes</b> is a central system or repository of data that is stored in its natural/raw format. A datalake acts as a single store for all enterprise data. Data is transformed using machine learning, advanced,analytics, and visualization. Several forms of data can be hosed in a datalake. These include structured data from relational databases, unstructured data, semi-structured data, and binary data. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>In an ever-increasing hyperconnected world, corporations and businesses are struggling to deal with the responsibilities of storage, management and quick availability of raw data. To break these data challenges down further:</p><br />
<ul><br />
<li>Data comes in many different structures.<br />
<ul><br />
<li>Unstructured</li><br />
<li>Semi-Sturctured</li><br />
<li>Structured</li><br />
</ul></li><br />
<li>Data comes from many disparate sources.<br />
<ul><br />
<li>Enterprise Applications</li><br />
<li>Raw Files</li><br />
<li>Operation and Security Logs</li><br />
<li>Financial Transactions</li><br />
<li>Internet of Things (IoT) Devices and Network Sensors</li><br />
<li>Websites</li><br />
<li>Scientific Research</li><br />
</ul></li><br />
<li>Data sources are often geographically distributed to multiple locations<br />
<ul><br />
<li>Datacenters</li><br />
<li>Remote Offices</li><br />
<li>Mobile Devices</li><br />
</ul></li><br />
</ul><br />
<br />
<p>In an effort to resolve these data challenges, a new way of managing data was created which drove data oriented companies to invent a new data storage mechanism called a Data Lake.</p><br />
<p class="expand mw-collapsible-content">Data Lakes are characterized as: </p><br />
<ul class="expand mw-collapsible-content"><br />
<li>Collect Everything<ul><br />
<li>A Data Lake contains all data; raw sources over extended periods of time as well as any processed data.</li><br />
</ul></li><br />
<li>Dive in anywhere<ul><br />
<li>A Data Lake enables users across multiple business units to refine, explore and enrich data on their terms.</li><br />
</ul></li><br />
<li>Flexible Access<ul><br />
<li>o A Data Lake enables multiple data access patterns across a shared infrastructure: batch, interactive, online, search, in-memory and other processing engines.</li><br />
</ul></li><br />
</ul><br />
<br />
<p>Data Lakes are essentially a technology platform for holding data. Their value to the business is only realized when applying data science skills to the lake. </p><br />
<br />
<p>To summarize, usecases for Data Lakes are still being discovered. Cloud providers are making it easier to procure Data Lakes and today Data Lakes are primarily used by Research Institutions, Financial Services, Telecom, Media, Retail, Manufacturing, Healthcare, Pharma, Oi l& Gas and Governments.<br />
</p><br />
<br />
<br />
<br />
<h2>Technology Brief</h2><br />
<p class="expand mw-collapsible-content">The most popular implementation of a Data Lake is through the open source platform called Apache Hadoop. Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. Hadoop was originally created by researchers at Google as a storage method to handle the indexing of websites on the Internet; At that time it was called the Google File System. </p><br />
<p>A Data Lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.”</p><br />
<p class="expand mw-collapsible-content">Data can flow into the Data Lake by either batch processing or real-time processing of streaming data. Additionally, data itself is no longer restrained by initial schema decisions and can be exploited more freely by the enterprise. Rising above this repository is a set of capabilities that allow IT to provide Data and Analytics as a Service (DAaaS), in a supply-demand model. IT takes the role of the data provider (supplier), while business users (data scientists, business analysts) are consumers.</p><br />
<p>The DAaaS model enables users to self-serve their data and analytic needs. Users browse the lake’s data catalog (a Datapedia) to find and select the available data and fill a metaphorical “shopping cart” (effectively an analytics sandbox) with data to work with. Once access is provisioned, users can use the analytics tools of their choice to develop models and gain insights. Subsequently, users can publish analytical models or push refined or transformed data back into the Data Lake to share with the larger community.</p><br />
<p class="expand mw-collapsible-content">Although provisioning an analytic sandbox is a primary use, the Data Lake also has other applications. For example, the Data Lake can also be used to ingest raw data, curate the data, and apply Export-Transform-Load (ETL). This data can then be loaded to an Enterprise Data Warehouse. To take advantage of the flexibility provided by the Data Lake, organizations need to customize and configure the Data Lake to their specific requirements and domains.</p><br />
<h2>Industry Usage</h2><br />
<p>There are a variety of ways Data Lakes are being used in the industry:</p><br />
<ul><br />
<li><p><b>Ingestion of semi-structured and unstructured data sources (aka big data)</b>such as equipment readings, telemetry data, logs, streaming data, and so forth. A Data Lake is a great solution for storing IoT (Internet of Things) type of data which has traditionally been more difficult to store, and can support near real-time analysis. Optionally, you can also add structured data (i.e., extracted from a relational data source) to a Data Lake if your objective is a single repository of all data to be available via the lake.</p></li><br />
<li><p><b>Experimental analysis </b>of data before its value or purpose has been fully defined. Agility is important for every business these days, so a Data Lake can play an important role in "proof of value" type of situations because of the "ELT" approach discussed above.</p></li><br />
<li><p><b>Advanced analytics support. </b>A Data Lake is useful for data scientists and analysts to provision and experiment with data.</p></li><br />
<li><p><b>Archival and historical data storage. </b>Sometimes data is used infrequently, but does need to be available for analysis. A Data Lake strategy can be very valuable to support an active archive strategy.</p></li><br />
<li><p><b>Distributed processing </b>capabilities associated with a logical data warehouse.</p></li><br />
</ul><br />
<p class="expand mw-collapsible-content"><b>How TD Bank Made Its Data Lake More Usa</b></p><br />
<p class="expand mw-collapsible-content">[[https://www.datanami.com/2017/10/03/td-bank-made-data-lake-usable]]<br>Toronto-Dominion Bank (TD Bank) is one of the largest banks in North America, with 85,000 employees, more than 2,400 locations between Canada and the United States, and assets nearing $1 trillion. In 2014, the company decided to standardize how it warehouses data for various business intelligence and regulatory reporting functions. The company purchased a Hadoop distribution and set off to build a large cluster that could function as a centralized lake to store data originating from a variety of departments.</p><br />
<br />
<h2>Canadian Government Use</h2><br />
<p>In 2019, the Treasury Board of Canada Secretariat (TBS), partnered with Shared Services Canada and other departments, to identify a business lead to develop a Data Lake (a repository of raw data) service strategy so that the GC can take advantage of big data and market innovation to foster better analytics and promote horizontal data-sharing. </p><br />
<p class="expand mw-collapsible-content">Big data is the technology that stores and processes data and information in datasets that are so large or complex that traditional data processing applications can’t analyze them. Big data can make available almost limitless amounts of information, improving data-driven decision-making and expanding open data initiatives. Business intelligence involves creating, aggregating, analyzing and visualizing data to inform and facilitate business management and strategy. TBS, working with departments, will lead the development of requirements for an enterprise analytics platform.</p><br />
<p>Data Lake development in the GC is a more recent initiative. This is mainly due to the GC focussing resources on the implementation of cloud initiatives. However, there are some GC departments engaged in developing Data Lake environments in tandem to cloud initiatives.</p><br />
<p class="expand mw-collapsible-content">Notably, the Employment and Social Development Canada (ESDC) is preparing the installment of multiple Data Lakes in order to enable a Data Lake Ecosystem and Data Analytics and Machine Learning toolset. This will enable ESDC to share information horizontally both effectively and safely, while enabling a wide variety of data analytics capabilities. ESDC aims to maintain current data and analytics capabilities up-to-date while exploring new ones to mitigate gaps and continuously evolve our services to meet client’s needs. </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="expand mw-collapsible-content">There are three common value propositions for pursuing Data Lakes. 1) It can provide an easy and accessible way to obtain data faster; 2) It can create a singular inflow point of data to help connect and merge information silos in an organization; and 3) It can provide an experimental environment for experienced data scientists to enable new analytical insights.</p><br />
<p> </p><br />
<p class="inline">Data Lakes can provide data to consumers more quickly by offering data in a more raw and easily accessible form. Data is stored in its native form with little to no processing, it is optimized to store vast amounts of data in their native formats. By allowing the data to remain in its native format, a much timelier stream of data is available for unlimited queries and analysis. A Data Lake can help data consumers bypass strict data retrieval and data structured applications such as a data warehouse and/or data mart. This has the effect of improving a business’ data flexibility.</p><p class="expand inline mw-collapsible-content">Some companies have in fact used Data Lakes to replace existing warehousing environments where implementing a new data warehouse is more cost prohibitive. A Data Lake can contain unrefined data, this is helpful when either a business data structure is unknown, or when a data consumer requires access to the data quickly. </p><br />
<p> </p><br />
<p class="inline">A Data Lake is not a single source of truth. A Data Lake is a central location in which data converges from all data sources and is stored, regardless of the data formatting. </p><p class="expand inline mw-collapsible-content">As a singular point for the inflow of data, sections of a business can pool their information together in the Data Lake and increase the sharing of information with other parts of the organization. In this way everyone in the organization has access to the data. A Data Lake can increase the horizontal data sharing within an organization by creating this singular data inflow point. Using a variety of storage and processing tools analysts can extract data value quickly in order to inform key business decisions.</p><br />
<p> </p><br />
<p class="expand inline mw-collapsible-content">A Data Lake is optimized for exploration and provides an experimental environment for experienced data scientists to uncover new insights from data. Analysts can overlay context on the data to extract value. All organizations want to increase analytics and operational agility.</p><p class="inline">The Data Lake architectural approach can store large volumes of data, this can be a way in which cross-cutting teams can pool their data in a central location and by complementing their systems of record with systems of insight. </p><p class="expand inline mw-collapsible-content">Data Lakes present the most potential benefits for experienced and competant data scientists. </p><p class="inline">Having structured, unstructured and semistructured data, usually in the same data set, can contain business, predictive, and prescriptive insights previously not possible from a structured platform as observed in data warehouses and data marts.</p><br />
<br />
<h4>Challenges</h4><br />
<p>Although Data Lake technology has many benefits for organizations dealing with big data it has its own challenges. For example:</p><br />
<b>Data Governance and Semantic Issues</b><br />
<p class="expand inline mw-collapsible-content">The biggest challenge for Data Lakes is to resolve assorted data governance requirements in a single centralized data platform. Data Lakes fail mostly when they lack governance, self-disciplined users, and a rational data flow.</p><p class="inline">Often, Data Lake implementations are focused on storing data instead of managing the data. Data Lakes are not optimized for semantic enforcement or consistency. They are made for semantic flexibility, to allow anyone to provide context to data if they have the skills to do so. </p><br />
<p>Putting data in the same place does not remove it’s ambiguity or meaning. Data Lakes provide unconstrained, “no compromises” storage model environment without the data governance assurances common to data warehouses or data marts. Proper meta data is essential for a Data Lake, without appropriate meta data the Data Lake will not work as intended. It is beneficial to think of meta data as the fish finder in the Data Lake.</p><br />
<b class="expand mw-collapsible-content">Lack of Quality and Trust in Data</b><br />
<p class="expand mw-collapsible-content">Data quality and trust in the data is a perennial issue for many organizations. Although data discovery tools can apply Machine Learning across related datasets from multiple data sources to identify anomalies (incorrect values, missing values, duplicates and outdated data), quality and trustworthiness of data continue to be an issue for Data Lakes who can easily become data dumping grounds. Some data is more accurate than others. This can present a real problem for anyone using multiple data sets and making decisions based upon analysis conducted with data of varying degrees of quality.</p><br />
<b>Data Swamps, Performance, and Flexibility Challenges</b><br />
<p class="expand inline mw-collapsible-content">Data stored in Data Lakes can sometimes become muddy when good data is mixed with bad data. Data Lake infrastructure is meant to store and process large amounts of data, usually in massive data files. </p><p class="inline">. A Data Lake is not optimized for a high number of users or diverse and simultaneous workloads due to intensive query tasks. This can result in performance degradation and failures are common when running extractions, transformations, and loading tasks all at the same time. On-premises Data Lakes face other performance challenges in that they have a static configuration. </p><br />
<b class="expand mw-collapsible-content">Data Hoarding and Storage Capacity</b><br />
<p class="expand mw-collapsible-content">Data stored in Data Lakes may actually never be used in production and stay unused indefinitely in the Data Lake. By storing massive amounts of historical data, the infinite Data Lake may skew analysis with data that is no longer relevant to the priorities of the business. In keeping the historical data the metadata describing it must be understood as well. This decreases the performance of the Data Lake by increasing the overall workload of employees to clean the datasets no longer in use for analysis.</p><br />
<p class="expand mw-collapsible-content">Storing increasingly massive amounts of data for an unlimited time will also lead to scalability and cost challenges. Scalability challenges are less of a risk in public cloud environments, but cost remains a factor. On-premises Data Lakes are more susceptible to cost challenges. This is because their cluster nodes require all three dimensions of computing (storage, memory and processing). Organizations of all kinds generate massive amounts of data (including meta data) and it is increasing exponentially.</p><br />
<p>The storage capacity of all this data (and future data) will be an ongoing challenge and one that will require constant management. While Data Lakes can and will be stored on the cloud, SSC as cloud broker for the GC will need to provide the appropriate infrastructure and scalability to clients.</p><br />
<b class="expand mw-collapsible-content">Advanced Users Required</b><br />
<p class="expand mw-collapsible-content">Data Lakes are not a platform to be explored by everyone. Data Lakes present an unrefined view of data that usually only the most highly skilled analysts are able to explore and engage in data refinement independent of any other formal system-of-record such as a data warehouse. </p><br />
<p class="expand mw-collapsible-content">Not just anyone in an organization is data-literate enough to derive value from large amounts of raw or uncurated data. The reality is only a handful of staff are skilled enough to navigate a Data Lake. Since Data Lakes store raw data their business value is entirely determined by the skills of Data Lake users. These skills are often lacking in an organization.</p><br />
<b>Data Security</b><br />
<p class="inline"></p>Data in a Data Lake lacks standard security protection with a relational database management system or an enterprise database. In practice, this means that the data is unencrypted and lacks access control.<p class="expand inline mw-collapsible-content">. Security is not just a binary solution. We have varying degrees of security (unclassified, secret, top secret, etc.) and all of which require different approaches. This will inevitably present challenges with the successful use of data from Data Lakes.To combat this, organizations will have to embrace a new security framework to be compatable with Data Lakes and Data Scientists.</p><br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Datalakes&diff=11137Technology Trends/Datalakes2019-07-18T17:56:03Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Datalakes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Lac de Donnés]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[KubernetesIMG.png|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Datalakes</b> is a central system or repository of data that is stored in its natural/raw format. A datalake acts as a single store for all enterprise data. Data is transformed using machine learning, advanced,analytics, and visualization. Several forms of data can be hosed in a datalake. These include structured data from relational databases, unstructured data, semi-structured data, and binary data. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>In an ever-increasing hyperconnected world, corporations and businesses are struggling to deal with the responsibilities of storage, management and quick availability of raw data. To break these data challenges down further:</p><br />
<ul><br />
<li>Data comes in many different structures.<br />
<ul><br />
<li>Unstructured</li><br />
<li>Semi-Sturctured</li><br />
<li>Structured</li><br />
</ul></li><br />
<li>Data comes from many disparate sources.<br />
<ul><br />
<li>Enterprise Applications</li><br />
<li>Raw Files</li><br />
<li>Operation and Security Logs</li><br />
<li>Financial Transactions</li><br />
<li>Internet of Things (IoT) Devices and Network Sensors</li><br />
<li>Websites</li><br />
<li>Scientific Research</li><br />
</ul></li><br />
<li>Data sources are often geographically distributed to multiple locations<br />
<ul><br />
<li>Datacenters</li><br />
<li>Remote Offices</li><br />
<li>Mobile Devices</li><br />
</ul></li><br />
</ul><br />
<br />
<p>In an effort to resolve these data challenges, a new way of managing data was created which drove data oriented companies to invent a new data storage mechanism called a Data Lake.</p><br />
<p class="expand mw-collapsible-content">Data Lakes are characterized as: </p><br />
<ul class="expand mw-collapsible-content"><br />
<li>Collect Everything<ul><br />
<li>A Data Lake contains all data; raw sources over extended periods of time as well as any processed data.</li><br />
</ul></li><br />
<li>Dive in anywhere<ul><br />
<li>A Data Lake enables users across multiple business units to refine, explore and enrich data on their terms.</li><br />
</ul></li><br />
<li>Flexible Access<ul><br />
<li>o A Data Lake enables multiple data access patterns across a shared infrastructure: batch, interactive, online, search, in-memory and other processing engines.</li><br />
</ul></li><br />
</ul><br />
<br />
<p>Data Lakes are essentially a technology platform for holding data. Their value to the business is only realized when applying data science skills to the lake. </p><br />
<br />
<p>To summarize, usecases for Data Lakes are still being discovered. Cloud providers are making it easier to procure Data Lakes and today Data Lakes are primarily used by Research Institutions, Financial Services, Telecom, Media, Retail, Manufacturing, Healthcare, Pharma, Oi l& Gas and Governments.<br />
</p><br />
<br />
<br />
<br />
<h2>Technology Brief</h2><br />
<p class="expand mw-collapsible-content">The most popular implementation of a Data Lake is through the open source platform called Apache Hadoop. Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. Hadoop was originally created by researchers at Google as a storage method to handle the indexing of websites on the Internet; At that time it was called the Google File System. </p><br />
<p>A Data Lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.”</p><br />
<p class="expand mw-collapsible-content">Data can flow into the Data Lake by either batch processing or real-time processing of streaming data. Additionally, data itself is no longer restrained by initial schema decisions and can be exploited more freely by the enterprise. Rising above this repository is a set of capabilities that allow IT to provide Data and Analytics as a Service (DAaaS), in a supply-demand model. IT takes the role of the data provider (supplier), while business users (data scientists, business analysts) are consumers.</p><br />
<p>The DAaaS model enables users to self-serve their data and analytic needs. Users browse the lake’s data catalog (a Datapedia) to find and select the available data and fill a metaphorical “shopping cart” (effectively an analytics sandbox) with data to work with. Once access is provisioned, users can use the analytics tools of their choice to develop models and gain insights. Subsequently, users can publish analytical models or push refined or transformed data back into the Data Lake to share with the larger community.</p><br />
<p class="expand mw-collapsible-content">Although provisioning an analytic sandbox is a primary use, the Data Lake also has other applications. For example, the Data Lake can also be used to ingest raw data, curate the data, and apply Export-Transform-Load (ETL). This data can then be loaded to an Enterprise Data Warehouse. To take advantage of the flexibility provided by the Data Lake, organizations need to customize and configure the Data Lake to their specific requirements and domains.</p><br />
<h2>Industry Usage</h2><br />
<p>There are a variety of ways Data Lakes are being used in the industry:</p><br />
<ul><br />
<li><p><b>Ingestion of semi-structured and unstructured data sources (aka big data)</b>such as equipment readings, telemetry data, logs, streaming data, and so forth. A Data Lake is a great solution for storing IoT (Internet of Things) type of data which has traditionally been more difficult to store, and can support near real-time analysis. Optionally, you can also add structured data (i.e., extracted from a relational data source) to a Data Lake if your objective is a single repository of all data to be available via the lake.</p></li><br />
<li><p><b>Experimental analysis </b>of data before its value or purpose has been fully defined. Agility is important for every business these days, so a Data Lake can play an important role in "proof of value" type of situations because of the "ELT" approach discussed above.</p></li><br />
<li><p><b>Advanced analytics support. </b>A Data Lake is useful for data scientists and analysts to provision and experiment with data.</p></li><br />
<li><p><b>Archival and historical data storage. </b>Sometimes data is used infrequently, but does need to be available for analysis. A Data Lake strategy can be very valuable to support an active archive strategy.</p></li><br />
<li><p><b>Distributed processing </b>capabilities associated with a logical data warehouse.</p></li><br />
</ul><br />
<p class="expand mw-collapsible-content"><b>How TD Bank Made Its Data Lake More Usa</b></p><br />
<p class="expand mw-collapsible-content">[[https://www.datanami.com/2017/10/03/td-bank-made-data-lake-usable]]<br>Toronto-Dominion Bank (TD Bank) is one of the largest banks in North America, with 85,000 employees, more than 2,400 locations between Canada and the United States, and assets nearing $1 trillion. In 2014, the company decided to standardize how it warehouses data for various business intelligence and regulatory reporting functions. The company purchased a Hadoop distribution and set off to build a large cluster that could function as a centralized lake to store data originating from a variety of departments.</p><br />
<br />
<h2>Canadian Government Use</h2><br />
<p>In 2019, the Treasury Board of Canada Secretariat (TBS), partnered with Shared Services Canada and other departments, to identify a business lead to develop a Data Lake (a repository of raw data) service strategy so that the GC can take advantage of big data and market innovation to foster better analytics and promote horizontal data-sharing. </p><br />
<p class="expand mw-collapsible-content">Big data is the technology that stores and processes data and information in datasets that are so large or complex that traditional data processing applications can’t analyze them. Big data can make available almost limitless amounts of information, improving data-driven decision-making and expanding open data initiatives. Business intelligence involves creating, aggregating, analyzing and visualizing data to inform and facilitate business management and strategy. TBS, working with departments, will lead the development of requirements for an enterprise analytics platform.</p><br />
<p>Data Lake development in the GC is a more recent initiative. This is mainly due to the GC focussing resources on the implementation of cloud initiatives. However, there are some GC departments engaged in developing Data Lake environments in tandem to cloud initiatives.</p><br />
<p class="expand mw-collapsible-content">Notably, the Employment and Social Development Canada (ESDC) is preparing the installment of multiple Data Lakes in order to enable a Data Lake Ecosystem and Data Analytics and Machine Learning toolset. This will enable ESDC to share information horizontally both effectively and safely, while enabling a wide variety of data analytics capabilities. ESDC aims to maintain current data and analytics capabilities up-to-date while exploring new ones to mitigate gaps and continuously evolve our services to meet client’s needs. </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="expand mw-collapsible-content">There are three common value propositions for pursuing Data Lakes. 1) It can provide an easy and accessible way to obtain data faster; 2) It can create a singular inflow point of data to help connect and merge information silos in an organization; and 3) It can provide an experimental environment for experienced data scientists to enable new analytical insights.</p><br />
<p> </p><br />
<p class="inline">Data Lakes can provide data to consumers more quickly by offering data in a more raw and easily accessible form. Data is stored in its native form with little to no processing, it is optimized to store vast amounts of data in their native formats. By allowing the data to remain in its native format, a much timelier stream of data is available for unlimited queries and analysis. A Data Lake can help data consumers bypass strict data retrieval and data structured applications such as a data warehouse and/or data mart. This has the effect of improving a business’ data flexibility.</p><p class="expand inline mw-collapsible-content">Some companies have in fact used Data Lakes to replace existing warehousing environments where implementing a new data warehouse is more cost prohibitive. A Data Lake can contain unrefined data, this is helpful when either a business data structure is unknown, or when a data consumer requires access to the data quickly. </p><br />
<p> </p><br />
<p class="inline">A Data Lake is not a single source of truth. A Data Lake is a central location in which data converges from all data sources and is stored, regardless of the data formatting. </p><p class="expand inline mw-collapsible-content">As a singular point for the inflow of data, sections of a business can pool their information together in the Data Lake and increase the sharing of information with other parts of the organization. In this way everyone in the organization has access to the data. A Data Lake can increase the horizontal data sharing within an organization by creating this singular data inflow point. Using a variety of storage and processing tools analysts can extract data value quickly in order to inform key business decisions.</p><br />
<p> </p><br />
<p class="expand inline mw-collapsible-content">A Data Lake is optimized for exploration and provides an experimental environment for experienced data scientists to uncover new insights from data. Analysts can overlay context on the data to extract value. All organizations want to increase analytics and operational agility.</p><p class="inline">The Data Lake architectural approach can store large volumes of data, this can be a way in which cross-cutting teams can pool their data in a central location and by complementing their systems of record with systems of insight. </p><p class="expand inline mw-collapsible-content">Data Lakes present the most potential benefits for experienced and competant data scientists. </p><p class="inline">Having structured, unstructured and semistructured data, usually in the same data set, can contain business, predictive, and prescriptive insights previously not possible from a structured platform as observed in data warehouses and data marts.</p><br />
<br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service<ref>Tsang, Daisy. (February 12th, 2018). Kubernetes vs. Docker: What Does It Really Mean? Sumo Logic. 2019. Retrieved 16-May-2019 from: <i>[https://www.sumologic.com/blog/kubernetes-vs-docker/ ]</i></ref>. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Datalakes&diff=11136Technology Trends/Datalakes2019-07-18T17:53:45Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Datalakes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Lac de Donnés]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[KubernetesIMG.png|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Datalakes</b> is a central system or repository of data that is stored in its natural/raw format. A datalake acts as a single store for all enterprise data. Data is transformed using machine learning, advanced,analytics, and visualization. Several forms of data can be hosed in a datalake. These include structured data from relational databases, unstructured data, semi-structured data, and binary data. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>In an ever-increasing hyperconnected world, corporations and businesses are struggling to deal with the responsibilities of storage, management and quick availability of raw data. To break these data challenges down further:</p><br />
<ul><br />
<li>Data comes in many different structures.<br />
<ul><br />
<li>Unstructured</li><br />
<li>Semi-Sturctured</li><br />
<li>Structured</li><br />
</ul></li><br />
<li>Data comes from many disparate sources.<br />
<ul><br />
<li>Enterprise Applications</li><br />
<li>Raw Files</li><br />
<li>Operation and Security Logs</li><br />
<li>Financial Transactions</li><br />
<li>Internet of Things (IoT) Devices and Network Sensors</li><br />
<li>Websites</li><br />
<li>Scientific Research</li><br />
</ul></li><br />
<li>Data sources are often geographically distributed to multiple locations<br />
<ul><br />
<li>Datacenters</li><br />
<li>Remote Offices</li><br />
<li>Mobile Devices</li><br />
</ul></li><br />
</ul><br />
<br />
<p>In an effort to resolve these data challenges, a new way of managing data was created which drove data oriented companies to invent a new data storage mechanism called a Data Lake.</p><br />
<p class="expand mw-collapsible-content">Data Lakes are characterized as: </p><br />
<ul class="expand mw-collapsible-content"><br />
<li>Collect Everything<ul><br />
<li>A Data Lake contains all data; raw sources over extended periods of time as well as any processed data.</li><br />
</ul></li><br />
<li>Dive in anywhere<ul><br />
<li>A Data Lake enables users across multiple business units to refine, explore and enrich data on their terms.</li><br />
</ul></li><br />
<li>Flexible Access<ul><br />
<li>o A Data Lake enables multiple data access patterns across a shared infrastructure: batch, interactive, online, search, in-memory and other processing engines.</li><br />
</ul></li><br />
</ul><br />
<br />
<p>Data Lakes are essentially a technology platform for holding data. Their value to the business is only realized when applying data science skills to the lake. </p><br />
<br />
<p>To summarize, usecases for Data Lakes are still being discovered. Cloud providers are making it easier to procure Data Lakes and today Data Lakes are primarily used by Research Institutions, Financial Services, Telecom, Media, Retail, Manufacturing, Healthcare, Pharma, Oi l& Gas and Governments.<br />
</p><br />
<br />
<br />
<br />
<h2>Technology Brief</h2><br />
<p class="expand mw-collapsible-content">The most popular implementation of a Data Lake is through the open source platform called Apache Hadoop. Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. Hadoop was originally created by researchers at Google as a storage method to handle the indexing of websites on the Internet; At that time it was called the Google File System. </p><br />
<p>A Data Lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.”</p><br />
<p class="expand mw-collapsible-content">Data can flow into the Data Lake by either batch processing or real-time processing of streaming data. Additionally, data itself is no longer restrained by initial schema decisions and can be exploited more freely by the enterprise. Rising above this repository is a set of capabilities that allow IT to provide Data and Analytics as a Service (DAaaS), in a supply-demand model. IT takes the role of the data provider (supplier), while business users (data scientists, business analysts) are consumers.</p><br />
<p>The DAaaS model enables users to self-serve their data and analytic needs. Users browse the lake’s data catalog (a Datapedia) to find and select the available data and fill a metaphorical “shopping cart” (effectively an analytics sandbox) with data to work with. Once access is provisioned, users can use the analytics tools of their choice to develop models and gain insights. Subsequently, users can publish analytical models or push refined or transformed data back into the Data Lake to share with the larger community.</p><br />
<p class="expand mw-collapsible-content">Although provisioning an analytic sandbox is a primary use, the Data Lake also has other applications. For example, the Data Lake can also be used to ingest raw data, curate the data, and apply Export-Transform-Load (ETL). This data can then be loaded to an Enterprise Data Warehouse. To take advantage of the flexibility provided by the Data Lake, organizations need to customize and configure the Data Lake to their specific requirements and domains.</p><br />
<h2>Industry Usage</h2><br />
<p>There are a variety of ways Data Lakes are being used in the industry:</p><br />
<ul><br />
<li><p><b>Ingestion of semi-structured and unstructured data sources (aka big data)</b>such as equipment readings, telemetry data, logs, streaming data, and so forth. A Data Lake is a great solution for storing IoT (Internet of Things) type of data which has traditionally been more difficult to store, and can support near real-time analysis. Optionally, you can also add structured data (i.e., extracted from a relational data source) to a Data Lake if your objective is a single repository of all data to be available via the lake.</p></li><br />
<li><p><b>Experimental analysis </b>of data before its value or purpose has been fully defined. Agility is important for every business these days, so a Data Lake can play an important role in "proof of value" type of situations because of the "ELT" approach discussed above.</p></li><br />
<li><p><b>Advanced analytics support. </b>A Data Lake is useful for data scientists and analysts to provision and experiment with data.</p></li><br />
<li><p><b>Archival and historical data storage. </b>Sometimes data is used infrequently, but does need to be available for analysis. A Data Lake strategy can be very valuable to support an active archive strategy.</p></li><br />
<li><p><b>Distributed processing </b>capabilities associated with a logical data warehouse.</p></li><br />
</ul><br />
<p class="expand mw-collapsible-content"><b>How TD Bank Made Its Data Lake More Usa</b></p><br />
<p class="expand mw-collapsible-content">[[https://www.datanami.com/2017/10/03/td-bank-made-data-lake-usable]]<br>Toronto-Dominion Bank (TD Bank) is one of the largest banks in North America, with 85,000 employees, more than 2,400 locations between Canada and the United States, and assets nearing $1 trillion. In 2014, the company decided to standardize how it warehouses data for various business intelligence and regulatory reporting functions. The company purchased a Hadoop distribution and set off to build a large cluster that could function as a centralized lake to store data originating from a variety of departments.</p><br />
<br />
<h2>Canadian Government Use</h2><br />
<p>In 2019, the Treasury Board of Canada Secretariat (TBS), partnered with Shared Services Canada and other departments, to identify a business lead to develop a Data Lake (a repository of raw data) service strategy so that the GC can take advantage of big data and market innovation to foster better analytics and promote horizontal data-sharing. </p><br />
<p class="expand mw-collapsible-content">Big data is the technology that stores and processes data and information in datasets that are so large or complex that traditional data processing applications can’t analyze them. Big data can make available almost limitless amounts of information, improving data-driven decision-making and expanding open data initiatives. Business intelligence involves creating, aggregating, analyzing and visualizing data to inform and facilitate business management and strategy. TBS, working with departments, will lead the development of requirements for an enterprise analytics platform.</p><br />
<p>Data Lake development in the GC is a more recent initiative. This is mainly due to the GC focussing resources on the implementation of cloud initiatives. However, there are some GC departments engaged in developing Data Lake environments in tandem to cloud initiatives.</p><br />
<p class="expand mw-collapsible-content">Notably, the Employment and Social Development Canada (ESDC) is preparing the installment of multiple Data Lakes in order to enable a Data Lake Ecosystem and Data Analytics and Machine Learning toolset. This will enable ESDC to share information horizontally both effectively and safely, while enabling a wide variety of data analytics capabilities. ESDC aims to maintain current data and analytics capabilities up-to-date while exploring new ones to mitigate gaps and continuously evolve our services to meet client’s needs. </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="expand mw-collapsible-content">There are three common value propositions for pursuing Data Lakes. 1) It can provide an easy and accessible way to obtain data faster; 2) It can create a singular inflow point of data to help connect and merge information silos in an organization; and 3) It can provide an experimental environment for experienced data scientists to enable new analytical insights.</p><br />
<p><br />
<p class="inline">Data Lakes can provide data to consumers more quickly by offering data in a more raw and easily accessible form. Data is stored in its native form with little to no processing, it is optimized to store vast amounts of data in their native formats. By allowing the data to remain in its native format, a much timelier stream of data is available for unlimited queries and analysis. A Data Lake can help data consumers bypass strict data retrieval and data structured applications such as a data warehouse and/or data mart. This has the effect of improving a business’ data flexibility.</p><p class="expand inline mw-collapsible-content">Some companies have in fact used Data Lakes to replace existing warehousing environments where implementing a new data warehouse is more cost prohibitive. A Data Lake can contain unrefined data, this is helpful when either a business data structure is unknown, or when a data consumer requires access to the data quickly. </p><p/p><br />
<br />
<p><p class="inline">A Data Lake is not a single source of truth. A Data Lake is a central location in which data converges from all data sources and is stored, regardless of the data formatting. </p><p class="expand inline mw-collapsible-content">As a singular point for the inflow of data, sections of a business can pool their information together in the Data Lake and increase the sharing of information with other parts of the organization. In this way everyone in the organization has access to the data. A Data Lake can increase the horizontal data sharing within an organization by creating this singular data inflow point. Using a variety of storage and processing tools analysts can extract data value quickly in order to inform key business decisions.</p></p><br />
<br />
<p><p class="expand inline mw-collapsible-content">A Data Lake is optimized for exploration and provides an experimental environment for experienced data scientists to uncover new insights from data. Analysts can overlay context on the data to extract value. All organizations want to increase analytics and operational agility.</p><p class="inline">The Data Lake architectural approach can store large volumes of data, this can be a way in which cross-cutting teams can pool their data in a central location and by complementing their systems of record with systems of insight. </p><p class="expand inline mw-collapsible-content">Data Lakes present the most potential benefits for experienced and competant data scientists. </p><p class="inline">Having structured, unstructured and semistructured data, usually in the same data set, can contain business, predictive, and prescriptive insights previously not possible from a structured platform as observed in data warehouses and data marts.</p></p><br />
<br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service<ref>Tsang, Daisy. (February 12th, 2018). Kubernetes vs. Docker: What Does It Really Mean? Sumo Logic. 2019. Retrieved 16-May-2019 from: <i>[https://www.sumologic.com/blog/kubernetes-vs-docker/ ]</i></ref>. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Datalakes&diff=11135Technology Trends/Datalakes2019-07-18T17:46:39Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Datalakes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Lac de Donnés]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[KubernetesIMG.png|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Datalakes</b> is a central system or repository of data that is stored in its natural/raw format. A datalake acts as a single store for all enterprise data. Data is transformed using machine learning, advanced,analytics, and visualization. Several forms of data can be hosed in a datalake. These include structured data from relational databases, unstructured data, semi-structured data, and binary data. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>In an ever-increasing hyperconnected world, corporations and businesses are struggling to deal with the responsibilities of storage, management and quick availability of raw data. To break these data challenges down further:</p><br />
<ul><br />
<li>Data comes in many different structures.<br />
<ul><br />
<li>Unstructured</li><br />
<li>Semi-Sturctured</li><br />
<li>Structured</li><br />
</ul></li><br />
<li>Data comes from many disparate sources.<br />
<ul><br />
<li>Enterprise Applications</li><br />
<li>Raw Files</li><br />
<li>Operation and Security Logs</li><br />
<li>Financial Transactions</li><br />
<li>Internet of Things (IoT) Devices and Network Sensors</li><br />
<li>Websites</li><br />
<li>Scientific Research</li><br />
</ul></li><br />
<li>Data sources are often geographically distributed to multiple locations<br />
<ul><br />
<li>Datacenters</li><br />
<li>Remote Offices</li><br />
<li>Mobile Devices</li><br />
</ul></li><br />
</ul><br />
<br />
<p>In an effort to resolve these data challenges, a new way of managing data was created which drove data oriented companies to invent a new data storage mechanism called a Data Lake.</p><br />
<p class="expand mw-collapsible-content">Data Lakes are characterized as: </p><br />
<ul class="expand mw-collapsible-content"><br />
<li>Collect Everything<ul><br />
<li>A Data Lake contains all data; raw sources over extended periods of time as well as any processed data.</li><br />
</ul></li><br />
<li>Dive in anywhere<ul><br />
<li>A Data Lake enables users across multiple business units to refine, explore and enrich data on their terms.</li><br />
</ul></li><br />
<li>Flexible Access<ul><br />
<li>o A Data Lake enables multiple data access patterns across a shared infrastructure: batch, interactive, online, search, in-memory and other processing engines.</li><br />
</ul></li><br />
</ul><br />
<br />
<p>Data Lakes are essentially a technology platform for holding data. Their value to the business is only realized when applying data science skills to the lake. </p><br />
<br />
<p>To summarize, usecases for Data Lakes are still being discovered. Cloud providers are making it easier to procure Data Lakes and today Data Lakes are primarily used by Research Institutions, Financial Services, Telecom, Media, Retail, Manufacturing, Healthcare, Pharma, Oi l& Gas and Governments.<br />
</p><br />
<br />
<br />
<br />
<h2>Technology Brief</h2><br />
<p class="expand mw-collapsible-content">The most popular implementation of a Data Lake is through the open source platform called Apache Hadoop. Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. Hadoop was originally created by researchers at Google as a storage method to handle the indexing of websites on the Internet; At that time it was called the Google File System. </p><br />
<p>A Data Lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.”</p><br />
<p class="expand mw-collapsible-content">Data can flow into the Data Lake by either batch processing or real-time processing of streaming data. Additionally, data itself is no longer restrained by initial schema decisions and can be exploited more freely by the enterprise. Rising above this repository is a set of capabilities that allow IT to provide Data and Analytics as a Service (DAaaS), in a supply-demand model. IT takes the role of the data provider (supplier), while business users (data scientists, business analysts) are consumers.</p><br />
<p>The DAaaS model enables users to self-serve their data and analytic needs. Users browse the lake’s data catalog (a Datapedia) to find and select the available data and fill a metaphorical “shopping cart” (effectively an analytics sandbox) with data to work with. Once access is provisioned, users can use the analytics tools of their choice to develop models and gain insights. Subsequently, users can publish analytical models or push refined or transformed data back into the Data Lake to share with the larger community.</p><br />
<p class="expand mw-collapsible-content">Although provisioning an analytic sandbox is a primary use, the Data Lake also has other applications. For example, the Data Lake can also be used to ingest raw data, curate the data, and apply Export-Transform-Load (ETL). This data can then be loaded to an Enterprise Data Warehouse. To take advantage of the flexibility provided by the Data Lake, organizations need to customize and configure the Data Lake to their specific requirements and domains.</p><br />
<h2>Industry Usage</h2><br />
<p>There are a variety of ways Data Lakes are being used in the industry:</p><br />
<ul><br />
<li><p><b>Ingestion of semi-structured and unstructured data sources (aka big data)</b>such as equipment readings, telemetry data, logs, streaming data, and so forth. A Data Lake is a great solution for storing IoT (Internet of Things) type of data which has traditionally been more difficult to store, and can support near real-time analysis. Optionally, you can also add structured data (i.e., extracted from a relational data source) to a Data Lake if your objective is a single repository of all data to be available via the lake.</p></li><br />
<li><p><b>Experimental analysis </b>of data before its value or purpose has been fully defined. Agility is important for every business these days, so a Data Lake can play an important role in "proof of value" type of situations because of the "ELT" approach discussed above.</p></li><br />
<li><p><b>Advanced analytics support. </b>A Data Lake is useful for data scientists and analysts to provision and experiment with data.</p></li><br />
<li><p><b>Archival and historical data storage. </b>Sometimes data is used infrequently, but does need to be available for analysis. A Data Lake strategy can be very valuable to support an active archive strategy.</p></li><br />
<li><p><b>Distributed processing </b>capabilities associated with a logical data warehouse.</p></li><br />
</ul><br />
<p class="expand mw-collapsible-content"><b>How TD Bank Made Its Data Lake More Usa</b></p><br />
<p class="expand mw-collapsible-content">[[https://www.datanami.com/2017/10/03/td-bank-made-data-lake-usable]]<br>Toronto-Dominion Bank (TD Bank) is one of the largest banks in North America, with 85,000 employees, more than 2,400 locations between Canada and the United States, and assets nearing $1 trillion. In 2014, the company decided to standardize how it warehouses data for various business intelligence and regulatory reporting functions. The company purchased a Hadoop distribution and set off to build a large cluster that could function as a centralized lake to store data originating from a variety of departments.</p><br />
<br />
<h2>Canadian Government Use</h2><br />
<p>In 2019, the Treasury Board of Canada Secretariat (TBS), partnered with Shared Services Canada and other departments, to identify a business lead to develop a Data Lake (a repository of raw data) service strategy so that the GC can take advantage of big data and market innovation to foster better analytics and promote horizontal data-sharing. </p><br />
<p class="expand mw-collapsible-content">Big data is the technology that stores and processes data and information in datasets that are so large or complex that traditional data processing applications can’t analyze them. Big data can make available almost limitless amounts of information, improving data-driven decision-making and expanding open data initiatives. Business intelligence involves creating, aggregating, analyzing and visualizing data to inform and facilitate business management and strategy. TBS, working with departments, will lead the development of requirements for an enterprise analytics platform.</p><br />
<p>Data Lake development in the GC is a more recent initiative. This is mainly due to the GC focussing resources on the implementation of cloud initiatives. However, there are some GC departments engaged in developing Data Lake environments in tandem to cloud initiatives.</p><br />
<p class="expand mw-collapsible-content">Notably, the Employment and Social Development Canada (ESDC) is preparing the installment of multiple Data Lakes in order to enable a Data Lake Ecosystem and Data Analytics and Machine Learning toolset. This will enable ESDC to share information horizontally both effectively and safely, while enabling a wide variety of data analytics capabilities. ESDC aims to maintain current data and analytics capabilities up-to-date while exploring new ones to mitigate gaps and continuously evolve our services to meet client’s needs. </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="expand mw-collapsible-content">There are three common value propositions for pursuing Data Lakes. 1) It can provide an easy and accessible way to obtain data faster; 2) It can create a singular inflow point of data to help connect and merge information silos in an organization; and 3) It can provide an experimental environment for experienced data scientists to enable new analytical insights.</p><br />
<br><br />
<p class="inline">Data Lakes can provide data to consumers more quickly by offering data in a more raw and easily accessible form. Data is stored in its native form with little to no processing, it is optimized to store vast amounts of data in their native formats. By allowing the data to remain in its native format, a much timelier stream of data is available for unlimited queries and analysis. A Data Lake can help data consumers bypass strict data retrieval and data structured applications such as a data warehouse and/or data mart. This has the effect of improving a business’ data flexibility.</p><p class="expand inline mw-collapsible-content">Some companies have in fact used Data Lakes to replace existing warehousing environments where implementing a new data warehouse is more cost prohibitive. A Data Lake can contain unrefined data, this is helpful when either a business data structure is unknown, or when a data consumer requires access to the data quickly. </p><br />
<br><br />
<p class="inline">A Data Lake is not a single source of truth. A Data Lake is a central location in which data converges from all data sources and is stored, regardless of the data formatting. </p><p class="expand inline mw-collapsible-content">As a singular point for the inflow of data, sections of a business can pool their information together in the Data Lake and increase the sharing of information with other parts of the organization. In this way everyone in the organization has access to the data. A Data Lake can increase the horizontal data sharing within an organization by creating this singular data inflow point. Using a variety of storage and processing tools analysts can extract data value quickly in order to inform key business decisions.</p><br />
<br><br />
<p class="expand inline mw-collapsible-content">A Data Lake is optimized for exploration and provides an experimental environment for experienced data scientists to uncover new insights from data. Analysts can overlay context on the data to extract value. All organizations want to increase analytics and operational agility.</p><p class="inline">The Data Lake architectural approach can store large volumes of data, this can be a way in which cross-cutting teams can pool their data in a central location and by complementing their systems of record with systems of insight. </p><p class="expand inline mw-collapsible-content">Data Lakes present the most potential benefits for experienced and competant data scientists. </p><p class="inline">Having structured, unstructured and semistructured data, usually in the same data set, can contain business, predictive, and prescriptive insights previously not possible from a structured platform as observed in data warehouses and data marts.</p><br />
<br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service<ref>Tsang, Daisy. (February 12th, 2018). Kubernetes vs. Docker: What Does It Really Mean? Sumo Logic. 2019. Retrieved 16-May-2019 from: <i>[https://www.sumologic.com/blog/kubernetes-vs-docker/ ]</i></ref>. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Datalakes&diff=11134Technology Trends/Datalakes2019-07-18T17:40:13Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Datalakes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Lac de Donnés]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[KubernetesIMG.png|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Datalakes</b> is a central system or repository of data that is stored in its natural/raw format. A datalake acts as a single store for all enterprise data. Data is transformed using machine learning, advanced,analytics, and visualization. Several forms of data can be hosed in a datalake. These include structured data from relational databases, unstructured data, semi-structured data, and binary data. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>In an ever-increasing hyperconnected world, corporations and businesses are struggling to deal with the responsibilities of storage, management and quick availability of raw data. To break these data challenges down further:</p><br />
<ul class="expand mw-collapsible-content"><br />
<li>Data comes in many different structures.<br />
<ul><br />
<li>Unstructured</li><br />
<li>Semi-Sturctured</li><br />
<li>Structured</li><br />
</ul></li><br />
<li>Data comes from many disparate sources.<br />
<ul><br />
<li>Enterprise Applications</li><br />
<li>Raw Files</li><br />
<li>Operation and Security Logs</li><br />
<li>Financial Transactions</li><br />
<li>Internet of Things (IoT) Devices and Network Sensors</li><br />
<li>Websites</li><br />
<li>Scientific Research</li><br />
</ul></li><br />
<li>Data sources are often geographically distributed to multiple locations<br />
<ul><br />
<li>Datacenters</li><br />
<li>Remote Offices</li><br />
<li>Mobile Devices</li><br />
</ul></li><br />
</ul><br />
<br />
<p>In an effort to resolve these data challenges, a new way of managing data was created which drove data oriented companies to invent a new data storage mechanism called a Data Lake.</p><br />
<p class="expand mw-collapsible-content">Data Lakes are characterized as: </p><br />
<ul class="expand mw-collapsible-content"><br />
<li>Collect Everything<ul><br />
<li>A Data Lake contains all data; raw sources over extended periods of time as well as any processed data.</li><br />
</ul></li><br />
<li>Dive in anywhere<ul><br />
<li>A Data Lake enables users across multiple business units to refine, explore and enrich data on their terms.</li><br />
</ul></li><br />
<li>Flexible Access<ul><br />
<li>o A Data Lake enables multiple data access patterns across a shared infrastructure: batch, interactive, online, search, in-memory and other processing engines.</li><br />
</ul></li><br />
</ul><br />
<br />
<p>Data Lakes are essentially a technology platform for holding data. Their value to the business is only realized when applying data science skills to the lake. </p><br />
<br />
<p>To summarize, usecases for Data Lakes are still being discovered. Cloud providers are making it easier to procure Data Lakes and today Data Lakes are primarily used by Research Institutions, Financial Services, Telecom, Media, Retail, Manufacturing, Healthcare, Pharma, Oi l& Gas and Governments.<br />
</p><br />
<br />
<br />
<br />
<h2>Technology Brief</h2><br />
<p class="expand mw-collapsible-content">The most popular implementation of a Data Lake is through the open source platform called Apache Hadoop. Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. Hadoop was originally created by researchers at Google as a storage method to handle the indexing of websites on the Internet; At that time it was called the Google File System. </p><br />
<p>A Data Lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data, and run different types of analytics—from dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions.”</p><br />
<p class="expand mw-collapsible-content">Data can flow into the Data Lake by either batch processing or real-time processing of streaming data. Additionally, data itself is no longer restrained by initial schema decisions and can be exploited more freely by the enterprise. Rising above this repository is a set of capabilities that allow IT to provide Data and Analytics as a Service (DAaaS), in a supply-demand model. IT takes the role of the data provider (supplier), while business users (data scientists, business analysts) are consumers.</p><br />
<p>The DAaaS model enables users to self-serve their data and analytic needs. Users browse the lake’s data catalog (a Datapedia) to find and select the available data and fill a metaphorical “shopping cart” (effectively an analytics sandbox) with data to work with. Once access is provisioned, users can use the analytics tools of their choice to develop models and gain insights. Subsequently, users can publish analytical models or push refined or transformed data back into the Data Lake to share with the larger community.</p><br />
<p class="expand mw-collapsible-content">Although provisioning an analytic sandbox is a primary use, the Data Lake also has other applications. For example, the Data Lake can also be used to ingest raw data, curate the data, and apply Export-Transform-Load (ETL). This data can then be loaded to an Enterprise Data Warehouse. To take advantage of the flexibility provided by the Data Lake, organizations need to customize and configure the Data Lake to their specific requirements and domains.</p><br />
<h2>Industry Usage</h2><br />
<p>There are a variety of ways Data Lakes are being used in the industry:</p><br />
<ul><br />
<li><p><b>Ingestion of semi-structured and unstructured data sources (aka big data)</b>such as equipment readings, telemetry data, logs, streaming data, and so forth. A Data Lake is a great solution for storing IoT (Internet of Things) type of data which has traditionally been more difficult to store, and can support near real-time analysis. Optionally, you can also add structured data (i.e., extracted from a relational data source) to a Data Lake if your objective is a single repository of all data to be available via the lake.</p></li><br />
<li><p><b>Experimental analysis </b>of data before its value or purpose has been fully defined. Agility is important for every business these days, so a Data Lake can play an important role in "proof of value" type of situations because of the "ELT" approach discussed above.</p></li><br />
<li><p><b>Advanced analytics support. </b>A Data Lake is useful for data scientists and analysts to provision and experiment with data.</p></li><br />
<li><p><b>Archival and historical data storage. </b>Sometimes data is used infrequently, but does need to be available for analysis. A Data Lake strategy can be very valuable to support an active archive strategy.</p></li><br />
<li><p><b>Distributed processing </b>capabilities associated with a logical data warehouse.</p></li><br />
</ul><br />
<b class="expand mw-collapsible-content">How TD Bank Made Its Data Lake More Usa</b><br />
<p class="expand mw-collapsible-content">[[https://www.datanami.com/2017/10/03/td-bank-made-data-lake-usable/]]<br>Toronto-Dominion Bank (TD Bank) is one of the largest banks in North America, with 85,000 employees, more than 2,400 locations between Canada and the United States, and assets nearing $1 trillion. In 2014, the company decided to standardize how it warehouses data for various business intelligence and regulatory reporting functions. The company purchased a Hadoop distribution and set off to build a large cluster that could function as a centralized lake to store data originating from a variety of departments.</p><br />
<br />
<h2>Canadian Government Use</h2><br />
<p>In 2019, the Treasury Board of Canada Secretariat (TBS), partnered with Shared Services Canada and other departments, to identify a business lead to develop a Data Lake (a repository of raw data) service strategy so that the GC can take advantage of big data and market innovation to foster better analytics and promote horizontal data-sharing. </p><br />
<p class="expand mw-collapsible-content">Big data is the technology that stores and processes data and information in datasets that are so large or complex that traditional data processing applications can’t analyze them. Big data can make available almost limitless amounts of information, improving data-driven decision-making and expanding open data initiatives. Business intelligence involves creating, aggregating, analyzing and visualizing data to inform and facilitate business management and strategy. TBS, working with departments, will lead the development of requirements for an enterprise analytics platform.</p><br />
<p>Data Lake development in the GC is a more recent initiative. This is mainly due to the GC focussing resources on the implementation of cloud initiatives. However, there are some GC departments engaged in developing Data Lake environments in tandem to cloud initiatives.</p><br />
<p class="expand mw-collapsible-content">Notably, the Employment and Social Development Canada (ESDC) is preparing the installment of multiple Data Lakes in order to enable a Data Lake Ecosystem and Data Analytics and Machine Learning toolset. This will enable ESDC to share information horizontally both effectively and safely, while enabling a wide variety of data analytics capabilities. ESDC aims to maintain current data and analytics capabilities up-to-date while exploring new ones to mitigate gaps and continuously evolve our services to meet client’s needs. </p><br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p class="expand mw-collapsible-content">There are three common value propositions for pursuing Data Lakes. 1) It can provide an easy and accessible way to obtain data faster; 2) It can create a singular inflow point of data to help connect and merge information silos in an organization; and 3) It can provide an experimental environment for experienced data scientists to enable new analytical insights.</p><br />
<p class="inline">Data Lakes can provide data to consumers more quickly by offering data in a more raw and easily accessible form. Data is stored in its native form with little to no processing, it is optimized to store vast amounts of data in their native formats. By allowing the data to remain in its native format, a much timelier stream of data is available for unlimited queries and analysis. A Data Lake can help data consumers bypass strict data retrieval and data structured applications such as a data warehouse and/or data mart. This has the effect of improving a business’ data flexibility.</p><p class="expand inline mw-collapsible-content">Some companies have in fact used Data Lakes to replace existing warehousing environments where implementing a new data warehouse is more cost prohibitive. A Data Lake can contain unrefined data, this is helpful when either a business data structure is unknown, or when a data consumer requires access to the data quickly. </p><br />
<p></p><br />
<p class="inline">A Data Lake is not a single source of truth. A Data Lake is a central location in which data converges from all data sources and is stored, regardless of the data formatting. </p><p class="expand inline mw-collapsible-content">As a singular point for the inflow of data, sections of a business can pool their information together in the Data Lake and increase the sharing of information with other parts of the organization. In this way everyone in the organization has access to the data. A Data Lake can increase the horizontal data sharing within an organization by creating this singular data inflow point. Using a variety of storage and processing tools analysts can extract data value quickly in order to inform key business decisions.</p><br />
<p></p><br />
<p class="expand inline mw-collapsible-content">A Data Lake is optimized for exploration and provides an experimental environment for experienced data scientists to uncover new insights from data. Analysts can overlay context on the data to extract value. All organizations want to increase analytics and operational agility.</p><p class="inline">The Data Lake architectural approach can store large volumes of data, this can be a way in which cross-cutting teams can pool their data in a central location and by complementing their systems of record with systems of insight. </p><p class="expand inline mw-collapsible-content">Data Lakes present the most potential benefits for experienced and competant data scientists. </p><p class="inline">Having structured, unstructured and semistructured data, usually in the same data set, can contain business, predictive, and prescriptive insights previously not possible from a structured platform as observed in data warehouses and data marts.</p><br />
<br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service<ref>Tsang, Daisy. (February 12th, 2018). Kubernetes vs. Docker: What Does It Really Mean? Sumo Logic. 2019. Retrieved 16-May-2019 from: <i>[https://www.sumologic.com/blog/kubernetes-vs-docker/ ]</i></ref>. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Datalakes&diff=11129Technology Trends/Datalakes2019-07-18T15:47:58Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Datalakes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Lac de Donnés]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[KubernetesIMG.png|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Datalakes</b> is a central system or repository of data that is stored in its natural/raw format. A datalake acts as a single store for all enterprise data. Data is transformed using machine learning, advanced,analytics, and visualization. Several forms of data can be hosed in a datalake. These include structured data from relational databases, unstructured data, semi-structured data, and binary data. </p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>In an ever-increasing hyperconnected world, corporations and businesses are struggling to deal with the responsibilities of storage, management and quick availability of raw data. To break these data challenges down further:</p><br />
<ul class="expand mw-collapsible-content"><br />
<li>Data comes in many different structures.<br />
<ul><br />
<li>Unstructured</li><br />
<li>Semi-Sturctured</li><br />
<li>Structured</li><br />
</ul></li><br />
<li>Data comes from many disparate sources.<br />
<ul><br />
<li>Enterprise Applications</li><br />
<li>Raw Files</li><br />
<li>Operation and Security Logs</li><br />
<li>Financial Transactions</li><br />
<li>Internet of Things (IoT) Devices and Network Sensors</li><br />
<li>Websites</li><br />
<li>Scientific Research</li><br />
</ul></li><br />
<li>Data sources are often geographically distributed to multiple locations<br />
<ul><br />
<li>Datacenters</li><br />
<li>Remote Offices</li><br />
<li>Mobile Devices</li><br />
</ul></li><br />
</ul><br />
<br />
<p></p><br />
<br />
<br />
<br />
<h2>Technology Brief</h2><br />
<p>The Kubernetes cluster or deployment can be broken down into several components. The Kubernetes “master” is the machine in charge of managing other “node” machines. The “node is the machine in charge of actually running tasks fed to it via the user or the “master”. The master and nodes can be either a physical or virtual machines. In each Kubernetes cluster, there is one master and multiple nodes machines. The main goal of Kubernetes is to achieve “Desired State Management”. The “master” is fed a specific configuration through its RESTful API which it exposes to the user, and the “master” is then responsible for running this configuration across its set of “node”. The nodes can be thought of as host of containers. They communicate with the “master” through the agent in each node --“Kubelet” process. To establish a specific configuration in Kubernetes, the “master is fed a deployment file with the “.yaml” extension. This file contains a variety of configuration information. Within this information are “Pods” and “replicas”. There is a concept of Pod in Kubernetes and it can be described as a logic collection of containers which are managed as a single application. Resources can be shared within a Pod, these resources include shared storage (Volumes), a unique cluster of IP addresses, and information about how to run each container. A Pod can be thought of as the basic unit of the Kubernetes object model, it represents the deployment of a single instance of an application in Kubernetes <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. A Pod can encapsulate one or more application containers. Two models exist for how Pods are deployed within a cluster. The “one-Pod-per-container” means a single pod will be associated with a single container. There can also be multiple containers that run within a single Pod, where these containers may need to communicate with one another as they share resources. In either model, the Pod can be thought of as a wrapper around the application containers. Kubernetes manages the Pod instances rather than managing the containers directly. The Pods are run on the Node machines to perform tasks. Replicas are simply instances of the Pods. Within the “.yaml” deployment file, specifications are instructing the “master” machine how many instances/replicas of each Pod to run, which is handled by a replication controller <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. When a node dies or a running Pod experiences an unexpected termination, the replication controller will take note take care of this by creating the appropriate number of Pods <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>.</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Kubernetes is an open source system and many companies have begun to adopt it into their existing architecture as well as adapt it to their specific needs. It was originally developed by Google and was made an open source project in 2014. The Cloud Native Computing Foundation is a project of the Linux Foundation providing a community for different companies who are seeking to develop Kubernetes and other container orchestration projects. Several major cloud providers and platforms including Google Cloud Compute, HP Helion Cloud, RedHat Openshift, VMware Cloud, and Windows Azure all support the use of Kubernetes<ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes.<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/ ]</i></ref>. A survey, performed by iDatalabs in 2017, found 2,867 companies are currently using Kubernetes. These companies are generally located in the United States and are also most the computer software industry. Companies on the list hire between 50 and 200 employees, and accumulate 1M-100M in revenue per year. Some of the major companies on this list include GoDaddy inc, Pivotal Software inc, Globant SA, and Splunk inc</p><p class="expand inline mw-collapsible-content">. Kubernetes own approximately 8.6% of the market share within the virtualization management software category <ref>Idatalabs.com. (2018). Kubernetes commands 8.62% market share in Virtualization Management Software<i>[https://idatalabs.com/tech/products/kubernetes] </i></ref>. </p><br />
<h2>Canadian Government Use</h2><br />
<p>There is a lack of documented Government of Canada (GC) initiatives and programs promoting the current and future use of Kubernetes technology. As a GC strategic IT item, Kubernetes is absent from both the GC’s Digital Operations Strategic Plan: 2018-2022 and the GC Strategic Plan for Information Management and Information Technology 2017 to 2021. This may be due to the fact that the GC is currently grappling with the implementation of Cloud Services, and the majority of resources and efforts are occupied with implementation challenges, as well as security concerns related to the protection of the information of Canadians.</p><br />
<p class="expand mw-collapsible-content">However, the inception of containers into the market has shown that large-scale organizations, who are involved in cloud-native application development as well as networking, can benefit greatly from the use of containers <ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/]</i></ref>. Although the infrastructure applications providing cloud services can be based solely on Virtual Machines (VMs), the maintenance costs associated with running different operating systems on individual VMs outweighs the benefit <ref>Heron, P. (2018). Experimenting with containerised infrastructure for GOV.UK - Inside GOV.UK. [online] Insidegovuk.blog.gov.uk<i>[https://insidegovuk.blog.gov.uk/2017/09/15/experimenting-with-containerised-infrastructure-for-gov-uk/ ]</i></ref>. Containers and Containerization is a replacement and/or complimentary architecture for VMs. As the GC moves toward cloud services and development of cloud-native applications, the use of containers and orchestrating them with Kubernetes can become an integral part the GC IT architecture. </p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p>The primary business value impact of Kubernetes is the technology’s portability, and mobility independent of the environment. Its ability to manage, and orchestrate an organization’s application containers is a marked benefit. Kubernetes secondary business value is that it enables enterprise high-velocity, meaning that every product team can safely ship updates many times a day, deploy instantly, observe results in real time, and use this feedback to roll containers forward or back with the goal to improve the customer experience as fast as possible<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>In the age of modern web services, users expect their applications to be available 24/7, and developers expect the ability to deploy new versions of those applications several times a day with minimal downtime. Containers have become one of the main ways in which to manage applications across enterprise IT infrastructure and also one of the most difficult areas to manage effectively.</p><br />
<p>Kubernetes, as an open source system, is a technology that can administer and manage a large number of containerized applications spread across clusters of servers while providing basic mechanisms for deployment, maintenance, and scaling of applications<ref>GitHub. (2019). Production-Grade Container Scheduling and Management. GitHub. 2019. Retrieved 16-May-2019 from: <i>[https://github.com/kubernetes/kubernetes ]</i></ref>. An application container is a standard unit of software that packages code and all its dependencies so the application runs quickly and reliably from one computing environment to another<ref>Docker. (2019). What is a Container? A Standardized Unit of Software. Docker Inc. 2019.Retrieved 16-May-2019 from: <i>[https://www.docker.com/resources/what-container ]</i></ref>. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way<ref>Kubernetes. (2019). Using Minikube to Create a Cluster. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-intro/ ]</i></ref>. </p><br />
<p>Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. This decoupling allows container-based applications to be deployed easily and consistently, regardless of whether the target environment is a private data center, the public cloud, or even a developer’s personal laptop<ref><i>[https://cloud.google.com/containers/ ]</i></ref>. An additional benefit to containerization is that the Operating System (OS) is not running as hard. </p><br />
<p class="inline">Since Kubernetes is open source, it allows the enterprise freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, and the ability to effortlessly move workloads<ref>Kubernetes. (2019). Production-Grade Container Orchestration. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/ ]</i></ref>. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes groups containers that make up an application into logical units for easy management and discovery. </p><p class="expand inline mw-collapsible-content">The abstractions in Kubernetes allows deployment of containerized applications to a cluster without tying them specifically to individual machines (i.e. Virtual Machines). Applications can be co-located on the same machines without impacting each other. This means that tasks from multiple users can be packed onto fewer machines. This provides greater efficiency and reduces the cost on hardware as less machines are used. </p><br />
<p>Kubernetes contains tools for orchestration, secrets management, service discovery, scaling and load balancing and includes automatic bin packing to place containers with the optimal resources, and it applies configurations via configuration management features<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes ]</i></ref>. It protects container workloads by rolling out or rolling back changes and offers availability and quality checks for containers -- replacing or restarting failed containers. As requirements change, a user can move container workloads in Kubernetes from one cloud provider or hosting infrastructure to another without changing the code<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. This is a great value to developers as their work is protected and an audit trail of changes is available.</p><br />
<p class="expand mw-collapsible-content">The core concepts of Kubernetes which enables high velocity are immutability, declarative configuration and self-healing systems<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>Containers and Kubernetes encourage developers to build distributed systems that adhere to the principles of immutable infrastructure. In immutable infrastructure an artifact created, will not be changed upon user modifications. To update applications in an immutable infrastructure, a new container image is built with a new tag, and is deployed, terminating the old container with the old image version. In this way, the enterprise always has an artifact record of what was done and if there was an error in the new image. If an error is detected the container is rolled back to the previous image<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. Anything that goes into a container has a text file. Text files can be treated like application source code and provisions immutability.</p><br />
<p class="expand mw-collapsible-content">Declarative configuration enables the user to describe exactly what state the system should be in. Traditional tools of development such as source control, unit tests etc. can be used with declarative configurations in ways that are impossible with imperative configurations. Imperative systems describe how to get from point A to B, but rarely include reverse instructions to get back. Kubernetes declarative configuration makes rollbacks fairly easy which is impossible with imperative configurations<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p class="expand mw-collapsible-content">Lastly, Kubernetes has a means of self-healing. When Kubernetes receives a desired state configuration, it does not simply take actions to make the current state match the desired state at a single time, but it will continuously take actions to ensure it stays that way as time passes by<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service<ref>Tsang, Daisy. (February 12th, 2018). Kubernetes vs. Docker: What Does It Really Mean? Sumo Logic. 2019. Retrieved 16-May-2019 from: <i>[https://www.sumologic.com/blog/kubernetes-vs-docker/ ]</i></ref>. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Datalakes&diff=11116Technology Trends/Datalakes2019-07-18T14:51:33Z<p>Kpere060: Created page with "test"</p>
<hr />
<div>test</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Kubernetes&diff=11113Technology Trends/Kubernetes2019-07-18T14:19:42Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Kubernetes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Kubernetes]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[File:KubernetesIMG.png|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Kubernetes</b> also known as K8s, is a portable, extensible open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>Application containers are increasingly being used in the infrastructure for cloud-native and microservice applications. Specifically, Docker is the engine most commonly used to create containers. Many application developers have noted the value of containers in dependency management as it packages the application and its dependencies, libraries, other binaries into a container to abstract away the differences in the OS distribution and underlying infrastructure. This solves many issues caused by differences in runtime environments when a software moves from one environment to another one.</p><br />
<p class="inline">Containers improve applications’ portability and scalability that enable applications to be released and updated in an easy and fast way without downtime. However, there still exits a demand in the management of containers when the services provided need to be deployed across a cluster of host servers to achieve high-availability and disaster recovery. This is where container deployment in cluster and management tools like Kubernetes provide their value. Developers can now begin to deploy and orchestrate services as a collection of containers across a cluster of servers. Container resource requirements can be explicitly declared that allows developers to bundle application code with an environment configurations. </p><p class="expand inline mw-collapsible-content">Also by increasing container density, resources can be used more efficiently and thus it in turn improves hardware usage <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. Containers provide applications with isolations, so that a development team can be made responsible for specific containers.</p><br />
<p> Kubernetes also offers self-healing, automatized rollout and rollback features, which greatly improve operation high-availability and flexibility. One of the biggest advantages of Kubernetes is the flexibility it provides. Many PaaS packaging dictate specific frameworks, are catered towards specific workloads, or impose limitations which language runtimes can be used <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. These issues are all eliminated with Kubernetes. Therefore, if an organization’s application is capable of being run on a container, Kubernetes is a viable option for container orchestration.</p><br />
<br />
<h2>Technology Brief</h2><br />
<p>The Kubernetes cluster or deployment can be broken down into several components. The Kubernetes “master” is the machine in charge of managing other “node” machines. The “node is the machine in charge of actually running tasks fed to it via the user or the “master”. The master and nodes can be either a physical or virtual machines. In each Kubernetes cluster, there is one master and multiple nodes machines. The main goal of Kubernetes is to achieve “Desired State Management”. The “master” is fed a specific configuration through its RESTful API which it exposes to the user, and the “master” is then responsible for running this configuration across its set of “node”. The nodes can be thought of as host of containers. They communicate with the “master” through the agent in each node --“Kubelet” process. To establish a specific configuration in Kubernetes, the “master is fed a deployment file with the “.yaml” extension. This file contains a variety of configuration information. Within this information are “Pods” and “replicas”. There is a concept of Pod in Kubernetes and it can be described as a logic collection of containers which are managed as a single application. Resources can be shared within a Pod, these resources include shared storage (Volumes), a unique cluster of IP addresses, and information about how to run each container. A Pod can be thought of as the basic unit of the Kubernetes object model, it represents the deployment of a single instance of an application in Kubernetes <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. A Pod can encapsulate one or more application containers. Two models exist for how Pods are deployed within a cluster. The “one-Pod-per-container” means a single pod will be associated with a single container. There can also be multiple containers that run within a single Pod, where these containers may need to communicate with one another as they share resources. In either model, the Pod can be thought of as a wrapper around the application containers. Kubernetes manages the Pod instances rather than managing the containers directly. The Pods are run on the Node machines to perform tasks. Replicas are simply instances of the Pods. Within the “.yaml” deployment file, specifications are instructing the “master” machine how many instances/replicas of each Pod to run, which is handled by a replication controller <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. When a node dies or a running Pod experiences an unexpected termination, the replication controller will take note take care of this by creating the appropriate number of Pods <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>.</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Kubernetes is an open source system and many companies have begun to adopt it into their existing architecture as well as adapt it to their specific needs. It was originally developed by Google and was made an open source project in 2014. The Cloud Native Computing Foundation is a project of the Linux Foundation providing a community for different companies who are seeking to develop Kubernetes and other container orchestration projects. Several major cloud providers and platforms including Google Cloud Compute, HP Helion Cloud, RedHat Openshift, VMware Cloud, and Windows Azure all support the use of Kubernetes<ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes.<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/ ]</i></ref>. A survey, performed by iDatalabs in 2017, found 2,867 companies are currently using Kubernetes. These companies are generally located in the United States and are also most the computer software industry. Companies on the list hire between 50 and 200 employees, and accumulate 1M-100M in revenue per year. Some of the major companies on this list include GoDaddy inc, Pivotal Software inc, Globant SA, and Splunk inc</p><p class="expand inline mw-collapsible-content">. Kubernetes own approximately 8.6% of the market share within the virtualization management software category <ref>Idatalabs.com. (2018). Kubernetes commands 8.62% market share in Virtualization Management Software<i>[https://idatalabs.com/tech/products/kubernetes] </i></ref>. </p><br />
<h2>Canadian Government Use</h2><br />
<p>There is a lack of documented Government of Canada (GC) initiatives and programs promoting the current and future use of Kubernetes technology. As a GC strategic IT item, Kubernetes is absent from both the GC’s Digital Operations Strategic Plan: 2018-2022 and the GC Strategic Plan for Information Management and Information Technology 2017 to 2021. This may be due to the fact that the GC is currently grappling with the implementation of Cloud Services, and the majority of resources and efforts are occupied with implementation challenges, as well as security concerns related to the protection of the information of Canadians.</p><br />
<p class="expand mw-collapsible-content">However, the inception of containers into the market has shown that large-scale organizations, who are involved in cloud-native application development as well as networking, can benefit greatly from the use of containers <ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/]</i></ref>. Although the infrastructure applications providing cloud services can be based solely on Virtual Machines (VMs), the maintenance costs associated with running different operating systems on individual VMs outweighs the benefit <ref>Heron, P. (2018). Experimenting with containerised infrastructure for GOV.UK - Inside GOV.UK. [online] Insidegovuk.blog.gov.uk<i>[https://insidegovuk.blog.gov.uk/2017/09/15/experimenting-with-containerised-infrastructure-for-gov-uk/ ]</i></ref>. Containers and Containerization is a replacement and/or complimentary architecture for VMs. As the GC moves toward cloud services and development of cloud-native applications, the use of containers and orchestrating them with Kubernetes can become an integral part the GC IT architecture. </p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p>The primary business value impact of Kubernetes is the technology’s portability, and mobility independent of the environment. Its ability to manage, and orchestrate an organization’s application containers is a marked benefit. Kubernetes secondary business value is that it enables enterprise high-velocity, meaning that every product team can safely ship updates many times a day, deploy instantly, observe results in real time, and use this feedback to roll containers forward or back with the goal to improve the customer experience as fast as possible<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>In the age of modern web services, users expect their applications to be available 24/7, and developers expect the ability to deploy new versions of those applications several times a day with minimal downtime. Containers have become one of the main ways in which to manage applications across enterprise IT infrastructure and also one of the most difficult areas to manage effectively.</p><br />
<p>Kubernetes, as an open source system, is a technology that can administer and manage a large number of containerized applications spread across clusters of servers while providing basic mechanisms for deployment, maintenance, and scaling of applications<ref>GitHub. (2019). Production-Grade Container Scheduling and Management. GitHub. 2019. Retrieved 16-May-2019 from: <i>[https://github.com/kubernetes/kubernetes ]</i></ref>. An application container is a standard unit of software that packages code and all its dependencies so the application runs quickly and reliably from one computing environment to another<ref>Docker. (2019). What is a Container? A Standardized Unit of Software. Docker Inc. 2019.Retrieved 16-May-2019 from: <i>[https://www.docker.com/resources/what-container ]</i></ref>. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way<ref>Kubernetes. (2019). Using Minikube to Create a Cluster. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-intro/ ]</i></ref>. </p><br />
<p>Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. This decoupling allows container-based applications to be deployed easily and consistently, regardless of whether the target environment is a private data center, the public cloud, or even a developer’s personal laptop<ref><i>[https://cloud.google.com/containers/ ]</i></ref>. An additional benefit to containerization is that the Operating System (OS) is not running as hard. </p><br />
<p class="inline">Since Kubernetes is open source, it allows the enterprise freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, and the ability to effortlessly move workloads<ref>Kubernetes. (2019). Production-Grade Container Orchestration. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/ ]</i></ref>. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes groups containers that make up an application into logical units for easy management and discovery. </p><p class="expand inline mw-collapsible-content">The abstractions in Kubernetes allows deployment of containerized applications to a cluster without tying them specifically to individual machines (i.e. Virtual Machines). Applications can be co-located on the same machines without impacting each other. This means that tasks from multiple users can be packed onto fewer machines. This provides greater efficiency and reduces the cost on hardware as less machines are used. </p><br />
<p>Kubernetes contains tools for orchestration, secrets management, service discovery, scaling and load balancing and includes automatic bin packing to place containers with the optimal resources, and it applies configurations via configuration management features<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes ]</i></ref>. It protects container workloads by rolling out or rolling back changes and offers availability and quality checks for containers -- replacing or restarting failed containers. As requirements change, a user can move container workloads in Kubernetes from one cloud provider or hosting infrastructure to another without changing the code<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. This is a great value to developers as their work is protected and an audit trail of changes is available.</p><br />
<p class="expand mw-collapsible-content">The core concepts of Kubernetes which enables high velocity are immutability, declarative configuration and self-healing systems<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>Containers and Kubernetes encourage developers to build distributed systems that adhere to the principles of immutable infrastructure. In immutable infrastructure an artifact created, will not be changed upon user modifications. To update applications in an immutable infrastructure, a new container image is built with a new tag, and is deployed, terminating the old container with the old image version. In this way, the enterprise always has an artifact record of what was done and if there was an error in the new image. If an error is detected the container is rolled back to the previous image<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. Anything that goes into a container has a text file. Text files can be treated like application source code and provisions immutability.</p><br />
<p class="expand mw-collapsible-content">Declarative configuration enables the user to describe exactly what state the system should be in. Traditional tools of development such as source control, unit tests etc. can be used with declarative configurations in ways that are impossible with imperative configurations. Imperative systems describe how to get from point A to B, but rarely include reverse instructions to get back. Kubernetes declarative configuration makes rollbacks fairly easy which is impossible with imperative configurations<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p class="expand mw-collapsible-content">Lastly, Kubernetes has a means of self-healing. When Kubernetes receives a desired state configuration, it does not simply take actions to make the current state match the desired state at a single time, but it will continuously take actions to ensure it stays that way as time passes by<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service<ref>Tsang, Daisy. (February 12th, 2018). Kubernetes vs. Docker: What Does It Really Mean? Sumo Logic. 2019. Retrieved 16-May-2019 from: <i>[https://www.sumologic.com/blog/kubernetes-vs-docker/ ]</i></ref>. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=File:KubernetesIMG.png&diff=11112File:KubernetesIMG.png2019-07-18T14:19:07Z<p>Kpere060: </p>
<hr />
<div></div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Kubernetes&diff=11111Technology Trends/Kubernetes2019-07-18T14:18:37Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Kubernetes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Kubernetes]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[File:KubernetesIMG.png|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Kubernetes</b>also known as K8s, is a portable, extensible open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>Application containers are increasingly being used in the infrastructure for cloud-native and microservice applications. Specifically, Docker is the engine most commonly used to create containers. Many application developers have noted the value of containers in dependency management as it packages the application and its dependencies, libraries, other binaries into a container to abstract away the differences in the OS distribution and underlying infrastructure. This solves many issues caused by differences in runtime environments when a software moves from one environment to another one.</p><br />
<p class="inline">Containers improve applications’ portability and scalability that enable applications to be released and updated in an easy and fast way without downtime. However, there still exits a demand in the management of containers when the services provided need to be deployed across a cluster of host servers to achieve high-availability and disaster recovery. This is where container deployment in cluster and management tools like Kubernetes provide their value. Developers can now begin to deploy and orchestrate services as a collection of containers across a cluster of servers. Container resource requirements can be explicitly declared that allows developers to bundle application code with an environment configurations. </p><p class="expand inline mw-collapsible-content">Also by increasing container density, resources can be used more efficiently and thus it in turn improves hardware usage <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. Containers provide applications with isolations, so that a development team can be made responsible for specific containers.</p><br />
<p> Kubernetes also offers self-healing, automatized rollout and rollback features, which greatly improve operation high-availability and flexibility. One of the biggest advantages of Kubernetes is the flexibility it provides. Many PaaS packaging dictate specific frameworks, are catered towards specific workloads, or impose limitations which language runtimes can be used <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. These issues are all eliminated with Kubernetes. Therefore, if an organization’s application is capable of being run on a container, Kubernetes is a viable option for container orchestration.</p><br />
<br />
<h2>Technology Brief</h2><br />
<p>The Kubernetes cluster or deployment can be broken down into several components. The Kubernetes “master” is the machine in charge of managing other “node” machines. The “node is the machine in charge of actually running tasks fed to it via the user or the “master”. The master and nodes can be either a physical or virtual machines. In each Kubernetes cluster, there is one master and multiple nodes machines. The main goal of Kubernetes is to achieve “Desired State Management”. The “master” is fed a specific configuration through its RESTful API which it exposes to the user, and the “master” is then responsible for running this configuration across its set of “node”. The nodes can be thought of as host of containers. They communicate with the “master” through the agent in each node --“Kubelet” process. To establish a specific configuration in Kubernetes, the “master is fed a deployment file with the “.yaml” extension. This file contains a variety of configuration information. Within this information are “Pods” and “replicas”. There is a concept of Pod in Kubernetes and it can be described as a logic collection of containers which are managed as a single application. Resources can be shared within a Pod, these resources include shared storage (Volumes), a unique cluster of IP addresses, and information about how to run each container. A Pod can be thought of as the basic unit of the Kubernetes object model, it represents the deployment of a single instance of an application in Kubernetes <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. A Pod can encapsulate one or more application containers. Two models exist for how Pods are deployed within a cluster. The “one-Pod-per-container” means a single pod will be associated with a single container. There can also be multiple containers that run within a single Pod, where these containers may need to communicate with one another as they share resources. In either model, the Pod can be thought of as a wrapper around the application containers. Kubernetes manages the Pod instances rather than managing the containers directly. The Pods are run on the Node machines to perform tasks. Replicas are simply instances of the Pods. Within the “.yaml” deployment file, specifications are instructing the “master” machine how many instances/replicas of each Pod to run, which is handled by a replication controller <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. When a node dies or a running Pod experiences an unexpected termination, the replication controller will take note take care of this by creating the appropriate number of Pods <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>.</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Kubernetes is an open source system and many companies have begun to adopt it into their existing architecture as well as adapt it to their specific needs. It was originally developed by Google and was made an open source project in 2014. The Cloud Native Computing Foundation is a project of the Linux Foundation providing a community for different companies who are seeking to develop Kubernetes and other container orchestration projects. Several major cloud providers and platforms including Google Cloud Compute, HP Helion Cloud, RedHat Openshift, VMware Cloud, and Windows Azure all support the use of Kubernetes<ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes.<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/ ]</i></ref>. A survey, performed by iDatalabs in 2017, found 2,867 companies are currently using Kubernetes. These companies are generally located in the United States and are also most the computer software industry. Companies on the list hire between 50 and 200 employees, and accumulate 1M-100M in revenue per year. Some of the major companies on this list include GoDaddy inc, Pivotal Software inc, Globant SA, and Splunk inc</p><p class="expand inline mw-collapsible-content">. Kubernetes own approximately 8.6% of the market share within the virtualization management software category <ref>Idatalabs.com. (2018). Kubernetes commands 8.62% market share in Virtualization Management Software<i>[https://idatalabs.com/tech/products/kubernetes] </i></ref>. </p><br />
<h2>Canadian Government Use</h2><br />
<p>There is a lack of documented Government of Canada (GC) initiatives and programs promoting the current and future use of Kubernetes technology. As a GC strategic IT item, Kubernetes is absent from both the GC’s Digital Operations Strategic Plan: 2018-2022 and the GC Strategic Plan for Information Management and Information Technology 2017 to 2021. This may be due to the fact that the GC is currently grappling with the implementation of Cloud Services, and the majority of resources and efforts are occupied with implementation challenges, as well as security concerns related to the protection of the information of Canadians.</p><br />
<p class="expand mw-collapsible-content">However, the inception of containers into the market has shown that large-scale organizations, who are involved in cloud-native application development as well as networking, can benefit greatly from the use of containers <ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/]</i></ref>. Although the infrastructure applications providing cloud services can be based solely on Virtual Machines (VMs), the maintenance costs associated with running different operating systems on individual VMs outweighs the benefit <ref>Heron, P. (2018). Experimenting with containerised infrastructure for GOV.UK - Inside GOV.UK. [online] Insidegovuk.blog.gov.uk<i>[https://insidegovuk.blog.gov.uk/2017/09/15/experimenting-with-containerised-infrastructure-for-gov-uk/ ]</i></ref>. Containers and Containerization is a replacement and/or complimentary architecture for VMs. As the GC moves toward cloud services and development of cloud-native applications, the use of containers and orchestrating them with Kubernetes can become an integral part the GC IT architecture. </p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p>The primary business value impact of Kubernetes is the technology’s portability, and mobility independent of the environment. Its ability to manage, and orchestrate an organization’s application containers is a marked benefit. Kubernetes secondary business value is that it enables enterprise high-velocity, meaning that every product team can safely ship updates many times a day, deploy instantly, observe results in real time, and use this feedback to roll containers forward or back with the goal to improve the customer experience as fast as possible<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>In the age of modern web services, users expect their applications to be available 24/7, and developers expect the ability to deploy new versions of those applications several times a day with minimal downtime. Containers have become one of the main ways in which to manage applications across enterprise IT infrastructure and also one of the most difficult areas to manage effectively.</p><br />
<p>Kubernetes, as an open source system, is a technology that can administer and manage a large number of containerized applications spread across clusters of servers while providing basic mechanisms for deployment, maintenance, and scaling of applications<ref>GitHub. (2019). Production-Grade Container Scheduling and Management. GitHub. 2019. Retrieved 16-May-2019 from: <i>[https://github.com/kubernetes/kubernetes ]</i></ref>. An application container is a standard unit of software that packages code and all its dependencies so the application runs quickly and reliably from one computing environment to another<ref>Docker. (2019). What is a Container? A Standardized Unit of Software. Docker Inc. 2019.Retrieved 16-May-2019 from: <i>[https://www.docker.com/resources/what-container ]</i></ref>. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way<ref>Kubernetes. (2019). Using Minikube to Create a Cluster. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-intro/ ]</i></ref>. </p><br />
<p>Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. This decoupling allows container-based applications to be deployed easily and consistently, regardless of whether the target environment is a private data center, the public cloud, or even a developer’s personal laptop<ref><i>[https://cloud.google.com/containers/ ]</i></ref>. An additional benefit to containerization is that the Operating System (OS) is not running as hard. </p><br />
<p class="inline">Since Kubernetes is open source, it allows the enterprise freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, and the ability to effortlessly move workloads<ref>Kubernetes. (2019). Production-Grade Container Orchestration. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/ ]</i></ref>. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes groups containers that make up an application into logical units for easy management and discovery. </p><p class="expand inline mw-collapsible-content">The abstractions in Kubernetes allows deployment of containerized applications to a cluster without tying them specifically to individual machines (i.e. Virtual Machines). Applications can be co-located on the same machines without impacting each other. This means that tasks from multiple users can be packed onto fewer machines. This provides greater efficiency and reduces the cost on hardware as less machines are used. </p><br />
<p>Kubernetes contains tools for orchestration, secrets management, service discovery, scaling and load balancing and includes automatic bin packing to place containers with the optimal resources, and it applies configurations via configuration management features<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes ]</i></ref>. It protects container workloads by rolling out or rolling back changes and offers availability and quality checks for containers -- replacing or restarting failed containers. As requirements change, a user can move container workloads in Kubernetes from one cloud provider or hosting infrastructure to another without changing the code<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. This is a great value to developers as their work is protected and an audit trail of changes is available.</p><br />
<p class="expand mw-collapsible-content">The core concepts of Kubernetes which enables high velocity are immutability, declarative configuration and self-healing systems<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>Containers and Kubernetes encourage developers to build distributed systems that adhere to the principles of immutable infrastructure. In immutable infrastructure an artifact created, will not be changed upon user modifications. To update applications in an immutable infrastructure, a new container image is built with a new tag, and is deployed, terminating the old container with the old image version. In this way, the enterprise always has an artifact record of what was done and if there was an error in the new image. If an error is detected the container is rolled back to the previous image<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. Anything that goes into a container has a text file. Text files can be treated like application source code and provisions immutability.</p><br />
<p class="expand mw-collapsible-content">Declarative configuration enables the user to describe exactly what state the system should be in. Traditional tools of development such as source control, unit tests etc. can be used with declarative configurations in ways that are impossible with imperative configurations. Imperative systems describe how to get from point A to B, but rarely include reverse instructions to get back. Kubernetes declarative configuration makes rollbacks fairly easy which is impossible with imperative configurations<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p class="expand mw-collapsible-content">Lastly, Kubernetes has a means of self-healing. When Kubernetes receives a desired state configuration, it does not simply take actions to make the current state match the desired state at a single time, but it will continuously take actions to ensure it stays that way as time passes by<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service<ref>Tsang, Daisy. (February 12th, 2018). Kubernetes vs. Docker: What Does It Really Mean? Sumo Logic. 2019. Retrieved 16-May-2019 from: <i>[https://www.sumologic.com/blog/kubernetes-vs-docker/ ]</i></ref>. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Kubernetes&diff=11110Technology Trends/Kubernetes2019-07-18T14:17:42Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Kubernetes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Kubernetes]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[File:KubernetesIMG.png|200px]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Kubernetes</b>also known as K8s, is a portable, extensible open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>Application containers are increasingly being used in the infrastructure for cloud-native and microservice applications. Specifically, Docker is the engine most commonly used to create containers. Many application developers have noted the value of containers in dependency management as it packages the application and its dependencies, libraries, other binaries into a container to abstract away the differences in the OS distribution and underlying infrastructure. This solves many issues caused by differences in runtime environments when a software moves from one environment to another one.</p><br />
<p class="inline">Containers improve applications’ portability and scalability that enable applications to be released and updated in an easy and fast way without downtime. However, there still exits a demand in the management of containers when the services provided need to be deployed across a cluster of host servers to achieve high-availability and disaster recovery. This is where container deployment in cluster and management tools like Kubernetes provide their value. Developers can now begin to deploy and orchestrate services as a collection of containers across a cluster of servers. Container resource requirements can be explicitly declared that allows developers to bundle application code with an environment configurations. </p><p class="expand inline mw-collapsible-content">Also by increasing container density, resources can be used more efficiently and thus it in turn improves hardware usage <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. Containers provide applications with isolations, so that a development team can be made responsible for specific containers.</p><br />
<p> Kubernetes also offers self-healing, automatized rollout and rollback features, which greatly improve operation high-availability and flexibility. One of the biggest advantages of Kubernetes is the flexibility it provides. Many PaaS packaging dictate specific frameworks, are catered towards specific workloads, or impose limitations which language runtimes can be used <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. These issues are all eliminated with Kubernetes. Therefore, if an organization’s application is capable of being run on a container, Kubernetes is a viable option for container orchestration.</p><br />
<br />
<h2>Technology Brief</h2><br />
<p>The Kubernetes cluster or deployment can be broken down into several components. The Kubernetes “master” is the machine in charge of managing other “node” machines. The “node is the machine in charge of actually running tasks fed to it via the user or the “master”. The master and nodes can be either a physical or virtual machines. In each Kubernetes cluster, there is one master and multiple nodes machines. The main goal of Kubernetes is to achieve “Desired State Management”. The “master” is fed a specific configuration through its RESTful API which it exposes to the user, and the “master” is then responsible for running this configuration across its set of “node”. The nodes can be thought of as host of containers. They communicate with the “master” through the agent in each node --“Kubelet” process. To establish a specific configuration in Kubernetes, the “master is fed a deployment file with the “.yaml” extension. This file contains a variety of configuration information. Within this information are “Pods” and “replicas”. There is a concept of Pod in Kubernetes and it can be described as a logic collection of containers which are managed as a single application. Resources can be shared within a Pod, these resources include shared storage (Volumes), a unique cluster of IP addresses, and information about how to run each container. A Pod can be thought of as the basic unit of the Kubernetes object model, it represents the deployment of a single instance of an application in Kubernetes <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. A Pod can encapsulate one or more application containers. Two models exist for how Pods are deployed within a cluster. The “one-Pod-per-container” means a single pod will be associated with a single container. There can also be multiple containers that run within a single Pod, where these containers may need to communicate with one another as they share resources. In either model, the Pod can be thought of as a wrapper around the application containers. Kubernetes manages the Pod instances rather than managing the containers directly. The Pods are run on the Node machines to perform tasks. Replicas are simply instances of the Pods. Within the “.yaml” deployment file, specifications are instructing the “master” machine how many instances/replicas of each Pod to run, which is handled by a replication controller <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. When a node dies or a running Pod experiences an unexpected termination, the replication controller will take note take care of this by creating the appropriate number of Pods <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>.</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Kubernetes is an open source system and many companies have begun to adopt it into their existing architecture as well as adapt it to their specific needs. It was originally developed by Google and was made an open source project in 2014. The Cloud Native Computing Foundation is a project of the Linux Foundation providing a community for different companies who are seeking to develop Kubernetes and other container orchestration projects. Several major cloud providers and platforms including Google Cloud Compute, HP Helion Cloud, RedHat Openshift, VMware Cloud, and Windows Azure all support the use of Kubernetes<ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes.<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/ ]</i></ref>. A survey, performed by iDatalabs in 2017, found 2,867 companies are currently using Kubernetes. These companies are generally located in the United States and are also most the computer software industry. Companies on the list hire between 50 and 200 employees, and accumulate 1M-100M in revenue per year. Some of the major companies on this list include GoDaddy inc, Pivotal Software inc, Globant SA, and Splunk inc</p><p class="expand inline mw-collapsible-content">. Kubernetes own approximately 8.6% of the market share within the virtualization management software category <ref>Idatalabs.com. (2018). Kubernetes commands 8.62% market share in Virtualization Management Software<i>[https://idatalabs.com/tech/products/kubernetes] </i></ref>. </p><br />
<h2>Canadian Government Use</h2><br />
<p>There is a lack of documented Government of Canada (GC) initiatives and programs promoting the current and future use of Kubernetes technology. As a GC strategic IT item, Kubernetes is absent from both the GC’s Digital Operations Strategic Plan: 2018-2022 and the GC Strategic Plan for Information Management and Information Technology 2017 to 2021. This may be due to the fact that the GC is currently grappling with the implementation of Cloud Services, and the majority of resources and efforts are occupied with implementation challenges, as well as security concerns related to the protection of the information of Canadians.</p><br />
<p class="expand mw-collapsible-content">However, the inception of containers into the market has shown that large-scale organizations, who are involved in cloud-native application development as well as networking, can benefit greatly from the use of containers <ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/]</i></ref>. Although the infrastructure applications providing cloud services can be based solely on Virtual Machines (VMs), the maintenance costs associated with running different operating systems on individual VMs outweighs the benefit <ref>Heron, P. (2018). Experimenting with containerised infrastructure for GOV.UK - Inside GOV.UK. [online] Insidegovuk.blog.gov.uk<i>[https://insidegovuk.blog.gov.uk/2017/09/15/experimenting-with-containerised-infrastructure-for-gov-uk/ ]</i></ref>. Containers and Containerization is a replacement and/or complimentary architecture for VMs. As the GC moves toward cloud services and development of cloud-native applications, the use of containers and orchestrating them with Kubernetes can become an integral part the GC IT architecture. </p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p>The primary business value impact of Kubernetes is the technology’s portability, and mobility independent of the environment. Its ability to manage, and orchestrate an organization’s application containers is a marked benefit. Kubernetes secondary business value is that it enables enterprise high-velocity, meaning that every product team can safely ship updates many times a day, deploy instantly, observe results in real time, and use this feedback to roll containers forward or back with the goal to improve the customer experience as fast as possible<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>In the age of modern web services, users expect their applications to be available 24/7, and developers expect the ability to deploy new versions of those applications several times a day with minimal downtime. Containers have become one of the main ways in which to manage applications across enterprise IT infrastructure and also one of the most difficult areas to manage effectively.</p><br />
<p>Kubernetes, as an open source system, is a technology that can administer and manage a large number of containerized applications spread across clusters of servers while providing basic mechanisms for deployment, maintenance, and scaling of applications<ref>GitHub. (2019). Production-Grade Container Scheduling and Management. GitHub. 2019. Retrieved 16-May-2019 from: <i>[https://github.com/kubernetes/kubernetes ]</i></ref>. An application container is a standard unit of software that packages code and all its dependencies so the application runs quickly and reliably from one computing environment to another<ref>Docker. (2019). What is a Container? A Standardized Unit of Software. Docker Inc. 2019.Retrieved 16-May-2019 from: <i>[https://www.docker.com/resources/what-container ]</i></ref>. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way<ref>Kubernetes. (2019). Using Minikube to Create a Cluster. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-intro/ ]</i></ref>. </p><br />
<p>Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. This decoupling allows container-based applications to be deployed easily and consistently, regardless of whether the target environment is a private data center, the public cloud, or even a developer’s personal laptop<ref><i>[https://cloud.google.com/containers/ ]</i></ref>. An additional benefit to containerization is that the Operating System (OS) is not running as hard. </p><br />
<p class="inline">Since Kubernetes is open source, it allows the enterprise freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, and the ability to effortlessly move workloads<ref>Kubernetes. (2019). Production-Grade Container Orchestration. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/ ]</i></ref>. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes groups containers that make up an application into logical units for easy management and discovery. </p><p class="expand inline mw-collapsible-content">The abstractions in Kubernetes allows deployment of containerized applications to a cluster without tying them specifically to individual machines (i.e. Virtual Machines). Applications can be co-located on the same machines without impacting each other. This means that tasks from multiple users can be packed onto fewer machines. This provides greater efficiency and reduces the cost on hardware as less machines are used. </p><br />
<p>Kubernetes contains tools for orchestration, secrets management, service discovery, scaling and load balancing and includes automatic bin packing to place containers with the optimal resources, and it applies configurations via configuration management features<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes ]</i></ref>. It protects container workloads by rolling out or rolling back changes and offers availability and quality checks for containers -- replacing or restarting failed containers. As requirements change, a user can move container workloads in Kubernetes from one cloud provider or hosting infrastructure to another without changing the code<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. This is a great value to developers as their work is protected and an audit trail of changes is available.</p><br />
<p class="expand mw-collapsible-content">The core concepts of Kubernetes which enables high velocity are immutability, declarative configuration and self-healing systems<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>Containers and Kubernetes encourage developers to build distributed systems that adhere to the principles of immutable infrastructure. In immutable infrastructure an artifact created, will not be changed upon user modifications. To update applications in an immutable infrastructure, a new container image is built with a new tag, and is deployed, terminating the old container with the old image version. In this way, the enterprise always has an artifact record of what was done and if there was an error in the new image. If an error is detected the container is rolled back to the previous image<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. Anything that goes into a container has a text file. Text files can be treated like application source code and provisions immutability.</p><br />
<p class="expand mw-collapsible-content">Declarative configuration enables the user to describe exactly what state the system should be in. Traditional tools of development such as source control, unit tests etc. can be used with declarative configurations in ways that are impossible with imperative configurations. Imperative systems describe how to get from point A to B, but rarely include reverse instructions to get back. Kubernetes declarative configuration makes rollbacks fairly easy which is impossible with imperative configurations<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p class="expand mw-collapsible-content">Lastly, Kubernetes has a means of self-healing. When Kubernetes receives a desired state configuration, it does not simply take actions to make the current state match the desired state at a single time, but it will continuously take actions to ensure it stays that way as time passes by<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service<ref>Tsang, Daisy. (February 12th, 2018). Kubernetes vs. Docker: What Does It Really Mean? Sumo Logic. 2019. Retrieved 16-May-2019 from: <i>[https://www.sumologic.com/blog/kubernetes-vs-docker/ ]</i></ref>. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Kubernetes&diff=11109Technology Trends/Kubernetes2019-07-18T14:16:02Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Kubernetes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Kubernetes]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[KubernetesIMG.png|200px]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Kubernetes</b>also known as K8s, is a portable, extensible open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>Application containers are increasingly being used in the infrastructure for cloud-native and microservice applications. Specifically, Docker is the engine most commonly used to create containers. Many application developers have noted the value of containers in dependency management as it packages the application and its dependencies, libraries, other binaries into a container to abstract away the differences in the OS distribution and underlying infrastructure. This solves many issues caused by differences in runtime environments when a software moves from one environment to another one.</p><br />
<p class="inline">Containers improve applications’ portability and scalability that enable applications to be released and updated in an easy and fast way without downtime. However, there still exits a demand in the management of containers when the services provided need to be deployed across a cluster of host servers to achieve high-availability and disaster recovery. This is where container deployment in cluster and management tools like Kubernetes provide their value. Developers can now begin to deploy and orchestrate services as a collection of containers across a cluster of servers. Container resource requirements can be explicitly declared that allows developers to bundle application code with an environment configurations. </p><p class="expand inline mw-collapsible-content">Also by increasing container density, resources can be used more efficiently and thus it in turn improves hardware usage <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. Containers provide applications with isolations, so that a development team can be made responsible for specific containers.</p><br />
<p> Kubernetes also offers self-healing, automatized rollout and rollback features, which greatly improve operation high-availability and flexibility. One of the biggest advantages of Kubernetes is the flexibility it provides. Many PaaS packaging dictate specific frameworks, are catered towards specific workloads, or impose limitations which language runtimes can be used <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. These issues are all eliminated with Kubernetes. Therefore, if an organization’s application is capable of being run on a container, Kubernetes is a viable option for container orchestration.</p><br />
<br />
<h2>Technology Brief</h2><br />
<p>The Kubernetes cluster or deployment can be broken down into several components. The Kubernetes “master” is the machine in charge of managing other “node” machines. The “node is the machine in charge of actually running tasks fed to it via the user or the “master”. The master and nodes can be either a physical or virtual machines. In each Kubernetes cluster, there is one master and multiple nodes machines. The main goal of Kubernetes is to achieve “Desired State Management”. The “master” is fed a specific configuration through its RESTful API which it exposes to the user, and the “master” is then responsible for running this configuration across its set of “node”. The nodes can be thought of as host of containers. They communicate with the “master” through the agent in each node --“Kubelet” process. To establish a specific configuration in Kubernetes, the “master is fed a deployment file with the “.yaml” extension. This file contains a variety of configuration information. Within this information are “Pods” and “replicas”. There is a concept of Pod in Kubernetes and it can be described as a logic collection of containers which are managed as a single application. Resources can be shared within a Pod, these resources include shared storage (Volumes), a unique cluster of IP addresses, and information about how to run each container. A Pod can be thought of as the basic unit of the Kubernetes object model, it represents the deployment of a single instance of an application in Kubernetes <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. A Pod can encapsulate one or more application containers. Two models exist for how Pods are deployed within a cluster. The “one-Pod-per-container” means a single pod will be associated with a single container. There can also be multiple containers that run within a single Pod, where these containers may need to communicate with one another as they share resources. In either model, the Pod can be thought of as a wrapper around the application containers. Kubernetes manages the Pod instances rather than managing the containers directly. The Pods are run on the Node machines to perform tasks. Replicas are simply instances of the Pods. Within the “.yaml” deployment file, specifications are instructing the “master” machine how many instances/replicas of each Pod to run, which is handled by a replication controller <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. When a node dies or a running Pod experiences an unexpected termination, the replication controller will take note take care of this by creating the appropriate number of Pods <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>.</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Kubernetes is an open source system and many companies have begun to adopt it into their existing architecture as well as adapt it to their specific needs. It was originally developed by Google and was made an open source project in 2014. The Cloud Native Computing Foundation is a project of the Linux Foundation providing a community for different companies who are seeking to develop Kubernetes and other container orchestration projects. Several major cloud providers and platforms including Google Cloud Compute, HP Helion Cloud, RedHat Openshift, VMware Cloud, and Windows Azure all support the use of Kubernetes<ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes.<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/ ]</i></ref>. A survey, performed by iDatalabs in 2017, found 2,867 companies are currently using Kubernetes. These companies are generally located in the United States and are also most the computer software industry. Companies on the list hire between 50 and 200 employees, and accumulate 1M-100M in revenue per year. Some of the major companies on this list include GoDaddy inc, Pivotal Software inc, Globant SA, and Splunk inc</p><p class="expand inline mw-collapsible-content">. Kubernetes own approximately 8.6% of the market share within the virtualization management software category <ref>Idatalabs.com. (2018). Kubernetes commands 8.62% market share in Virtualization Management Software<i>[https://idatalabs.com/tech/products/kubernetes] </i></ref>. </p><br />
<h2>Canadian Government Use</h2><br />
<p>There is a lack of documented Government of Canada (GC) initiatives and programs promoting the current and future use of Kubernetes technology. As a GC strategic IT item, Kubernetes is absent from both the GC’s Digital Operations Strategic Plan: 2018-2022 and the GC Strategic Plan for Information Management and Information Technology 2017 to 2021. This may be due to the fact that the GC is currently grappling with the implementation of Cloud Services, and the majority of resources and efforts are occupied with implementation challenges, as well as security concerns related to the protection of the information of Canadians.</p><br />
<p class="expand mw-collapsible-content">However, the inception of containers into the market has shown that large-scale organizations, who are involved in cloud-native application development as well as networking, can benefit greatly from the use of containers <ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/]</i></ref>. Although the infrastructure applications providing cloud services can be based solely on Virtual Machines (VMs), the maintenance costs associated with running different operating systems on individual VMs outweighs the benefit <ref>Heron, P. (2018). Experimenting with containerised infrastructure for GOV.UK - Inside GOV.UK. [online] Insidegovuk.blog.gov.uk<i>[https://insidegovuk.blog.gov.uk/2017/09/15/experimenting-with-containerised-infrastructure-for-gov-uk/ ]</i></ref>. Containers and Containerization is a replacement and/or complimentary architecture for VMs. As the GC moves toward cloud services and development of cloud-native applications, the use of containers and orchestrating them with Kubernetes can become an integral part the GC IT architecture. </p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p>The primary business value impact of Kubernetes is the technology’s portability, and mobility independent of the environment. Its ability to manage, and orchestrate an organization’s application containers is a marked benefit. Kubernetes secondary business value is that it enables enterprise high-velocity, meaning that every product team can safely ship updates many times a day, deploy instantly, observe results in real time, and use this feedback to roll containers forward or back with the goal to improve the customer experience as fast as possible<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>In the age of modern web services, users expect their applications to be available 24/7, and developers expect the ability to deploy new versions of those applications several times a day with minimal downtime. Containers have become one of the main ways in which to manage applications across enterprise IT infrastructure and also one of the most difficult areas to manage effectively.</p><br />
<p>Kubernetes, as an open source system, is a technology that can administer and manage a large number of containerized applications spread across clusters of servers while providing basic mechanisms for deployment, maintenance, and scaling of applications<ref>GitHub. (2019). Production-Grade Container Scheduling and Management. GitHub. 2019. Retrieved 16-May-2019 from: <i>[https://github.com/kubernetes/kubernetes ]</i></ref>. An application container is a standard unit of software that packages code and all its dependencies so the application runs quickly and reliably from one computing environment to another<ref>Docker. (2019). What is a Container? A Standardized Unit of Software. Docker Inc. 2019.Retrieved 16-May-2019 from: <i>[https://www.docker.com/resources/what-container ]</i></ref>. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way<ref>Kubernetes. (2019). Using Minikube to Create a Cluster. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-intro/ ]</i></ref>. </p><br />
<p>Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. This decoupling allows container-based applications to be deployed easily and consistently, regardless of whether the target environment is a private data center, the public cloud, or even a developer’s personal laptop<ref><i>[https://cloud.google.com/containers/ ]</i></ref>. An additional benefit to containerization is that the Operating System (OS) is not running as hard. </p><br />
<p class="inline">Since Kubernetes is open source, it allows the enterprise freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, and the ability to effortlessly move workloads<ref>Kubernetes. (2019). Production-Grade Container Orchestration. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/ ]</i></ref>. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes groups containers that make up an application into logical units for easy management and discovery. </p><p class="expand inline mw-collapsible-content">The abstractions in Kubernetes allows deployment of containerized applications to a cluster without tying them specifically to individual machines (i.e. Virtual Machines). Applications can be co-located on the same machines without impacting each other. This means that tasks from multiple users can be packed onto fewer machines. This provides greater efficiency and reduces the cost on hardware as less machines are used. </p><br />
<p>Kubernetes contains tools for orchestration, secrets management, service discovery, scaling and load balancing and includes automatic bin packing to place containers with the optimal resources, and it applies configurations via configuration management features<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes ]</i></ref>. It protects container workloads by rolling out or rolling back changes and offers availability and quality checks for containers -- replacing or restarting failed containers. As requirements change, a user can move container workloads in Kubernetes from one cloud provider or hosting infrastructure to another without changing the code<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. This is a great value to developers as their work is protected and an audit trail of changes is available.</p><br />
<p class="expand mw-collapsible-content">The core concepts of Kubernetes which enables high velocity are immutability, declarative configuration and self-healing systems<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>Containers and Kubernetes encourage developers to build distributed systems that adhere to the principles of immutable infrastructure. In immutable infrastructure an artifact created, will not be changed upon user modifications. To update applications in an immutable infrastructure, a new container image is built with a new tag, and is deployed, terminating the old container with the old image version. In this way, the enterprise always has an artifact record of what was done and if there was an error in the new image. If an error is detected the container is rolled back to the previous image<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. Anything that goes into a container has a text file. Text files can be treated like application source code and provisions immutability.</p><br />
<p class="expand mw-collapsible-content">Declarative configuration enables the user to describe exactly what state the system should be in. Traditional tools of development such as source control, unit tests etc. can be used with declarative configurations in ways that are impossible with imperative configurations. Imperative systems describe how to get from point A to B, but rarely include reverse instructions to get back. Kubernetes declarative configuration makes rollbacks fairly easy which is impossible with imperative configurations<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p class="expand mw-collapsible-content">Lastly, Kubernetes has a means of self-healing. When Kubernetes receives a desired state configuration, it does not simply take actions to make the current state match the desired state at a single time, but it will continuously take actions to ensure it stays that way as time passes by<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service<ref>Tsang, Daisy. (February 12th, 2018). Kubernetes vs. Docker: What Does It Really Mean? Sumo Logic. 2019. Retrieved 16-May-2019 from: <i>[https://www.sumologic.com/blog/kubernetes-vs-docker/ ]</i></ref>. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Kubernetes&diff=11108Technology Trends/Kubernetes2019-07-18T14:15:03Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Kubernetes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Kubernetes]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[[KubernetesIMG.png|200px]]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Kubernetes</b>also known as K8s, is a portable, extensible open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>Application containers are increasingly being used in the infrastructure for cloud-native and microservice applications. Specifically, Docker is the engine most commonly used to create containers. Many application developers have noted the value of containers in dependency management as it packages the application and its dependencies, libraries, other binaries into a container to abstract away the differences in the OS distribution and underlying infrastructure. This solves many issues caused by differences in runtime environments when a software moves from one environment to another one.</p><br />
<p class="inline">Containers improve applications’ portability and scalability that enable applications to be released and updated in an easy and fast way without downtime. However, there still exits a demand in the management of containers when the services provided need to be deployed across a cluster of host servers to achieve high-availability and disaster recovery. This is where container deployment in cluster and management tools like Kubernetes provide their value. Developers can now begin to deploy and orchestrate services as a collection of containers across a cluster of servers. Container resource requirements can be explicitly declared that allows developers to bundle application code with an environment configurations. </p><p class="expand inline mw-collapsible-content">Also by increasing container density, resources can be used more efficiently and thus it in turn improves hardware usage <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. Containers provide applications with isolations, so that a development team can be made responsible for specific containers.</p><br />
<p> Kubernetes also offers self-healing, automatized rollout and rollback features, which greatly improve operation high-availability and flexibility. One of the biggest advantages of Kubernetes is the flexibility it provides. Many PaaS packaging dictate specific frameworks, are catered towards specific workloads, or impose limitations which language runtimes can be used <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. These issues are all eliminated with Kubernetes. Therefore, if an organization’s application is capable of being run on a container, Kubernetes is a viable option for container orchestration.</p><br />
<br />
<h2>Technology Brief</h2><br />
<p>The Kubernetes cluster or deployment can be broken down into several components. The Kubernetes “master” is the machine in charge of managing other “node” machines. The “node is the machine in charge of actually running tasks fed to it via the user or the “master”. The master and nodes can be either a physical or virtual machines. In each Kubernetes cluster, there is one master and multiple nodes machines. The main goal of Kubernetes is to achieve “Desired State Management”. The “master” is fed a specific configuration through its RESTful API which it exposes to the user, and the “master” is then responsible for running this configuration across its set of “node”. The nodes can be thought of as host of containers. They communicate with the “master” through the agent in each node --“Kubelet” process. To establish a specific configuration in Kubernetes, the “master is fed a deployment file with the “.yaml” extension. This file contains a variety of configuration information. Within this information are “Pods” and “replicas”. There is a concept of Pod in Kubernetes and it can be described as a logic collection of containers which are managed as a single application. Resources can be shared within a Pod, these resources include shared storage (Volumes), a unique cluster of IP addresses, and information about how to run each container. A Pod can be thought of as the basic unit of the Kubernetes object model, it represents the deployment of a single instance of an application in Kubernetes <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. A Pod can encapsulate one or more application containers. Two models exist for how Pods are deployed within a cluster. The “one-Pod-per-container” means a single pod will be associated with a single container. There can also be multiple containers that run within a single Pod, where these containers may need to communicate with one another as they share resources. In either model, the Pod can be thought of as a wrapper around the application containers. Kubernetes manages the Pod instances rather than managing the containers directly. The Pods are run on the Node machines to perform tasks. Replicas are simply instances of the Pods. Within the “.yaml” deployment file, specifications are instructing the “master” machine how many instances/replicas of each Pod to run, which is handled by a replication controller <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. When a node dies or a running Pod experiences an unexpected termination, the replication controller will take note take care of this by creating the appropriate number of Pods <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>.</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Kubernetes is an open source system and many companies have begun to adopt it into their existing architecture as well as adapt it to their specific needs. It was originally developed by Google and was made an open source project in 2014. The Cloud Native Computing Foundation is a project of the Linux Foundation providing a community for different companies who are seeking to develop Kubernetes and other container orchestration projects. Several major cloud providers and platforms including Google Cloud Compute, HP Helion Cloud, RedHat Openshift, VMware Cloud, and Windows Azure all support the use of Kubernetes<ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes.<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/ ]</i></ref>. A survey, performed by iDatalabs in 2017, found 2,867 companies are currently using Kubernetes. These companies are generally located in the United States and are also most the computer software industry. Companies on the list hire between 50 and 200 employees, and accumulate 1M-100M in revenue per year. Some of the major companies on this list include GoDaddy inc, Pivotal Software inc, Globant SA, and Splunk inc</p><p class="expand inline mw-collapsible-content">. Kubernetes own approximately 8.6% of the market share within the virtualization management software category <ref>Idatalabs.com. (2018). Kubernetes commands 8.62% market share in Virtualization Management Software<i>[https://idatalabs.com/tech/products/kubernetes] </i></ref>. </p><br />
<h2>Canadian Government Use</h2><br />
<p>There is a lack of documented Government of Canada (GC) initiatives and programs promoting the current and future use of Kubernetes technology. As a GC strategic IT item, Kubernetes is absent from both the GC’s Digital Operations Strategic Plan: 2018-2022 and the GC Strategic Plan for Information Management and Information Technology 2017 to 2021. This may be due to the fact that the GC is currently grappling with the implementation of Cloud Services, and the majority of resources and efforts are occupied with implementation challenges, as well as security concerns related to the protection of the information of Canadians.</p><br />
<p class="expand mw-collapsible-content">However, the inception of containers into the market has shown that large-scale organizations, who are involved in cloud-native application development as well as networking, can benefit greatly from the use of containers <ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/]</i></ref>. Although the infrastructure applications providing cloud services can be based solely on Virtual Machines (VMs), the maintenance costs associated with running different operating systems on individual VMs outweighs the benefit <ref>Heron, P. (2018). Experimenting with containerised infrastructure for GOV.UK - Inside GOV.UK. [online] Insidegovuk.blog.gov.uk<i>[https://insidegovuk.blog.gov.uk/2017/09/15/experimenting-with-containerised-infrastructure-for-gov-uk/ ]</i></ref>. Containers and Containerization is a replacement and/or complimentary architecture for VMs. As the GC moves toward cloud services and development of cloud-native applications, the use of containers and orchestrating them with Kubernetes can become an integral part the GC IT architecture. </p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p>The primary business value impact of Kubernetes is the technology’s portability, and mobility independent of the environment. Its ability to manage, and orchestrate an organization’s application containers is a marked benefit. Kubernetes secondary business value is that it enables enterprise high-velocity, meaning that every product team can safely ship updates many times a day, deploy instantly, observe results in real time, and use this feedback to roll containers forward or back with the goal to improve the customer experience as fast as possible<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>In the age of modern web services, users expect their applications to be available 24/7, and developers expect the ability to deploy new versions of those applications several times a day with minimal downtime. Containers have become one of the main ways in which to manage applications across enterprise IT infrastructure and also one of the most difficult areas to manage effectively.</p><br />
<p>Kubernetes, as an open source system, is a technology that can administer and manage a large number of containerized applications spread across clusters of servers while providing basic mechanisms for deployment, maintenance, and scaling of applications<ref>GitHub. (2019). Production-Grade Container Scheduling and Management. GitHub. 2019. Retrieved 16-May-2019 from: <i>[https://github.com/kubernetes/kubernetes ]</i></ref>. An application container is a standard unit of software that packages code and all its dependencies so the application runs quickly and reliably from one computing environment to another<ref>Docker. (2019). What is a Container? A Standardized Unit of Software. Docker Inc. 2019.Retrieved 16-May-2019 from: <i>[https://www.docker.com/resources/what-container ]</i></ref>. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way<ref>Kubernetes. (2019). Using Minikube to Create a Cluster. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-intro/ ]</i></ref>. </p><br />
<p>Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. This decoupling allows container-based applications to be deployed easily and consistently, regardless of whether the target environment is a private data center, the public cloud, or even a developer’s personal laptop<ref><i>[https://cloud.google.com/containers/ ]</i></ref>. An additional benefit to containerization is that the Operating System (OS) is not running as hard. </p><br />
<p class="inline">Since Kubernetes is open source, it allows the enterprise freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, and the ability to effortlessly move workloads<ref>Kubernetes. (2019). Production-Grade Container Orchestration. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/ ]</i></ref>. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes groups containers that make up an application into logical units for easy management and discovery. </p><p class="expand inline mw-collapsible-content">The abstractions in Kubernetes allows deployment of containerized applications to a cluster without tying them specifically to individual machines (i.e. Virtual Machines). Applications can be co-located on the same machines without impacting each other. This means that tasks from multiple users can be packed onto fewer machines. This provides greater efficiency and reduces the cost on hardware as less machines are used. </p><br />
<p>Kubernetes contains tools for orchestration, secrets management, service discovery, scaling and load balancing and includes automatic bin packing to place containers with the optimal resources, and it applies configurations via configuration management features<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes ]</i></ref>. It protects container workloads by rolling out or rolling back changes and offers availability and quality checks for containers -- replacing or restarting failed containers. As requirements change, a user can move container workloads in Kubernetes from one cloud provider or hosting infrastructure to another without changing the code<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. This is a great value to developers as their work is protected and an audit trail of changes is available.</p><br />
<p class="expand mw-collapsible-content">The core concepts of Kubernetes which enables high velocity are immutability, declarative configuration and self-healing systems<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>Containers and Kubernetes encourage developers to build distributed systems that adhere to the principles of immutable infrastructure. In immutable infrastructure an artifact created, will not be changed upon user modifications. To update applications in an immutable infrastructure, a new container image is built with a new tag, and is deployed, terminating the old container with the old image version. In this way, the enterprise always has an artifact record of what was done and if there was an error in the new image. If an error is detected the container is rolled back to the previous image<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. Anything that goes into a container has a text file. Text files can be treated like application source code and provisions immutability.</p><br />
<p class="expand mw-collapsible-content">Declarative configuration enables the user to describe exactly what state the system should be in. Traditional tools of development such as source control, unit tests etc. can be used with declarative configurations in ways that are impossible with imperative configurations. Imperative systems describe how to get from point A to B, but rarely include reverse instructions to get back. Kubernetes declarative configuration makes rollbacks fairly easy which is impossible with imperative configurations<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p class="expand mw-collapsible-content">Lastly, Kubernetes has a means of self-healing. When Kubernetes receives a desired state configuration, it does not simply take actions to make the current state match the desired state at a single time, but it will continuously take actions to ensure it stays that way as time passes by<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service<ref>Tsang, Daisy. (February 12th, 2018). Kubernetes vs. Docker: What Does It Really Mean? Sumo Logic. 2019. Retrieved 16-May-2019 from: <i>[https://www.sumologic.com/blog/kubernetes-vs-docker/ ]</i></ref>. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Kubernetes&diff=11107Technology Trends/Kubernetes2019-07-18T14:14:04Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Kubernetes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Kubernetes]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[KubernetesIMG.png|200px]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Kubernetes</b>also known as K8s, is a portable, extensible open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>Application containers are increasingly being used in the infrastructure for cloud-native and microservice applications. Specifically, Docker is the engine most commonly used to create containers. Many application developers have noted the value of containers in dependency management as it packages the application and its dependencies, libraries, other binaries into a container to abstract away the differences in the OS distribution and underlying infrastructure. This solves many issues caused by differences in runtime environments when a software moves from one environment to another one.</p><br />
<p class="inline">Containers improve applications’ portability and scalability that enable applications to be released and updated in an easy and fast way without downtime. However, there still exits a demand in the management of containers when the services provided need to be deployed across a cluster of host servers to achieve high-availability and disaster recovery. This is where container deployment in cluster and management tools like Kubernetes provide their value. Developers can now begin to deploy and orchestrate services as a collection of containers across a cluster of servers. Container resource requirements can be explicitly declared that allows developers to bundle application code with an environment configurations. </p><p class="expand inline mw-collapsible-content">Also by increasing container density, resources can be used more efficiently and thus it in turn improves hardware usage <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. Containers provide applications with isolations, so that a development team can be made responsible for specific containers.</p><br />
<p> Kubernetes also offers self-healing, automatized rollout and rollback features, which greatly improve operation high-availability and flexibility. One of the biggest advantages of Kubernetes is the flexibility it provides. Many PaaS packaging dictate specific frameworks, are catered towards specific workloads, or impose limitations which language runtimes can be used <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. These issues are all eliminated with Kubernetes. Therefore, if an organization’s application is capable of being run on a container, Kubernetes is a viable option for container orchestration.</p><br />
<br />
<h2>Technology Brief</h2><br />
<p>The Kubernetes cluster or deployment can be broken down into several components. The Kubernetes “master” is the machine in charge of managing other “node” machines. The “node is the machine in charge of actually running tasks fed to it via the user or the “master”. The master and nodes can be either a physical or virtual machines. In each Kubernetes cluster, there is one master and multiple nodes machines. The main goal of Kubernetes is to achieve “Desired State Management”. The “master” is fed a specific configuration through its RESTful API which it exposes to the user, and the “master” is then responsible for running this configuration across its set of “node”. The nodes can be thought of as host of containers. They communicate with the “master” through the agent in each node --“Kubelet” process. To establish a specific configuration in Kubernetes, the “master is fed a deployment file with the “.yaml” extension. This file contains a variety of configuration information. Within this information are “Pods” and “replicas”. There is a concept of Pod in Kubernetes and it can be described as a logic collection of containers which are managed as a single application. Resources can be shared within a Pod, these resources include shared storage (Volumes), a unique cluster of IP addresses, and information about how to run each container. A Pod can be thought of as the basic unit of the Kubernetes object model, it represents the deployment of a single instance of an application in Kubernetes <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. A Pod can encapsulate one or more application containers. Two models exist for how Pods are deployed within a cluster. The “one-Pod-per-container” means a single pod will be associated with a single container. There can also be multiple containers that run within a single Pod, where these containers may need to communicate with one another as they share resources. In either model, the Pod can be thought of as a wrapper around the application containers. Kubernetes manages the Pod instances rather than managing the containers directly. The Pods are run on the Node machines to perform tasks. Replicas are simply instances of the Pods. Within the “.yaml” deployment file, specifications are instructing the “master” machine how many instances/replicas of each Pod to run, which is handled by a replication controller <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. When a node dies or a running Pod experiences an unexpected termination, the replication controller will take note take care of this by creating the appropriate number of Pods <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>.</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Kubernetes is an open source system and many companies have begun to adopt it into their existing architecture as well as adapt it to their specific needs. It was originally developed by Google and was made an open source project in 2014. The Cloud Native Computing Foundation is a project of the Linux Foundation providing a community for different companies who are seeking to develop Kubernetes and other container orchestration projects. Several major cloud providers and platforms including Google Cloud Compute, HP Helion Cloud, RedHat Openshift, VMware Cloud, and Windows Azure all support the use of Kubernetes<ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes.<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/ ]</i></ref>. A survey, performed by iDatalabs in 2017, found 2,867 companies are currently using Kubernetes. These companies are generally located in the United States and are also most the computer software industry. Companies on the list hire between 50 and 200 employees, and accumulate 1M-100M in revenue per year. Some of the major companies on this list include GoDaddy inc, Pivotal Software inc, Globant SA, and Splunk inc</p><p class="expand inline mw-collapsible-content">. Kubernetes own approximately 8.6% of the market share within the virtualization management software category <ref>Idatalabs.com. (2018). Kubernetes commands 8.62% market share in Virtualization Management Software<i>[https://idatalabs.com/tech/products/kubernetes] </i></ref>. </p><br />
<h2>Canadian Government Use</h2><br />
<p>There is a lack of documented Government of Canada (GC) initiatives and programs promoting the current and future use of Kubernetes technology. As a GC strategic IT item, Kubernetes is absent from both the GC’s Digital Operations Strategic Plan: 2018-2022 and the GC Strategic Plan for Information Management and Information Technology 2017 to 2021. This may be due to the fact that the GC is currently grappling with the implementation of Cloud Services, and the majority of resources and efforts are occupied with implementation challenges, as well as security concerns related to the protection of the information of Canadians.</p><br />
<p class="expand mw-collapsible-content">However, the inception of containers into the market has shown that large-scale organizations, who are involved in cloud-native application development as well as networking, can benefit greatly from the use of containers <ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/]</i></ref>. Although the infrastructure applications providing cloud services can be based solely on Virtual Machines (VMs), the maintenance costs associated with running different operating systems on individual VMs outweighs the benefit <ref>Heron, P. (2018). Experimenting with containerised infrastructure for GOV.UK - Inside GOV.UK. [online] Insidegovuk.blog.gov.uk<i>[https://insidegovuk.blog.gov.uk/2017/09/15/experimenting-with-containerised-infrastructure-for-gov-uk/ ]</i></ref>. Containers and Containerization is a replacement and/or complimentary architecture for VMs. As the GC moves toward cloud services and development of cloud-native applications, the use of containers and orchestrating them with Kubernetes can become an integral part the GC IT architecture. </p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p>The primary business value impact of Kubernetes is the technology’s portability, and mobility independent of the environment. Its ability to manage, and orchestrate an organization’s application containers is a marked benefit. Kubernetes secondary business value is that it enables enterprise high-velocity, meaning that every product team can safely ship updates many times a day, deploy instantly, observe results in real time, and use this feedback to roll containers forward or back with the goal to improve the customer experience as fast as possible<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>In the age of modern web services, users expect their applications to be available 24/7, and developers expect the ability to deploy new versions of those applications several times a day with minimal downtime. Containers have become one of the main ways in which to manage applications across enterprise IT infrastructure and also one of the most difficult areas to manage effectively.</p><br />
<p>Kubernetes, as an open source system, is a technology that can administer and manage a large number of containerized applications spread across clusters of servers while providing basic mechanisms for deployment, maintenance, and scaling of applications<ref>GitHub. (2019). Production-Grade Container Scheduling and Management. GitHub. 2019. Retrieved 16-May-2019 from: <i>[https://github.com/kubernetes/kubernetes ]</i></ref>. An application container is a standard unit of software that packages code and all its dependencies so the application runs quickly and reliably from one computing environment to another<ref>Docker. (2019). What is a Container? A Standardized Unit of Software. Docker Inc. 2019.Retrieved 16-May-2019 from: <i>[https://www.docker.com/resources/what-container ]</i></ref>. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way<ref>Kubernetes. (2019). Using Minikube to Create a Cluster. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-intro/ ]</i></ref>. </p><br />
<p>Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. This decoupling allows container-based applications to be deployed easily and consistently, regardless of whether the target environment is a private data center, the public cloud, or even a developer’s personal laptop<ref><i>[https://cloud.google.com/containers/ ]</i></ref>. An additional benefit to containerization is that the Operating System (OS) is not running as hard. </p><br />
<p class="inline">Since Kubernetes is open source, it allows the enterprise freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, and the ability to effortlessly move workloads<ref>Kubernetes. (2019). Production-Grade Container Orchestration. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/ ]</i></ref>. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes groups containers that make up an application into logical units for easy management and discovery. </p><p class="expand inline mw-collapsible-content">The abstractions in Kubernetes allows deployment of containerized applications to a cluster without tying them specifically to individual machines (i.e. Virtual Machines). Applications can be co-located on the same machines without impacting each other. This means that tasks from multiple users can be packed onto fewer machines. This provides greater efficiency and reduces the cost on hardware as less machines are used. </p><br />
<p>Kubernetes contains tools for orchestration, secrets management, service discovery, scaling and load balancing and includes automatic bin packing to place containers with the optimal resources, and it applies configurations via configuration management features<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes ]</i></ref>. It protects container workloads by rolling out or rolling back changes and offers availability and quality checks for containers -- replacing or restarting failed containers. As requirements change, a user can move container workloads in Kubernetes from one cloud provider or hosting infrastructure to another without changing the code<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. This is a great value to developers as their work is protected and an audit trail of changes is available.</p><br />
<p class="expand mw-collapsible-content">The core concepts of Kubernetes which enables high velocity are immutability, declarative configuration and self-healing systems<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>Containers and Kubernetes encourage developers to build distributed systems that adhere to the principles of immutable infrastructure. In immutable infrastructure an artifact created, will not be changed upon user modifications. To update applications in an immutable infrastructure, a new container image is built with a new tag, and is deployed, terminating the old container with the old image version. In this way, the enterprise always has an artifact record of what was done and if there was an error in the new image. If an error is detected the container is rolled back to the previous image<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. Anything that goes into a container has a text file. Text files can be treated like application source code and provisions immutability.</p><br />
<p class="expand mw-collapsible-content">Declarative configuration enables the user to describe exactly what state the system should be in. Traditional tools of development such as source control, unit tests etc. can be used with declarative configurations in ways that are impossible with imperative configurations. Imperative systems describe how to get from point A to B, but rarely include reverse instructions to get back. Kubernetes declarative configuration makes rollbacks fairly easy which is impossible with imperative configurations<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p class="expand mw-collapsible-content">Lastly, Kubernetes has a means of self-healing. When Kubernetes receives a desired state configuration, it does not simply take actions to make the current state match the desired state at a single time, but it will continuously take actions to ensure it stays that way as time passes by<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service<ref>Tsang, Daisy. (February 12th, 2018). Kubernetes vs. Docker: What Does It Really Mean? Sumo Logic. 2019. Retrieved 16-May-2019 from: <i>[https://www.sumologic.com/blog/kubernetes-vs-docker/ ]</i></ref>. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=File:EN_-_Kubernetes_v0.1_EN.pdf&diff=11106File:EN - Kubernetes v0.1 EN.pdf2019-07-18T14:11:28Z<p>Kpere060: </p>
<hr />
<div></div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Kubernetes&diff=11105Technology Trends/Kubernetes2019-07-18T14:10:51Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Kubernetes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Kubernetes]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[KubernetesIMG.png|200px]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>May 5, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 18, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Kubernetes_v0.1_EN.pdf|Kubernetes.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Kubernetes</b> also known as K8s, is a portable, extensible open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>Application containers are increasingly being used in the infrastructure for cloud-native and microservice applications. Specifically, Docker is the engine most commonly used to create containers. Many application developers have noted the value of containers in dependency management as it packages the application and its dependencies, libraries, other binaries into a container to abstract away the differences in the OS distribution and underlying infrastructure. This solves many issues caused by differences in runtime environments when a software moves from one environment to another one.</p><br />
<p class="inline">Containers improve applications’ portability and scalability that enable applications to be released and updated in an easy and fast way without downtime. However, there still exits a demand in the management of containers when the services provided need to be deployed across a cluster of host servers to achieve high-availability and disaster recovery. This is where container deployment in cluster and management tools like Kubernetes provide their value. Developers can now begin to deploy and orchestrate services as a collection of containers across a cluster of servers. Container resource requirements can be explicitly declared that allows developers to bundle application code with an environment configurations. </p><p class="expand inline mw-collapsible-content">Also by increasing container density, resources can be used more efficiently and thus it in turn improves hardware usage <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. Containers provide applications with isolations, so that a development team can be made responsible for specific containers.</p><br />
<p> Kubernetes also offers self-healing, automatized rollout and rollback features, which greatly improve operation high-availability and flexibility. One of the biggest advantages of Kubernetes is the flexibility it provides. Many PaaS packaging dictate specific frameworks, are catered towards specific workloads, or impose limitations which language runtimes can be used <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. These issues are all eliminated with Kubernetes. Therefore, if an organization’s application is capable of being run on a container, Kubernetes is a viable option for container orchestration.</p><br />
<br />
<h2>Technology Brief</h2><br />
<p>The Kubernetes cluster or deployment can be broken down into several components. The Kubernetes “master” is the machine in charge of managing other “node” machines. The “node is the machine in charge of actually running tasks fed to it via the user or the “master”. The master and nodes can be either a physical or virtual machines. In each Kubernetes cluster, there is one master and multiple nodes machines. The main goal of Kubernetes is to achieve “Desired State Management”. The “master” is fed a specific configuration through its RESTful API which it exposes to the user, and the “master” is then responsible for running this configuration across its set of “node”. The nodes can be thought of as host of containers. They communicate with the “master” through the agent in each node --“Kubelet” process. To establish a specific configuration in Kubernetes, the “master is fed a deployment file with the “.yaml” extension. This file contains a variety of configuration information. Within this information are “Pods” and “replicas”. There is a concept of Pod in Kubernetes and it can be described as a logic collection of containers which are managed as a single application. Resources can be shared within a Pod, these resources include shared storage (Volumes), a unique cluster of IP addresses, and information about how to run each container. A Pod can be thought of as the basic unit of the Kubernetes object model, it represents the deployment of a single instance of an application in Kubernetes <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. A Pod can encapsulate one or more application containers. Two models exist for how Pods are deployed within a cluster. The “one-Pod-per-container” means a single pod will be associated with a single container. There can also be multiple containers that run within a single Pod, where these containers may need to communicate with one another as they share resources. In either model, the Pod can be thought of as a wrapper around the application containers. Kubernetes manages the Pod instances rather than managing the containers directly. The Pods are run on the Node machines to perform tasks. Replicas are simply instances of the Pods. Within the “.yaml” deployment file, specifications are instructing the “master” machine how many instances/replicas of each Pod to run, which is handled by a replication controller <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. When a node dies or a running Pod experiences an unexpected termination, the replication controller will take note take care of this by creating the appropriate number of Pods <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>.</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Kubernetes is an open source system and many companies have begun to adopt it into their existing architecture as well as adapt it to their specific needs. It was originally developed by Google and was made an open source project in 2014. The Cloud Native Computing Foundation is a project of the Linux Foundation providing a community for different companies who are seeking to develop Kubernetes and other container orchestration projects. Several major cloud providers and platforms including Google Cloud Compute, HP Helion Cloud, RedHat Openshift, VMware Cloud, and Windows Azure all support the use of Kubernetes<ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes.<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/ ]</i></ref>. A survey, performed by iDatalabs in 2017, found 2,867 companies are currently using Kubernetes. These companies are generally located in the United States and are also most the computer software industry. Companies on the list hire between 50 and 200 employees, and accumulate 1M-100M in revenue per year. Some of the major companies on this list include GoDaddy inc, Pivotal Software inc, Globant SA, and Splunk inc</p><p class="expand inline mw-collapsible-content">. Kubernetes own approximately 8.6% of the market share within the virtualization management software category <ref>Idatalabs.com. (2018). Kubernetes commands 8.62% market share in Virtualization Management Software<i>[https://idatalabs.com/tech/products/kubernetes] </i></ref>. </p><br />
<h2>Canadian Government Use</h2><br />
<p>There is a lack of documented Government of Canada (GC) initiatives and programs promoting the current and future use of Kubernetes technology. As a GC strategic IT item, Kubernetes is absent from both the GC’s Digital Operations Strategic Plan: 2018-2022 and the GC Strategic Plan for Information Management and Information Technology 2017 to 2021. This may be due to the fact that the GC is currently grappling with the implementation of Cloud Services, and the majority of resources and efforts are occupied with implementation challenges, as well as security concerns related to the protection of the information of Canadians.</p><br />
<p class="expand mw-collapsible-content">However, the inception of containers into the market has shown that large-scale organizations, who are involved in cloud-native application development as well as networking, can benefit greatly from the use of containers <ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/]</i></ref>. Although the infrastructure applications providing cloud services can be based solely on Virtual Machines (VMs), the maintenance costs associated with running different operating systems on individual VMs outweighs the benefit <ref>Heron, P. (2018). Experimenting with containerised infrastructure for GOV.UK - Inside GOV.UK. [online] Insidegovuk.blog.gov.uk<i>[https://insidegovuk.blog.gov.uk/2017/09/15/experimenting-with-containerised-infrastructure-for-gov-uk/ ]</i></ref>. Containers and Containerization is a replacement and/or complimentary architecture for VMs. As the GC moves toward cloud services and development of cloud-native applications, the use of containers and orchestrating them with Kubernetes can become an integral part the GC IT architecture. </p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p>The primary business value impact of Kubernetes is the technology’s portability, and mobility independent of the environment. Its ability to manage, and orchestrate an organization’s application containers is a marked benefit. Kubernetes secondary business value is that it enables enterprise high-velocity, meaning that every product team can safely ship updates many times a day, deploy instantly, observe results in real time, and use this feedback to roll containers forward or back with the goal to improve the customer experience as fast as possible<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>In the age of modern web services, users expect their applications to be available 24/7, and developers expect the ability to deploy new versions of those applications several times a day with minimal downtime. Containers have become one of the main ways in which to manage applications across enterprise IT infrastructure and also one of the most difficult areas to manage effectively.</p><br />
<p>Kubernetes, as an open source system, is a technology that can administer and manage a large number of containerized applications spread across clusters of servers while providing basic mechanisms for deployment, maintenance, and scaling of applications<ref>GitHub. (2019). Production-Grade Container Scheduling and Management. GitHub. 2019. Retrieved 16-May-2019 from: <i>[https://github.com/kubernetes/kubernetes ]</i></ref>. An application container is a standard unit of software that packages code and all its dependencies so the application runs quickly and reliably from one computing environment to another<ref>Docker. (2019). What is a Container? A Standardized Unit of Software. Docker Inc. 2019.Retrieved 16-May-2019 from: <i>[https://www.docker.com/resources/what-container ]</i></ref>. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way<ref>Kubernetes. (2019). Using Minikube to Create a Cluster. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-intro/ ]</i></ref>. </p><br />
<p>Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. This decoupling allows container-based applications to be deployed easily and consistently, regardless of whether the target environment is a private data center, the public cloud, or even a developer’s personal laptop<ref><i>[https://cloud.google.com/containers/ ]</i></ref>. An additional benefit to containerization is that the Operating System (OS) is not running as hard. </p><br />
<p class="inline">Since Kubernetes is open source, it allows the enterprise freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, and the ability to effortlessly move workloads<ref>Kubernetes. (2019). Production-Grade Container Orchestration. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/ ]</i></ref>. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes groups containers that make up an application into logical units for easy management and discovery. </p><p class="expand inline mw-collapsible-content">The abstractions in Kubernetes allows deployment of containerized applications to a cluster without tying them specifically to individual machines (i.e. Virtual Machines). Applications can be co-located on the same machines without impacting each other. This means that tasks from multiple users can be packed onto fewer machines. This provides greater efficiency and reduces the cost on hardware as less machines are used. </p><br />
<p>Kubernetes contains tools for orchestration, secrets management, service discovery, scaling and load balancing and includes automatic bin packing to place containers with the optimal resources, and it applies configurations via configuration management features<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes ]</i></ref>. It protects container workloads by rolling out or rolling back changes and offers availability and quality checks for containers -- replacing or restarting failed containers. As requirements change, a user can move container workloads in Kubernetes from one cloud provider or hosting infrastructure to another without changing the code<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. This is a great value to developers as their work is protected and an audit trail of changes is available.</p><br />
<p class="expand mw-collapsible-content">The core concepts of Kubernetes which enables high velocity are immutability, declarative configuration and self-healing systems<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>Containers and Kubernetes encourage developers to build distributed systems that adhere to the principles of immutable infrastructure. In immutable infrastructure an artifact created, will not be changed upon user modifications. To update applications in an immutable infrastructure, a new container image is built with a new tag, and is deployed, terminating the old container with the old image version. In this way, the enterprise always has an artifact record of what was done and if there was an error in the new image. If an error is detected the container is rolled back to the previous image<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. Anything that goes into a container has a text file. Text files can be treated like application source code and provisions immutability.</p><br />
<p class="expand mw-collapsible-content">Declarative configuration enables the user to describe exactly what state the system should be in. Traditional tools of development such as source control, unit tests etc. can be used with declarative configurations in ways that are impossible with imperative configurations. Imperative systems describe how to get from point A to B, but rarely include reverse instructions to get back. Kubernetes declarative configuration makes rollbacks fairly easy which is impossible with imperative configurations<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p class="expand mw-collapsible-content">Lastly, Kubernetes has a means of self-healing. When Kubernetes receives a desired state configuration, it does not simply take actions to make the current state match the desired state at a single time, but it will continuously take actions to ensure it stays that way as time passes by<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service<ref>Tsang, Daisy. (February 12th, 2018). Kubernetes vs. Docker: What Does It Really Mean? Sumo Logic. 2019. Retrieved 16-May-2019 from: <i>[https://www.sumologic.com/blog/kubernetes-vs-docker/ ]</i></ref>. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Kubernetes&diff=11104Technology Trends/Kubernetes2019-07-18T14:02:30Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Kubernetes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Kubernetes]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>June 25, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 17, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Digital_Twins_v0.1_EN_Published.pdf|Digital Twins.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Kubernetes</b>also known as K8s, is a portable, extensible open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>Application containers are increasingly being used in the infrastructure for cloud-native and microservice applications. Specifically, Docker is the engine most commonly used to create containers. Many application developers have noted the value of containers in dependency management as it packages the application and its dependencies, libraries, other binaries into a container to abstract away the differences in the OS distribution and underlying infrastructure. This solves many issues caused by differences in runtime environments when a software moves from one environment to another one.</p><br />
<p class="inline">Containers improve applications’ portability and scalability that enable applications to be released and updated in an easy and fast way without downtime. However, there still exits a demand in the management of containers when the services provided need to be deployed across a cluster of host servers to achieve high-availability and disaster recovery. This is where container deployment in cluster and management tools like Kubernetes provide their value. Developers can now begin to deploy and orchestrate services as a collection of containers across a cluster of servers. Container resource requirements can be explicitly declared that allows developers to bundle application code with an environment configurations. </p><p class="expand inline mw-collapsible-content">Also by increasing container density, resources can be used more efficiently and thus it in turn improves hardware usage <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. Containers provide applications with isolations, so that a development team can be made responsible for specific containers.</p><br />
<p> Kubernetes also offers self-healing, automatized rollout and rollback features, which greatly improve operation high-availability and flexibility. One of the biggest advantages of Kubernetes is the flexibility it provides. Many PaaS packaging dictate specific frameworks, are catered towards specific workloads, or impose limitations which language runtimes can be used <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. These issues are all eliminated with Kubernetes. Therefore, if an organization’s application is capable of being run on a container, Kubernetes is a viable option for container orchestration.</p><br />
<br />
<h2>Technology Brief</h2><br />
<p>The Kubernetes cluster or deployment can be broken down into several components. The Kubernetes “master” is the machine in charge of managing other “node” machines. The “node is the machine in charge of actually running tasks fed to it via the user or the “master”. The master and nodes can be either a physical or virtual machines. In each Kubernetes cluster, there is one master and multiple nodes machines. The main goal of Kubernetes is to achieve “Desired State Management”. The “master” is fed a specific configuration through its RESTful API which it exposes to the user, and the “master” is then responsible for running this configuration across its set of “node”. The nodes can be thought of as host of containers. They communicate with the “master” through the agent in each node --“Kubelet” process. To establish a specific configuration in Kubernetes, the “master is fed a deployment file with the “.yaml” extension. This file contains a variety of configuration information. Within this information are “Pods” and “replicas”. There is a concept of Pod in Kubernetes and it can be described as a logic collection of containers which are managed as a single application. Resources can be shared within a Pod, these resources include shared storage (Volumes), a unique cluster of IP addresses, and information about how to run each container. A Pod can be thought of as the basic unit of the Kubernetes object model, it represents the deployment of a single instance of an application in Kubernetes <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. A Pod can encapsulate one or more application containers. Two models exist for how Pods are deployed within a cluster. The “one-Pod-per-container” means a single pod will be associated with a single container. There can also be multiple containers that run within a single Pod, where these containers may need to communicate with one another as they share resources. In either model, the Pod can be thought of as a wrapper around the application containers. Kubernetes manages the Pod instances rather than managing the containers directly. The Pods are run on the Node machines to perform tasks. Replicas are simply instances of the Pods. Within the “.yaml” deployment file, specifications are instructing the “master” machine how many instances/replicas of each Pod to run, which is handled by a replication controller <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. When a node dies or a running Pod experiences an unexpected termination, the replication controller will take note take care of this by creating the appropriate number of Pods <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>.</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Kubernetes is an open source system and many companies have begun to adopt it into their existing architecture as well as adapt it to their specific needs. It was originally developed by Google and was made an open source project in 2014. The Cloud Native Computing Foundation is a project of the Linux Foundation providing a community for different companies who are seeking to develop Kubernetes and other container orchestration projects. Several major cloud providers and platforms including Google Cloud Compute, HP Helion Cloud, RedHat Openshift, VMware Cloud, and Windows Azure all support the use of Kubernetes<ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes.<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/ ]</i></ref>. A survey, performed by iDatalabs in 2017, found 2,867 companies are currently using Kubernetes. These companies are generally located in the United States and are also most the computer software industry. Companies on the list hire between 50 and 200 employees, and accumulate 1M-100M in revenue per year. Some of the major companies on this list include GoDaddy inc, Pivotal Software inc, Globant SA, and Splunk inc</p><p class="expand inline mw-collapsible-content">. Kubernetes own approximately 8.6% of the market share within the virtualization management software category <ref>Idatalabs.com. (2018). Kubernetes commands 8.62% market share in Virtualization Management Software<i>[https://idatalabs.com/tech/products/kubernetes] </i></ref>. </p><br />
<h2>Canadian Government Use</h2><br />
<p>There is a lack of documented Government of Canada (GC) initiatives and programs promoting the current and future use of Kubernetes technology. As a GC strategic IT item, Kubernetes is absent from both the GC’s Digital Operations Strategic Plan: 2018-2022 and the GC Strategic Plan for Information Management and Information Technology 2017 to 2021. This may be due to the fact that the GC is currently grappling with the implementation of Cloud Services, and the majority of resources and efforts are occupied with implementation challenges, as well as security concerns related to the protection of the information of Canadians.</p><br />
<p class="expand mw-collapsible-content">However, the inception of containers into the market has shown that large-scale organizations, who are involved in cloud-native application development as well as networking, can benefit greatly from the use of containers <ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/]</i></ref>. Although the infrastructure applications providing cloud services can be based solely on Virtual Machines (VMs), the maintenance costs associated with running different operating systems on individual VMs outweighs the benefit <ref>Heron, P. (2018). Experimenting with containerised infrastructure for GOV.UK - Inside GOV.UK. [online] Insidegovuk.blog.gov.uk<i>[https://insidegovuk.blog.gov.uk/2017/09/15/experimenting-with-containerised-infrastructure-for-gov-uk/ ]</i></ref>. Containers and Containerization is a replacement and/or complimentary architecture for VMs. As the GC moves toward cloud services and development of cloud-native applications, the use of containers and orchestrating them with Kubernetes can become an integral part the GC IT architecture. </p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p>The primary business value impact of Kubernetes is the technology’s portability, and mobility independent of the environment. Its ability to manage, and orchestrate an organization’s application containers is a marked benefit. Kubernetes secondary business value is that it enables enterprise high-velocity, meaning that every product team can safely ship updates many times a day, deploy instantly, observe results in real time, and use this feedback to roll containers forward or back with the goal to improve the customer experience as fast as possible<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>In the age of modern web services, users expect their applications to be available 24/7, and developers expect the ability to deploy new versions of those applications several times a day with minimal downtime. Containers have become one of the main ways in which to manage applications across enterprise IT infrastructure and also one of the most difficult areas to manage effectively.</p><br />
<p>Kubernetes, as an open source system, is a technology that can administer and manage a large number of containerized applications spread across clusters of servers while providing basic mechanisms for deployment, maintenance, and scaling of applications<ref>GitHub. (2019). Production-Grade Container Scheduling and Management. GitHub. 2019. Retrieved 16-May-2019 from: <i>[https://github.com/kubernetes/kubernetes ]</i></ref>. An application container is a standard unit of software that packages code and all its dependencies so the application runs quickly and reliably from one computing environment to another<ref>Docker. (2019). What is a Container? A Standardized Unit of Software. Docker Inc. 2019.Retrieved 16-May-2019 from: <i>[https://www.docker.com/resources/what-container ]</i></ref>. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way<ref>Kubernetes. (2019). Using Minikube to Create a Cluster. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/create-cluster/cluster-intro/ ]</i></ref>. </p><br />
<p>Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. This decoupling allows container-based applications to be deployed easily and consistently, regardless of whether the target environment is a private data center, the public cloud, or even a developer’s personal laptop<ref><i>[https://cloud.google.com/containers/ ]</i></ref>. An additional benefit to containerization is that the Operating System (OS) is not running as hard. </p><br />
<p class="inline">Since Kubernetes is open source, it allows the enterprise freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, and the ability to effortlessly move workloads<ref>Kubernetes. (2019). Production-Grade Container Orchestration. Kubernetes. 2019. ICP license: 京ICP备17074266号-3. Retrieved 16-May-2019 from: <i>[https://kubernetes.io/ ]</i></ref>. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes groups containers that make up an application into logical units for easy management and discovery. </p><p class="expand inline mw-collapsible-content">The abstractions in Kubernetes allows deployment of containerized applications to a cluster without tying them specifically to individual machines (i.e. Virtual Machines). Applications can be co-located on the same machines without impacting each other. This means that tasks from multiple users can be packed onto fewer machines. This provides greater efficiency and reduces the cost on hardware as less machines are used. </p><br />
<p>Kubernetes contains tools for orchestration, secrets management, service discovery, scaling and load balancing and includes automatic bin packing to place containers with the optimal resources, and it applies configurations via configuration management features<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes ]</i></ref>. It protects container workloads by rolling out or rolling back changes and offers availability and quality checks for containers -- replacing or restarting failed containers. As requirements change, a user can move container workloads in Kubernetes from one cloud provider or hosting infrastructure to another without changing the code<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. This is a great value to developers as their work is protected and an audit trail of changes is available.</p><br />
<p class="expand mw-collapsible-content">The core concepts of Kubernetes which enables high velocity are immutability, declarative configuration and self-healing systems<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p>Containers and Kubernetes encourage developers to build distributed systems that adhere to the principles of immutable infrastructure. In immutable infrastructure an artifact created, will not be changed upon user modifications. To update applications in an immutable infrastructure, a new container image is built with a new tag, and is deployed, terminating the old container with the old image version. In this way, the enterprise always has an artifact record of what was done and if there was an error in the new image. If an error is detected the container is rolled back to the previous image<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. Anything that goes into a container has a text file. Text files can be treated like application source code and provisions immutability.</p><br />
<p class="expand mw-collapsible-content">Declarative configuration enables the user to describe exactly what state the system should be in. Traditional tools of development such as source control, unit tests etc. can be used with declarative configurations in ways that are impossible with imperative configurations. Imperative systems describe how to get from point A to B, but rarely include reverse instructions to get back. Kubernetes declarative configuration makes rollbacks fairly easy which is impossible with imperative configurations<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<p class="expand mw-collapsible-content">Lastly, Kubernetes has a means of self-healing. When Kubernetes receives a desired state configuration, it does not simply take actions to make the current state match the desired state at a single time, but it will continuously take actions to ensure it stays that way as time passes by<ref>Jayanandana, Nilesh. (May 2nd, 2018). Benefits of Kubernetes. Medium Newspaper. Retrieved 16-May-2019 from: <i>[https://medium.com/platformer-blog/benefits-of-kubernetes-e6d5de39bc48]</i></ref>. </p><br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure<ref>Williams, Alex, et al. Kubernetes Deployment & Security Patterns. The New Stack. 2019. 20180622. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/kubernetes-deployment-and-security-patterns/]</i></ref>. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud]</i></ref>. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm<ref>Rouse, Margaret, et al. (August 2017). Kubernetes. TechTarget Inc. 2019. Retrieved 16-May-2019 from: <i>[https://searchitoperations.techtarget.com/definition/Google-Kubernetes]</i></ref>. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service<ref>Tsang, Daisy. (February 12th, 2018). Kubernetes vs. Docker: What Does It Really Mean? Sumo Logic. 2019. Retrieved 16-May-2019 from: <i>[https://www.sumologic.com/blog/kubernetes-vs-docker/ ]</i></ref>. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services<ref>Treasury Board of Canada Secretariat. December 3, 2018. Directive on Management of Information Technology. Treasury Board of Canada Secretariat. Government of Canada. Retrieved 27-Dec-2018 from: <i>[https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249 ]</i></ref>. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments<ref>Williams, Alex, et al. The State of the Kubernetes Ecosystem. The New Stack. thenewstack.io. Retrieved 15-May-2019 from: <i>[https://thenewstack.io/ebooks/kubernetes/state-of-kubernetes-ecosystem/ ]</i></ref>. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more<ref>Chemitiganti, Vamsi, and Fray, Peter. (February 20th, 2019). 7 Key Considerations for Kubernetes in Production. The New Stack. 2019. Retrieved 16-May-2019 from: <i>[https://thenewstack.io/7-key-considerations-for-kubernetes-in-production/]</i></ref>.. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Kubernetes&diff=11092Technology Trends/Kubernetes2019-07-17T19:50:56Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Kubernetes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Kubernetes]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>June 25, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 17, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Digital_Twins_v0.1_EN_Published.pdf|Digital Twins.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Kubernetes</b>also known as K8s, is a portable, extensible open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>Application containers are increasingly being used in the infrastructure for cloud-native and microservice applications. Specifically, Docker is the engine most commonly used to create containers. Many application developers have noted the value of containers in dependency management as it packages the application and its dependencies, libraries, other binaries into a container to abstract away the differences in the OS distribution and underlying infrastructure. This solves many issues caused by differences in runtime environments when a software moves from one environment to another one.</p><br />
<p class="inline">Containers improve applications’ portability and scalability that enable applications to be released and updated in an easy and fast way without downtime. However, there still exits a demand in the management of containers when the services provided need to be deployed across a cluster of host servers to achieve high-availability and disaster recovery. This is where container deployment in cluster and management tools like Kubernetes provide their value. Developers can now begin to deploy and orchestrate services as a collection of containers across a cluster of servers. Container resource requirements can be explicitly declared that allows developers to bundle application code with an environment configurations. </p><p class="expand inline mw-collapsible-content">Also by increasing container density, resources can be used more efficiently and thus it in turn improves hardware usage <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. Containers provide applications with isolations, so that a development team can be made responsible for specific containers.</p><br />
<p> Kubernetes also offers self-healing, automatized rollout and rollback features, which greatly improve operation high-availability and flexibility. One of the biggest advantages of Kubernetes is the flexibility it provides. Many PaaS packaging dictate specific frameworks, are catered towards specific workloads, or impose limitations which language runtimes can be used <ref>Clayton, T. and Watson, R. (2018). Using Kubernetes to Orchestrate Container-Based Cloud and Microservices Applications. [online] Gartner.com. Available at: <i>[https://www.gartner.com/doc/3873073/using-kubernetes-orchestrate-containerbased-cloud] </i></ref>. These issues are all eliminated with Kubernetes. Therefore, if an organization’s application is capable of being run on a container, Kubernetes is a viable option for container orchestration.</p><br />
<br />
<h2>Technology Brief</h2><br />
<p>The Kubernetes cluster or deployment can be broken down into several components. The Kubernetes “master” is the machine in charge of managing other “node” machines. The “node is the machine in charge of actually running tasks fed to it via the user or the “master”. The master and nodes can be either a physical or virtual machines. In each Kubernetes cluster, there is one master and multiple nodes machines. The main goal of Kubernetes is to achieve “Desired State Management”. The “master” is fed a specific configuration through its RESTful API which it exposes to the user, and the “master” is then responsible for running this configuration across its set of “node”. The nodes can be thought of as host of containers. They communicate with the “master” through the agent in each node --“Kubelet” process. To establish a specific configuration in Kubernetes, the “master is fed a deployment file with the “.yaml” extension. This file contains a variety of configuration information. Within this information are “Pods” and “replicas”. There is a concept of Pod in Kubernetes and it can be described as a logic collection of containers which are managed as a single application. Resources can be shared within a Pod, these resources include shared storage (Volumes), a unique cluster of IP addresses, and information about how to run each container. A Pod can be thought of as the basic unit of the Kubernetes object model, it represents the deployment of a single instance of an application in Kubernetes <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. A Pod can encapsulate one or more application containers. Two models exist for how Pods are deployed within a cluster. The “one-Pod-per-container” means a single pod will be associated with a single container. There can also be multiple containers that run within a single Pod, where these containers may need to communicate with one another as they share resources. In either model, the Pod can be thought of as a wrapper around the application containers. Kubernetes manages the Pod instances rather than managing the containers directly. The Pods are run on the Node machines to perform tasks. Replicas are simply instances of the Pods. Within the “.yaml” deployment file, specifications are instructing the “master” machine how many instances/replicas of each Pod to run, which is handled by a replication controller <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>. When a node dies or a running Pod experiences an unexpected termination, the replication controller will take note take care of this by creating the appropriate number of Pods <ref>Kubernetes.io. (2018). Kubernetes Basics - Kubernetes. [online] Available at: <i>[https://kubernetes.io/docs/tutorials/kubernetes-basics/] </i></ref>.</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Kubernetes is an open source system and many companies have begun to adopt it into their existing architecture as well as adapt it to their specific needs. It was originally developed by Google and was made an open source project in 2014. The Cloud Native Computing Foundation is a project of the Linux Foundation providing a community for different companies who are seeking to develop Kubernetes and other container orchestration projects. Several major cloud providers and platforms including Google Cloud Compute, HP Helion Cloud, RedHat Openshift, VMware Cloud, and Windows Azure all support the use of Kubernetes<ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes.<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/ ]</i></ref>. A survey, performed by iDatalabs in 2017, found 2,867 companies are currently using Kubernetes. These companies are generally located in the United States and are also most the computer software industry. Companies on the list hire between 50 and 200 employees, and accumulate 1M-100M in revenue per year. Some of the major companies on this list include GoDaddy inc, Pivotal Software inc, Globant SA, and Splunk inc</p><p class="expand inline mw-collapsible-content">. Kubernetes own approximately 8.6% of the market share within the virtualization management software category <ref>Idatalabs.com. (2018). Kubernetes commands 8.62% market share in Virtualization Management Software<i>[https://idatalabs.com/tech/products/kubernetes] </i></ref>. </p><br />
<h2>Canadian Government Use</h2><br />
<p>There is a lack of documented Government of Canada (GC) initiatives and programs promoting the current and future use of Kubernetes technology. As a GC strategic IT item, Kubernetes is absent from both the GC’s Digital Operations Strategic Plan: 2018-2022 and the GC Strategic Plan for Information Management and Information Technology 2017 to 2021. This may be due to the fact that the GC is currently grappling with the implementation of Cloud Services, and the majority of resources and efforts are occupied with implementation challenges, as well as security concerns related to the protection of the information of Canadians.</p><br />
<p class="expand mw-collapsible-content">However, the inception of containers into the market has shown that large-scale organizations, who are involved in cloud-native application development as well as networking, can benefit greatly from the use of containers <ref>CENGN. (2018). CENGN and CloudOps Collaborate to Train Industry on Docker and Kubernetes<i>[Available at: https://www.cengn.ca/docker-kubernetes-training-jan18/]</i></ref>. Although the infrastructure applications providing cloud services can be based solely on Virtual Machines (VMs), the maintenance costs associated with running different operating systems on individual VMs outweighs the benefit <ref>Heron, P. (2018). Experimenting with containerised infrastructure for GOV.UK - Inside GOV.UK. [online] Insidegovuk.blog.gov.uk<i>[https://insidegovuk.blog.gov.uk/2017/09/15/experimenting-with-containerised-infrastructure-for-gov-uk/ ]</i></ref>. Containers and Containerization is a replacement and/or complimentary architecture for VMs. As the GC moves toward cloud services and development of cloud-native applications, the use of containers and orchestrating them with Kubernetes can become an integral part the GC IT architecture. </p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p>The primary business value impact of Kubernetes is the technology’s portability, and mobility independent of the environment. Its ability to manage, and orchestrate an organization’s application containers is a marked benefit. Kubernetes secondary business value is that it enables enterprise high-velocity, meaning that every product team can safely ship updates many times a day, deploy instantly, observe results in real time, and use this feedback to roll containers forward or back with the goal to improve the customer experience as fast as possible. </p><br />
<p>In the age of modern web services, users expect their applications to be available 24/7, and developers expect the ability to deploy new versions of those applications several times a day with minimal downtime. Containers have become one of the main ways in which to manage applications across enterprise IT infrastructure and also one of the most difficult areas to manage effectively.</p><br />
<p>Kubernetes, as an open source system, is a technology that can administer and manage a large number of containerized applications spread across clusters of servers while providing basic mechanisms for deployment, maintenance, and scaling of applications. An application container is a standard unit of software that packages code and all its dependencies so the application runs quickly and reliably from one computing environment to another. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way. </p><br />
<p>Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. This decoupling allows container-based applications to be deployed easily and consistently, regardless of whether the target environment is a private data center, the public cloud, or even a developer’s personal laptop. An additional benefit to containerization is that the Operating System (OS) is not running as hard. </p><br />
<p class="inline">Since Kubernetes is open source, it allows the enterprise freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, and the ability to effortlessly move workloads. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes groups containers that make up an application into logical units for easy management and discovery. </p><p class="expand inline mw-collapsible-content">The abstractions in Kubernetes allows deployment of containerized applications to a cluster without tying them specifically to individual machines (i.e. Virtual Machines). Applications can be co-located on the same machines without impacting each other. This means that tasks from multiple users can be packed onto fewer machines. This provides greater efficiency and reduces the cost on hardware as less machines are used. </p><br />
<p>Kubernetes contains tools for orchestration, secrets management, service discovery, scaling and load balancing and includes automatic bin packing to place containers with the optimal resources, and it applies configurations via configuration management features. It protects container workloads by rolling out or rolling back changes and offers availability and quality checks for containers -- replacing or restarting failed containers. As requirements change, a user can move container workloads in Kubernetes from one cloud provider or hosting infrastructure to another without changing the code. This is a great value to developers as their work is protected and an audit trail of changes is available.</p><br />
<p class="expand mw-collapsible-content">The core concepts of Kubernetes which enables high velocity are immutability, declarative configuration and self-healing systems. </p><br />
<p>Containers and Kubernetes encourage developers to build distributed systems that adhere to the principles of immutable infrastructure. In immutable infrastructure an artifact created, will not be changed upon user modifications. To update applications in an immutable infrastructure, a new container image is built with a new tag, and is deployed, terminating the old container with the old image version. In this way, the enterprise always has an artifact record of what was done and if there was an error in the new image. If an error is detected the container is rolled back to the previous image. Anything that goes into a container has a text file. Text files can be treated like application source code and provisions immutability.</p><br />
<p class="expand mw-collapsible-content">Declarative configuration enables the user to describe exactly what state the system should be in. Traditional tools of development such as source control, unit tests etc. can be used with declarative configurations in ways that are impossible with imperative configurations. Imperative systems describe how to get from point A to B, but rarely include reverse instructions to get back. Kubernetes declarative configuration makes rollbacks fairly easy which is impossible with imperative configurations. </p><br />
<p class="expand mw-collapsible-content">Lastly, Kubernetes has a means of self-healing. When Kubernetes receives a desired state configuration, it does not simply take actions to make the current state match the desired state at a single time, but it will continuously take actions to ensure it stays that way as time passes by. </p><br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour [1]. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward [1]. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Kubernetes&diff=11076Technology Trends/Kubernetes2019-07-17T18:31:58Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Kubernetes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Kubernetes]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>June 25, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 17, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Digital_Twins_v0.1_EN_Published.pdf|Digital Twins.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Kubernetes</b>also known as K8s, is a portable, extensible open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>Application containers are increasingly being used in the infrastructure for cloud-native and microservice applications. Specifically, Docker is the engine most commonly used to create containers. Many application developers have noted the value of containers in dependency management as it packages the application and its dependencies, libraries, other binaries into a container to abstract away the differences in the OS distribution and underlying infrastructure. This solves many issues caused by differences in runtime environments when a software moves from one environment to another one.</p><br />
<p class="inline">Containers improve applications’ portability and scalability that enable applications to be released and updated in an easy and fast way without downtime. However, there still exits a demand in the management of containers when the services provided need to be deployed across a cluster of host servers to achieve high-availability and disaster recovery. This is where container deployment in cluster and management tools like Kubernetes provide their value. Developers can now begin to deploy and orchestrate services as a collection of containers across a cluster of servers. Container resource requirements can be explicitly declared that allows developers to bundle application code with an environment configurations. </p><p class="expand inline mw-collapsible-content">Also by increasing container density, resources can be used more efficiently and thus it in turn improves hardware usage [1]. Containers provide applications with isolations, so that a development team can be made responsible for specific containers.</p><br />
<p>Kubernetes also known as K8s, is a portable, extensible open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads. Kubernetes also offers self-healing, automatized rollout and rollback features, which greatly improve operation high-availability and flexibility. One of the biggest advantages of Kubernetes is the flexibility it provides. Many PaaS packaging dictate specific frameworks, are catered towards specific workloads, or impose limitations which language runtimes can be used [1]. These issues are all eliminated with Kubernetes. Therefore, if an organization’s application is capable of being run on a container, Kubernetes is a viable option for container orchestration.</p><br />
<br />
<h2>Technology Brief</h2><br />
<p>The Kubernetes cluster or deployment can be broken down into several components. The Kubernetes “master” is the machine in charge of managing other “node” machines. The “node is the machine in charge of actually running tasks fed to it via the user or the “master”. The master and nodes can be either a physical or virtual machines. In each Kubernetes cluster, there is one master and multiple nodes machines. The main goal of Kubernetes is to achieve “Desired State Management”. The “master” is fed a specific configuration through its RESTful API which it exposes to the user, and the “master” is then responsible for running this configuration across its set of “node”. The nodes can be thought of as host of containers. They communicate with the “master” through the agent in each node --“Kubelet” process. To establish a specific configuration in Kubernetes, the “master is fed a deployment file with the “.yaml” extension. This file contains a variety of configuration information. Within this information are “Pods” and “replicas”. There is a concept of Pod in Kubernetes and it can be described as a logic collection of containers which are managed as a single application. Resources can be shared within a Pod, these resources include shared storage (Volumes), a unique cluster of IP addresses, and information about how to run each container. A Pod can be thought of as the basic unit of the Kubernetes object model, it represents the deployment of a single instance of an application in Kubernetes [8]. A Pod can encapsulate one or more application containers. Two models exist for how Pods are deployed within a cluster. The “one-Pod-per-container” means a single pod will be associated with a single container. There can also be multiple containers that run within a single Pod, where these containers may need to communicate with one another as they share resources. In either model, the Pod can be thought of as a wrapper around the application containers. Kubernetes manages the Pod instances rather than managing the containers directly. The Pods are run on the Node machines to perform tasks. Replicas are simply instances of the Pods. Within the “.yaml” deployment file, specifications are instructing the “master” machine how many instances/replicas of each Pod to run, which is handled by a replication controller [8]. When a node dies or a running Pod experiences an unexpected termination, the replication controller will take note take care of this by creating the appropriate number of Pods [8].</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Kubernetes is an open source system and many companies have begun to adopt it into their existing architecture as well as adapt it to their specific needs. It was originally developed by Google and was made an open source project in 2014. The Cloud Native Computing Foundation is a project of the Linux Foundation providing a community for different companies who are seeking to develop Kubernetes and other container orchestration projects. Several major cloud providers and platforms including Google Cloud Compute, HP Helion Cloud, RedHat Openshift, VMware Cloud, and Windows Azure all support the use of Kubernetes [7]. A survey, performed by iDatalabs in 2017, found 2,867 companies are currently using Kubernetes. These companies are generally located in the United States and are also most the computer software industry. Companies on the list hire between 50 and 200 employees, and accumulate 1M-100M in revenue per year. Some of the major companies on this list include GoDaddy inc, Pivotal Software inc, Globant SA, and Splunk inc</p><p class="expand inline mw-collapsible-content">. Kubernetes own approximately 8.6% of the market share within the virtualization management software category [9]. </p><br />
<h2>Canadian Government Use</h2><br />
<p>There is a lack of documented Government of Canada (GC) initiatives and programs promoting the current and future use of Kubernetes technology. As a GC strategic IT item, Kubernetes is absent from both the GC’s Digital Operations Strategic Plan: 2018-2022 and the GC Strategic Plan for Information Management and Information Technology 2017 to 2021. This may be due to the fact that the GC is currently grappling with the implementation of Cloud Services, and the majority of resources and efforts are occupied with implementation challenges, as well as security concerns related to the protection of the information of Canadians.</p><br />
<p class="expand mw-collapsible-content">However, the inception of containers into the market has shown that large-scale organizations, who are involved in cloud-native application development as well as networking, can benefit greatly from the use of containers [7]. Although the infrastructure applications providing cloud services can be based solely on Virtual Machines (VMs), the maintenance costs associated with running different operating systems on individual VMs outweighs the benefit [6]. Containers and Containerization is a replacement and/or complimentary architecture for VMs. As the GC moves toward cloud services and development of cloud-native applications, the use of containers and orchestrating them with Kubernetes can become an integral part the GC IT architecture. </p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p>The primary business value impact of Kubernetes is the technology’s portability, and mobility independent of the environment. Its ability to manage, and orchestrate an organization’s application containers is a marked benefit. Kubernetes secondary business value is that it enables enterprise high-velocity, meaning that every product team can safely ship updates many times a day, deploy instantly, observe results in real time, and use this feedback to roll containers forward or back with the goal to improve the customer experience as fast as possible. </p><br />
<p>In the age of modern web services, users expect their applications to be available 24/7, and developers expect the ability to deploy new versions of those applications several times a day with minimal downtime. Containers have become one of the main ways in which to manage applications across enterprise IT infrastructure and also one of the most difficult areas to manage effectively.</p><br />
<p>Kubernetes, as an open source system, is a technology that can administer and manage a large number of containerized applications spread across clusters of servers while providing basic mechanisms for deployment, maintenance, and scaling of applications. An application container is a standard unit of software that packages code and all its dependencies so the application runs quickly and reliably from one computing environment to another. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way. </p><br />
<p>Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. This decoupling allows container-based applications to be deployed easily and consistently, regardless of whether the target environment is a private data center, the public cloud, or even a developer’s personal laptop. An additional benefit to containerization is that the Operating System (OS) is not running as hard. </p><br />
<p class="inline">Since Kubernetes is open source, it allows the enterprise freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, and the ability to effortlessly move workloads. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes groups containers that make up an application into logical units for easy management and discovery. </p><p class="expand inline mw-collapsible-content">The abstractions in Kubernetes allows deployment of containerized applications to a cluster without tying them specifically to individual machines (i.e. Virtual Machines). Applications can be co-located on the same machines without impacting each other. This means that tasks from multiple users can be packed onto fewer machines. This provides greater efficiency and reduces the cost on hardware as less machines are used. </p><br />
<p>Kubernetes contains tools for orchestration, secrets management, service discovery, scaling and load balancing and includes automatic bin packing to place containers with the optimal resources, and it applies configurations via configuration management features. It protects container workloads by rolling out or rolling back changes and offers availability and quality checks for containers -- replacing or restarting failed containers. As requirements change, a user can move container workloads in Kubernetes from one cloud provider or hosting infrastructure to another without changing the code. This is a great value to developers as their work is protected and an audit trail of changes is available.</p><br />
<p class="expand mw-collapsible-content">The core concepts of Kubernetes which enables high velocity are immutability, declarative configuration and self-healing systems. </p><br />
<p>Containers and Kubernetes encourage developers to build distributed systems that adhere to the principles of immutable infrastructure. In immutable infrastructure an artifact created, will not be changed upon user modifications. To update applications in an immutable infrastructure, a new container image is built with a new tag, and is deployed, terminating the old container with the old image version. In this way, the enterprise always has an artifact record of what was done and if there was an error in the new image. If an error is detected the container is rolled back to the previous image. Anything that goes into a container has a text file. Text files can be treated like application source code and provisions immutability.</p><br />
<p class="expand mw-collapsible-content">Declarative configuration enables the user to describe exactly what state the system should be in. Traditional tools of development such as source control, unit tests etc. can be used with declarative configurations in ways that are impossible with imperative configurations. Imperative systems describe how to get from point A to B, but rarely include reverse instructions to get back. Kubernetes declarative configuration makes rollbacks fairly easy which is impossible with imperative configurations. </p><br />
<p class="expand mw-collapsible-content">Lastly, Kubernetes has a means of self-healing. When Kubernetes receives a desired state configuration, it does not simply take actions to make the current state match the desired state at a single time, but it will continuously take actions to ensure it stays that way as time passes by. </p><br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br><b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour [1]. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges. </p><br><b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge. </p><br><b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br><b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward [1]. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service. </p><br><b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060https://wiki.gccollab.ca/index.php?title=Technology_Trends/Kubernetes&diff=11073Technology Trends/Kubernetes2019-07-17T18:19:08Z<p>Kpere060: </p>
<hr />
<div>{{DISPLAYTITLE:<span style="position: absolute; clip: rect(1px 1px 1px 1px); clip: rect(1px, 1px, 1px, 1px);">{{FULLPAGENAME}}</span>}}<br />
[[fr:Tendances_Technologiques/Chaîne_de_Blocs]]<br />
<br />
<div class="mw-collapsible" data-expandtext="Show Detailed View" data-collapsetext="Hide Detailed View"><br />
<div class="sidetable"><br />
<table class="wikitable"><br />
<tr><br />
<th class="breadcrumb" colspan="2"><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Technology_Trends|Technology Trends]]</th><br />
<th> / </th><br />
<th>Kubernetes</th><br />
</tr><br />
</table><br />
<table class="breadcrumb-table"><br />
<tr><br />
<th>[[Tendances_Technologiques|Tendances Technologiques]]</th><br />
<th> / </th><br />
<th>[[Tendances_Technologiques|Kubernetes]]</th><br />
</tr><br />
</table><br />
</th><br />
</tr><br />
<tr><td colspan="2" class="logo">[]</td></tr><br />
<tr><br />
<th>Status</th><br />
<td>Translation</td><br />
</tr><br />
<tr><br />
<th>Initial release</th><br />
<td>June 25, 2019</td><br />
</tr><br />
<tr><br />
<th>Latest version</th><br />
<td>July 17, 2019</td><br />
</tr><br />
<tr><br />
<th>Official publication</th><br />
<td>[[Media:EN_-_Digital_Twins_v0.1_EN_Published.pdf|Digital Twins.pdf]]</td><br />
</tr><br />
<tr><td colspan="2" class="disclaimer"><table><tr><br />
<td>[[File:Traffic_cone.png|40px]]</td><br />
<td class="disclaimerText">This page is a work in progress. We welcome your feedback. Please use the discussion page for suggestions and comments. When the page is approved and finalized, we will send it for translation.</td><br />
</tr></table></td></tr><br />
</table><br />
</div><br />
<br />
<br><p><b>Kubernetes</b>also known as K8s, is a portable, extensible open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads.</p><br />
<br />
<div class="mw-collapsible-toggle btn" style="float: left; display: block;"><br />
<div class="toggle mw-collapsible-toggle-collapsed" role="button" tabindex="0"><span class="mw-collapsible-text">Hide Detailed View</span></div><br />
</div><br><br><br />
<br />
<h2>Business Brief</h2><br />
<p>Application containers are increasingly being used in the infrastructure for cloud-native and microservice applications. Specifically, Docker is the engine most commonly used to create containers. Many application developers have noted the value of containers in dependency management as it packages the application and its dependencies, libraries, other binaries into a container to abstract away the differences in the OS distribution and underlying infrastructure. This solves many issues caused by differences in runtime environments when a software moves from one environment to another one.</p><br />
<p class="inline">Containers improve applications’ portability and scalability that enable applications to be released and updated in an easy and fast way without downtime. However, there still exits a demand in the management of containers when the services provided need to be deployed across a cluster of host servers to achieve high-availability and disaster recovery. This is where container deployment in cluster and management tools like Kubernetes provide their value. Developers can now begin to deploy and orchestrate services as a collection of containers across a cluster of servers. Container resource requirements can be explicitly declared that allows developers to bundle application code with an environment configurations. </p><p class="expand inline mw-collapsible-content">Also by increasing container density, resources can be used more efficiently and thus it in turn improves hardware usage [1]. Containers provide applications with isolations, so that a development team can be made responsible for specific containers.</p><br />
<p>Kubernetes also known as K8s, is a portable, extensible open-source platform for managing containerized workloads and services that facilitates both declarative configuration and automation. Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads. Kubernetes also offers self-healing, automatized rollout and rollback features, which greatly improve operation high-availability and flexibility. One of the biggest advantages of Kubernetes is the flexibility it provides. Many PaaS packaging dictate specific frameworks, are catered towards specific workloads, or impose limitations which language runtimes can be used [1]. These issues are all eliminated with Kubernetes. Therefore, if an organization’s application is capable of being run on a container, Kubernetes is a viable option for container orchestration.</p><br />
<br />
<h2>Technology Brief</h2><br />
<p>The Kubernetes cluster or deployment can be broken down into several components. The Kubernetes “master” is the machine in charge of managing other “node” machines. The “node is the machine in charge of actually running tasks fed to it via the user or the “master”. The master and nodes can be either a physical or virtual machines. In each Kubernetes cluster, there is one master and multiple nodes machines. The main goal of Kubernetes is to achieve “Desired State Management”. The “master” is fed a specific configuration through its RESTful API which it exposes to the user, and the “master” is then responsible for running this configuration across its set of “node”. The nodes can be thought of as host of containers. They communicate with the “master” through the agent in each node --“Kubelet” process. To establish a specific configuration in Kubernetes, the “master is fed a deployment file with the “.yaml” extension. This file contains a variety of configuration information. Within this information are “Pods” and “replicas”. There is a concept of Pod in Kubernetes and it can be described as a logic collection of containers which are managed as a single application. Resources can be shared within a Pod, these resources include shared storage (Volumes), a unique cluster of IP addresses, and information about how to run each container. A Pod can be thought of as the basic unit of the Kubernetes object model, it represents the deployment of a single instance of an application in Kubernetes [8]. A Pod can encapsulate one or more application containers. Two models exist for how Pods are deployed within a cluster. The “one-Pod-per-container” means a single pod will be associated with a single container. There can also be multiple containers that run within a single Pod, where these containers may need to communicate with one another as they share resources. In either model, the Pod can be thought of as a wrapper around the application containers. Kubernetes manages the Pod instances rather than managing the containers directly. The Pods are run on the Node machines to perform tasks. Replicas are simply instances of the Pods. Within the “.yaml” deployment file, specifications are instructing the “master” machine how many instances/replicas of each Pod to run, which is handled by a replication controller [8]. When a node dies or a running Pod experiences an unexpected termination, the replication controller will take note take care of this by creating the appropriate number of Pods [8].</p><br />
<h2>Industry Usage</h2><br />
<p class="inline">Kubernetes is an open source system and many companies have begun to adopt it into their existing architecture as well as adapt it to their specific needs. It was originally developed by Google and was made an open source project in 2014. The Cloud Native Computing Foundation is a project of the Linux Foundation providing a community for different companies who are seeking to develop Kubernetes and other container orchestration projects. Several major cloud providers and platforms including Google Cloud Compute, HP Helion Cloud, RedHat Openshift, VMware Cloud, and Windows Azure all support the use of Kubernetes [7]. A survey, performed by iDatalabs in 2017, found 2,867 companies are currently using Kubernetes. These companies are generally located in the United States and are also most the computer software industry. Companies on the list hire between 50 and 200 employees, and accumulate 1M-100M in revenue per year. Some of the major companies on this list include GoDaddy inc, Pivotal Software inc, Globant SA, and Splunk inc</p><p class="expand inline mw-collapsible-content">. Kubernetes own approximately 8.6% of the market share within the virtualization management software category [9]. </p><br />
<h2>Canadian Government Use</h2><br />
<p>There is a lack of documented Government of Canada (GC) initiatives and programs promoting the current and future use of Kubernetes technology. As a GC strategic IT item, Kubernetes is absent from both the GC’s Digital Operations Strategic Plan: 2018-2022 and the GC Strategic Plan for Information Management and Information Technology 2017 to 2021. This may be due to the fact that the GC is currently grappling with the implementation of Cloud Services, and the majority of resources and efforts are occupied with implementation challenges, as well as security concerns related to the protection of the information of Canadians.</p><br />
<p class="expand mw-collapsible-content">However, the inception of containers into the market has shown that large-scale organizations, who are involved in cloud-native application development as well as networking, can benefit greatly from the use of containers [7]. Although the infrastructure applications providing cloud services can be based solely on Virtual Machines (VMs), the maintenance costs associated with running different operating systems on individual VMs outweighs the benefit [6]. Containers and Containerization is a replacement and/or complimentary architecture for VMs. As the GC moves toward cloud services and development of cloud-native applications, the use of containers and orchestrating them with Kubernetes can become an integral part the GC IT architecture. </p><br />
<br />
<h2>Implications for Government Agencies</h2><br />
<h3>Shared Services Canada (SSC)</h3><br />
<h4>Value Proposition</h4><br />
<p>The primary business value impact of Kubernetes is the technology’s portability, and mobility independent of the environment. Its ability to manage, and orchestrate an organization’s application containers is a marked benefit. Kubernetes secondary business value is that it enables enterprise high-velocity, meaning that every product team can safely ship updates many times a day, deploy instantly, observe results in real time, and use this feedback to roll containers forward or back with the goal to improve the customer experience as fast as possible. </p><br />
<p>In the age of modern web services, users expect their applications to be available 24/7, and developers expect the ability to deploy new versions of those applications several times a day with minimal downtime. Containers have become one of the main ways in which to manage applications across enterprise IT infrastructure and also one of the most difficult areas to manage effectively.</p><br />
<p>Kubernetes, as an open source system, is a technology that can administer and manage a large number of containerized applications spread across clusters of servers while providing basic mechanisms for deployment, maintenance, and scaling of applications. An application container is a standard unit of software that packages code and all its dependencies so the application runs quickly and reliably from one computing environment to another. Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way. </p><br />
<p>Containers offer a logical packaging mechanism in which applications can be abstracted from the environment in which they actually run. This decoupling allows container-based applications to be deployed easily and consistently, regardless of whether the target environment is a private data center, the public cloud, or even a developer’s personal laptop. An additional benefit to containerization is that the Operating System (OS) is not running as hard. </p><br />
<p class="inline">Since Kubernetes is open source, it allows the enterprise freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, and the ability to effortlessly move workloads. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host. Kubernetes groups containers that make up an application into logical units for easy management and discovery. </p><p class="expand inline mw-collapsible-content">The abstractions in Kubernetes allows deployment of containerized applications to a cluster without tying them specifically to individual machines (i.e. Virtual Machines). Applications can be co-located on the same machines without impacting each other. This means that tasks from multiple users can be packed onto fewer machines. This provides greater efficiency and reduces the cost on hardware as less machines are used. </p><br />
<p>Kubernetes contains tools for orchestration, secrets management, service discovery, scaling and load balancing and includes automatic bin packing to place containers with the optimal resources, and it applies configurations via configuration management features. It protects container workloads by rolling out or rolling back changes and offers availability and quality checks for containers -- replacing or restarting failed containers. As requirements change, a user can move container workloads in Kubernetes from one cloud provider or hosting infrastructure to another without changing the code. This is a great value to developers as their work is protected and an audit trail of changes is available.</p><br />
<p class="expand mw-collapsible-content">The core concepts of Kubernetes which enables high velocity are immutability, declarative configuration and self-healing systems. </p><br />
<p>Containers and Kubernetes encourage developers to build distributed systems that adhere to the principles of immutable infrastructure. In immutable infrastructure an artifact created, will not be changed upon user modifications. To update applications in an immutable infrastructure, a new container image is built with a new tag, and is deployed, terminating the old container with the old image version. In this way, the enterprise always has an artifact record of what was done and if there was an error in the new image. If an error is detected the container is rolled back to the previous image. Anything that goes into a container has a text file. Text files can be treated like application source code and provisions immutability.</p><br />
<p class="expand mw-collapsible-content">Declarative configuration enables the user to describe exactly what state the system should be in. Traditional tools of development such as source control, unit tests etc. can be used with declarative configurations in ways that are impossible with imperative configurations. Imperative systems describe how to get from point A to B, but rarely include reverse instructions to get back. Kubernetes declarative configuration makes rollbacks fairly easy which is impossible with imperative configurations. </p><br />
<p class="expand mw-collapsible-content">Lastly, Kubernetes has a means of self-healing. When Kubernetes receives a desired state configuration, it does not simply take actions to make the current state match the desired state at a single time, but it will continuously take actions to ensure it stays that way as time passes by. </p><br />
<h4>Challenges</h4><br />
<p>The greatest challenge in regards to Kubernetes is its complexity. However, security, storage and networking, maturity, and competing enterprise transformation priorities are also challenges facing the Kubernetes technology.</p><br />
<b>Kubernetes Complexity and Analyst Experience</b><br />
<p>There is the challenge of a lack of organizational and analyst experience with container management and in using Kubernetes. Managing, updating, and changing a Kubernetes cluster can be operationally complex, more so if the analysts have a lack of experience. The system itself does provide a solid base of infrastructure for a Platform as a Service (PaaS) framework, which can reduce the complexity for developers. However, testing within a Kubernetes environment is still a complex task. Although its use cases in testing are well noted, testing several moving parts of an infrastructure to determine proper application functionality is still a more difficult endeavour [1]. This means a lot of new learning will be needed for operations teams developing and managing Kubernetes infrastructure. The larger the company, the more likely the Kubernetes user is to face container challenges. </p><br />
<b>Security</b><br />
<p>In a distributed, highly scalable environment, traditional and typical security patterns will not cover all threats. Security will have to be aligned for containers and in the context of Kubernetes. It is critical for operations teams to understand Kubernetes security in terms of containers, deployment, and network security. Security perimeters are porous, containers must be secured at the node level, but also through the image and registry. Security practices in the context of various deployment models will be a persistent challenge. </p><br />
<b>Storage & Networking</b><br />
<p>Storage and networking technologies are pillars of data center infrastructure, but were designed originally for client/server and virtualized environments. Container technologies are leading companies to rethink how storage and networking technologies function and operate. Architectures are becoming more application-oriented and storage does not necessarily live on the same machine as the application or its services. Larger companies tend to run more containers, and to do so in scaled-out production environments requires new approaches to infrastructure. </p><br />
<p>Some legacy systems can run containers and only sometimes can VMs can be replaced by containers. There may be significant engineering consequences to existing legacy systems if containerization and Kubernetes is implemented in a legacy system not designed to handle that change. Some Legacy systems may require refactoring and making it more suitable for containerization. Some pieces of a system may be able to be broken off and containerized. In general, anything facing the internet should be run in containers.</p><br />
<b>Maturity</b><br />
<p>Kubernetes maturity as a technology is still being tested by organizations. For now, Kubernetes is the market leader and the standardized means of orchestrating containers and deploying distributed applications. Google is the primary commercial organization behind Kubernetes; however they do not support Kubernetes as a software product. It offers a commercial managed Kubernetes service known as GKE but not as a software. This can be viewed as both a strength and a weakness. Without commercialization, the user is granted more flexibility with how Kubernetes can be implemented in their infrastructure; However, without a concrete set of standards of the services that Kubernetes can offer, there is a risk that Google’s continuous support cannot be guaranteed. Its donation of Kubernetes code and intellectual property to the Cloud Native Computing Foundation does minimize this risk since there is still an organization enforcing the proper standards and verifying services Kubernetes can offer moving forward [1]. It is also important to note that the organizational challenges that Kubernetes users face have been more dependent on the size of the organization using it.</p><br />
<p>Kubernetes faces competition from other scheduler and orchestrator technologies, such as Docker Swarm and Mesosphere DC/OS. While Kubernetes is sometimes used to manage Docker containers, it also competes with the native clustering capabilities of Docker Swarm. However, Kubernetes can be run on a public cloud service or on-premises, is highly modular, open source, and has a vibrant community. Companies of all sizes are investing into it, and many cloud providers offer Kubernetes as a service. </p><br />
<b class="expand mw-collapsible-content">Competing Enterprise Transformation Priorities</b><br />
<p class="expand mw-collapsible-content">The last challenge facing Kubernetes initiative development and implementation is its place in an organization’s IT transformation priority list. Often there are many higher priority initiatives that can take president over Kubernetes projects.</p><br />
<br />
<br />
<h4>Considerations</h4><br />
<b>Strategic Resourcing and Network Planning</b><br />
<p>A strategic approach to Kubernetes investments will need to be developed to ensure opportunities are properly leveraged. The GC invests a significant portion of its annual budget on IT and supporting infrastructure. Without strategic Kubernetes direction the fragmented approaches to IT investments, coupled with rapid developing technology and disjointed business practices, can undermine effective and efficient delivery of GC programs and services. A clear vision and mandate for how Kubernetes will transform services, and what the end-state Kubernetes initiative is supposed to look like, is a prominent consideration. </p><br />
<p>SSC should consider defining a network strategy for Kubernetes adoption. Multiple factors should be taken into account, including the amount of resources, funding, and expertise that will be required for the development and experimentation with Kubernetes technologies. Calculation of resource requirements including CPU, memory, storage, etc. at the start of Kubernetes projects is imperative. Considerations include whether or not an in-house Kubernetes solution is required or if a solution can be procured. Other strategy considerations include analyzing different orchestration approaches for different application use cases.</p><br />
<b>Complexity and Skills Gap</b><br />
<p>Kubernetes is a good technology and the de facto standard for orchestrating containers, and containers are the future of modern software delivery. But it is notoriously complex to manage for enterprise workloads, where Service Level Agreements (SLAs) are critical. The operational pain of managing production-grade Kubernetes is further complicated by the industry-wide talent scarcity and skills gap. Most organizations today struggle to hire Kubernetes experts, and even these “experts” lack advanced Kubernetes experience to ensure smooth operations at scale. SSC will need to be cautious in implementing Kubernetes and having the right staff experienced and comfortable in its use.</p><br />
<b>Customization and Integration Still Required</b><br />
<p>Kubernetes technology and ecosystem are evolving rapidly, because of its relatively new state, it is hard to find packaged solutions with complete out-of-the-box support for complex, large-scale enterprise scenarios. As a large and sophisticated enterprise organization, SSC will need to devote significant resources on customization and training. Enterprise Architecture pros will need to focus on the whole architecture of cloud-native applications as well as keep a close watch on technology evolution and industry. </p><br />
<p>Implementation usually takes longer than expected, however the consensus in the New Stack’s Kubernetes User Experience Survey is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources. Additionally, implementation takes longer than expected. The consensus is that Kubernetes reduces code deployment times, and increases the frequency of those deployments. However, in the short run, the implementation phase does consume more human resources.</p><br />
<b>Pilot Small and Scale Success</b><br />
<p>SSC may wish to consider evaluating the current Service Catalogue in order to determine where Kubernetes can be leveraged first to improve efficiencies, reduce costs, and reduce administrative burdens of existing services as well as how a new Kubernetes service could be delivered on a consistent basis. Any new procurements of devices or platforms should have high market value and can be on-boarded easily onto the GC network. SSC should avoid applying in-house Kubernetes for production mission-critical apps. Failure of in-house deployments is high and thus should be avoided. SSC should pilot and establish a Kubernetes test cluster. With all new cloud-based technologies, piloting is preferred. Focus should first be on a narrow set of objectives and a single application scenario to stand up a test cluster.</p><br />
<b>Implement Robust Monitoring, Logging, and Audit Practices and Tools</b><br />
<p>Monitoring provides visibility and detailed metrics of Kubernetes infrastructure. This includes granular metrics on usage and performance across all cloud providers or private data centers, regions, servers, networks, storage, and individual VMs or containers. Improving data center efficiency and utilization on both on-premises and public cloud resources is the goal. Additionally, logging is a complementary function and required capability for effective monitoring is also a goal. Logging ensures that logs at every layer of the architecture are all captured for analysis, troubleshooting and diagnosis. Centralized, distributed, log management and visualization is a key capability. Lastly, routine auditing, no matter the checks and balances put in place, will cover topics that normal monitoring will not cover. Traditionally, auditing is as a manual process, but the automated tooling in the Kubernetes space is quickly improving.</p><br />
<b>Security</b><br />
<p>Security is a critical part of cloud native applications and Kubernetes is no exception. Security is a constant throughout the container lifecycle and it is required throughout the design, development, DevOps, and infrastructure choices for container-based applications. A range of technology choices are available to cover various areas such as application-level security and the security of the container and infrastructure itself. Different tools that provide certification and security for what goes inside the container itself (such as image registry, image signing, packaging), Common Vulnerability Exposures/Enumeration (CVE) scans, and more. SSC will need to ensure appropriate security measures are used with any new Kubernetes initiatives, including the contents of the containers being orchestrated.</p><br />
<br />
<h2>References</h2><br />
<br />
</div><br />
<br />
{{#css:<br />
<br />
div>a>img {<br />
width: 100%;<br />
height: auto;<br />
}<br />
<br />
#firstHeading::after{<br />
content:"Kubernetes";<br />
}<br />
<br />
/* PC Formatting */<br />
<br />
.sidetable{ float: right; width: 25%; cursor: auto; text-align: center;}<br />
.wikitable{ margin: 0px 0px 0px 0px; }<br />
.logo{ text-align: center; }<br />
.disclaimer{ border: 1px solid red; background-color: rgba(255, 0, 0, 0.3); }<br />
<br />
.breadcrumb{<br />
background-color: rgb(248, 249, 250);<br />
border: 1px solid rgb(162, 169, 177);<br />
}<br />
.breadcrumb-table{ margin: auto; }<br />
<br />
.expand{ background-color: rgba(242, 109, 33, 0.2); }<br />
.mw-collapsible-text{ text-align:left; }<br />
.inline{ display: inline; }<br />
.inline-spacer{ display: block; }<br />
<br />
.btn {<br />
display: inline-block;<br />
margin-bottom: 0;<br />
font-weight: 400;<br />
text-align: center;<br />
white-space: nowrap;<br />
vertical-align: middle;<br />
-ms-touch-action: manipulation;<br />
touch-action: manipulation;<br />
cursor: pointer;<br />
background-image: none;<br />
border: 1px solid transparent;<br />
padding: 6px 12px;<br />
font-size: 14px;<br />
line-height: 1.42857143;<br />
border-radius: 4px;<br />
-webkit-user-select: none;<br />
-moz-user-select: none;<br />
-ms-user-select: none;<br />
user-select: none;<br />
<br />
width: 230px;<br />
color: rgba(242, 109, 33, 1);<br />
background-color: rgba(242, 109, 33, 0.1);<br />
border-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.btn:hover {<br />
color: rgba(255, 255, 255, 1);<br />
background-color: rgba(242, 109, 33, 1);<br />
text-decoration: none;<br />
}<br />
<br />
.hypecycle {<br />
width: 80%;<br />
margin: auto;<br />
}<br />
<br />
.container{<br />
width: 100%;<br />
display: block;<br />
}<br />
<br />
.row{<br />
width: 100%;<br />
display: table;<br />
content: " ";<br />
}<br />
<br />
.col-sm-8{<br />
width: 66.6666666666%;<br />
float:left;<br />
}<br />
<br />
.col-sm-4{<br />
width: 33.33333333333%;<br />
float:left;<br />
}<br />
<br />
.hypecycleTable{<br />
width: 100%;<br />
}<br />
<br />
/* Mobile Formatting */<br />
@media (max-width: 992px){<br />
.sidetable{ width: 80%; margin-left: 10%; margin-right: 10%;}<br />
<br />
.container{<br />
display: block;<br />
clear: both;<br />
}<br />
<br />
.col-sm-8,<br />
.col-sm-6,<br />
.col-sm-4{<br />
display: block;<br />
width: 100%;<br />
clear: both;<br />
margin: auto;<br />
}<br />
}<br />
@media (max-width: 600px){<br />
.sidetable{ width: 100%; margin-left: 0px; margin-right: 0px;}<br />
}<br />
<br />
}}</div>Kpere060