Cortana Analytics Suite. Microsoft Data Platform User Group

Transkript

1 Cortana Analytics Suite Microsoft Data Platform User Group

2 Cortana Analytics Suite Information Management Big Data Stores Machine Learning and Analytics Dashboards and Visualizations Business apps Azure Data Factory Azure Machine Learning Power BI Personal Digital Assistant Custom apps Azure Data Catalog Azure Event Hub Azure Data Lake Store Azure SQL Data Warehouse ` Azure HDInsight (Hadoop) Azure Data Lake Analytics Azure Stream Analytics Cortana Perceptual Intelligence Face, vision Speech, text Business Scenarios People Sensors and devices Recommendations, customer churn, forecasting, etc. Automated Systems DATA INTELLIGENCE ACTION

3 A data lake Uses Bottoms-Up Approach Ingest regardless of requirements Store in native format without schema definition Analyze Using analytic engines like Hadoop Devices Batch queries Interactive queries Real-time analytics Machine Learning Data warehouse

4 Data Lake Store uses HDFS

5 Analytics: Two Form Factors HDInsight managed Hadoop clusters ADLA analytics service n1 n2 n3 n4 HDInsight Cluster lots of containers YARN Layer Hive/Pig/etc. job U-SQL/Hive/Pig job ADLA Account Input File Output File Storage (Blob or ADLS)

6 ADLA complements HDInsight ADLA and HDInsight target the same scenarios, tools, and customers HDInsight is for developers familiar with the Open Source stack and tools: Java, Eclipse, Hive, etc. ADLA enables customers to leverage their existing experience with Languages (C# & SQL) and Tools (PowerShell). HDInsight clusters offer customization, control, and flexibility in a managed Hadoop cluster form factor ADLA offers convenience, efficiency, automatic scale, and management in a job service form factor

7 Hvordan vi har sett på teknologiene

8 Struktur Oppsett Bruksscenario Forkunnskaper Modenhet Grensesnitt Support Lockin Bruk Ytelse Rapportering til powerbi/excel Where Join Prismodell 8

9 Noen glimt fra prosjektet

10 Data Lake Store - bruksscenario Datalager for filer, eller ADLA tabeller Datakilde og mål for HDInsight (Hbase) og Data Lake Analytics Kan være grunnlag for rapportering i PowerBI Støtte for PowerQuery er planlagt i februar

11 Data Lake - oppsett Enkelt oppsett Egen konfigurasjon gjøres for kjøring av jobber Parallelisering Priority Default: Maks 3 samtidige jobber Maks 20 ADLAU 11

12 Data Lake Analytics - Bruksscenario Data Lake Analytics med U-SQL brukes for å lese filer, behandle de, og skrive resultat til fil eller Data Lake Analytics tabell Starter med Extractor Innebygd extractor for separerte filer (flate filer) Finnes eksempel på JSON og XML extrator Gjør videre transformsteg med C# eller U-SQL syntaks Laster til fil eller U-SQL tabell U-SQL tabell lagres i Data Lake Store. Kan sikre bedre clustering av lagring Kan brukes til å kombinere filer, gjøre uttrekk og mye av det som gjøres med standard ETL. 12

13 Noen U-SQL jobber vi har jobbet med Jobbnavn RemoveHeaderSplitColum ns SplitFileToDailyFile SelectTest SplitFile JoinFile CreatePOCTable IncrementalLoad PrepareHiveFileData Beskrivelse Fjerner header fra fil og splitter resterende fil i kolonner med antatt kommaseparert format Splitter File (samlet datasett) til datasett for enkeltdager Gjør test på ytelse for WHERE-betingelse Deler opp File i ny fil som har unike PortfolioID og PortfolioName, før JoinFile Joiner sammen to filer Lager tabell for POC Laster filsett fra EventHub-mappe til fil og U-SQL tabell Flytter data til mappe for Hivetabell 13

14 U-SQL; nesten som SQL Hive Select targetdate, currency, market, instrumenttype, rowtag, cast(avgprice as decimal(30,15)) from hiveonefile where currency = "EUR" and market = "CO2" and instrumenttype = "FUT" and Rowtag in ( "NEDEC5", "NEDEC6", "NEDEC7", "NEDEC8" ) Data Lake Analytics = EXTRACT Area String, AvgPrice String, USING = SELECT TargetDate, Currency, Market, InstrumentType, RowTag, Convert.ToDecimal(AvgPrice) AS AvgPrice WHERE Currency == "EUR" AND Market == "CO2" AND InstrumentType == "FUT" AND (RowTag == "NEDEC5" OR RowTag == "NEDEC6" OR RowTag == "NEDEC7" OR RowTag == "NEDEC8") ; USING Outputters.Csv(); 14

15 Custom U-SQL row processor U-SQL //Read some = EXTRACT txt string USING Extractors.Text(delimiter: '\b'); //remove header = SELECT * WHERE NOT (txt == "sep=," OR txt.substring(0,4) == "Area"); //split = PRODUCE Area String, AvgPrice String, AvgpriceBuy String, USING new USQL_UDO.SplitColumns(); USQL_UDO.SplitColumns() namespace USQL_UDO { public class SplitColumns : IProcessor { public override IRow Process(IRow input, IUpdatableRow output) { string txt = input.get<string>("txt"); string[] stringseparators = new string[] { "," }; string[] columns; columns = txt.split(stringseparators, StringSplitOptions.None); for (int i = 0; i<columns.length; i++) { output.set(i, columns[i]); } return output.asreadonly(); } } } 15

16 Incremental Load string = "/InputFiles/EventHub/"; string + "{date:yyyy}{date:mm}{date:dd}-{*}"; //Read some = EXTRACT txt string, date DateTime //date settes fra ADL_File-strengen USING Extractors.Text(delimiter: '\b'); //create new = SELECT * UNION SELECT * //insert into table INSERT INTO DatalakePOC.dbo.Result SELECT * //Read existing = SELECT * FROM DatalakePOC.dbo.TRMResult; //output table data to file TO "/OutputFiles/TableOutput.csv" USING Outputters.Csv(); //empty existing table TRUNCATE TABLE Result; 16

17 Hvorfor bruke en ADLA tabell? 17

18 Bruk Visual Studio! 18

19 Analyse av kjøretid 19

20 Sammenligning av teknologier 20

21 Sammenligning av teknologier Teknologi Oppsett Forkunn skaper Modenhet Grenses nitt Support Lockin Ytelse Rapport Ytelse WHERE Ytelse JOIN Pris ADLS 11 m N/A N/A Lav ADLA N/A 2 m 3 m Lav Hbase m 2 m 3 m Medium Spark N/A 2 s 38 s Høy ADWH 14 m 3 m 56 s Medium 21

22 For mer info Helge Rege Gårdsvoll 22

23 About Capgemini With more than 180,000 people in over 40 countries, Capgemini is one of the world's foremost providers of consulting, technology and outsourcing services. The Group reported 2014 global revenues of EUR billion. Together with its clients, Capgemini creates and delivers business and technology solutions that fit their needs and drive the results they want. A deeply multicultural organization, Capgemini has developed its own way of working, the Collaborative Business Experience, and draws on Rightshore, its worldwide delivery model Learn more about us at The information contained in this presentation is proprietary. Copyright 2015 Capgemini. All rights reserved. Rightshore is a trademark belonging to Capgemini.