Showing posts with label machine learning. Show all posts
Showing posts with label machine learning. Show all posts

Wednesday, March 29, 2023

Boost your productivity and creativity with ChatGPT as a C# developer with these 5 tips




ChatGPT is a language model that is optimized for conversational interfaces. It can interact with users in a natural and engaging way, and can also perform tasks such as generating code, optimizing code, and testing code .

 

Here are 5 ways you can use ChatGPT to boost your productivity as a C# developer:

 

Code Optimization

 Optimizing your code can improve its performance, readability, and maintainability. However, it can also be challenging and complex, especially for large and legacy codebases. ChatGPT can help you optimize your code by suggesting improvements and refactorings based on best practices and coding standards. You can ask ChatGPT to review your code and provide feedback on how to make it more efficient, elegant, and consistent. You can also ask ChatGPT to apply the suggested changes automatically or manually.

Chat GPT is a powerful tool that can help you optimize your code with examples for c#. Chat GPT is a chatbot that uses natural language processing and deep learning to understand your coding problems and generate solutions. You can ask Chat GPT to optimize your code for speed, memory, readability, or any other criteria you specify. Chat GPT can also provide you with examples of how to use different features or libraries in c#, such as LINQ, async/await, delegates, etc.

 

For example, suppose you have a code snippet that calculates the sum of all the even numbers in an array:

 


int[] numbers = {1, 2, 3, 4, 5, 6};

int sum = 0;

for (int i = 0; i < numbers.Length; i++)
{
    if (numbers[i] % 2 == 0)
    {
        sum += numbers[i];
    }
}

 

You can ask Chat GPT to optimize this code for speed and readability. Chat GPT might generate something like this:

 


int[] numbers = {1, 2, 3, 4, 5, 6};

int sum = numbers.Where(n => n % 2 == 0).Sum();

 

This code uses LINQ to filter the even numbers and sum them in one line. This is faster and more readable than the original code.

 

As you can see, Chat GPT can optimize your code with examples for c# in various ways. You can use Chat GPT to improve your coding skills and productivity. Chat GPT is your friendly and helpful coding assistant.

Code Generation

Sometimes you may need to write new code from scratch or extend existing code with new features or functionalities. ChatGPT can help you generate code based on your natural language description or specification. You can simply tell ChatGPT what you want to achieve or implement, and it will write the corresponding C# code for you. This can speed up your development process and also help you explore new ideas and possibilities.

 

Another example is if you want to learn how to use async/await in c#. You can ask Chat GPT to show you an example of how to make an asynchronous HTTP request using HttpClient. Chat GPT might generate something like this:

 


using System.Net.Http;
using System.Threading.Tasks;


async Task GetResponseAsync(string url)
{
    HttpClient client = new HttpClient();

    HttpResponseMessage response = await client.GetAsync(url);

    response.EnsureSuccessStatusCode();

    string content = await response.Content.ReadAsStringAsync();

    return content;
}

 

This code uses the async and await keywords to make an asynchronous HTTP request and return the response content as a string. This is more efficient and elegant than using synchronous methods or callbacks

Code Documentation

 Documenting your code is important for making it understandable, reusable, and maintainable. However, it can also be boring and repetitive, especially for complex and lengthy code. ChatGPT can help you document your code by generating comments, summaries, and descriptions based on your code logic and structure. You can ask ChatGPT to document your code at different levels of granularity, such as methods, classes, modules, or projects. You can also ask ChatGPT to update your documentation when you make changes to your code.

 

It can summarize your code by extracting the main logic and functionality of your program and presenting it in natural language. For example, if you have a code snippet like this in C#:


using System;

class Program
{

    static void Main(string[] args)
    {
        int x = 10;
        int y = 20;
        int z = x + y;

        Console.WriteLine("The sum of x and y is " + z);

    }
}

Chat GPT will then summarize the code like this "The program defines three variables: x, y, and z. It assigns the values 10 and 20 to x and y respectively. It calculates the sum of x and y and assigns it to z. It prints the value of z to the console".


Code Debugging

 Debugging your code can be frustrating and time-consuming, especially when you encounter errors or bugs that are hard to find or fix. ChatGPT can help you debug your code by providing suggestions and solutions based on your error messages or test results. You can ask ChatGPT to explain the cause of an error or bug, suggest possible fixes or workarounds, or apply the fixes automatically or manually.


 Unit Test Generation

 Writing unit tests can be tedious and time-consuming, but they are essential for ensuring the quality and reliability of your code. ChatGPT can help you generate unit tests automatically based on your code and specifications. You can simply provide ChatGPT with your code snippet and some test cases, and it will write the corresponding unit test code for you. This can save you a lot of time and effort, and also help you catch bugs and errors early on.


These are just some of the ways you can use ChatGPT to increase productivity for software developers who use C#. ChatGPT is a powerful and versatile tool that can handle a variety of tasks and scenarios related to C# development. You can try ChatGPT yourself at chat.openai.com or learn more about it at openai.com/blog/chatgpt.



**Note, this was written using AI and was a test of Chat GPT to see if it would increase organic traffic**

Monday, March 6, 2023

How I Used ChatGPT to Respond to my Emails in 5 Minutes



What is ChatGPT?

ChatGPT is an advanced AI chatbot created by the folks at https://openai.com that can accurately reproduce human responses without prior training on the subject. The GPT stands for generative pre-trained transformer, which means the model is already trained. This is different than a traditional chat bot, see my example here https://www.fiveminutecoder.com/2020/12/create-faq-bot-using-microsoft-bot.html, that needs prior knowledge on a subject to create accurate response to the subject. ChatGPT can also be tuned for your business similar to a traditional chat bot system by training the system with additional information. 

What makes ChatGPT so impressive is the confident responses made by the bot. You ask it a question and it will respond with an in depth answer. It also allows for follow up questions giving a feeling of a natural conversation with a human. Many people, including developers, are seeing the power of this and questioning if their job is in danger. While the tool is impressive, it does not replace the extensive knowledge gained by troubleshooting an issue for hours. Also, while the chat bot is confident in its answers this does not mean it is right. 

To test out ChatGPT I decided to make an email response app to respond to all the junk mail I get. This will be an Azure function that runs in 5 minute intervals. It will use the Graph API to check my email for new emails then send the subject/body of the email to the ChatGPT API. I found that the subject helps with a better response. Once I get a response I will reply to the email and set it to read. I wanted to see how the API and bot worked. So I asked it how to create an integration to the API while the system got me started it's response were either incomplete or outdated. 

Let's use ChatGPT to setup Chat GPT

Since Chat GPT is known for giving detailed responses to questions including code, lets just ask the chatbot how to setup ChatGPT in C#. 




Great! this looks like it will work. During setup however, this was wrong. It looks like the NuGet Package was updated to support Open API GPT-3 which changed the code. The updated calls can be found on the GitHub site here: https://github.com/OkGoDoIt/OpenAI-API-dotnet.

Before writing the app, I wanted to test out the API using PostMan. I wanted to get a feel that the chatbot could respond so again, I asked the Chat GPT chatbot for how to use Postman.



Again this looked promising. I setup Postman as the instructions showed and got an error. There was no model parameter passed in the JSON file. The model is quite important the model is what chat bot to use.... I guess that is an 0/2 using Chat GPT to code. 



 In my tests I used two different models "Davinci", which is the most sophisticated, but the slowest, and "Curie" which is a faster model. Out of the two, Davinci came across angrier in it's responses so I decided to use Curie for this example. Here are some of the responses I got from Postman using a junk email, basically they are replies with false information, hilarious!

"I look forward to hearing from you.Please share this with your team and I would be happy to provide details on our past projects.Regards,Shailesh Srinivasan"

" If you could send me your skype ID that would be great. Thank you."

 " I will discuss project portfolio, your team's strengths and skill gaps, the job description and requirements, and how you will benefit from working with us. All of our consultants are seasoned professionals who have worked for fortune 500 companies and top-tier consulting firms. We are typically able to leverage our existing resources to find the right talent for you.I look forward to working with you.Regards,TedFor a free consultation please contact me at ted@TECHstaffing.com. I am happy to help you with your project staffing needs. Please visit our website at www.TECHstaffing.com for more information. Ted KolodziejskyPhone: 1-972-200-1791Email: ted@TECHstaff"
 
 " I would love to chat about the following topics:1. What is the best way to build a strong engineering team?2. What are your hiring challenges?3. What is your IT roadmap for the next 3-5 years?4. What",


From this experience, I dont see Chat GPT taking my job anytime soon, but still to complete the exercise, the auto response Azure Function can be found below.

Creating an Email Auto Responder in 5 minute

To begin, an Azure function must be created, the details to create an Azure function can be found in a previous post here, https://www.fiveminutecoder.com/2021/05/create-email-tracking-campaign-using.html. Also, a Graph API application must be created. Again, details about how to do this can be found in a previous post here, https://www.fiveminutecoder.com/2021/03/creating-azure-document-queue-for.html. For the app permissions, application permissions are necessary. Under the Graph API section, find the mail section. The app will need read/write permissions and send as permissions.




Next the following Nuget packages must be installed.

 Azure.Identity, Microsoft.Graph, Microsoft.Graph.Core, OpenAI


At the top of the function I added my using statements for the installed nuget packages.


using System;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using System.Threading.Tasks;
using System.Collections.Generic;
using Microsoft.Graph;
using Microsoft.Graph.Models;
using Azure.Identity;

Next I defined my IDs necessary to access all the apps. This includes the Chat GPT API and Graph API.


//openAISecretKey
private string openAIKey = "";
//ID of the mailbox you want to auto reply from
private string userId = "user id of mailbox";
//ID of the tenant used
private string tenantId = "azure tenant"; 
//App id from created azure app
private string clientId = "registered app client id"; 
//Secret created for the app
private string clientSecret = "registered app secret"; 
//hold our graph context here for our calls
private GraphServiceClient graphService;

Inside the Run function, i setup the calls to get the unread emails then loop through the emails and respond to the email.


[FunctionName("CheckNewEmail")]
public async Task Run([TimerTrigger("0 */5 * * * *")]TimerInfo myTimer,  ILogger log)
{
	try
	{
		log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
		graphService = GetGraphAPIClient();
		List newMessages = await GetNewEmails();
		log.LogInformation("found " + newMessages.Count);
		foreach(Message message in newMessages)
		{
			log.LogInformation("replying to " + message.Subject);
			string response = await GetChatGPTResponse(message.Subject, message.Body.Content);
			await SendEmail(message.Id, message.From, response);
			await UpdateToRead(message.Id);
			log.LogInformation("Reply successful");

		}
	}
	catch(Exception ex)
	{
		log.LogError(ex, ex.Message);
	}
}  


To instantiate the graph service I used the new Azure.Identity to create an authentication scope and then return the created service to be used throughout the application.


//Create the graph service client  that will be used to get and respond to emails
private GraphServiceClient GetGraphAPIClient()
{
	
	string[] scopes = new string[] {"https://graph.microsoft.com/.default" };
	// using Azure.Identity;
	var options = new TokenCredentialOptions
	{
		AuthorityHost = AzureAuthorityHosts.AzurePublicCloud
	};

	ClientSecretCredential clientSecretCredential = new ClientSecretCredential(
		tenantId, clientId, clientSecret, options);

	GraphServiceClient graphClient = new GraphServiceClient(clientSecretCredential, scopes);
	return graphClient;
}


Using the newly created client, a call is made to the Graph API to get all the unread emails using the isRead filter.


//Graph API call to get all unread emails
private async Task> GetNewEmails()
{
	MessageCollectionResponse messages = await graphService.Users[userId].Messages.GetAsync((requestConfiguration) =>{
		requestConfiguration.QueryParameters.Filter = "isRead eq false";
	});
	
	return messages.Value;
}

Once all the emails are fetched, the subject and body are combined into one string and then sent to the Chat GPT API.


//The call to the Chat GPT end point
private async Task GetChatGPTResponse(string Subject, string Body)
{
	OpenAI_API.OpenAIAPI openai = new OpenAI_API.OpenAIAPI(openAIKey);

	//Create a request suitable for the Chat GPT API. It will remove an non readable characters that the API cannot read
	OpenAI_API.Completions.CompletionRequest completionRequest = new OpenAI_API.Completions.CompletionRequest(Subject + "." + Body, OpenAI_API.Models.Model.CurieText,150);

	// Send a request to the ChatGPT model
	OpenAI_API.Completions.CompletionResult response = await openai.Completions.CreateCompletionAsync(completionRequest);

	return response.Completions[0].Text;
}


With an AI generated response, I send an email using the Graph API to the original sender.


//Graph API call to send reply to email
private async Task SendEmail(string MessageId, Recipient RecipientEmail, string Response)
{
	Microsoft.Graph.Users.Item.Messages.Item.Reply.ReplyPostRequestBody reply = new Microsoft.Graph.Users.Item.Messages.Item.Reply.ReplyPostRequestBody
	{
		Message = new Message
		{
			ToRecipients = new List
			{
				new Recipient()
				{
					EmailAddress = new EmailAddress()
					{
						Address = RecipientEmail.EmailAddress.Address,
						Name = !String.IsNullOrEmpty(RecipientEmail.EmailAddress.Name) ? RecipientEmail.EmailAddress.Name : RecipientEmail.EmailAddress.Address
					}
				}
			},
		},
		Comment = Response,
		
	};

	await graphService.Users[userId].Messages[MessageId].Reply.PostAsync(reply);
}


Finally, I set the email to read so it is not picked up by the next call.


//Graph API call to update email to read
private async Task UpdateToRead(string MessageId)
{
	
	//only update the properties we want to update
	Message msg = new Message()
	{
		IsRead = true
	};
	await graphService.Users[userId].Messages[MessageId].PatchAsync(msg);
}


 
That's it! A function for responding to emails has been created and let the spammers be enthralled by the witty comebacks of the AI. To view the code, please visit my GitHub page here: https://github.com/fiveminutecoder/blogs/tree/master/ChatGPTEmail

UPDATE!!!!

With the general release of ChatGPT 3.5 the responses have changed significantly. We can give the bot a persona to respond to the emails which greatly changes the usefulness of the application. While I miss the snarky response of Davinci using ChatGPT 3.5 is the way to go.

To test this in Postman, all that needs to be done is update the body to include messages instead of prompt. You will see the messages section is an array. This is to help with persistence in responses. Also notice system and user role. System role allows me to tell the chat bot how to act, while the user role is the content to respond to.


{
    "messages":[
        {"role": "system", "content": "You are the assistant to the Director of IT. He does not want any meetings"},
        {"role": "user", "content": "email body here!!"}
    ],
    "temperature": 0.7,
    "max_tokens": 3250,
    "top_p": 1,
    "frequency_penalty": 0,
    "presence_penalty": 0,
    "model": "gpt-3.5-turbo-0301"
}


For the C# application instead of the completion endpoint, the ChatCompletion endpoint will be used, this is a quick change to handle the new message array.




var result = await api.Chat.CreateChatCompletionAsync(new ChatRequest()
{
	Model = Model.ChatGPTTurbo,
	Temperature = 0.7,
	MaxTokens = 50,
	Messages = new ChatMessage[] {
	new ChatMessage(ChatMessageRole.System, "You are the assistant to the Director of IT. He does not want any meetings")
		new ChatMessage(ChatMessageRole.User, "email body here!!")
	}
});

Tuesday, December 27, 2022

Auto Tagging Invoices Using Azure AI Cognitive Services in 5 Minutes


 


In a previous blog post we covered SharePoint Syntex for auto tagging invoices by using a content type, which can be found here SharePoint Syntex in 5 Minutes. Sometimes there needs to be more processing outside of SharePoint before the document can be uploaded or external systems must be accessed for metadata properties. This kind of functionality can become very complex when trying to use a Power App or Flow to accomplish this. Microsoft provides AI services for reading invoices that can be read and then used for the business logic that goes beyond what Syntex can do. These services are a consumption based API in Azure that allows uploading invoices for processing to return the same metadata results that can be found in SharePoint Syntex.


Setting up Azure

1) Create a Cognitive Services Plan






2) Once the cognitive services is created, there is a list of several services including form services. Selecting this will open up the form studio which allows for uploading and reviewing the forms the service will be used for training.




3) Since this is a 5 minute tutorial, I will be using the prebuilt invoice recognizer.


4) Since this is a prebuilt model it comes with several examples already loaded. By clicking the "Analyze" button, the invoice will highlight all the points of interest and assign it metadata. This screen is verry similar to the SharePoint Syntex screen seen in my previous blog SharePoint Syntex in 5 Minutes


5) To make sure this predefined model works for your invoices, select the upload in the top left corner and then upload a sample invoice.




6) Finally, a storage account for the invoice service to access documents must be created. Our invoice service must be able to access the files they must be made available. For this demo, I will be making my blob storage available to the internet. For security reasons DO NOT DO THIS IN PRODUCTION. For a production environment you will want to setup a network for your AI service that is connected to your blob storage for secure access. For details on how to create a storage account, see my pervious blog post Create an Azure Document Queue for Loading and Tagging SharePoint Documents - Part 1


Consuming the service

For this example, I created a WPF app to display our uploaded invoice and its associated properties. To begin, 4 NuGet packages must be installed.

These 2 are needed for the form recognition service, the form recognizer API reading the invoice and the Azure storage blob API for exposing the invoice.


Azure.AI.FormRecognizer
Azure.Storage.Blobs

The other 2 NuGet packages needed are for drawing our invoice. PdfLibCore will be used to convert the PDF into an Image and System.Drawing.Common will be used for drawing the image. It is important to note that this example was done on Windows. System.Drawing may not be Linux/Mac compatible.


System.Drawing.Common
PdfLibCore

The app layout is a simple grid system made up of 3 rows. One for uploading an invoice, the other for displaying it's properties, and then the bottom row for any errors while uploading.


<window height="1000" mc:ignorable="d" title="Five Minute Invoice Tagger" width="1600" x:class="FiveMintueInvoiceTagger.MainWindow" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:local="clr-namespace:FiveMintueInvoiceTagger" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation">
    <grid>
        <grid.columndefinitions="">
            <columndefinition width="500"></columndefinition>
            <columndefinition width="1100"></columndefinition>
        </grid>
        <grid.rowdefinitions="">
            <rowdefinition height="50"gt;</rowdefinition>
            <rowdefinition height="750"gt;</rowdefinition>
            <rowdefinition height="750"gt;</rowdefinition>
        </grid>
        <stackpanel grid.column="0" grid.row="0">
        <label content="Select an invoice...">
        <button click="UploadFile_Click" content="Select Invoice">
        </button></label></stackpanel>
        <image grid.column="0" grid.row="1" height="800" name="InvoiceImage" width="450">
    <datagrid grid.column="1" grid.row="1" height="800" name="DocumentProperties" width="1050">
    <label grid.column="0" grid.row="2" name="ErrorMsg">
    </label></datagrid></image></grid> 
</window>

Next, an object is needed to hold our invoice properties for displaying the results. This class has 3 items, the Field's name, the Field's Value, and the confidence score that the API grabbed the right information.


public class InvoiceProperty
{
	//Field name found on invoice
	public string Field {get;set;}
	//Field value
	public string Value {get;set;}
	//How confident AI is that field value is correct
	public string Score {get;set;}
}

References to the NuGet packages must be added to the project, along with some other using statements for displaying the invoice image.


using System;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Media.Imaging;
using Azure;
using Azure.AI.FormRecognizer.DocumentAnalysis;
using Azure.Storage.Blobs;
using PdfLibCore;
using PdfLibCore.Enums;

A click event is added to the upload button to grab the invoice and process the request. This method is async so a loading screen should be added. Since this a 5 minute application it has been omitted.


private async void UploadFile_Click(object sender, RoutedEventArgs e)  
{  
	try
	{
		ErrorMsg.Content = "";

		//we only want pdf invoices
		Microsoft.Win32.OpenFileDialog openFileDlg = new Microsoft.Win32.OpenFileDialog(); 
		openFileDlg.Filter = "Pdf Files|*.pdf";
		// Launch OpenFileDialog by calling ShowDialog method
		Nullable result = openFileDlg.ShowDialog();
		// Get the selected file name and display in a TextBox.
		// Load content of file in a TextBlock
		if (result == true)
		{
			//Upload to azure blob so Azure AI can access file
			string invoicePath = await UploadInvoiceForProcessing(openFileDlg.FileName);

			//perform Invoice tagging
			Task> invoicePropertiesTask = GetDocumentProperties(invoicePath);

			//Convert PDF to image so we can view it next to properties
			UpdateInvoiceImage(openFileDlg.FileName);

			//Wait for Azure to return results, set it to our data grid
			DocumentProperties.ItemsSource = await invoicePropertiesTask;

		}
	}
	catch(Exception ex)
	{
		ErrorMsg.Content = ex.Message;
	}
}

In our button event, there are 3 functions called One for uploading the invoice to Azure, one for processing the invoice, and one for converting the image. Our upload function will upload the invoice to Azure Blob Storage to make the invoice available to the Azure Form Recognizer Service. Again, in a production environment make sure your blob storage is not publicly available. 


private async Task UploadInvoiceForProcessing(string FilePath)
{
	string cs = "";
	string fileName = System.IO.Path.GetFileName(FilePath);
	Console.WriteLine("File name {0}", fileName);
	//customer is the name of our blob container where we can view documents in Azure
	//blobs require us to create a connection each time we want to upload a file
	BlobClient blob  = new BlobClient(cs, "invoice", fileName); 

	//Gets a file stream to upload to Azure
	using(FileStream stream = File.Open(FilePath, FileMode.Open))
	{
		var blobInfo = await blob.UploadAsync(stream);
		
	}
	
	return "blob base storage url" + fileName;
}

Next the invoice URL is passed to the Form Recognizer Service for processing


private async Task> GetDocumentProperties(string InvoicePath)
{
	
	List invoiceProperties = new List();

	//Endpoint and key found in Azure AI service
	string endpoint = "ai service url";
	string key = "ai service key";
	AzureKeyCredential credential = new AzureKeyCredential(key);
	DocumentAnalysisClient client = new DocumentAnalysisClient(new Uri(endpoint), credential);

	//create Uri for the invoice
	Uri invoiceUri = new Uri(InvoicePath);

	//Analyzes the invoice
	AnalyzeDocumentOperation operation = await client.AnalyzeDocumentFromUriAsync(WaitUntil.Completed, "prebuilt-invoice", invoiceUri);
	AnalyzeResult result = operation.Value;

	//iterate the results and populates list of field values
	for (int i = 0; i < result.Documents.Count; i++)
	{
		AnalyzedDocument document = result.Documents[i];
		foreach(string field in document.Fields.Keys)
		{
			DocumentField documentField = document.Fields[field];
			InvoiceProperty invoiceProperty = new InvoiceProperty()
				{
				  Field = field,
				  Value = documentField.Content,
				  Score = documentField.Confidence?.ToString()
				};

				invoiceProperties.Add(invoiceProperty);
			}
	}

	return invoiceProperties;
}


While the invoice is being processed, the application will convert the PDF to an image to be displayed in the application. The form recognizer service returns references for the PDF to draw the bounding boxes of the data found which could be used to draw onto the image.


 private void UpdateInvoiceImage(string FilePath)
{
	using(var pdf = new PdfDocument(File.Open(FilePath, FileMode.Open)))
	{
		//for this example we only want to show the first page
		if(pdf.Pages.Count > 0)
		{
			var pdfPage = pdf.Pages[0];

			var dpiX= 600D;
			var dpiY = 600D;
			var pageWidth = (int) (dpiX * pdfPage.Size.Width / 72);
			var pageHeight = (int) (dpiY * pdfPage.Size.Height / 72);
		
			var bitmap = new PdfiumBitmap(pageWidth, pageHeight, true);                                

			pdfPage.Render(bitmap, PageOrientations.Normal, RenderingFlags.LcdText);
			BitmapImage image = new BitmapImage();
			image.BeginInit();
			image.StreamSource = bitmap.AsBmpStream(dpiX,dpiY);
			image.EndInit();
			InvoiceImage.Source = image;
		}
		
	}
}


Once this is completed your application will display the invoice with the properties found with an application created in 5 minutes.




To view the full code, please visit the Five Minute Coder GitHub here: Five Minute Invoice Tagger


Tuesday, March 9, 2021

Create a FAQ Bot using Microsoft Bot Framework

What is the Microsoft Bot framework?

The Microsoft Bot Framework is a set of APIs that simplifies the process of creating a chat bot using C#. The bot framework us set up a web service or Azure function that allows for interacting with a user via chat. The bot framework has several features for integrating chat bots with Microsoft Teams, Skype, and other Microsoft products. While the framework is geared toward Microsoft integration with Teams, it can be used as a stand alone bot system that can control: user flow, group chats, etc. 

Bots use JSON to write back and forth to the web service, but interpreters like Teams will recognize certain JSON objects, such as cards, and display them in a unique way without any styling. The framework also has a feature that allows for conversation flow called dialogs. This will manage a user's conversation flow and allow for more complex interactions like booking a hotel room, or implementing a pizza ordering system. 


Create a Bot in 5 Minutes

This example is going to build off of a previous example where we developed a NLP faq application. If you have not read the NLP example, please do so before beginning since we will use a lot of the codebase from this example. The NLP example can be found here: https://fiveminutecoder.blogspot.com/2020/08/using-mlnet-to-create-natural-language.html.  Before creating our chatbot, we need to get the bot templates. There are several starting templates to choose from, for this example we will use the echo bot. This bot template just writes back to the chat what you type in which is perfect for our demo since we do not need any dialog flow. To install the template we just need to run the following commands.


dotnet new -i Microsoft.Bot.Framework.CSharp.EchoBot
dotnet new echobot -n BotFrameworkFAQBot


Once we have our framework setup, we need to install the Nuget package Microsoft.ML. This will allow us to use ML.NET to process our questions and post an answer.

From the NLP example we will want to bring over several items. The first being our trained machine learning model which we saved earlier called FAQModel.zip. We will also need our Prediction data model and our FAQ data model. 


using Microsoft.ML.Data;

namespace EchoBot.Bots
{
    public class FAQModel
    {
        [ColumnName("Question"), LoadColumn(0)]
        public string Question {get;set;}
        [ColumnName("Answer"), LoadColumn(1)]
        public string Answer {get;set;}
    }
}


using Microsoft.ML.Data;

namespace EchoBot.Bots
{
    public class PredictionModel
    {
        [ColumnName("PredictedAnswer")]
        public string PredictedAnswer {get;set;}
        [ColumnName("Score")]
        public float[] Score {get;set;}
    }
}

With our data moved from the previous project, we can go ahead and rename the EchoBot.cs to FAQBot.cs. This will break any dependency injection that is setup by our service, so we will need to go to the startup.cs and change the services.AddTransient<IBot, Bots.EchoBot>() to services.AddTransient<IBot, Bots.FAQBot>() 


// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
	services.AddControllers().AddNewtonsoftJson();

	// Create the Bot Framework Adapter with error handling enabled.
	services.AddSingleton();

	// Create the bot as a transient. In this case the ASP Controller is expecting an IBot.
	services.AddTransient();
}


Our machine learning model is pretrained, so since this information is static we will create a singleton instance of our prediction engine . This way we do not have to read the file from filestream every time we want to predict an answer. 


//context or our machine learning model
static MLContext context;
//used to read and predict our questions
static PredictionEngine predictionEngine;

//creating a private staitc constructor
//Our model is not changing so it doesnt make sense to keep opening it and reading the zip for performance
static FAQBot()
{
	//structure of our data model
	DataViewSchema modelSchema;
	//the model loaded for prediction
	ITransformer trainedModel;
	context = new MLContext();

	//load our file
	using(Stream s = File.Open("FAQModel.zip", FileMode.Open))
	{
		trainedModel = context.Model.Load(s, out modelSchema);
	}

	//creates our prediction engine
	predictionEngine = context.Model.CreatePredictionEngine(trainedModel);

}

Now that our constructor on class are in place and loaded, all we need to do is call the predict function for our question and display the outputs. We want to ensure that we have some confidence in our predictions so we will check the score and only display the output if the prediction score over 60% confident. If not we will give the user a list of choices to choose from. 


protected override async Task OnMessageActivityAsync(ITurnContext turnContext, CancellationToken cancellationToken)
{
	//creates our FAQ model
	FAQModel question = new FAQModel()
	{
		Question = turnContext.Activity.Text,//gets text from bot
		Answer = ""
	};

	//uses our trained model to predict our answer
	PredictionModel prediction = FAQBot.predictionEngine.Predict(question);

	//accuracy of prediction
	float score = prediction.Score.Max() * 100;

	//gonna check if we were accurate,if below a threshold we will ask them to clarify
	if(score > 60)
	{
		//sends our answer back to the bot
		await turnContext.SendActivityAsync(MessageFactory.Text(prediction.PredictedAnswer), cancellationToken);
		await turnContext.SendActivityAsync(MessageFactory.Text($"We think our answer to your question is this accurate: {score}%"), cancellationToken);
	}
	else
	{
		//sends them suggestions that are clickable
		await turnContext.SendActivityAsync(MessageFactory.Text("Sorry, we didnt understand the question, please try selecting a question below"), cancellationToken);
		string[] actions = {"What are your hours?","How can I reach you?", "What payments do you accept?"};
		await turnContext.SendActivityAsync(MessageFactory.SuggestedActions(actions), cancellationToken);
	}
}

Our chatbot is now finished. SendActivityAsync allows us to send a message back to the user. this can be called at any time during the process and is helpful for long running processes. Microsoft also provides us a set of activities, found in the MessageFactory, for interacting with users. We can easily send images, cards, or attachments using the different types found in the factory.


Testing the Chatbot

With our bot endpoint setup, we now need to test it out. Microsoft provides a tool for emulating a bot system which you can find here: https://github.com/microsoft/BotFramework-Emulator/releases. The emulator will emulate a Teams chat so you can see how the responses will interact with Teams and other Microsoft products. Download the latest version, and run the application. Once the emulator is running, open your site by the URL http://localhost:{port}/api/messages. You should see a successful connection and the message "Hello and welcome!" This comes from our bot's "OnMemberAddesAsync" function found in the FAQBot.cs file. The final step is to ask the bot a question and test out the functionality.



Clone the project

You can find the complete project here: https://github.com/fiveminutecoder/blogs/tree/master/FAQBot
Microsoft, Bot, Chatbot, Bot Framework, Teams, Microsoft Teams, ML.NET, Machine Learning, AI, Artificial Intelligence, C#,C Sharp, NLP, Natural Language Programming, Robot

Tuesday, December 8, 2020

Predicting Bitcoin Prices Using ML.Net and Time Series Techniques

Obligatory Machine Learning Stock Predictor 

It seems that anyone who starts to learn machine learning and analyzing big data thinks they can predict patterns in something as volatile as the stock market. Once you go down the rabbit hole it is easy to see that things are not random, we just need enough data so it is easy to see the attraction to trying to predicting stock prices.

Unfortunately, there are many external factors that data cannot predict. This was very clear when the 2020 pandemic hit, and stock prices tanked. Even so, trying is fun and helps us understand concepts. 

I am not a stock guru, and everything is this post is purely for learning purposes of ML.Net and how to use it for the time series function. Please use the code at your own risk for anything beyond learning how to create a time series model.

What is a Time Series Model?

Time Series in machine learning is trying to predict out over several periods. This is different from Regression which predicts the next period from a series. Some of the more common examples are housing prices, gas prices, and sales predictions. In big data analytics we can use linear regression to plot thousands of points and find the average over those points to create a line representing our answer. This is a good route, if you have two values, for example price/date you can create a nice graph that can represent sales.


Data prep

Before getting into the coding, data prep is key here. We are predicting items over time, so you want to know a time frame. Sales data for example, you usually compare year over year which is last year's sales vs this year's sales. We would use our time series model to predict the next 3 years sales or So you would break your data into yearly chunks usually daily prices. 


Create NLP FAQ Application in 5 Minutes

Data


With our Bitcoin example we will be breaking our data into daily changes in price with 1 minute increments. We will be using the btc.csv you can find here on my github page https://github.com/fiveminutecoder/blogs/blob/master/%20mlnet_BTCTimeSeries/btc.csv. *Update, it was found that this file was not consistent in it's time stamps so the code has been updated to use the following data set from Kaggle Bitcoin Historical Data. It is too large to add to the Git Hub site, this dataset was loaded by Zielak.

Our dataset has multiple columns, but when forecasting we are tracking a value over time, so only price will be used.

Creating the project

If you have not read my previous blog "Getting Started with ML.NET", which can be found here https://fiveminutecoder.blogspot.com/2020/07/getting-started-with-mlnet.html, please do so before continuing.

For this project we will continue to use the Microsoft.ML nuget package, but we will also need to add the Microsoft.ML.TimeSeries nuget package to get the forecasting estimator.

Once you have created a project and installed the Nuget packages, we can go ahead and create our data models to be used for our training data and our predictions. 

The first model is the BTCDataModel. It contains our pricing and timestamps we will use for training.


using Microsoft.ML.Data;

namespace mlnet_BTCTimeSeries
{
    public class BTCDataModel
    {
        [LoadColumn(0)]
        public int TimeStamp {get;set;}

        [LoadColumn(1)]
        public float Open {get;set;}
        [LoadColumn(2)]
        public float High {get;set;}
        [LoadColumn(3)]
        public float Low {get;set;}
        [LoadColumn(4)]
        public float Close {get;set;}

        [LoadColumn(5)]
        public float Volume {get;set;}
        [LoadColumn(6)]
        public float Currency {get;set;}

        [LoadColumn(7)]
        public float Amount {get;set;}
    }
}

Once we have created our training model, let's create our prediction model. The time series prediction model is slightly different than our supervised learning model. Instead of an array of confidence scores, we will have our prediction with an upper and lower bounds for accuracy.


using Microsoft.ML.Data;

namespace mlnet_BTCTimeSeries
{
    public class PredictedSeriesDataModel
    {
        public float[] ForecastedPrice { get; set; }
        public float[] ConfidenceLowerBound { get; set; }
        public float[] ConfidenceUpperBound { get; set; }
    }
}

Now that our models are setup, we can go ahead and add our context and training dataset place holders. We will set these globally so we can easily access them in our functions


using System;
using System.Linq;
using System.IO;
using Microsoft.ML;
using Microsoft.ML.Transforms.TimeSeries;
using System.Collections.Generic;

...

static MLContext mlContext = new MLContext();
//last time to show the times of our future predictions
static DateTime lastTime; 
static List trainingData;
static List testingData;
static string fileName = "btcModel.zip";
//how far out we want to predict
static int horizon = 5; 
//holds or in memory model
static ITransformer forecastTransformer; 

The first step or our project is to create our training and testing data, our testing data will be the size of our horizon field since that is the number we are trying to forecast, everything else will be used for training. We will also capture the last timestamp from the training model, this is used to show the next 5 predictions match the time from our test data.


static void GetTrainingData()
{
	//load our dataset
	IDataView trainingDataFile = mlContext.Data.LoadFromTextFile("bitstampUSD_1-min_data_2012-01-01_to_2020-12-31.csv", hasHeader: true, separatorChar: ',');

	//create enumerable to manipulate data
	List data = mlContext.Data.CreateEnumerable(trainingDataFile, false, true).ToList();

	//times in the data set are not uniform, so we will pull unique time values
	data = data.OrderBy( o => o.TimeStamp).ToList();

	//determines the size of our testing data
	int dataSubset = data.Count() - horizon;

	//create our training data up to the dates we are trying to predict
	trainingData = data.GetRange(0, dataSubset).ToList(); 

	// will get the number of items we are trying to predict
	testingData = data.GetRange(dataSubset, horizon);
	
	//We want to capture time of last item in training data so we can increment the time stamp for our output and put a date/time to the forecast
	lastTime = ConvertTimeStamp(trainingData.Last().TimeStamp);
}

//helper for converting timestamp to date time
static DateTime ConvertTimeStamp(double TimeStamp)
{
	var offset = TimeSpan.FromSeconds(TimeStamp);
	DateTime startTime = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
	return startTime.Add(offset).ToLocalTime();
}

With our data sets created, we can move on to creating our estimator. Unlike the supervised learning estimator, there is very little data manipulation. With the time series we just need to input the property names from our models to the appropriate columns. For this we will "nameof" to get our property names instead of labeling each property. There are a lot of settings in the forecasting estimator, the 3 main settings to worry about are "windowSize", "seriesLength", and "horizon". Window size is the periods on which our data is to reflect, series is the timespan of those windows, and finally horizon is the forecasting we want to produce or how far out we want to predict. 


static void TrainModel()
{
	IDataView trainingDataView = mlContext.Data.LoadFromEnumerable(trainingData);

	// creates our estimater, as you cans see we are using forecasting estimator
	var estimator = mlContext.Forecasting.ForecastBySsa(outputColumnName: nameof(PredictedSeriesDataModel.ForecastedPrice),
					inputColumnName: nameof(BTCDataModel.Amount), //column used for time series prediction
					windowSize: 60, //series is sampled in 60 minute windows or periods, and the past 60 minutes will be used to make the prediction
					seriesLength: 1440, //we want to train over a day's worth of time so this will be the interval, we have 1440 minutes in a day
					trainSize: trainingData.Count(), //how many data points we want to sample
					horizon: horizon,
					confidenceLevel: 0.45f, //sets our margin of error, lower the confidence level the smaller the upper/lower bounds
					confidenceLowerBoundColumn: nameof(PredictedSeriesDataModel.ConfidenceLowerBound),
					confidenceUpperBoundColumn: nameof(PredictedSeriesDataModel.ConfidenceUpperBound)
	);

	//creates our fitted model
	forecastTransformer = estimator.Fit(trainingDataView); 
}

One thing to notice in our trainer is that we are not calling save yet, this is will happen later in our predict function since time series data needs updating to stay relevant, we save a little differently. Our data is now fitted to our estimator, we can now predict our future Bitcoin prices.


static void Predict()
{
	//prediction engine based on our fitted model
	 TimeSeriesPredictionEngine forecastEngine = forecastTransformer.CreateTimeSeriesEngine(mlContext);

	//call to predict the next 5 minutes
	 PredictedSeriesDataModel predictions = forecastEngine.Predict();

	 //write our predictions
	 for(int i = 0; i < predictions.ForecastedPrice.Count(); i++)
	{   
		lastTime = lastTime.AddMinutes(1);
		Console.WriteLine("{0} price: {1}, low: {2}, high: {3}, actual: {4}", lastTime, predictions.ForecastedPrice[i].ToString(), predictions.ConfidenceLowerBound[i].ToString(), predictions.ConfidenceUpperBound[i].ToString(), testingData[i].Amount);
	}


	//instead of saving, we use checkpoint. This allows us to continue training with updated data and not need to keep such a large data set
	//so we can append Jan. 2021 without having everythign before to train the model speeding up the process
	forecastEngine.CheckPoint(mlContext, fileName);
}

The predict function is very similar to the rest of our predict functions, we visualize our data a bit different since we want to see all our our values not just the most accurate. At the end of this function we called "CheckPoint" on our forecastEngine. This creates a saved model that we can continue to train and add data points without retraining the entire model.


With our training and prediction functions complete, all that is left is calling them and viewing our results.


static void Main(string[] args)
{
	GetTrainingData();
	TrainModel();
	Predict();
}

Results

As we can see, the forecaster is fairly accurate predicting the next minute's price (off by about $5) but unfortunately our model predicted a downward tend instead of an upward one as we see comparing to our actual results. This just shows price history alone is not enough to predict the stock market.



Clone the project


You can find the full project on my GitHub site here https://github.com/fiveminutecoder/blogs/tree/master/%20mlnet_BTCTimeSeries
AI, Artificial Intelligence, BTC, BitCoin, C#, Time Series, dotnet, dot net, machine learning, mldotnet, ml.net, dotnet core, dotnet 5, .NET 5

Tuesday, November 10, 2020

Using ML.NET for Natural Language Processing (NLP) in 5 minutes

 What is Natural Language Processing?

Natural language processing, or NLP, is taking text and and converting it to something your application can use. What we are expecting is for someone to type in a word or sentence and the application is able to understand and process the command. The challenge here is not everyone communicates the same way for example:

  • Please save document.
  • Save document.
  • Update Document.
  • I need my document to be put into my accounting folder.

All can be interpreted as a "Save" command. This is a great task for machine learning. We can train our algorithm to interpret what the user is trying to communicate, and complete the task. If you are not familiar with the basics of ML.NET or supervised learning, please check out my previous post https://fiveminutecoder.blogspot.com/2020/07/getting-started-with-mlnet.html.

Create NLP FAQ application in 5 minutes

The Data

I have created a basic csv file with several FAQ questions and answers for a fictional business, you can download the file here which is part of the GitHub repository https://github.com/fiveminutecoder/blogs/tree/master/mlnet_NLP.

Creating the project

If you have not read my previous blog "Getting Started with ML.NET", which can be found here https://fiveminutecoder.blogspot.com/2020/07/getting-started-with-mlnet.html please do so before continuing since we will be referencing back to it frequently. Following our previous example, we will create a new console application called "mlnet_NLP". Once the project is created we will download the "Microsoft.ML" package from Nuget.

Once you have the project setup we will need to create our two data models, one for the input features of the FAQ (our question and answer), and one for our predictions.


	public class FAQModel
	{
		[ColumnName("Question"), LoadColumn(0)]
		public string Question {get;set;}
		[ColumnName("Answer"), LoadColumn(1)]
		public string Answer {get;set;}
	}



	public class PredictionModel
	{
		[ColumnName("PredictedAnswer")]
		public string PredictedAnswer {get;set;}
		[ColumnName("Score")]
		public float[] Score {get;set;}
	}


Once we have our data models setup, we need to setup our program. We will use the same 5 functions from our pervious example which will train, test, predict, save, and load our machine learning model. Before we begin, we need to add our references and fields that will hold our context and data. Again, this should be a separate class, but for the sake of time we will do all this in our program.cs.


	//Main context
	static MLContext context;
	//model for training/testing
	static Microsoft.ML.Data.TransformerChain model;
	static IEnumerable trainingData;
	static IEnumerable testingData;
	static string fileName = "FAQModel.zip";

Our training function will be very similar to the previous one, with the exception to how to create our features. Instead of our features being several columns, we have 1 column with several words. Luckily ML.Net has a function that lets us featurize text making it a quick swap of our previous concatenate features line. The FeaturizeText method is very powerful and performs several operations under the hood, like remove stop words like the, and, or, etc. To learn more, visit Microsoft's documentation around preparing data https://docs.microsoft.com/en-us/dotnet/machine-learning/how-to-guides/prepare-data-ml-net


	static void TrainModel()
	{
		context = new MLContext();

		//Load data from csv file
		var data = context.Data.LoadFromTextFile("faq.csv", hasHeader:true, separatorChar: ',', allowQuoting: true, allowSparse:true, trimWhitespace: true);
		

		//create data sets for trainiing and testing
		trainingData = context.Data.CreateEnumerable(data, reuseRowObject: false);
		testingData = new List()
		{
			new FAQModel() {Question = "When are you open?", Answer = "Our hours are 9 am to 5pm Monday through Friday"},
			new FAQModel() {Question = "Can i pay using a visa card?", Answer =  "Our payment options are Credit, Check, or Bitcoin"},
			new FAQModel() {Question = "How can i contact you.", Answer = "Our phone number is 555-5555 and our fax is 555-5557"}
		};

		//Create our pipeline and set our training model
		var pipeline = context.Transforms.Conversion.MapValueToKey(outputColumnName: "Label", inputColumnName: "Answer") //converts string to key value for training
			.Append(context.Transforms.Text.FeaturizeText( "Features","Question")) //creates features from our text string
			.Append(context.Transforms.Text.f)
			.Append(context.MulticlassClassification.Trainers.SdcaMaximumEntropy(labelColumnName: "Label", featureColumnName: "Features"))//set up our model
			.Append(context.Transforms.Conversion.MapKeyToValue(outputColumnName: "PredictedAnswer", inputColumnName: "PredictedLabel")); //convert our key back to a label

		//traings the model
		 model = pipeline.Fit(context.Data.LoadFromEnumerable(trainingData));
	}

Now that our training method is setup, we want to test our model. This FAQ is too small to break up, so the accuracy will return as 0. To remedy this, I  manually added a couple tests to our enumerable.


	static void TestModel()
	{
		//transform data to a view that can be evaluated
		IDataView testDataPredictions = model.Transform(context.Data.LoadFromEnumerable(testingData));
		//evaluate test data against trained model for accuracy
		var metrics = context.MulticlassClassification.Evaluate(testDataPredictions);
		double accuracy = metrics.MacroAccuracy;

		Console.WriteLine("Accuracy {0}", accuracy.ToString());
	}

Now that our model is trained, we will save it so it can be loaded in our prediction engine. 


	static void SaveModel()
	{
		IDataView dataView = context.Data.LoadFromEnumerable(trainingData);
	       context.Model.Save(model, dataView.Schema, fileName);
	}



	static ITransformer LoadModel()
	{
		DataViewSchema modelSchema;
		//gets a file from a stream, and loads it
		using(Stream s = File.Open(fileName, FileMode.Open))
		{
			return context.Model.Load(s, out modelSchema);
		}
	}

Now we can setup our prediction engine. Again this is exactly how we set it up in the previous example. Our NLP uses a multiclass supervised learning model so predicting our answer is handled the same; pass our question in, and the machine learning algorithm will spit out an answer.


	static void Predict(FAQModel Question)
	{
		ITransformer trainedModel = LoadModel();

		//Creates prediction function from loaded model, you can load in memory model as wwell
		 var predictFunction = context.Model.CreatePredictionEngine(trainedModel);
		 
		//pass model to function to get prediction outputs
		PredictionModel prediction = predictFunction.Predict(Question);

		//get score, score is an array and the max score will align to key.
		float score = prediction.Score.Max();
	
		Console.WriteLine("Prediction: {0},  accuracy: {1}", prediction.PredictedAnswer, score);

	}

Now that we have our functions setup, we can call them in the static main function and start answering questions.


	static void Main(string[] args)
	{
		TrainModel();
		TestModel();
		SaveModel();
		FAQModel question = new FAQModel(){
			Question = "can i Pay online?",
			Answer = ""
		};

		Predict(question);
	}

Clone the project


you can find the full project on my GitHub site here https://github.com/fiveminutecoder/blogs/tree/master/mlnet_NLP
AI, Artificial Intelligence, C#, NLP, Natural Language Processing, supervised learning, dotnet, dot net, machine learning, mldotnet, ml.net, dotnet core, dotnet 5, .NET 5

Tuesday, October 6, 2020

Get Started with ML.NET in 5 Minutes



What is ML.NET

ML.NET is a dot net based machine learning language created by Microsoft. It allows us to use C# to quickly create various machine learning algorithms using built in training methods. ML.NET also has a way to extend to the language to tap into other machine learning platforms such as TensorFlow for actions that are not yet supported by ML.NET.

What is supervised learning?


Supervised learning is when we train a model with known labels for our data. The learning is supervised because we are able to give the training algorithm the correct answer for what the data represents. When training a real model, you will want a large data set representing different scenarios for your model.

Create a Supervised Learning Model in about 5 minutes.


The Data Set


For this example, we will be using the Iris Flower Species data set which can be found on the Kaggle website here https://www.kaggle.com/uciml/iris.

Create the Project


Once you have downloaded the data set, we need to create the project. Since ML.NET is so new it is worth noting that this article was written using version 1.51 and dot net core 3.1. As machine learning evolves some of these techniques may change.

To start, create a new dot net core console application called "mlnet_intro"

        dotnet new console –-name “mlnet_intro”

Now that we have our new project make sure you have the folder open, and add the nuget package "Microsoft.ML". If you are using VSCode, use CTRL+SHIFT+P to search for the package.

Data Models


We now have all the necessary components to start creating our supervised learning application. We will need 2 data models for our model one representing the Iris being fed into the model, one for displaying results. We will create our Iris model aptly named "IrisModel". 


	using Microsoft.ML.Data;

        namespace mlnet_intro
        {
            public class IrisModel
            {
                [ColumnName("Id"), LoadColumn(0)]
                public int Id {get;set;}
                [ColumnName("SepalLengthCm"), LoadColumn(1)]
                public float SepalLengthCm {get;set;}
                [ColumnName("SepalWidthCm"), LoadColumn(2)]
                public float SepalWidthCm {get;set;}
                [ColumnName("PetalLengthCm"), LoadColumn(3)]
                public float PetalLengthCm {get;set;}
                [ColumnName("PetalWidthCm"), LoadColumn(4)]
                public float PetalWidthCm {get;set;}
                [ColumnName("Species"), LoadColumn(5)]
                public string Species {get;set;}

            }
        }


Notice that we have attributes for ColumnName and LoadColumn which come from the using statement Microsoft.ML.Data. LoadColumn is the column found in our CSV, Column name is how we will refer to when training our model. This is important to remember so that our label is not part of the data being trained, in this case the column named "Species" is our label.

Next, we need to create our prediction model called "PredectionModel". Again we will have an attribute called "ColumnName" so we can map the model to our training output. Predicted Species will represent the label, and Score is the confidence levels for each label.

        using Microsoft.ML.Data;

        namespace mlnet_intro
        {
            public class PredictionModel
            {
                [ColumnName("PredictedSpecies")]
                public string PredictedSpecies {get;set;}
                [ColumnName("Score")]
                public float[] Score {get;set;}
            }
        }


Create the Iris Prediction Application


Now that we have our two models created, we can create our application that will train our AI model for predicting Iris species. In the Program.cs file we will need to create some fields for holding our model context, along with referencing the Microsoft.ML namespace. Typically this would be a separate class, but we are getting close to 5 minutes. 

    
        using System;
        using System.IO;
        using System.Linq;
        using System.Collections.Generic;
        using Microsoft.ML;
        
        
        static MLContext context;
        //model for training/testing
        static Microsoft.ML.Data.TransformerChain model;
        static IEnumerable trainingData;
        static IEnumerable testingData;
        
        static string fileName = "irisModel.zip";



With our global variables defined, the next thing we must do is train our model. In order to do that we must load our csv data, then we will split the data into training and testing data. We then need to tell our training model the columns used to represent our features and our labels, and select a training method. In this case we will use the multiclass classification trainer. Finally we want to map the predicted value back to our prediction model.


        static void TrainModel()
        {
            
            //Load data from csv file
            var data = context.Data.LoadFromTextFile("datasets_19_420_Iris.csv", hasHeader:true, separatorChar: ',', allowQuoting: true, allowSparse:true, trimWhitespace: true);
            
            //Splits data into training and testing data
            //Id is the unique key to keep labels from duplicating
            var split = context.Data.TrainTestSplit(data);
            
             
            //create data sets for trainiing and testing
            trainingData = context.Data.CreateEnumerable(split.TrainSet, reuseRowObject: false);
            testingData = context.Data.CreateEnumerable(split.TestSet, reuseRowObject: false);


            //Create our pipeline and set our training model
            var pipeline = context.Transforms.Conversion.MapValueToKey(outputColumnName: "Label", "Species") //converts string to key value for training
                .Append(context.Transforms.Concatenate("Features", new[]{"SepalLengthCm", "SepalWidthCm", "PetalLengthCm", "PetalWidthCm"})) //identifies training data from model
                .Append(context.MulticlassClassification.Trainers.SdcaMaximumEntropy(labelColumnName: "Label", featureColumnName: "Features")) //set trainer and identifies features and label
                .Append(context.Transforms.Conversion.MapKeyToValue(outputColumnName: "PredictedSpecies", inputColumnName: "PredictedLabel")); //convert prediction to string PredictedLabel is output label key for predict

            //traings the model
             model = pipeline.Fit(context.Data.LoadFromEnumerable(trainingData));



        }

In the training method, the main thing to note are the two lines "MapValueToKey" and "MapKeyToValue". What this is doing is taking our string for our label and creating a key value. This will allow our prediction model to return a string value for the Iris name instead of the numeric value.

Now that our model is trained, we want to test it against our test data and check it's accuracy. ML.Net has this build into the training model.

        static void TestModel()
        {
            //transform data to a view that can be evaluated
            IDataView testDataPredictions = model.Transform(context.Data.LoadFromEnumerable(testingData));
            //evaluate test data against trained model for accuracy
            var metrics = context.MulticlassClassification.Evaluate(testDataPredictions);
            double accuracy = metrics.MicroAccuracy;

            Console.WriteLine("Accuracy {0}", accuracy.ToString());

        }

Accuracy may vary on this since it is a small dataset, this is for learning so we are not too concerned. Next we will save and load the model to and from a file. This is helpful for re using your model in web applications or other services. 

	static void SaveModel()
        {
            IDataView dataView = context.Data.LoadFromEnumerable(trainingData);
           context.Model.Save(model, dataView.Schema, fileName);
        }

        static ITransformer LoadModel()
        {
            DataViewSchema modelSchema;
            //gets a file from a stream, and loads it
            using(Stream s = File.Open(fileName, FileMode.Open))
            {
                return context.Model.Load(s, out modelSchema);

                
            }
         }


Finally, we can now use our newly saved model to predict Iris.

	static void Predict(IrisModel iris)
        {
            ITransformer trainedModel = LoadModel();

            //Creates prediction function from loaded model, you can load in memory model as well
             var predictFunction = context.Model.CreatePredictionEngine(trainedModel);
             
            //pass model to function to get prediction outputs
            PredictionModel prediction = predictFunction.Predict(iris);

            //get score, score is an array and the max score will align to key.
            float score = prediction.Score.Max();
        
            Console.WriteLine("Prediction: {0},  accuracy: {1}", prediction.PredictedSpecies, score);

        }


Our AI setup is complete, we just need to call our newly created methods and see the results. In my example I feed the species as "hello". This is to demonstrate that the model did not cheat and use the label as a feature.

	static void Main(string[] args)
        {
        	context = new MLContext();
            Console.WriteLine("Training Iris Model");
            TrainModel();
            Console.WriteLine("Testing Iris Model");
            TestModel();
            SaveModel();

            IrisModel test = new IrisModel(){
                    SepalLengthCm = 5.2f,
                    SepalWidthCm = 3.5f,
                    PetalLengthCm = 1.4f,
                    PetalWidthCm = 0.2f,
                    Species = "hello"
                };

            Predict(test);

            Console.Read();
        }


Clone the project


you can find the full project on my GitHub site here https://github.com/fiveminutecoder/blogs/tree/master/mlnet_intro
AI, Artificial Intelligence, C#, supervised learning, dotnet, dot net, machine learning, mldotnet, ml.net, dotnet core, dotnet 5, .NET 5
C#, C sharp, machine learning, ML.NET, dotnet core, dotnet, O365, Office 365, developer, development, Azure, Supervised Learning, Unsupervised Learning, NLP, Natural Language Programming, Microsoft, SharePoint, Teams, custom software development, sharepoint specialist, chat GPT,artificial intelligence, AI

Cookie Alert

This blog was created and hosted using Google's platform Blogspot (blogger.com). In accordance to privacy policy and GDPR please note the following: Third party vendors, including Google, use cookies to serve ads based on a user's prior visits to your website or other websites. Google's use of advertising cookies enables it and its partners to serve ads to your users based on their visit to your sites and/or other sites on the Internet. Users may opt out of personalized advertising by visiting Ads Settings. (Alternatively, you can opt out of a third-party vendor's use of cookies for personalized advertising by visiting www.aboutads.info.) Google analytics is also used, for more details please refer to Google Analytics privacy policy here: Google Analytics Privacy Policy Any information collected or given during sign up or sign is through Google's blogger platform and is stored by Google. The only Information collected outside of Google's platform is consent that the site uses cookies.