Blog

MSFT CLU FIRST LOOK : Excellent improvement!

Brad Crain, 5 min read

As mentioned in my last post I need to modify our insurance chatbot so it uses CLU rather than LUIS.  MSFT’s documentation and tools are nicely designed for this task and the migration of the LUIS schema to CLU’s worked flawlessly.  Once I had that data in CLU, it took me a bit of time to read MSFT’s CLU online documentation which was fine. Once I had that knowledge I did the required labeling, entity creation, and model training using CLU’s application interface.  Finally, I deployed my model.

Next major development steps will be to enhance our chatbot authoring solution to have a new CLU option where a user can define the CLU prediction endpoint and related confirmation information. And, my bot runtime will need to be extended to use CLU if that is the AI option selected by the chatbot author.  Before doing this work, I thought I would take a look at how CLU compares to LUIS in predicting intents and entities in conversational utterances.  

For my simple benchmarking, I had the exact same utterances for the entities in LUIS and CLU.  Using LUIS and CLU’s application testing features, I experimented with entering different test cases (e.g. “I am new to insurance and have no ideas what is a deductible. Please explain it to me”). My quick spot test indicates the CLU is a great improvement over LUIS in this capability! Utterances that LUIS couldn’t map to the correct entity were properly handled by CLUS.  My short test found that CLU is much better at classifying intents correctly, is more confident/certain is these findings, and the ability to identify entities has improved as well.   I was very impressed as our upcoming CLU based chatbot will be much better solution than the one using LUIS. 

Two examples illustrating these findings:

Results:

LUIS did not find the correct intent

CLU did find the intent: Information_EB.Insurance.Deductible.Explain.Req, Confidence: 85.19%

Results:

LUIS did not find correct intent

CLU did find the intent intentrequestFunStuff, Confidence: 94.20% and it also identified giggle correctly as a Humor entity

LUIS integrated testing feature - showing incorrect intent match for my utterance

CLU integrated testing feature - showing correct intent match for my utterance, along with a high confidence score