[RavenclawDev 184] Re: grounding_manager_configuration setting inconfig file of DM

[Antoine Raux]

I think the problem is either in the grammar or in Helios. Helios uses
features such as "how many top level slots does the parse contain", "how
many words are covered by the parse", . and combines them (along with more
typical speech recognition features) to produce the confidence score. What
seems to be happening here is that the grammar is such that the parse for
"WHERE IS CLASS", while producing the right concepts, leads to low
confidence.

Three ways to solve this: 1) hack Helios to always give perfect confidence
for parsed input (that is *not* a long term solution, I wouldn't recommend
it), 2) fix your grammar so that it produces parses that are more like what
Helios expect, 3) fix the Helios model to match your grammar. The problem
with 3) is that the confidence model in Helios is trained on data so you'd
need data for your system before being able to retrain Helios. So for now
I'd try to go for 2).

 

Could you send the parse for these two utterances and you grammar?

 

antoine

 

  _____  

From: Svetlana Stenchikova [mailto:svetastenchikova at gmail.com] 
Sent: Wednesday, November 08, 2006 5:45 PM
To: Antoine Raux
Cc: ravenclaw-developers at cs.cmu.edu
Subject: Re: [RavenclawDev 182] Re: grounding_manager_configuration setting
inconfig file of DM

 

In the case of WHERE CLASS, the conversation continues correctly, but in
WHERE IS CLASS it  it goes into default explicit confirmation for CLASS (I
just added it - thanks for the hint)

I don't think it is the desirable behavior though. These typed (or perfectly
recognized) concepts. Why should it need to confirm them?

thanks
Svetlana

On 11/8/06, Antoine Raux < <mailto:antoine at cs.cmu.edu>  antoine at cs.cmu.edu>
wrote:

What exactly is happening after you type either of those?

In the second case (WHERE IS CLASS) RC is trying to perform an explicit
confirmation "I think you said you wanted information about a class. Is this
correct?", whereas in the first case it's just trying to do an implicit
confirmation "Information about a class. <moves on to the next task related
question>".

For any of these to work properly, you need to have the ExplicitConfirm.pm
and ImplicitConfirm.pm updated in the NLG. They should map the concept name
to the proper confirmation prompt (these Rosetta modeuls work the same as
Inform.pm or Request.pm). Maybe the problem is that you're missing those? It
wouldn't be a major problem in the implicit confirmation case because the
dialog moves on anyway but it would in the explicit confirmation case
because RC waits for a user answer (it creates a new request agent and puts
it on top of the stack).

 

So the thing is, when you start dealing with speech, you need to have all
those grounding models and their associated prompts in the NLG set
correctly. It takes a little while but then RC does the whole grounding job
for you.

I guess we really need some documentation about this (Dan??).

 

Hope this helps.

antoine

 

  _____  

From: Svetlana Stenchikova [mailto:svetastenchikova at gmail.com] 
Sent: Wednesday, November 08, 2006 4:51 PM
To: Antoine Raux
Cc: ravenclaw-developers at cs.cmu.edu
Subject: Re: [RavenclawDev 179] grounding_manager_configuration setting
inconfig file of DM

 

I am not sure which entry to look at.  

Could this be because I am binding multiple  concepts?
It is not a problem for me now as I can fix it by taking out
grounding_manager_configuration from config. 
I am just concerned that I will deal with these issues with speech.


>From a first glance at the helios log:
WHERE CLASS

confidence    0.62288

WHERE IS CLASS

confidence    0.28883


In the dialog.log I get entries:

I think this is for WHERE CLASS:
****************************************

[GMX at 16:11:10.083] Grounding model /RavenCalendar/question_type_string
[TYPE=concept_default;POLICY=expl;EM=greedy;EP=0.20]:
Full state: 
grounded = false
top_confidence = 0.622880
updated = true
Belief state: INACTIVE:0.00  CONFIDENT:0.62  UNCONFIDENT:0.38  GROUNDED:0.00

Action values (dumped below):
  ACCEPT:2.4576
  EXPL_CONF:0.6568

Suggested action: ACCEPT

[GMX at 16:11:10.083] Concept /RavenCalendar/the_event.event_type dumped below:
CLASS|0.62
[GMX at 16:11:10.083] Grounding model /RavenCalendar/the_event.event_type
[TYPE=concept_default;POLICY=expl_impl;EM=greedy;EP=0.20]:
Full state: 
grounded = false
top_confidence = 0.622880
updated = true
Belief state: INACTIVE:0.00  CONFIDENT:0.62  UNCONFIDENT:0.38  GROUNDED:0.00

Action values (dumped below):
  ACCEPT:-2.1822
  EXPL_CONF:0.6568
  IMPL_CONF:1.8856

Suggested action: IMPL_CONF



*****************************************

and this for WHERE IS CLASS



[GMX at 16:11:13.314] Grounding model /RavenCalendar/question_type_string
[TYPE=concept_default;POLICY=expl;EM=greedy;EP=0.20]:
Full state: 
grounded = false
top_confidence = 0.000000
updated = false
Belief state: INACTIVE:1.00  CONFIDENT:0.00  UNCONFIDENT:0.00  GROUNDED:0.00

Action values (dumped below):
  ACCEPT:10.0000
  EXPL_CONF: -

Suggested action: ACCEPT

[GMX at 16:11:13.314] Concept /RavenCalendar/the_event.event_type dumped below:
<UNDEFINED>
[GMX at 16:11:13.315] Grounding model /RavenCalendar/the_event.event_type
[TYPE=concept_default;POLICY=expl_impl;EM=greedy;EP=0.20]:
Full state: 
grounded = false
top_confidence = 0.000000


[GMX at 16:11:37.467] Concept /RavenCalendar/question_type_string dumped below:
where|0.29
[GMX at 16:11:37.467] Grounding model /RavenCalendar/question_type_string
[TYPE=concept_default;POLICY=expl;EM=greedy;EP=0.20]:
Full state: 
grounded = false
top_confidence = 0.288830
updated = true
Belief state: INACTIVE:0.00  CONFIDENT:0.29  UNCONFIDENT:0.71  GROUNDED:0.00

Action values (dumped below):
  ACCEPT:-4.2234
  EXPL_CONF:5.6675

Suggested action: EXPL_CONF







On 11/8/06, Antoine Raux <antoine at cs.cmu.edu> wrote:

Hi Svetlana,

 

Have you looked at the confidence value for the input in both cases? This is
one of the features that Helios sends to the DM for each input. It is
possible that "where class" has a low confidence for some reason (e.g.
segmented parse.) and RC rejects it. Normally if your grammar gives a full
parse for "where class" that shouldn't happen (at least on typed input) but.

 

antoine

 

  _____  

From: ravenclaw-developers-bounces at LOGANBERRY.srv.cs.cmu.edu
[mailto:ravenclaw-developers-bounces at LOGANBERRY.srv.cs.cmu.edu] On Behalf Of
Svetlana Stenchikova
Sent: Wednesday, November 08, 2006 2:50 PM
To: ravenclaw-developers at cs.cmu.edu
Subject: [RavenclawDev 179] grounding_manager_configuration setting inconfig
file of DM

 


I have encountered an issue with binding a concept in DM depending on a user
utterance.

in my DM I have a grammar mapping. "@(...)[Question.where_event]>where,"


The user can ask a question in multiple ways:
where class
where is class
where is the class
what is the location the class
..

for all of these entries PHOENIX produces identical parse

[Question] ( [where_event] ( WHERE ) )
Features
[EventTypeSpec] ( [event_type] ( [event_type_singular] ( MEETING ) ) )

" } )

When a configuration file passed to the DM in -config has an entry:
grounding_manager_configuration =
turns:default;concepts:default;beliefupdatingmodel:npu


if a user types:
"where is the class"    DM does not bind the concepts, but if the user types

"where class"             DM binds the concepts.

When the grounding_manager_configuration is taken out of DM, the binding
succeeds for both utterances.

Can this be explained?  Do you think there may be a problem with my grammar
or something in my DM? 
The grammar of the Phoenix parser should not matter because in all cases the
parses are identical, right?

thank you

Svetlana

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.srv.cs.cmu.edu/pipermail/ravenclaw-developers/attachments/20061108/650f792f/attachment-0001.html