[RavenclawDev 196] Re: grounding_manager_configuration settinginconfig file of DM
Dan Bohus
dbohus at cs.cmu.edu
Sat Nov 25 15:21:48 EST 2006
Hi Antoine, Svetlana,
Antoine, thanks a lot for answering all these questions that Svetlana
had about RavenClaw. Sorry I haven't been able to be more responsive,
I've been really busy preparing job talks, etc. In any case, Svetlana,
if you have more questions or need more help regarding helios and the
confidence annotation model it uses, please let us know. I will try to
be better at responding :)
Hope things are well,
Cheers,
Dan,
________________________________
From: ravenclaw-developers-bounces at LOGANBERRY.srv.cs.cmu.edu
[mailto:ravenclaw-developers-bounces at LOGANBERRY.srv.cs.cmu.edu] On
Behalf Of Antoine Raux
Sent: Thursday, November 09, 2006 10:19 AM
To: ravenclaw-developers at cs.cmu.edu
Subject: [RavenclawDev 189] Re: grounding_manager_configuration
settinginconfig file of DM
Okay, let me reclarify this ;)
There is no "hard" assumption about getting a single parse in Helios.
What happens is that from the data it has been trained on, Helios
*learned* that when you have more than one top-level node, it is more
likely that your recognition hypothesis has errors (same thing for when
you have words not covered by the parse). So rather than a requirement
of Helios, this is an artifact of the training data we used in it. This
means that once you get data for your system, you can retrain Helios to
estimate confidence *given* your grammar. This answers your last
question: the full system, including Helios and RavenClaw could work
with a keyword spotting like grammar. Whether this would work better or
worse than a full fledge NL grammar needs to be seen empirically (this
is quite an interesting question actually)...
Of course the problem is that you need a system to get data in the first
place... One approach to get to that is to start by basically ignoring
confidence altogether in RavenClaw. This is done by first setting the
rejection threshold to 0 in your dialogue specification (sorry I don't
have access to my source code here but I think you need to add something
like DEFAULT_REJECTION_THRESHOLD(0) to the CORE_CONFIGURATION section).
The second thing you need to do is to use a heuristic-based grounding
policy (this is what we do in Let's Go). This requires a little more
changes and potentially getting the latest version of RavenClaw (I'm not
sure which one you have). I'll get back to you on that later.
antoine
________________________________
From: Svetlana Stenchikova [mailto:svetastenchikova at gmail.com]
Sent: Thursday, November 09, 2006 9:36 AM
To: Antoine Raux
Cc: ravenclaw-developers at cs.cmu.edu
Subject: Re: [RavenclawDev 184] Re: grounding_manager_configuration
setting inconfig file of DM
Thank you Antoine, this clarifies the things.
So, currently Helios assumes that there should be a single parse for an
utterance. I can make these changes.
Do you think that if Helios did not have this requirement, the
recognition would deteriorate?
I though that since in RavenClaw you can set the nodes of the dialog
tree where a concept is recognized, this limits the number of concepts
that are in focus at each point and recognition of a full sentence may
not be necessary to disambiguate between hypotheses.
I think it would be interesting to investigate if RavenClaw would work
with simple grammars that pick out concepts from an utterance. (Assuming
Helios is modified to allow this). Or do you think there is a flaw in
this idea?
thanks,
Svetlana
On 11/8/06, Antoine Raux <antoine at cs.cmu.edu> wrote:
I see. I assume [Question] and [EventTypeSpec] are top-level slots in
your grammar right? The "IS" in "WHERE IS CLASS" doesn't get covered by
any parse? This looks like a key-phrase-spotting-like approach. While
this is perfectly okay in the sense that Phoenix will return the right
semantic structure, we try in general to get our grammars to cover as
much as possible of the input. The idea is that the fact that we get a
full parse or a fragmented parse gives an indication on how confident we
should be in the recognizer output (assuming your user speaks natural
English, I'd be more confident in an ASR result like "WHERE IS THE
CLASS" than in "WHERE CLASS" or "WHERE BLAHBLAH CLASS"). So in your
case, you can either extend your [where_event] rules to include an
(optional) IS:
[where_event]
(where *is)
(at which location *is)
(what is the location *is)
(which location *is)
(what location *is)
;
or (even better) you can create a top-level rule like:
[Question]
([where_event] *is [EventType])
([whattime_event] *is [EventType])
([describe_event] [EventType])
;
This way you can still bind [where_event] and [EventType] to different
concepts (in different agents) but your parse will cover more of your
input, under a single top-level slot (which means that you can be pretty
confident that your input was correctly recognized). Of course it
requires more work to actually cover most of what users might say (all
the variants of "WHERE IS THE MEETING", "IN WHICH ROOM IS THE MEETING",
"WHERE WILL THE MEETING BE HELD AT"...) but the best you can do that,
the better your system will behave, both in terms of understanding what
the user means and in terms of engaging in confirmations at the right
time.
Does that clarify things a bit?
antoine
________________________________
From: Svetlana Stenchikova [mailto:svetastenchikova at gmail.com]
Sent: Wednesday, November 08, 2006 9:17 PM
To: Antoine Raux
Cc: ravenclaw-developers at cs.cmu.edu
Subject: Re: [RavenclawDev 184] Re: grounding_manager_configuration
setting inconfig file of DM
Thanks Antoine,
Helios does not have access to the Phoenix grammar does it? It only
gets the string and the output parse, right?
Here is my grammar excerpt. (the full grammar file is attached)
I am binding Question and EventTypeSpec concepts in separate agents, so
phoenix outputs:
(Maybe this is a problem?)
[Question] ( [where_event] ( WHERE ) )
Features
[EventTypeSpec] ( [event_type] ( [event_type_singular] ( MEETING ) ) )
Question and EventTypeSpec are declared in form
[Question]
([whattime_event])
([describe_event])
([where_event])
;
[where_event]
(where)
(at which location)
(what is the location)
(which location)
(what location)
;
# SS: I used to have this, but it did not matter:
# (where *is *the *EVENT)
# (*at which location *is *the *EVENT)
[EventTypeSpec]
([event_type])
;
[event_type]
([event_type_singular])
([event_type_plural])
;
On 11/8/06, Antoine Raux <antoine at cs.cmu.edu> wrote:
I think the problem is either in the grammar or in Helios. Helios uses
features such as "how many top level slots does the parse contain", "how
many words are covered by the parse", ... and combines them (along with
more typical speech recognition features) to produce the confidence
score. What seems to be happening here is that the grammar is such that
the parse for "WHERE IS CLASS", while producing the right concepts,
leads to low confidence.
Three ways to solve this: 1) hack Helios to always give perfect
confidence for parsed input (that is *not* a long term solution, I
wouldn't recommend it), 2) fix your grammar so that it produces parses
that are more like what Helios expect, 3) fix the Helios model to match
your grammar. The problem with 3) is that the confidence model in Helios
is trained on data so you'd need data for your system before being able
to retrain Helios. So for now I'd try to go for 2).
Could you send the parse for these two utterances and you grammar?
antoine
________________________________
From: Svetlana Stenchikova [mailto:svetastenchikova at gmail.com]
Sent: Wednesday, November 08, 2006 5:45 PM
To: Antoine Raux
Cc: ravenclaw-developers at cs.cmu.edu
Subject: Re: [RavenclawDev 182] Re: grounding_manager_configuration
setting inconfig file of DM
In the case of WHERE CLASS, the conversation continues correctly, but in
WHERE IS CLASS it it goes into default explicit confirmation for CLASS
(I just added it - thanks for the hint)
I don't think it is the desirable behavior though. These typed (or
perfectly recognized) concepts. Why should it need to confirm them?
thanks
Svetlana
On 11/8/06, Antoine Raux < antoine at cs.cmu.edu
<mailto:antoine at cs.cmu.edu> > wrote:
What exactly is happening after you type either of those?
In the second case (WHERE IS CLASS) RC is trying to perform an explicit
confirmation "I think you said you wanted information about a class. Is
this correct?", whereas in the first case it's just trying to do an
implicit confirmation "Information about a class... <moves on to the
next task related question>".
For any of these to work properly, you need to have the
ExplicitConfirm.pm and ImplicitConfirm.pm updated in the NLG. They
should map the concept name to the proper confirmation prompt (these
Rosetta modeuls work the same as Inform.pm or Request.pm). Maybe the
problem is that you're missing those? It wouldn't be a major problem in
the implicit confirmation case because the dialog moves on anyway but it
would in the explicit confirmation case because RC waits for a user
answer (it creates a new request agent and puts it on top of the stack).
So the thing is, when you start dealing with speech, you need to have
all those grounding models and their associated prompts in the NLG set
correctly. It takes a little while but then RC does the whole grounding
job for you...
I guess we really need some documentation about this (Dan??)...
Hope this helps...
antoine
________________________________
From: Svetlana Stenchikova [mailto:svetastenchikova at gmail.com]
Sent: Wednesday, November 08, 2006 4:51 PM
To: Antoine Raux
Cc: ravenclaw-developers at cs.cmu.edu
Subject: Re: [RavenclawDev 179] grounding_manager_configuration setting
inconfig file of DM
I am not sure which entry to look at.
Could this be because I am binding multiple concepts?
It is not a problem for me now as I can fix it by taking out
grounding_manager_configuration from config.
I am just concerned that I will deal with these issues with speech.
>From a first glance at the helios log:
WHERE CLASS
confidence 0.62288
WHERE IS CLASS
confidence 0.28883
In the dialog.log I get entries:
I think this is for WHERE CLASS:
****************************************
[GMX at 16:11:10.083] Grounding model /RavenCalendar/question_type_string
[TYPE=concept_default;POLICY=expl;EM=greedy;EP=0.20]:
Full state:
grounded = false
top_confidence = 0.622880
updated = true
Belief state: INACTIVE:0.00 CONFIDENT:0.62 UNCONFIDENT:0.38
GROUNDED:0.00
Action values (dumped below):
ACCEPT:2.4576
EXPL_CONF:0.6568
Suggested action: ACCEPT
[GMX at 16:11:10.083] Concept /RavenCalendar/the_event.event_type dumped
below:
CLASS|0.62
[GMX at 16:11:10.083] Grounding model /RavenCalendar/the_event.event_type
[TYPE=concept_default;POLICY=expl_impl;EM=greedy;EP=0.20]:
Full state:
grounded = false
top_confidence = 0.622880
updated = true
Belief state: INACTIVE:0.00 CONFIDENT:0.62 UNCONFIDENT:0.38
GROUNDED:0.00
Action values (dumped below):
ACCEPT:-2.1822
EXPL_CONF:0.6568
IMPL_CONF:1.8856
Suggested action: IMPL_CONF
*****************************************
and this for WHERE IS CLASS
[GMX at 16:11:13.314] Grounding model /RavenCalendar/question_type_string
[TYPE=concept_default;POLICY=expl;EM=greedy;EP=0.20]:
Full state:
grounded = false
top_confidence = 0.000000
updated = false
Belief state: INACTIVE:1.00 CONFIDENT:0.00 UNCONFIDENT:0.00
GROUNDED:0.00
Action values (dumped below):
ACCEPT:10.0000
EXPL_CONF: -
Suggested action: ACCEPT
[GMX at 16:11:13.314] Concept /RavenCalendar/the_event.event_type dumped
below:
<UNDEFINED>
[GMX at 16:11:13.315] Grounding model /RavenCalendar/the_event.event_type
[TYPE=concept_default;POLICY=expl_impl;EM=greedy;EP=0.20]:
Full state:
grounded = false
top_confidence = 0.000000
[GMX at 16:11:37.467] Concept /RavenCalendar/question_type_string dumped
below:
where|0.29
[GMX at 16:11:37.467] Grounding model /RavenCalendar/question_type_string
[TYPE=concept_default;POLICY=expl;EM=greedy;EP=0.20]:
Full state:
grounded = false
top_confidence = 0.288830
updated = true
Belief state: INACTIVE:0.00 CONFIDENT:0.29 UNCONFIDENT:0.71
GROUNDED:0.00
Action values (dumped below):
ACCEPT:-4.2234
EXPL_CONF:5.6675
Suggested action: EXPL_CONF
On 11/8/06, Antoine Raux <antoine at cs.cmu.edu> wrote:
Hi Svetlana,
Have you looked at the confidence value for the input in both cases?
This is one of the features that Helios sends to the DM for each input.
It is possible that "where class" has a low confidence for some reason
(e.g. segmented parse...) and RC rejects it. Normally if your grammar
gives a full parse for "where class" that shouldn't happen (at least on
typed input) but...
antoine
________________________________
From: ravenclaw-developers-bounces at LOGANBERRY.srv.cs.cmu.edu
[mailto:ravenclaw-developers-bounces at LOGANBERRY.srv.cs.cmu.edu] On
Behalf Of Svetlana Stenchikova
Sent: Wednesday, November 08, 2006 2:50 PM
To: ravenclaw-developers at cs.cmu.edu
Subject: [RavenclawDev 179] grounding_manager_configuration setting
inconfig file of DM
I have encountered an issue with binding a concept in DM depending on a
user utterance.
in my DM I have a grammar mapping. "@(...)[Question.where_event]>where,"
The user can ask a question in multiple ways:
where class
where is class
where is the class
what is the location the class
..
for all of these entries PHOENIX produces identical parse
[Question] ( [where_event] ( WHERE ) )
Features
[EventTypeSpec] ( [event_type] ( [event_type_singular] ( MEETING ) ) )
" } )
When a configuration file passed to the DM in -config has an entry:
grounding_manager_configuration =
turns:default;concepts:default;beliefupdatingmodel:npu
if a user types:
"where is the class" DM does not bind the concepts, but if the user
types
"where class" DM binds the concepts.
When the grounding_manager_configuration is taken out of DM, the binding
succeeds for both utterances.
Can this be explained? Do you think there may be a problem with my
grammar or something in my DM?
The grammar of the Phoenix parser should not matter because in all cases
the parses are identical, right?
thank you
Svetlana
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.srv.cs.cmu.edu/pipermail/ravenclaw-developers/attachments/20061125/759dbfed/attachment-0001.html
More information about the Ravenclaw-developers
mailing list