[RavenclawDev 189] Re: grounding_manager_configuration setting inconfig file of DM

Thu Nov 9 10:19:11 EST 2006

Okay, let me reclarify this ;)

There is no "hard" assumption about getting a single parse in Helios. What
happens is that from the data it has been trained on, Helios *learned* that
when you have more than one top-level node, it is more likely that your
recognition hypothesis has errors (same thing for when you have words not
covered by the parse). So rather than a requirement of Helios, this is an
artifact of the training data we used in it. This means that once you get
data for your system, you can retrain Helios to estimate confidence *given*
your grammar. This answers your last question: the full system, including
Helios and RavenClaw could work with a keyword spotting like grammar.
Whether this would work better or worse than a full fledge NL grammar needs
to be seen empirically (this is quite an interesting question actually).

Of course the problem is that you need a system to get data in the first
place. One approach to get to that is to start by basically ignoring
confidence altogether in RavenClaw. This is done by first setting the
rejection threshold to 0 in your dialogue specification (sorry I don't have
access to my source code here but I think you need to add something like
DEFAULT_REJECTION_THRESHOLD(0) to the CORE_CONFIGURATION section). The
second thing you need to do is to use a heuristic-based grounding policy
(this is what we do in Let's Go). This requires a little more changes and
potentially getting the latest version of RavenClaw (I'm not sure which one
you have). I'll get back to you on that later.

antoine

  _____  

From: Svetlana Stenchikova [mailto:svetastenchikova at gmail.com] 
Sent: Thursday, November 09, 2006 9:36 AM
To: Antoine Raux
Cc: ravenclaw-developers at cs.cmu.edu
Subject: Re: [RavenclawDev 184] Re: grounding_manager_configuration setting
inconfig file of DM

Thank you Antoine, this clarifies the things.

So, currently Helios assumes that there should be a single parse for an
utterance. I can make these changes.

Do you think that if Helios did not have this requirement, the recognition
would deteriorate?

I though that since in RavenClaw you can set the nodes of the dialog tree
where a concept is recognized, this limits the number of concepts that are
in focus at each point and recognition of a full sentence may not be
necessary to disambiguate between hypotheses. 

I think it would be interesting to investigate if RavenClaw would work with
simple grammars that pick out concepts from an utterance. (Assuming Helios
is modified to allow this). Or do you think there is a flaw in this idea?

thanks,
Svetlana

On 11/8/06, Antoine Raux <antoine at cs.cmu.edu> wrote:

I see. I assume [Question] and [EventTypeSpec] are top-level slots in your
grammar right? The "IS" in "WHERE IS CLASS" doesn't get covered by any
parse? This looks like a key-phrase-spotting-like approach. While this is
perfectly okay in the sense that Phoenix will return the right semantic
structure, we try in general to get our grammars to cover as much as
possible of the input. The idea is that the fact that we get a full parse or
a fragmented parse gives an indication on how confident we should be in the
recognizer output (assuming your user speaks natural English, I'd be more
confident in an ASR result like "WHERE IS THE CLASS" than in "WHERE CLASS"
or "WHERE BLAHBLAH CLASS"). So in your case, you can either extend your
[where_event] rules to include an (optional) IS:

[where_event]
    (where *is)
    (at which location *is)
    (what is the location *is)
    (which location *is)
    (what location *is)
;

or (even better) you can create a top-level rule like:

[Question]

            ([where_event] *is [EventType])

            ([whattime_event] *is [EventType])

            ([describe_event] [EventType])

;

This way you can still bind [where_event] and [EventType] to different
concepts (in different agents) but your parse will cover more of your input,
under a single top-level slot (which means that you can be pretty confident
that your input was correctly recognized). Of course it requires more work
to actually cover most of what users might say (all the variants of "WHERE
IS THE MEETING", "IN WHICH ROOM IS THE MEETING", "WHERE WILL THE MEETING BE
HELD AT".) but the best you can do that, the better your system will behave,
both in terms of understanding what the user means and in terms of engaging
in confirmations at the right time.

Does that clarify things a bit?

antoine

  _____  

From: Svetlana Stenchikova [mailto:svetastenchikova at gmail.com] 
Sent: Wednesday, November 08, 2006 9:17 PM
To: Antoine Raux
Cc: ravenclaw-developers at cs.cmu.edu
Subject: Re: [RavenclawDev 184] Re: grounding_manager_configuration setting
inconfig file of DM

Thanks Antoine,

Helios does not have access to the Phoenix grammar does it?  It only gets
the string and the output parse, right?
Here is my grammar excerpt. (the full grammar file is attached)

I am binding Question and EventTypeSpec concepts in separate agents, so
phoenix outputs:
(Maybe this is a problem?)

[Question] ( [where_event] ( WHERE ) )
Features
[EventTypeSpec] ( [event_type] ( [event_type_singular] ( MEETING ) ) )

Question and EventTypeSpec are declared in form

[Question]
    ([whattime_event]) 
    ([describe_event])
    ([where_event])
;

[where_event]
    (where)
    (at which location)
    (what is the location)
    (which location)
    (what location)
;

# SS: I used to have this, but it did not matter:
#    (where *is *the *EVENT)
#    (*at which location *is *the *EVENT)

[EventTypeSpec]
    ([event_type])
;

[event_type]
    ([event_type_singular])
    ([event_type_plural])
;

On 11/8/06, Antoine Raux <antoine at cs.cmu.edu> wrote:

I think the problem is either in the grammar or in Helios. Helios uses
features such as "how many top level slots does the parse contain", "how
many words are covered by the parse", . and combines them (along with more
typical speech recognition features) to produce the confidence score. What
seems to be happening here is that the grammar is such that the parse for
"WHERE IS CLASS", while producing the right concepts, leads to low
confidence.

Three ways to solve this: 1) hack Helios to always give perfect confidence
for parsed input (that is *not* a long term solution, I wouldn't recommend
it), 2) fix your grammar so that it produces parses that are more like what
Helios expect, 3) fix the Helios model to match your grammar. The problem
with 3) is that the confidence model in Helios is trained on data so you'd
need data for your system before being able to retrain Helios. So for now
I'd try to go for 2).

Could you send the parse for these two utterances and you grammar?

antoine

  _____  

From: Svetlana Stenchikova [mailto:svetastenchikova at gmail.com] 
Sent: Wednesday, November 08, 2006 5:45 PM
To: Antoine Raux
Cc: ravenclaw-developers at cs.cmu.edu
Subject: Re: [RavenclawDev 182] Re: grounding_manager_configuration setting
inconfig file of DM

In the case of WHERE CLASS, the conversation continues correctly, but in
WHERE IS CLASS it  it goes into default explicit confirmation for CLASS (I
just added it - thanks for the hint)

I don't think it is the desirable behavior though. These typed (or perfectly
recognized) concepts. Why should it need to confirm them?

thanks
Svetlana

On 11/8/06, Antoine Raux < antoine at cs.cmu.edu <mailto:antoine at cs.cmu.edu> >
wrote:

What exactly is happening after you type either of those?

In the second case (WHERE IS CLASS) RC is trying to perform an explicit
confirmation "I think you said you wanted information about a class. Is this
correct?", whereas in the first case it's just trying to do an implicit
confirmation "Information about a class. <moves on to the next task related
question>".

For any of these to work properly, you need to have the ExplicitConfirm.pm
and ImplicitConfirm.pm updated in the NLG. They should map the concept name
to the proper confirmation prompt (these Rosetta modeuls work the same as
Inform.pm or Request.pm). Maybe the problem is that you're missing those? It
wouldn't be a major problem in the implicit confirmation case because the
dialog moves on anyway but it would in the explicit confirmation case
because RC waits for a user answer (it creates a new request agent and puts
it on top of the stack).

So the thing is, when you start dealing with speech, you need to have all
those grounding models and their associated prompts in the NLG set
correctly. It takes a little while but then RC does the whole grounding job
for you.

I guess we really need some documentation about this (Dan??).

Hope this helps.

antoine

  _____  

From: Svetlana Stenchikova [mailto:svetastenchikova at gmail.com] 
Sent: Wednesday, November 08, 2006 4:51 PM
To: Antoine Raux
Cc: ravenclaw-developers at cs.cmu.edu
Subject: Re: [RavenclawDev 179] grounding_manager_configuration setting
inconfig file of DM

I am not sure which entry to look at.  

Could this be because I am binding multiple  concepts?
It is not a problem for me now as I can fix it by taking out
grounding_manager_configuration from config. 
I am just concerned that I will deal with these issues with speech.

>From a first glance at the helios log:
WHERE CLASS

confidence    0.62288

WHERE IS CLASS

confidence    0.28883

In the dialog.log I get entries:

I think this is for WHERE CLASS:
****************************************

[GMX at 16:11:10.083] Grounding model /RavenCalendar/question_type_string
[TYPE=concept_default;POLICY=expl;EM=greedy;EP=0.20]:
Full state: 
grounded = false
top_confidence = 0.622880
updated = true
Belief state: INACTIVE:0.00  CONFIDENT:0.62  UNCONFIDENT:0.38  GROUNDED:0.00

Action values (dumped below):
  ACCEPT:2.4576
  EXPL_CONF:0.6568

Suggested action: ACCEPT

[GMX at 16:11:10.083] Concept /RavenCalendar/the_event.event_type dumped below:
CLASS|0.62
[GMX at 16:11:10.083] Grounding model /RavenCalendar/the_event.event_type
[TYPE=concept_default;POLICY=expl_impl;EM=greedy;EP=0.20]:
Full state: 
grounded = false
top_confidence = 0.622880
updated = true
Belief state: INACTIVE:0.00  CONFIDENT:0.62  UNCONFIDENT:0.38  GROUNDED:0.00

Action values (dumped below):
  ACCEPT:-2.1822
  EXPL_CONF:0.6568
  IMPL_CONF:1.8856

Suggested action: IMPL_CONF

*****************************************

and this for WHERE IS CLASS

[GMX at 16:11:13.314] Grounding model /RavenCalendar/question_type_string
[TYPE=concept_default;POLICY=expl;EM=greedy;EP=0.20]:
Full state: 
grounded = false
top_confidence = 0.000000
updated = false
Belief state: INACTIVE:1.00  CONFIDENT:0.00  UNCONFIDENT:0.00  GROUNDED:0.00

Action values (dumped below):
  ACCEPT:10.0000
  EXPL_CONF: -

Suggested action: ACCEPT

[GMX at 16:11:13.314] Concept /RavenCalendar/the_event.event_type dumped below:
<UNDEFINED>
[GMX at 16:11:13.315] Grounding model /RavenCalendar/the_event.event_type
[TYPE=concept_default;POLICY=expl_impl;EM=greedy;EP=0.20]:
Full state: 
grounded = false
top_confidence = 0.000000

[GMX at 16:11:37.467] Concept /RavenCalendar/question_type_string dumped below:
where|0.29
[GMX at 16:11:37.467] Grounding model /RavenCalendar/question_type_string
[TYPE=concept_default;POLICY=expl;EM=greedy;EP=0.20]:
Full state: 
grounded = false
top_confidence = 0.288830
updated = true
Belief state: INACTIVE:0.00  CONFIDENT:0.29  UNCONFIDENT:0.71  GROUNDED:0.00

Action values (dumped below):
  ACCEPT:-4.2234
  EXPL_CONF:5.6675

Suggested action: EXPL_CONF

On 11/8/06, Antoine Raux <antoine at cs.cmu.edu> wrote:

Hi Svetlana,

Have you looked at the confidence value for the input in both cases? This is
one of the features that Helios sends to the DM for each input. It is
possible that "where class" has a low confidence for some reason (e.g.
segmented parse.) and RC rejects it. Normally if your grammar gives a full
parse for "where class" that shouldn't happen (at least on typed input) but.

antoine

  _____  

From: ravenclaw-developers-bounces at LOGANBERRY.srv.cs.cmu.edu
[mailto:ravenclaw-developers-bounces at LOGANBERRY.srv.cs.cmu.edu] On Behalf Of
Svetlana Stenchikova
Sent: Wednesday, November 08, 2006 2:50 PM
To: ravenclaw-developers at cs.cmu.edu
Subject: [RavenclawDev 179] grounding_manager_configuration setting inconfig
file of DM

I have encountered an issue with binding a concept in DM depending on a user
utterance.

in my DM I have a grammar mapping. "@(...)[Question.where_event]>where,"

The user can ask a question in multiple ways:
where class
where is class
where is the class
what is the location the class
..

for all of these entries PHOENIX produces identical parse

[Question] ( [where_event] ( WHERE ) )
Features
[EventTypeSpec] ( [event_type] ( [event_type_singular] ( MEETING ) ) )

" } )

When a configuration file passed to the DM in -config has an entry:
grounding_manager_configuration =
turns:default;concepts:default;beliefupdatingmodel:npu

if a user types:
"where is the class"    DM does not bind the concepts, but if the user types

"where class"             DM binds the concepts.

When the grounding_manager_configuration is taken out of DM, the binding
succeeds for both utterances.

Can this be explained?  Do you think there may be a problem with my grammar
or something in my DM? 
The grammar of the Phoenix parser should not matter because in all cases the
parses are identical, right?

thank you

Svetlana

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.srv.cs.cmu.edu/pipermail/ravenclaw-developers/attachments/20061109/c168e890/attachment-0001.html