[RavenclawDev 185] Re: grounding_manager_configuration setting inconfig file of DM
Svetlana Stenchikova
svetastenchikova at gmail.com
Wed Nov 8 21:16:40 EST 2006
Thanks Antoine,
Helios does not have access to the Phoenix grammar does it? It only gets
the string and the output parse, right?
Here is my grammar excerpt. (the full grammar file is attached)
I am binding Question and EventTypeSpec concepts in separate agents, so
phoenix outputs:
(Maybe this is a problem?)
[Question] ( [where_event] ( WHERE ) )
Features
[EventTypeSpec] ( [event_type] ( [event_type_singular] ( MEETING ) ) )
Question and EventTypeSpec are declared in form
[Question]
([whattime_event])
([describe_event])
([where_event])
;
[where_event]
(where)
(at which location)
(what is the location)
(which location)
(what location)
;
# SS: I used to have this, but it did not matter:
# (where *is *the *EVENT)
# (*at which location *is *the *EVENT)
[EventTypeSpec]
([event_type])
;
[event_type]
([event_type_singular])
([event_type_plural])
;
On 11/8/06, Antoine Raux <antoine at cs.cmu.edu> wrote:
>
> I think the problem is either in the grammar or in Helios. Helios uses
> features such as "how many top level slots does the parse contain", "how
> many words are covered by the parse", … and combines them (along with more
> typical speech recognition features) to produce the confidence score. What
> seems to be happening here is that the grammar is such that the parse for
> "WHERE IS CLASS", while producing the right concepts, leads to low
> confidence.
>
> Three ways to solve this: 1) hack Helios to always give perfect confidence
> for parsed input (that is **not** a long term solution, I wouldn't
> recommend it), 2) fix your grammar so that it produces parses that are more
> like what Helios expect, 3) fix the Helios model to match your grammar. The
> problem with 3) is that the confidence model in Helios is trained on data so
> you'd need data for your system before being able to retrain Helios. So for
> now I'd try to go for 2).
>
>
>
> Could you send the parse for these two utterances and you grammar?
>
>
>
> antoine
>
>
> ------------------------------
>
> *From:* Svetlana Stenchikova [mailto:svetastenchikova at gmail.com]
> *Sent:* Wednesday, November 08, 2006 5:45 PM
> *To:* Antoine Raux
> *Cc:* ravenclaw-developers at cs.cmu.edu
> *Subject:* Re: [RavenclawDev 182] Re: grounding_manager_configuration
> setting inconfig file of DM
>
>
>
> In the case of WHERE CLASS, the conversation continues correctly, but in
> WHERE IS CLASS it it goes into default explicit confirmation for CLASS (I
> just added it - thanks for the hint)
>
> I don't think it is the desirable behavior though. These typed (or
> perfectly recognized) concepts. Why should it need to confirm them?
>
> thanks
> Svetlana
>
> On 11/8/06, *Antoine Raux* < antoine at cs.cmu.edu> wrote:
>
> What exactly is happening after you type either of those?
>
> In the second case (WHERE IS CLASS) RC is trying to perform an explicit
> confirmation "I think you said you wanted information about a class. Is this
> correct?", whereas in the first case it's just trying to do an implicit
> confirmation "Information about a class… <moves on to the next task related
> question>".
>
> For any of these to work properly, you need to have the ExplicitConfirm.pmand
> ImplicitConfirm.pm updated in the NLG. They should map the concept name to
> the proper confirmation prompt (these Rosetta modeuls work the same as
> Inform.pm or Request.pm). Maybe the problem is that you're missing those?
> It wouldn't be a major problem in the implicit confirmation case because the
> dialog moves on anyway but it would in the explicit confirmation case
> because RC waits for a user answer (it creates a new request agent and puts
> it on top of the stack).
>
>
>
> So the thing is, when you start dealing with speech, you need to have all
> those grounding models and their associated prompts in the NLG set
> correctly. It takes a little while but then RC does the whole grounding job
> for you…
>
> I guess we really need some documentation about this (Dan??)…
>
>
>
> Hope this helps…
>
> antoine
>
>
> ------------------------------
>
> *From:* Svetlana Stenchikova [mailto:svetastenchikova at gmail.com]
> *Sent:* Wednesday, November 08, 2006 4:51 PM
> *To:* Antoine Raux
> *Cc:* ravenclaw-developers at cs.cmu.edu
> *Subject:* Re: [RavenclawDev 179] grounding_manager_configuration setting
> inconfig file of DM
>
>
>
> I am not sure which entry to look at.
>
> Could this be because I am binding multiple concepts?
> It is not a problem for me now as I can fix it by taking out
> grounding_manager_configuration from config.
> I am just concerned that I will deal with these issues with speech.
>
>
> >From a first glance at the helios log:
> WHERE CLASS
>
> confidence 0.62288
>
> WHERE IS CLASS
>
> confidence 0.28883
>
>
> In the dialog.log I get entries:
>
> I think this is for WHERE CLASS:
> ****************************************
>
> [GMX at 16:11:10.083] Grounding model /RavenCalendar/question_type_string
> [TYPE=concept_default;POLICY=expl;EM=greedy;EP=0.20]:
> Full state:
> grounded = false
> top_confidence = 0.622880
> updated = true
> Belief state: INACTIVE:0.00 CONFIDENT:0.62 UNCONFIDENT:0.38 GROUNDED:
> 0.00
> Action values (dumped below):
> ACCEPT:2.4576
> EXPL_CONF:0.6568
>
> Suggested action: ACCEPT
>
> [GMX at 16:11:10.083] Concept /RavenCalendar/the_event.event_type dumped
> below:
> CLASS|0.62
> [GMX at 16:11:10.083] Grounding model /RavenCalendar/the_event.event_type
> [TYPE=concept_default;POLICY=expl_impl;EM=greedy;EP=0.20]:
> Full state:
> grounded = false
> top_confidence = 0.622880
> updated = true
> Belief state: INACTIVE:0.00 CONFIDENT:0.62 UNCONFIDENT:0.38 GROUNDED:
> 0.00
> Action values (dumped below):
> ACCEPT:-2.1822
> EXPL_CONF:0.6568
> IMPL_CONF:1.8856
>
> Suggested action: IMPL_CONF
>
>
>
> *****************************************
>
> and this for WHERE IS CLASS
>
>
>
> [GMX at 16:11:13.314] Grounding model /RavenCalendar/question_type_string
> [TYPE=concept_default;POLICY=expl;EM=greedy;EP=0.20]:
> Full state:
> grounded = false
> top_confidence = 0.000000
> updated = false
> Belief state: INACTIVE:1.00 CONFIDENT:0.00 UNCONFIDENT:0.00 GROUNDED:
> 0.00
> Action values (dumped below):
> ACCEPT:10.0000
> EXPL_CONF: -
>
> Suggested action: ACCEPT
>
> [GMX at 16:11:13.314] Concept /RavenCalendar/the_event.event_type dumped
> below:
> <UNDEFINED>
> [GMX at 16:11:13.315] Grounding model /RavenCalendar/the_event.event_type
> [TYPE=concept_default;POLICY=expl_impl;EM=greedy;EP=0.20]:
> Full state:
> grounded = false
> top_confidence = 0.000000
>
>
> [GMX at 16:11:37.467] Concept /RavenCalendar/question_type_string dumped
> below:
> where|0.29
> [GMX at 16:11:37.467] Grounding model /RavenCalendar/question_type_string
> [TYPE=concept_default;POLICY=expl;EM=greedy;EP=0.20]:
> Full state:
> grounded = false
> top_confidence = 0.288830
> updated = true
> Belief state: INACTIVE:0.00 CONFIDENT:0.29 UNCONFIDENT:0.71 GROUNDED:
> 0.00
> Action values (dumped below):
> ACCEPT:-4.2234
> EXPL_CONF:5.6675
>
> Suggested action: EXPL_CONF
>
>
>
>
>
> On 11/8/06, *Antoine Raux* <antoine at cs.cmu.edu> wrote:
>
> Hi Svetlana,
>
>
>
> Have you looked at the confidence value for the input in both cases? This
> is one of the features that Helios sends to the DM for each input. It is
> possible that "where class" has a low confidence for some reason (e.g.
> segmented parse…) and RC rejects it. Normally if your grammar gives a full
> parse for "where class" that shouldn't happen (at least on typed input) but…
>
>
>
> antoine
>
>
> ------------------------------
>
> *From:* ravenclaw-developers-bounces at LOGANBERRY.srv.cs.cmu.edu [mailto:
> ravenclaw-developers-bounces at LOGANBERRY.srv.cs.cmu.edu] *On Behalf Of *Svetlana
> Stenchikova
> *Sent:* Wednesday, November 08, 2006 2:50 PM
> *To:* ravenclaw-developers at cs.cmu.edu
> *Subject:* [RavenclawDev 179] grounding_manager_configuration setting
> inconfig file of DM
>
>
>
>
> I have encountered an issue with binding a concept in DM depending on a
> user utterance.
>
> in my DM I have a grammar mapping. "@(...)[Question.where_event]>where,"
>
>
> The user can ask a question in multiple ways:
> where class
> where is class
> where is the class
> what is the location the class
> ..
>
> for all of these entries PHOENIX produces identical parse
>
> [Question] ( [where_event] ( WHERE ) )
> Features
> [EventTypeSpec] ( [event_type] ( [event_type_singular] ( MEETING ) ) )
>
> " } )
>
> When a configuration file passed to the DM in -config has an entry:
> grounding_manager_configuration =
> turns:default;concepts:default;beliefupdatingmodel:npu
>
>
> if a user types:
> "where is the class" DM does not bind the concepts, but if the user
> types
> "where class" DM binds the concepts.
>
> When the grounding_manager_configuration is taken out of DM, the binding
> succeeds for both utterances.
>
> Can this be explained? Do you think there may be a problem with my
> grammar or something in my DM?
> The grammar of the Phoenix parser should not matter because in all cases
> the parses are identical, right?
>
> thank you
>
> Svetlana
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.srv.cs.cmu.edu/pipermail/ravenclaw-developers/attachments/20061108/d9beebf3/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: RavenCalendarTask.gra
Type: application/octet-stream
Size: 21178 bytes
Desc: not available
Url : http://mailman.srv.cs.cmu.edu/pipermail/ravenclaw-developers/attachments/20061108/d9beebf3/RavenCalendarTask-0002.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: RavenCalendarTask.forms
Type: application/octet-stream
Size: 1384 bytes
Desc: not available
Url : http://mailman.srv.cs.cmu.edu/pipermail/ravenclaw-developers/attachments/20061108/d9beebf3/RavenCalendarTask-0003.obj
More information about the Ravenclaw-developers
mailing list