1175269 - Add MathML roles/subroles for NSAccessibility

Status: RESOLVED FIXED

(Core :: Disability Access APIs, defect)

Description

[Frédéric Wang (:fredw)]

This is the part of bug 1001641 that only deals with roles. Let's see what VoiceOver can do with that.

Comment 1

[Frédéric Wang (:fredw)]

Attachment: Patch

https://treeherder.mozilla.org/#/jobs?repo=try&revision=23b07f7a76f9

Comment 2

[Frédéric Wang (:fredw)]

Attachment: Patch V2 - try to get the fence/separator properties

Apple sometimes refines AXMathOperator to AXMathFenceOperator or AXMathSeparatorOperator. I'm not sure how it is used by ATs and whether it's really important to have it now, but this second version tries to get these info from the MathML layout code.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=282ebdbb198a

But actually, what I suspect Jonathan wanted to do was to create AXMathFenceOperator or AXMathSeparatorOperator accessibles for the open/close/separators of the <mfenced> element. This will probably be harder to do since the MathML rendering code only creates nsMathMLChar objects, that are not currently exposed to the outside. I'll open a separate bug for that.

Comment 3

[Frédéric Wang (:fredw)]

Attachment: Basic testcase for mac roles

Here is a testcase for basic MathML expressions with what I believe the role/subroles should look like, based on what I see in the WebKit code in http://trac.webkit.org/browser/trunk/Source/WebCore/accessibility/mac/WebAccessibilityObjectWrapperMac.mm#L2125

Comment 4

[Frédéric Wang (:fredw)]

@marco: Can you please check whether the roles in the accessible trees are consistent in the testcase for the two Firefox try builds VS Safari? Probably the 3 last tests may render differently and the <mfenced> case will not work well in any platforms, but I believe we can ignore these for now.

Also, it would be good to see whether that improves how VoiceOver reads MathML in Gecko, although we may not have exactly the same result as VoiceOver because the patch does not include accessible relations.

Comment 5

[Frédéric Wang (:fredw)]

Attachment: Basic testcase for mac roles

Comment 6

[Frédéric Wang (:fredw)]

(In reply to Frédéric Wang (:fredw) from comment #2)
> But actually, what I suspect Jonathan wanted to do was to create
> AXMathFenceOperator or AXMathSeparatorOperator accessibles for the
> open/close/separators of the <mfenced> element. This will probably be harder
> to do since the MathML rendering code only creates nsMathMLChar objects,
> that are not currently exposed to the outside. I'll open a separate bug for
> that.

So this is bug 1175747. The test for WebKit Mac (which I think is currently broken) is http://trac.webkit.org/browser/trunk/LayoutTests/platform/mac/accessibility/mathml-elements.html

<math id="fenced">
  <mfenced open="{" close="}" separators=",,"><mi>2</mi><mi>a</mi><mi>e</mi></mfenced>
</math>

var fenced = accessibilityController.accessibleElementById("fenced").childAtIndex(0);
shouldBe("fenced.role", "'AXRole: AXGroup'");
shouldBe("fenced.subrole", "'AXSubrole: AXMathFenced'");
shouldBe("fenced.stringAttributeValue('AXMathFencedOpen')", "'{'");
shouldBe("fenced.stringAttributeValue('AXMathFencedClose')", "'}'");
var fenceValues = new Array("{", "2", ",", "a", ",", "e", "}");
for (var k = 0; k < fenceValues.length; k++) {
   var child = fenced.childAtIndex(k);
   shouldBe("child.childAtIndex(0).stringValue", "'AXValue: " + fenceValues[k] + "'");
}      

I'm not sure what will be exactly the subroles, but as I understand '{' and '}' are AXMathFenceOperator and ',' is AXMathSeparatorOperator.

Comment 7

[Marco Zehe (:MarcoZ)]

I just tried the try-server build, and VoiceOver doesn't see any of the MathML stuff at all. But I don't know why that might be, because at the same time, the table itself isn't rendered to VoiceOver correctly yet because we are not exposing tables correctly yet in general. I tried the second build, for patch v2. Does it make sense at all to test the one for the first patch?

Comment 8

[Frédéric Wang (:fredw)]

Attachment: Basic testcase for mac roles (with lists)

Comment 9

[alexander :surkov (:asurkov)]

Comment on attachment 8623445 [details] [diff] [review]
Patch

Review of attachment 8623445 [details] [diff] [review]:
-----------------------------------------------------------------

looks good, I saw it working with relations patch applied, so rs=me

Comment 10

[Frédéric Wang (:fredw)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=23b07f7a76f9

Comment 11

[Frédéric Wang (:fredw)]

(In reply to Marco Zehe (:MarcoZ) from comment #7)
> But I don't know why that might be, because at the same
> time, the table itself isn't rendered to VoiceOver correctly yet because we
> are not exposing tables correctly yet in general.

We also saw that bug with Alex yesterday. Is there a bug entry for that?

Comment 12

[Marco Zehe (:MarcoZ)]

(In reply to Frédéric Wang (:fredw) from comment #11)
> (In reply to Marco Zehe (:MarcoZ) from comment #7)
> > But I don't know why that might be, because at the same
> > time, the table itself isn't rendered to VoiceOver correctly yet because we
> > are not exposing tables correctly yet in general.
> 
> We also saw that bug with Alex yesterday. Is there a bug entry for that?

Yes, bug 744790.

Comment 13

[Frédéric Wang (:fredw)]

Attachment: Patch V3 - do not expose the AXMathFence subrole

Per bug 1001641 comment 15, it's best not to expose the AXMathFence role for now.

Comment 14

[alexander :surkov (:asurkov)]

Comment on attachment 8624957 [details] [diff] [review]
Patch V3 - do not expose the AXMathFence subrole

Review of attachment 8624957 [details] [diff] [review]:
-----------------------------------------------------------------

fine with me

Comment 15

[Frédéric Wang (:fredw)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2e65f422018e

Comment 16

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8ce4a0a6691b

Comment 17

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/8ce4a0a6691b