770460 - Test suite names in dev.tree-management & graphs.mozilla.org are confusing for developers

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Ed Morley [:emorley]]

This isn't the first time this kind of situation has arisen:

(2012-07-03 10:53) mattwoodrow: edmorley: Do you have any idea what Android Txul shows up as on tbpl?
(2012-07-03 10:53) mattwoodrow: I have emails about a regression there, but I don't see it
(2012-07-03 10:59) edmorley: mattwoodrow: sorry I can't seem to find which test suit it's under, jmaher should know
(2012-07-03 10:59) edmorley: mattwoodrow: the suite/test names are a bit of a muddle tbh
(2012-07-03 11:01) edmorley: mattwoodrow: ah although bug 397952 seems to give a clue

Before I found bug 397952, I had:
* searched in mozilla-central
* looked at the sub-testsuites listed in TBPL in the summary box when you click on talos test runs
* tried looking in http://hg.mozilla.org/build/talos/file/tip/talos/run_tests.py and poking around the rest of the repo over hgweb (I don't have it checked out so couldn't text-search)
* searched Google/MDN for an answer

But yet the only thing I could find explaining what Txul might be, was the attachment name in bug 397952.

If we are expecting developers to care about the dev.tree-management mails they get sent, we really need a better story than this. If they can't even find the test to run it locally, then they are not going to bother checking regressions.

Whether it's making sure that everywhere uses the same test names (ie fix outdated references to Txul if it's now called twinopen as bug 397952 would imply), or just creating an MDN page: "List of all testsuites (and nicknames used) and how to run each individually" I don't really mind :-)

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Does this page clear up any confusion: https://wiki.mozilla.org/Buildbot/Talos ?  If not, lets make sure it does.

If it does, then we need to figure out what next steps we can take to make the dev.tree-management emails better, likewise the tbpl information.

Comment 2

[Jeff Hammel]

(In reply to Joel Maher (:jmaher) from comment #1)
> Does this page clear up any confusion:
> https://wiki.mozilla.org/Buildbot/Talos ?  If not, lets make sure it does.
> 
> If it does, then we need to figure out what next steps we can take to make
> the dev.tree-management emails better, likewise the tbpl information.

I'll add the graphserver names for these tests soon from: http://hg.mozilla.org/graphs/raw-file/tip/sql/data.sql

Comment 3

[Jeff Hammel]

Attachment: table describing how things are named such a way

So the short story is: Things are confusing because they're confusing

The longer story is: ...but they don't need to be, this is a path we've chosen

I've made a table that describes the information workflow from buildbot -> talos -> graphserver + TBPL . The graphserver part still isn't quite right: I'll work on this going forward, but wanted to put up what I have.

In short:
- buildbot specifies a suite
- ... which is translated into PerfConfigurator directives
- ...... which are used to run talos tests
- ...... which with marked up names report to graphserver
- ...... which are mapped to graphserver test names and counters for the test
- ...... whose links are posted to TBPL using TinderboxPrint
- ... which also post to TBPL for the whole suite
- ... which are translated into some number

I'll put together a blog post and a bit on https://wiki.mozilla.org/Buildbot/Talos when I'm done(ish).  Suffice it to say that this would be hard to untangle even if we were doing a good job of A. documenting what we're doing; and B. making it easy to consume our software.

Comment 4

[Ed Morley [:emorley]]

Wow! Thank you for looking into this Jeff :-)

Comment 5

[Jeff Hammel]

(In reply to Ed Morley [:edmorley] from comment #4)
> Wow! Thank you for looking into this Jeff :-)

Its turtles all the way down ;)  I'll endeavor to get this wrapped up this week, but I have talos code to write for now

Comment 6

[Jonathan Griffin (:jgriffin)]

nice work!  We face a very similar problem with WOO, as we are correlating data from different sources, which unfortunately do not use a consistent naming convention for things like test names.

Comment 7

[Jeff Hammel]

Attachment: WIP

There is still a lot to do here: counters, paint, nochrome,etc. This gets the graphserver names slightly better but...its still not wonderful

Comment 8

[Jeff Hammel]

Attachment: next version

This is getting closer.  There are still some, shall we say?, wrongities in it

Comment 9

[Jeff Hammel]

Attachment: accurate version

This ain't perfect, but ABICT it is at least accurate, if somewhat incomplete, and I need to move on to other things for now so I'm leaving this one here.

Comment 10

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #9)
> Created attachment 645490 [details] [diff] [review]
> accurate version
> 
> This ain't perfect, but ABICT it is at least accurate, if somewhat
> incomplete, and I need to move on to other things for now so I'm leaving
> this one here.

Future versions will be at http://k0s.org/mozilla/talos/talosnames.html or somewhere else sensible and linkable to

Comment 11

[Ed Morley [:emorley]]

Thank you :-D

Comment 12

[Jeff Hammel]

I've also updated https://wiki.mozilla.org/Buildbot/Talos with as many of the perfomatic names as I could.  The page is somewhat misleading, at many parts, and http://k0s.org/mozilla/talos/talosnames.html is canonical if confusing.

See also http://k0s.org/mozilla/blog/20120724135349 for how I gather this information.

There is a lot of low-hanging fruit we can do to make this easier, but I'm wondering how good of a stop-gap :jmaher's and my work here is and if it makes sense to close the bug and ticket subsequent issues?

Comment 13

[Jeff Hammel]

Oh, and forgot to mention the code is at http://k0s.org/mozilla/hg/talosnames if anyone wants to play with it

Comment 14

[Jeff Hammel]

So I have a live instance at http://k0s.org:8080/

What this does:
- gathers the suite names and CLI parameters from buildbot-configs
- determines the TBPL names from the buildbot suite names
- determines the talos test names by running PerfConfigurator
- determines the graphserver names by looking at data.sql
- has a TBPL-centric view

What this doesn't do but easily could:
- dynamically reload the page
- display the graphserver magic id numbers

What this doesn't do but could with work:
- currently, things are targetted at the lastest resources (buildbot-configs, etc). There is no way of knowing if there are the ones in production (AFAIK).  

I'm happy to take more feature requests if this is useful.  I really think the ideal system is one where you don't need a complicated website to sort all of this stuff out, but until that day...

Should we close this bug?

Comment 15

[Ed Morley [:emorley]]

That's great - thank you for your work on this Jeff! :-D

If you wouldn't mind adding a link to http://k0s.org:8080/ to wherever you feel is most appropriate on https://wiki.mozilla.org/Buildbot/Talos, I think we can then close this out :-)

Comment 16

[Jeff Hammel]

I've added at https://wiki.mozilla.org/Buildbot/Talos#Talos_Tests . As said, this isn't really ideal for the reasons above, but I'm more than happy to tackle the remaining side issues if they're priority for anyone.

Comment 17

[Jeff Hammel]

Closing.