Closed Bug 770460 Opened 12 years ago Closed 12 years ago

Test suite names in dev.tree-management & graphs.mozilla.org are confusing for developers

Categories

(Testing :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: emorley, Unassigned)

Details

Attachments

(1 file, 3 obsolete files)

This isn't the first time this kind of situation has arisen:

(2012-07-03 10:53) mattwoodrow: edmorley: Do you have any idea what Android Txul shows up as on tbpl?
(2012-07-03 10:53) mattwoodrow: I have emails about a regression there, but I don't see it
(2012-07-03 10:59) edmorley: mattwoodrow: sorry I can't seem to find which test suit it's under, jmaher should know
(2012-07-03 10:59) edmorley: mattwoodrow: the suite/test names are a bit of a muddle tbh
(2012-07-03 11:01) edmorley: mattwoodrow: ah although bug 397952 seems to give a clue

Before I found bug 397952, I had:
* searched in mozilla-central
* looked at the sub-testsuites listed in TBPL in the summary box when you click on talos test runs
* tried looking in http://hg.mozilla.org/build/talos/file/tip/talos/run_tests.py and poking around the rest of the repo over hgweb (I don't have it checked out so couldn't text-search)
* searched Google/MDN for an answer

But yet the only thing I could find explaining what Txul might be, was the attachment name in bug 397952.

If we are expecting developers to care about the dev.tree-management mails they get sent, we really need a better story than this. If they can't even find the test to run it locally, then they are not going to bother checking regressions.

Whether it's making sure that everywhere uses the same test names (ie fix outdated references to Txul if it's now called twinopen as bug 397952 would imply), or just creating an MDN page: "List of all testsuites (and nicknames used) and how to run each individually" I don't really mind :-)
Does this page clear up any confusion: https://wiki.mozilla.org/Buildbot/Talos ?  If not, lets make sure it does.

If it does, then we need to figure out what next steps we can take to make the dev.tree-management emails better, likewise the tbpl information.
(In reply to Joel Maher (:jmaher) from comment #1)
> Does this page clear up any confusion:
> https://wiki.mozilla.org/Buildbot/Talos ?  If not, lets make sure it does.
> 
> If it does, then we need to figure out what next steps we can take to make
> the dev.tree-management emails better, likewise the tbpl information.

I'll add the graphserver names for these tests soon from: http://hg.mozilla.org/graphs/raw-file/tip/sql/data.sql
So the short story is: Things are confusing because they're confusing

The longer story is: ...but they don't need to be, this is a path we've chosen

I've made a table that describes the information workflow from buildbot -> talos -> graphserver + TBPL . The graphserver part still isn't quite right: I'll work on this going forward, but wanted to put up what I have.

In short:
- buildbot specifies a suite
- ... which is translated into PerfConfigurator directives
- ...... which are used to run talos tests
- ...... which with marked up names report to graphserver
- ...... which are mapped to graphserver test names and counters for the test
- ...... whose links are posted to TBPL using TinderboxPrint
- ... which also post to TBPL for the whole suite
- ... which are translated into some number

I'll put together a blog post and a bit on https://wiki.mozilla.org/Buildbot/Talos when I'm done(ish).  Suffice it to say that this would be hard to untangle even if we were doing a good job of A. documenting what we're doing; and B. making it easy to consume our software.
Attachment #640914 - Attachment is patch: false
Wow! Thank you for looking into this Jeff :-)
(In reply to Ed Morley [:edmorley] from comment #4)
> Wow! Thank you for looking into this Jeff :-)

Its turtles all the way down ;)  I'll endeavor to get this wrapped up this week, but I have talos code to write for now
nice work!  We face a very similar problem with WOO, as we are correlating data from different sources, which unfortunately do not use a consistent naming convention for things like test names.
Attached file WIP (obsolete) —
There is still a lot to do here: counters, paint, nochrome,etc. This gets the graphserver names slightly better but...its still not wonderful
Attachment #640914 - Attachment is obsolete: true
Attachment #641530 - Attachment is patch: false
Attachment #641530 - Attachment mime type: text/plain → text/html
Attached file next version (obsolete) —
This is getting closer.  There are still some, shall we say?, wrongities in it
Attached patch accurate versionSplinter Review
This ain't perfect, but ABICT it is at least accurate, if somewhat incomplete, and I need to move on to other things for now so I'm leaving this one here.
Attachment #641530 - Attachment is obsolete: true
Attachment #645080 - Attachment is obsolete: true
(In reply to Jeff Hammel [:jhammel] from comment #9)
> Created attachment 645490 [details] [diff] [review]
> accurate version
> 
> This ain't perfect, but ABICT it is at least accurate, if somewhat
> incomplete, and I need to move on to other things for now so I'm leaving
> this one here.

Future versions will be at http://k0s.org/mozilla/talos/talosnames.html or somewhere else sensible and linkable to
Thank you :-D
I've also updated https://wiki.mozilla.org/Buildbot/Talos with as many of the perfomatic names as I could.  The page is somewhat misleading, at many parts, and http://k0s.org/mozilla/talos/talosnames.html is canonical if confusing.

See also http://k0s.org/mozilla/blog/20120724135349 for how I gather this information.

There is a lot of low-hanging fruit we can do to make this easier, but I'm wondering how good of a stop-gap :jmaher's and my work here is and if it makes sense to close the bug and ticket subsequent issues?
Oh, and forgot to mention the code is at http://k0s.org/mozilla/hg/talosnames if anyone wants to play with it
So I have a live instance at http://k0s.org:8080/

What this does:
- gathers the suite names and CLI parameters from buildbot-configs
- determines the TBPL names from the buildbot suite names
- determines the talos test names by running PerfConfigurator
- determines the graphserver names by looking at data.sql
- has a TBPL-centric view

What this doesn't do but easily could:
- dynamically reload the page
- display the graphserver magic id numbers

What this doesn't do but could with work:
- currently, things are targetted at the lastest resources (buildbot-configs, etc). There is no way of knowing if there are the ones in production (AFAIK).  

I'm happy to take more feature requests if this is useful.  I really think the ideal system is one where you don't need a complicated website to sort all of this stuff out, but until that day...

Should we close this bug?
That's great - thank you for your work on this Jeff! :-D

If you wouldn't mind adding a link to http://k0s.org:8080/ to wherever you feel is most appropriate on https://wiki.mozilla.org/Buildbot/Talos, I think we can then close this out :-)
I've added at https://wiki.mozilla.org/Buildbot/Talos#Talos_Tests . As said, this isn't really ideal for the reasons above, but I'm more than happy to tackle the remaining side issues if they're priority for anyone.
Closing.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: