Does the GLXP scorecard need a new grading curve?

The web site Evadot recently published a comprehensive “team scorecard” ranking all the current teams participating in the Google Lunar X PRIZE. The scorecard lists 22 teams and their cumulative scores based on the following metrics:

  • Funding – 20 possible points – Measures how far along the teams are in their acquisition of funding based on their publicly stated estimated mission costs
  • Innovation – 10 possible points – Measures how much innovation is being used across the entire project. This includes new inventions and clever reuses of existing resources and technology
  • Social Savvy – 10 possible points – It’s 2010 and connecting with people will require the use of social networks and other avenues in order to gain mindshare of both influential thinkers and the “people on the street”
  • Connections – 10 possible points – Measures how connected are the people involved in the team leadership to the outside help and expertise they will need to execute their mission.
  • Progress – 10 possible points – Measures our perception of their progress to being able to launch.
  • Feeling – 10 possible points – Measures just our gut feeling about the team. Things like that look in a leader’s eyes when they speak.
  • Inspiration – 10 possible points – Measures the ability to inspire others.
  • Rover/Lander Completion – 10 possible points – How complete is the actual build.
  • Participatory Exploration – 10 possible points – Measures the teams involvement in involving others. People need to feel directly connected to the exploration of space in order to have a long term impact on their thinking.

It’s certainly a comprehensive examination of the teams, and Michael Doornbos deserves credit for putting it together. However, if the goal is to measure which teams are closest to winning the prize, the categories and their weighting should be reconsidered. Some comments:

1) While the scorecard weights funding more than any other category, it’s still not weighted heavily enough. Getting enough funding to carry out a mission is perhaps the most difficult aspect of the competition, given that none of the teams are independently wealthy or (with, perhaps, the exception of new team Rocket City Space Pioneers) have the backing of major corporations. You can have a great concept, an impressive social media strategy, and inspiration oozing out your virtual pores, but without money you’re never getting off the ground.

2) Similarly, hardware development should be weighted more: it’s a key differentiator between teams making serious progress towards going to the Moon versus those with flashy web sites and gorgeous illustrations, but nothing else.

3) Several of the other categories should be weighted less, or even combined or eliminated: social savvy, connections, feeling, and inspiration among them. Social media is nice to have, but beyond the requirements set forth by the competition it’s not essential. And some of the metrics are admittedly extremely subjective (see “feeling”).

4) Since progress is captured in other areas, such as funding and hardware development, having a separate progress category seems redundant.

A simplistic alternative would be to give one-third weight to funding, one-third to hardware development, and one-third distributed among the other categories. Even that, though, may underweight funding and hardware.

8 comments to Does the GLXP scorecard need a new grading curve?

  • Thanks for the comments on the scorecard Jeff. You’re a terrific resource

    I gave the creation of our scorecard serious consideration over a long period of time. Initially, I had a much more simple set of metrics like you suggest here, but after careful consideration, I concluded something different was in order:

    This competition is not just a technical competition. It’s not really about launching a cheap rover to the moon, making some tracks and sending video, and then collecting 20 million dollars. The spirit of the competition is to provide a springboard both in proving the commercial concept of space exploration, and putting space exploration back into the minds and hearts of people. If it was about the prize, there are certainly easier ways to get 20 million dollars. Sending rovers to other worlds is not new either.

    The competition is designed to be more than that.

    We’ve made a major mistake the last 35 years in turning space exploration from an inspiring and exciting set of events into a sterile set of engineering and funding goals. This is just not good enough. As a people, we deserve to be inspired by the exploration of space and together should want to push the limits of possibilities.

    While I believe that tracking it on technical merits shouldn’t be the point, it is certainly something we can do. We’ve had a lot of suggestions and we’ve decided to provide more visualisations to the large amount of data we’ve collected. As you’ll see later in the week(hopefully, might be early next week), we’re going to be including several visual representations that will include a weighted score on funding and engineering just like you suggest. Mostly we’ve had requests to see the score category breakdowns by team so we can see where improvements need to be made or where our data might be inaccurate :-)

    Thanks again for you comments. Sparking a discussion was really the point and I’m thrilled that you’re willing to discuss it out in the open.

    • Bennett Dawson

      Michael,

      This has to be the finest response to a critique that I’ve ever read. I predict big things in your future, with the whole world watching. Great job.

      Bennett

    • Nathan

      Hm, I do understand your point, but I think that the critique still stands. The development of space has been plagued by two seemingly opposite trends. As you rightly point out, reducing things to dry metrics of engineering and funding can strip space of all its inspirational elements, robbing it of the socio-cultural-psychological-etc. meaning that it really ought to have. That’s mistake #1.

      But mistake #2 has been to attempt to create a bunch of socio-cultural-psychological-etc. meaning in space-related activities which are ultimately nothing more than unrealistic fantasies (and occasionally outright scams). this is something that both public and private agencies have been guilty of. When these schemes inevitably fizzle out, then the cultural capital that they attempted to create is lost — if not outright reversed. So I would argue that a team which has masterful social outreach, inspirational messages, participatory involvement, etc. — but is ultimately based on nothing more than a bunch of technical and financial piffle — is not actually doing the development of space any favors. Quite the opposite, in fact.

      So I would suggest that a scorecard should first account for the technical and financial prospects of the team, and only then should it take into account the social factors. Personally, I’d suggest a formula that looks something like this:

      Technical Score “T” (0% – 100%): 50% financial, 20% rover, 20% overall completion, 10% industry connections
      Social Score “S” (0% – 100%):1/3rd social savvy, 1/3rd inspiration, 1/3rd participation
      Combined score: T + (T * S)

      This would ensure that teams which have no technical or financial merit are not erroneously awarded social merit.

  • That’s a pretty interesting formula and thought. I’ll give it a think.

    Thanks!

  • Phil Stooke

    Good point, Nathan. Something along those lines would be very sensible. And the whole scorecard idea is a great way to apply some pressure to teams who are keeping way too quiet about their activities. I can’t say I would rank the teams quite the same way as the graphs show it myself, but the basic idea is excellent.

  • [...] funding, social, and more). Jeff Foust over at the NewSpace Journal (and of Space Politics) posted a commentary/critique of the scorecard’s weighting, with an interesting and thorough response from Doornbos in the comments. Great discussion [...]

  • Just Ducky

    When my son was young he dreamed of becoming an astronaut. In recent years however, the scientific community has neglected the wonder and adventure of space exploration. Space exploration is more than crunching numbers, it requires a strong sense of courage and the ability to capture the hearts of the people and the next generation to maintain social and economic relevance. Just because there is an economic driver that doesn’t mean there will be a strong enough monetary force to overcome social resistance or even ambivalence. The goals of space exploration and commercialization require political and popular acceptance. Twitter, Face Book, business and politics all intertwine. It is the social savvy that is capturing the imagination, excitement, energy, and inspiration of the youth. It is these factors as well as the Lunar X Prize competition that will allow this endeavor to be more than a singular event. Right now it is about the competition, the race to the moon, what is the next step. To keep it real you need to involve the people by being socially relevant. Numbers are important but not motivating to my teenagers. My son and daughter are following the progress of a group of international college students, Part Time Scientist, because of their social savvy, inspiration, and their ability to engage with the people. Michael is telling the story of the people, which at one time, was NASA’s secret sauce. Space travel became boring over the years, however, space travel is alive again. We want to hear from all the teams, get to know the people and their stories, and be educated through frequent blogs. Currently, my kids know more about the lives of the Chilean minors than they know about their United State’s GLXP team members. We want to know and experience more through learning about the technology but just as importantly we want to be connected to the people.

Leave a Reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>