SELIS V2 / eval

Evaluation — SELIS V1 proposal draft

Proposal Idea

What the evaluated draft proposes, limited to the points that affect a criterion. The criterion each point affects is given in brackets.

Concept. A feasibility study of a "high-fidelity lunar surface simulation environment". The result is an interactive digital twin of one landing site at the lunar South Pole, used as a sandbox for analysis, planning and training. Users place assets in the scene and review terrain, illumination and operations in context. The product name SELIS is not used in the draft.

Stated objective and maturity. A proof of concept at TRL 3, starting from TRL 2, with "quantified accuracy, uncertainty and computing requirements". The planned end product is an installable, integrated prototype with user trials. (C1, C2)

Main features

Feature What the draft proposes Affects
Site terrain One 20 x 20 km South Pole site from at least two DEM products, imagery and context maps, in one coordinate system, with sources and uncertainty recorded. One shape-from-shading enhancement trial on a small tile. C1
Synthetic detail A separate 100 x 100 m robotics patch with procedural rocks and microtopography, labelled as synthetic. C1
Illumination and visibility SPICE geometry and terrain ray tracing: shadows, solar availability, Earth and relay line of sight; a six-hourly reference year; 18.6-year statistics at asset locations; Earthshine, first-bounce light and a 1D thermal calculation at selected points. C1, C2
Infrastructure placement Multi-objective ranking of candidate layouts for solar arrays, communication equipment and other assets. C1, C2
Robotics simulation A generic rover with lunar gravity, terrain contact and camera, depth and IMU outputs, at 30 frames per second on a desktop workstation. C1
VR/AR review Two headsets with controllers for immersive site review. The draft makes this the priority of WP4. C1, C2 (duplication), C3 (hardware cost)
Printed terrain model A small 3D-printed relief with projected data layers, as a showcase. C2, C3
Computer-vision example One vision model trained on synthetic images from the scene and tested on held-out simulated routes. C1 (scope), C2
Terrestrial example One limited example of use for industrial robotics on Earth. C2

Uniqueness claim. The integration of data sets and simulation functions in one environment. The draft names no competitor and does not use the four data classes as its headline; it keeps only "measured" against "synthetic". (C1, C2)

Users and market. No named end customer and no letter of interest. A survey of at least 30 organisations, 8 interviews and a user group of 3 organisations are planned inside the activity. Revenue scenarios after the study: EUR 0, 40,000 or 110,000 a year, from paid site packages and simulation services. No ESA programme or mission is named. (C2)

1. Marks

Criterion Weight Mark Weighted
C1 Technical quality and team 40% 45 18.0
C2 Prospects, programmatic fit, maturity 30% 35 10.5
C3 Management, cost, planning 25% 52 13.0
C4 Administrative, contract, IPR 5% 40 2.0
Overall 43.5

2. C1 — Technical quality and team (45)

Weaknesses, most damaging first:

  1. Section 1.4.1 is not a state of the art. It names no competitor, product or reference and describes the bidder's own approach in the future tense. research/findings/A-competitors.md lists about 20 tools, including DUST, CNES Spaceship FR, ADTLAS, Moon Trek and PANGU.
  2. The scope is far beyond 2.36 full-time equivalents in 12 months. It covers multi-DEM terrain, illumination with 18.6-year statistics, Earthshine, thermal, layout optimisation, rover physics, VR and AR, a printed model, a trained vision model, a terrestrial example and a 30-organisation survey. The critical functions are not identified.
  3. The end point reads above TRL 3. "Installable proof of concept", "feature-complete research release" and an "integrated prototype" with user trials describe an integrated system. Section 1.3 gives no evidence for start TRL 2; the existing stereo DEMs and PV demos are not mentioned.
  4. Requirements have no accuracy thresholds. The only numbers are sizes and 30 frames per second. Requirements 5 and 6 are a facility and a deliverable. Verification is "review" and "inspect". No customer justifies any requirement.
  5. Team and company are empty. Section 1.8 is a placeholder and every person is "[NAME]". The Simulation Engineer, 1,680 hours, is not a person recorded in this repository.
  6. Work package descriptions are too thin to judge effort. Two or three sentences cover 1,032 hours, and outputs carry no identifiers.

Strengths:

3. C2 — Prospects, programmatic fit and maturity (35)

Weaknesses, most damaging first:

  1. Duplication is not addressed, and the draft prioritises the overlapping part. WP4 makes VR/AR the priority. Findings A say an XR or training environment duplicates ESA-funded ADTLAS and the ESA XR Lab, and recommend reducing the 3D environment to a viewer.
  2. The one differentiator is nearly gone. The four data classes with per-cell provenance and confidence appear only in passing; "derived" never appears.
  3. No ESA programme or mission is named. Argonaut, MAGPIE and LUVMI-M are in research/findings/C-users-and-needs.md but not in the draft. Section 1.6.1 defers the question to WP1, after award.
  4. No end customer. No name, no letter, and the Lithuanian lead is absent. The draft says recruitment targets are "not commitments".
  5. The business case is weak relative to EUR 153,220. Revenue scenarios of EUR 0, 40,000 and 110,000 a year have no sources. Most of section 1.6.2 is an outreach work plan, and the terrestrial robotics fallback dilutes the space link.

Strength:

4. C3 — Management, organisation, cost and planning (52)

Weaknesses, most damaging first:

  1. EUR 19,000 of generic hardware (workstation, headsets, projector) is 12% of the price and is excluded by the call. The draft admits eligibility is unconfirmed and lists the equipment as deliverable HW2.
  2. Time shares are not credible. The Lead is at 100% while being owner-manager, project manager, main developer and manager of four work packages. Rates are identical for all roles and not justified.
  3. The Gantt is a dot table with no dependencies, critical path or contingency. The vision-model experiment runs to Month 12 while final drafts are due two weeks before the final review.
  4. Six work packages, four of them under EUR 21,000, against the target of four to five.

Strengths:

5. C4 — Administrative, contract and IPR compliance (40)

Weaknesses:

  1. The cover letter is blank apart from title and company, and all forms are unsigned.
  2. The AI statement in section 2.4.3 says the proposal was generated with OpenAI Codex. That statement belongs in cover letter point 9, which is empty, and the two must agree.
  3. The background IP table holds placeholders, and section 4.1.1 mentions an "RFI commercial licence".
  4. Notes to the author remain throughout ("confirm before submission", "revise the scope rather than re-label it").

6. Likely negotiation questions

  1. How does this differ from ADTLAS, DUST and Spaceship FR?
  2. Which functions are critical, and what measured data proves each?
  3. Who is the Simulation Engineer, and how is each person engaged and paid?
  4. Why should ESA pay for a workstation and headsets?
  5. Who will use the result, and on which mission?

7. Top five fixes, by expected gain in the weighted mark

  1. Cut the scope to the traceable site package with illumination time series. Reduce VR/AR, the printed model and the vision model to a viewer, or drop them. Rewrite section 1.6.1 around complementarity and named missions.
  2. Rewrite section 1.4.1 from research/findings/A-competitors.md, with sources.
  3. Name the team, correct the credentials, and add the Lithuanian customer with a letter of interest.
  4. Remove or re-base the hardware, and reduce the 100% time shares.
  5. Give each requirement a threshold and a measured reference, and state the start-TRL evidence.

8. Comparing later evaluations

Save each later evaluation in eval/ with the same headings and the same marks table. Compare the four criterion marks and the overall mark first, then check which weaknesses listed here are closed, which remain and which are new.