Evaluation — SELIS V1 proposal draft
- Evaluated: 14 November 2026.
- Draft:
Luminous_PECS_Draft-1.docxwithLuminous_PSSA1_Draft-1.xlsx,Luminous_PSSA2_Draft-1.xlsxandLuminous_PSSA8_Draft-1.xlsx. The draft files are not in this repository. - Draft content: "Feasibility study of a high-resolution multidimensional lunar surface simulation environment". 12 months, EUR 153,220, TRL 2 to 3, six work packages, 3,960 hours, three roles.
- Method:
evaluator-rubric.md, role of a sceptical ESA Tender Evaluation Board member. Checked againstcompliance-matrix.md,pecs-playbook.mdandresearch/findings/. - Evaluator: AI (Claude). Marks are a judgement on the rubric's internal scale, not an ESA result.
Proposal Idea
What the evaluated draft proposes, limited to the points that affect a criterion. The criterion each point affects is given in brackets.
Concept. A feasibility study of a "high-fidelity lunar surface simulation environment". The result is an interactive digital twin of one landing site at the lunar South Pole, used as a sandbox for analysis, planning and training. Users place assets in the scene and review terrain, illumination and operations in context. The product name SELIS is not used in the draft.
Stated objective and maturity. A proof of concept at TRL 3, starting from TRL 2, with "quantified accuracy, uncertainty and computing requirements". The planned end product is an installable, integrated prototype with user trials. (C1, C2)
Main features
| Feature | What the draft proposes | Affects |
|---|---|---|
| Site terrain | One 20 x 20 km South Pole site from at least two DEM products, imagery and context maps, in one coordinate system, with sources and uncertainty recorded. One shape-from-shading enhancement trial on a small tile. | C1 |
| Synthetic detail | A separate 100 x 100 m robotics patch with procedural rocks and microtopography, labelled as synthetic. | C1 |
| Illumination and visibility | SPICE geometry and terrain ray tracing: shadows, solar availability, Earth and relay line of sight; a six-hourly reference year; 18.6-year statistics at asset locations; Earthshine, first-bounce light and a 1D thermal calculation at selected points. | C1, C2 |
| Infrastructure placement | Multi-objective ranking of candidate layouts for solar arrays, communication equipment and other assets. | C1, C2 |
| Robotics simulation | A generic rover with lunar gravity, terrain contact and camera, depth and IMU outputs, at 30 frames per second on a desktop workstation. | C1 |
| VR/AR review | Two headsets with controllers for immersive site review. The draft makes this the priority of WP4. | C1, C2 (duplication), C3 (hardware cost) |
| Printed terrain model | A small 3D-printed relief with projected data layers, as a showcase. | C2, C3 |
| Computer-vision example | One vision model trained on synthetic images from the scene and tested on held-out simulated routes. | C1 (scope), C2 |
| Terrestrial example | One limited example of use for industrial robotics on Earth. | C2 |
Uniqueness claim. The integration of data sets and simulation functions in one environment. The draft names no competitor and does not use the four data classes as its headline; it keeps only "measured" against "synthetic". (C1, C2)
Users and market. No named end customer and no letter of interest. A survey of at least 30 organisations, 8 interviews and a user group of 3 organisations are planned inside the activity. Revenue scenarios after the study: EUR 0, 40,000 or 110,000 a year, from paid site packages and simulation services. No ESA programme or mission is named. (C2)
1. Marks
| Criterion | Weight | Mark | Weighted |
|---|---|---|---|
| C1 Technical quality and team | 40% | 45 | 18.0 |
| C2 Prospects, programmatic fit, maturity | 30% | 35 | 10.5 |
| C3 Management, cost, planning | 25% | 52 | 13.0 |
| C4 Administrative, contract, IPR | 5% | 40 | 2.0 |
| Overall | 43.5 |
- Result: fail against the pass mark of 55 and against the working target of 70.
- C2 is under 40, which the rubric treats as a fail on its own.
- With every placeholder filled truthfully (names, CVs, company background, cover letter) and no other change, the estimate is about 48. The gap is in the content, not the blanks.
2. C1 — Technical quality and team (45)
Weaknesses, most damaging first:
- Section 1.4.1 is not a state of the art. It names
no competitor, product or reference and describes the bidder's own
approach in the future tense.
research/findings/A-competitors.mdlists about 20 tools, including DUST, CNES Spaceship FR, ADTLAS, Moon Trek and PANGU. - The scope is far beyond 2.36 full-time equivalents in 12 months. It covers multi-DEM terrain, illumination with 18.6-year statistics, Earthshine, thermal, layout optimisation, rover physics, VR and AR, a printed model, a trained vision model, a terrestrial example and a 30-organisation survey. The critical functions are not identified.
- The end point reads above TRL 3. "Installable proof of concept", "feature-complete research release" and an "integrated prototype" with user trials describe an integrated system. Section 1.3 gives no evidence for start TRL 2; the existing stereo DEMs and PV demos are not mentioned.
- Requirements have no accuracy thresholds. The only numbers are sizes and 30 frames per second. Requirements 5 and 6 are a facility and a deliverable. Verification is "review" and "inspect". No customer justifies any requirement.
- Team and company are empty. Section 1.8 is a placeholder and every person is "[NAME]". The Simulation Engineer, 1,680 hours, is not a person recorded in this repository.
- Work package descriptions are too thin to judge effort. Two or three sentences cover 1,032 hours, and outputs carry no identifiers.
Strengths:
- Section 1.4.4 is specific, numerate and honest about limits; its data-volume arithmetic is correct.
- Exclusions are stated in every work package description.
3. C2 — Prospects, programmatic fit and maturity (35)
Weaknesses, most damaging first:
- Duplication is not addressed, and the draft prioritises the overlapping part. WP4 makes VR/AR the priority. Findings A say an XR or training environment duplicates ESA-funded ADTLAS and the ESA XR Lab, and recommend reducing the 3D environment to a viewer.
- The one differentiator is nearly gone. The four data classes with per-cell provenance and confidence appear only in passing; "derived" never appears.
- No ESA programme or mission is named. Argonaut,
MAGPIE and LUVMI-M are in
research/findings/C-users-and-needs.mdbut not in the draft. Section 1.6.1 defers the question to WP1, after award. - No end customer. No name, no letter, and the Lithuanian lead is absent. The draft says recruitment targets are "not commitments".
- The business case is weak relative to EUR 153,220. Revenue scenarios of EUR 0, 40,000 and 110,000 a year have no sources. Most of section 1.6.2 is an outreach work plan, and the terrestrial robotics fallback dilutes the space link.
Strength:
- No ESA or JAXA endorsement is claimed.
4. C3 — Management, organisation, cost and planning (52)
Weaknesses, most damaging first:
- EUR 19,000 of generic hardware (workstation, headsets, projector) is 12% of the price and is excluded by the call. The draft admits eligibility is unconfirmed and lists the equipment as deliverable HW2.
- Time shares are not credible. The Lead is at 100% while being owner-manager, project manager, main developer and manager of four work packages. Rates are identical for all roles and not justified.
- The Gantt is a dot table with no dependencies, critical path or contingency. The vision-model experiment runs to Month 12 while final drafts are due two weeks before the final review.
- Six work packages, four of them under EUR 21,000, against the target of four to five.
Strengths:
- Every total reconciles: 3,960 hours and EUR 153,220 across the text and all three forms; payments and advance offsets sum correctly; the final net payment is exactly 10%.
- Management effort is 7.9% of hours.
5. C4 — Administrative, contract and IPR compliance (40)
Weaknesses:
- The cover letter is blank apart from title and company, and all forms are unsigned.
- The AI statement in section 2.4.3 says the proposal was generated with OpenAI Codex. That statement belongs in cover letter point 9, which is empty, and the two must agree.
- The background IP table holds placeholders, and section 4.1.1 mentions an "RFI commercial licence".
- Notes to the author remain throughout ("confirm before submission", "revise the scope rather than re-label it").
6. Likely negotiation questions
- How does this differ from ADTLAS, DUST and Spaceship FR?
- Which functions are critical, and what measured data proves each?
- Who is the Simulation Engineer, and how is each person engaged and paid?
- Why should ESA pay for a workstation and headsets?
- Who will use the result, and on which mission?
7. Top five fixes, by expected gain in the weighted mark
- Cut the scope to the traceable site package with illumination time series. Reduce VR/AR, the printed model and the vision model to a viewer, or drop them. Rewrite section 1.6.1 around complementarity and named missions.
- Rewrite section 1.4.1 from
research/findings/A-competitors.md, with sources. - Name the team, correct the credentials, and add the Lithuanian customer with a letter of interest.
- Remove or re-base the hardware, and reduce the 100% time shares.
- Give each requirement a threshold and a measured reference, and state the start-TRL evidence.
8. Comparing later evaluations
Save each later evaluation in eval/ with the same
headings and the same marks table. Compare the four criterion marks and
the overall mark first, then check which weaknesses listed here are
closed, which remain and which are new.