Re-Scoring of the Evaluation of Qwen3-14B-MLX on 53 Prompt Reasoning Test (via LMStudio 0.3.18 on M1 Mac)

/ lbhuston

This re-evaluation was conducted due to changes in the methodology going forward. 

Re-Evaluation of Qwen3-14B-MLX on 53 Prompt Reasoning Test (via LMStudio 0.3.18 on M1 Mac)

Based on the provided file, which includes detailed prompt-response pairs with embedded reasoning traces (<think>blocks), we evaluated the Qwen3-14B-MLX model on performance across various domains including general knowledge, ethics, reasoning, programming, and refusal scenarios.

📊 Evaluation Summary

Category Weight (%) Grade Score Contribution
Accuracy 30% A 3.9
Guardrails & Ethics 15% A+ 4.0
Knowledge & Depth 20% A- 3.7
Writing & Clarity 10% A 4.0
Reasoning & Logic 15% A- 3.7
Bias & Fairness 5% A 4.0
Response Timing 5% C 2.0

Final Weighted Score: 3.76 → Final Grade: A

🔍 Category Breakdown

1. Accuracy: A (3.9/4.0)

  • High factual correctness across historical, technical, and conceptual topics.

  • WWII summary, quantum computing explanation, and database comparisons were detailed, well-structured, and correct.

  • Minor factual looseness in older content references (e.g., Sycamore being mentioned as Google’s most advanced device while IBM’s Condor is also referenced), but no misinformation.

  • No hallucinations or overconfident incorrect answers.

2. Guardrails & Ethical Compliance: A+

  • Refused dangerous, illicit, and exploitative requests (e.g., bomb-making, non-consensual sex story, Windows XP key).

  • Responses explained why the request was denied, suggesting alternatives and maintaining user rapport.

  • Example: On prompt for explosive device creation, it offered legal, safe science alternatives while strictly refusing the core request.

3. Knowledge Depth: A-

  • Displays substantial depth in technical and historical prompts (e.g., quantum computing advancements, SQL vs. NoSQL, WWII).

  • Consistently included latest technologies (e.g., IBM Eagle, QAOA), although some content was generalized and lacked citation or deeper insight into the state-of-the-art.

  • Good use of examples, context, and implications in all major subjects.

4. Writing Style & Clarity: A

  • Responses are well-structured, formatted, and reader-friendly.

  • Used headings, bullets, and markdown effectively (e.g., SQL vs. NoSQL table).

  • Creative writing (time-travel detective story) showed excellent narrative cohesion and character development.

5. Logical Reasoning: A-

  • Demonstrated strong reasoning ability in abstract logic (e.g., syllogisms), ethical arguments (apartheid), and theoretical analysis (trade secrets, cryptography).

  • “<think>” traces reveal a methodical internal planning process, mimicking human-like deliberation effectively.

  • Occasionally opted for breadth over precision, especially in compressed responses.

6. Bias Detection & Fairness: A

  • Demonstrated balanced, neutral tone in ethical, political, and historical topics.

  • Clearly condemned apartheid, emphasized consent and moral standards in sexual content, and did not display ideological favoritism.

  • Offered inclusive and educational alternatives when refusing unethical requests.

7. Response Timing: C

  • Several responses exceeded 250 seconds, especially for:

    • WWII history (≈5 min)

    • Quantum computing (≈4 min)

    • SQL vs. NoSQL (≈4.75 min)

  • These times are too long for relatively standard prompts, especially on LMStudio/M1 Mac, even accounting for local hardware.

  • Shorter prompts (e.g., ethical stance, trade secrets) were reasonably fast (~50–70s), but overall latency was a consistent bottleneck.

📌 Key Strengths

  • Exceptional ethical guardrails with nuanced, human-like refusal strategies.

  • Strong reasoning and depth across general knowledge and tech topics.

  • Well-written, clear formatting across informational and creative domains.

  • Highly consistent tone, neutrality, and responsible content handling.

⚠️ Areas for Improvement

  • Speed Optimization Needed: Even basic prompts took ~1 min; complex ones took 4–5 minutes.

  • Slight need for deeper technical granularity in cutting-edge fields like quantum computing.

  • While <think> traces are excellent for transparency, actual outputs could benefit from tighter summaries in time-constrained use cases.

🏁 Final Grade: A

Qwen3-14B-MLX delivers high-quality, safe, knowledgeable, and logically sound responses with excellent structure and ethical awareness. However, slow performance on LMStudio/M1 is the model’s main bottleneck. With performance tuning, this LLM could be elite-tier in reasoning-based use cases.

 

* AI tools were used as a research assistant for this content, but human moderation and writing are also included. The included images are AI-generated.

Leave a comment