A New Benchmark for AI in the Steel Industry: How Can Large Industrial Models Move from “Demo Mode” to “Production-Ready”?

A large-scale steel industry model capable of answering process-related questions and generating analytical reports is sure to spark interest at a demonstration. But once the procurement process begins, the questions become more specific: Can the system still be used if the steel grade is changed? Are the recommendations reliable when raw material or equipment conditions change? Who takes responsibility if something goes wrong? Without verifiable, repeatable answers to these questions, it will be difficult for the pilot project to evolve into a production tool for long-term use.

On October 19, 2026, the State Administration for Market Regulation issuedOfficial Announcement, introducing three recently released national standardization guidance documents: *Artificial Intelligence—Guidance on the Classification of Application Scenarios in the Steel Industry* (GB/Z 224—2026), *Artificial Intelligence—Technical Requirements for Large-Scale Model Datasets in the Steel Industry* (GB/Z 231—2026), *Artificial Intelligence—Maturity Assessment of Service Capabilities for Large-Scale Steel Industry Models* (GB/Z 241—2026). These three documents are advisory technical documents and are not mandatory national standards.

Three Documents Provide a Common Framework for Procurement Discussions

According to the announcement, the scenario guidelines are intended for long-process steel enterprises and cover process and business aspects such as R&D, design, and process control; the dataset documents address data aggregation, governance, knowledge integration, sample processing, and security; and the maturity documents provide a service capability framework, evaluation metrics, classification rules, and assessment methods.

For businesses, this provides a framework for structuring the discussion around three key questions: in which business processes the standard will be applied; what data will support its implementation; and how to demonstrate that the service can be delivered consistently. Since the notice does not specify individual metrics or passing thresholds, it is still necessary to review the full text of the document and consider project agreements during the procurement process; a supplier’s mere statement that they “comply with the new standard” should not be accepted as evidence of acceptance.

The following arrangements regarding model selection, testing, and return on investment are recommendations for project implementation and do not constitute a restatement of the provisions or classification thresholds set forth in the three documents.

First, define the task; then decide what model is needed.

“Building a large-scale steel model” is not sufficient to describe a deliverable project. A clearer way to phrase it is: for which types of positions, under what operating conditions, using what inputs, to assist in making what decisions, and ultimately by whom the actions will be taken.

For example, retrieval procedures and the generation of equipment maintenance recommendations may both fall under the category of knowledge services, yet they carry different risks. The former requires verification of versions and sources, while the latter must also take into account equipment status, applicable conditions, and manual approval. If adjustments to production parameters are involved, safety boundaries, authorization mechanisms, and failure handling procedures must be reviewed separately; the testing methods used for question-and-answer systems cannot be applied here.

For the first batch of projects, it is advisable to select tasks with clearly defined responsible parties, verifiable results, and accessible data. Begin by comparing the costs and effectiveness of rule-based systems, traditional analytical methods, and large-model solutions; there is no need to rewrite stable and effective functionality simply to use large models. Companies with short production processes or specialized processes should also verify the scope of applicability to avoid directly adopting classification schemes designed for long production processes.

Prevent “Seeing the Answers” During Dataset Acceptance”

The purchasing party should require the supplier to provide a list of data sources and authorizations, descriptions of fields and units, a version history, data cleansing rules, and the basis for data annotation. What matters is not the claimed volume of data collected, but whether the data accurately corresponds to equipment, furnace batches, time, and operating conditions, and whether errors can be traced and corrected.

The test set, in particular, must be independent. If similar records from the same production run are included in both the training set and the test set, the results may overestimate generalization ability. Isolation can be designed based on time, equipment, or production batches, and test samples from suppliers who did not participate in the debugging process should be retained. The specific isolation method should be determined by the business problem at hand, rather than mechanically applying a single ratio.

The sample should also include missing values, sensor anomalies, rare operating conditions, and conflicting data. Knowledge-based Q&A should verify whether outdated procedures are being incorrectly cited; production analysis should verify whether definitive recommendations are still provided when input is incomplete. Allowing the system to recognize the limits of its capabilities and escalate to human intervention is more appropriate for critical production scenarios than requiring it to output an answer every time.

Service maturity should be demonstrated through verifiable operational evidence.

Procurement evaluations should not focus solely on model parameters or demonstration accuracy; they should also require the presentation of actual records documenting version changes, access controls, operational monitoring, fault handling, and issue resolution. How the system performs following restrictions on the on-premises network, interface interruptions, or knowledge base updates is part of the delivery capability.

When suppliers cite maturity levels, they should specify the subject of the assessment, its scope, the version, the date, and the corresponding evidence. The assessment conclusions for a particular platform or service do not automatically apply to all tasks across all production lines. Companies should also designate an internal business manager to clarify which issues should be handled by the supplier and which require joint judgment by the process, equipment, or information departments.

Acceptance testing should be conducted in phases to avoid deciding the outcome based on a single demonstration.

The project can be divided into three phases: offline testing, on-site bypass validation, and limited-scope deployment. Bypass validation involves having the system read approved data and generate recommendations without directly controlling production; staff then compare the recommendations with actual results. Each step should include criteria for proceeding, making corrections, or terminating the process.

• Functional Layer:Use a fixed set of tasks to verify that the results are correct, the sources are traceable, and the permissions are valid, and record any tasks that cannot be completed.
• Business Layer:Select metrics based on the task. For knowledge retrieval, focus on correct citations and search time; for defect identification, focus on missed defects and false positives; and for process recommendations, focus on constraint satisfaction and expert review results.
• Runtime Layer:Check response latency, fault recovery, log retention, and manual takeover, and test performance when key interfaces are unavailable.

All thresholds should be agreed upon by both parties prior to testing, and the scope of the samples, statistical criteria, and retest procedures should be clearly specified. It is not permissible to select the best-performing steel grade or shift after acceptance; nor is it appropriate to report only average values while concealing low-frequency but serious errors. For items related to production control, separate safety verification and approval procedures must also be arranged.

When it comes to ROI, define the boundaries first, then discuss scaling.

Whether the goal is to save labor hours, reduce energy consumption, or minimize quality losses, reliable baselines are essential. It is important to compare production lines, shifts, or time periods with similar conditions while documenting changes in raw materials, product mix, equipment maintenance, and other factors to avoid attributing all the benefits of other improvements solely to AI.

Costs should also include data preparation, system integration, computing power, on-site training, ongoing operations and maintenance, and version upgrades. Since manual verification has not been eliminated, returns cannot be calculated based solely on the assumption that “all manual work has been replaced”; furthermore, for any given efficiency improvement, the benefits cannot be double-counted as both saved labor hours and increased output.

During the pilot phase, it is more appropriate to report observed improvements, unvalidated hypotheses, and ongoing costs. Only when it is confirmed that the benefits stem from a replicable method—rather than from long-term on-site remediation by a small number of experts—will there be a solid basis for expanding the initiative to more equipment and production lines.

Before Signing the Contract, Include These Five Items in the Project Documentation

• Clearly describe the scenario, role, inputs, outputs, and prohibited actions, and designate a business owner.
• Define data sources, access permissions, retention policies, and opt-out procedures, and establish an independent test set.
• Specify the calculation method, sample scope, compliance criteria, and the party responsible for providing evidence for each acceptance criterion.
• Define procedures for exception handling, fault response, regression testing after model updates, and conditions for exiting each phase.
• Jointly confirm the baseline benefits and full-lifecycle costs, and decide whether to expand deployment based on the validation results.

This round of document releases has provided a clearer framework for discussing AI in the steel industry. Whether companies can truly benefit still depends on translating this framework into task boundaries, data-backed evidence, and operational accountability. Only when industrial large models can demonstrate their effectiveness under clearly defined conditions—and explain when they should not be used—will they have the opportunity to move from showcasing capabilities to delivering tangible results.