Automated sorting and quality grading of agricultural produce are crucial for providing commodities with consistent quality to the consumers and markets. Machine vision has been playing a key role in this quest by presenting technological solutions that provide robust, consistent, and accurate decisions with minimal human intervention. An end-to-end quality inspection system should recognize the type of agricultural product and then perform quality grading. Accordingly, in this proof-o- -concept study, a deep learning-based end-to-end solution for quality inspection of agricultural produce is presented, where an initial system automatically sorts fruitsvegetables, while a second system grades apples by skin quality. Experimental evaluations show that the presented end-to-end solution achieves accurate and promising results, and thus holds high-potential for offering high-impact, traceable and generalizable answers for the industry.