Skip to content
J.R.
日本語 EN 中文

01 / 05 · 2019.07 — 2020.07

Beijing Yunshangzuo Technology Co., Ltd.

Applied algorithms and full-stack work in AI × education: real-time classroom-behaviour analysis and alerts with Mask R-CNN + ResNet, an internal labelling and training platform, the stats backend of an online-class monitoring system, face verification and screenshot-recognition services.

Role
Software Engineer
Industry
AI × education
Location
Beijing
Team
5 people
Tenure
1 yr 1 mo

Key results

  • 6 classes

    Classroom behaviours (heads up, head down, standing, slumped, turned away, hand raised); confidence below 0.95 is "uncertain", never forced

  • 21 routes

    Stats-engine backend for online-class monitoring (~1,650 lines of Tornado) on Elasticsearch aggregations

  • 10,000+

    Screenshots I labelled myself for the WeChat-screenshot recogniser (plus 1,000+ icons)

Projects here

PROJECTS · 4
Data labelling: Mask R-CNN pre-labels, and only boxes and behaviour classes are corrected in Labelme

2019 – 2020 · Full-stack engineer (algorithms, backend, internal training platform) · Team 5

AI classroom behaviour analysis system

Classroom cameras recognise student behaviour (heads up, head down, hand raised, slumped on desk, turned away), face recognition maps it to individual students, and the system offers live alerts and multi-level class reports. I built two subsystems: a school-facing real-time analysis and alert platform, and an internal data-labelling and model-training platform.

  • Two-stage recognition: Mask R-CNN (R-101-FPN) finds students, ResNet18 classifies behaviour into 6 classes (confidence below 0.95 becomes "uncertain") — more accurate than one detector doing both, and changing the behaviour set only means retraining the small classifier
  • Real-time analysis: OpenCV reads the RTSP stream and drops frames whenever the video timestamp lags wall-clock time; one GPU process per class, with an nvidia-smi free-memory check before launch
  • Tracking and identity: wrote my own tracker using bidirectional overlap over a 5-second window; face_recognition (dlib) 128-d features are matched to the class face gallery, and the most frequent ID within a track wins — so identity survives head-down and side-face frames
  • Live alerts for slumping / turning away: the student's face crop (base64) goes into the alert; latest results and per-frame history are stored separately in MongoDB
  • Tornado backend: accounts with three roles, alerts and reports at four levels (student, class, grade, school), a WeChat mini-program API and an open API for third-party platforms; designed API field protocols against the product manager's prototype
  • Analysed behaviour data with scikit-learn and produced weekly, monthly and term reports per student, pushed to parents' WeChat mini program
  • Internal labelling and training platform: video trimming → Mask R-CNN pre-labelling (labelme JSON) → human correction → cropping → dataset building → ResNet18 training (resumable, loss/accuracy curves) → validation; the whole-lesson offline analysis was the prototype of the real-time algorithm
  • Delivery: compiled the core tracking algorithm to .so with Cython, ran under supervisor and used a licence dongle, deployed on-premise at several customers
  • Python
  • Tornado
  • PyTorch
  • maskrcnn-benchmark
  • face_recognition (dlib)
  • OpenCV
  • FFmpeg
  • MongoDB
  • scikit-learn
  • Cython
  • ECharts
  • jQuery

Live demos

  • Real-time platform: class monitoring overview
  • Principal's dashboard
  • Multi-class live feeds

System architecture

Capture layer Streaming & inference layer Data layer Service layer Clients Classroom camera RTSP video stream Demo video stream infinitevideo.sh FFmpeg loops the stream Live path (started when ClassDetailHandler opens a class) FFmpeg RTSP → RTMP relay nginx-rtmp rtmp://…/mytv/ H5 player ckplayer/video.js Start / stop analysis process (nohup + kill, check free GPU memory first) GPU inference process · one per class · front_back_frame ①OpenCV read + frame-drop control Skip frames when the video timestamp lags real time, keeping it real-time ②Mask R-CNN (R-101-FPN) detects students Keep "person" only · conf ≥ 0.7 · de-dup overlapping boxes (0.8) ③ResNet18 behaviour (6 classes) Pad single-person crop to square → 224×224 · conf < 0.95 = "uncertain" ④face_recognition identifies students CNN face location → 128-d feature → match class face gallery ⑤Cross-frame tracking + in-track identity vote 5-second bidirectional-overlap association · most frequent student ID in a track wins Slumped / turned away → crop face, write base64 into alert Write MongoDB realtimewarning Latest frame result per class record_pics Per-frame history pg_classes Class/stream/faces school/class/student School · class · student records students_performances Behaviour results per lesson bf_kaoqin … Customer-specific data Tornado Web :8080 · pymongo Account system Secure cookie · three roles Alert system Poll /getfaster Report views Student / class / grade / school Customer-specific pages Customer A · Customer B Mini-program API xcx.py Partner open API jyd_api.py Amaze UI · ECharts Principal / teacher Browser Mini program Parent / teacher Partner Third-party systems Operations & delivery supervisor keeps 4 processes alive (web · analysis · stream · VPN) | Cython compiles the core algorithm to .so | dongle licensing | Linux + NVIDIA GPU (CUDA)
1Trim video /ffmpeg · /ffmpegGetVideotimes Read duration, cut clips by start/end time 2Model auto pre-labelling /labelme · infer2labelme_mask Mask R-CNN inference writes labelme JSON 3Human correction labelme desktop tool Only fix boxes and behaviour classes 4Crop & sort /cutimg · /gray Crop one person per box, file by behaviour 5Build dataset /dataset · DataSet.py Check class folders are numbered from 0 6Train classifier /train · resnet_trainer ResNet18 · SGD · 50 epochs · resumable 7Validate /behavior · test_jr2_fang Adjustable threshold, batch testing 8Whole-lesson offline analysis /upVideo · class_recognition Track → cluster → identify → store ↓ Outputs the ResNet classifier .pth, deployed to the real-time platform ↓ The algorithm later became a real-time version Support: class & stream management (/addClass /modifyClassStream /removeClass) · student & face-photo management (/addStu /modifyStu /removeStu) · model & folder browsing (/getMaskmodel /getFolder)
Classroom video material Camera recordings / on-site capture Backend training platform Label · build datasets · train · validate Model files detector.pth (Mask R-CNN) classifier.pth Real-time analysis platform Real-time inference · alerts · reports New classroom video keeps arriving after go-live and flows back as training data
An online English class in progress (teacher and students pixelated)

2020 · Stats-engine backend, admin frontend, part of the algorithm analysis

AI diagnostic online-class monitoring system

For online English classes (lessons moved online because of the pandemic): teacher and student audio/video are captured live and structured by ASR / CV / VAD. Red-line behaviours (speaking Chinese, the teacher talking too long, too little student participation, leaving the camera, black screen or drop-out) trigger near-real-time alerts, and teaching-quality reports are compared and ranked by headquarters/branch/campus, course, teacher and student.

  • Stats-engine backend (Tornado, ~1,650 lines, 21 routes): lists, details and history trends for classes, teachers and students, teaching-attitude time series, model-class TopN
  • Elasticsearch queries: bool + terms + range, nested avg sub-aggregations under terms (per-class teacher happiness probability, word count, speech rate), scan/scroll for large result sets; CV and ASR results merged and sorted in the application layer
  • Admin frontend (vue-element-admin 4.4): role-based dynamic route permissions, data dashboard, class/teacher/student lists and details, rankings, trends, filters; ECharts visualisation, Docker and multi-environment config
  • Part of the algorithm analysis: combined speaking share, fluency and positive emotion to find ~30-second "highlight" clips, flagging noise, courseware audio interference and parent presence, for the progress report sent to parents
  • Python
  • Tornado
  • Elasticsearch
  • Redis
  • Vue 2
  • Element UI
  • ECharts
  • MQTT
  • Docker
  • Alibaba Cloud

System architecture

From capture to alert display
Capture side Ingest · storage · events Structured algorithms (gRPC) Search · analysis · display Browser <video> web-video-capture.js 1 frame/s · audio every 10 s ~30 kbps up · <5% overhead IOAuthProxy Auth service :9001 Aliyun OSS Public cloud MinIO On-premise deployment Notification Hub OSS trigger → MNS ASR WorkerZH/EN speech-to-text CV WorkerFace · pose · expression Sound (VAD)Speech start / end ElasticsearchAs time-series DB Analyse TaskQueue Analyse Enginestats_engine :9002 ReportDB+ ClassDB timetable Management Backend Vue MQTT event queue 1883 / 8083 / 8084 Request token webpwebm Red-line event WebSocket push → alert on teacher page

2020 · Algorithm and service development

FaceID face verification and search

Face verification for an online-education customer: confirm from class video that the person on camera is the registered student, with full-gallery, incremental and per-person face search to catch one person with several accounts or several people sharing one.

  • Inference flow: random frame sampling → YuFaceDetectNet detection with 5 landmarks → affine alignment → occlusion / pose-blur / age models filter → ResNet18 features compared by cosine similarity; the youngest face (< 17 years) is taken as the student, excluding parents
  • Mixed-framework inference: PyTorch for detection and features, MXNet Gluon for occlusion, pose and age, all on CPU; iterated from v1.0 to v3.4 (dlib 68-point + Laplacian blur check → new models and alignment → three quality models → rewritten full-gallery search and video validation)
  • Features live in Redis with global, incremental and per-person search; a full pairwise pass is estimated at about 40 million comparisons per the code comments
  • Commercial licence control: call counts, expiry and daily quota are AES-encrypted in Redis, SETNX gives a distributed lock, and the device SN is verified; the core module ships as a Cython-built .so
  • Config is read in the order command line > environment variables > JSON/YAML; Docker deployment pulls model weights from OSS at build time
  • Python
  • PyTorch
  • MXNet
  • dlib
  • ResNet18
  • Redis
  • Cython
  • Docker
  • OSS

System architecture

Verification & search inference flow (v3)
Video class video file Random frames resize to 320×240 Face detection YuFaceDetectNet · PyTorch 5 landmarks Affine alignment per face Quality filters (MXNet Gluon · CPU) Occlusion eyes / nose / mouth > 0.5 skip Pose / blur yaw / pitch / roll / blur over limit skip Age youngest (< 17) = the student ResNet18 features PyTorch · cosine similarity Matching & storage First time: enrol feature stored in a Redis hash Later: verify / search global · incremental · per person Licence control + Cython delivery AES quota · SETNX lock · .so
A sample Moments-ad screenshot (people pixelated)

2020 · Training, inference, service and deployment

WeChat screenshot recognition (Moments ad-delivery checks)

Automatic recognition of uploaded WeChat screenshots to verify Moments ad delivery: classify the screen type (chat / feed list / post detail) and extract the post time, ad copy, poster ID, QR-code link and like/comment counts.

  • YOLOv5s as the main detector (7 UI-element classes) with rules to decide the screen type; labelled 10,000+ screenshots and 1,000+ like/comment icons myself, with k-means anchor boxes
  • Post detail: PaddleOCR (PP-OCR, DB detection + CRNN recognition) reads post time, ad copy and poster ID by layout position; QR decoding; the bottom-right is cropped, like/comment icons are detected with YOLOv5 and the digits read by a 3-layer CNN
  • Tornado + PyTorch + PaddlePaddle service: token and MD5 integrity checks, nanoid request tracing, originals in OSS, every result and latency written to Elasticsearch for debugging production errors
  • Upgraded the PP-OCR model release, improving poster-ID and post-time recognition
  • Python
  • PyTorch
  • YOLOv5
  • PaddleOCR
  • PaddlePaddle
  • Tornado
  • Elasticsearch
  • OSS
  • Docker

System architecture

Screenshot recognition flow
Screenshot uploadTornado POST Validate + store in OSStoken · MD5 · nanoid YOLOv5s main detector7 icon classes / QR code Rule-basedBy position Chat · Plus or three-dot icon present Feed list · Camera icon → detect group visibility Other Post detail · QR code position on the detail page PaddleOCR (DB detection + CRNN recognition) Post time by position · copy by position relative to QR Poster ID via the "ID" keyword QR decoding Crop detected box → decode URL Like / comment counts Crop bottom-right → YOLOv5 finds icons → 3-layer CNN reads digits (98.25% validation) Aggregate and return JSON · log all results and latency to Elasticsearch · custom error codes (212 / 218 …)

System data flow

Abstract sketch · no internal details

  1. Classroom camera

    Live RTSP stream

  2. OpenCV reader

    Drops frames when late

  3. Mask R-CNN detection

    Locate students · conf ≥ 0.7

    ▲ My part

  4. ResNet18 behaviour

    6 classes · slump / turn → alert

    ▲ My part

  5. Face ID + tracking

    128-d features · in-track vote

    ▲ My part

  6. MongoDB → Tornado

    Live alerts · reports · mini program

What I learned here

After graduating I joined an AI × education company and, away from campus, learned and used more languages on the job. I implemented object detection and classification (Mask R-CNN, ResNet) with PyTorch inside the business logic; web services ran on Tornado and the company frontend moved from HTML / JS / CSS to Vue, which broadened my computer-science base.

Classroom-behaviour analysis taught me to split a problem: detect people first, then classify behaviour, and put unsure cases in their own bucket instead of forcing a verdict; cross-frame tracking and in-track voting make up for weak single frames, and frame dropping keeps the system real-time. It was also my first systematic taste of delivery engineering — Cython to protect algorithm source, supervisor for process management, on-premise deployment at several customers, and Elasticsearch aggregation queries for the online-class monitor.

TECH STACK

  • Python
  • PyTorch
  • Mask R-CNN
  • ResNet
  • YOLOv5
  • MXNet
  • PaddleOCR
  • dlib
  • Tornado
  • Elasticsearch
  • MongoDB
  • Redis
  • Cython
  • Vue 2
  • Docker