# Rendering Multiple NES Tetris thingos

Thread in 'Discussion' started by XaeL, 18 Mar 2020.

1. ### XaeL

@Jago / K

Hi, just chiming in here,

Also i've already implemented a NesTris99 implementation:
Here's 16+ players playing over the internet

And here's 16+ games made from just scraping video straight off twitch:

I'm glad that real programmers are doing it properly.

@Jago
For the Nestris89 stuff it does the following
1) Convert from any video source (screen cap, or direct from capture card) to game state in json. You could for example have one laptop with 4 capture cards and then 4 daemons running.
2) Send it off via network
3) Render in Unity3D

Right now Nestris89 can be played on internet or wherever, we've used it for live qualification twice

The only thing left to implement is kick/banning clients, and password protection / rooms.

There are already a few modes:
* Nestris99 (highest score wins, you all start at the same time)
* Qualification (Timed mode where highest score wins)
* Free play (just displays all fields)

Also the server sends a sync signal to clients in Nestris99 mode, so they can hear when to press start. It then queues the current state so that the games actually start simultaneously as despite games starting within an 5-10 second window across the world.

GyRo likes this.
2. ### K

Effectively we are working on the same kind of solution.
it depend about your goal acuracy. I just took some random example from your videos but i see a lot of artifact that i wouldn't like to see on the solution i'm working on for CTEC :
 this is a I falling piece This is a falling O piece :

furthermore, i don't have "unlimited amount of calculation power" : the openCV python program is currently running on a raspberry Pi 3 hooked to the console. i also send data through network but JSON is just a waste of bandwith. I encoded each frame into a "byte array" lowering down each message to around 80 bytes.
Right now i'm just finishing some stuff on the unity program, then i'll work again on the frame analysis.. there are lot or improvement room, and i aim to get thing to run on a raspberry pi Zero, wich is slower and single core..
My visualisation app is also in unity. but i not put any animation or playfield switching position because the team want things to stay static.

so far here is a POC with 18 games running at the same time on my laptop :

even if it's from the same replay from file sent trough network.. the unity app can maintain 60fps and there are lot of room for optimization.
that screenshot is from last week or something.. now the digits are displayed and a specific "curtain drop" animation added. almost finished the CTEC logo animation too.

Last edited: 27 Mar 2020
3. ### K

Yo,
Just to precise my previous post, don't feel any animosity about it, but just wanted to precise that computervision acuracy isn't "pretty trivial".
I like what you did so far about the online mode. Providing a solution for all the online tournament running atm is really cool.

4. ### XaeL

I didn't get any notification; eugh

That artifacts are due to deinterlacing. You can guess what an interlaced stream does to an average of 5px for guessing block color.
In a tournament setting you can control capture card / deinterlacing. This is run by non Tech savvy internet users so the data source can be non ideal. Garbage in garbage out
The streams that don't do that are obviously either emulator or a properly deinterlaced output.

My field data is also exactly 80 bytes. I send store it as json internally and can output as either json or bitpacked. JSON is great if you're writing a generic pack to pipe to say a web browser or something.
I'm guessing you're also doing 4bits per cell (primary secondary white or black). I was considering further optimization like field diffs or active piece + previous field, but didnt feel it was necessary at this point.
You can further optimize by only tracking active piece and sending field once per piece, scanning score/field/lines once per piece etc. Atm i scan every frame which is expensive since OCR is the most expensive part.

The OpenCV python program is currently running on a raspberry Pi 3 hooked to the console

What are you using OpenCV for? I found that it was faster to just have a 256x224 size image and compare pixels. For text detection i did my own rudimentary template matching (same as opencv, i'm assuming its similar speed).
For C like speed if you're doing things in python you can use numba which precompiles to native C. If you're literally piping it into and out of OpenCV, then it's already almost native code the entire way.
If you're using OpenCV and grabbing a capture card, then you are all good. My approach supports capturecard/screencapture/windowcapture,
which all have decreasing speed (especially screen/windowcapture where the resoution is way bigger than the 256x224 required, so i have to add an expensive nearest-neighbor downscaling stage)

Running it off a pi zero is definitely ambitious. I can see a nice tournament setup of a cheap capture card + pi zero + wifi card hooked up together, then everything streaming to the unity display server.

As per Tetris Concept tradition i'd say our things are 99% similar to each other (i'm using python/opencv + unity too). It wouldnt be hard for either of us to add the other's functionality.

But my project's pretty dead coz no one uses it. There was a big hit last year when it was released but it fell by the wayside. I think if i added a proper automated ladder that might help.
I'm glad that you have a real tournament that would actually use it. There's a lot of cool tech that you guys do (e.g. custom carts) that people don't really notice.

Future extensions for my project are :
1) emulator support. You can read the data straight out of emulator so you don't have to do OCR which is expensive
2) proper ladder multiplayer support. It only supports qualification/KOTH atm.

For your use case it would be nice to use Unity3d's 3d effects.
I use 3d blocks but with orthographic camera. You can then do cool shots (think TGM 3 Ti intro scene) pans very easily.

You can also add custom animations. for example blocks exploding or such. No one really pays attention during an 20+ field screen anyway, so it would be good to add random effects (say 5-10% of the time) to draw peoples attention, and also make it obvious which players are post transition. I know the TO just want 20 screens statically, but i think automatically determining a "main" game or main two games to make slightly larger when people hit transition would be more interesting to watch. I mean Look at my actual qualifier video that i posted above. What were the things you noticed/didnt notice? was a live scoreboard good, or would it be better in a different screen? Did your eyes just gaze at whoever was transitioning? What could you do to aid it?

Last edited: 31 Mar 2020
5. ### Muf

I strongly think that 3D effects will just detract from the retro aesthetic. What's the last NES game that had 3D effects? I can't remember either.

As for keeping attention, I don't think adding visual fireworks (literally) will make the presentation more clear to the viewer. I think it will just make it more chaotic with so many fields in view already. I think there are two subtle effects that could work:
- Flash the level box with the NES level up sound when transition occurs
- Flash the score box with some other retro sound effect when a new high score occurs

Other than that, you have to remember that a real tournament setting is very different from just a demo video showing random names. Every player has their own story and people have favourites and players they want to see succeed, so they will gravitate towards watching those players, while ignoring names they don't recognise. I think having 8 fields on screen is the sweet spot: for CTEC we are planning to have 8 qualifying stations that are level 18 only, and only show those on stream. The high level players will be playing on those stations, and because of the speed, stations will cycle players more often, and it's more likely exciting games will be shown.

6. ### XaeL

Just a few things on OCR optimization (from my own github discussions):

Level 1 optimization
* I'm guessing you don't scan piece distribution.
* OCR -> text is the most expensive thing.
* You can ocr score only once per piece placed. (probably need to scan all 6, or you can use math to figure out how many digits based on previous score and current level)
* You can ocr lines only if score changed
* You can skip OCR of all the digits in lines, just get last digit on line clear
* You can skip OCR of level; just increment every 10 lines

This cuts down OCR'ing 3+6+2 (11) digits ->
6 digits on every piece
7 digits on every line clear
7 digits on every level up.

Level 2 optimization
Then more optimization, you can change the order:
1) Scan last digit of lines on every piece placed, and the last 2 digits of score.
2) if lines++, you can calculate all the score digits except for the last two (softdrop)
This means you only have to scan 1 digit of lines on every piece, and 2 digits of score
this is 3 digits per piece, instead of 11 digits/frame.

For piece distribution, i currently have two methods:
1) scan all digits every frame (expensive, 21 digits)
2) scan top of field (assume 0 dropped frames) to determine piece that spawned (checking 4-6 cells for non-black)

For "piece has been placed" i have one method:
scan a 2x4 grid of cells every frame.
If they were all blank last frame, and are now not blank, its a new piece.

Let me know your thoughts (i haven't implemented half the stuff, because performance was acceptable. You're using a RaspPi though so hopefully these help )