<- Back to the Teaching of Morimoto Lab

Processing03

You need your headset for this lesson.

---

---

Sound

Basis of sound

Sound is defined as a vibration traveling through a medium (air).
Sound is typically represented as a plot of pressure over time.

The periodic pattern of the waveform repeats.

Typically, sounds occurring in the natural world contain many discrete frequency components.
As such, the below figure is called (frequency) spectrum (=周波数スペクトル).

Point: Three components of sound
Amplitude -> Volume (intensity)　大きさ
Frequency -> Pitch　高さ
Shape of waveform -> Tone　音色

[Fourier transform & image compression.]

Similarly, images also consist of waves.

---

Topics of Text & Sound

I’d like to introduce topics ranging from traditional representation to the latest service based on research and artistic works!

TextAlive is a system to create PV semi-automatically.
Let’s see! http://textalive.jp/

• In an actual live stage of an idol singer group, a movie created with TextAlive was used.
  
  • No Maps IDOL DIVERSITY「BELLRING少女ハート」のステージで、TextAliveで制作された映像が使われました！

It is based on Songle which is a web service or research to analyze music by Dr. Goto of AIST(産総研).

• Goto-san’s home page has many interesting projects related to music analysis.
• It allows us to detect the “music map” that includes
  
  • structural segment (サビ、A/Bメロ…),
  • melody, beat (拍子),
  • chords (和音).

Kinetic typography is related to these topics.

• Motion typography
  
  • scrolling typography : such as STAR WARS’s opening?
  • dynamic layout : texts move in relation to one another. (I couldn’t find any examples…)
• Fluid typography : youtube (<-morphing is used)

PV: androp

• Bell
• World.Words.Lights.
  
  • This is not very related to the topics here…
• In their PVs, interesting techniques are used sometimes. NAVERまとめ

VJ software:

• software to create videos for video jockeys.
  
  • nicoわりとすごいVJソフトを作ってみた

Toshio Iwai　岩井俊夫

• 映像装置としてのピアノ [web][yt:Ryuichi]
• テーブルの上の音楽 [web]
• TENORI-ON [yt]
• エレクトロプラクトン [yt]

---

Methods & Class of Sound

Official reference of Sound

Words:
I/O … Input / Output
Band … frequency band. A part of frequency. 周波数帯域。
Reverve … 残響
Oscillators … Transform waveform
Pulse … a brief sudden change of a wave
Envelope … 包括線。yourube

Let’s see demos from Sample.

1. Library > Sound > Analysis > AmplitudeRMS
   
   • The size of ellipse is changed by the volume.
2. Library > Sound > Analysis > FFTSpectrum
   
   • Display the spectrum analysed by Fast Fourier Transform.
3. Library > Sound > Demos > Envelopes
   
   • Change attackTime, sustainTime/Level, releaseTime.
4. Library > Sound > Demos > Keyboard
   
   • like a piano
5. Library > Sound > Demos > Sampler
   
   • sampler: musical instrument using sound recordings (=samples).
   • sequencer: it can record, edit, or play by your action.
6. Library > Sound > IO > AudioInput
   
   • needs your microphone.

7. Library > Sound > Soundfile > Sample

   • See the methods of SoundFile class.

   High, low, band pass filter:
   High pass : Only high frequency can pass.
   Low pass : Only low frequency can pass.
   Band pass : Only specified band’s frequency can pass.

   See youtube for sounds of each frequency. (high freq is high pitch.)

   See youtube for H/L/B pass filter.

Ex1: Make an application where music and graphics interact such as the ones below.

• music -> graphics
• graphics -> music
• music <-> graphics
• music, graphics <-> music, graphics
• etc.

---

3D

The followings are partially extracted from Official tutorial of P3D.

• There are four render modes:

  1. the default renderer
  2. P2D
  3. P3D
  4. PDF

• We can specify the render mode via the size() function.

void setup() {
  size(200,200,P3D);
}

• 2D and 3D coordinate
  (the left and right in the figure below).

• How to draw rect() in 3D?

We use rect() as well as 2D drawing with translate(moveX, moveY, moveZ).

translate( moveX, moveY, moveZ);
rect( posX, posY, w, h);

translate() can have 2 or 3 parameters. See the reference.

Below is a sample program.

float x,y,z;

void setup() {
  size(200,200,P3D);
  x = width/2;
  y = height/2;
  z = 0;
}

void draw() {
  translate(x,y,z);
  rectMode(CENTER);
  rect(0,0,100,100);

  z++; // The rectangle moves forward as z increments.
}

• rotate() in 2D. rotateX(), rotateY(), rotateZ() in 3D.

size(200, 200, P3D);
background(100);
rectMode(CENTER);
fill(51);
stroke(255);

translate(100, 100, 0);
rotateZ(PI/8);
rect(0, 0, 100, 100);

rotateX(PI/8);

rotateY(PI/8);

translate(100, 100, 0);
rotateX(PI/8);
rotateY(PI/8);
rotateZ(PI/8);
rect(0, 0, 100, 100);

3D shapes

3D primitives are box(), sphere(), sphereDetail().
See “3D Primitives” here.
See “Lights, Camera” also.

Below is a sample.

size(640,360,P3D);
background(0);
lights();   //This is new! 
//Sets the default light such as 
//ambientLight(128, 128, 128) and 
//directionalLight(128, 128, 128, 0, 0, -1), 
//lightFalloff(1, 0, 0), and 
//lightSpecular(0, 0, 0).

pushMatrix();
translate(130, height/2, 0);
rotateY(1.25);
rotateX(-0.4);
noStroke();
box(100);   //This is new!
popMatrix();

pushMatrix();
translate(500, height*0.35, -200);
noFill();
stroke(255);
sphere(280);//This is new!
popMatrix();

---

beginShape(), endShape(), and vertex() allowed to draw custom 3D shapes.
Below is a sample.

size(640, 360, P3D);
background(0);

translate(width/2, height/2, 0);
stroke(255);
rotateX(PI/2);
rotateZ(-PI/6);
noFill();

beginShape();
vertex(-100, -100, -100);
vertex( 100, -100, -100);
vertex(   0,    0,  100);

vertex( 100, -100, -100);
vertex( 100,  100, -100);
vertex(   0,    0,  100);

vertex( 100, 100, -100);
vertex(-100, 100, -100);
vertex(   0,   0,  100);

vertex(-100,  100, -100);
vertex(-100, -100, -100);
vertex(   0,    0,  100);
endShape();

See examples of RGB cube, Vertices, Toroid, Isocahedra for more about 3D shapes.
[official web example page]

Texture Mapping (UV mapping)

Sample > Topics > Textures > TextureQuad

void draw() {
  background(0);
  translate(width / 2, height / 2);
  beginShape();
  texture(img);
  vertex(-100, -100, 0, 0,   0);
  vertex( 100, -100, 0, 400, 0);
  vertex( 100,  100, 0, 400, 400);
  vertex(-100,  100, 0, 0,   400);
  endShape();
}

Here, vertex(x, y, z, u, v) is used. x, y, z are coordinates of the 3D space and u, v are x, y position of the texture.

Check the examples:

• Texture Triangle
• Texture Cylinder
• Texture Cube
• Textured Sphere

[2D Texture mapping: See the reference]

See the “Textures” section of the tutorial.

Lighting

Copy the below code and run.
Then, push a mouse button to enable lights().

void setup() {
  size(200, 200, P3D);
}

void draw() {
  background(0);
  translate(100, 100, 0);
  if (mousePressed) {
    lights();
  }
  noStroke();
  fill(255);
  sphere(50);
}

Ex2: Generate the below images by using ambientLight(), derectionalLight(), spotLight(), pointLight().

Perspective

There is perspective(the left) and orthographics(the right).

See the reference of perspective() and ortho().

void setup()  {
  size(640, 360, P3D);
  noStroke();
  fill(204);
}

void draw()  {
  background(0);
  lights();

  if(mousePressed) {
    float fov = PI/3.0; 
    float cameraZ = (height/2.0) / tan(fov/2.0); 
    perspective(fov, float(width)/float(height), cameraZ/2.0, cameraZ*2.0); 
  } else {
    ortho(0, width, 0, height); 
  }
  translate(width/2, height/2, 0);
  rotateX(-PI/6); 
  rotateY(PI/3); 
  box(160); 
}

Ex3: By editing the above example, arrange the position to see the box in both cases of perspective and orthographic.

Camera

The default settings of camera() are shown below:

• Eye position: width/2, height/2, (height/2) / tan(PI/6)
• Scene center: width/2, height/2, 0
• Upwards axis: 0, 1, 0

See the reference of camera().

void setup() {
  size(640, 360, P3D);
}

void draw() {
  background(0);
  camera(mouseX, height/2, (height/2) / tan(PI/6), width/2, height/2, 0, 0, 1, 0);
  translate(width/2, height/2, -100);
  stroke(255);
  noFill();
  box(200);
}

Ex4: Run the above code and then change the fourth parameter of camera() to mouseX.

• The fourth parameter is the center position of the 3D scene in x-axis.

loadShape() (.obj, .svg)

See the official reference of loadShadpe().

Sample > Shape > LoadDisplayOBJ is the below.

PShape rocket;//PShape is new.
float ry;   //Rotation of Y-axis.

public void setup() {
  size(640, 360, P3D);    
  rocket = loadShape("rocket.obj");//new.
}

public void draw() {
  background(0);
  lights();

  translate(width/2, height/2 + 100, -200);
  rotateZ(PI);//PI is initially defined
  rotateY(ry);
  shape(rocket);

  ry += 0.02;
}

PShape: Datatype for storing shapes such as svg, obj. [ref]
Here, we mainly use loadShape() with prepared files.
However, createShape() is also useful to make by kinds and so on [ref].
Kinds : POINT, LINE, TRIANGLE, QUAD, RECT, ELLIPSE, ARC, BOX, SPHERE

Ex5: Download obj files from Perfume global site and display Perfume by coding.

Note: Not only obj files, but also .mtl (material file) and texture files in the same folder should be placed in the same folder (named “data”) of your program.

(Perfume data)
In the obj file, mtl file is used.
In the mtl file, the texture file is used.

Note: Open the obj file by using text editor. You can see the contents shown below.

mtlib "material file's name"
g "group name"
usemtl "material name"

v x1 y1 z1 #position of vertex 
v x2 y2 z2
#omit other v
vt u1 v1 #corresponding texture uv
vt u2 v2
#omit other vt
vn x1 y1 z1 #vertex normal
vn x2 y2 z2
#omit other vn
f "index of v"/"index of vt"/"index of vertex normal" (continue as many as the number of the vertex of the face/polygon (n角形))
#omit other faces

To access each vertices in the .obj, use vertex().

PShape aa;

public void setup() {
  size(1000, 600, P3D);
  aa = loadShape("data/model1.obj"); 
  stroke(0);
}

public void draw() {
  background(0);
  lights();

  translate(width/2, height/2, -200);
  rotateZ(PI);
  rotateY(PI);
  shape(aa);  

  int childCount = aa.getChildCount();//face num
  beginShape(TRIANGLES);
  for (int i=5000; i<childCount; i++) 
  {
    //if (rocket[0].getChild(i).getVertexCount() ==3){      
    for (int j=0; j<aa.getChild(i).getVertexCount(); j++) 
    {
      PVector p = aa.getChild(i).getVertex(j);
      vertex(p.x, p.y, p.z);
    }
    //}
  }
  endShape();
}

• beginShape(), vertex(), endShape()
• getChild():
  
  • ”Child” is a face/polygon/mesh defined as PShape. Since a model (.obj) consists of many faces, these faces are defined as indexed children.
  • You can add a child. addChild()
  • getChildCount(): The number of children (faces).
• getVertexCount(): The number of vertex (in this case, 3)

The above program is very slow. The below program is fast.

PShape aa;
PShape bb;

public void setup() {
  size(1000, 600, P3D);

  aa = loadShape("data/model1.obj"); 
  bb=createShape();
  bb.beginShape(TRIANGLES);
  bb.stroke(128);//put between begin and endShape.

  int childCount = aa.getChildCount();//face num

  for (int i=5000; i<childCount; i++)//insufficient memory if i=0 
  {          
    for (int j=0; j<aa.getChild(i).getVertexCount(); j++) 
    {
      PVector p = aa.getChild(i).getVertex(j); 
      bb.vertex(p.x, p.y, p.z);
      //PVector tem = new PVector(p.x+50, p.y, p.z+150);
      //if (i<6000) {
      //  aa.getChild(i).setVertex(j, tem);
      //}      
    }
  }
  bb.endShape();
}

public void draw() {
  background(0);
  lights();
  translate(width/2, height/2, -200);
  rotateZ(PI);
  rotateY(PI);
  shape(aa);  
  shape(bb);
}

• PShape class has methods of beginShape(), vertex(), endShape() also and we can use them as well.
• Why i=5000? : Because an error would occur from insufficient memory.
• Remove double slashes in the double for loops to enable that part.
  

3D Motion

Download the program using the motion data of Perfume from here.

1. Click “Clone or download”
2. Click “Download ZIP”
3. Unzip the file.
4. Open : example-processing-master > p5f_sample > p5f_sample.pde
   
   • But they won’t dance…
   • In the data folder, there are the motion data of A/ B/ C_test.bvh.

Download the dance and sound data from here.

1. Select “Download” from the top-left.
2. Check “You agree to our terms”
3. Click “BVH” & “SOUND”
   
   • In the “bvhfiles” folder, there are aachan/ kashiyuka/ nocchi.bvh.
   • In the sound folder, there is Perfume_globalsite_sound.wav.

Ex6: Edit the above example program to change bvh files to aachan/ kashiyuka/ nocchi.bvh and play the sound.

• The result of this exercise would be like the below video.

Unfortunately… it is not easy to apply motion data (.bvh) to obj file by Processing.

3D Motion: Access each joint of BVH

Open bvh file with a text editor such as the one below. At a first part of bvh file, there are joints and their relative positions which is called a hierarchical tree structure. There are parent and child nodes.

HIERARCHY
ROOT Hips
{
 OFFSET 0.000000 0.000000 0.000000
 CHANNELS 6 Xposition Yposition Zposition Yrotation Xrotation Zrotation
 JOINT Chest
 {
  OFFSET 0.000000 10.678932 0.006280
  CHANNELS 3 Yrotation Xrotation Zrotation
  JOINT Chest2
  {
   OFFSET 0.000000 10.491159 -0.011408
   CHANNELS 3 Yrotation Xrotation Zrotation
   JOINT Chest3
   {
    OFFSET 0.000000 9.479342 0.000000
    CHANNELS 3 Yrotation Xrotation Zrotation
    JOINT Chest4
    {
     OFFSET 0.000000 9.479342 0.000000
     CHANNELS 3 Yrotation Xrotation Zrotation
     JOINT Neck
     {
      OFFSET 0.000000 13.535332 0.000000
      CHANNELS 3 Yrotation Xrotation Zrotation
      JOINT Head
      {
       OFFSET 0.000000 8.819083 -0.027129
       CHANNELS 3 Yrotation Xrotation Zrotation
       End Site
       {
        OFFSET 0.000000 16.966594 -0.014170
       }
      }
     }
     JOINT RightCollar
     {
      OFFSET -3.012546 7.545150 0.000000
      CHANNELS 3 Yrotation Xrotation Zrotation
(*snip*)

Note: The name of joints are defined arbitrary by the editor. (It is not common)

The last part of the bvh file is such as the below. This part records each relative behavior (rotations) of the joints of each frame. Each value corresponds to each joint rotation.

0.000000 83.315683 0.000000 0.000000 -5.826342 0.000000 0.000000 11.116423 0.000000 0.000000 -5.290081 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 12.665069 0.000000 0.000000 0.575450 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 2.337306 0.000000 0.000000 -2.337306 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 -2.337306 0.000000 0.000000 2.337306 0.000000 0.000000 0.000000 8.578828 0.000000 0.000000 1.905768 0.000000 0.000000 -4.658254 0.000000 0.000000 0.000000 0.000000 0.000000 8.578828 0.000000 0.000000 1.905768 0.000000 0.000000 -4.658254 0.000000 0.000000 0.000000 0.000000
3.126870 82.488588 -2.512073 -163.432317 -12.583494 2.411893 -0.022186 6.091839 -0.870750 -0.000485 0.270951 -0.038541 -0.000879 0.439592 -0.062531 -0.046942 5.579781 -0.796611 -2.351908 19.167381 -0.239515 -4.228814 -0.565389 0.309274 1.982247 0.541421 8.873998 -15.617168 -1.897286 79.435858 14.419189 10.956827 7.499878 4.796990 18.335287 -11.894136 1.117364 1.544991 -10.761212 7.067530 -1.172001 -76.267319 -15.993750 13.473308 -8.174009 -7.019157 11.160696 9.245920 -5.344010 13.426208 -18.847884 0.430844 0.586119 -1.442370 10.955876 -5.488756 15.073039 0.012261 -0.024437 -0.003367 1.696452 22.047257 7.925247 7.575236 -2.347691 2.505695 -0.230943 -6.957240 -14.392770 -0.020288 0.030451 -0.023185
(*snip*)

Note:
Here, these relative angles are defined against the parent nodes.
It is not an absolute value!

In this case, joints are defined such as the below left.

Let’s change the color of the head ellipse such as the above right.
See the program of PBvh class.

public class PBvh
{
  public BvhParser parser;
  //BVHを解析するjavaクラス。ファイルの読み込みなども。

  public PBvh(String[] data)
  {
    parser = new BvhParser();
    parser.init();
    parser.parse( data );//ファイルの読み込み
  }

  public void update( int ms )
  {//以下2行で指定した秒数のポーズに更新
    parser.moveMsTo( ms );
    parser.update();
  }

  public void draw()
  {    
    fill(color(255));
    imageMode(CENTER);

    //BvhBone: モーションの骨格情報を保持するクラス
    //以下のようなfor文を拡張for文と言う。
    //getBones()で取得した全ての骨格情報から一つの骨格を取り出し、forで全ての骨格を一つ一つ見ていく。
    for ( BvhBone b : parser.getBones())
    {      
      pushMatrix();
      translate(b.absPos.x, b.absPos.y, b.absPos.z);//absPos: 骨格の位置情報
      ellipse(0, 0, 2, 2);
      popMatrix();

      //骨格は連結しているので、親子関係を持つ（例：肩の子が肘、肘の子が手首...）
      //以下の場合、子がない、つまり先端の骨格の場合、{}の処理をする
      if (!b.hasChildren())
      {
        //print(b.getName()+", ");
        //print(b.getYrotation()+", ");

        //getName()には==ではだめ。equalsを使用
        if (b.getName().equals("Head")) {        
         fill(color(255, 0, 0));
        }

        pushMatrix();
        translate( b.absEndPos.x, b.absEndPos.y, b.absEndPos.z);//先端の位置情報
        ellipse(0, 0, 10, 10);
        popMatrix();

        fill(color(255));//色を変えたときに元に戻す
      }
    }
  }

Ex7: Change the ellipse of head to an image file.

• (Advanced) Apply rotation of the head joint to the image.

Hint:
b.getX/Y/Zrotation() are relative angles against the parent joints.
Absolute angles would be convenient here.
There is an example below.

PMatrix3D mat = b.global_matrix;
applyMatrix(mat);          
//draw something here.

Draw motion lines with a dynamic array

Motion lines are usually used for representations of comic books.

It is one of the hot topics in CG research suitable such as non-photorealistic rendering (non-photo or NPR or stylized rendering). Some works transform 3D scenes to 2D images.

[SIGGRAPH 2016 Technical paper, SketchMo] (not the above)

Here, we would like to draw a 3D motion line of “RightWrist” such as the below.

1. Collect the last 10 positions of RightWrist moving along the time.
2. Draw lines between these 10 positions every frame.

However, counting and collecting the positions are a bit bothersome with the array that we used (=static array: the number of the data is fixed).

Dynamic array can change the number of its data anytime.
ArrayList<> is one of the data type of dynamic arrays.
The below is the minimum of how to use this.

ArrayList<PVector> mLines = new ArrayList<PVector>();//define and new

PVector tem =new PVector(x, y, z); 
mLines.add(tem);//add tem on the back of the array.

mLines.remove(0);//0th data is removed
---
mLine.size()//The current size (number) of data.

[Reference of ArrayList]

Ex8: Draw a motion line.

• (Advanced) Draw motion lines of multiple joints.
• (Additional) Try anything that will improve the appearance of the motion lines such as the color, stroke weight, and so on.

//Hint to draw 
if (mLines.size()>=2) {   
    noFill();
    stroke(255);
    strokeWeight(5);
    beginShape();    
    for (int i=0 ; i < mLines.size(); i++) {
      PVector tem = mLines.get(i);
      vertex(tem.x, tem.y, tem.z);
      //curveVertex(tem.x, tem.y, tem.z);//Try.
    }
    endShape();
    strokeWeight(1);    
  }

The result of changing the stroke weight.

---

References

• Workshop/perfume_and_bvh by Assoc. Prof. Baba
• Perfume global site project #001 by anno lab
• Perfume Global Site Project #001 (Anno Perfume 養成ギプス) メイキング

全部芸工の先輩だ！

---

Homework1: Add the best exercises in this page to your web site

Homework2: Describe the final work in this class on your web site

• In the same page with the best exercises.
• Make a list of your ideas (more than four) to create a final work in this class. It is ok if it is an abstract idea.
• If you have more obvious plan, please describe it.

Homework3: Make a one minute presentation with your web page

• Until the next class.
• Appeal your Processing skills with your best exercises. (It would be evaluated as an achievement)
• Tell us what you would like to do for the final work. (It would be considered to make groups)
• Related words
  
  • fast forward: In SIGGRAPH, paper contributors make 30 sec presentation and briefly present their works.
    
    • [SIGGRAPH 2014 - Papers Fast Forward, Presenters Time Lapse]
    • [SIGGRAPH 2015 - Exhibits Fast Forward]
    • [Siggraph Asia 2012: Fast Forward]
  • Elevator pitch: a short summary used to quickly and simply define a process, product, service, organization, or event and its value proposition.

Contact

• Email: morimoto@design.kyushu-u.ac.jp
• Slack
  
  • コンピュータアニメーション演習: computeranimation16.slack.com
  • 視覚芸術基礎Ⅱ: bdva2.slack.com